Jinfer (Qxotic) Models

Qxotic provides jinfer, an engine that runs Large Language Models (LLMs) locally and purely in Java, embedded in your Quarkus application. This extension integrates jinfer’s LangChain4j support, exposing chat, streaming chat, embedding, scoring (rerank), audio transcription, and text-to-speech models backed by in-process GGUF inference.

Prerequisites

Java 25

jinfer requires Java 25 or later. The extension is only built and active on JDK 25+.

Vector API and native access

jinfer uses the Java Vector API (incubator) and the Foreign Function & Memory API. The required JVM flags are:

--add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED

  • In dev mode and continuous testing, the extension passes these flags automatically.

  • In a production jar, --enable-native-access=ALL-UNNAMED is written into the runnable jar manifest automatically, but --add-modules jdk.incubator.vector cannot be expressed in a manifest (a JDK limitation). You must pass it yourself:

    java --add-modules jdk.incubator.vector -jar quarkus-run.jar

    or via the JAVA_OPTS_APPEND environment variable (honored by the Quarkus container images). The jdk.incubator.vector module is required: if it is absent, creating a model fails fast at startup with a clear error (jinfer needs the Vector API: start the JVM with --add-modules jdk.incubator.vector).

Configuration

Models are selected with a HuggingFace/ModelScope GGUF reference (for example owner/repo:Q8_0) via the model property, or a local file via model-path:

quarkus.langchain4j.jinfer.chat-model.model=LiquidAI/LFM2.5-8B-A1B-GGUF:Q8_0
quarkus.langchain4j.jinfer.embedding-model.model=Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0

Models are downloaded by jinfer into its own resumable, sha256-verified cache on first use. Set the JINFER_OFFLINE=1 environment variable to forbid network access.

Model files are large (often several GB) and may take time to download on first use.

Set quarkus.langchain4j.jinfer.enable-integration=false to disable the models (they then throw when used), which is useful for tests.

Configuration property fixed at build time - All other configuration properties are overridable at runtime

Configuration property

Type

Default

quarkus.langchain4j.jinfer.chat-model.enabled

Whether the chat model should be enabled.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_CHAT_MODEL_ENABLED

boolean

true

quarkus.langchain4j.jinfer.embedding-model.enabled

Whether the embedding model should be enabled.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_EMBEDDING_MODEL_ENABLED

boolean

true

quarkus.langchain4j.jinfer.scoring-model.enabled

Whether the scoring (rerank) model should be enabled.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_SCORING_MODEL_ENABLED

boolean

true

quarkus.langchain4j.jinfer.transcription-model.enabled

Whether the audio transcription model should be enabled.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_TRANSCRIPTION_MODEL_ENABLED

boolean

true

quarkus.langchain4j.jinfer.tts-model.enabled

Whether the text-to-speech model should be enabled.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_TTS_MODEL_ENABLED

boolean

true

quarkus.langchain4j.jinfer.enable-integration

Whether to enable the integration. Defaults to true, which means requests are made to jinfer. Set to false to disable all models (they will throw when used).

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_ENABLE_INTEGRATION

boolean

true

quarkus.langchain4j.jinfer.chat-model.model

The HuggingFace/ModelScope GGUF model reference, e.g. owner/repo:Q8_0. Required unless model-path() is set.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_CHAT_MODEL_MODEL

string

quarkus.langchain4j.jinfer.chat-model.model-path

Path to a local GGUF file. Takes precedence over model(); when set, no download occurs.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_CHAT_MODEL_MODEL_PATH

path

quarkus.langchain4j.jinfer.chat-model.context-capacity

The context window size in tokens. 0 uses the model’s full context.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_CHAT_MODEL_CONTEXT_CAPACITY

int

quarkus.langchain4j.jinfer.chat-model.temperature

The sampling temperature.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_CHAT_MODEL_TEMPERATURE

double

${quarkus.langchain4j.temperature:0.7}

quarkus.langchain4j.jinfer.chat-model.top-p

The nucleus sampling top-p value.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_CHAT_MODEL_TOP_P

double

quarkus.langchain4j.jinfer.chat-model.top-k

The top-k sampling value.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_CHAT_MODEL_TOP_K

int

quarkus.langchain4j.jinfer.chat-model.min-p

The min-p sampling value.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_CHAT_MODEL_MIN_P

double

quarkus.langchain4j.jinfer.chat-model.max-output-tokens

The maximum number of tokens to generate.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_CHAT_MODEL_MAX_OUTPUT_TOKENS

int

quarkus.langchain4j.jinfer.chat-model.max-reasoning-tokens

The maximum number of reasoning tokens. -1 uncaps reasoning.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_CHAT_MODEL_MAX_REASONING_TOKENS

int

quarkus.langchain4j.jinfer.chat-model.seed

The RNG seed for sampling.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_CHAT_MODEL_SEED

long

quarkus.langchain4j.jinfer.chat-model.prompt-cache

Path to a read-only cached-prompt artifact to mount.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_CHAT_MODEL_PROMPT_CACHE

path

quarkus.langchain4j.jinfer.chat-model.thinking

Whether to enable the model’s "thinking"/reasoning mode.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_CHAT_MODEL_THINKING

boolean

quarkus.langchain4j.jinfer.embedding-model.model

The HuggingFace/ModelScope GGUF model reference, e.g. owner/repo:Q8_0. Required unless model-path() is set.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_EMBEDDING_MODEL_MODEL

string

quarkus.langchain4j.jinfer.embedding-model.model-path

Path to a local GGUF file. Takes precedence over model(); when set, no download occurs.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_EMBEDDING_MODEL_MODEL_PATH

path

quarkus.langchain4j.jinfer.embedding-model.context-capacity

The context window size in tokens.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_EMBEDDING_MODEL_CONTEXT_CAPACITY

int

quarkus.langchain4j.jinfer.scoring-model.model

The HuggingFace/ModelScope GGUF model reference, e.g. owner/repo:Q8_0. Required unless model-path() is set.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_SCORING_MODEL_MODEL

string

quarkus.langchain4j.jinfer.scoring-model.model-path

Path to a local GGUF file. Takes precedence over model(); when set, no download occurs.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_SCORING_MODEL_MODEL_PATH

path

quarkus.langchain4j.jinfer.scoring-model.context-capacity

The context window size in tokens.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_SCORING_MODEL_CONTEXT_CAPACITY

int

quarkus.langchain4j.jinfer.scoring-model.instruction

An optional reranker instruction. Rejected if the reranker has no instruction slot.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_SCORING_MODEL_INSTRUCTION

string

quarkus.langchain4j.jinfer.transcription-model.model

The HuggingFace/ModelScope GGUF model reference. Required unless model-path() is set.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_TRANSCRIPTION_MODEL_MODEL

string

quarkus.langchain4j.jinfer.transcription-model.model-path

Path to a local GGUF file. Takes precedence over model(); when set, no download occurs.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_TRANSCRIPTION_MODEL_MODEL_PATH

path

quarkus.langchain4j.jinfer.tts-model.model

The HuggingFace/ModelScope GGUF model reference. Required unless model-path() is set.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_TTS_MODEL_MODEL

string

quarkus.langchain4j.jinfer.tts-model.model-path

Path to a local GGUF file. Takes precedence over model(); when set, no download occurs.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_TTS_MODEL_MODEL_PATH

path

quarkus.langchain4j.jinfer.tts-model.speed

The speech speed multiplier (must be greater than 0).

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_TTS_MODEL_SPEED

double

quarkus.langchain4j.jinfer.tts-model.max-input-chars

The maximum number of input characters.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER_TTS_MODEL_MAX_INPUT_CHARS

int

Named model config

Type

Default

quarkus.langchain4j.jinfer."model-name".enable-integration

Whether to enable the integration. Defaults to true, which means requests are made to jinfer. Set to false to disable all models (they will throw when used).

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__ENABLE_INTEGRATION

boolean

true

quarkus.langchain4j.jinfer."model-name".chat-model.model

The HuggingFace/ModelScope GGUF model reference, e.g. owner/repo:Q8_0. Required unless model-path() is set.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__CHAT_MODEL_MODEL

string

quarkus.langchain4j.jinfer."model-name".chat-model.model-path

Path to a local GGUF file. Takes precedence over model(); when set, no download occurs.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__CHAT_MODEL_MODEL_PATH

path

quarkus.langchain4j.jinfer."model-name".chat-model.context-capacity

The context window size in tokens. 0 uses the model’s full context.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__CHAT_MODEL_CONTEXT_CAPACITY

int

quarkus.langchain4j.jinfer."model-name".chat-model.temperature

The sampling temperature.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__CHAT_MODEL_TEMPERATURE

double

${quarkus.langchain4j.temperature:0.7}

quarkus.langchain4j.jinfer."model-name".chat-model.top-p

The nucleus sampling top-p value.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__CHAT_MODEL_TOP_P

double

quarkus.langchain4j.jinfer."model-name".chat-model.top-k

The top-k sampling value.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__CHAT_MODEL_TOP_K

int

quarkus.langchain4j.jinfer."model-name".chat-model.min-p

The min-p sampling value.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__CHAT_MODEL_MIN_P

double

quarkus.langchain4j.jinfer."model-name".chat-model.max-output-tokens

The maximum number of tokens to generate.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__CHAT_MODEL_MAX_OUTPUT_TOKENS

int

quarkus.langchain4j.jinfer."model-name".chat-model.max-reasoning-tokens

The maximum number of reasoning tokens. -1 uncaps reasoning.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__CHAT_MODEL_MAX_REASONING_TOKENS

int

quarkus.langchain4j.jinfer."model-name".chat-model.seed

The RNG seed for sampling.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__CHAT_MODEL_SEED

long

quarkus.langchain4j.jinfer."model-name".chat-model.prompt-cache

Path to a read-only cached-prompt artifact to mount.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__CHAT_MODEL_PROMPT_CACHE

path

quarkus.langchain4j.jinfer."model-name".chat-model.thinking

Whether to enable the model’s "thinking"/reasoning mode.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__CHAT_MODEL_THINKING

boolean

quarkus.langchain4j.jinfer."model-name".embedding-model.model

The HuggingFace/ModelScope GGUF model reference, e.g. owner/repo:Q8_0. Required unless model-path() is set.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__EMBEDDING_MODEL_MODEL

string

quarkus.langchain4j.jinfer."model-name".embedding-model.model-path

Path to a local GGUF file. Takes precedence over model(); when set, no download occurs.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__EMBEDDING_MODEL_MODEL_PATH

path

quarkus.langchain4j.jinfer."model-name".embedding-model.context-capacity

The context window size in tokens.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__EMBEDDING_MODEL_CONTEXT_CAPACITY

int

quarkus.langchain4j.jinfer."model-name".scoring-model.model

The HuggingFace/ModelScope GGUF model reference, e.g. owner/repo:Q8_0. Required unless model-path() is set.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__SCORING_MODEL_MODEL

string

quarkus.langchain4j.jinfer."model-name".scoring-model.model-path

Path to a local GGUF file. Takes precedence over model(); when set, no download occurs.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__SCORING_MODEL_MODEL_PATH

path

quarkus.langchain4j.jinfer."model-name".scoring-model.context-capacity

The context window size in tokens.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__SCORING_MODEL_CONTEXT_CAPACITY

int

quarkus.langchain4j.jinfer."model-name".scoring-model.instruction

An optional reranker instruction. Rejected if the reranker has no instruction slot.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__SCORING_MODEL_INSTRUCTION

string

quarkus.langchain4j.jinfer."model-name".transcription-model.model

The HuggingFace/ModelScope GGUF model reference. Required unless model-path() is set.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__TRANSCRIPTION_MODEL_MODEL

string

quarkus.langchain4j.jinfer."model-name".transcription-model.model-path

Path to a local GGUF file. Takes precedence over model(); when set, no download occurs.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__TRANSCRIPTION_MODEL_MODEL_PATH

path

quarkus.langchain4j.jinfer."model-name".tts-model.model

The HuggingFace/ModelScope GGUF model reference. Required unless model-path() is set.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__TTS_MODEL_MODEL

string

quarkus.langchain4j.jinfer."model-name".tts-model.model-path

Path to a local GGUF file. Takes precedence over model(); when set, no download occurs.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__TTS_MODEL_MODEL_PATH

path

quarkus.langchain4j.jinfer."model-name".tts-model.speed

The speech speed multiplier (must be greater than 0).

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__TTS_MODEL_SPEED

double

quarkus.langchain4j.jinfer."model-name".tts-model.max-input-chars

The maximum number of input characters.

Environment variable: QUARKUS_LANGCHAIN4J_JINFER__MODEL_NAME__TTS_MODEL_MAX_INPUT_CHARS

int