Jlama

Jlama provides a way to run large language models (LLMs) locally and in pure Java and embedded in your Quarkus application. You can run many models such as LLama3, Mistral, Granite and many others on your machine.

Prerequisites

To use Jlama it is necessary to run on Java 21 or later. This is because it utilizes the new Vector API for faster inference. Note that the Vector API is still a Java preview features, so it is required to explicitly enable it.

Since the Vector API are still a preview feature in Java 21, and up to the latest Java 23, it is necessary to enable it on the JVM by launching it with the following flags:

--enable-preview --enable-native-access=ALL-UNNAMED --add-modules jdk.incubator.vector

Dev Mode

Quarkus LangChain4j automatically handles the pulling of the models configured by the application, so there is no need for users to do so manually. Furthermore, the extension properly configures the launch of Java process in order to ensure that the C2 compiler will be enabled (as without it, Jlama is virtually unusable).

Models are generally very large and can take time to download while also consuming a large chunk of disk space. Model location can be controlled using quarkus.langchain4j.jlama.models-path property.

Using Jlama

To let Jlama running inference on your models, add the following dependency into your project:

<dependency>
    <groupId>io.quarkiverse.langchain4j</groupId>
    <artifactId>quarkus-langchain4j-jlama</artifactId>
    <version>0.25.0.CR1</version>
</dependency>

If no other LLM extension is installed, AI Services will automatically utilize the configured Jlama model.

By default, the extension uses as model TinyLlama-1.1B-Chat-v1.0-Jlama-Q4. You can change it by setting the quarkus.langchain4j.jlama.chat-model.model-name property in the application.properties file:

quarkus.langchain4j.jlama.chat-model.model-name=tjake/granite-3.0-2b-instruct-JQ4

Configuration

Several configuration properties are available:

Configuration property fixed at build time - All other configuration properties are overridable at runtime

Configuration property

Type

Default

quarkus.langchain4j.jlama.include-models-in-artifact

Determines whether the necessary Jlama models are downloaded and included in the jar at build time. Currently, this option is only valid for fast-jar deployments.

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA_INCLUDE_MODELS_IN_ARTIFACT

boolean

true

quarkus.langchain4j.jlama.chat-model.enabled

Whether the model should be enabled

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA_CHAT_MODEL_ENABLED

boolean

true

quarkus.langchain4j.jlama.embedding-model.enabled

Whether the model should be enabled

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA_EMBEDDING_MODEL_ENABLED

boolean

true

quarkus.langchain4j.jlama.chat-model.model-name

Model name to use

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA_CHAT_MODEL_MODEL_NAME

string

tjake/TinyLlama-1.1B-Chat-v1.0-Jlama-Q4

quarkus.langchain4j.jlama.embedding-model.model-name

Model name to use

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA_EMBEDDING_MODEL_MODEL_NAME

string

intfloat/e5-small-v2

quarkus.langchain4j.jlama.models-path

Location on the file-system which serves as a cache for the models

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA_MODELS_PATH

path

${user.home}/.langchain4j/models

quarkus.langchain4j.jlama.chat-model.temperature

What sampling temperature to use, between 0.0 and 1.0. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.

It is generally recommended to set this or the top-k property but not both.

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA_CHAT_MODEL_TEMPERATURE

double

0.3f

quarkus.langchain4j.jlama.chat-model.max-tokens

The maximum number of tokens to generate in the completion.

The token count of your prompt plus max_tokens cannot exceed the model’s context length

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA_CHAT_MODEL_MAX_TOKENS

int

quarkus.langchain4j.jlama.enable-integration

Whether to enable the integration. Set to false to disable all requests.

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA_ENABLE_INTEGRATION

boolean

true

quarkus.langchain4j.jlama.log-requests

Whether Jlama should log requests

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA_LOG_REQUESTS

boolean

false

quarkus.langchain4j.jlama.log-responses

Whether Jlama client should log responses

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA_LOG_RESPONSES

boolean

false

Named model config

Type

Default

quarkus.langchain4j.jlama."model-name".chat-model.model-name

Model name to use

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA__MODEL_NAME__CHAT_MODEL_MODEL_NAME

string

tjake/TinyLlama-1.1B-Chat-v1.0-Jlama-Q4

quarkus.langchain4j.jlama."model-name".embedding-model.model-name

Model name to use

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA__MODEL_NAME__EMBEDDING_MODEL_MODEL_NAME

string

intfloat/e5-small-v2

quarkus.langchain4j.jlama."model-name".chat-model.temperature

What sampling temperature to use, between 0.0 and 1.0. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.

It is generally recommended to set this or the top-k property but not both.

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA__MODEL_NAME__CHAT_MODEL_TEMPERATURE

double

0.3f

quarkus.langchain4j.jlama."model-name".chat-model.max-tokens

The maximum number of tokens to generate in the completion.

The token count of your prompt plus max_tokens cannot exceed the model’s context length

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA__MODEL_NAME__CHAT_MODEL_MAX_TOKENS

int

quarkus.langchain4j.jlama."model-name".enable-integration

Whether to enable the integration. Set to false to disable all requests.

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA__MODEL_NAME__ENABLE_INTEGRATION

boolean

true

quarkus.langchain4j.jlama."model-name".log-requests

Whether Jlama should log requests

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA__MODEL_NAME__LOG_REQUESTS

boolean

false

quarkus.langchain4j.jlama."model-name".log-responses

Whether Jlama client should log responses

Environment variable: QUARKUS_LANGCHAIN4J_JLAMA__MODEL_NAME__LOG_RESPONSES

boolean

false

Document Retriever and Embedding

Jlama also provides embedding models. By default, it uses intfloat/e5-small-v2.

You can change the default embedding model by setting the quarkus.langchain4j.jlama.embedding-model.model-name property in the application.properties file:

quarkus.langchain4j.log-requests=true
quarkus.langchain4j.log-responses=true

quarkus.langchain4j.jlama.chat-model.model-name=tjake/granite-3.0-2b-instruct-JQ4
quarkus.langchain4j.jlama.embedding-model.model-name=intfloat/e5-small-v2

If no other LLM extension is installed, retrieve the embedding model as follows:

@Inject EmbeddingModel model; // Injects the embedding model