Llama3.java
Llama3.java provides a way to run large language models (LLMs) locally and in pure Java and embedded in your Quarkus application. You can run various models such as LLama3, Mistral on your machine.
Prerequisites
To use Llama3.java it is necessary to run on Java 21 or later. This is because it utilizes the new Vector API for faster inference. Note that the Vector API is still a Java preview features, so it is required to explicitly enable it.
Since the Vector API are still a preview feature in Java 21, and up to the latest Java 23, it is necessary to enable it on the JVM by launching it with the following flags:
--enable-preview --enable-native-access=ALL-UNNAMED --add-modules jdk.incubator.vectoror equivalently to configure the quarkus-maven-plugin in the pom.xml file of your project as it follows:
<configuration>
<jvmArgs>--enable-preview --enable-native-access=ALL-UNNAMED</jvmArgs>
<modules>
<module>jdk.incubator.vector</module>
</modules>
</configuration>Dev Mode
Quarkus LangChain4j automatically handles the pulling of the models configured by the application, so there is no need for users to do so manually.
When running Quarkus in dev mode C2 compilation is not enabled and this can make Llama3.java excessively slow. This limitation will be fixed with Quarkus 3.17 when <forceC2>true</forceC2> is set.
|
Models are huge, so make sure you have enough disk space. Models location can be controlled using quarkus.langchain4j.llama3.models-path property.
|
| Due to model’s large size, pulling them can take time |
Native mode
Currently, Llama3.java only works in native mode with Early Access version’s of Oracle GraalVM 24 (which can be easily downloaded with SDKMan).
To achieve the best performance in native mode, it is suggested to configure the application with the following:
quarkus.native.additional-build-args=-O3,-march=nativeUsing Llama3.java
To let Llama3.java running inference on your models, add the following dependency into your project:
<dependency>
<groupId>io.quarkiverse.langchain4j</groupId>
<artifactId>quarkus-langchain4j-llama3-java</artifactId>
<version>0.23.0.CR1</version>
</dependency>If no other LLM extension is installed, AI Services will automatically utilize the configured Llama3.java model.
By default, the extension uses as model mukel/Llama-3.2-1B-Instruct-GGUF.
You can change it by setting the quarkus.langchain4j.llama3.chat-model.model-name property in the application.properties file:
quarkus.langchain4j.llama3.chat-model.model-name=mukel/Llama-3.2-3B-Instruct-GGUFConfiguration
Several configuration properties are available:
Configuration property fixed at build time - All other configuration properties are overridable at runtime
Configuration property |
Type |
Default |
|---|---|---|
Determines whether the necessary Jlama models are downloaded and included in the jar at build time. Currently, this option is only valid for Environment variable: |
boolean |
|
Whether the model should be enabled Environment variable: |
boolean |
|
Model name to use Environment variable: |
string |
|
Quantization of the model to use Environment variable: |
string |
|
Llama3.java supports AOT model preloading, enabling 0-overhead, instant inference, with minimal TTFT (time-to-first-token). A specialized, larger binary will be generated, with no parsing overhead for that particular model. It can still run other models, although incurring the usual parsing overhead. Environment variable: |
boolean |
|
Location on the file-system which serves as a cache for the models Environment variable: |
path |
|
Temperature in [0,inf] Environment variable: |
double |
|
Number of steps to run for < 0 = limited by context length Environment variable: |
int |
|
Whether to enable the integration. Set to Environment variable: |
boolean |
|
Whether Jlama should log requests Environment variable: |
boolean |
|
Whether Jlama client should log responses Environment variable: |
boolean |
|
Type |
Default |
|
Model name to use Environment variable: |
string |
|
Quantization of the model to use Environment variable: |
string |
|
Llama3.java supports AOT model preloading, enabling 0-overhead, instant inference, with minimal TTFT (time-to-first-token). A specialized, larger binary will be generated, with no parsing overhead for that particular model. It can still run other models, although incurring the usual parsing overhead. Environment variable: |
boolean |
|
Temperature in [0,inf] Environment variable: |
double |
|
Number of steps to run for < 0 = limited by context length Environment variable: |
int |
|
Whether to enable the integration. Set to Environment variable: |
boolean |
|
Whether Jlama should log requests Environment variable: |
boolean |
|
Whether Jlama client should log responses Environment variable: |
boolean |
|