IBM watsonx.ai
You can develop generative AI solutions with foundation models in IBM watsonx.ai. You can use prompts to generate, classify, summarize, or extract content from your input text. Choose from IBM models or open source models from Hugging Face. You can tune foundation models to customize your prompt output or optimize inferencing performance.
Supported only for IBM watsonx as a service on IBM Cloud. |
Using watsonx.ai
To employ watsonx.ai LLMs, integrate the following dependency into your project:
<dependency>
<groupId>io.quarkiverse.langchain4j</groupId>
<artifactId>quarkus-langchain4j-watsonx</artifactId>
<version>0.23.3</version>
</dependency>
If no other extension is installed, AI Services will automatically utilize the configured watsonx dependency.
Configuration
To use the watsonx.ai dependency, you must configure some required values in the application.properties
file.
Base URL
The base-url
property depends on the region of the provided service instance, use one of the following values:
-
Frankfurt: https://eu-de.ml.cloud.ibm.com
-
London: https://eu-gb.ml.cloud.ibm.com
-
Sydney: https://au-syd.ml.cloud.ibm.com
quarkus.langchain4j.watsonx.base-url=https://us-south.ml.cloud.ibm.com
Project ID
To prompt foundation models in watsonx.ai, you need to pass the identifier of a project.
To get the ID of a project, complete the following steps:
-
Go to https://dataplatform.cloud.ibm.com/projects/?context=wx.
-
Open the project, and then click the Manage tab.
-
Copy the project ID from the Details section of the General page.
quarkus.langchain4j.watsonx.project-id=23d...
If you like, you can use the Space ID property with quarkus.langchain4j.watsonx.space-id
|
Modes of Interaction
The watsonx.ai
module provides two modes for interacting with large language models: chat
and generation
. These modes allow you to tailor interactions based on the complexity of your use case and the level of control you require over the prompt structure.
You can select the interaction mode using the property quarkus.langchain4j.watsonx.mode
.
-
chat
: This mode abstracts the complexity of tagging by automatically formatting prompts (default value). -
generation
: In this mode, you must explicitly structure the prompts using the required model-specific tags. This provides full control over the format of the prompt but requires in-depth knowledge of the model being used. For best results, always refer to the documentation provided for each model to maximize the effectiveness of your prompts.
Each mode uses its own property namespace for customization:
-
For
chat
mode the properties are underquarkus.langchain4j.watsonx.chat-model
. -
For
generation
mode properties are underquarkus.langchain4j.watsonx.generation-model
.
Chat Mode Example
The following example demonstrates how to configure and use chat
mode:
quarkus.langchain4j.watsonx.base-url=${BASE_URL}
quarkus.langchain4j.watsonx.api-key=${API_KEY}
quarkus.langchain4j.watsonx.project-id=${PROJECT_ID}
quarkus.langchain4j.watsonx.chat-model.model-id=mistralai/mistral-large
quarkus.langchain4j.watsonx.mode=chat // You can omit this property, as 'chat' is the default mode.
@RegisterAiService
public interface AiService {
@SystemMessage("You are a helpful assistant")
public String chat(@MemoryId String id, @UserMessage message);
}
Generation Mode Example
The following example demonstrates how to configure and use generation
mode:
quarkus.langchain4j.watsonx.base-url=${BASE_URL}
quarkus.langchain4j.watsonx.api-key=${API_KEY}
quarkus.langchain4j.watsonx.project-id=${PROJECT_ID}
quarkus.langchain4j.watsonx.generation-model.model-id=mistralai/mistral-large
quarkus.langchain4j.watsonx.mode=generation
@RegisterAiService(chatMemoryProviderSupplier = RegisterAiService.NoChatMemoryProviderSupplier.class)
public interface AiService {
@SystemMessage("<s>[INST] You are a helpful assistant [/INST]</s>")
@UserMessage("[INST] What is the capital of {capital}? [/INST]")
public String askCapital(String capital);
}
The @SystemMessage and @UserMessage annotations are joined by default with no separator (an empty string "" ). If you want to change this behavior, use the property quarkus.langchain4j.watsonx.generation-model.prompt-joiner=<value> .
|
All configuration properties
Configuration property fixed at build time - All other configuration properties are overridable at runtime
Configuration property |
Type |
Default |
---|---|---|
Whether the model should be enabled. Environment variable: |
boolean |
|
Whether the embedding model should be enabled. Environment variable: |
boolean |
|
Whether the scoring model should be enabled. Environment variable: |
boolean |
|
Specifies the mode of interaction with the LLM model. This property allows you to choose between two modes of operation:
Environment variable: |
string |
|
Base URL of the watsonx.ai API. Environment variable: |
string |
|
IBM Cloud API key. To create a new API key, follow this link. Environment variable: |
string |
|
Timeout for watsonx.ai calls. Environment variable: |
|
|
The version date for the API of the form YYYY-MM-DD. Environment variable: |
string |
|
The space that contains the resource. Either Environment variable: |
string |
|
The space that contains the resource. Either To look up your project id, click here. Environment variable: |
string |
|
Whether the watsonx.ai client should log requests. Environment variable: |
boolean |
|
Whether the watsonx.ai client should log responses. Environment variable: |
boolean |
|
Whether to enable the integration. Defaults to Environment variable: |
boolean |
|
Base URL of the IAM Authentication API. Environment variable: |
||
Timeout for IAM authentication calls. Environment variable: |
|
|
Grant type for the IAM Authentication API. Environment variable: |
string |
|
Model id to use. To view the complete model list, click here. Environment variable: |
string |
|
Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim. Possible values: Environment variable: |
double |
|
Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message. Environment variable: |
boolean |
|
An integer specifying the number of most likely tokens to return at each token position, each with an associated log probability. The option Possible values: Environment variable: |
int |
|
The maximum number of tokens that can be generated in the chat completion. The total length of input tokens and generated tokens is limited by the model’s context length. Environment variable: |
int |
|
How many chat completion choices to generate for each input message. Note that you will be charged based on the number of generated tokens across all of the choices. Keep n as 1 to minimize costs Environment variable: |
int |
|
Positive values penalize new tokens based on whether they appear in the text so far, increasing the model’s likelihood to talk about new topics. Possible values: Environment variable: |
double |
|
What sampling temperature to use. Higher values like We generally recommend altering this or Possible values: Environment variable: |
double |
|
An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So We generally recommend altering this or Possible values: Environment variable: |
double |
|
Specifies the desired format for the model’s output. Allowable values: Applicable in modes: Environment variable: |
string |
|
Whether chat model requests should be logged. Environment variable: |
boolean |
|
Whether chat model responses should be logged. Environment variable: |
boolean |
|
Model id to use. To view the complete model list, click here. Environment variable: |
string |
|
Represents the strategy used for picking the tokens during generation of the output text. During text generation when parameter value is set to Allowable values: Environment variable: |
string |
|
Represents the factor of exponential decay. Larger values correspond to more aggressive decay. Possible values: Environment variable: |
double |
|
A number of generated tokens after which this should take effect. Possible values: Environment variable: |
int |
|
The maximum number of new tokens to be generated. The maximum supported value for this field depends on the model being used. How the "token" is defined depends on the tokenizer and vocabulary size, which in turn depends on the model. Often the tokens are a mix of full words and sub-words. Depending on the users plan, and on the model being used, there may be an enforced maximum number of new tokens. Possible values: Environment variable: |
int |
|
If stop sequences are given, they are ignored until minimum tokens are generated. Possible values: Environment variable: |
int |
|
Random number generator seed to use in sampling mode for experimental repeatability. Possible values: Environment variable: |
int |
|
Stop sequences are one or more strings which will cause the text generation to stop if/when they are produced as part of the output. Stop sequences encountered prior to the minimum number of tokens being generated will be ignored. Possible values: Environment variable: |
list of string |
|
A value used to modify the next-token probabilities in Possible values: Environment variable: |
double |
|
The number of highest probability vocabulary tokens to keep for top-k-filtering. Only applies for Possible values: Environment variable: |
int |
|
Similar to Possible values: Environment variable: |
double |
|
Represents the penalty for penalizing tokens that have already been generated or belong to the context. The value Possible values: Environment variable: |
double |
|
Represents the maximum number of input tokens accepted. This can be used to avoid requests failing due to input being longer than configured limits. If the text is truncated, then it truncates the start of the input (on the left), so the end of the input will remain the same. If this value exceeds the maximum sequence length (refer to the documentation to find this value for the model) then the call will fail if the total number of tokens exceeds the maximum sequence length. Zero means don’t truncate. Possible values: Environment variable: |
int |
|
Pass Environment variable: |
boolean |
|
Whether chat model requests should be logged. Environment variable: |
boolean |
|
Whether chat model responses should be logged. Environment variable: |
boolean |
|
Delimiter used to concatenate the ChatMessage elements into a single string. By setting this property, you can define your preferred way of concatenating messages to ensure that the prompt is structured in the correct way. Environment variable: |
string |
` ` |
Model id to use. To view the complete model list, click here. Environment variable: |
string |
|
Represents the maximum number of input tokens accepted. This can be used to avoid requests failing due to input being longer than configured limits. If the text is truncated, then it truncates the end of the input (on the right), so the start of the input will remain the same. If this value exceeds the maximum sequence length (refer to the documentation to find this value for the model) then the call will fail if the total number of tokens exceeds the maximum sequence length. Environment variable: |
int |
|
Whether embedding model requests should be logged. Environment variable: |
boolean |
|
Whether embedding model responses should be logged. Environment variable: |
boolean |
|
Model id to use. To view the complete model list, click here. Environment variable: |
string |
|
Represents the maximum number of input tokens accepted. This can be used to avoid requests failing due to input being longer than configured limits. If the text is truncated, then it truncates the end of the input (on the right), so the start of the input will remain the same. If this value exceeds the maximum sequence length (refer to the documentation to find this value for the model) then the call will fail if the total number of tokens exceeds the maximum sequence length. Environment variable: |
int |
|
Whether embedding model requests should be logged. Environment variable: |
boolean |
|
Whether embedding model responses should be logged. Environment variable: |
boolean |
|
Type |
Default |
|
Specifies the mode of interaction with the LLM model. This property allows you to choose between two modes of operation:
Environment variable: |
string |
|
Base URL of the watsonx.ai API. Environment variable: |
string |
|
IBM Cloud API key. To create a new API key, follow this link. Environment variable: |
string |
|
Timeout for watsonx.ai calls. Environment variable: |
|
|
The version date for the API of the form YYYY-MM-DD. Environment variable: |
string |
|
The space that contains the resource. Either Environment variable: |
string |
|
The space that contains the resource. Either To look up your project id, click here. Environment variable: |
string |
|
Whether the watsonx.ai client should log requests. Environment variable: |
boolean |
|
Whether the watsonx.ai client should log responses. Environment variable: |
boolean |
|
Whether to enable the integration. Defaults to Environment variable: |
boolean |
|
Base URL of the IAM Authentication API. Environment variable: |
||
Timeout for IAM authentication calls. Environment variable: |
|
|
Grant type for the IAM Authentication API. Environment variable: |
string |
|
Model id to use. To view the complete model list, click here. Environment variable: |
string |
|
Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim. Possible values: Environment variable: |
double |
|
Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message. Environment variable: |
boolean |
|
An integer specifying the number of most likely tokens to return at each token position, each with an associated log probability. The option Possible values: Environment variable: |
int |
|
The maximum number of tokens that can be generated in the chat completion. The total length of input tokens and generated tokens is limited by the model’s context length. Environment variable: |
int |
|
How many chat completion choices to generate for each input message. Note that you will be charged based on the number of generated tokens across all of the choices. Keep n as 1 to minimize costs Environment variable: |
int |
|
Positive values penalize new tokens based on whether they appear in the text so far, increasing the model’s likelihood to talk about new topics. Possible values: Environment variable: |
double |
|
What sampling temperature to use. Higher values like We generally recommend altering this or Possible values: Environment variable: |
double |
|
An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So We generally recommend altering this or Possible values: Environment variable: |
double |
|
Specifies the desired format for the model’s output. Allowable values: Applicable in modes: Environment variable: |
string |
|
Whether chat model requests should be logged. Environment variable: |
boolean |
|
Whether chat model responses should be logged. Environment variable: |
boolean |
|
Model id to use. To view the complete model list, click here. Environment variable: |
string |
|
Represents the strategy used for picking the tokens during generation of the output text. During text generation when parameter value is set to Allowable values: Environment variable: |
string |
|
Represents the factor of exponential decay. Larger values correspond to more aggressive decay. Possible values: Environment variable: |
double |
|
A number of generated tokens after which this should take effect. Possible values: Environment variable: |
int |
|
The maximum number of new tokens to be generated. The maximum supported value for this field depends on the model being used. How the "token" is defined depends on the tokenizer and vocabulary size, which in turn depends on the model. Often the tokens are a mix of full words and sub-words. Depending on the users plan, and on the model being used, there may be an enforced maximum number of new tokens. Possible values: Environment variable: |
int |
|
If stop sequences are given, they are ignored until minimum tokens are generated. Possible values: Environment variable: |
int |
|
Random number generator seed to use in sampling mode for experimental repeatability. Possible values: Environment variable: |
int |
|
Stop sequences are one or more strings which will cause the text generation to stop if/when they are produced as part of the output. Stop sequences encountered prior to the minimum number of tokens being generated will be ignored. Possible values: Environment variable: |
list of string |
|
A value used to modify the next-token probabilities in Possible values: Environment variable: |
double |
|
The number of highest probability vocabulary tokens to keep for top-k-filtering. Only applies for Possible values: Environment variable: |
int |
|
Similar to Possible values: Environment variable: |
double |
|
Represents the penalty for penalizing tokens that have already been generated or belong to the context. The value Possible values: Environment variable: |
double |
|
Represents the maximum number of input tokens accepted. This can be used to avoid requests failing due to input being longer than configured limits. If the text is truncated, then it truncates the start of the input (on the left), so the end of the input will remain the same. If this value exceeds the maximum sequence length (refer to the documentation to find this value for the model) then the call will fail if the total number of tokens exceeds the maximum sequence length. Zero means don’t truncate. Possible values: Environment variable: |
int |
|
Pass Environment variable: |
boolean |
|
Whether chat model requests should be logged. Environment variable: |
boolean |
|
Whether chat model responses should be logged. Environment variable: |
boolean |
|
Delimiter used to concatenate the ChatMessage elements into a single string. By setting this property, you can define your preferred way of concatenating messages to ensure that the prompt is structured in the correct way. Environment variable: |
string |
` ` |
Model id to use. To view the complete model list, click here. Environment variable: |
string |
|
Represents the maximum number of input tokens accepted. This can be used to avoid requests failing due to input being longer than configured limits. If the text is truncated, then it truncates the end of the input (on the right), so the start of the input will remain the same. If this value exceeds the maximum sequence length (refer to the documentation to find this value for the model) then the call will fail if the total number of tokens exceeds the maximum sequence length. Environment variable: |
int |
|
Whether embedding model requests should be logged. Environment variable: |
boolean |
|
Whether embedding model responses should be logged. Environment variable: |
boolean |
|
Model id to use. To view the complete model list, click here. Environment variable: |
string |
|
Represents the maximum number of input tokens accepted. This can be used to avoid requests failing due to input being longer than configured limits. If the text is truncated, then it truncates the end of the input (on the right), so the start of the input will remain the same. If this value exceeds the maximum sequence length (refer to the documentation to find this value for the model) then the call will fail if the total number of tokens exceeds the maximum sequence length. Environment variable: |
int |
|
Whether embedding model requests should be logged. Environment variable: |
boolean |
|
Whether embedding model responses should be logged. Environment variable: |
boolean |
|
About the Duration format
To write duration values, use the standard You can also use a simplified format, starting with a number:
In other cases, the simplified format is translated to the
|