Using Quarkus Fault Tolerance with AI Services

Quarkus Fault Tolerance helps you make your AI service interactions more robust and resilient to failure. This is particularly useful when working with external model providers that may be unavailable or experience latency.

To enable fault tolerance for AI services, follow these steps:

Add the quarkus-smallrye-fault-tolerance extension to your project:

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-smallrye-fault-tolerance</artifactId>
</dependency>

Use fault tolerance annotations such as @Fallback, @Timeout, or @Retry on your AI service methods:

package io.quarkiverse.langchain4j.samples;

import java.time.temporal.ChronoUnit;

import org.eclipse.microprofile.faulttolerance.Fallback;
import org.eclipse.microprofile.faulttolerance.Timeout;

import dev.langchain4j.service.SystemMessage;
import dev.langchain4j.service.UserMessage;
import io.quarkiverse.langchain4j.RegisterAiService;

@RegisterAiService
public interface AiServiceWithFaultTolerance {

    @SystemMessage("You are a professional poet")
    @UserMessage("Write a poem about {topic}. The poem should be {lines} lines long.")
    @Timeout(value = 60, unit = ChronoUnit.SECONDS)
    @Fallback(fallbackMethod = "fallback")
    String writeAPoem(String topic, int lines);

    default String fallback(String topic, int lines) {
        return "I'm sorry, I can't write a poem about " + topic;
    }
}

Understanding the Annotations

@Fallback: provides an alternative response if the AI interaction fails
@Timeout: limits how long the LLM call is allowed to run
@Retry: automatically retries the call in case of transient failure

Use @Timeout carefully when using tools or multi-step function calls, as those may require more time to complete due to multiple background interactions with your application’s tools or services.

Example: Handling Disabled Integrations

If model integrations are disabled (e.g. via quarkus.langchain4j.openai.enable-integration=false), calling the AI service will throw a ModelDisabledException. You can use @Fallback to gracefully recover:

package io.quarkiverse.langchain4j.samples;

import org.eclipse.microprofile.faulttolerance.Fallback;

import dev.langchain4j.model.ModelDisabledException;
import dev.langchain4j.service.SystemMessage;
import dev.langchain4j.service.UserMessage;
import io.quarkiverse.langchain4j.RegisterAiService;

@RegisterAiService
public interface AiServiceWithFaultToleranceOnlyOnDisabledIntegration {
    @SystemMessage("You are a professional poet")
    @UserMessage("Write a poem about {topic}. The poem should be {lines} lines long.")
    @Fallback(fallbackMethod = "fallback", applyOn = ModelDisabledException.class)
    String writeAPoem(String topic, int lines);

    // In this case, the fallback is only applied if a ModelDisabledException is thrown
    default String fallback(String topic, int lines) {
        return "I'm sorry, I can't write a poem about " + topic;
    }
}

Going Further