End-to-end example: prove your resilience annotations work

This walkthrough uses the sample application of the integration-tests module to verify, for real, that MicroProfile Fault Tolerance annotations react to failures: a @Timeout on a slow business method, then a @Retry + @Fallback on a failing database.

A fault injected at the inbound REST boundary (HTTP_IN) hits the request before the resource runs, so the Fault Tolerance annotations below it never see it. Both scenarios therefore arm a deeper layer — SERVICE or DATABASE — whose faults are injected inside or below the guarded methods (see Chaos layers).

Start the sample application

./mvnw -pl integration-tests quarkus:dev

Open the Dev UI at http://localhost:8080/q/dev and the Goblin Chaos Dashboard. The sample application has an in-memory H2 datasource, so every layer is available.

Scenario 1: a slow service triggers @Timeout

SampleService.timed() is guarded by @Timeout(400ms); GET /api/service/timeout calls it.

  1. In Chaos layers, check Service and make sure Database is unchecked (the default Inbound REST and Outbound HTTP can stay checked). At level 100 the deepest armed layer wins the whole request: with Database armed, this request — which never touches the database — would resolve to the database layer and not be assaulted at all.

  2. Enable Latency and set the range to 600-700 ms, above the 400 ms timeout. Keep the level at 100.

  3. Call the endpoint:

    curl -w "\nstatus=%{http_code} time=%{time_total}s\n" http://localhost:8080/api/service/timeout

  4. The call fails with a 500 after about 0.4 s instead of 0.6 s: @Timeout interrupted the injected delay. The Assault History shows a latency entry on io.quarkiverse.goblin.it.SampleService.timed whose duration is the delay actually endured (about 400 ms), and the status bar counts it under service.

Scenario 2: a failing database triggers @Retry then @Fallback

SampleService.databasePingWithRetry() is guarded by @Retry(maxRetries = 2) and @Fallback; it calls SampleRepository, which acquires a JDBC connection. GET /api/db/retry calls it.

  1. In Chaos layers, check Database.

  2. Disable Latency and enable Exception (the default java.lang.RuntimeException is fine). Keep the level at 100.

  3. Call the endpoint:

    curl -w "\nstatus=%{http_code}\n" http://localhost:8080/api/db/retry

  4. The response is 200 with database fallback reply. The Assault History shows three exception entries on Database <default> connection: the first attempt and the two retries each acquired a connection and failed, then the fallback answered; the status bar counts them under database. Compare with GET /api/db/ping, which has no Fault Tolerance annotation and fails with a 500.

Make it intermittent

Set the level to 30 and call the endpoints repeatedly: some attempts now succeed, so @Retry recovers before the fallback, and only part of the calls hit the timeout — the intermittent, non-deterministic failures that are the hardest to get right in production.

The same scenarios can run in a @QuarkusTest: set quarkus.goblin.test.enabled=true, arm the layer and the assault through the injected AssaultEngine (engine.getMutableConfig()), and assert on the response and on engine.getHistory(). See GoblinServiceLayerIntegrationTest and GoblinDatabaseLayerIntegrationTest in the integration-tests module.