How it works

Goblin operates at three boundaries:

  • the incoming HTTP boundary, through a JAX-RS ContainerRequestFilter / ContainerResponseFilter (the HTTP_IN layer);

  • the outgoing HTTP boundary, through a ClientRequestFilter (outgoing REST Client calls) and, for Vert.x WebClient calls, an opt-in request interceptor attached in application code (the HTTP_OUT layer);

  • the service boundary, through a CDI interceptor woven at build time on eligible application bean methods (the SERVICE layer).

For the HTTP boundaries:

  • For outbound MicroProfile REST Client calls the ClientRequestFilter is registered as a global provider at build time and intercepts every call automatically.

  • For Vert.x WebClient calls the application arms the client once with GoblinWebClient.enable(webClient); the interceptor is then attached through Vert.x’s internal WebClientInternal mechanism.

    1. At build time, the deployment module registers the client filter, the service interceptor binding and the goblin feature with Quarkus; the server-side JAX-RS filter is a @Provider discovered by Quarkus REST.

    2. At runtime startup, the AssaultEngine is initialized from the configuration in application.properties. In dev mode it first restores any persisted state file (.goblin-state.json) and registers a change listener so that every subsequent config mutation automatically persists the state to disk.

    3. For every incoming HTTP request, the filter checks whether chaos is active, whether this request should be affected (based on the level percentage), and which layer the request resolves to (see Chaos layers). The resolved layer is carried on the request stack for the whole processing chain.

    4. If the HTTP_IN layer won, all enabled assault types are applied in order: latency first (delay), then exception/HTTP-status/dependency-degradation (which abort the request). The response body and response header assaults are applied on the way out, in the ContainerResponseFilter, rewriting the entity and its headers before they reach the client (header rules run after the body transformation, so an explicit header change always wins).

    5. An assault record is added to the history buffer (capped at 1000 entries).

The server-side request filter runs once the request is matched to a resource method, before any endpoint logic executes. This means:

  • Exception and HTTP_STATUS assaults completely bypass your endpoint code — no side effects, no partial state mutations.

  • Latency assaults add delay before processing, simulating network or upstream slowness.

  • On a Vert.x event-loop thread (a non-blocking endpoint) the latency assault is skipped, with a WARN logged once, instead of blocking the loop; the other assaults still apply.

For every outbound call — MicroProfile REST Client or Vert.x WebClient — the HTTP_OUT layer, the same level gate and the client-side toggles are checked, then latency and/or exception are applied. The REST Client filter runs before the request is sent; the WebClient interceptor runs just after the request is prepared, before it is dispatched over the wire. This means:

  • Client-side exceptions abort the outbound call before it is dispatched — the downstream service never sees the request.

  • REST Client latency blocks the calling thread; when the call is made from an event-loop thread the latency is skipped (WARN logged once).

  • WebClient latency is applied without blocking the event loop: a timer is scheduled on the calling Vert.x context, with a blocking fallback only when no Vert.x context is active.

Chaos layers

Chaos is not tied to a single injection point: you pick which layers to arm, and every inbound request — or every consumed message — resolves to the deepest armed layer whose probability roll passes, so a fault rooted at the bottom propagates naturally through the layers above it. The layers, from deepest to shallowest:

  1. DATABASE — JDBC connection acquisition, through an Agroal pool interceptor (requires a JDBC datasource)

  2. MESSAGING — @Incoming consumer methods (requires quarkus-messaging)

  3. SERVICE — CDI interceptor on targeted beans

  4. HTTP_OUT — outbound REST Client / Vert.x WebClient calls (opt-in client toggles)

  5. HTTP_IN — inbound REST endpoints (default)

DATABASE and MESSAGING are only offered when the application has the matching extension; otherwise they are shown as unavailable in the Dev UI and never selected, so arming them falls through to the next available layer.

A layer only takes part when it is actionable: at least one of the assaults its hook can inject is enabled. The DATABASE, MESSAGING and SERVICE hooks only inject latency and exceptions, so with only the HTTP status assault enabled they are skipped and the request resolves to HTTP_IN.

There are two entry points, each resolving among its own candidate layers:

  • an inbound HTTP request resolves among DATABASE, SERVICE and HTTP_IN;

  • a consumed message resolves among DATABASE, MESSAGING and SERVICE.

Two layers ({HTTP_IN, HTTP_OUT}) are armed by default, which reproduces exactly the pre-layers behaviour. For every incoming request the engine walks the enabled layers from the deepest up, re-rolling nextInt(100) < level independently for each layer — the roll is never cached. The deepest available and actionable layer whose roll passes wins and becomes the armed layer for the whole request; the shallower layers pass through unimpeded, and the fault thrown at the bottom then propagates up through the SERVICE and HTTP_IN machinery. With level = 100 the deepest available and actionable layer always wins; with a lower level the distribution is biased toward the bottom.

  • HTTP_IN armed — the ContainerRequestFilter applies the server-side assaults exactly as before.

  • HTTP_OUT armed — the client filter / WebClient interceptor apply latency/exception on outgoing calls. HTTP_OUT is not part of the per-request resolution: every outgoing call rolls its own gate, independently of the layer the inbound request resolved to, so an outbound fault can add up with a SERVICE or HTTP_IN one.

  • SERVICE armed — the business bean interceptor fires (see below); the HTTP_IN filter stands down.

  • DATABASE armed — the next JDBC connection acquisitions of the request fail or are delayed (see below).

  • MESSAGING armed — the consumer invocation fails or is delayed before the consumer body runs (see below).

The decision is bound to the thread and to the CDI request context that made it: a decision read from another request context is discarded, so a worker thread left with a stale decision (an exception escaped a resource before the response filter could clear it) never assaults an unrelated message consumer or scheduled job.

Service-layer assaults

The SERVICE layer injects faults inside an application bean, on the CDI invocation itself, before your method runs. At build time a binding is woven onto eligible bean methods; at runtime a CDI interceptor (GoblinServiceInterceptor, @Priority(4100)) applies the configured latency and exception assaults when the request resolved to the SERVICE layer.

  • Placement inside Fault Tolerance. Jakarta interceptors invoke lower priorities first, so 4100 sits inside the MicroProfile Fault Tolerance interceptor (4010): a thrown exception is observed by @Retry, @CircuitBreaker and @Fallback, and the injected latency is covered by @Timeout and @Bulkhead.

  • Eligibility. A bean method is decorated only when the class belongs to the application root archive and the usual goblin.target rules pass (include-packages / exclude-packages / exclude-annotations). JAX-RS resources (@Path), @Provider`s, interfaces, static and private methods are never decorated — HTTP endpoints belong to the HTTP_IN layer and sit above the guarded beans (a `@Path declared on an implemented interface counts too). Methods named as the fallbackMethod of a @Fallback (on a method or on the class) in the same class, and FallbackHandler implementations, are skipped too, so a fallback can answer the original failure instead of being assaulted itself. The service-assault binding is only woven in dev and test builds, never in a production build.

  • One fault per call chain. Only the outermost intercepted call of a request is assaulted: when a decorated bean calls another decorated bean, the inner call runs untouched. Each further outermost call of the same request (a @Retry attempt, or a second bean called by the resource) draws level again.

  • Synchronous scope (for now). The decision is bound to the request thread, so @Asynchronous / non-blocking methods are not covered by Phase 1. On a Vert.x event-loop thread latency assaults are skipped (WARN logged once) instead of blocking the loop.

  • History. Each service assault is recorded like any other, with a fully qualified com.example.ClassName.method descriptor — including the actually applied latency duration for the latency assault. Since the rule falls inside FT, a @Retry-guarded method produces one record per attempt.

Database-layer assaults

The DATABASE layer injects faults where a slow or unreachable database really surfaces: when a JDBC connection is acquired from the Agroal pool. It therefore sits below Hibernate ORM, Panache and plain JDBC code alike.

  • Hook. One Agroal pool interceptor is registered per JDBC datasource (default and named) at build time, only when quarkus-agroal is present. Hibernate Reactive and the reactive SQL clients do not use Agroal and are not covered.

  • Granularity. Agroal notifies an acquisition once per getConnection() outside a transaction, and once per transaction inside one. The first acquisition of a request uses the per-request decision; every further acquisition (typically a @Retry attempt opening a new transaction) draws level again.

  • Faults. latency delays the acquisition; exception throws the configured exception out of getConnection() (the connection is released back to the pool). Fault Tolerance annotations on the repository or service above observe it, exactly as they would observe a real connection failure.

  • History. Records are labelled Database <datasource> connection, e.g. Database <default> connection, and tagged source=database in metrics and traces.

  • Scope. Only database access made within an inbound HTTP request or a consumed message is assaulted: a startup task or a scheduled job has no resolved layer.

Messaging-layer assaults

The MESSAGING layer targets the consumer side of Quarkus Messaging: the @Incoming methods of the application.

  • Entry point. A consumed message has no inbound HTTP request, so each consumer invocation is its own pseudo-request: it resolves among DATABASE, MESSAGING and SERVICE, and the decision stays in scope for the whole consumer call — a DATABASE or SERVICE fault fires inside message processing exactly as inside a REST call.

  • Placement outside Fault Tolerance. The interceptor runs at @Priority(4005), outside the MicroProfile Fault Tolerance interceptor (4010): a messaging-layer fault stands for the delivery of the message failing, so it is handled by the messaging failure strategy (nack, dead-letter queue, failure-strategy=ignore…​) rather than by a @Retry on the consumer. A SERVICE fault on the same consumer still runs inside Fault Tolerance.

  • Eligibility. The binding is woven at build time on the @Incoming methods of the application root archive, honouring the same goblin.target rules as the service layer, only when quarkus-messaging is present.

  • Synchronous scope. For a consumer returning Uni / CompletionStage only the synchronous part of the call is covered. Latency needs a blocking consumer (@Blocking, @RunOnVirtualThread): on an event-loop thread it is skipped rather than blocking the loop.

  • History. Records are labelled Messaging <class>.<method> and tagged source=messaging.

Runtime config modification

All assault parameters can be modified at runtime through the Dev UI or via JSON-RPC. The initial config is loaded from application.properties at startup, then held in a mutable in-memory config (MutableAssaultConfig). This means you can:

  • Enable latency AND exception simultaneously to simulate a slow failure.

  • Toggle assault types on/off without restarting.

  • Change the latency range, exception class, or HTTP status code on the fly.

  • Drop the target level from 100% to 10% to test intermittent failures.

Changes take effect on the next request — no restart, no redeployment. A WARN log confirms every change in the console.

State persistence

Runtime config changes made through the Dev UI are automatically persisted to a .goblin-state.json file in the project working directory. On the next startup, Goblin checks for this file and restores the previous configuration instead of starting fresh from application.properties.

This means your chaos scenario survives restarts — no need to reconfigure the dashboard every time you restart in dev mode.

The persisted state only covers the assault configuration (profile, toggles, latency range, exception settings, HTTP status settings, response body mode/percentage, response header rules, armed layers, target level). The global enabled/active flag is deliberately excluded: it is always taken from quarkus.goblin.enabled at startup and, during a session, from the Dev UI toggle — a state file can never reactivate an extension you disabled via quarkus.goblin.enabled=false. A pending Dev UI auto-off is not stored in the file either: it survives a live reload of the same dev-mode process, not a new start.

State persistence is dev-mode only: the change listener is only registered in dev mode, and the state file is only read on a dev-mode startup. Chaos itself only activates in dev mode, and in test mode on opt-in (quarkus.goblin.test.enabled=true) — a packaged production application always starts with the engine inactive. Integration or system tests always start from application.properties and never read nor overwrite the chaos configuration you saved from the Dev UI.

Add .goblin-state.json to your .gitignore to avoid committing local chaos state to version control.

If the state file is corrupted or unreadable, Goblin logs a warning and falls back to the configuration from application.properties.

Observability: the AssaultObserver SPI

Every recorded assault and every active-state change is broadcast to the registered AssaultObserver beans (io.quarkiverse.goblin.AssaultObserver, default no-op methods). Observers run synchronously on the request path: the engine calls each bean and skips any failing observer (the failure is only logged at DEBUG) so observability can never break an assault. The engine injects Instance<AssaultObserver>, so any @ApplicationScoped implementation is picked up automatically; outside the CDI container (plain unit tests) the notification simply does nothing.

The optional quarkus-goblin-metrics and quarkus-goblin-opentelemetry modules are the current consumers (see Metrics (Micrometer / Prometheus) and Tracing (OpenTelemetry)); post-assault assertions will plug into the same hook.

Adding your own assault type

Goblin ships with six built-in assault types (latency, exception, HTTP status, dependency degradation, response body and response headers) and lets you register your own. Implement the io.quarkiverse.goblin.assault.Assault interface, annotate the class @ApplicationScoped, and it is discovered automatically at build time — no other wiring required:

  • type() — the AssaultType this assault represents. AssaultType is a closed enum: a custom assault reuses one of the existing constants. Note that the per-request gate only opens when at least one of the built-in toggles is enabled, so a custom assault must be paired with (or guarded by) one of them.

  • isEnabled(config) — whether the assault should run for the current mutable configuration.

  • recordLabel() — the label used in the assault history and the Markdown report.

  • order() — the execution position in the chain. Latency (order 10) runs first; assaults that abort the request run afterwards.

  • apply(context) — perform the assault. Return AssaultOutcome.CONTINUE to let the next enabled type run, or ABORTED to stop processing. To short-circuit the request call context.getRequestContext().abortWith(…​); to throw a simulated failure, throw a RuntimeException. The engine does not record anything on your behalf: call context.getEngine().recordAssault(context.getMethodName(), recordLabel()) when the assault fires, so it shows up in the history, the counters, the metrics and the traces.

At build time Goblin scans the application index for implementors of Assault and registers them as beans, so a custom type provided by your application is picked up automatically. The filter resolves the ordered list of assaults for every request and applies the enabled ones in order() sequence.