How it works

Goblin operates at three boundaries:

  • the incoming HTTP boundary, through a JAX-RS ContainerRequestFilter / ContainerResponseFilter (the HTTP_IN layer);

  • the outgoing HTTP boundary, through a ClientRequestFilter (outgoing REST Client calls) and, for Vert.x WebClient calls, an opt-in request interceptor attached in application code (the HTTP_OUT layer);

  • the service boundary, through a CDI interceptor woven at build time on eligible application bean methods (the SERVICE layer).

For the HTTP boundaries:

  • For outbound MicroProfile REST Client calls the ClientRequestFilter is registered as a global provider at build time and intercepts every call automatically.

  • For Vert.x WebClient calls the application arms the client once with GoblinWebClient.enable(webClient); the interceptor is then attached through Vert.x’s internal WebClientInternal mechanism.

    1. At build time, the deployment module registers the client filter, the service interceptor binding and the goblin feature with Quarkus; the server-side JAX-RS filter is a @Provider discovered by Quarkus REST.

    2. At runtime startup, the AssaultEngine is initialized from the configuration in application.properties. In dev mode it first restores any persisted state file (.goblin-state.json) and registers a change listener so that every subsequent config mutation automatically persists the state to disk.

    3. For every incoming HTTP request, the filter checks whether chaos is active, whether this request should be affected (based on the level percentage), and which layer the request resolves to (see Chaos layers). The resolved layer is carried on the request stack for the whole processing chain.

    4. If the HTTP_IN layer won, all enabled assault types are applied in order: latency first (delay), then exception/HTTP-status/dependency-degradation (which abort the request). The response body and response header assaults are applied on the way out, in the ContainerResponseFilter, rewriting the entity and its headers before they reach the client (header rules run after the body transformation, so an explicit header change always wins).

    5. An assault record is added to the history buffer (capped at 1000 entries).

The server-side request filter runs once the request is matched to a resource method, before any endpoint logic executes. This means:

  • Exception and HTTP_STATUS assaults completely bypass your endpoint code — no side effects, no partial state mutations.

  • Latency assaults add delay before processing, simulating network or upstream slowness.

  • On a Vert.x event-loop thread (a non-blocking endpoint) the latency assault is skipped, with a WARN logged once, instead of blocking the loop; the other assaults still apply.

For every outbound call — MicroProfile REST Client or Vert.x WebClient — the HTTP_OUT layer, the same level gate and the client-side toggles are checked, then latency and/or exception are applied. The REST Client filter runs before the request is sent; the WebClient interceptor runs just after the request is prepared, before it is dispatched over the wire. This means:

  • Client-side exceptions abort the outbound call before it is dispatched — the downstream service never sees the request.

  • REST Client latency blocks the calling thread; when the call is made from an event-loop thread the latency is skipped (WARN logged once).

  • WebClient latency is applied without blocking the event loop: a timer is scheduled on the calling Vert.x context, with a blocking fallback only when no Vert.x context is active.

Chaos layers

Chaos is not tied to a single injection point: you pick which layers to arm, and every inbound request — or every consumed message — resolves to the deepest armed layer whose probability roll passes, so a fault rooted at the bottom propagates naturally through the layers above it. The layers, from deepest to shallowest:

  1. DATABASE — JDBC connection acquisition, through an Agroal pool interceptor (requires a JDBC datasource)

  2. MESSAGING — @Incoming consumer methods (requires quarkus-messaging)

  3. SERVICE — CDI interceptor on targeted beans

  4. HTTP_OUT — outbound REST Client / Vert.x WebClient calls (opt-in client toggles)

  5. HTTP_IN — inbound REST endpoints (default)

DATABASE and MESSAGING are only offered when the application has the matching extension; otherwise they are shown as unavailable in the Dev UI and never selected, so arming them falls through to the next available layer.

A layer only takes part when it is actionable: at least one of the assaults its hook can inject is enabled. The DATABASE, MESSAGING and SERVICE hooks only inject latency and exceptions, so with only the HTTP status assault enabled they are skipped and the request resolves to HTTP_IN.

There are two entry points, each resolving among its own candidate layers:

  • an inbound HTTP request resolves among DATABASE, SERVICE and HTTP_IN;

  • a consumed message resolves among DATABASE, MESSAGING and SERVICE.

Two layers ({HTTP_IN, HTTP_OUT}) are armed by default, which reproduces exactly the pre-layers behaviour. For every incoming request the engine walks the enabled layers from the deepest up, re-rolling nextInt(100) < level independently for each layer — the roll is never cached. The deepest available and actionable layer whose roll passes wins and becomes the armed layer for the whole request; the shallower layers pass through unimpeded, and the fault thrown at the bottom then propagates up through the SERVICE and HTTP_IN machinery. With level = 100 the deepest available and actionable layer always wins; with a lower level the distribution is biased toward the bottom.

  • HTTP_IN armed — the ContainerRequestFilter applies the server-side assaults exactly as before.

  • HTTP_OUT armed — the client filter / WebClient interceptor apply latency/exception on outgoing calls. HTTP_OUT is not part of the per-request resolution: every outgoing call rolls its own gate, independently of the layer the inbound request resolved to, so an outbound fault can add up with a SERVICE or HTTP_IN one.

  • SERVICE armed — the business bean interceptor fires (see below); the HTTP_IN filter stands down.

  • DATABASE armed — the next JDBC connection acquisitions of the request fail or are delayed (see below).

  • MESSAGING armed — the consumer invocation fails or is delayed before the consumer body runs (see below).

The decision is bound to the thread and to the CDI request context that made it: a decision read from another request context is discarded, so a worker thread left with a stale decision (an exception escaped a resource before the response filter could clear it) never assaults an unrelated message consumer or scheduled job.

Service-layer assaults

The SERVICE layer injects faults inside an application bean, on the CDI invocation itself, before your method runs. At build time a binding is woven onto eligible bean methods; at runtime a CDI interceptor (GoblinServiceInterceptor, @Priority(4100)) applies the configured latency and exception assaults when the request resolved to the SERVICE layer.

  • Placement inside Fault Tolerance. Jakarta interceptors invoke lower priorities first, so 4100 sits inside the MicroProfile Fault Tolerance interceptor (4010): a thrown exception is observed by @Retry, @CircuitBreaker and @Fallback, and the injected latency is covered by @Timeout and @Bulkhead.

  • Eligibility. A bean method is decorated only when the class belongs to the application root archive and the usual goblin.target rules pass (include-packages / exclude-packages / exclude-annotations). JAX-RS resources (@Path), @Provider`s, interfaces, records, static and private methods are never decorated — HTTP endpoints belong to the HTTP_IN layer and sit above the guarded beans (a `@Path declared on an implemented interface counts too). Neither are classes that can never be CDI beans (anonymous, local and non-static inner classes, such as the anonymous MeterFilter a producer method returns), constructors nor synthetic methods. Methods named as the fallbackMethod of a @Fallback (on a method or on the class) in the same class, and FallbackHandler implementations, are skipped too, so a fallback can answer the original failure instead of being assaulted itself. The service-assault binding is only woven in dev and test builds, never in a production build.

  • One fault per call chain. Only the outermost intercepted call of a request is assaulted: when a decorated bean calls another decorated bean, the inner call runs untouched. Each further outermost call of the same request (a @Retry attempt, or a second bean called by the resource) draws level again.

  • Synchronous scope (for now). The decision is bound to the request thread, so @Asynchronous / non-blocking methods are not covered by Phase 1. On a Vert.x event-loop thread latency assaults are skipped (WARN logged once) instead of blocking the loop.

  • History. Each service assault is recorded like any other, with a fully qualified com.example.ClassName.method descriptor — including the actually applied latency duration for the latency assault. Since the rule falls inside FT, a @Retry-guarded method produces one record per attempt.

Database-layer assaults

The DATABASE layer injects faults where a slow or unreachable database really surfaces: when a JDBC connection is acquired from the Agroal pool. It therefore sits below Hibernate ORM, Panache and plain JDBC code alike.

  • Hook. One Agroal pool interceptor is registered per JDBC datasource (default and named) at build time, only when quarkus-agroal is present, and a call to it is woven at the very start of Agroal’s connection acquisition, in dev and test builds only. Hibernate Reactive and the reactive SQL clients do not use Agroal and are not covered.

  • Granularity. Every getConnection() call of the request is a candidate. The first one uses the per-request decision; every further one (typically a @Retry attempt opening a new transaction) draws level again.

  • Faults. latency delays the acquisition; exception throws the configured exception out of getConnection(). Both fire before the pool checks a connection out or enlists it in the transaction, so the pool is left exactly as it was: no connection is destroyed and the pool metrics (agroal_active_count…​) stay accurate. Fault Tolerance annotations on the repository or service above observe the exception, exactly as they would observe a real connection failure.

  • History. Records are labelled Database <datasource> connection, e.g. Database <default> connection, and tagged source=database in metrics and traces.

  • Scope. Only database access made within an inbound HTTP request or a consumed message is assaulted: a startup task or a scheduled job has no resolved layer.

Messaging-layer assaults

The MESSAGING layer targets the consumer side of Quarkus Messaging: the @Incoming methods of the application.

  • Entry point. A consumed message has no inbound HTTP request, so each consumer invocation is its own pseudo-request: it resolves among DATABASE, MESSAGING and SERVICE, and the decision stays in scope for the whole consumer call — a DATABASE or SERVICE fault fires inside message processing exactly as inside a REST call.

  • Placement outside Fault Tolerance. The interceptor runs at @Priority(4005), outside the MicroProfile Fault Tolerance interceptor (4010): a messaging-layer fault stands for the delivery of the message failing, so it is handled by the messaging failure strategy (nack, dead-letter queue, failure-strategy=ignore…​) rather than by a @Retry on the consumer. A SERVICE fault on the same consumer still runs inside Fault Tolerance.

  • Eligibility. The binding is woven at build time on the @Incoming methods of the application root archive, honouring the same goblin.target rules as the service layer, only when quarkus-messaging is present.

  • Synchronous scope. For a consumer returning Uni / CompletionStage only the synchronous part of the call is covered. Latency needs a blocking consumer (@Blocking, @RunOnVirtualThread): on an event-loop thread it is skipped rather than blocking the loop.

  • History. Records are labelled Messaging <class>.<method> and tagged source=messaging.

Runtime config modification

All assault parameters can be modified at runtime through the Dev UI or via JSON-RPC. The initial config is loaded from application.properties at startup, then held in a mutable in-memory config (MutableAssaultConfig). This means you can:

  • Enable latency AND exception simultaneously to simulate a slow failure.

  • Toggle assault types on/off without restarting.

  • Change the latency range, exception class, or HTTP status code on the fly.

  • Drop the target level from 100% to 10% to test intermittent failures.

Changes take effect on the next request — no restart, no redeployment. A WARN log confirms every change in the console.

State persistence

Runtime config changes made through the Dev UI are automatically persisted to a .goblin-state.json file in the project working directory. On the next startup, Goblin checks for this file and restores the previous configuration instead of starting fresh from application.properties.

This means your chaos scenario survives restarts — no need to reconfigure the dashboard every time you restart in dev mode.

The persisted state only covers the assault configuration (profile, toggles, latency range, exception settings, HTTP status settings, response body mode/percentage, response header rules, armed layers, target level). The global enabled/active flag is deliberately excluded: it is always taken from quarkus.goblin.enabled at startup and, during a session, from the Dev UI toggle — a state file can never reactivate an extension you disabled via quarkus.goblin.enabled=false. The two deactivations taken during a session are not stored in the file either. They are held in JVM-wide system properties, which survive a live reload — the one that recreates the engine but keeps the JVM — and disappear with the process, so a new start reads the configuration again:

  • the auto-off (goblin.auto-off.deadline): a pending deadline keeps its countdown, and an auto-off that already fired keeps chaos off until it is switched on again;

  • a manual deactivation (goblin.manual-off): clicking Deactivate or Disable all — or calling setActive(false) through JSON-RPC or Dev MCP — keeps chaos off across live reloads, and an explicit activation clears it so the activation survives them too. Only a deactivation is ever recorded, so the property can hold chaos off, never arm it.

Neither is written outside dev mode: a test application started by continuous testing runs in the same JVM and keeps its own state, so a test’s setActive(false) neither inherits nor overwrites the dev session’s decision. getStatus reports which of these applies as inactiveReason (manual, auto-off, disabled, test-mode, launch-mode), so an inactive dashboard is never a mystery.

State persistence is dev-mode only: the change listener, registered in dev and test mode so that the observers see every configuration change (see Observability: the AssaultObserver SPI), only writes the state file in dev mode, and the state file is only read on a dev-mode startup. Chaos itself only activates in dev mode, and in test mode on opt-in (quarkus.goblin.test.enabled=true) — a packaged production application always starts with the engine inactive. Integration or system tests always start from application.properties and never read nor overwrite the chaos configuration you saved from the Dev UI.

Add .goblin-state.json and .goblin/ (the saved scenarios, see Scenarios) to your .gitignore to avoid committing local chaos state to version control.

If the state file is corrupted or unreadable, Goblin logs a warning and falls back to the configuration from application.properties.

Observability: the AssaultObserver SPI

Every recorded assault and every active-state change is broadcast to the registered AssaultObserver beans (io.quarkiverse.goblin.AssaultObserver, default no-op methods). Observers run synchronously on the request path: the engine calls each bean and skips any failing observer (the failure is only logged at DEBUG) so observability can never break an assault. The engine injects Instance<AssaultObserver>, so any @ApplicationScoped implementation is picked up automatically; outside the CDI container (plain unit tests) the notification simply does nothing.

Every change of the assault configuration is broadcast too, through onConfigChange(AssaultConfigChange): the event carries the configuration before and after the change, both read-only snapshots, and the time it was published. That is enough for an application or a test to record the exact attack it went through, and to replay it after a fix:

@ApplicationScoped
public class AttackRecorder implements AssaultObserver {

    @Override
    public void onConfigChange(AssaultConfigChange change) {
        LOG.infof("Goblin attack changed: %s -> %s", change.previousDescription(), change.currentDescription());
    }
}

  • One change, one notification. A single setter call (a Dev UI toggle, a JSON-RPC or Dev MCP tool) is one notification. A change staged and published at once — applyConfig, a Dev UI import, a profile with its parameters — is one notification too, however many fields it changes. Two changes made at the same time from two threads can be folded into one notification, from the oldest previous to the latest current configuration: the intermediate state is then not reported on its own.

  • Not a request-path call. The observer runs on the thread that changed the configuration, never on a request thread. It must still return quickly: the caller waits for it.

  • Read-only. Both configurations are frozen snapshots: their setters throw UnsupportedOperationException, so an observer can keep them but never changes the live configuration through them.

  • Dev and test mode. The notification fires in test mode as well, so a @QuarkusTest can assert on the configuration an assault ran with, while the .goblin-state.json persistence stays dev-only.

  • What is not a change. The configuration loaded at startup, its startup validation and the profile label restored from the state file are not notified: the first notification is the first change made once the application runs. Activating or deactivating chaos is reported by onActiveChange, not here.

  • Failures are logged, never propagated. Unlike the request-path notifications, an observer failing on a configuration change is logged at WARN — it is losing the one event it exists for — and still never reaches the caller, which goes on applying its change. The other observers are notified all the same.

The optional quarkus-goblin-metrics and quarkus-goblin-opentelemetry modules are the current consumers of the assault and activation notifications (see Metrics (Micrometer / Prometheus) and Tracing (OpenTelemetry)); experiment recording plugs into the same hook.

Adding your own assault type

Goblin ships with six built-in assault types (latency, exception, HTTP status, dependency degradation, response body and response headers) and lets you register your own. Implement the io.quarkiverse.goblin.assault.Assault interface, annotate the class @ApplicationScoped, and it is discovered automatically at build time — no other wiring required:

  • type() — the AssaultType this assault represents. AssaultType is a closed enum: a custom assault reuses one of the existing constants. Note that the per-request gate only opens when at least one of the built-in toggles is enabled, so a custom assault must be paired with (or guarded by) one of them.

  • isEnabled(config) — whether the assault should run for the current mutable configuration.

  • recordLabel() — the label used in the assault history and the Markdown report.

  • order() — the execution position in the chain. Latency (order 10) runs first; assaults that abort the request run afterwards.

  • apply(context) — perform the assault. Return AssaultOutcome.CONTINUE to let the next enabled type run, or ABORTED to stop processing. To short-circuit the request call context.getRequestContext().abortWith(…​); to throw a simulated failure, throw a RuntimeException. The engine does not record anything on your behalf: call context.getEngine().recordAssault(context.getMethodName(), recordLabel()) when the assault fires, so it shows up in the history, the counters, the metrics and the traces.

At build time Goblin scans the application index for implementors of Assault and registers them as beans, so a custom type provided by your application is picked up automatically. The filter resolves the ordered list of assaults for every request and applies the enabled ones in order() sequence.