Metrics (Micrometer / Prometheus)

Table of Contents

Adding the dependency
Exposed metrics
Reading the latency histogram
Assault signals for dashboards
Internals

Goblin can surface its assault activity as Micrometer metrics, so the chaos you inject shows up in your existing Prometheus / Grafana dashboards alongside your application’s regular metrics.

The integration is optional and lives in the quarkus-goblin-metrics module. Adding it to the classpath is enough: it observes the assaults through the engine’s observer SPI and never alters the assault behavior.

Adding the dependency

<dependency>
    <groupId>io.quarkiverse.goblin</groupId>
    <artifactId>quarkus-goblin-metrics</artifactId>
    <version>${goblin.version}</version>
</dependency>

The module only brings in the Micrometer API (quarkus-micrometer), so it never adds a registry nor an endpoint to your application on its own. Add the registry you use — for Prometheus, io.quarkus:quarkus-micrometer-registry-prometheus, which exposes /q/metrics. Any other Micrometer backend (e.g. a custom MeterRegistry) picks up the same meters: the metrics are registered against the application’s MeterRegistry bean.

Exposed metrics

Metric

Description

Reading the latency histogram

Micrometer’s Prometheus export keeps _sum, _count and the _bucket series cumulative, while _max is the largest delay recorded over a sliding window of about two minutes (Micrometer’s default distribution expiry). When no latency is injected for that long, _max therefore drops back to 0 — this is the standard Micrometer behavior, identical to Quarkus’s own http_server_requests_seconds_max, not a broken probe. Quantiles are computed in PromQL from the _bucket series over the range you choose. There is no _min line.

Percentile and min/max over a rolling window, e.g. per source:

histogram_quantile(0.99, sum(rate(goblin_latency_injected_seconds_bucket[5m])) by (le, source))  # p99 of injected latency
max_over_time(goblin_latency_injected_seconds_max[10m])                                         # max over the last 10 minutes, across the _max resets

Because the histogram is enabled by default, on every scrape you get the _bucket series (goblin_latency_injected_seconds_bucket{source="…"}, with Micrometer’s default boundaries from ~1 ms up to 30 s), which is also what Grafana heatmaps expect. To narrow them, produce a MeterFilter:

@Produces
@Singleton
MeterFilter goblinLatencyBuckets() {
    return new MeterFilter() {
        @Override
        public DistributionStatisticConfig configure(Meter.Id id, DistributionStatisticConfig config) {
            if (id.getName().equals("goblin.latency.injected.seconds")) {
                return DistributionStatisticConfig.builder()
                        .minimumExpectedValue((double) Duration.ofMillis(10).toNanos())
                        .maximumExpectedValue((double) Duration.ofSeconds(10).toNanos())
                        .build()
                        .merge(config);
            }
            return config;
        }
    };
}

Assault signals for dashboards

Typical alerts:

goblin_assaults_total > 0 — chaos was injected (expect it while testing).
goblin_active == 0 — the engine is off, none of the other signals are changing.
Sudden spikes on goblin_latency_injected_seconds — latency assault is degrading the injected delays.

These metrics are also the foundation for the planned post-assault assertions (issue #50): declaring "inject 500 ms latency, expect a fallback to fire" becomes verifiable from the fault-tolerance signals combined with these metrics.

Internals

The quarkus-goblin-metrics module is a plain jar containing one @ApplicationScoped bean (io.quarkiverse.goblin.metrics.GoblinMetricsObserver) that implements the engine’s AssaultObserver SPI (io.quarkiverse.goblin.AssaultObserver, default no-op methods). The observer is fed by the engine on every recorded assault and on every active-state change, so it has zero coupling to the JAX-RS filters or the client interceptor.