Testing Pool Exhaustion

A connection pool has a limit on how many database connections it can provide. If every connection is busy waiting for a hung database, the next request must wait for one to become available. The pool’s acquisition timeout limits that wait.

This guide tests two questions:

  • Does a request fail within a bounded time when every connection is busy?

  • Can a request that does not use that pool still succeed?

The main example uses a pool with two connections. Two requests occupy those connections; a third request should fail after the pool’s one-second acquisition timeout. A separate example tests a pool that cannot open new connections.

Prerequisites

Use the NetworkFaultsProfile test profile and inject NetworkFaults, as described in the Quick start.

The sample application provides these endpoints:

Endpoint Purpose

/audit/count

Runs a query using the named audit datasource.

/audit/slow-count

Uses the same datasource, but runs a query with pg_sleep(0.5) to keep a connection busy during setup.

/orders/price

Calls a REST dependency and does not use the audit pool.

/orders/count

Runs a query using the default datasource, used in the second example.

Fill the Pool with Blocked Requests

Configure Timeouts You Can Tell Apart

The audit datasource uses PostgreSQL and the following settings:

quarkus.datasource."audit".db-kind=postgresql
quarkus.datasource."audit".jdbc.max-size=2
quarkus.datasource."audit".jdbc.acquisition-timeout=PT1S
quarkus.datasource."audit".jdbc.additional-jdbc-properties.socketTimeout=10
quarkus.datasource."audit".jdbc.additional-jdbc-properties.connectTimeout=10

PT1S means one second; the PostgreSQL driver’s timeout properties above are in seconds.

Setting Role in this test

Pool size: 2

Two concurrent requests can occupy every connection.

Acquisition timeout: 1 second

The third request gives up waiting for a free connection.

Driver timeouts: 10 seconds

The first two requests remain blocked long enough to test the third request.

Keeping these timeouts far apart makes the result easier to interpret. If acquiring a connection and waiting for the database both time out after one second, elapsed time alone cannot tell you which timeout ended the request.

Establish Both Connections Before Applying the Fault

A pool with max-size=2 does not necessarily have two connections open already. Create them by sending two overlapping requests to /audit/slow-count, then wait for both requests to finish successfully. At that point, the pool has two established connections available for the test.

This setup matters because opening a new connection involves a database handshake. If the fault hangs that handshake, the test can end up measuring connection creation instead of waiting for a busy connection to become free. Do not assume that the acquisition timeout also bounds every step of connection creation; configure the driver’s connection and login timeouts for that case. See Client Timeout Defaults.

Block Two Requests, Then Send a Third

Apply hangs(onNextWrite()) after the warm-up. The fault holds each connection when the application next writes to it, so the two /audit/count requests occupy the pool while waiting for database communication to continue.

Use the proxy’s observations to know when to proceed:

  • awaitConnections(2, …​) waits for two open connections. They may still be idle.

  • awaitHeldConnections(2, …​) waits for two connections held by the fault. With hangs(onNextWrite()), they are being used by the requests you just started.

Send the third request only after both connections are held. A fixed sleep cannot reliably establish that condition on a busy build machine.

Target audit = faults.target(DataSource.class, new DataSourceLiteral("audit"));
try (ExecutorService callers = Executors.newFixedThreadPool(2)) {
    List<Future<Integer>> warmUp = new ArrayList<>();
    for (int i = 0; i < 2; i++) {
        warmUp.add(callers.submit(() -> when().get("/audit/slow-count").andReturn().statusCode()));
    }
    audit.traffic().awaitConnections(2, Duration.ofSeconds(5));
    for (Future<Integer> status : warmUp) {
        assertEquals(200, status.get(5, TimeUnit.SECONDS));
    }

    try (ActiveFault f = audit.hangs(onNextWrite())) {
        List<Future<Integer>> held = new ArrayList<>();
        for (int i = 0; i < 2; i++) {
            held.add(callers.submit(() -> when().get("/audit/count").andReturn().statusCode()));
        }
        audit.traffic().awaitHeldConnections(2, Duration.ofSeconds(5)); (1)

        long ms = Stopwatch.millis(() -> when()
                .get("/audit/count").then().statusCode(500)); (2)
        assertTrue(ms >= 800 && ms < 3_000);

        when().get("/orders/price").then().statusCode(200); (3)

        f.end(); (4)
        for (Future<Integer> status : held) {
            status.get(5, TimeUnit.SECONDS);
        }
    }
}
1 Both pool connections are busy before the third request starts.
2 The third request cannot obtain a connection. This sample returns 500 after the acquisition timeout; the timing bounds allow slack around the configured one second.
3 A request that does not use the exhausted pool still succeeds while the first two requests are blocked.
4 End the fault before joining the blocked requests, so they can finish and the executor can shut down.

The third request runs on the test thread. The executor has only two threads because it is responsible only for the two requests that occupy the pool.

The 500 response is this sample application’s error handling. In your application, assert the response your API is supposed to return when a connection cannot be acquired. Keep timing assertions broad enough for a slow build machine, while still well below the driver timeout.

Understand Cleanup and What the Test Proves

Ending a fault restores traffic and runs the extension’s cleanup, including flushing the Agroal pool. The first two requests may therefore finish with a closed-connection error. The final Future.get(…​) calls check that they finish and surface exceptions from the submitted tasks; they do not assert successful HTTP responses. See What happens when a fault ends.

The resources are nested so that the fault ends before the executor closes, including when an assertion fails. Calling f.end() explicitly releases the blocked requests before joining them; closing the fault again at the end of the block is safe.

This test demonstrates that waiting for the exhausted pool is bounded and that the independent endpoint still works under these two blocked requests. That separation is often called a bulkhead: a failure affecting one resource should leave other work able to run. It does not establish how much concurrent load the application can tolerate. For example, enough blocked database requests could also consume a shared worker pool; testing that requires a separate load scenario.

Test a Pool That Cannot Open New Connections

A pool can also fail to serve a request because it has no usable connections and cannot create replacements. This case needs only one request.

The default datasource in the sample has max-size=4, acquisition-timeout=PT2S, and PostgreSQL connect and socket timeouts of two seconds. It also enables foreground validation with quarkus.datasource.jdbc.foreground-validation-interval=PT0.001S, so Agroal detects the connections reset by dropConnections() when it tries to reuse them.

Target database = faults.target(DataSource.class);
when().get("/orders/count").then().statusCode(200);

database.dropConnections(); // invalidate existing pooled connections
try (ActiveFault f = database.connectTimeout()) { // block replacement connections
    long ms = Stopwatch.millis(() -> when()
            .get("/orders/count").then().statusCode(500));
    assertTrue(ms >= 1_500 && ms < 4_000);
    assertEquals(0, database.traffic().connections());
}

when().get("/orders/count").then().statusCode(200);

dropConnections() resets the existing connections once. connectTimeout() then causes new TCP connection attempts to time out, leaving the pool without a usable connection for the request. After the fault ends, the final request checks that the pool can recover.

This sample checks a bounded failure around its configured two-second timeouts. Because the acquisition and driver connect timeouts are both two seconds, this example’s elapsed time alone does not identify which timeout fired. Use the first example when you need to distinguish pool exhaustion from a driver timeout.

connectTimeout() is available on Linux only. The source test uses @EnabledOnOs(OS.LINUX). See Faults.

Going Further