Using HashiCorp Vault
HashiCorp Vault is a multi-purpose tool aiming at protecting sensitive data, such as credentials, certificates, access tokens, encryption keys, … In the context of Quarkus, several use cases are supported:
-
mounting a map of properties stored into the Vault kv secret engine as an Eclipse MicroProfile config source
-
fetching credentials from Vault when configuring an Agroal datasource, as documented in the Vault Datasource Guide
-
fetching credentials from Vault when connecting to a RabbitMQ broker, as documented in the RabbitMQ Dynamic Credentials Guide
-
accessing Vault kv secret engine programmatically
-
support for the TOTP Secret Engine
-
support for the Transit Secret Engine as documented in the Vault Transit Secret Engine Guide
-
support for the PKI Secret Engine as documented in the Vault PKI Secret Engine Guide
-
support for several authentication methods as documented in the Vault Authentication guide
Under the hood, the Quarkus Vault extension takes care of authentication when negotiating a client Vault token plus any transparent token or lease renewals according to ttl and max-ttl.
Prerequisites
To complete this guide, you need:
-
roughly 20 minutes
-
an IDE
-
JDK 11+ installed with
JAVA_HOME
configured appropriately -
Apache Maven 3.8.1+
-
Docker installed
Starting Vault
Using Dev Services, Quarkus will take care of starting Vault in dev and test mode. |
Let’s start Vault in development mode:
docker run --rm --cap-add=IPC_LOCK -e VAULT_ADDR=http://localhost:8200 -p 8200:8200 -d --name=dev-vault vault:1.6.0
You can check that vault is running with:
docker logs dev-vault
You should see:
==> Vault server configuration:
Api Address: http://0.0.0.0:8200
Cgo: disabled
Cluster Address: https://0.0.0.0:8201
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level: info
Mlock: supported: true, enabled: false
Storage: inmem
Version: Vault v1.6.0
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.
You may need to set the following environment variable:
$ export VAULT_ADDR='http://0.0.0.0:8200'
The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.
Unseal Key: 0lZ2/vzpa92pH8gersSn2h9b5tmzd4m5sqIdMC/4PDs=
Root Token: s.5VUS8pte13RqekCB2fmMT3u2
Development mode should NOT be used in production installations!
==> Vault server started! Log data will stream in below:
In development mode, Vault gets configured with several options that makes it convenient:
-
Vault is already initialized with one key share (whereas in normal mode this has to be done explicitly and the number of key shares is 5 by default)
-
the unseal key and the root token are displayed in the logs (please write down the root token, we will need it in the following step)
-
Vault is unsealed
-
in-memory storage
-
TLS is disabled
-
a kv secret engine v2 is mounted at
secret/
By default Quarkus assumes that a kv secret engine in version 2 mounted at path |
In the following step, we are going to add a userpass
authentication that we will use from the Quarkus application,
to access a secret stored in the kv secret engine.
First open a shell inside the vault container:
docker exec -it dev-vault sh
Set the VAULT_TOKEN
with the value that was printed in the logs:
export VAULT_TOKEN=s.5VUS8pte13RqekCB2fmMT3u2
You can check Vault’s status using the CLI command vault status
:
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.6.0
Cluster Name vault-cluster-b07e80d8
Cluster ID 55bd74b6-eaaf-3862-f7ce-3473ab86c57f
HA Enabled false
Now let’s add a secret configuration for our application:
vault kv put secret/myapps/vault-quickstart/config a-private-key=123456
We have defined a path secret/myapps/vault-quickstart
in Vault that we need to give access to from the Quarkus application.
Create a policy that gives read access to secret/myapps/vault-quickstart
and subpaths:
cat <<EOF | vault policy write vault-quickstart-policy -
path "secret/data/myapps/vault-quickstart/*" {
capabilities = ["read", "create"]
}
EOF
When using a kv secret engine version 2, secrets are written and fetched at path |
And finally, let’s enable the userpass auth secret engine, and create user bob
with access to the vault-quickstart-policy
:
vault auth enable userpass
vault write auth/userpass/users/bob password=sinclair policies=vault-quickstart-policy
Quarkus supports additional Vault Authentication methods. Check the Vault Authentication guide for details. |
To check that the configuration is correct, try logging in:
vault login -method=userpass username=bob password=sinclair
You should see:
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.s93BVzJPzBiIGuYJHBTkG8Uw
token_accessor OKNipTAgxWbxsrjixASNiwdY
token_duration 768h
token_renewable true
token_policies ["default" "vault-quickstart-policy"]
identity_policies []
policies ["default" "vault-quickstart-policy"]
token_meta_username bob
Now set VAULT_TOKEN
to the token
above (instead of the root token), and try reading the vault-quickstart secret config:
export VAULT_TOKEN=s.s93BVzJPzBiIGuYJHBTkG8Uw
vault kv get secret/myapps/vault-quickstart/config
You should see:
======== Data ========
Key Value
--- -----
a-private-key 123456
Create a Quarkus application with a secret configuration
mvn io.quarkus.platform:quarkus-maven-plugin:3.12.0:create \
-DprojectGroupId=org.acme \
-DprojectArtifactId=vault-quickstart \
-DclassName="org.acme.quickstart.GreetingResource" \
-Dpath="/hello" \
-Dextensions="resteasy,vault"
cd vault-quickstart
If you already have your Quarkus project configured, you can add the vault
extension
to your project by running the following command in your project base directory:
./mvnw quarkus:add-extension -Dextensions="vault"
This will add the following to your pom.xml
:
<dependency>
<groupId>io.quarkiverse.vault</groupId>
<artifactId>quarkus-vault</artifactId>
<version>4.1.0</version>
</dependency>
Configure access to Vault from the application.properties
:
# vault url
quarkus.vault.url=http://localhost:8200
# vault authentication
quarkus.vault.authentication.userpass.username=bob
quarkus.vault.authentication.userpass.password=sinclair
# path within the kv secret engine where is located the vault-quickstart secret configuration
quarkus.vault.secret-config-kv-path=myapps/vault-quickstart/config
Using Dev Services, Quarkus will take care of configuring the Vault connection in dev and test mode. |
This should mount whatever keys are stored in secret/myapps/vault-quickstart
as Config properties.
Let’s verify that by adding a new endpoint in GreetingResource:
@Path("/hello")
public class GreetingResource {
@ConfigProperty(name = "a-private-key")
String privateKey;
...
@GET
@Path("/private-key")
@Produces(MediaType.TEXT_PLAIN)
public String privateKey() {
return privateKey;
}
}
Now compile the application and run it:
./mvnw clean install
java -jar target/quarkus-app/quarkus-run.jar
Finally test the new endpoint:
curl http://localhost:8080/hello/private-key
You should see:
123456
The Vault Config Source is configured with ordinal |
Only scalar types are supported for secrets value from Vault. |
Programmatic access to the KV secret engine
Sometimes secrets are retrieved from an arbitrary path that is known only at runtime through an application
specific property, or a method argument for instance.
In that case it is possible to inject a Quarkus VaultKVSecretEngine
, and retrieve secrets programmatically.
For instance, in GreetingResource
, add:
@Inject
VaultKVSecretEngine kvSecretEngine;
...
@GET
@Path("/secrets/{vault-path}")
@Produces(MediaType.TEXT_PLAIN)
public String getSecrets(@PathParam("vault-path") String vaultPath) {
return kvSecretEngine.readSecret("myapps/vault-quickstart/" + vaultPath).toString();
}
Add a new key in Vault:
vault kv put secret/myapps/vault-quickstart/private mysecret=abc
Restart the application after rebuilding it, and test it with the new endpoint:
curl http://localhost:8080/hello/secrets/private
You should see:
{mysecret=abc}
TOTP Secrets Engine
The Vault TOTP secrets engine generates time-based credentials according to the TOTP standard.
Vault TOTP supports both the generator scenario (like Google Authenticator) and the provider scenario (like the Google.com sign in).
The Vault extension integrates with the Vault TOTP secret engine by providing an io.quarkus.vault.VaultTOTPSecretEngine
class.
import io.quarkus.vault.VaultTOTPSecretEngine;
import io.quarkus.vault.secrets.totp.CreateKeyParameters;
import io.quarkus.vault.secrets.totp.KeyConfiguration;
import io.quarkus.vault.secrets.totp.KeyDefinition;
@Inject
VaultTOTPSecretEngine vaultTOTPSecretEngine;
CreateKeyParameters createKeyParameters = new CreateKeyParameters("Google", "test@gmail.com");
createKeyParameters.setPeriod("30m");
/** Google Authentication logic */
final Optional<KeyDefinition> myKey = vaultTOTPSecretEngine
.createKey("my_key_2", createKeyParameters); (1) (2)
final String keyCode = vaultTOTPSecretEngine.generateCode("my_key_2"); (3)
/** Google Login logic */
boolean valid = vaultTOTPSecretEngine.validateCode("my_key_2", keyCode); (4)
1 | Create a key to generate codes. |
2 | KeyDefinition class contains an embeddable base64 QR code that can be used by third-party code generators. |
3 | Generates a code (not using third-party generator). |
4 | Validates that the code is valid. |
Vault Health Check
If you are using the quarkus-smallrye-health
extension, quarkus-vault
can add a readiness health check
to validate the connection to the Vault server. This is disabled by default.
If enabled, when you access the /q/health/ready
endpoint of your application you will have information about the connection validation status.
This behavior can be enabled by setting the quarkus.vault.health.enabled
property to true
in your application.properties
.
Only if Vault is initialized, unsealed and active, the health endpoint returns that Vault is ready to serve requests.
You can change a bit this behaviour by using quarkus.vault.health.stand-by-ok
and quarkus.vault.health.performance-stand-by-ok
to true
in your application.properties
.
- stand-by-ok
-
Specifies if being a standby should still return the active status code instead of the standby status code.
- performance-stand-by-ok
-
Specifies if being a performance standby should still return the active status code instead of the performance standby status code.
You can inject io.quarkus.vault.VaultSystemBackendEngine
to run system operations programmatically.
When the readiness probe is failing in Kubernetes, then the application is not reachable. This means that if Vault is failing, all services depending on Vault will become unreachable and maybe this is not the desired state, so use this flag according to your requirements. |
TLS
In production mode, TLS should be activated between the Quarkus application and Vault to prevent man-in-the-middle attacks.
There are several ways to configure the Quarkus application:
-
through the standard
javax.net.ssl.trustStore
system property, which should refer to a JKS truststore containing the trusted certificates -
using property
quarkus.vault.tls.ca-cert
, which should refer to a pem encoded file.
If quarkus.vault.tls.ca-cert
is not set and the Quarkus application is using the Kubernetes authentication,
TLS will be active and use the CA certificate bundle located in /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
.
If you want to disable this behavior (for instance when using a trust store), you need to set it explicitly using:
quarkus.vault.tls.use-kubernetes-ca-cert=false
.
The last relevant property is quarkus.vault.tls.skip-verify
, which allows to communicate with Vault using TLS,
but without checking the certificate authenticity. This may be convenient in development, but is strongly
discouraged in production as it is not more secure than talking to Vault in plain HTTP.
Vault Provisioning
Beside the typical client use cases, the Quarkus extension can be used to provision Vault as well, for instance as part of a CD pipeline. Specific CDI beans and operations support this scenario:
-
VaultSystemBackendEngine
: create Vault Policies. See the Vault documentation. -
VaultKubernetesAuthService
. See the Vault documentation.-
Configure the Kubernetes Auth Method (Kubernetes host, certificates, keys, …)
-
Create Kubernetes Auth Roles (association between Kubernetes service accounts, Kubernetes namespaces and Vault policies)
-
-
VaultTransitSecretEngine
: CRUD operations on keys. See the Vault documentation.
For instance:
@Inject
VaultSystemBackendEngine vaultSystemBackendEngine;
@Inject
VaultKubernetesAuthService vaultKubernetesAuthService;
...
String rules = "path \"transit/*\" {\n" +
" capabilities = [ \"create\", \"read\", \"update\" ]\n" +
"}";
String policyName = "sys-test-policy";
vaultSystemBackendEngine.createUpdatePolicy(policyName, rules);
String roleName = "test-auth-k8s";
List<String> boundServiceAccountNames = asList("vault-auth");
List<String> boundServiceAccountNamespaces = asList("default");
List<String> tokenPolicies = asList(policyName);
vaultKubernetesAuthService.createRole(roleName, new VaultKubernetesAuthRole()
.setBoundServiceAccountNames(boundServiceAccountNames)
.setBoundServiceAccountNamespaces(boundServiceAccountNamespaces)
.setTokenPolicies(tokenPolicies));
Like any client, a provisioning program would have to authenticate using one of the supported Auth methods, and get the appropriate grants through one or more policies. Example:
# create Policies
path "sys/policy/*" {
capabilities = ["read", "create", "update", "delete"]
}
# create Kubernetes Auth Roles
path "auth/kubernetes/role/*" {
capabilities = ["read", "create", "update", "delete"]
}
You should adjust to the minimal set of access rights depending on your specific use case.
Dev Services (Configuration Free Vault)
Quarkus supports a feature called Dev Services that allows you to create various containers without any config.
What that means in practice is that if you have Docker running and have not configured quarkus.vault.url
,
Quarkus will automatically start a Vault container when running tests or in dev mode, and automatically
configure the connection.
When running the production version of the application, the Vault connection needs to be configured as normal,
so if you want to include a production Vault config in your application.properties
and continue to use Dev Services
we recommend that you use the %prod.
profile to define your Vault connection settings.
Shared server
Most of the time you need to share the server between applications. Dev Services for Vault implements a service discovery mechanism for your multiple Quarkus applications running in dev mode to share a single server.
Dev Services for Vault starts the container with the quarkus-dev-service-vault label which is used to identify the container.
|
If you need multiple (shared) servers, you can configure the quarkus.vault.devservices.service-name
attribute and indicate the server name.
It looks for a container with the same value, or starts a new one if none can be found.
The default service name is vault
.
Sharing is enabled by default in dev mode, but disabled in test mode.
You can disable the sharing with quarkus.vault.devservices.shared=false
.
Automatic Secret Engine Provisioning
To help with provisioning the automatically managed Vault instance, you can enable certain secret engines.
As mentioned above Vault containers started in dev mode automatically mount the kv secret engine v2
at |
Enable the Transit Secret Engine at transit/
using:
quarkus.vault.devservices.transit-enabled=true
Enable the PKI Secret Engine at pki/
using:
quarkus.vault.devservices.pki-enabled=true
Conclusion
As a general matter, you should consider reading the extensive Vault documentation and apply best practices.
The features exposed today through the Vault extension covers only a small fraction of what the product is capable of. Still, it addresses already some of the most common microservices scenarii (e.g. sensitive configuration and database credentials), which goes a long way towards creating secured applications.
Configuration Reference
Configuration property fixed at build time - All other configuration properties are overridable at runtime
Type |
Default |
|
---|---|---|
Whether or not an health check is published in case the smallrye-health extension is present. Environment variable: |
boolean |
|
Specifies if being a standby should still return the active status code instead of the standby status code. Environment variable: |
boolean |
|
Specifies if being a performance standby should still return the active status code instead of the performance standby status code. Environment variable: |
boolean |
|
If DevServices has been explicitly enabled or disabled. DevServices is generally enabled by default, unless there is an existing configuration present. When DevServices is enabled Quarkus will attempt to automatically configure and start a vault instance when running in Dev or Test mode and when Docker is running. Environment variable: |
boolean |
|
The container image name to use, for container based DevServices providers. Environment variable: |
string |
|
Indicates if the Vault instance managed by Quarkus Dev Services is shared. When shared, Quarkus looks for running containers using label-based service discovery. If a matching container is found, it is used, and so a second one is not started. Otherwise, Dev Services for Vault starts a new container. The discovery uses the Container sharing is only used in dev mode. Environment variable: |
boolean |
|
The value of the This property is used when you need multiple shared Vault instances. Environment variable: |
string |
|
Optional fixed port the dev service will listen to. If not defined, the port will be chosen randomly. Environment variable: |
int |
|
Should the Transit secret engine be enabled Environment variable: |
boolean |
|
Should the PKI secret engine be enabled Environment variable: |
boolean |
|
Custom container initialization commands Environment variable: |
list of string |
|
Microprofile Config ordinal. This is provided as an alternative to the The default value is higher than the file system or jar ordinals, but lower than env vars. Environment variable: |
int |
|
Vault server url.
<p>
Example: https://localhost:8200
<p>
See also the documentation for the Environment variable: |
||
Renew grace period duration. <p> This value if used to extend a lease before it expires its ttl, or recreate a new lease before the current lease reaches its max_ttl. By default Vault leaseDuration is equal to 7 days (ie: 168h or 604800s). If a connection pool maxLifetime is set, it is reasonable to set the renewGracePeriod to be greater than the maxLifetime, so that we are sure we get a chance to renew leases before we reach the ttl. In any case you need to make sure there will be attempts to fetch secrets within the renewGracePeriod, because that is when the renewals will happen. This is particularly important for db dynamic secrets because if the lease reaches its ttl or max_ttl, the password of the db user will become invalid and it will be not longer possible to log in. This value should also be smaller than the ttl, otherwise that would mean that we would try to recreate leases all the time. Environment variable: |
|
|
Vault config source cache period.
<p>
Properties fetched from vault as MP config will be kept in a cache, and will not be fetched from vault
again until the expiration of that period.
This property is ignored if Environment variable: |
|
|
List of comma separated vault paths in kv store,
where all properties will be available as MP config properties as-is, with no prefix.
<p>
For instance, if vault contains property Environment variable: |
list of string |
|
Maximum number of attempts when fetching MP Config properties on the initial connection. Environment variable: |
int |
|
Used to hide confidential infos, for logging in particular. Possible values are: <p> * low: display all secrets. * medium: display only usernames and lease ids (ie: passwords and tokens are masked). * high: hide lease ids and dynamic credentials username. Environment variable: |
|
|
Kv secret engine version. <p> see https://www.vaultproject.io/docs/secrets/kv/index.html Environment variable: |
int |
|
KV secret engine path.
<p>
This value is used when building the url path in the KV secret engine programmatic access
(i.e. Environment variable: |
string |
|
Timeout to establish a connection with Vault. Environment variable: |
|
|
Request timeout on Vault. Environment variable: |
|
|
List of remote hosts that are not proxied when the client is configured to use a proxy. This list serves the same purpose as the JVM Entries can use the * wildcard character for pattern matching, e.g *.example.com matches www.example.com. Environment variable: |
list of string |
|
The proxy host. If set the client is configured to use a proxy. Environment variable: |
string |
|
The port the proxy is listening on, 3128 by default. Environment variable: |
int |
|
List of comma separated vault paths in kv store,
where all properties will be available as prefixed MP config properties.
<p>
For instance if the application properties contains
Environment variable: |
list of string |
required |
Dynamic credentials' role.
<p>
Roles are defined by the secret engine in use. For example, Environment variable: |
string |
|
Mount of dynamic credentials secrets engine, for example Environment variable: |
string |
|
Path of dynamic credentials request.
<p>
Request paths are dictated by the secret engine in use. For standard secret engines this should be
left as the default of Environment variable: |
string |
|
A path in vault kv store, where we will find the kv-key.
<p>
One of Environment variable: |
string |
|
Key name to search in vault path Environment variable: |
string |
|
Type |
Default |
|
Vault Enterprise namespace
<p>
If set, this will add a Environment variable: |
string |
|
Type |
Default |
|
Vault token, bypassing Vault authentication (kubernetes, userpass or approle). This is useful in development where an authentication mode might not have been set up. In production we will usually prefer some authentication such as userpass, or preferably kubernetes, where Vault tokens get generated with a TTL and some ability to revoke them. Lease renewal does not apply. Environment variable: |
string |
|
Client token wrapped in a wrapping token, such as what is returned by: vault token create -wrap-ttl=60s -policy=myapp client-token and client-token-wrapping-token are exclusive. Lease renewal does not apply. Environment variable: |
string |
|
Role Id for AppRole auth method. This property is required when selecting the app-role authentication type. Environment variable: |
string |
|
Secret Id for AppRole auth method. This property is required when selecting the app-role authentication type. Environment variable: |
string |
|
Wrapping token containing a Secret Id, obtained from: vault write -wrap-ttl=60s -f auth/approle/role/myapp/secret-id secret-id and secret-id-wrapping-token are exclusive Environment variable: |
string |
|
Allows configure Approle authentication mount path. Environment variable: |
string |
|
User for userpass auth method. This property is required when selecting the userpass authentication type. Environment variable: |
string |
|
Password for userpass auth method. This property is required when selecting the userpass authentication type. Environment variable: |
string |
|
Wrapping token containing a Password, obtained from: vault kv get -wrap-ttl=60s secret/ The key has to be 'password', meaning the password has initially been provisioned with: vault kv put secret/ password= password and password-wrapping-token are exclusive Environment variable: |
string |
|
Allows configure userpass authentication mount path. Environment variable: |
string |
|
Kubernetes authentication role that has been created in Vault to associate Vault policies, with Kubernetes service accounts and/or Kubernetes namespaces. This property is required when selecting the Kubernetes authentication type. Environment variable: |
string |
|
Location of the file containing the Kubernetes JWT token to authenticate against in Kubernetes authentication mode. Environment variable: |
string |
|
Allows configure Kubernetes authentication mount path. Environment variable: |
string |
|
Type |
Default |
|
Allows to bypass certificate validation on TLS communications. If true this will allow TLS communications with Vault, without checking the validity of the certificate presented by Vault. This is discouraged in production because it allows man in the middle type of attacks. Environment variable: |
boolean |
|
Certificate bundle used to validate TLS communications with Vault. The path to a pem bundle file, if TLS is required, and trusted certificates are not set through javax.net.ssl.trustStore system property. Environment variable: |
string |
|
If true and Vault authentication type is kubernetes, TLS will be active and the cacert path will be set to /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. If set, this setting will take precedence over property quarkus.vault.tls.ca-cert. This means that if Vault authentication type is kubernetes and we want to use quarkus.vault.tls.ca-cert or system property javax.net.ssl.trustStore, then this property should be set to false. Environment variable: |
boolean |
|
Type |
Default |
|
Specifies the name of the key to use. By default this will be the property key alias. Used when the same transit key is used with different configurations. Such as in:
Environment variable: |
string |
|
Set to true when the input is already hashed. Applies to sign operations. Environment variable: |
boolean |
|
When using a RSA key, specifies the RSA signature algorithm. Applies to sign operations. Environment variable: |
string |
|
Specifies the hash algorithm to use for supporting key types. Applies to sign operations. Environment variable: |
string |
|
Specifies the type of key to create for the encrypt operation. Applies to encrypt operations. Environment variable: |
string |
|
If enabled, the key will support convergent encryption, where the same plaintext creates the same ciphertext. Applies to encrypt operations. Environment variable: |
string |
About the Duration format
To write duration values, use the standard You can also use a simplified format, starting with a number:
In other cases, the simplified format is translated to the
|