Quick Guide to Vault Integration
Overview
This article is about a quick recipe for configuring a Salt Master server and Vault service to retrieve secrets from Vault using KV Secrets engine V2 with Approle authentication method and store those secrets in pillar using SDB.
The goal of this setup is to demonstrate usage and test capabilities. It is not intended to serve as a guide on setting up Vault in a production environment.
Included in this guide:
- Salt Master version 2019.2.0
- Vault 1.1.0 configured in server mode
- AppRole Auth method
- KV v2 secrets engine configuration (versioning support)
- Specific policy for Salt Master with limited access granted by AppRole
- Salt Master uses Pillar and SDB runner to retrieve secrets
- Secrets are retrieved by Salt Master and presented to minions via pillar, minions have not access to Vault server
- Troubleshooting with tcpdump
- Using Vault API
- Setting up basic token auth method
- Running Vault in dev mode
IMPORTANT: Vault secrets engine KV V2 adds versioning support. Something important to be aware of is that it adds a /data/ to your secrets path, so for instance if your secret path is salt/users_secrets, the Vault API calls must be pointed to salt/data/users_secrets. Some tools can do this for you automatically and silently, but if you make direct curl calls to Vault API you must be aware of the correct path. |
Not included in this guide:
- Vault installation and advanced deployment scenarios
- Other Vault secrets engines and authentication methods
Assumptions:
- Salt Master v2019.2.0 and Salt Minion(s) installed and configured properly (no additional requisites)
- Vault 1.1.0 running under default configuration (new install)
Steps:
Vault:
- Enable KV version 2 secrets engine for salt/ path
- Create and upload policy file for "saltmaster" role
- Create AppRole saltmaster
- Get role_id and secret_id
- Provision secrets data under salt/ path
- Test AppRole login and get data using granted token
Salt Master:
- Configure Vault module and SDB
- Test sdb runner to get secrets from Vault
- Configure pillar to use sdb to get secrets from Vault
- Create and run state consuming pillar data
Step-by-step:
In Vault server:
1. Enable KV Version 2 secret engine for salt/ path. All secrets will reside under salt/ path. Then, list secrets to verify configuration:
2. Create policy file for "saltmaster". The default policy is restrictive not allowing access to kv secrets.
3. Copy the following content:
5. PUT and GET sample data:
6. Enable AppRole authentication method:
7. Create Approle for saltmaster. Pay attention to the settings. Feel free to change values to your needs except for token_num_uses and token use. Token use can't be limited, just keep token_num_uses=0.
8. OPTIONAL: Verify role settings:
9. Get role-id and secret-id:
10. OPTIONAL: Test login with AppRole and validate assigned token settings:
11. OPTIONAL: Use token and get sample secret. (Be careful to request secrets data before the token expires, especially if you set your token_ttl too short (in seconds or minutes). In this guide, it's set to 1 hour:
In Salt Master server:
1. Configure SDB and Vault in Salt Master:
2. OPTIONAL: Configure peer settings in Salt Master. Note that this is only needed if planning to run vault modules from minions, which is not applicable in this case. For instance:
# salt salt vault.read_secret "salt/data/user1"
# In this case we don't want to run on the minion, just the master to retrieve secrets via pillar
3. Restart the Salt Master. Optional running in foreground may be a good choice before testing.
To run in foreground, after stopping the service, run: # salt-master -l debug
4. Get sample secret using SDB runner:
5. OPTIONAL: If peer.conf is enabled to allow minions, use vault modules. Otherwise it will return None:
6. Assign pillar in pillar top file.
# The pillar file will be called srv1_access in this example:
7. Create a pillar file:
NOTE: |tojson filter required as of 2019.2.0 |
8. Refresh and get pillar data:
9. Create a test state file to show and use secret:
10. Run state:
Summary:
Appendix - Useful Vault CLI and API commands
Setting environment variables for Vault server address and token
Check Vault status
Vault CLI Help
Creating, updating and deleting secrets in KV V2 Secret engine
Using Vault REST API
Appendix - Vault dev mode
In this mode, Vault runs entirely in-memory and starts unsealed with a single unseal key. The root token is already authenticated to the CLI, so you can immediately begin using Vault.
Appendix - Troubleshooting Vault with tcpdump
tcpdump has proven to be extremely useful in troubleshooting Vault, such as finding token permissions and denials issues:
// create token request with configured token s.pAYzPUPh62YtlXycOtOvfaC8
POST /v1/auth/token/create HTTP/1.1
Host: 127.0.0.1:8200
Content-Length: 145
Accept-Encoding: gzip, deflate
X-Vault-Token: s.pAYzPUPh62YtlXycOtOvfaC8
User-Agent: python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-957.5.1.el7.x86_64
Connection: keep-alive
Accept: */*
Content-Type: application/json
{"meta": {"saltstack-minion": "salt", "saltstack-user": "root", "saltstack-jid": "20190405022639705519"}, "policies": ["default"], "num_uses": 1}
// token created reply, new token s.inLJpTqPdYFUcEeC4y7gKVqV
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 05 Apr 2019 02:26:40 GMT
Content-Length: 490
{"request_id":"1208f0d0-be0f-0bf6-a30c-0f070623e2fb","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.inLJpTqPdYFUcEeC4y7gKVqV","accessor":"aqsxFnlmCZDiM4ib1VggHwdi","policies":["default"],"token_policies":["default"],"metadata":{"saltstack-jid":"20190405022639705519","saltstack-minion":"salt","saltstack-user":"root"},"lease_duration":2764800,"renewable":true,"entity_id":"","token_type":"service","orphan":false}}
// get Vault secret request with generated client token s.inLJpTqPdYFUcEeC4y7gKVqV
GET /v1/kv/data/user1 HTTP/1.1
Host: 127.0.0.1:8200
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-957.5.1.el7.x86_64
Connection: keep-alive
X-Vault-Token: s.inLJpTqPdYFUcEeC4y7gKVqV
Content-Type: application/json
// get Vault secret reply with requested data
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 05 Apr 2019 02:26:40 GMT
Content-Length: 319
{"request_id":"5f0e343b-5feb-460e-7aba-8d4dd18b22af","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":{"desc":"test user","password":"p4ssw0rd"},"metadata":{"created_time":"2019-04-03T06:51:11.496386892Z","deletion_time":"","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}
// Read secrets with AppRole auth method
// Requests a new token using role_id and secret_id
POST /v1/auth/approle/login HTTP/1.1
Host: 127.0.0.1:8200
Content-Length: 104
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-957.5.1.el7.x86_64
Connection: keep-alive
Content-Type: application/json
{"secret_id": "30041a3b-68c1-0e09-3050-b75e1c692f95", "role_id": "ed300913-f084-fc55-2874-991b0c537ae9"}
// Gets new token from Vault
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 05 Apr 2019 12:14:01 GMT
Content-Length: 461
{"request_id":"a3e6eea7-696a-c417-ebd7-d6aaf74b09e2","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.um3buHrDeWnTmiS7uRyOws2d","accessor":"iC0wpLCeSsy7n4eeRrPw61wn","policies":["default"],"token_policies":["default"],"metadata":{"role_name":"saltmaster"},"lease_duration":2764800,"renewable":true,"entity_id":"8a77c857-a123-86f7-94bc-9dc78b1a3558","token_type":"service","orphan":true}}
POST /v1/auth/token/create HTTP/1.1
Host: 127.0.0.1:8200
Content-Length: 145
Accept-Encoding: gzip, deflate
X-Vault-Token: s.um3buHrDeWnTmiS7uRyOws2d
User-Agent: python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-957.5.1.el7.x86_64
Connection: keep-alive
Accept: */*
Content-Type: application/json
{"meta": {"saltstack-minion": "salt", "saltstack-user": "root", "saltstack-jid": "20190405121401463486"}, "policies": ["default"], "num_uses": 1}
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 05 Apr 2019 12:14:01 GMT
Content-Length: 526
{"request_id":"03b9b403-c945-5da0-f66a-d66a5b08783b","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.rE210yeLMcyu2pe57JurHsNy","accessor":"iHPXjBeemjFfUV3GGFB5mzxY","policies":["default"],"token_policies":["default"],"metadata":{"saltstack-jid":"20190405121401463486","saltstack-minion":"salt","saltstack-user":"root"},"lease_duration":2764800,"renewable":true,"entity_id":"8a77c857-a123-86f7-94bc-9dc78b1a3558","token_type":"service","orphan":false}}
// Request for data using new token
GET /v1/kv/data/user1 HTTP/1.1
Host: 127.0.0.1:8200
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-957.5.1.el7.x86_64
Connection: keep-alive
X-Vault-Token: s.rE210yeLMcyu2pe57JurHsNy
Content-Type: application/json
// Data request reply
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 05 Apr 2019 12:14:01 GMT
Content-Length: 319
{"request_id":"a79687ab-1bdf-33d2-532f-ad0ed89324ff","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":{"desc":"test user","password":"p4ssw0rd"},"metadata":{"created_time":"2019-04-03T06:51:11.496386892Z","deletion_time":"","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}
Appendix - Configuring Salt SDB/Vault modules for Vault basic token authentication method
The simplest configuration is to create a token in Vault and configure Salt Vault/SDB to use the given token:
Once you have a valid token, set /etc/salt/master.d/vault.conf to use this token and auth method: token.
SALT + Vault at large scale
- Performance issues and bottlenecks seen at large implementations due high number of requests to backend.
- Causes slowness rendering the pillar process, which is perceived as slow response from Salt.
- Customers may want to tweak vault/sdb/pillar modules to alter behavior demanding less calls to Vault and adding some sort of caching in the middle.
- Data structure is also important. Performance is quite different making 1 call to retrieve 100 secrets (many secrets in 1 key) rather than making 100 calls to retrieve 1 secret (1 secret per key). In other words: less calls turns out in better perf.
- Vault Agent Cache, new in version 1.1 https://learn.hashicorp.com/vault/secrets-management/agent-caching . It includes auto auth and caching, which is worth a try.
Actual customer feedback:
- Currently, sdb.get makes two requests to fetch the data, first check expiry of token, then fetch data. We implemented in-memory caching of vault token to skip the validation request.
- Checking token expiry and token fetching only on request failures.
- Upgraded 'disk pillar cache' to be 'encrypted disk pillar cache' using AES; the key to decrypt the files needs to be retrieved from vault.
All together this added functionality drastically decreased our pillar rendering time, where we use vault backend to retrieve values.
Before fix (no caching):
Initial rendering: ~10 min
Subsequent rendering: ~10 min
After fix (encrypted caching):
Initial rendering: ~40 sec
Subsequent rendering from cache: ~2 sec
This new pillar caching requires the following changes in master.conf file:
pillar_cache: True
pillar_cache_ttl: 3600
pillar_cache_backend: disk
pillar_cache_secret: 'sdb://vault/path/to/your/pillar/decryption/key'
Performance Test Cases
Substantial performance difference between each of the following 3 cases, depending on how you built the data in Vault and in pillar. For the same 100 secrets, it varies from 0.7 second to 4 seconds.
Case 1:
Vault loaded with 100 objects/different keys (diff access paths).
Pillar built by 100 individual pillar files (1 vault secret per pillar file).
MEASURED TIME for pillar.items: 0m3.619s
SUMMARY: Worst case, Salt Master has a lot of overhead due to reading 100 pillar files and establishing many connections to Vault
Case 2:
Vault loaded with 100 objects/different keys
Pillar loaded by 1 pillar file (100 vault secrets in 100 pillar entries in 1 pillar file)
MEASURED TIME for pillar.items: 0m2.703s
SUMMARY: Not optimal, Salt Master has less overhead (only 1 pillar file) but still many connections to Vault
Case 3:
Vault loaded with a single 1 key object (1 access path) with 100 secrets
Pillar loaded by 1 pillar file, it retrieves all the secrets at once from the single Vault object
MEASURED TIME for pillar.items: real 0m0.717s
SUMMARY: BEST CASE, less overhead due single pillar file and 1 Vault object to retrieve