SaltStack Support

Quick Guide to Vault Integration

Overview 

This article is about a quick recipe for configuring a Salt Master server and Vault service to retrieve secrets from Vault using KV Secrets engine V2 with Approle authentication method and store those secrets in pillar using SDB.

The goal of this setup is to demonstrate usage and test capabilities, not intended to serve as a guide on setting up Vault in production environment.

Included in this guide: Edit section

  • Salt Master version 2019.2.0
  • Vault 1.1.0 configured in server mode 
  • Approle Auth method
  • KV v2 secrets engine configuration (versioning support)
  • Specific policy for Salt Master with limited access granted by approle
  • Salt Master uses Pillar and SDB runner to retrieve secrets
  • Secrets are retrieved by Salt Master and presented to minions via pillar, minions have not access to Vault server
  • Troubleshooting with tcpdump
  • Using Vault API
  • Setting up basic token auth method
  • Running Vault in dev mode
IMPORTANT: Vault secrets engine KV V2 adds versioning support, something important to remark to be aware is that it adds a /data/ to your secrets path, so for instance if your secret path is salt/users_secrets, the Vault API calls must be pointed to salt/data/users_secrets, some tools can do this for you automatically and silently, but if you make direct curl calls to Vault API you must be aware of the correct path. 

 

Not included in this guide: 

  • Vault installation and advanced deployment scenarios
  • Other Vault secrets engines and authentication methods

Assumptions: 

  • Salt Master v2019.2.0 and Salt Minion(s) installed and configured properly (no additional requisites)
  • Vault 1.1.0 running under default configuration (new install)

 

Steps: 

Vault:

  • Enable KV version 2 secretn engine for salt/ path
  • Create and upload policy file for "saltmaster" role
  • Create approle saltmaster
  • Get role_id and secret_id
  • Provision secrets data under salt/ path
  • Test approle login and get data usign granted token

Salt Master:

  • Configure Vault module and SDB
  • Test sdb runner to get secrets from Vault
  • Configure pillar to use sdb to get secrets from Vault
  • Create and run state consuming pillar data

 

Step-by-step: 

In Vault server: 

# Enable KV Version 2 secret engine for salt/ path, all our secrets will reside under salt/ path, then list secrets to verify configuration

vault secrets enable -version=2 -path=salt/ kv
vault secrets list -detailed
 

# Create policy file for "saltmaster" (default policy is restrictive not allowing access to kv secrets)

vi /etc/vault/saltmaster.hcl

# Copy the following content

# This section grants all access on "salt/*". 
# Further restrictions can be applied to this broad policy, as shown below.

# Minimum required access for KV secrets engine V1 and V2

# V1
path "salt/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "salt/" {
  capabilities = ["list"]
}

# V2
path "salt/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "salt/data/" {
  capabilities = ["list"]
}

# THIS ENTRY ALLOW SALTUTIL.PILLAR_REFRESH TO WORK PROPERLY
path "auth/token/create" {
 capabilities = ["create", "read", "update"]
}
# Upload policy saltmaster
 vault policy write saltmaster /etc/vault/saltmaster.hcl

# PUT and GET sample data

vault kv put salt/user1 password=p4ssw0rd desc="test user"
vault kv get salt/user1

# Enable approle authentication method

vault auth enable approle

# Create approle for saltmaster, pay attention to settings, feel free to change values to your needs, except for token_num_uses, token use can't be limited, just keep token_num_uses=0

 vault write auth/approle/role/saltmaster policies="saltmaster" token_num_uses=0 secret_id_num_uses=0 token_ttl=3600 secret_id_ttl=31536000 token_type=service role_name=saltmaster

# OPTIONAL Verify role settings

vault read auth/approle/role/saltmaster
Key                      Value
---                      -----
bind_secret_id           true
bound_cidr_list          <nil>
local_secret_ids         false
period                   0s
policies                 [saltmaster]
secret_id_bound_cidrs    <nil>
secret_id_num_uses       0
secret_id_ttl            8760h
token_bound_cidrs        <nil>
token_max_ttl            0s
token_num_uses           0
token_ttl                1h
token_type               service

# Get role-id and secret-id

vault read auth/approle/role/saltmaster/role-id
vault write -f auth/approle/role/saltmaster/secret-id

# OPTIONAL Test login with approle and validate assigned token settings

vault write auth/approle/login role_id=9c138de4-d3ad-bbdb-e72c-e928bfcd98e0 secret_id=a52a574f-231a-5fc0-76d0-d4736a3cd72e
Key                     Value
---                     -----
token                   s.CTT20bP4uF8auuH44cPeopUR
token_accessor          Xq2Hu2rNfaweeaMPm0JxKoCS
token_duration          1h
token_renewable         true
token_policies          ["default" "saltmaster"]
identity_policies       []
policies                ["default" "saltmaster"]
token_meta_role_name    saltmaster

# OPTIONAL Use token and get sample secret (be sure of requesting secrets data before token expires, specially if you set your token_ttl too short (seconds or minutes), in this guide it's set to 1 hour

export VAULT_TOKEN=s.CTT20bP4uF8auuH44cPeopUR
vault kv get salt/user1
====== Metadata ======
Key              Value
---              -----
created_time     2019-04-18T19:17:56.647818266Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
desc        test user
password    p4ssw0rd
 

In Salt Master server: 

# Configure SDB and Vault in Salt Master

 vi /etc/salt/master.d/vault.conf
 
# SDB CONFIG
myvault:
  driver: vault

# VAULT CONFIG FOR TOKEN OR APPROLE AUTH MODE
vault:
  url: http://127.0.0.1:8200
  auth:
    method: approle
    role_id: 9c138de4-d3ad-bbdb-e72c-e928bfcd98e0
    secret_id: a52a574f-231a-5fc0-76d0-d4736a3cd72e
  policies:
    - saltmaster
 

# OPTIONAL Configure peer settings in Salt Master (only needed if planning to run vault modules from minions, not in this case), for instance:
#       salt salt vault.read_secret "salt/data/user1"
#  In this case we don't want to run on the minion, just the master to retrieve secrets via pillar

 vi /etc/salt/master.d/peer_run.conf
 peer_run:
  .*:
    - vault.generate_token

# Restart Salt Master (optional running in foreground may be a good choice before testing

# To run in foreground, after stopping the service, run

# salt-master -l debug

service salt-master stop
service salt-master start

# Get sample secret using SDB runner

 salt-run sdb.get 'sdb://myvault/salt/data/user1' --out=json { "data": { "password": "p4ssw0rd", "desc": "test user" }, "metadata": { "created_time": "2019-04-18T19:17:56.647818266Z", "destroyed": false, "version": 1, "deletion_time": "" } }

# OPTIONAL If peer.conf enabled to allow minions use vault modules (otherwise will return None)

salt minion vault.read_secret "salt/data/user1"
minion:
    ----------
    data:
        ----------
        desc:
            test user
        password:
            p4ssw0rd
    metadata:
        ----------
        created_time:
            2019-04-18T19:17:56.647818266Z
        deletion_time:
        destroyed:
            False
        version:
            1
 

# Assign pillar in pillar top file

# The pillar file will be called srv1_access in this example

vi /srv/pillar/top.sls 
 base:
  '*':
    - srv1_access
 

# Create pillar file

NOTE: |tojson filter required as of 2019.2.0

 

vi /srv/pillar/srv1_access.sls
{% set srv1_access = salt['sdb.get']('sdb://myvault/salt/data/user1') %}
srv1_access: {{ srv1_access['data']|tojson }}

# Refresh and get pillar data

salt \* saltutil.refresh_pillar
salt \* pillar.items
minion:
    ----------
    srv1_access:
        ----------
        desc:
            test user
        password:
            p4ssw0rd
 

# Create test state file to show and use secret

vi /srv/salt/show_secrets.sls

show_secrets:
  test.configurable_test_state:
    - name: Show them secrets
    - changes: False
    - result: True
    - comment: {{ pillar['srv1_access']['password'] }}

# Run state

salt minion state.sls show_secrets
minion:
----------
          ID: show_secrets
    Function: test.configurable_test_state
        Name: Show them secrets
      Result: True
     Comment: p4ssw0rd
     Started: 21:46:17.853706
    Duration: 0.589 ms
     Changes:

Summary for salt
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:   0.589 ms
 

Summary: 

This quick recipe demonstrated how to obtain secrets from Vault stored in KV v2 secret engine using Salt SDB runner and Salt Pillar features and how to use the secret in a Salt state.

Appendix - Useful Vault CLI and API commands 

 Setting environment variables for Vault server address and token:

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN=<root vault token>

Check Vault status

vault status
 

Vault CLI Help

vault path-help (any path)
vault path-help secret
vault kv --help
 

Creating, updating and deletiing secrets in KV V2 Secret engine

# Write secret
vault kv put secret/cookie recipe=sugar

# List secrets
vault kv list secret

# Read secrets
vault kv get -format json secret/cookie

# Write/Update secret
vault kv put secret/cookie recipe="sugar,flour" cooktime="45 minutes"

# Delete secret, destroy version, delete metadata
vault kv delete secret/cookie
vault kv destroy -version=2 secret/cookie
vault kv metadata delete secret/cookie
 

Using Vault REST API

# Secret Config info
curl -H "X-Vault-Token:$VAULT_TOKEN" http://127.0.0.1:8200/v1/secret/config

# Get secret
curl -H "X-Vault-Token:$VAULT_TOKEN" http://127.0.0.1:8200/v1/secret/data/cookie

# Write secret
curl -X POST -H "X-Vault-Token:$VAULT_TOKEN" -d '{"data":{"password":"somesecret","desc":"test user"}}' http://127.0.0.1:8200/v1/secret/data/user2

# Delete secret
curl --request DELETE -H "X-Vault-Token:$VAULT_TOKEN" http://127.0.0.1:8200/v1/secret/data/user2

# Destroy secret version
curl -X POST -H "X-Vault-Token:$VAULT_TOKEN" -d '{"versions": [1, 2]}' http://127.0.0.1:8200/v1/secret/destroy/user2

# Enable AppRole auth method
$ curl --silent --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type": "approle"}' http://127.0.0.1:8200/v1/sys/auth/approle

# Create/Update role saltmaster
$ curl --silent --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type": "approle"}' http://127.0.0.1:8200/v1/sys/auth/approle
payload_approle_saltmaster.json:
{
"policies":"default",
"secret_id_ttl": 0,
"secret_id_num_uses": 0,
"token_type":"service",
"token_num_uses": 0,
"token_ttl": 0,
"token_max_ttl": 0,
"role_name":"saltmaster"
}

# List roles
curl --silent --header "X-Vault-Token:$VAULT_TOKEN" --request LIST http://127.0.0.1:8200/v1/auth/approle/role | jq '.'

# Fetch role_id and secret_id
curl --silent --header "X-Vault-Token: $VAULT_TOKEN" http://127.0.0.1:8200/v1/auth/approle/role/saltmaster/role-id | jq '.'
curl --silent --header "X-Vault-Token: $VAULT_TOKEN" --request POST http://127.0.0.1:8200/v1/auth/approle/role/saltmaster/secret-id | jq '.'\

# Login (get token) with AppRole
curl --silent --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"role_id":"ed300913-f084-fc55-2874-991b0c537ae9","secret_id":"30041a3b-68c1-0e09-3050-b75e1c692f95"}' http://127.0.0.1:8200/v1/auth/approle/login | jq '.'

# Get token with basic token auth method
curl --silent --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data @payload_token.json http://127.0.0.1:8200/v1/auth/token/create | jq '.'
payload_token.json:
 {
"policies": [ "default" ],
"metadata": {
"user": "adrian"
},
"ttl": "4h",
"renewable": true,
"num_uses": 0
}

Appendix - Vault dev mode

In this mode, Vault runs entirely in-memory and starts unsealed with a single unseal key. The root token is already authenticated to the CLI, so you can immediately begin using Vault.

vault server -dev

Appendix - Troubleshooting Vault with tcpdump 

Edit section

tcpdump has proven to be extremely useful in troubleshooting Vault, for instance to find token permissions and denials issues

tcpdump -A -i lo port 8200 -n

// create token request with configured token s.pAYzPUPh62YtlXycOtOvfaC8

POST /v1/auth/token/create HTTP/1.1
Host: 127.0.0.1:8200
Content-Length: 145
Accept-Encoding: gzip, deflate
X-Vault-Token: s.pAYzPUPh62YtlXycOtOvfaC8
User-Agent: python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-957.5.1.el7.x86_64
Connection: keep-alive
Accept: */*
Content-Type: application/json

{"meta": {"saltstack-minion": "salt", "saltstack-user": "root", "saltstack-jid": "20190405022639705519"}, "policies": ["default"], "num_uses": 1}

 

// token created reply, new token s.inLJpTqPdYFUcEeC4y7gKVqV

HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 05 Apr 2019 02:26:40 GMT
Content-Length: 490

{"request_id":"1208f0d0-be0f-0bf6-a30c-0f070623e2fb","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.inLJpTqPdYFUcEeC4y7gKVqV","accessor":"aqsxFnlmCZDiM4ib1VggHwdi","policies":["default"],"token_policies":["default"],"metadata":{"saltstack-jid":"20190405022639705519","saltstack-minion":"salt","saltstack-user":"root"},"lease_duration":2764800,"renewable":true,"entity_id":"","token_type":"service","orphan":false}}

 

// get Vault secret request with generated client token s.inLJpTqPdYFUcEeC4y7gKVqV

GET /v1/kv/data/user1 HTTP/1.1
Host: 127.0.0.1:8200
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-957.5.1.el7.x86_64
Connection: keep-alive
X-Vault-Token: s.inLJpTqPdYFUcEeC4y7gKVqV
Content-Type: application/json

// get Vault secret reply with requested data

HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 05 Apr 2019 02:26:40 GMT
Content-Length: 319

{"request_id":"5f0e343b-5feb-460e-7aba-8d4dd18b22af","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":{"desc":"test user","password":"p4ssw0rd"},"metadata":{"created_time":"2019-04-03T06:51:11.496386892Z","deletion_time":"","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}

 

// Read secrets with AppRole auth method

// Requests a new token using role_id and secret_id

POST /v1/auth/approle/login HTTP/1.1
Host: 127.0.0.1:8200
Content-Length: 104
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-957.5.1.el7.x86_64
Connection: keep-alive
Content-Type: application/json

{"secret_id": "30041a3b-68c1-0e09-3050-b75e1c692f95", "role_id": "ed300913-f084-fc55-2874-991b0c537ae9"}

 

// Gets new token from Vault

HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 05 Apr 2019 12:14:01 GMT
Content-Length: 461

{"request_id":"a3e6eea7-696a-c417-ebd7-d6aaf74b09e2","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.um3buHrDeWnTmiS7uRyOws2d","accessor":"iC0wpLCeSsy7n4eeRrPw61wn","policies":["default"],"token_policies":["default"],"metadata":{"role_name":"saltmaster"},"lease_duration":2764800,"renewable":true,"entity_id":"8a77c857-a123-86f7-94bc-9dc78b1a3558","token_type":"service","orphan":true}}

 

POST /v1/auth/token/create HTTP/1.1
Host: 127.0.0.1:8200
Content-Length: 145
Accept-Encoding: gzip, deflate
X-Vault-Token: s.um3buHrDeWnTmiS7uRyOws2d
User-Agent: python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-957.5.1.el7.x86_64
Connection: keep-alive
Accept: */*
Content-Type: application/json

{"meta": {"saltstack-minion": "salt", "saltstack-user": "root", "saltstack-jid": "20190405121401463486"}, "policies": ["default"], "num_uses": 1}

 

HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 05 Apr 2019 12:14:01 GMT
Content-Length: 526

{"request_id":"03b9b403-c945-5da0-f66a-d66a5b08783b","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.rE210yeLMcyu2pe57JurHsNy","accessor":"iHPXjBeemjFfUV3GGFB5mzxY","policies":["default"],"token_policies":["default"],"metadata":{"saltstack-jid":"20190405121401463486","saltstack-minion":"salt","saltstack-user":"root"},"lease_duration":2764800,"renewable":true,"entity_id":"8a77c857-a123-86f7-94bc-9dc78b1a3558","token_type":"service","orphan":false}}

 

// Request for data using new token

GET /v1/kv/data/user1 HTTP/1.1
Host: 127.0.0.1:8200
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-957.5.1.el7.x86_64
Connection: keep-alive
X-Vault-Token: s.rE210yeLMcyu2pe57JurHsNy
Content-Type: application/json

// Data request reply

HTTP/1.1 200 OK
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 05 Apr 2019 12:14:01 GMT
Content-Length: 319

{"request_id":"a79687ab-1bdf-33d2-532f-ad0ed89324ff","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":{"desc":"test user","password":"p4ssw0rd"},"metadata":{"created_time":"2019-04-03T06:51:11.496386892Z","deletion_time":"","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}

 

Appendix - Configuring Salt SDB/Vault modules for Vault basic token authentication method 

This is the simplest configuration, just create token in vault and configure Salt Vault/SDB to use given token

vault token create -policy=default -use-limit=0
Key Value
--- -----
token s.4iqUXQvThDmpA9bTewOfjAuT
token_accessor QfDsotw0X18OuszEhXAS41Ka
token_duration 768h
token_renewable true
token_policies ["default"]
identity_policies []
policies ["default"]

VAULT_TOKEN=s.4iqUXQvThDmpA9bTewOfjAuT vault token lookup

Once you have a valid token, set /etc/salt/master.d/vault.conf to use this token and auth method: token

vi /etc/salt/master.d/vault.conf
# SDB CONFIG
myvault:
  driver: vault
# VAULT CONFIG FOR TOKEN OR APPROLE AUTH MODEvault:
  url: http://127.0.0.1:8200
  auth:
    method: token
    token: <TOKEN>
  policies:
    - default
 

SALT + Vault at large scale 

  • Performance issues and bottlenecks seen at large implementations due high number of requests to backend
  • Causes slowness rendering pillar process, perceived as slow response from Salt
  • Customers may want to tweak vault/sdb/pillar modules to alter behavior demanding less calls to Vault and adding some sort of caching in the middle
  • Data structure is also important, performance is quite different making 1 call to retrieve 100 secrets (many secrets in 1 key) than making 100 calls to retrieve 1 secret (1 secret per key).    Summary: less calls turns out in better perf.
  • Vault Agent Cache, new in version 1.1  https://learn.hashicorp.com/vault/secrets-management/agent-caching  (auto auth and caching), worth a try

 

Actual customer feedback:

    • Currently sdb.get makes two requests to fetch the data, first check expiry of token, then fetch data.  We implemented in-memory caching of vault token to skip the validation request
    • Checking token expiry and token fetching only on request failures
    • Upgraded 'disk pillar cache'  to be 'encrypted disk pillar cache' using AES; the key to decrypt the files needs to be retrieved from vault.  

Altogether this added functionality drastically decreased our pillar rendering time, where we use vault backend to retrieve values.

Before fix (no caching):

Initial rendering:  ~10 min

Subsequent rendering: ~10 min

 

After fix (encrypted caching):

Initial rendering:  ~40 sec

Subsequent rendering from cache:  ~2 sec

 

This new pillar caching requires the following changes in master.conf file:

pillar_cache: True

pillar_cache_ttl: 3600

pillar_cache_backend: disk

pillar_cache_secret: 'sdb://vault/path/to/your/pillar/decryption/key'

  

Edit section

Performance Test Cases 

Edit section

Substancial performance difference between each of the following 3 cases, depending on how you built the data in Vault and in pillar. For the same 100 secrets, it varies from 0.7 second to 4 seconds.

Case 1:

Vault loaded with 100 objects/different keys (diff access paths)

Pillar built by 100 individual pillar files (1 vault secret per pillar file).

MEASURED TIME for pillar.items:                   0m3.619s

SUMMARY: Worst case, Salt Master has a lot of overhead due to reading 100 pillar files and establishing many connections to Vault

 

Case 2:

Vault loaded with 100 objects/different keys

Pillar loaded by 1 pillar file (100 vault secrets in 100 pillar entries in 1 pillar file)

MEASURED TIME for pillar.items:                   0m2.703s

SUMMARY: Not optimal, Salt Master has less overhead (only 1 pillar file) but still many connections to Vault

 

Case 3:

Vault loaded with a single 1 key object (1 access path) with 100 secrets

Pillar loaded by 1 pillar file, it retrieves all the secrets at once from the single Vault object

MEASURED TIME for pillar.items: real 0m0.717s

SUMMARY: BEST CASE, less overhead due single pillar file and 1 Vault object to retrieve

Related articles