Optimizing Pillar: How to avoid Pillar overload
Abstract
This article describes common practices to reduce Pillar usage, a common issue found especially in large environments, including moving non-sensitive data from Pillar to map files (yaml, json, jinja files) imported by the states that require the data. In addition SDB can be used to move data away from Pillar to other sources.
The basics
If at this moment you are asking yourself what is Pillar,
Salt pillar is a system that lets you define secure data that are ‘assigned’ to one or more minions using targets. Salt pillar data stores values such as ports, file paths, configuration parameters, and passwords.
Pay attention to the word 'secure' in the previous definition, it is a key in understanding the goal of Pillar usage.
Pillar can be used to store any sort of data, but ideally it should be used primarily for sensitive data, due the additional security layer it provides.
For further information, please see these articles:
What's the problem
Pillar is great! So great that it is commonly overused, causing an overloaded Pillar which may lead to performance issues in your Salt Master servers, as the Pillar rendering process gets heavier and slower over time.
A common issue often seen is that Pillar is initially small and everything works great, as a limited group of people builds initial Pillar data, but over time, more and more people add Pillar data as more apps and services get managed via Salt, until Pillar grows so much that it gets out of control, at which point rendering issues become a problem.
Remember Pillar should be used mainly for sensitive data; if your Pillar is large, it is likely that much of the data in Pillar is not that sensitive.
The solution
The solution is simple: reduce Pillar load by moving non-sensitive data to other resources, usually map files accompanying, and imported by, state files.
At first, given Pillar has grown so large, it may seem to be a daunting task, so it's time to sit down and do a proper assessment. If possible, gather information from the people involved in Pillar data provisioning to understand what's there. Prioritize by criticality, sensitiveness, most commonly used, etc.
A simple assessment could be to find out Pillar state files with more data, running the following command from the pillar root directory (usually /srv/salt):
In this guide we will focus in standard Pillar, but don't forget to review any external Pillar too.
What's next
Let's take vault_config.sls pillar for this example. This data is used by a Vault state to install and configure Vault service. Let's review the content and classify what's sensitive and must remain in Pillar, and what data can be moved to map files.
Assuming the Vault state file is under /srv/salt/vault, put the map file in /srv/salt/vault/files/vault_defaults.yaml
Clear the pillar vault file, removing the data that was moved to map file:
Test
Conclusion:
The state executed successfully, retrieving all the required data from the map file. The map file has most values set locally except for the selected sensitive data svc_address and api_addr that remain in pillar.
The map file is the single point in which all the state data is defined, whether locally or from pillar, making the state file more flexible to changes in your data sources.
Next question would be, what else can we do to clean up pillar, can we retrieve secrets or sensitive information from other sources other than Pillar? The answer is yes.
For instance, you can move the data to an external source like a database and use SDB to retrieve the information.
How to use SDB to retrieve secrets instead of Pillar?
The SDB interface is designed to store and retrieve data that, unlike pillars and grains, is not necessarily minion-specific. The initial design goal was to allow passwords to be stored in a secure database, such as one managed by the keyring package, rather than as plain-text files. However, as a generic database interface, it could conceptually be used for a number of other purposes.
For detailed information on SDB and available modules, please visit:
- https://docs.saltstack.com/en/latest/topics/sdb/
- https://docs.saltstack.com/en/latest...all/index.html
Let's use SDB to interface with Hashicorp Vault to retrieve secrets in this example.
Salt SDB Vault configuration is intentionally out of scope of this article, look for other articles explaining this setup in this library. Please follow Salt and Vault documentation for further information. Adjust the steps as needed.
Start by provisioning the remaining pillar data to Vault:
Change pillar file to retrieve data from SDB, refresh pillar and get pillar items:
Conclusion: While pillar still in use, the source of data is in an external source retrieved to Pillar by SDB interface.
If the minions were allowed to reach the Vault server, the vault.conf file can be placed in the minions config, and let the minions use SDB to retrieve the secrets while rendering the state file locally, allowing a complete removal of Pillar data from the formula.
Change the /srv/salt/vault/files/vault_defaults.yaml map file to retrieve data from SDB for svc_address and api_addr
Remove the data from pillar.
Conclusion:
With this latest setup we have completely eliminated the use of Pillar by moving pillar data to map files and secrets to an external source consumed by SDB from minions instead of Pillar.