Troubleshooting "Connection reset by peer" or "413 Request Entity Too Large" in Salt Master logs
This article explains how to troubleshoot and resolve a "Connection reset by peer" or "413 Request Entity Too Large" error message as part of a log message containing sseapi
in the logged function.
The Problem
Messages like the following appear in the Salt Master logs, possibly frequently.
2020-04-24 15:34:09,018 [salt.loaded.ext.engines.sseapi_engine_sync:524 ][ERROR ][87781] Failed to send master fileserver data to SSE: 413 Request Entity Too Large
2020-04-24 14:28:52,249 [salt.loaded.ext.engines.sseapi_engine_sync:396 ][ERROR ][50] Failed to retrieve commands from SSE: [Errno 104] Connection reset by peer
2020-04-24 14:28:52,249 [salt.loaded.ext.returner.sseapi_return:207 ][ERROR ][153] Failed to save events: [Errno 104] Connection reset by peer
Explanation
Although "Connection reset by peer" can be caused by various network issues, a common cause of these messages in a SaltStack Enterprise environment is a Salt Master attempting to send a message to the SaltStack Enterprise API that exceeds it's configured maximum body size.
An "HTTP 413" message pinpoints more directly that a message is too large for the SaltStack Enterprise API.
Resolution
On the SaltStack Enterprise server, edit /etc/raas/raas
and find the webserver_max_body_size
setting. Its default is empty, which effectively sets it to 100MB. The setting is denominated in bytes. To raise this limit to 256MB, for example, modify the configuration as follows.
# maximum amount of data for body, when None uses tornado default
webserver_max_body_size: 268435456
Restart raas
with systemctl restart raas
for the change to take effect.