Troubleshooting "Connection reset by peer" or "413 Request Entity Too Large" in Salt Master logs

This article explains how to troubleshoot and resolve a "Connection reset by peer" or "413 Request Entity Too Large" error message as part of a log message containing sseapi in the logged function.

The Problem

Messages like the following appear in the Salt Master logs, possibly frequently.

2020-04-24 15:34:09,018 [salt.loaded.ext.engines.sseapi_engine_sync:524 ][ERROR ][87781] Failed to send master fileserver data to SSE: 413 Request Entity Too Large

2020-04-24 14:28:52,249 [salt.loaded.ext.engines.sseapi_engine_sync:396 ][ERROR ][50] Failed to retrieve commands from SSE: [Errno 104] Connection reset by peer
2020-04-24 14:28:52,249 [salt.loaded.ext.returner.sseapi_return:207 ][ERROR ][153] Failed to save events: [Errno 104] Connection reset by peer

Explanation

Although "Connection reset by peer" can be caused by various network issues, a common cause of these messages in a SaltStack Enterprise environment is a Salt Master attempting to send a message to the SaltStack Enterprise API that exceeds it's configured maximum body size.

An "HTTP 413" message pinpoints more directly that a message is too large for the SaltStack Enterprise API.

Resolution

On the SaltStack Enterprise server, edit /etc/raas/raas and find the webserver_max_body_size setting. Its default is empty, which effectively sets it to 100MB. The setting is denominated in bytes. To raise this limit to 256MB, for example, modify the configuration as follows.

# maximum amount of data for body, when None uses tornado default
webserver_max_body_size: 268435456

Restart raas with systemctl restart raas for the change to take effect.