Troubleshooting sseapi_engine_sync "403 Forbidden" errors in salt master logs

Applicable versions: 6.1.0, 6.2.0, 6.3.0

This article explains how to troubleshoot and resolve sseapi_engine_sync "403 Forbidden" errors in the Salt master logs, and similar classes of errors such as an "Error getting raas lock" error message, various HTTP 401 and 403 responses, and potential other errors related to a master-to-raas connection.

The Problem

A message like the following is appearing in the salt master logs, possibly frequently.

2020-05-22 13:23:20,921 [salt.loaded.ext.engines.sseapi_engine_sync:396 ][ERROR ][22779] Failed to retrieve commands from SSE: 403 Forbidden

There may be a corresponding message in the raas logs:

2020-05-22 13:23:20,907 [pack.packed.rest.rpc][INFO :6 ][Webserver:1004] RPC Called: master.get_master_jwt
2020-05-22 13:23:20,920 [pack.packed.auth.master][INFO :6 ][Webserver:1004] Decryption failure

Explanation

This issue is typically caused by a mismatch between the public key being used by the master to authenticate to SaltStack Enterprise and the public key that has been accepted in SaltStack Enterprise for that master.

Resolution

(Note: You may need to preface these commands with sudo if running as a non-root user.)

1. Stop the salt master:

   systemctl stop salt-master

2. On the salt master, remove the public key for SSE authentication:

   rm /etc/salt/pki/master/sseapi_key.pub

3. In SSE, under "Administration"/"Master Keys" ("System Administration"/"Master Keys" in 6.1.0) delete the key for the associated master:
   
4. Start the salt master:

   systemctl start salt-master

5. In SSE, the new master key will show up in Pending; accept it: