r/Backend • u/Innowise_ • 9h ago
We made a production incident worse by restarting the services
We had a fun one on a mobile banking support project. Not fun at the time, obviously.
About 40 minutes after a release, transfers started slowing down. Then came the 502s. It was payday too, so traffic was already higher than normal.
First reaction was to restart the services.
Bad call.
Every restart dropped the warm database connections, and the load spiked again. We were basically making things worse while trying to fix them.
Turned out the release had changed the DB connection pool settings. Once we found that in the deployment logs and rolled back, things settled down within a few minutes.
The other mistake was communication. The engineer working on the incident was also trying to answer questions from the bank's operations team, which had support lines filling up.
We changed that afterward. One person handles the incident, another handles updates. Even if the update is just "still investigating."
Curious if anyone else has had a seemingly harmless recovery step make an outage worse.