r/Backend • • 6h ago

We made a production incident worse by restarting the services

0 Upvotes

We had a fun one on a mobile banking support project. Not fun at the time, obviously.

About 40 minutes after a release, transfers started slowing down. Then came the 502s. It was payday too, so traffic was already higher than normal.

First reaction was to restart the services.

Bad call.

Every restart dropped the warm database connections, and the load spiked again. We were basically making things worse while trying to fix them.

Turned out the release had changed the DB connection pool settings. Once we found that in the deployment logs and rolled back, things settled down within a few minutes.

The other mistake was communication. The engineer working on the incident was also trying to answer questions from the bank's operations team, which had support lines filling up.

We changed that afterward. One person handles the incident, another handles updates. Even if the update is just "still investigating."

Curious if anyone else has had a seemingly harmless recovery step make an outage worse.


r/Backend • • 17h ago

How are your companies actually adopting AI? Are we just too slow in banking?

13 Upvotes

I'm a software developer working at a bank, and I feel like we're way behind when it comes to AI adoption.

Every time AI comes up, the conversation is mostly around security, compliance, data privacy, etc. I understand these are important, especially in banking, but it feels like we're moving extremely slowly.

For example, our bank deployed a GPT model through Azure AI Foundry and built an internal ChatGPT-like interface for employees. That's pretty much the extent of AI adoption I've seen so far.

Meanwhile, I keep seeing people talk about using Claude Code, Cursor, AI agents, and building AI-powered features using LLM APIs.

I'm curious how things work at other companies.

• Are you actually building AI-powered features using APIs from OpenAI, Anthropic, etc.?

• How does your company approach new AI initiatives? Do developers get to experiment and propose ideas, or does everything have to go through multiple approval processes?

• How are you handling security and compliance concerns when integrating AI into existing applications?

• Are companies actually using AI agents and automation in production, or is most of it still experimentation?

I'm especially interested in hearing from developers working at banks, fintech companies, or other large enterprises.

I want to understand what real-world AI adoption looks like beyond just building an internal ChatGPT wrapper.

How is your company approaching this? Are you seeing meaningful AI adoption, or is it mostly hype and internal chatbots?


r/Backend • • 8h ago

Java Backend Developers — What Problems Do You Wish Someone Would Solve?

0 Upvotes

​

Hi everyone,

I'm looking for a project idea that solves a genuine problem faced by developers, engineering teams, or businesses.

I want to build something beyond a standard CRUD application, with a strong focus on backend engineering.

I'm interested in exploring areas such as:

- API design and reliability

- Distributed systems and event-driven architecture

- Database performance and data consistency

- Developer productivity tools

- Monitoring, debugging, and automation

I'd particularly appreciate suggestions based on problems you've personally experienced at work or in your own projects.

Three questions:

  1. What repetitive or frustrating technical problem do you face regularly?

  2. What tool or workflow do you wish existed, or worked better?

  3. If someone built a small, reliable solution to this problem, would you actually use it?

I'm open to using Java or another language if the problem calls for it.

Thanks!


r/Backend • • 10h ago

PHP incidents - your credentials might have been hacked

Thumbnail
0 Upvotes

r/Backend • • 19h ago

Opiniones sobre mi CV de Backend

Post image
0 Upvotes

r/Backend • • 15h ago

Will the recent PHP security vulnerability impact the PHP job market?

Thumbnail
0 Upvotes

With the recent PHP security vulnerability making headlines, I'm wondering how much impact this could have on the PHP job market in the long run.

I'm currently working as a PHP developer, and with all the AI disruption already happening in IT, news like this makes me question the future of the tech stack even more.

Could this push companies towards other technologies, or is it just another security issue that will eventually get patched and forgotten?

Would love to hear from experienced PHP developers and people involved in hiring. Are you seeing any changes in demand for PHP developers? Is it still worth investing in PHP/Laravel, or should developers seriously start preparing to switch stacks?


r/Backend • • 20h ago

Web App are overrated, Deploy django in VMs ($8 - $16) are enough

Thumbnail
0 Upvotes

r/Backend • • 3h ago

becoming a backend dev

2 Upvotes

hi all! I am interested in becoming a backend dev as I like the logic side and problem solving aspects.

Is learning the JS, MySQL/PostgreSQL (or MongoDB), and Node/Express enough to get started in that field?
I have taken a bootcamp for full-stack web development and I really liked the backend side of things and would like to pursue that but I am not sure with junior roles becoming more scarce.

Thank you!


r/Backend • • 18h ago

Timeout and retry math: why a 2-second timeout can turn into a 6-second request

0 Upvotes

A small calculation that changed how I look at timeouts. Take a request that has to answer the user in 3 seconds.

Step 1. Your service calls another service, and you’ve set a 2-second timeout on that call. That looks safe, since 2 is less than 3.

Step 2. The call is wrapped in a retry, with up to 2 retries after the first attempt. A timed-out attempt is retried.

Step 3. Worst case, all three attempts time out:

worst case = timeout × (retries + 1)
           = 2s × (2 + 1)
           = 6s

The user’s budget was 3 seconds. In the worst case you spend twice that waiting on one dependency, and you haven’t started counting the rest of the request yet.

Step 4. Now add a second dependency with the same settings. If both are called one after the other and both are slow, the worst case is 12 seconds. Each setting looked reasonable on its own, and the total is nowhere near the budget.

A few things this seems to imply:

  • timeouts and retries have to be chosen together, from the outside in: start with the budget the caller gave you, then divide it
  • a retry is only useful if there’s time left to make it, so a deadline that is passed along and checked before each attempt works better than a fixed timeout per call (I believe gRPC propagates deadlines like this, but check your stack)
  • retrying a call that’s already slow because the dependency is overloaded makes it worse. A small cap on retries, with a bit of random delay between them, helps avoid everyone hammering it at once
  • retries are only safe for operations that can be repeated, so idempotency comes first

One more thing to check is what your HTTP client does by default. As far as I know, several common Java clients have no timeout at all unless you set one, which means the “worst case” can be unbounded. I’d look this up for the exact client and version you use, since defaults vary.

What I’m not sure about is how to split a budget across calls that run in parallel versus in sequence, and whether people set it by hand or compute it somewhere central.

How do you handle it in your services? Do you have a standard timeout and retry policy, or does each call get decided where it’s written?


r/Backend • • 23h ago

How many backend services is too many for one AI product?

0 Upvotes

I'm curious where backend folks draw the line between "best tool for each job" and just too much operational surface area.

A typical AI app can easily end up with Postgres/document DB + vector DB + auth + object storage + search + LLM gateway + OCR/document service + workers. Each component is reasonable by itself. Together, a small product suddenly has a platform team's architecture.

That pain is why I started Singlebase - one backend API for the common app + AI primitives. I'm not arguing everything should literally be one database/process. More that the developer-facing surface can be unified while implementations stay specialized underneath.

Would you rather have one opinionated backend abstraction, or keep each system explicit?