Hear me out...

When I first saw the movie The Matrix, in the early scene where Neo is called into his boss's office and told that if he behaves well, he has a future at that no-name Corp., or otherwise he'll be fired, I joked and laughed, like, "Yeah, sure, a tech professional could just walk out into the opposite building, and they would beg him to join since everyone is looking to hire engineers."

Today, this scene is closer to reality than I'd like to admit. I personally know Senior DevOps engineers who are solid folks, and it took them 8 months to land a job at a salary that sucked. Our industry is not regulated, the barriers to entry and outsourcing are being lowered by AI, and we have a massive oversupply of engineers. This is just a reality.

Before tech, I worked in another industry that also experienced a major decline. I saw it coming and made a decision at the right time, before it became obvious. I moved to tech years ago and was lucky enough to become a staff engineer. Now I see the same pattern. I think we have 5 to 10 years before salaries in this field will be $70k-$80k for Senior Engineers in the US (which is around a 2x decline) you can do the math for other countries.

My advice is work on a backup plan now. Invest, save, but also learn a trade. I'm studying medicine this year. Fuck it. Six years of study while working remotely, and I'm opening my own dentistry clinic. I'm in my late 30s, but it's still better to have a plan. My wife is also changing careers to become a registered nurse.

This is my plan. You might think of something else, but my point is that it was a good ride. We did a lot of nice work and, IMHO, got easy money, but it's over, and it's time to move on.

GDG on Campus IIIT Nagpur. We're running **KEEP ALIVE '26**, a national incident-response hackathon for students, at TantraFiesta 2026.

**What's different:** you don't build an app. Your team gets a broken microservices system's alerts, Grafana dashboards, logs and traces, works out what broke and why, and presents the fix. It's what an on-call SRE does on a bad night.

- **Who:** UG, PG and diploma students, teams of 2–3 from the same college

- **Level:** 6–12 months of DevOps learning; no production experience needed

- **Round 1:** a 30-minute online team assessment on Unstop, open till Sun 18 Oct, 11:59 PM IST

- **Finale:** top 15 teams at IIIT Nagpur on 26 Oct

- **Prizes:** ₹50,000 prize pool (₹25,000 cash + goodies for every finalist)

- **Partners:** Grafana (Official Platform Partner), CNCG Noida (Community Partner)

Register: https://unstop.com/hackathons/keep-alive-26-iiitn-1766226

Happy to answer anything in the comments, including how the incident platform works.

I've recommended The Site Reliability Workbook to a lot of people, and most never get past chapter 2. So I built an interactive study guide that covers the core ideas in about 50 minutes:

\- SLIs, SLOs, SLAs and error budgets, each explained in plain language first
\- An error budget simulator: trigger incidents and watch the budget policy kick in
\- A burn-rate explorer showing which multiwindow alerts would fire
\- Toil vs. engineering sorting, incident command roles, blameless postmortems, canarying, and how SRE teams engage and grow
\- A checkpoint at the end of each module, plus a final cheat sheet and a 30-day starter checklist

It's a single HTML page. No account, no tracking, and progress saves in your browser. It's an independent study guide, not affiliated with Google or O'Reilly, and it links to the free official version of the book.

Link: https://jazeer888.github.io/field-courses/
Repo (MIT code, CC BY 4.0 content): https://github.com/jazeer888/field-courses

Full disclosure: I wrote it with help from Claude and reviewed it against the book, but I'd really value corrections from people who run SRE in production. If anything is wrong or oversimplified, tell me here or open an issue and I'll fix it.

I’m a infra engineer at a UK tech company,. My contract title is Systems Engineer: infrastructure, DevOps, systems. I was not hired as a software developer.

From beginning of this year, I built a full software platform for the company on my own initiative, with no support from anyone. I did it largely out of hours, often around 12 hours a day, 7 days a week, to hit deadlines. I paid for the agentic AI tools myself and was never reimbursed. The platform is now a live company product, and stuff worth several million are lined up to run on it.

I asked for compensation. The company first said it couldn’t pay anything: no raise, regrade, bonus or equity. They told me UK law says anything done under employment belongs to the company. After I said I’d step back and focus on my hired role, they said they want me to continue. They’ve now offered a salary review “in a couple of months” plus share options, with a written proposal to follow.

Relevant contract clauses:

• IP clause: all “Materials” created “in the course of employment… or otherwise relating to the business of the Company” vest in the company, with an assignment of future rights.
• “You may be required to perform other duties not within the scope of your normal duties.”
• Extra hours are expected “without any additional compensation.”
• Expenses are reimbursed only with pre-approval.

My questions:

1. Does the “otherwise relating to the business” wording really capture work outside my job role, done on my own time? Or could it be challenged as too broad? I’m aware of Noah v Shuba vs Missing Link v Magee.
2. Is there any route to compensation for copyright or software under UK law? I understand Shanks v Unilever / Patents Act s.40 only applies to patents.
3. What should I check or push for in the share options offer: percentage, strike price, vesting, leaver terms?
4. Is it worth paying for a solicitor at this stage, and what kind (employment or IP)?

Not looking to burn bridges, just want to understand my position before I respond to their written offer. Thanks.

Hi, I'm Sergio, author of kubelatch.

After years handing out Kubernetes clusters across clouds, I kept seeing the same thing: admin kubeconfigs and random ServiceAccount tokens copied onto laptops and into CI, and audit logs where everyone is admin (or even worse, no audit logs at all).

AI agents made it worse. Claude Code needs the cluster to debug a pod, and if I give it my admin kubeconfig, nobody can tell what I did from what it did.

So I built kubelatch, an impersonating proxy in front of your clusters' API servers:

• Each person gets their own token that expires on its own (30 days max by default, configurable by settings) and can be revoked instantly.
• GitHub Actions swaps its OIDC token for a 1-hour credential, no secrets stored.
• Agents connect through an MCP server with scoped permissions, and every write waits for your approval with the diff in front of you.
• Every request in the audit log has a real name on it.

Heads up: it's not open source. It's free for 2 clusters and 5 people, and it already runs in production. I'm looking for people to break it, and the local kind tutorial takes about 15 minutes.

https://kubelatch.com
Local tutorial: https://docs.kubelatch.com/getting-started/local/
Repo: https://github.com/picaportelabs/kubelatch

How are you handling cluster access right now? Has anyone wired an agent to a real cluster yet?

Generating terraform for a console created resource is useful, getting it into the modules platform team actually maintains where things get interesting.

Firefly docs say it can generate calls to existing terraform modules along with import blocks. That caught my attention because a separate pile of generated HCL for every resource would still leave plenty of clean up.

Take an existing S3 bucket with a policy the approved module doesn’t expose, or a security group shared with resources in another stack. Capturing the current configuration gets you part of the way, deciding which module should own it and reviewing any changes needed to fit that module still matter.

For anyone who used firefly for this would be interested in the work between generating the code and merging the PR. Particularly module inputs that need adjusting and dependencies managed elsewhere and anything unexpected in the plan.

A walkthrough of one resource going from unmanaged to maintained in an existing module would be helpful.

Honestly,

9/10 of theses roles are getting outsourced. We are screwed and when you do land a role you get laid off a year later so you have to save the good salary incase it takes a while to get another role.

Skyportal's agent asks before it runs a command on your hosts, unless the command is on an allow list. It also has a built-in check that guesses whether a command changes data, and that guess beats the allow lists. This week it flagged a read-only command as a file overwrite because of a 2>&1;, so no allow rule could have approved it.

So we're removing the guess. A command will run without asking only if every part of it is on an allow list and nothing matches a deny rule. Everything else asks. A few things always ask, whatever the rules say: credential files, secrets, RBAC objects, kubectl exec, and deletes you can't undo. And if the rules file is broken, everything on that repo asks.

The cost is that a broad entry like sudo will mean exactly that, with nothing behind it to catch sudo rm. If you've put policy in front of automation on hosts, did you keep a "looks dangerous" check as a backstop, or trust the allow list and review it like code?

When you're reviewing a Terraform PR you didn't write, the plan output is only part of the work. You still need to understand how the affected resources fit into the rest of the infrastructure.

I built Rootform to make that easier.

It's a local CLI that reads Terraform and OpenTofu plan or state exports and produces an interactive architecture you can explore, compare and check against policies.

In one of the sample plans, an Event Grid subscription switches from one Service Bus queue to another. Rootform shows the old and new delivery relationships alongside the unchanged source topic. You can inspect the evidence behind those relationships instead of treating every Terraform dependency as an architectural connection.

The part I think is particularly useful for platform teams is the output: you can export the entire interactive review as a single HTML file. A teammate can open it in their browser without installing Rootform or running another service.

It also integrates with GitHub Actions for CI workflows, including architecture comparisons and policy checks. The analysis runs locally, without executing Terraform or accessing your cloud accounts.

Interactive Playground (no signup): https://rootform.dev/playground/

GitHub: https://github.com/rootform-dev/rootform

For those managing Terraform across multiple teams: how do you currently give reviewers the architectural context behind a PR? Do you rely on plan comments, documentation, diagrams, or something else?

#devops #terraform #platformengineer

Our canary deployments catch obvious breakage, but subtle behavior changes often only show up when real traffic hits edge cases that a small canary percentage rarely sees. Increasing the canary percentage might help catch more of the issues, but it also increases the blast radius if something goes wrong.

Our team have hud running on the canary pods, which helps us spot changes in function behavior before the overall error rate starts moving. The tricky part is knowing when there is enough traffic to trust the comparison and when a difference is worth rolling back over.

For teams running canaries in production, how do you handle this? Do you rely on traffic thresholds, behavioral comparisons, or a mix of both? Also curious what signals have led to unnecessary rollbacks and what subtle regressions were hardest to catch.

When something breaks, the actual workflow usually looks something like:

- An alert fires.

- You jump into logs.

- Check the database.

- Check deployment status.

- Figure out what changed.

- Open another dashboard.

- Make a fix.

- Test it.

- Deploy.

- Verify that the incident is actually resolved.

The problem isn't necessarily that any individual tool is bad.

The problem is the context switching between them.

This reel shows a simple production incident: a database connection pool gets exhausted and errors begin appearing.

The incident moves through the workflow:

Incident open → fixing → shipping → resolved

In the scenario shown, Escanor gets from the alert to the resolved state in 43 seconds.

That's the direction we're building toward with Escanor: an engineering operating system where the infrastructure around your applications is connected instead of scattered across a collection of dashboards and tools.

We're still building it, so this isn't a claim that every production incident can magically be solved in 43 seconds. The number in the video represents this particular workflow/scenario.

The bigger idea is:

Production engineering shouldn't feel like searching through ten different systems just to understand what happened.

Escanor is our attempt to bring that workflow into one connected place.

Would love to hear from engineers/SREs: what's the most annoying part of handling a production incident in your current stack?

#DevOps #SRE #Infrastructure #DeveloperTools #SoftwareEngineering

I've been in the field for more than 15 years. I started as a SysAdmin, then moved into DevOps. Platform Engineering as a concept sounds interesting, but most of what I read ends up the same way: "and then we connect all of this to Backstage."

That doesn't match what I'm seeing. Our engineers don't want a web portal for provisioning infrastructure. They want to do it from their LLM of choice. Meanwhile, Security wants full traceability: who requested what, which identity made the change, and through which path.

So the interface seems to be the least important part, and the value has to sit underneath it.

For those of you doing platform work without Backstage (or without any portal): what does your platform actually consist of, and how do you show its value at work?

As developers, we often end up working across multiple tools to manage an application and its infrastructure.

One platform for source control, another for deployments, separate dashboards for cloud resources, databases, logs, and monitoring. Each tool may work perfectly well on its own, but switching between them adds friction, especially when you're trying to understand what's happening across an entire system.

That got us thinking: what if infrastructure management could feel like one connected experience instead of a collection of disconnected dashboards?

That's the idea behind Escanor, a project I'm building.

Our goal is to create a unified infrastructure management experience that helps developers interact with and understand their infrastructure without constantly jumping between tools.

We're approaching this as an engineering problem, not just a dashboard redesign. The challenge is making integrations, infrastructure visibility, and operational workflows work together while preserving the security and reliability that production systems require.

We're still building and refining the product, so I'm not going to claim we've solved every part of this problem. There are real challenges around permissions, integration reliability, observability, and safely executing infrastructure operations.

I'd genuinely like to hear from people who manage production systems:

\- How many tools do you typically use to manage your infrastructure?

\- What's the most frustrating part of switching between dashboards?

\- Would you find a unified infrastructure interface useful, or do you prefer keeping each provider's native tooling separate?

\- What would you need to see before trusting a platform like this with production infrastructure?

I'm the founder of Escanor, and I'm interested in hearing honest technical feedback — including reasons why this approach might not work.

I can imagine removing almost every retry loop and SDK-specific integration from an application and still ending up with the wrong abstraction.

The distinction I'd draw is between a mechanism and the meaning of an operation.

Connection management, credential retrieval, transient backoff, and common telemetry are reasonable things to standardize. Deciding whether an order can be submitted twice, whether a stale result is acceptable, or what compensates for a partial update depends on the business operation.

That's what interests me about the Dapr/Diagrid approach: APIs and policies can make repeated infrastructure work a shared capability. I don't take that as a reason to hide every failure mode behind "the platform handles it."

For example, a platform can own a pub/sub integration while the application owns the event contract and duplicate handling. It can persist a workflow while the application defines when the process is complete. It can apply a timeout without knowing whether the remote action still needs reconciliation.

I'd want those responsibilities documented at the boundary, with enough context in errors and traces to understand what happened underneath. Otherwise the code gets shorter but the incident requires more people to explain it.

The abstraction I'd trust is one that makes ordinary work easier and unusual failures understandable. Where have you drawn that line successfully, and which piece of infrastructure logic did you deliberately keep in application code?

Three layers each allowing three total attempts can turn one logical operation into 27 downstream attempts. That's the example I keep in mind whenever "we added retries" sounds like an uncomplicated reliability improvement.

I don't think there's one correct layer for every retry, but I do want one understandable budget.

A client might handle a brief connection failure. A queue might redeliver a job after a worker dies. A workflow might retry an activity after a longer wait. The business operation still needs to define whether repetition is safe and when another attempt stops making sense.

The separation in Dapr and Diagrid Catalyst is useful to look at here: call-level resiliency policies and workflow activity retries have different scopes. They can also compose. Moving retry configuration out of business code doesn't automatically prevent multiplication.

The questions I'd want answered are: does the end-to-end deadline survive each hop, how many attempts can actually reach the dependency, and does the operation keep the same identity across those attempts? Backoff and jitter help spread retries out; they don't make an unsafe write safe.

I'd lean toward a shared platform default for transient infrastructure failures, with application-owned rules for business recovery. But those two policies have to be visible together.

How are you making that practical? Central retry budgets, a designated retry-owning layer, or explicitly disabling retries everywhere except one place?

Hi everyone,
I’m looking for some advice from people working in software development, cloud engineering and platform engineering in the UK.
I’m currently working as a Senior/Lead .NET Developer at a large financial services company. My main experience is with C#, .NET applications, AWS serverless services and infrastructure using AWS CDK. My current role is relatively comfortable, with a reasonable work-life balance, but I’m starting to feel that my career growth is somewhat limited by the opportunities available within my existing team.Also a lot of development work is just small enhancements so am not really using most of my skillset and always have this insecurity of not being a full stack dev .
I’m considering an internal move to a Platform Engineering/Team Lead role with a salary of around £65k.
From what I understand, the role would involve:
Managing and mentoring a small team of developers/engineers.
Supporting a critical, approximately 10-year-old legacy application.
AWS infrastructure maintenance, server patching and AMI updates.
Managing Jenkins pipelines, deployments and occasional Terraform changes.
Production support, incident management and root-cause analysis.
Working with third-party teams responsible for application enhancements.
It sounds like there would be less hands-on application development than in my current role, and potentially some on-call responsibilities. There would also be an opportunity to learn more about infrastructure, operations and people management.
My main question is whether moving towards cloud/platform engineering is a better long-term career move than continuing down the .NET development and technical leadership path.
A few things I’d particularly appreciate your thoughts on:
Job demand: Are Cloud Engineers and Platform Engineers currently in greater demand than experienced .NET developers and .NET Lead Developers in the UK?
Career progression: Which path generally offers better opportunities over the next 5–10 years?
Technical skills: Would this particular role help me move towards Cloud Architect, Platform Architect or Engineering Manager positions, or could the operational and legacy-system focus limit my future options?
Salary: Is £65k reasonable for this kind of role, considering the team leadership and production responsibilities?
Work-life balance: How demanding are platform roles with production support and on-call responsibilities compared with senior software development roles?
I have a young family, so work-life balance matters to me, but I also don’t want to stay too comfortable and find that my career has stagnated a few years down the line.
Would you make this move, or would you focus on progressing towards a .NET Technical Lead/Architect role while continuing to build cloud skills?
I’d appreciate honest advice, particularly from people who have moved between software development and cloud/platform engineering.
Thanks!

I like the idea of publishing a fact once and letting independent consumers react to it. What worries me is using events to make dependencies less visible rather than less important.

Suppose an order event eventually triggers inventory reservation, billing, and a notification. The producer no longer waits on three HTTP calls, but somebody still needs to know when the business process is complete, which consumers are behind, and what to do when only two of them succeed.

There's also a difference between replaying events to rebuild a read model and replaying them through a consumer that sends emails or triggers another external action. The same retained event can be harmless input in one context and a duplicate side effect in another.

A common API for pub/sub can simplify integrations. That's the part of Dapr and and Diagrid Catalyst I find interesting. It doesn't make broker-specific ordering, retention, or delivery behavior irrelevant, and it doesn't establish the meaning of an event for us.

For a platform team, I'd prioritize a visible event contract, ownership, and a safe replay procedure before making it effortless to add another topic.

My concern isn't that event-driven systems are a mistake. It's that "decoupled" can become the new "independently deployable": technically true at one layer while business changes still require coordination everywhere. How do you make those remaining dependencies visible?

If several services still have to be deployed together, changed by the same people, and debugged as on unit, I'm not sure what the network boundaries bought us.

Independent scaling or failure isolation could still justify them. But "we have separate repositories" doesn't sound like enough on its own.

The test I'd use is a routine product change. How many teams, deployments, schema agreements, and rollback plans does it require? If splitting a module that ordinary change slower without improving a concrete operating constraint, I'd rather keep the boundary inside a modular monolith.

A platform can change the cost calculation. Shared service invocation, state access, and operational tooling can reduce repeated integration work. Dapr and Diagrid Catalyst are examples of that approach. What they can't do is decide that inventory and billing have the right ownership boundary, or make a cross-service business invariant disappear.

That distinction matters to me because it's easy to buy better plumbing for an awkward architecture and call the architecture fixed.

From a platform-team perspective, would you first invest in making service creation cheap, or in making it harder to create a service without a reason? I'd be interested in a case where consolidating services improved delivery, even though the infrastructure team could already operate them reliably.

For context we built a golden path for aws infra so teams can spin up envs and services without touching tickets. Modules, tagging standards, default regions, sane iam, the usual platform engineering stuff. Adoption has been good, people like it.

Now that usage is higher, IaC governance is getting weird. We are seeing stacks in random regions, modules forked in private repos, owner tags missing, and folks tweaking security groups in the console because the paved road doesnt cover their edge case. None of it is huge on its own, together it feels like drift we did not sign up to babysit.

If your org has strong developer self service, where do you put the real infrastructure guardrails, in the modules, in ci checks, in cloud side policies, or somewhere else, and who is allowed to break them without the platform team turning into a change board again? appreciate any thoughts

I think the history from traditional Ops to DevOps and then Platform Engineering is best understood as a dialectic driven mostly by technological change.

Traditional Ops concentrated infrastructure knowledge in a specialised team because operating infrastructure required direct administration. Automation, cloud and infrastructure as code changed that. DevOps emerged as the antithesis: operational ownership could move closer to development teams.
The operational work did not disappear, though. Much of its complexity was simply redistributed to developers.

Platform Engineering became the synthesis once cloud APIs, declarative systems and orchestration made it possible to encode operational knowledge into software and expose it through self-service. Operations could specialise again without returning to tickets and manual handoffs.

This is why I do not think the conceptual distance between the old sysadmin and the platform engineer is that large. The function is broadly similar. What changed was the technology through which that function could be organised and delivered.

AI may now be starting the next movement in the same dialectic. Agents do not necessarily need portals, fixed workflows or abstractions designed primarily for humans. They can discover tools, combine APIs and construct their own path through a system.

But this does not remove the need for a platform. It changes what the platform must provide. Instead of defining only human-oriented golden paths, it increasingly needs to expose machine-readable contracts, capabilities, relationships, constraints and evidence.

The next stage of Platform Engineering may therefore be less about building interfaces that hide complexity from developers, and more about defining the operational space within which agents can act without having to guess.

I've spent the last three years on the platform team at a pretty boring old corporate, building their internal workload platform. No SLAs and not much real impact, but the Kubernetes work itself was interesting (we ran a managed control plane though).

Now I have an offer from a fast-growing scale-up as a DevOps engineer, with a 60% pay increase. The role is mostly self-hosted pipelines, code scanning and similar tooling, all running on Kubernetes with lots of potential for impact.

My question: if I ever want to go back to Kubernetes work, will that still be an option, or will I have branded myself as a "DevEx guy"? I'd still be working with K8s every day, but more as the thing that runs the tools rather than something I'm building a platform abstraction on top of.

hey yall,

im a lead SRE at a global fortune 500 that you have heard of haha - i'm a bit of a lurker here

but just wanted to talk about runbooks and documentation for SOPs.

my feeling is that confluence docs kinda suck and runbooks are kind of a mess, our operators are jumping between the docs and their shell, the docs are often times missing a bunch of detail, they are brittle, poorly maintained, half the time the details are hidden in tribal knowledge and when dealing with the pressure of an active inc we notice the pain even more. extracting critical details & commands out of an inc can also be painful and doesnt always translate to improved operational readiness next time.

i admit we're not very mature.. but was curious if its only me feeling like this?

what are you guys doing with runbooks to solve these issues?

I've been working in software/infrastructure engineering for around 14 years, with a focus on cloud, DevOps, platform engineering and large-scale infrastructure.

Over the years I've worked on everything from smaller cloud migrations to enterprise and financial-sector environments, including AWS/GCP migrations, Kubernetes, Terraform, CI/CD, networking, security, reliability and cost optimization.

I'm currently putting together the practical knowledge I've accumulated into a free technical book.

I'm not planning to make it a personal career story or another basic “learn AWS/GCP” tutorial.

The idea is to focus on how engineers actually solve infrastructure problems:

• How to systematically debug production issues
• How to troubleshoot Kubernetes
• AWS/GCP networking and connectivity problems
• IAM and permission failures
• Terraform/IaC problems
• CI/CD and deployment failures
• Cloud migration problems
• Scaling and reliability
• Observability and incident investigation
• Designing reusable platform infrastructure
• How to approach an unfamiliar production system
• Practical lessons that aren't obvious from vendor documentation

I'm also building some small prototypes/labs for myself while writing, so I can test the ideas rather than just writing theory.

Before I spend a lot of time putting the whole thing together, I'd like to hear from other engineers:

If you could have one practical DevOps/Platform Engineering book that focuses on real problem-solving rather than certification theory, what topics would you want it to cover?

And if you already have a favorite resource for this kind of material, I'd be interested in hearing what you think it does well or what is missing.

I’m trying to understand what companies actually expect from a **Staff / Sr Staff Platform Engineer** at the ₹1Cr+ compensation level in India.

For people currently in these roles or hiring for them:

**What does your day-to-day work actually look like?**

How much of the job is **hands-on coding** vs architecture/design/reviews?

How much **system design** is expected at Staff level?

Are Staff Platform Engineers expected to be strong in **distributed systems**, or is deep infrastructure/platform expertise more important?

If you’re currently a Staff+ Platform/Infrastructure Engineer in India, I’d especially appreciate hearing:

**What were you doing before becoming Staff, and what changed in your responsibilities after reaching Staff level?**