Building SRE from zero

How I stood up a Site Reliability Engineering function from nothing in my first six months, starting with no SLOs, no runbooks and a noisy pager.

• Role: Head of Site Reliability Engineering
• Timeline: first six months in the role
• Team: grew from 2 to 4
• Stack: Kubernetes on AWS, Grafana-based observability, incident.io, GitLab

The six months in three numbers

• 0 to 20 service tiers with SLOs and error budgets
• 0% to 56% of alert rules linked to a runbook
• 2 to 4 people on the SRE team, now a one-in-four on-call rotation

The starting point

When I joined, there was no SRE practice. One SRE had been hired, but nothing had started. The tools were there: Grafana for metrics and incident.io for incidents. What was missing was everything that turns tools into reliability.

There were no SLOs, so nobody could say what “reliable enough” meant. The incident tool was noisy, with almost all of its alerts coming from a single source. Not one alert rule linked to a runbook, so every incident depended on whoever happened to know the system. And nobody had measured time to recovery, so there was no way to tell whether things were getting better or worse.

How I ran it

I treated the function like a product with a roadmap. At day 120 I wrote down six commitments and the gaps I’d rather show than hide. Leadership approved a six-month roadmap. At six months I reported against both, with every number pulled from the systems, not recalled, including the ones that didn’t go my way.

What we built

• Reliability targets. SLOs with burn-rate alerts and error budgets for 20 service tiers: customer-facing services, data and messaging systems, and the monitoring stack itself. Each one went through the same lifecycle: define the SLI, agree the target with leadership, deploy the rules, then add a dashboard and a runbook.
• Alerts that point somewhere. The alert rules were reorganised by component and labelled so an incident lands on the right service. More than half of them now link to a runbook, up from none.
• Less noise. Two tuning changes cut the noisiest alerts by about 99%. One of them came from an engineer eight days into the job, using historical data to reset a threshold.
• An on-call people can live with. On-call grew to a one-in-four rotation with a weekly written handover, and no page has gone unanswered.
• Postmortems with owners. Our serious-incident postmortems now share one structure (summary, impact, contributing factors), and the latest add what went well, what was hard, and an action table with owners.
• Routine checks, automated. The daily cluster health check runs on a schedule in CI with a results history, and releases now get a health check before and after deploy, based on live error budgets.

The team did most of the building. Almost nine in ten changes to the alerting rules came from them, not me. My job was to set the direction, agree the targets and clear the way.

What changed

Monthly alert volume fell by more than 80%. Reliability now has targets the business has agreed to, and most alerts come with a runbook. We also measure time to resolve for the first time. It hasn’t improved yet, and I’d rather say that than dress it up. Now that it’s measured, it can be managed.

The hard part

The hardest part wasn’t technical. It was drawing the line between SRE and the infrastructure team: who owns which dashboards, and who owns what when something fails. Some of that is still being settled. An ownership question left open today becomes an incident question later.

What I’d do differently

Capture a baseline on day one, even an ugly one. Nobody measured recovery time before I arrived, and I didn’t in my first weeks either, so I can’t show a before and after for it. And put every piece of work in the tracker from the start: a few months in, I found most of our open roadmap items had no ticket behind them.

Reliability isn’t a tool you install. It’s a set of agreements about what matters, and the habits that keep them.

Want to talk about building SRE at your company? Get in touch.