Tracking Service Level Objectives (SLOs) for Reliable Systems

"How reliable is this service?" seems like a simple question, but most teams can't answer it objectively. Service Level Objectives (SLOs) provide a framework for measuring reliability and making data-driven decisions about risk.

What Are SLOs?

An SLO is a target reliability level for a service. It answers: "What level of reliability does this service need to meet?"

Key Concepts

SLI (Service Level Indicator) — the metric you measure

• Availability: successful requests / total requests
• Latency: requests under threshold / total requests
• Error rate: errors / total requests

SLO (Service Level Objective) — the target you set

• 99.9% availability
• 99.5% of requests under 200ms
• Less than 0.1% error rate

Error Budget — the allowed unreliability

• 99.9% SLO = 0.1% error budget = 43 minutes/month

Why SLOs Matter

Objective Decision Making

SLOs replace subjective judgments with data:

Without SLOs: "We think the service is pretty reliable"

With SLOs: "The service has consumed 80% of its error budget this month"

Risk Management

SLOs help balance reliability with velocity:

• High SLO (99.99%) — slow down, minimize risk
• Low SLO (99.9%) — move faster, accept more risk
• Error budget remaining — indicates how much risk you can take

Team Alignment

SLOs create shared understanding:

• Everyone knows what "reliable" means
• Decisions are based on data, not opinions
• Priorities are clear based on error budget status

Defining SLOs

Step 1: Identify Critical Services

Not everything needs a 99.99% SLO:

• User-facing API — high SLO (99.9%+)
• Internal tools — moderate SLO (99.5%)
• Batch processing — lower SLO (99%)
• Development environments — no SLO

Step 2: Choose the Right SLI

Pick metrics that reflect user experience:

Availability SLI

successful_requests / total_requests

Exclude health checks and synthetic monitoring from the calculation.

Latency SLI

requests_under_threshold / total_requests

Choose a threshold based on user expectations, not technical capabilities.

Error Rate SLI

error_requests / total_requests

Define what counts as an error (5xx responses, timeouts, etc.)

Step 3: Set Realistic Targets

Base targets on:

• User expectations — what do customers need?
• Current performance — where are you today?
• Business requirements — what's the cost of downtime?
• Technical capabilities — what can you achieve?

Common targets:

• 99.9% — 8.76 hours downtime/year
• 99.95% — 4.38 hours downtime/year
• 99.99% — 52.6 minutes downtime/year
• 99.999% — 5.26 minutes downtime/year

Step 4: Define Time Windows

Choose appropriate measurement windows:

• Rolling window — last 30 days (recommended)
• Calendar month — aligned with business cycles
• Quarterly — for long-term trends

Implementing SLOs in Datadog

Creating SLOs

1. Go to Service Management > SLOs
2. Click New SLO
3. Choose your SLI type:
   • Metric-based
   • Query-based
   • Browser-based
4. Configure your query
5. Set your target (e.g., 99.9%)
6. Choose time window
7. Save

Metric-Based SLO Example

# Availability SLO
Query: sum:requests{service:api,status:2xx}.as_count() /
       sum:requests{service:api}.as_count()
Target: 99.9%
Window: 30 days rolling

Latency SLO Example

# Latency SLO
Query: sum:requests{service:api,duration:<200}.as_count() /
       sum:requests{service:api}.as_count()
Target: 99.5%
Window: 30 days rolling

Error Budget Alerting

Alert when error budget is being consumed:

# Alert when 50% of error budget consumed
Alert condition: error_budget_remaining < 50%
Alert message: "API service has consumed 50% of its monthly error budget"

SLO Best Practices

Start Simple

Don't try to implement SLOs for everything at once:

1. Week 1: Define SLOs for top 3 critical services
2. Week 2: Set up monitoring and alerts
3. Week 3: Create dashboards for visibility
4. Week 4: Review and adjust based on findings

Use Error Budgets Wisely

The error budget is a tool, not a goal:

• Spend budget on risky deployments and improvements
• Conserve budget when approaching limits
• Protect budget during critical business periods

Review Regularly

SLOs should evolve:

• Monthly — review error budget consumption
• Quarterly — adjust targets based on learnings
• Annually — re-evaluate SLO strategy

Document Everything

Keep SLO documentation updated:

• Which services have SLOs
• How SLIs are calculated
• Who owns each SLO
• How to respond to SLO violations

Common SLO Mistakes

Setting targets too high. 99.99% is aspirational. Start with achievable targets and improve.

Measuring the wrong thing. SLOs should reflect user experience, not internal metrics.

Ignoring error budgets. If you never spend your error budget, your SLO is too conservative.

No action on violations. SLO violations should trigger real responses, not just alerts.

Too many SLOs. Focus on what matters. 5-10 SLOs per team is plenty.

SLO-Based Alerting

Burn Rate Alerts

Alert on how fast you're consuming error budget:

# High burn rate alert
Condition: burn_rate > 2x for 1 hour
Meaning: You'll exhaust error budget in 15 days at this rate

# Critical burn rate
Condition: burn_rate > 10x for 5 minutes
Meaning: Emergency - error budget being consumed rapidly

Multi-Window Alerts

Combine fast and slow burn rates:

• Fast burn — detects immediate issues
• Slow burn — detects gradual degradation
• Both together — reduces false positives

SLO Dashboard

Create a dashboard showing:

Current Status

• SLO achievement percentage
• Error budget remaining
• Burn rate trend

Historical View

• SLO trend over time
• Error budget consumption pattern
• Incident correlation

Team View

• SLOs by team/service
• Ownership and accountability
• Performance comparison

Getting Started

1. Identify your 3 most critical services
2. Define availability SLOs for each
3. Set realistic targets (start with 99.9%)
4. Monitor SLO achievement
5. Alert on error budget consumption
6. Review monthly and adjust

SLOs transform reliability from a vague concept to a measurable, manageable metric. Start simple, measure consistently, and improve based on data.