Q: What's the difference between monitoring and observability? "Monitoring tells you whether something is wrong using pre-defined metrics and dashboards. Observability gives you the ability to ask new questions about your system after the fact, by collecting metrics, logs, and traces with consistent context. Observability is necessary because cloud-native systems fail in ways you can't predict, so you can't pre-build a dashboard for every possible failure."
Q: How would you instrument a new service? "I'd start with the four golden signals: latency, traffic, errors, and saturation. For latency I'd capture distribution percentiles, not just averages — p50, p95, p99 — because averages hide the long tail that affects users. Then I'd add business-level metrics specific to the service, like checkout completion rate for an e-commerce service. Logging would be structured JSON with consistent fields like trace ID. Tracing via OpenTelemetry so we can follow requests across services."
Q: How do you reduce alert noise? "Three things. First, alert on symptoms not causes — alert when users see errors, not when CPU is high. Second, tie alerts to SLOs and error budget burn rate so we're alerting on user impact, not noise. Third, every alert needs a runbook entry. If we can't write a runbook, we can't justify the alert."