Paige Cruz

Cognitive apprenticeship is the philosophy that it is more effective to learn in context and real-world situations compared to following a tutorial in a sandboxed environment. In a nutshell it is “learning-through-guided-experience” and shines when teaching problem-solving processes experts use to handle complex tasks like say…investigating an alert by spelunking in production observability and monitoring data. Learn how Alert Triage Hour of Power became a can’t miss meeting of camaraderie and system surprises!

There's a reason "i test in prod" isn't a cheeky take but a lived reality. And that reason is there is no place like production. Not local dev or staging or other environment. This became clear when I deployed a tiny config change that passed all checks, reviews and pre-production environments that triggered a SEV-1. Examining each step of the journey from PR to production I uncovered the snafu that had occurred (unsurprisingly it relates to overwriting key blocks on nested YAML files). I’ll share how difficult it was to reconstruct the chain of events in the system compared to the ideal case of a highly observable system and how to share your own incident learnings since we all test in prod!

The signal to noise ratio can be frustratingly high for on-call alerts and downright mystifying when joining a new company or rotation. If you treat Pagerduty like the boy who cried wolf you’ve certainly mired in monitoring debt. Monitoring debit is tragically common and with investment mitigatable! Just how much of a risk are false positives? The data shows that a high number of false alerts trains workers to assume most alerts will be false. Repetition of the same alerts causes even greater alert fatigue. For clinicians the likelihood of acknowledging an alert dropped 30% for each reminder. Paige will share the nuts and bolts of how to start auditing your own alerts to begin your journey towards an on-call oasis.