Reminded of that time when the director of Engineering at a company I worked for had to tell all of Engineering that they were required to check their e-mails twice a day, no exceptions. #SlackLife
@Hypolite Petovan Alerts are things that cause alarm. Email and slack both can only contain alerts if they also are alerting! They need to create attention, not fall into background noise.
But yeah, we do both, and I'm always weirded out by the manual slack.
@silverwizard The alerts we issue should always be followed by actions, these aren't automated alerts based on arbitrary metrics level. We check 10 minutes before the top of the hour that the data needed for the next hour is available, and if it's not, someone needs to press a couple buttons possibly hundreds of miles away from me. We definitely factored in alert fatigue but these are business critical.
@Hypolite Petovan that feels like something where if it's not available it should be kicking off a system! Feels like computer work - and I'd be fighting to make it that
silverwizard
Fear and Tooting in Las Vegas
•#SlackLife
silverwizard likes this.
silverwizard
Hypolite Petovan
•@silverwizard That’s why we do both at my job! 🙌
It still didn’t prevent an SRE from claiming no alert was raised. 🤔
silverwizard
@Hypolite Petovan Alerts are things that cause alarm. Email and slack both can only contain alerts if they also are alerting! They need to create attention, not fall into background noise.
But yeah, we do both, and I'm always weirded out by the manual slack.
Hypolite Petovan
•silverwizard
Hypolite Petovan
•silverwizard likes this.
silverwizard
Hypolite Petovan likes this.
Hypolite Petovan
•silverwizard likes this.
silverwizard
@Hypolite Petovan Just shark your wires! It'll be fine!
Or use some screen scraping! We're too afraid of writing software that lies to other software! Mouse clicks are just programs!
Hypolite Petovan likes this.