Reminded of that time when the director of Engineering at a company I worked for had to tell all of Engineering that they were required to check their e-mails twice a day, no exceptions. #SlackLife
@Hypolite Petovan Alerts are things that cause alarm. Email and slack both can only contain alerts if they also are alerting! They need to create attention, not fall into background noise.
But yeah, we do both, and I'm always weirded out by the manual slack.
@silverwizard The alerts we issue should always be followed by actions, these aren't automated alerts based on arbitrary metrics level. We check 10 minutes before the top of the hour that the data needed for the next hour is available, and if it's not, someone needs to press a couple buttons possibly hundreds of miles away from me. We definitely factored in alert fatigue but these are business critical.
@Hypolite Petovan that feels like something where if it's not available it should be kicking off a system! Feels like computer work - and I'd be fighting to make it that
silverwizard
in reply to silverwizard • •Fear and Tooting in Las Vegas
in reply to silverwizard • • •#SlackLife
silverwizard likes this.
silverwizard
in reply to Fear and Tooting in Las Vegas • •Hypolite Petovan
in reply to silverwizard • • •@silverwizard That’s why we do both at my job! 🙌
It still didn’t prevent an SRE from claiming no alert was raised. 🤔
silverwizard
in reply to Hypolite Petovan • •@Hypolite Petovan Alerts are things that cause alarm. Email and slack both can only contain alerts if they also are alerting! They need to create attention, not fall into background noise.
But yeah, we do both, and I'm always weirded out by the manual slack.
Hypolite Petovan
in reply to silverwizard • • •silverwizard
in reply to Hypolite Petovan • •Hypolite Petovan
in reply to silverwizard • • •silverwizard likes this.
silverwizard
in reply to Hypolite Petovan • •Hypolite Petovan likes this.
Hypolite Petovan
in reply to silverwizard • • •silverwizard likes this.
silverwizard
in reply to Hypolite Petovan • •@Hypolite Petovan Just shark your wires! It'll be fine!
Or use some screen scraping! We're too afraid of writing software that lies to other software! Mouse clicks are just programs!
Hypolite Petovan likes this.