I was sizing a storage pipeline this week. A Batch job downloads a daily snapshot, about 10 GB, and lands it in S3. Downstream services read recent files; older ones rotate to cold storage automatically. Three components: a schedule rule, a job, and a lifecycle policy.
I ran the numbers. 30 days of snapshots is roughly 300 GB, about $7 a month in S3 Standard. After rotation, Glacier brings that under $1. Requests, uploads, transitions are all negligible. The whole thing costs less per year than a meal.
My first impulse was still to write an aggregation step. The snapshots are full-state dumps. I could diff consecutive days, keep only the deltas, drop the unchanged portions. The engineering seemed obvious, almost obligatory.
5:1 compression would save about $5 a month.
| Monthly | Annual | |
|---|---|---|
| Raw snapshots, 30 days in Standard, then Glacier | $7 | $84 |
| With 5:1 aggregation | $2 | $24 |
| Savings | $5 | $60 |
For $60 a year I would need to maintain a transformation job, handle schema changes, and give up the ability to reprocess raw data when the aggregation turns out wrong. I closed the notebook and left the snapshots as they were.
The decision took minutes. The discomfort lasted longer.
I have felt this before. A colleague allocates generously and I notice myself wanting to tighten it, not because of a constraint anyone has stated, but because something in me insists that uncompressed data sitting on a disk is waste. The arithmetic says otherwise. The arithmetic is right. The feeling does not update.
I think I know where this comes from. For my cohort, disk was the resource you learned to respect early. 10 GB was a serious number. You compressed, pruned, garbage-collected, not for elegance, but because the drive would fill. Cloud storage removed the constraint years ago. The reflex stayed.
Engineers a generation younger seem to carry the same pattern, aimed at a different target. RAM is the resource they feel in their hands. They will optimize allocations that do not need optimizing, avoid caching strategies that the numbers plainly support, long after memory becomes cheap and wide. They will know it is cheap. Their hands will not agree.
My grandparents’ generation had a version of this with food. People who lived through real shortage, not inconvenience but the kind where villages went quiet, do not throw away rice. They finish what is on the plate, store what cannot be finished, save containers and bags against a future they already know is unlikely. The body learned “not enough” and did not accept the correction.
The scarcity was not natural. Forced collectivization, fabricated production reports, grain exports continued while people starved. Tens of millions died. The policy that caused it is still described in official language as exploration, as tuition, as a natural disaster with an unfortunate duration.
Taiwan had its own terror in the same decades. The state killed and imprisoned dissidents, made silence a survival skill for a generation of intellectuals. It did not starve people. And it was eventually named: archives opened, victims listed, a public record assembled, however incomplete. The wound got a word for itself. That does not heal it, but it lets someone trace a reflex back to its origin.
When the record denies the wound, the reflex still runs. You finish every grain. You cannot explain why, exactly, because the explanation was removed from the language you were given.
I left my snapshots uncompressed. The lifecycle policy handles the rest. There is no aggregation step to maintain, no schema-change bugs to chase, no loss of raw data.
The discomfort faded once I recognized what it was. Not a signal about the pipeline. An echo from a time when the constraint was real.