Double Buffering with Cloudflare Queues
bellone.comI think this is a clever happy path solution, but it isn't very defensive against problems. If the cloud cron doesn't fire on time or the consumer fails then the queue won't be drained in time. That will cause the events to be batched in the wrong hour.
I don't see how you are building the batches, but there's a lot of complexity around batch sizes, acknowledging that the messages were consumed, etc. Having a single dataset.json.gz per hour feels like a hint this isn't done safely. I'd double check that carefully to avoid losing data.
One tip about compression, I've been really pleased with zstd compression of json/jsonl data.
- your blog page has errored out
A stateless pattern I use to batch telemetry at the edge while driving Cloudflare write costs to zero.
Cool pattern. Critics will say “falsehoods programmers believe about time” but it’s a cloud service, any time oddity that would break this would also break a lot more important stuff, so it’s fine.
Ugly code though.
// env QUEUES = [EVEN_QUEUE, ODD_QUEUE] // producers env.QUEUES[hour % 2].send … // consumer env.QUEUES[1 + hour % 2]> env.QUEUES[1 + hour % 2]
// results in index 1 or 2 as js has precedence for */% over +-
So something like the following, for your desired visual succinctness? ...
env.QUEUES[hour % 2 ^ 1]
// results in desired js index 0 or 1
I wonder if any of you using AI code assistance regularly might tell an old hack like me if your code review tools would have caught such a bug?
Can you explain the pricing here?
Cloudflare R2 bills on writes ($4.50/M), reads ($0.36/M), and storage ($0.015/GB/mo). The free tier gives you 1M writes, 10M reads, and 10GB per month. If you batch uploads hourly, you only make ~720 writes a month. That completely eliminates write costs, leaving storage as the only expense.