Ask HN: How to Design an analytical system
Hi,
I'm currently trying to improve a system we use at my company to generate reports based on events data we gather. Our system ingests a lot of events coming from different entities every day and at night it generates reports for each of theses entities. There are several challenges in this generation:
- standard data that can be counted or summed - maps data that group by values with some keys - Aggregation functions such as mean, median... that must be run last after all the processing is done - Cross dependencies on 2 differents kinds of events (that aren't usually processed together and require 2 db requests) - General scale of the task that requires everything to be page/cursor based
I'm wondering if anyone here already designed such a system, and if so how did you approach it?
I think it should be a fairly common task among tech companies but I've yet to find articles stating good practices and things to watch for.
No comments yet.