Show HN: Trifle – Open-source analytics that stores answers, not events
trifle.ioTrifle is an open-source time-series analytics library that aggregates nested counters instead of storing raw events. All in the database you already have. After rebuilding it twice over 10 years, it now tracks ~1B events a day at my day job.
It started in 2015 as my own Rails APM. I plugged into ActiveSupport::Notifications, got a few small users, and one bigger one whose scraping app broke everything. That sparked the core idea: aggregate counters into pre-defined time buckets, so a single write increments multiple buckets at once. The APM eventually faded away without much traction.
Later in 2021 I needed analytics at my day job. Instead of going for something out there I revised the idea of Trifle as a more generic analytics library, borrowing some data warehouse ideas. First used Redis, then Postgres, eventually MongoDB. Hence why Trifle::Stats comes with multiple drivers that keep the DSL unified while storage layer changes with your needs. In our case (huge write volume, some reads) PG read faster but slowed on large writes.
The nested values are the whole trick here. Single:
Trifle::Stats.track(
key: 'requests::aws::s3_uploads',
values: {
count: 1,
status: { request.response_code => 1 },
size: payload.bytes,
duration: { sum: request.duration, count: 1 }
}
)
{ count: 14, status: { 200: 12, 500: 2 }, size: 5628341, duration: { sum: 43, count: 14 } }
Success rate is never stored, but it is calculated by dividing 200s over total number of requests. Same with average duration: sum over count. You ask for a metrics key, granularity and timeframe and you get back aggregated values at each point. Ready for charts or to answer "Average response time over last 30 days".
There's a Series wrapper for aggregating and formatting values for charts in a simple call. And as building dashboards is not as much fun for other devs as I thought, I built Trifle App - a visual layer with dashboards, scheduled digests and alerts. It's written in Elixir, so I ported the library to Elixir too. And later to Go for a CLI. All three are compatible, write in one and read in another.
Today we track activity from over 100M background jobs a day which turns into about 1B events. It runs surprisingly cheap when you're willing to trade some safety away (turn off journaling and write concerns in Mongo). 3-node Hetzner MongoDB cluster where the primary does 20% utilization costs us around $1k/month.
It has its limitations. Payloads can't hold tens of thousands of keys. Documents becomes too large to update efficiently. Some planning ahead is needed. And then there are no dimensions. Sometimes you can nest them (country - there are only so many countries), sometimes it's better to have dedicated metrics key per dimension (customer - growing forever). That multiplies tracked events, hence 1B events from 100M jobs.
The libraries are MIT. The App is source-available under ELv2 - free to self-host and paid cloud if you want it managed. I build this on the side with no investor money to burn on a free service.
Happy to answer anything about architecture, storage models, my failures or why I didn't give up on this yet. What is the stack that is required to setup your opensource repo? and what's the machine config you recommend. btw, I liked the pricing page. I am planning to setup a similar pricing page my project deepsql.ai (dba agent for postgres and mysql). The library has no stack, it's just a set of plugins you can either use directly (Ruby, Elixir, Go) or with a framework (Rails, Sinatra, Phoenix, Ash, etc). They write into a database you already run. Setup is adding the plugin and pointing it at your DB. Redis, Postgres, MongoDB, MySQL or SQLite. As for when you would feel some load from it, it really depends on your volume. I would say somewhere around 100k+ events a day is where the load starts being noticeable. Until then I wouldn't worry much about extra load. If you want to self-host the App, it's pretty lightweight. While it's an Elixir app and you could compile and run it yourself, it comes packaged as a Docker image and it has a Kubernetes/Helm deployment documented. You can either point it to your own Postgres or it will launch its own so it can hold some of its own data like users, dashboards, monitors, etc. For 2-3 users one instance is plenty and you can easily run that on 1 CPU and 2GB RAM. Double that if you want redundancy. Good luck with deepsql! A billion events is only ~4GB on disk. With disk so cheap (even now), why throw that away immediately instead of at least waiting 30 days, 1 year, etc. until you know you have your analysis right? That's a fair point, and honestly disk cost rarely is the blocker these days. ~4GB assumes a few bytes per event, in our case it is closer to 100GB a day. That would be stored as raw text. Being able to rebuild that would require us to have some kind of pipeline that can do that effectively and that kind of defeats the point of Trifle. That said, it really depends on the use case. For us the last 24h-48h holds the most value and anything older than that is just a reference where margin of error is acceptable. Even if we correct the historical data from 2 weeks ago, it wouldn't change the decisions we make today. For others it may be the opposite. Tbh nothing stops you from doing both. Trifle is just a library that writes into your own database. If your data requires occasional re-analysis, write them into their own events table. Then you have the ability to rebuild the stats as you need to. I've seen both sides of the table, and as Trifle code lives with the rest of the code, it is simpler to handle this yourself than try to do this somehow via the App (which is optional in the first place) and UI. Mentioning below in case it helps someone looking into such estimates: For a website analytics product that I was building, where every page-view/event stores many values such as browser, device, os, page-params, referrer.. and other typical stuff, it was coming at about 200GB/Billion events, after DB-level compression... the database obviously was columnar. At these levels of storage needs, things start to cost real deal. With expected monthly intake of just around 10 Billion events, that ramps up to 2TB added every month. Because of this, I had to move to S3-scaled/backed, nvme-cached architecture, but, that made query times fluctuate to some noticeable extent. So, that ~4GB-only is for teeny-tiny data per metric sample.. probably 3-4 numeric values at max. How is this different fro Prometheus and its associated time series db? that looks like exactly how prom works, give or take. (and some of the negatives around stats that come with it, e.g. p95 calculation) you’re right that there are some similarities here. you can do histogram same way in Trifle just as in Prometheus. and the p95 issue is in both. sum and counts can only get you that far. you can get normal approximation of percentiles, but its a compromise you make. where the differences are clearer is who is it aimed for. Prometheus is a server you need to operate while Trifle pushes data into a database you already own. and then there’s a part that Prometheus scrapes for your data and you need a pushgateway to be able to push to it. one is build for infrastructure monitoring and the other is a library you use to push your increments. Exactly. That is the first question the web page should answer. That's a good callout. I had a basic comparison stuffed away on a different page, but you're right that it belongs on the landing page. so I've moved it there as well. I'll work on more in-depth comparisons against specific tools. Thanks! Beautiful website, it's clear you care. thanks! it went through couple redesigns. I'm still not sure if it communicates the message clearly. I'll keep iterating! what happens to your database with all of these metrics coming in? edit: could loading a performance dashboard affect peoples ability to buy their product? it will get busier. the goal is for write to cause the small load every time action happens. this way there is never single heavy query that would cause big load at read time. the main issue with MongoDB I've seen is that writes tend to slow down once you go over 1000+ keys in the aggregated payload. this can easily happen if you have something like yearly bucket. on the other side Redis is not affected by it as increments are done individually, so writes stay flat. about your update: dashboards are the easy side here. rendering one is a lookup for few points from pre-aggregated buckets. all you need is a key and a timeframe with granularity. unless you are trying to get something like 1 month worth of per-minute data, it will be quick. if you want real isolation, you can give driver dedicated user with its own limits or point it to dedicated database. we started with Redis to keep it easy, then Postgres as I wanted to ensure persistance and ended up with MongoDB which handled the writes with ease.