Ask HN: How to merge time series data from devices that can have offline time?
Let's say I have devices like Raspberry Pis that collect data, e.g. a Prometheus instance on the device scraping local metrics.
I find the concept of federation in Prometheus very interesting as it offers a way to provide the data of many Prometheus instances to a point where I can query all at once.
Now I've put quite some time into research about the following problem, but did not find very much so far:
- These devices (more or less "IoT"-ish), are expected to have outage times due to network issues
- Since they have a local Prometheus server, there is a local time series db that stores the metrics of this device
- How would I provide this data to a central Prometheus (or similar) instance that is able to "merge" all devices, also taking data with past timestamps into account (where the timestamps can have different ranges in the past, depending on the outage duration etc.)
I can't believe there are only few others with problems like this. Since I am lacking knowledge in this topic, maybe someone can give me some help or hints?
My goal is to provide a time series db that always offers the most recent data available for each device.
"store-and-forward" should be the right search term, but I didn't find a lot besides "proprietary" solutions for time series... I'd like to stay close to a "standard way" (TM).
I would really appreciate any hints or input!
No comments yet.