Show HN: Dolce – Get notified if something (bad) happens to your containers
github.comHi HN,
while I was on vacation this year the shop of my wife was suddenly down and we only noticed once the emails of customers rolled in. After fixing it on my phone I thought there has to be a better way to notice when your docker container die or are in a restart loop for whatever reason.
Now I know there are lots of tools like healthcheck.io and monocker, however none seemed to match my simple use case of getting notified _via email_ when the running– or health-status of any docker container changes. So I did what every SE would do in that case and hacked another solution but this one is mine :).
I also always wanted to do a project with deno and get up to speed on typescript. This was also a great reason to finally look into using GitHub Actions to automate some of the tedious tasks. Fun fact: After reading on here about Deno Queues, I rewrote the message buffering and backoff completely and could greatly simplify the code.
TL;DR - What is this: Get notification via
- EMail (BYOSMTP) - Discord - Telegram if any (or only selected) containers on your host die or change their health status.
I also tried to handle unexpected restarts of the container the tool is running in gracefully without loosing any events that happened before and even while the service was down. To make sure you are not spammed by notifications if a container goes into a restart loop, there is also a configurable exponential backoff function.
I tried to make using the tool as easy as possible (by getting inspired on how monocker is set up). I would love to get some feedback and hear what you think (also if I just missed the perfect tool for the job: let me know, however the journey was quite fun). Nice tool Would it have been sufficient to just use health checks against api calls on the site? That way you get notified as soon as the site is down. Well, sort of. The original downtime was caused by the reverse proxy dying for no apparent reason. This state could be detected by healthchecks. However the quick solution after the second time the container went away while on holiday was to use ‚restart: unless-stopped‘ and I could see a case where the container was down 90% of the time but just-so-happend to be freshly restarted while the healthchecks were looking. That’s why I decided to use the docker API’s /events endpoint to get notified about all changes. Using dockers healthcheck features you can even monitor the availability of the http server with this solution as well. I did not find a project that went the same route. The downside to this approach of course is, that if docker goes down you won’t get notified, also if you just use curl inside the container for healthchecks, you won’t notice external network problems. So something that is hosted on the outside like healthchecks.io is a great addition. This is really cool - great work! Thanks, appreciate it.
It was a fun exercise and if you end up with a tool that you and hopefully others can use even better.