A simple core.async job system in Clojure
blog.janetacarr.comI appreciate this emphasis on simplicity and I recommend it to others. I've seen trends in the other direction which I think are dangerous. I notice that as some companies adopted a microservices approach there was a tendency to allow in more technologies than necessary.
"Premature polyglot programming" is a disease that afflicts certain startups. While any sufficiently big company will be polyglot, a small startup needs to stay focused on a limited tech stack, for as long as possible. After all, each new technology requires someone on staff who has the skill for that technology, and when your startup is small, your talent pool will also be small, and so it becomes common that you only have 1 person on the team who knows how to run some particular technology (Kafka, RabbitMQ, Redshift, DynamoDB, MongoDB, Tornado, etc).
So you end up with a lot of single point of failures, where the "single point of failure" is the single engineer who knows how to run the technology that you've made critical to the company. Be wary. Avoid this if possible.
I notice, especially, as the tech industry developed better tools for managing complex devops situations (Docker, Kubernetes, Terraform) there was a tendency from some engineers to think "Nowadays it is easy to run 10 different technologies, therefore we should run 10 different technologies." Be wary of this.
Janet Carr's emphasis on simplicity is something we should all imitate.
I agree, and a lot of this applies to large companies as well.
Of course larger orgs will have higher tolerance for a wider set of technologies, but this tolerance is not infinite, and choosing tech because that one team was a fan creates staffing problems over time, especially as the original devs move on. Eventually you get “estate sprawl” that’s difficult to manage.
Choosing to minimise dependencies is definitely a (good) choice. The key is the right tech for the task at hand. Equally org-wide “use Oracle for everything” mandates aren’t helpful.
Balancing these at a large org definitely needs good engineering leadership!
Wow, thanks for the kind words.
I've been bitten by the ops hellscape of microservices and various tech many times throughout my career, and it's definitely shaped how I think about designing and building software now.
This is what I think about most when looking at languages. “How painful would it be if I implemented the minimum now and wanted to add more capability later?”
Some languages make it very painful to not have a mostly complete design upfront and then you’re trapped adding complexity just in case.
You can use this same line of reasoning to avoid using Clojure entirely.
Yeah but that's not nearly as fun!
Very true.
100% agreed. Unfortunately overdoing it on complexity early on seems to lead to a loss of the faith in engineering decision making even after overengineering leadership is removed.
core.async was actually the "killer app" that sold Clojure to me. I've always liked Go's CSP-style concurrency, but I wanted a proper functional language to go with it. It turns out that Channels/Queues are just an insanely good abstraction for getting threads to talk to each other. The closest general-purpose library I've found that helps bring that to the rest of the world has been ZeroMQ, which is great but not as easy and nice to use as core.async.
That said, I really don't think RabbitMQ is so bad. The default docker configuration for it is fine for most cases you're likely to come across for all but the biggest applications, and I do think that having the ability to have jobs restart/requeued when a worker crashes out of the box is worth the tradeoffs for slightly increased complexity. I usually just use a single docker compose and glue everything together in one big ol' YAML.
Still, there's obvious value in avoiding dependencies, so it of course depends on the size and scope of your project. If you think you're going to end up distributing this over 100 nodes, something like RabbitMQ or ActiveMQ might be worth playing with, but if you're just doing relatively small (at you appear to be in this project), it's probably the correct choice to mimic whatever behavior you need with core.async.
> core.async was actually the "killer app"
For ClojureScript, it was React. Immutable state / functional programming fits so nicely, and hiccup syntax (JSX as simple vectors) is the perfect declarative (yet transformable) DOM structure.
Want to put in a plug for Electric Clojure. More composable than React (mix server and client code in the same function!) and more performant. I've had a lot of fun using it over the past year, and it's IMHO the most transformative tech I've seen in web dev.
Wrt the original article, effect systems like missionary (which electric is based on) can be used to complement or replace queues, often resulting in simpler code.
Oh absolutely, no argument here at all.
I don't do a lot of frontend, but for anything that does involve frontend I kind of refuse to use anything but re-frame. It took a bit of dogma-acceptance, but once I did I really liked it for all the reasons you listed, and in particular that there's no special JSX crap, and instead just using vanilla vectors and symbols. This has the interesting advantage where you can generate your HTML logic without actually having to import the re-frame/re-agent logic, which can be useful if you want to decouple logic from rendering.
I haven't done any benchmarks, but I haven't noticed re-frame being slower than vanilla React, it seems to be a comparable speed, but as stated I don't do much frontend.
Yes! core.async is a great tool and I've used it to solve a number of problems effectively. I'm very happy with what I get: reliable, predictable systems.
A real "whoa" moment comes when you realize you can also use core.async in ClojureScript :-)
Thank you for this post Janet.
I think my technical perspectives have moved in a similar direction: keep things extremely simple with minimum moving parts.
I maintained (automated upgrades) a RabbitMQ cluster and while it is powerful software it is operationally expensive. For a side project you probably just batch process in a cron.
If I were to take the approach in this blog post I would want everyone on the team to be extremely familiar with the model of task running: stuck jobs, timeouts, duplicate jobs, client disconnects and retries, stuck "poison" jobs seem like issues you might face.
I'm glad you liked it!
I'm using a PaaS, so I didn't want to pay the extra money for a cron job. Maybe not a wise a choice, but here we are.
Totally agree about the issues I might face. I'm glad that the Clojure REPL is a thing, so I can test out all of a job's functionality before sending it off to async land.
I appreciate Janet’s aversion to adding RabbitMQ to the system. I had to use RabbitMQ with Common Lisp a year ago and in was a pain.
It looks like Clojure has better RabbitMQ client options than Common Lisp, but still, very cool to build something on core.async and keep things cleaner and simpler.