Run, Build, and Grow Small Systems Without Leaving Your Text Editor

5 min read Original article ↗
Albert ZakAlbert Zak

Run, Build, and Grow
Small Systems
Without Leaving Your Text Editor

Presenting at LIVE @ SPLASH 2024

Watch on YouTube (15 min) >

Screenshot of an interactive ClojureScript programming environment with self-rewriting expressions, live text, lightweight distributed programming etc

Clojure, but more alive

See the value of any expression on any node – in your code.

A REPL, but it's connected to all your nodes at the same time (1:n)

  • Query macro expression
    • rewrites itself to show the data inline
  • Declarative deployment
    • Say "this function should run on that node"
  • Supervision & reconciliation
    • Stop a process? Comment it out.
  • Effects via object capabilities
    • All host platform caps passed in as argument
  • The editor is also a node
    • You can run code in it
  • Define and use custom inline editor visualizations
  • Values have history

Taglines#

Status#

Hacked together with SCI, rewrite-clj, zprint, Lezer and CodeMirror. Everything barely holds together.

I am ashamed of the code. But I might clean it up and release it if enough people are interested.

There's a babashka script ./new-commit-every-minute.bb that does what you'd think.

I'm building this in public, scrappily.

Follow along, or get notified when this is ready:

Related work#

More related work that's missing from the video#

Future work (todo)#

  • error handling - at least like Sentry, maybe even a poor Smalltalk debugger or Common Lisp condition system
  • persistence story - accretive bitemporal facts like XTDB or something local-first?
  • communication - infer netcode like Electric
  • codebase management - hashed like Unison or Scrapscript
  • collaboration / ocap code sharing / delegate authority to change part of the system
  • control over when and where changes deploy: staged rollout, feature flags, "under construction" development preview areas
  • actually secure security
  • release as extensions for common editors (maybe as language server?)

FAQ#

  • What is this? A Language? An IDE? A new REPL?
    • It's just ClojureScript, with a handful of additions:
      • ! to redeploy the next top level form on every change
      • ? the query macro, rewrites itself to show values inline
      • node to assign functions to physical places
      • context scopes queried values to only show those matching some node context
      • with-context attaches custom metadata to node context
    • You can call it a different kind of REPL
      • or a programming environment with a runtime
      • for building distributed information processing systems
  • Is there any chance something like this can become an Emacs mode, VS Code extension, or NeoVim plugin?
    • Yes. Editor plugins are necessary. It should be fairly straightforward to port, apart from the inline html views.
      • The editor API surface for plaintext interactions is tiny right now: callback on every change, get buffer contents as string, (async) replace position span with string.
      • Inlining graphics in text always breaks the editing flow anyways. I don't like the heavyweight feel of blocks, so I never use this feature.
      • Just checking the rough shape of data in plain text is good enough usually.
      • It's also easy to hook the query macro into Clojure's tap system, so non-text output renders in a sidecar viewer like Portal (or Clerk, but haven't tried that)
  • It's always connected to all the running processes?
    • Yes, conceptually
      • nodes can be anywhere, behind firewalls, in production, etc.
        • communication goes through a central broker
    • It feels like a REPL that's connected to all nodes at the same time.
  • Can I just eval forms like normal?
    • Not trivially. I'm still thinking about the UX of targeting specific nodes.
    • Evaluation in this system should feel more like nodes pulling your specification (code).
      • Instead of you imperatively pushing code to nodes that they should execute,
      • they all pull the latest code from you
        • (and decide themselves how to best fulfill it)
  • Can it do transient sessions, like browser clients?
    • Not yet. The set of all nodes has to be hardcoded.
      • But the solution is simple, and I'll implement it soon
        • Introduce random sessions IDs
        • and keep the (then non-unique) node IDs to tell them to run the same code
  • Does it run on Clojure on the JVM / CLR / babashka?
    • Almost. There's no reason it couldn't run anywhere SCI does.
  • Why Clojure / Lisp?
    • (Watch any of Rich Hickey's Talks)
    • s-expressions
      • code being data
      • trivial for the query macro to rewrite
    • Clojure emphasizes simple plain data throughout your system
      • plain data is trivial to inspect
        • and to hold on to
        • copy/pasteable
        • serializable
        • safely experiment with it
      • functions take plain data and return new plain data
      • see the data in every step in between
    • semi-structural navigation via keyboard shortcuts
      • just works across code and data
  • What does it take to set up and connect a node to this system (eg how are the node's capabilities specified)?
    • it's a library you embed into your application. call it with:
      • a random node ID
      • the central message broker's URL
      • and an optional map (dict/object) of this node's capabilities for effects you want to expose
        • caps are function closures or references to host platform objects
    • or just download ready-to-use runtimes (just give them node IDs)
      • docker containers
      • standalone binary (Windows, Linux, Mac)
      • single board computer SD card image (e.g. Raspberry Pi)
      • babashka library/pod?

Pixel art of a cherry tree. Created by Christina Felicitas Zak after Bret Victor's demo in 'Inventing on Principle'