About a year ago Prolucid adopted Apache Storm as our platform of choice for event stream processing and F# as our language of choice for all of our “cloud” development.
FsStorm was an essential part that let us iterate, scale and deliver quickly, but even from the earliest days it was obvious that the developer experience could be improved. Unfortunately, it meant a complete rewrite of FsStorm:
- FsStorm DSL is a really thin layer on top of Nimbus API model:
- has explicit IDs when describing components in a topology
- uses strings in all the names
- matching of inputs/outputs is not guaranteed
- FsStorm uses Json AST as it’s public API:
- messages, tuples, configuration
- serialization mechanism is hard-baked into the API
We’ve worked around some of the problems, usually by writing more code.
It actually makes sense that Storm itself doesn’t care about the type of the tuples/fields. It runs on JVM, which is very much typed, and it relies on sub-class polymorphism to make things tick. However the public API for the tuples looks like an afterthought in every language. But we figured, there is this “compiler” that can do “type checking” for us, let’s make it work! Maybe we can even make it faster if we replace Json with Protobuf?
Coming up with the new DSL that would allow the components to consume and emit tuples of various (static) types on multiple streams was an interesting experience and led to some strange places. A lot has been written on F# DSLs, but none of that applied directly. Can I use “just functions”? Do I need a type provider? A computation expression? A compiler as a service?
After a few false starts I found the desired paradigm that could be expressed in F# succinctly. As it usually happens, once I gave up on certain notions (building “any purpose” graph from a single source in this case), the result was pretty simple. And so, after a few weeks of journey and discovery, we are releasing FsShelter: a way to program Storm with F# in a statically typed fashion.
Many thanks to Tomas Petricek, Scott Wlaschin, Andrew Cherry and Erik Tsarpalis, without them FsShelter wouldn’t have been possible.
FsShelter is currently in beta and any feedback is welcome and appreciated.