Clojure REBL [video]
youtube.comI need a Clojure job, their technology seems like it's from the year 3000 and I'm over here in PHP land banging rocks together
Their idiomatic database can travel through time, they focus intently on composition, referential integrity, immutability to the database level
They have specs which can validate things typically better than types and they compose
They focus on shared protocols which gives them amazing leverage, like the internet works because we all cooperate via the http protocol
I've been looking quite closely at datomic recently and that database looks ducking magic
Now datafy and nav will give rise to generalised browsers of anything Clojure can touch, conceptually it's kind of like my file system, my SQL browser, my web browser, my profilers, my debuggers and any number of other things I haven't imagined, all merged into one
My thoughts exactly!
I’ve only been playing with Clojure for a few months, but it feels like a spaceship. This addition just adds to the mystique.
My first thought upon seeing the REBL browser in action was that it looks like those protocols enable something like Smalltalk’s amazing integrated full-system dev environment—the most intriguing aspect of that language to me by far—to the Clojure world, without having to buy into the “system as image” paradigm that Smalltalk seems to require. Can anyone with Smalltalk experience comment on how accurate this hunch is? It just screams “best of both worlds” to this noob.
EDIT: I should have read the other comments first: it looks like this is a common sentiment, which is very encouraging. I’d still like to hear the first impressions of a Smalltalk veteran.
i tried installing clojure using leiningen at work and was greeted with over a hundred lines of incomprehensible java stack traces. that doesn't feel futuristic to me.
If it had todo with clojars’ URL resolving to localhost, I had the same issue trying to install recently. No obviously cause, nor a solution besides using sudo to install AFAICT. But yeah, not a good introduction to the language.
Clojure 1.10 (being released very soon) has a lot of work to make the commonly cited stacktrace problems better FWIW.
Clojure specs take me 10x longer to write than any strongly typed structure I ever made in Haskell. Not a fan, but I use it anyway with great distaste.
The funny thing is most of this has been available to Lisp programmers since the 80s.
Probably with BBN Lisp (later called Interlisp) in the end 60s.
I don't think that's true. Where did the old Lisps implement something like Datomic or Spec? Where did the old Lisps focus on immutability and composition? They didn't, they were generally imperative, even though they had "functions". Even Scheme is much more mutable than Clojure.
I just finished watching, and I'm super excited about this idea. There's been a lot of talk in the Clojure community about how to improve documentation, and then Rich comes along and thinks hard about it for a long time and then drops a general purpose data browser. I'm really excited by the possibilities this brings to the table.
> This is design work. Thanks Rich.
Well said! It feels to me that the power of a whole language with simplicity as a core philosophy is now starting to snowball up some really powerful tooling - since small changes can create big improvements.
We've got some new developers at work who are learning Clojure and so I've been rewatching some old Rich Hickey videos (Hammock Driven Development) and it's cool to see the payoff 7 years later of what he was talking about back then.
I'm curious on just how new this is. Feels like very similar things have been demoed before in environments like Dr Racket.
Still lots of fun, mind you. And anything to bring these environments to more folks. I just worry about things that only demo well if well rehearsed. Which seems to be more technology than makes sense.
This stuff dates back to the Lisp Machine at least. But Clojure is the only relevant Lisp left, so no exciting Lisp technology actually officially exists until Rich Hickey invents it.
Oh come on, there's still the undying Common Lisp, and also Emacs Lisp has a lot of relevance, even if indirectly.
Satire is hard to detect...
Oh. I can hear it now.
While we're talking about this conference, rich hickey just changed how I think about domain modelling forever with this talk, released about an hour ago: https://m.youtube.com/watch?v=YR5WdGrpoug
Strongly recommend a watch particularly if you're modelling something in a typed language
I was thinking of the exact same thing when writing my new project in Python, and implemented a poor-man's version using typing.NamedTuple. But my idea was mainly from the extensive records from Elm. Although Elm also have Maybe, but the idioms of "making impossible things unrepresentative" eliminates a lot of unwanted usage.
Agreed, just finished watching, and that's an amazing talk.
Cool stuff! HATEOAS [1] for REPLs?
"A [REST] (REBL) client enters a [REST application] (REBL browser) through a [simple fixed URL] (initial EDN [2] data). [All] future actions the client may take are discovered within [resource representations] (metadata) returned [from the server]. The media types used for these representations, and the link relations they may contain, are standardized. The client transitions through application states by selecting from [the links within a representation] (the annotated data) or by manipulating the representation in other ways afforded by its media type. In this way, [RESTful] (REBL) interaction is driven by [hypermedia] (metadata), rather than out-of-band information.
1: https://en.wikipedia.org/wiki/HATEOAS , the one thing that makes REST REST but nobody implements in their "RESTful APIs".
Glad to see more work being done on the general purpose data browser concept.
I've been spending occasional free moments working on one since 2014[0], and this is the first time I've come across something with any significant overlap in conception[1].
It still blows my mind that I can describe the concept to programmers (and/or show a video demo) and have the most typical response be along the lines of, "but what would you use it for?" (To be fair though: this is certainly in part—though not totally—due to the particulars of the poor job I've done attempting to communicate the idea, e.g. by demoing visualizations of algorithms on random data.)
It would be easier to answer what you wouldn't use it for. When is it not useful to see the data your code is operating on? When in the video he talked about having one monitor for the data browser and one for emacs, that made complete sense to me. I see writing programs in the future turning out that way: you get a dedicated, mostly passive display of the data your code is operating while you program.
It's almost like we're so trained at having to deal with invisible program data it ceases to be obvious that in building something whose fundamental goal is to transform data (i.e. computer programs), one would benefit enormously through the ability to see the data being transformed.
Prior to this the best way I had of describing the purpose was to point to the Chrome dev tools object explorer and Firefox's console.table(...)[2]. But this is the first thing I've seen also taking on the 'unifying' aspect of the project, based on—as was pointed out in the video—the fact that data tends to come in a limited number of shapes.
[0] http://symbolflux.com/projects/avd
[1] There are still significant differences though—for one thing mine is focused much more on time-evolution of data. Where he showed the quick display of a graph of a set of numeric values for instance, I'm building a corresponding feature but it would instead work with a single numeric variable and display a graph of its values over time.
[2] https://developer.mozilla.org/en-US/docs/Web/API/Console/tab...
> It still blows my mind [...] the typical response is along the lines of, "but what would you use it for?" [...] When is it not useful to see the data your code is operating on?
Status Quo? (the status quo being... print statements?)
Another easy example of a long available better model that is still niche/ignored: pipelines in PowerShell [1] are clearly superior in many fronts, but people prefer keep parsing plain text with regular expressions rather than learn a new shell language.
1: https://docs.microsoft.com/en-us/powershell/scripting/gettin...
It shouldn't be such a shock that people prefer text. Text is the universal format. The vast majority of programming tools operate on it. When someone comes up with a new tool, if they support text and STDIN/STDOUT/STDERR then that's it, they're done.
PowerShell sounds great and I'm sure there are some people who are very productive with it but it's ultimately a silo. It's never going to be as low-friction to support as text.
Text is the universal serialization format. Doesn't mean it has to be unstructured, as is the UNIX tradition.
It took a look at your first link - Lucidity looks pretty neat, but is there a way to try it out? I didn't see it in your GitHub.
Unfortunately there isn't at the moment. The setup process is still fairly involved. Hopefully in the next couple of months I'll have something easier for others to use.
This is very similar to my own work for the last many years, and I'm sad and delighted he's managed to pull it off in such a clojure-native, slick looking way. Well done. I haven't finished watching the video yet, but it's a little deflating. Still though, if others had been doing this years ago I wouldn't have been scratching that itch.
It sounds like you'd easily be able to extend your work to the datify/nav protocols so your work wouldn't really be a waste. His viewer seems more a proof-of-concept.
Doesn't feel like a waste ;) The time spent investigating different GUIs in Clojure or integrating languages with better GUIs with Clojure feels does like a waste. I eventually settled on plain ol Swing but JavaFX looks quite nice.
clojure.spec was a nice step up to natively dealing with data 'shapes', I'm hoping the datify/nav protocols will be a step up to what I'm doing as well. It's basically a generalised browser over data with different print representations, collection handling, publishing, muxing, etc. The REBL will hopefully raise consciousness of just how abysmal the representation and navigation of the web is today.
From the license:
> You may not use REBL for commercial use.
Sometimes it seems as if Cognitect just doesn't want people to use Clojure. It's like they looked at the results from the survey[0], created a tool that appears to address some of the main gripes that people have, and then proceeded to say "screw you" to all of the people that use Clojure commercially and may actually pay for Datomic...
[0]: https://danielcompton.net/2018/03/28/clojure-survey-2018
I know how easy it is to come up with interpretations like that but the site guidelines ask you to push pause before simply posting them. They have a destructive effect, one that compounds nonlinearly.
"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."
Haskell's motto is "avoid success at all costs", but it feels like it applies more to Clojure.
I genuinely believe that Clojure could have taken over the world if it had a more sensible open-source governance model.
Closed source I understand, they must have something in there they don't want to give away completely for free. But "no commercial use" just cuts out something like 90% of the user base. Because depending on your definition of those terms it would also apply to anyone building anything with it that they plan on selling in the future. Even paid tutorials on how to use it would be out of the question.
It's like "how to not get people to use your tool" 101
I was at the talk, and IIRC he said something about developing it while working with a client. It's quite likely it's a contract obligation.
I don't know what world releasing a tool that addresses some of the main gripes that people have is a "screw you", but it doesn't seem like a pleasant one to inhabit. The important bits are in Clojure 1.10.
I think the exciting thing is the datafy protocol, not necessarily this particular viewer. What he demoed seemed much more like a proof-of-concept. Like he said, he's excited to see what the Clojurescript community comes up with, and so am I. I might take a shot at building a browser in Emacs, though that takes away some more dynamic display opportunities.
An emacs based browser for this would be awesome!
My wild guess is that it will be added rather soon than later to CIDER. The protocols are part of Clojure, and Emacs is great with browsing buffers, so it doesn't seem very hard to implement.
That said I'm not Emacs expert so it may be harder than I imagine.
Everything except HTML and charts will be easy - after all, if something can be represented as or drawn with plain text, Emacs is probably the best tool for working with it. HTML has some half-decent previewers available, but those are essentially text browsers in elisp. Unfortunately, painting arbitrary images gets unwieldy with Emacs. You can get away with ASCII column charts, but forget scatter plots or sparklines. But for viewing that, I wouldn't mind an external application to put on another screen.
Honest question, how do we interpret "commercial use"?
I'd assume it means you cannot sell REBL itself or a derivative, not that you cannot use it as a developer tool to build your project, which kind of makes sense.
the REBL is just a javaFX program that builds upon `datafy` and `nav` connecting with the REPL. I think soon we will get other implementations (CLJS!).
I wouldn't want to read too much into this particular prototype of REBL. I think they don't want to have people re-package this iteration and sell it as-is.
Should it be read as "you may not resell it" or "you may not use it while being paid by someone"? It's a bit unclear.
Doesnt that just mean you cant depoloy REBL as part of a comercial product? Im sure it doesnt mean you can't use the tool at work. Not that I like seeing restrictions in any licences but REBL isnt the cool thing here, its datify and nav.
"Commercial use" is clearly any work-related use. (You are also not allowed to redistribute it at all)
I'd really like to hear that stated unambiguously by someone at Cognitect. It just can't be right. No sane person would think they could enforce such a rule and no sane person would be scared to "violate" it. There's no trace of information indicating what it was used for or when. If there is, no one should use it anyway.
Stated unambiguously by someone at Cognitect: https://github.com/cognitect-labs/REBL-distro/commit/144acd0...
Smalltalk has had this type of thing for years...
But Smalltalk is all about managing state and Clojure is all about immutability, no?
For the most part sure. I was pointing more to the nifty left to right viewer the presenter was showing and more generally the qualities of a "live" system.
Wish he had released the tooling. The REBL system looks a bit like the smalltalk virtual machine mixed with elements of Elm's time-traveling debugger. There has been a real need for serious error handling in clojure for a long time (since the stack traces and exceptions are borderline unintelligible). This thing makes it look much more user friendly.
They tried to fix the errors with Spec but honestly I don't think that solution worked that well. Spec is great for generative testing and other solutions but no one wants to write specs for every single small little potential error that could happen.
The tooling is in datafy and nav, which are in Clojure 1.10. Everything in the UI just flows from those protocols.
What is "not data" that can be the result of eval; at first I thought he was talking about side effects, but 5 minutes after he first said that I'm still not sure what he was talking about.
Clojure has facilities for seamless interop with Java (or whatever platform it's hosted on). A lot of times when inter-operating with Java libraries, you get back an opaque Object with associated methods when you'd normally expect a transparent Clojure map or some other pure data structure if you were interacting with a purely Clojure library. Datafy is just a protocol that allows you to transform those opaque Objects into transparent data structures.
Clojure separates data from the functions that operate on it. In contrast, Objects intertwine those two things. You can roughly think of the "data" part of an Object as its attributes, and the "function" part its methods.
I still fail to see how methods are not data in the general sense, but while trying to answer my own question, it seems that "data" in clojure means roughly "Either an atomic type[1] or a collection"
1: an "atom" in clojure is not what it is in traditional lisp lingo; is there a name in clojure for type that is roughly the union of: symbol, number, character, boolean ? perhaps "primitive"?
This concept is pretty subtle. Rich Hickey does a great job at explaining it (as always) in "Are We There Yet?"[1]. He deconstructs objects into state (attributes) and behavior (methods), and then uses values (or data) to represent state, and functions to represent behavior. Give that a watch a couple of times to get a better handle on what all this means from the Clojure perspective.
[edit] To answer your other question, a value that isn't a collection in Clojure is called a scalar.
1: https://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hi...
Oh, so in TFA, Stuart is using the term "data" to mean "Value" in the sense that Rich talks about (Are We There Yet is only one of of many of his talks about values (I think he had one called something like The Value of Values).
Within Clojure almost all data is directly serializable to data literals. Java objects don't have this property directly and are guaranteed to exposed everything via methods. Datafy and Nav just give Clojure the power to codify your observations. You're right for the most part, this is just the tools for interop (and a bunch of cool stuff you can build on top of it).
Perhaps “value”? The language is data-oriented, and everything you mentioned in [1] except for “symbol” easily falls within that. In Clojure, a symbol always evaluates to a specific value too, so it seems like a good fit.
A Clojure atom is one of the few types that is not immutable, but the way it is updated is pretty strictly enforced by the language, and references to it at any given point in time are still immutable.
I don’t have any insight into your broader question.
I'm not entirely sure I get the distinction you are making. Sounds like you are just saying that not everything prints well. Which, yeah, that is true. As noted in the speech, sometimes it is just cumbersome to see printed.
When clojure meets smalltalk
And I am so stoked for it. Smalltalk did a few things right that have still not been properly copied by other systems, but smalltalk itself was doomed to obscurity the second all the major implementations went for the Common Lisp-style "ship the system as an image" brain damage.
While clojure is far from perfect, it actually has a story for shipping your code, in a form where you don't need to embarrass yourself when your customer gets it, into servers, web pages, desktops and cell phones.
Can you say more about what makes it “far from perfect” besides the error messages? What are its other weaknesses? I’m still in the honeymoon phase and want to hear more from folks who have shipped code.
I personally hate the startup time. Mainly because it prevents me from using it in places I really want it.
It's gotten a lot better over the years, but it's still pretty bad.
Now you can use GraalVM (with a few caveats) to native-compile Clojure code, though. It's still in its early stages, but it's possible.
Given that this is pretty much an inferior version of the Pharo Glamorous Toolkit [1], which has been available for years, it is very disappointing to not see a reference/attribution. Especially taking into account that the - pioneering - research happened in the Smalltalk camp. If I wanted to be cynical I would say that presenting half-assed re-implementations of good ideas found in non-popular projects (Smalltalk, Common Lisp) in an easily-digestible manner with a veneer of freshness, is very prevalent in the Clojure domain.
I find REBL to be inferior because it largely misses the "moldable tool" point of GT. REBL focuses on the data (almost solely one could say) but presents a rigid UI - other than the data-specific part - whilst GT allows _everything_ to be molded at runtime which is of course the better approach. GT obsessively focuses on the interplay between data and the person interacting with said data. In systems theory terms, person and data mesh together into cybernetic entanglement.
I interpreted the talk as the tool viewer was a proof-of-concept in order to advocate the benefits of using the nav and datafy protocols. I believe Stu is not claiming its particularly novel, but instead wants to inspire tool makers to use the protocols and make better tools, hence his mentioning of his curiosity of what the Clojurescript guys could make. I mean I had the idea of a data viewer with some partial implementation with no knowledge of GT, and another person in this thread has been working on a similar idea, the tool concept isn't that novel (based on the features you describe). On the other hand, Clojure being a dynamic language by default makes any open source dataviewer written in Clojure automatically "moldable". Further, because of the protocols, its easy to swap between different people's implementations of the viewers. Clojurescript in particular is at an advantage because of the vast web resources and Clojurescript can compile itself.
So is this like exploring the web via a browser using hateoas & mime types? But instead exploring edn on rebl using clojure's 'new' data protocols? Feels old hat... But anything to shine a light on data oriented programming (and clojure sure is beautiful)
I think is pretty cool, yes. But fix error messages please.
Error messages have been greatly improved in Clojure version 1.10[1][2]
[edit] more context
1: https://github.com/clojure/clojure/blob/master/changes.md#21...
2: https://mobile.twitter.com/puredanger/status/102810365424144...
.
I’d say that is just spreading FUD around. How come I see plenty of job postings? Not interested myself, as I’m happily running my own SaaS written in Clojure and ClojureScript, but seriously: if you want a job writing Clojure, it’s not that hard. I mean, if Walmart and Citibank are hiring Clojure developers, it’s not exactly „in decline”.
I really, really like Clojure. I just wish I had something to use it for. I've done lots of Common Lisp development previously and REPL-based programming is just great. I can't really do it in Python and it makes me sad.
> I can't really do it in Python and it makes me sad.
That really bugs me with a great many popular languages. There's a REPL, but the libraries and tooling don't really embrace it.
The only tools I've found with decent support for Python and other languages are all emacs based. Same goes for the Clojure and CL ones, actually. It's no surprise, I suppose, because this style of programming is so natural to an emacs user.
But it's difficult with Python as soon as you try to do anything beyond a single module. CL and Clojure have proper packages and namespaces so it works fine. But in Python you're limited to working on the context of a single module. If you modify a module that was imported then you have to restart the REPL because you can't reload modules. It just doesn't work.
In CL I would just load the entire system then eval whatever part I want to. From that point I would never not have the current version of the system completely loaded into the running image. Much the same with emacs itself. It is a running image of your emacs lisp system and you can just modify it and eval as-you-go. Does any other language even come close to this?
>Does any other language even come close to this?
Smalltalk
Interesting. But one of the points of Clojure is to get away from the object-oriented way so I'm not sure that I'd like Smalltalk.
You have to try it to really understand what OO actually is. Its not the OO you might be familiar with..
Yeah that is why it supports a subset of CLOS.
Erlang and elixir.
If I understand it correctly, Smalltalk is really nothing like Java and the mainstream OOP that Clojure is a reaction to.
Smalltalk is dynamic and Java is static.
Smalltalk, it is an experience, given that language, IDE and runtime/OS are all blended together, while Java can only do so much, even with the more advanced IDEs.
Clojure might be a reaction to C++/Java/C# OOP style, but it surely adopts Common Lisp OOP style.
I think if things have private state that can't be seen externally and that's idiomatic then there's a conflict, if not then there isn't. Do you think it's worth learning Smalltalk? Is it still possible?
Have a look at http://pharo.org/documentation (a modern smalltalk implementation), they have some good materials, also a discord server.
I think it's worth learning, even if you never use it.
> But in Python you're limited to working on the context of a single module. If you modify a module that was imported then you have to restart the REPL because you can't reload modules. It just doesn't work.
This is so frustrating, the Python REPL does some kind of module caching, and I haven't found how to disable it. When I asked on IRC a few years ago, I was told that I was using the REPL for the wrong purpose.
Have you considered using ipython as your REPL? I use ipython and in my profile, I have the following setting. This enables auto-reload and it works almost all the time.
Here is the option I have in my profile file. You can run this command inside the ipython session as well:
%autoreload 2
The times I have seen auto-reload fail is when there is lot of class creation magic going on or while dealing with global connection objects which sql alchemy might be creating. But, other than that it works very nicely.
I stumbled upon importlib.reload(module_name) in the Spyder IPython console, which I use as a REPL alongside the code editor in Spyder. REPL based programming has spoiled me and I find myself drifting towards lispy things like Clojure for the REPL alone, nevermind all the other good things.
Haskell (in the form of intero+emacs at least) also has very nice REPL integration and will automatically reload any changed modules and any modules that depend on the changed modules. The debugging facilities are nowhere near what CL/SLIME offers, but I find I need a debugger much less often in Haskell anyway.
If you look at commercial tooling for Common Lisp, they can do a little more than plain old Emacs.
It doesn't help that most languages that advertise having a REPL really have more of an interactive shell than a true read-eval-print-loop. Lisps do a really good job of making a distinction between the reader, evaluator, and the printer. It makes it really easy to create tools that use just one or two of these or insert themselves between the different layers to expand the REPLs capabilities.
Yes indeed, and now that repl.it is popular with noob programmers I think that distinction has been lost. Who told them that what they are offering is a REPL? Sigh...
Yeah, PHP supports a REPL too! I sometimes found it useful when working on mediawiki to debug issues when a full blown debugger wasn't necessary. And if you were familiar with the various internal API's it was pretty easy to do simple maintenance tasks through the REPL without needing to write a script.
Which one do you use? I ended up rolling my own years ago and it's worked out fine (other than the fact I've never gotten readln support working), but I've noticed a proliferation of other solutions since about the release of PHP 7 and wonder if they're an improvement.
I used Boris, and the php debugger as a pretty effective shell sometimes. mediawiki had eval.php, which is apparently now shell.php
https://m.mediawiki.org/wiki/Special:MyLanguage/Manual:Shell...
Based on my experience with that I’d definitely add a shell utility to any large application I was working on.
I use ipython with auto reload and it works really well in general. Not sure how anyone lives without it.
I made a video to demonstrate it some time ago. Apologies for the dreadful sound quality. https://youtu.be/k-mAuNY9szI
OK yeah this looks good. Was expecting a tech demo but this is such a sales pitch glorifying a pretty common solution to a pretty common problem. The Intro on eval result not being a data is something that I don't get. If it's not data then what is it? Atleast what is it for the thing that's presenting it? All these while the folks were flaunting how awesome the REPL experience has been and suddenly Stu gives a small intro on how painful it is to eval a code. Just to glorify how awesome this REBL wud be? . I generally love most talks from clj community but not this one.
> The Intro on eval result not being a data is something that I don't get. If it's not data then what is it?
For an example - while technically a set of all prime numbers is data - you can't just view it in the REPL as a whole. So I interpret the "not data" notion as a shortcut to saying - a subset of data and optionally some way to navigate throgh the rest of it.
Unwritten rule of HN is that, one must be a blind supporter of anything clojure, Haskell and must be a blind hater of anything java, Javascript. Downvoters:care to provide reasons??
It's probably the accusational tone and sweeping generalisation coupled with the absence of proof, data, or at least examples to back it up. Also, it doesn't even tangentially touch on the actual subject of the post.