Some Context
Janet is a Lisp inspired by Clojure, implemented in transparent C89, making small (1 mb) binaries which use a few mb of RAM. Many people love this project and craft amazing things with love and joy.
Our community is growing and has a few big efforts. I think reaching consensus on goals will help us make informed decisions and achieve them. To that end, I interviewed Bakpakin hoping to jump-start a conversation.
In the past, he wrote a comment asking for and noting some short term goals and explaining some values:
While adoption is always nice, Janet is not a commercial endeavor and my main goal is “to make a useful lisp-like programming language for scripting and extending C applications”. The hope is that if the software is good, users will come. To that end, I think the core language is mostly there, and adding more functionality is encroaching on “bloat”. However, we are not there yet and there is room to improve.
However, wider adoption does make Janet more useful (more people adding code, more insight, more manpower, etc.), so “adoption” is just one factor that make Janet better software (and perhaps a better community).
Our sister project Fennel has rationale and values docs, Clojure has a rationale too. I’m not sure whether we need such a document, but they’re nifty.
Interview
You once PMed me:
I have been slow to respond to Janet inquiries and discussions recently (such is life), and as such have tried to delegate some power and responsibility to people who have been both competent and consistent.
Technomancy mentioned you two spoke about how to make a project live beyond us. To this effect, active community building could help. I’ll ask questions about your thoughts on the project to jump start some consensus building. To what extent is this just your personal project which we like, and you do things to just for you vs. a community thing?
I try to have empathy for the user. Frankly, I have more empathy for existing users than new users. Existing users are more important than new users. There is no corporate interest, I’m not selling anything so there’s no need to grow or sell more. We just make it good for the people who care. I’m more caring about that than making it good for myself.
With making things last, the idea is enabling forwards compatibility and non-breaking changes, making sure there are resources for future programmers even when we’ve all moved on. Documentation shouldn’t disappear, forum posting should at least be public if not well organized.
I’ll talk about how this project came about, where it went and how I see and “run” the community. I say “run” in quotes because in truth, I don’t really run it, which causes some confusion.
This started out with Fennel. Later, Phil found it and the compiler has since been rewritten. I don’t think any of my original code has been left. So that’s sort of a separate project now.
My Fennel efforts became Janet. It’s a real language, a real compiler, a real interpreter. Importantly, it does not serve any corporate interest. There’s no overarching goal beyond being a general purpose scripting language. When people ask “what’s the point? What’s the goal?” The goal is to be a tool that’s useful. I get contributions via PRs or suggestions and if they match my vision, I incorporate them. At the beginning, people advised me to set up a website and donations, which in retrospect weren’t necessary but it’s really good to have a website, documentation and community!
I don’t want the community owned by people. But in a certain sense, everything is professionally attatched to me, so I moderate to make sure nothing crazy’s going on.
As far as project direction, there’s a deference to taste lest it go off the rails and someone fills it with 10 features specific to their needs. I want to point out that the Janet project is done in a sense. We have incremental updates and people will always want to fix things, but big sweeping changes would be better served by a new project, which I don’t have a strict need for right now; Janet works for me. A lot of the big changes I’d make would be for the runtime, low level things, improving the compiler. But if you have suggestions, I’m happy to hear them. No one’s immune to a well written argument. If you post the problem you have, what you ran into and have a good idea, that’s really convincing! Changes solving concrete problems are more convincing than changes helping to solve general problems.
Real improvements to Janet require starting over to enable nice changes and make it run faster without compromising its use. But a new garbage collector, internal representations JIT-compiler would break backwards compatability, Right now, the goal’s no significant backward incompatible changes (barring fixing “broken” stuff). I don’t think there will ever be Janet 2. It’d have to be a new project. Stuff that can’t meet that level of stability is generally moved to spork or external packages.
There have been some attempts to rationalize documentation and vocabulary. What are your thoughts about this?
I care about documentation, but frankly it’s just a lot of work. There is inconsistent naming and terminology, because most of it was written probably 5 years ago while the language has evolved. But the fact that it’s still there is a testament that the language hasn’t change too much. If we go back, we could clarify things, but I have not put a ton of effort into modifying the prose, so there’s a lack of precision in it. I really appreciate when people put in effort to, you know, keep things nice and help with documentation and community. I know I’ve described the project like anarchy, but I’m still very appreciative of the help. I just mean I’m not your boss is what I’m saying.
Our documentation and discussions are also all scattered. I first set up an IRC channel, then the freenode fiasco happened. I found IRC a bit archaic, what with the bouncers etc. so I set up GitHub discussions which GitHub introduced during the drama but we eventually settled on Zulip.
Community management… what does it mean to not be owned by people?
There’s no formal governance structure. If you own the domain, you own it. I prefer that kind of organization because when legal issues appear, that’s how it’ll work. If something happens on my domain, it’s my problem. I guess there’s not a place for a larger organization right now.
When I started the project, I didn’t care about copyleft, but if I were doing this all over again I might go GPL. Anyway, if you don’t like it, you can fork it. If you want to merge the forks later and I like the idea, we’ll do that. But I’ve gotten plenty of good suggestions related to garbage collection, compilers and performance which sound good, but there’s always a long tail of little problems and a lot of work. I am generally hesitant of change that I am uncomfortable of going in to fix myself if there is an issue, but if something is well reasoned and tested, that is pretty convincing and more likely to get merged.
The Go people have the goal of keeping the compiler simple over any little optimization, do you agree with that?
Yes.
Really where I care about the quality, there’s the Janet codebase, specifically C which has to be good because of security repercussions. I also care about Spork, our contrib library. If I were to redo it, Spork would be part of Janet, one repository, versioned and tested together. So that’s where I try to maintain the level of quality. That means it’s tested, matches things, there’s not bad performance, specifically big O performance. Janet’s not ever going to win any speed contest. Here are some C style points:
- Don’t break userspace (existing programs)
- Keep the doc strings short and mostly plain text - they are included in binaries
- C code is C99 + platform specifics, mainly for MSVC, clang, and gcc
- C code - correctness is more important that making it pretty or “clean”. Don’t do anything too stupid, but high performance is not a primary goal.
- Please don’t mess with the build system too much
There’s a contribution guide, git history and example code too. At the end of the day, it’s hard to just codify every little opinion I have, so I try not to be too nit-picky.
As for things which other people like and I don’t, people want immutable data structures to just work and be fast. The truth is, I’ve never seen that just work and be fast, ever, anywhere. There are some technical reasons why it’s not great for Janet.
I like Clojure’s “change by accretion”, trying to add new things rather than breaking old things, function2 instead of breaking old code. The package managers are a salient example of this with our tools.
What Clojure practices do you like or dislike?
I like the aesthetics, general syntax and macros. A lot of this comes from Common Lisp, but terser than Common Lisp’s verbose default functions. So like Clojure, terse code is another a goal.
We have a very small core language, 13 special forms. We had less but grew it out. We build that out into 600 built-in forms.
But that’s just the Syntax, the runtime is more like Lua, Python or your favorite scripting language. There’s a simple garbage collector, no object system beneath it which would make it more like Clojure. Lately, I’ve been mostly using it like Python for file processing and text manipulation especially.
Package management is hot right now, receiving a lot of effort and forming big catalyst for this interview. We had JPM and now “modern” bundles and janet-pm. My undestanding’s that the bundle stuff does everything but networking and UI which a Janet package manager should add.
Regarding things like package managers, there were technical problems with our original PM JPM being hard to configue. We had many bug reports where people were misconfiguring it by accident or accidentially missed a small skip when following documentation. Another issue’s that it was tied to C tool chains, to GCC, Clang how your system is set up. The second iteration decoupled that so you could use one part e.g. just the build part. I have a local project where I don’t use all the package stuff, just C compiler extractions. Many people do the opposite, using PM but don’t care about C compilers.
There are a lot of ways to look at this. In the end, a package manager is just something which moves files around. I’ll give you an example research/self learning project of mine with computer graphics, which uses Janet and the new bundle stuff. What it doesn’t do is handle dependencies or package management, it just uses the build system. It uses bundles but doesn’t download from the internet. That distinguishes what I think is a common use case.
This project links to some system libraries but also a bunch of assets like images 3D models, shaders etc. that I want copied over when I install it. So the bundle isn’t one module, it’s a collection of things which are versioned and released together while a module is a single import. There’s no requirement that a bundle only has one module while a bundle could have a hundred or no modules, like a system font if you’re being creative. There’s no requirement to use the dependency system we established with JPM, you coud just vendor your dependencies and put them into subversion. (I know the open source world just uses git, but in industry you see a lot of other things. So it should be agnostic to git in a way many nascent projects are not.) It’s also not dependent on any centralized file server, which is expensive infrastructure and a legal entity.
I don’t use so many dependencies in general. But frankly, there’s enough people on the internet giving advice on how to do software engineering without evidence, so I won’t go too far into that. It is harder to test with a lot of dependencies.
Since bundle focuses on shared/universal features, making it easier to make a package manager would you prefer many coexisting or for everyone to coalesce on a single one?
I have some points about PMs which I know are a little unpopular, which causes some friction. I don’t want to be building something I don’t think’s a good thing to build. But clearly people kind of want that. But I also know it’s a pain in the butt to manage things yourself for every project, use git sub modules and custom build systems…. It would be great if there were only one.
The Janet binary itself has bootstrapping capabilities to build a real full package manager which does things people really want vs. what I want which is…. I don’t really care. I just use git to vendor in dependencies (editor note: He manually uses git clone or just copypastes the necessary code in) and not worry about other people’s problems to be frank.
There’s a good point for why I’m doing this. Other people have been very nice to help, I know Bob Tolbert, Michael Camilleri and Sogaiu have been fixing up my 80% code and I’m very thankful for that.
There are 2 parts of the tool. There’s a package manager which I’m not a big fan of and a build tool, which is very necessary.
Janet being a lisp, the build tool should be self-hosted, so the REPL, the runtime can build things, call to subprocesses, coordinate its real language. That’s a great example of metaprogramming.
Package managers on the other hand aren’t useful without strong curation, which is difficut and expensive. You need a human in the loop. There are also second order effects like name squatting. I don’t want to build a package manager without a lot of curation. What we have already is already a bit unnecessary, like the package listing. You can’t just put your own package with malware on there, Pepe checks first. A few have commit privilege; I see commit rights as full trust for the repository. So, there’s a lot more human than technical aspects of a package manager.
Many people have proposed rationalizing naming conventions, adding exclamation marks to set etc. Obviously that would break a lot of things, what about aliasing them? Is that pointless because current users are already happy?
It doesn’t technically break things, but keep in mind the namespace of all default symbols. As it grows, you’re taking away common or short binding names from people.
Many don’t like the term “namespace”.
To me a namespace is a space which contains names. There’s no concrete abstraction there. There’s just a set of names, of strings you can choose from.
Weren’t you working on a JIT?
There’s a branch with a DSL to compile eventually to machine code, but I won’t finish anytime soon.
You’ve written that you prefer shorter docstrings, because every byte goes into every Janet binary. So small binaries is another goal?
It is in part about saving disk space but more about saving RAM. I wouldn’t say minimalism is a goal for Janet, but we should not be particularly wasteful. If binary sizes and ram requirements grow significantly over time, that is a form of breaking compatibility on less powerful systems.
What inspired Janet’s fiber approach?
I’d been playing around with Lua coroutines, its version of green fibers. I ran into a problem where I was trying to use them for both an event loop and error handling but you can’t because you can only yield one kind of yield. That’s where the signals come in. That has its own complications; it’s not super simple. But it allows you to use fibers for many different things and build up your own schedule. It came from Lua, a bit of Erlang. There used to be a version of Janet with native threads directly, with a thread module. But actually that might have been a good idea… The event loop is nice until it isn’t, if your operating system doesn’t play nice with it…
That inspired all the dyn stuff too.
As opposed to var, dynamics will automatically fix themselves when you handle an error which could leave a var in a bad state when messsing with other threads, runtimes, parallel processes… It’s not as efficient though.
Why did you walk away from traditional lisp syntax for ~ and ;?
The syntax was more of an accident - the original plan for syntax was actually going to be more like Python or shell, hence # for comments, with lisp syntax being used to bootstrap things. However, there was no need to change things later.
The community has some idiosyncratic tastes like def everywhere.
There’s a really simple technical reasons for def. It’s just that everything has to use what’s in the file ahead of it because we don’t have mutual recurrence nor a linking stage, so boot.janet uses a lot of def.
I guess the whole community just really liked boot.janet and ignored e.g. gendoc which uses big lets!