Ki Editor - an editor that operates on the AST

Mine are:

Cmd+Shift+V - Stacked clipboard, you can start typing to search or hit a number to choose what to paste (keeps everything you've copied/cut inside jetbrains for a while)

Cmd+Shift+E - Recent locations, you can start typing to search - shows little buffers of where you've been recently

Cmd+Shift+A - Action tab of the command palette - fuzzy search for any command (really the only shortcut you need, other than maybe Shift+Shift for main command palette shortcut)

--- Through the Action bar...

Local History / Local History of Selection - you can start typing to search quite far back the history of all changes of the current file or selection - you can also right click a folder or the project and do the same. Much finer grained than git.

The general concept of being able to search for something and edit directly in the buffer of the search results.

Mathematica is the earliest thing I am aware of with this feature where it was Alt+. to expand selection in their notebook interface starting in the early 90s. But the thing I miss most that I still can't shake the muscle memory of after almost a decade of not using much Mathematica, is that single/double/triple/n-click scaled this way as well. So double-click selected a whole word (as in all editors), triple-click selected all the comma-separated multiple args of a function, 4-click for f(a,r,g,s), and so on.

Also available as `incremental selection` in Neovim via tree-sitter.

I work on AST based revision control. I have a stack of ideas on how to achieve the same Ctrl+W effect with commits/diffs/cherry-picks. All still in flux. If you have some thoughts to share, please do.

[1]: https://github.com/gritzko/librdx/tree/master/be#readme

I use it constantly in helix too. The vscode one is meh. I think I saw a discussion in github once about switching to tree-sitter, which would improve AST-related actions. I don't think it went anywhere though.

I love AST aware editing. I think it's one reason it's always been so nice to edit lisps. Stuff that is complicated to describe in javascript (and doesn't have LSP support) p much requires a whole AST parser, but in lisp it's just a simple list operation. When I go back typescript after a weekend of clojure, I reeaally miss slurp! and other paredit commands

I agree, top feature. Combined with things like "extract method" makes mundane refactorings super fast.

Yes, Java IDEs have had these since sometime in the 2000s.

To me, it feels like Zed and VsCode perform most operations in a general way on the text; they don't seem to (in Python and Rust at least) have an understanding of the code structure in the way JB does. (And based on some digging on Ki the way it does as well?) So, I would bet they are using that text-based model, which would be hit/miss here.

One problem is I got so used to Ctrl-W that I use it in other applications and usually wind up inadvertently closing the tab.

This has got to be my favourite feature of IntelliJ, along with the dumb context actions menu

Cool feature. what's the key for it in visual studio code?

Ty! I have been missing out. Adding this to the repertoire.

JetBrains also experimented with AST-based editing: https://www.jetbrains.com/help/mps/fast-track-to-mps.html#st...

An overview video: https://www.youtube.com/watch?v=XGm_khXZl44

I tried it, but it just was too clumsy. Sometimes refactoring/editing needs to go through phases where the AST is invalid, and MPS makes that just too clumsy.

But with AI this might be a different story.

Ah yes.

The shortcut I use the most in Jetbrains IDEs. Also the one I miss the most in VSCode (whatever is present there just doesn't seem to work right).

Also the shortcut that has caused me to close so many browser tabs inadvertently...

> The challenge is getting this to be a useable way of entering programs.

Well exactly.

When the path between Program A and Program B can only be valid programs, you are going to end up with either a much longer, less intuitive path, or deleting everything and starting again. It can also be quite possible to invent structures which are valid but have no valid path to creating them.

Pantograph[0] seems to be a more recent attempt to implement the same idea. It is still not a general editor but generalizing it to ranges of tree selections looks promising

[0]: https://pantographeditor.github.io/Pantograph/

If you want to relive it then simh¹ with mame² might be an option. There appears to be some support for VT11³, along with docs⁴ to use as a starting point.

No, I'm not claiming to have read all one hundred pages already. However, from what I have read I'd love to see a functional demo.

¹ https://simh.trailing-edge.com/

² https://www.mamedev.org/

³ https://github.com/simh/simh/blob/master/PDP11/pdp11_vt.c

⁴ https://wiki.mamedev.org/index.php/MAME_and_SIMH

I actually used a language and editor like this in a previous large company.

It was an experimental language that they wanted to replace their current application level language and was built ontop of JetBrains MPS https://www.jetbrains.com/mps/ which has that feature among others.

My personal opinion after working with that lang is all of this is that its theoretically interesting. But a dead end in practice along with visual coding etc.

The universality, simplicity, and legibility of text as interface for both humans and machines is too hard to beat. I think LLMs is just the most recent example of this.

Things that don't work in practice:

- You need a special editor, its heavy and slow, you lose all the ecosystem around them.

- You can't just cat or inspect the raw file, you can't see it in terminal at all.

- You need a new version control system, review system, and people need to learn it.

- You can't use any existing tools for working with code, you are essentially starting from scratch, and lose all the benefits of all the work everyone else is doing around tooling, dev saas etc.

- humans don't think in trees, they don't think in syntactically correct programs. Its actually absurdly frustrating writing a program in only syntactically correct edits. 99% of the time you are coding, your work is probably syntactically and semantically incorrect.

The tooling that is able to either make the right tradeoff around strictness or allow the users to make the that tradeoff is what ends up being used in practice. A good example of this typescript with gradual typing, python with type annotations, etc.

I think these editors just fall in a bit too far right on the strictness scale.

Now you're out of the realm of Ki, but what you're talking about is still being worked on in the modern era, by me! I'm building BABLR which is a modern follow-up to your idea, built on top of a powerful, generic system of parsers which has gaps/holes but no error recovery, so that it works with trees which may be incomplete but which must not be invalid.

The hard part is that we need to be able to talk about these structures. Even just here on this forum we need to be able to communicate precisely about them. I often use · as a typesetting symbol so that I can easily write and read expressions like 2 + · which you would read as "two plus gap". The · symbol is only for typesetting as I say thought because you it's not safe to assume that any one character is reserved for our use in every programming language. Instead we wrap the parts that aren't syntactic in quotes and we use <//> as the symbol for a gap so that it looks more like this:

  <*> "2 + " <//> </>

We can parse 2 + · into a proper tree now:

  <BinaryExpression>
    left: <*Number "2" />
    #: " "
    operator: <* "+" />
    #: " "
    right: <//>
  </>

Wow! I stumbled onto your paper a while ago when I was looking into structural editing and thinking of a masters in CS, exploring editor interfaces / feedback loops.

(Maybe your paper is famous and it’s not wild that I read it, but it was wild to see after so many years)

I never took that path, spent time in tech industry confused why people didn’t seem interested in structural editing and better editing tools.

Out of curiosity, how do you think LLMs and genai affect the value of structural editors and similar tooling?

Part of me wants to stay disciplined— of course it’s valuable to work efficiently and work on the AST and with a repl. The other part of me gets paid to work on essentially a punch card system (building dev, ship to prod and see what happens)

> I guess it's hard for me to edit things that I don't see right in front of me or aren't super simple changes (like name changes). Or at least, basic things I can reason about (such as finding by regex then deleting by textobject or something).

This is actually what's nice about tools like ast-grep. The pattern language reads almost like the code itself so you can see the transformation right in front of you (at least for small-scale cases) and reason about it. TypeScript examples:

  # convert guard clauses to optional chaining
  ast-grep -pattern '$A && $A.$B' --rewrite '$A?.$B' -lang ts

  # convert self-assignment to nullish coalescing assignment
  ast-grep -pattern '$X = $X ?? $Y' --rewrite '$X ??= $Y' -l ts

  # convert arrow functions to function declarations (need separate patterns for async & for return-type-annotated though)
  ast-grep -pattern 'const $NAME = ($$$PARAMS) => { $$$BODY }' --rewrite 'function $NAME($$$PARAMS) { $$$BODY }' -l ts

  # convert indexOf checks to .includes()
  ast-grep -pattern '$A.indexOf($B) !== -1' --rewrite '$A.includes($B)' -l ts

Sadly coding agents are still pretty bad at writing ast-grep patterns probably due to sparse training data. Hopefully that improves. The tool itself is solid!

TBH, it's actually not as hard as you think, most of the time, what I do is just select the whole syntax node and delete it, copy it, or replace it, and only 20% of the time would actually require deliberate understanding of how the AST is structured in the current language I'm coding in.

Maybe, or maybe not. I think there are many different kinds of work all lumped together under the "software engineering" umbrella, and tools which are indispensable for some people can seem unimpressive or unnecessary to others. We're not all spending our time in the same way, working on the same kinds of tasks.

I don't find myself doing huge structural refactoring often enough that it would be worth my while to learn a specialized tool for doing it; I'm a quick typist, and it's easy enough to just blat through the changes by hand on the rare occasion such a thing is necessary. But I don't work on giant line-of-business apps or sprawling web services, and I can see someone who did a lot of that sort of work having a different take on the matter.

I feel the same way. Learning Elixir and getting into macro writing was really helpful for me. Lisp is also the same way though for me the syntax of Elixir resonated a lot better for me. Many paths to the destination and all that, but figured I'd share in case it's helpful to someone

4. All of the above.

Which is Emacs.

Exactly the same for me. So far, there have been many challengers but the same champion (subjective, I know).

I am working on a modal code editor project that you might find interesting then. It also operates on an AST directly, which is represented as UI nodes which closely resemble normal text layout. Email in profile if you’d like to give it a try and possibly give early feedback (still in early development).

Vim improvement

So Ed Visual Mode Improved Improved!

This one?

https://marketplace.visualstudio.com/items?itemName=ki-edito...

Thank you for that! I find it really daunting to try a new editor, but having a vs code extension as an entry point really helps me and increases the chances of trying it out soon!

I wrote the vscode extension for Ki. It would almost certainly be usable with Zed since it talks with the editor over a websocket. Check out the code in the Ki repo.

It's a shame that editors are build around the editing model, so much time has been wasted by recreating everything. Neovim is a perfect editor in many aspects but editing model is not one of them, in my opinion. Helix or even Ki would not be needed if Neovim allowed to completely replace the vim part of it with plugins

An editor that tries to be layout agnostic is almost certainly going to be a nightmare for people like me who set the layout in the keyboard itself. I downloaded the editor (I'm on Windows at the moment), and tried it with my keyboard set to Dvorak, which was plainly broken. I'm sure there's a way to fix the mapping, but when software thinks it's smarter than you, you end up feeling pretty dumb.

Yeah and using combobulate feels so natural, I just wish it was easier to add support for other languages. In addition to the navigation, and simple edits (e.g. M-k kills node) you have Searching/editing/etc. using tree-sitter queries directly within emacs, possibly replacing other uses of external tree-sitter based tools.

It makes editing supported languages almost as pleasant as editing Lisp.

With how well the integration already work in existing editors, I'm curious how the UX can be improved further by an editor made specifically with AST editing in mind.

I think there's still value in generic AST-level operations. Like expand selection and shrink selection. Or select around the current node vs select inside (whatever that means for the current node type).

In Ki you don't have to know the name of the syntax node, you can just press `d m`, and the editor will show the labels of all the syntax node visible that you can jump to.

That comparison table is strange and sometimes wrong. Neovim for example detects and updates external file changes by default. And the coherence of the keybindings in Ki is "Great" bit vim/helix:

> As you can see, there's no single logical categorization for these keymaps, they are either lowercase-uppercase, normal-alt, left-right bracket, or outright unexplainable.

Word, End, Back, Change Word and even Change Inner (, etc are very logical to me and I feel like I'm talking to the editor when editing. I get that it doesn't make sense when one has learned another way to do it, but it does make total sense you just have to make an effort to try and understand it.

It's like learning and always driving automatic then calling manual "outright unexplainable". You simply learned another way and are conditioned into believing that's the one true way. It shows the creator comes from VSCode (multi-cursor is a useless feature, just use s/search/replace and get used to macros and a whole new world will open).

https://github.com/mickeynp/combobulate

You just complete to a parsable structure.

This stuff reminds me of intentional programming (Charles Simonyi). That never really panned out but the premise with intentional programming was that programming is about AST transformations that are correct (sort of like refactoring) but all the way down to the machine code level. You get these increasingly more complex languages that are defined in terms of building blocks that are ASTs.

Refactoring of course is based on the same notion that manipulating code correctly at the AST level is a lot easier than messing around with text files. That came out of the smalltalk world originally which also had very fancy IDEs already in the nineties where everything (including the IDE) was something you could mess with. Refactorings were defined as syntactically correct transformation of one AST into another (rename, extract class, move function to parent class, etc.).

Charles Simony had a solution for what you describe as well. Just serialize stuff to text if somebody wants to "edit" and then parse it back when they are done. What happens to the text in between of course is the programmer's problem.

I assume it uses an error tolerant parser, same as most language servers.

Ex: https://github.com/microsoft/tolerant-php-parser/blob/main/d...

Have you actually tried it?

True, but muscle memory and a couple keyboard shortcuts (or heck, even using the mouse to select and drag the block) is always going to be faster then describing the changes you want and reviewing the output, at least for simple stuff.

There was Weave which just uses git for VCS and tree-sitter for conflicts. Gritzko is on-thread with his Beagle system, and I'm here too for BABLR ; )

Diffsitter and/or weave?

Please feel free to ask for help in https://ki-editor.zulipchat.com/join/zzhagqzl6wyzpqfeqxcsrki... :)

What can you tell us about it? What worked and what didn't? What was hard?

We have one underway in https://codeberg.org/alicealysia/ki-bindings.nvim

> Note that the Ki keybindings cannot be simply implemented in Vim/Helix via key- remapping, due to the lack of the concept of Selection Mode, and implementing that requires major architectural changes in the core.

It's funny you mention that. I am currently working on a neovim plugin which implements the ki-editor keymap.

https://codeberg.org/alicealysia/ki-bindings.nvim

Unfortunately however, there's some shortcomings of neovim which make its control scheme impossible (at least not without some workarounds)

One of the biggest is ki's momentary layers feature.

Ki uses the kitty keyboard protocol to detect when a key is being held and when it is released in order to allow for unique actions when multiple keys are pressed simultaneously, and this functionality is a big part of what makes it so ergonomic.

For example, tapping c will copy the currently highlighted text, but pressing c and k simultaneously will duplicate the currently selected text to a new line below the current line.

While I'm currently chipping away at a pull request to introduce similar functionality to neovim itself, in the meantime, I'm needing to work around the issue by making a lot of concessions.