crystal i by asterite · Pull Request #10910 · crystal-lang/crystal

5 min read Original article ↗

The experimental crystal i (interactive) command is here!

What is crystal i

crystal i stands for "interactive crystal". It can be used in two modes right now:

  1. crystal i: opens an interactive crystal session (REPL) similar to irb from Ruby
  2. crystal i file.cr: runs a file in interpreted mode

In any of these two modes, you can use debugger in your code to debug it at that point. This is similar tu Ruby's pry. There you can use these commands (similar to pry too):

  • step: go to the next line/instruction, possibly going inside a method
  • next: go to the next line/instruction, doesn't enter into methods
  • finish: exit the current method
  • continue: resume execution
  • whereami: show where the debugger is

By the way, before this PR you could also use debugger and lldb or gdb would stop at that point, if compiled with -d (otherwise it seems the program just crashes, maybe we should just ignore debugger in that case: something to improve outside of this PR)

Overview and some technical details

When running in interpreted mode, semantic analysis is done as usual, but instead of then using LLVM to generate code, we compile code to bytecode (custom bytecode defined in this PR, totally unrelated to LLVM). Then there's an interpreter that understands this bytecode. All of this for a stack-based machine. If you know nothing about interpreters and you want to learn more, I recommend reading the excellent https://www.craftinginterpreters.com/a-bytecode-virtual-machine.html

The memory layout of things in interpreted mode matches the memory layout of compiled mode: a union of Int32 | String occupies 16 bytes in both modes, and there's an 8 bytes tag for the type ID. That said, the values for type IDs might not match between the two versions, and this makes the two worlds kind of incompatible (they can't interact with each other.)

C calls, and C callbacks, are implemented using libffi together with dlopen.

Status

I'm just opening this PR to give a chance for others to start looking and commenting at the code, and ask questions if things are not clear. That said, I'd still like to take some time to clean up and comment a lot of the code before merging this. A lot of this was rushed because I wanted to have a big chunk ready for the conference.

Some things that are missing to implement related to the language:

  • Raising and rescuing exceptions
  • Closures
  • Calling C functions that are outside of LibC (or outside the functions already included in the compiler, I think, because LibGC works)
  • Some other primitives (some atomic functions, some math functions) and I'm sure some others more

And then some things that I know don't work fine right now:

  • defining types and methods inside crystal i doesn't work super well, specially if you redefine methods or if you extend types (this is the real challenge, and we could maybe just postpone this one for how long as we want)
  • it seems that defining variables when in debugger mode (let's call it pry from now on) doesn't work. I think this one should be easy to fix but I didn't have time to look into it
  • free variables don't work well in pry don't work well (I'm not sure how to fix this, I think it requires a change also to the "real" Crystal)
  • if an error happens (exception, or maybe something doesn't compile), I think it gets stuck in that mode (this should be easy to fix too)
  • macro finished isn't executed yet
  • at_exit isn't executed yet (because main isn't executed at all)

Focus

For now, I'd like to focus on polishing the existing code and making sure it's 100% understandable and documented (some comments are already outdated, some method names are long and not nice, there might be some redundant code, and I'm sure there's already unclear code). After that we can merge this, put an "experimental" label next to the command, and then implement the missing things or fix existing bugs in separate PRs, which are going to be easier to review and merge like that.

I'm also open for suggestions on how to improve the code, be it for code clarity of for performance.

Also sorry beforehand for the commit (mis)organization and commit messages, this was all very experimental.

Should this be in a separate repository?

This PR has some changes to the standard library, mainly some methods and functions need to be emulated in interpreted mode, and there's a new flag?(:interpreted) thing you can use. These could be in a separate PR, and then we could maybe extract the tool to a separate repository. But I think that having interpreted mode in the main repository, like every other tool, is fine from now. I believe moving tools outside of this repository is a separate discussion. But for instance, eventually crystal play could start using interpreted mode under the hood, or maybe you will be able to do crystal spec -i ... to run in interpreted mode. So, again, having all of this in this repository is probably fine.

About FFI, dlopen and readline

Right now I put them in the top-level just for simplicity. We should probably move them under the Crystal namespace because we don't want to publicly support these.

There are now inside the Crystal namespace ✅

More thoughts

  • It's clear I never ran the full spec suit locally in this branch 😊

Steps before merging this PR (will be updated as we work on it)

  • Install readline and libffi on CI
  • Document every nook and cranny of the interpreter