Exploring Toit executables

I recently developed a small command-line tool called ai, utilizing OpenAI on the command line. Although there are many similar tools available, I wanted to create my own. Toit already includes an OpenAI package, and the remaining code is just a handful of lines. This also lets me tailor the tool’s API to suit my requirements. Those interested in the code can access it here.

Press enter or click to view image in full size

Vessel with tiger decoration.

The program’s concept was to use GPT-3.5 when called with ai and GPT-4 when called with ai4. At startup, the program just has to examine its name to determine the appropriate model to use.

Once written, I shared the executable with my brother, who immediately noticed that it wasn’t working correctly for him:

% echo "What gpt version are you?" | ~/local/bin/ai4
I am based on the GPT-3 model.

For fun he attempted to inspect the internals using ghidra, but his efforts were not successful. This prompted my curiosity about the complexity of inspecting Toit executables.

The Toit SDK provides a compiler capable of generating executables for various platforms. Rather than generating machine code, the compiler produces bytecode for the virtual machine, which is then loaded into prebuilt “vessels” designed to accommodate the bytecodes. The sdk/vessels directory contains prebuilt vessels for the host platform and optionally for different platforms/architectures. When downloading the Toit SDK from GitHub’s release page, a selection of vessels is already packaged with the SDK. When building locally, these vessels can also be downloaded separately.

Although Toit lacks a tool specifically for inspecting executables, it includes a program named toitp in the tools directory, which can be used to examine snapshots (bytecodes). By extracting the bytecodes from the executables, we can analyze them using toitp. The challenge lies in identifying where within the executables the bytecodes are stored. Interestingly, the simplest method to determine the offset is to compare two distinct executables (or the empty vessel and the executable). The first altered offset will indicate the offset of the snapshot-size, which is located just before the snapshot data itself (refer to the code):

  int snapshot_size = reinterpret_cast<uint32*>(vessel_snapshot_data)[0];
  uint8* snapshot = &vessel_snapshot_data[4];

One minor hurdle remains: identifying the correct vessel to compare with the executable. For convenience, the SDK includes prebuilt vessels of various sizes capable of holding different bytecodes. The smallest vessel is named vessel128, and the largest is vessel8192, with vessels for snapshots of size 256, 512, and 1024 in between. Given the vessels’ distinct sizes, we can compare them with the executable sizes to find a match.

ᐅ ls -l ~/local/bin/ai /opt/toit-sdk/vessels
-rwxr-xr-x 1 flo  flo  1624104 Mar 17 16:46 /home/flo/local/bin/ai/opt/toit-sdk/vessels:
total 15492
-rwxr-xr-x 1 root root 2148496 Mar  9 16:13 vessel1024
-rwxr-xr-x 1 root root 1230992 Mar  9 16:13 vessel128
-rwxr-xr-x 1 root root 1361960 Mar  9 16:13 vessel256
-rwxr-xr-x 1 root root 1624104 Mar  9 16:13 vessel512
-rwxr-xr-x 1 root root 9488528 Mar  9 16:13 vessel8192

As per the comparison, vessel512 seems to be the appropriate choice. Let’s compare the executables with this vessel:

ᐅ cmp ~/local/bin/ai /opt/toit-sdk/vessels/vessel512
/home/flo/local/bin/ai /opt/toit-sdk/vessels/vessel512 differ: byte 1096898, line 6611

It appears that the executables differ at byte position 1096898, which should thus be the snapshot size. Consequently, we can extract everything following the snapshot size:

ᐅ dd if=$HOME/local/bin/ai bs=1 skip=1096900 of=/tmp/ai.snapshot
527204+0 records in
527204+0 records out
527204 bytes (527 kB, 515 KiB) copied, 1.03678 s, 508 kB/s

The /tmp/ai.snapshot file now contains the bytecodes, along with some extraneous content at the end. Fortunately, toitp ignores trailing garbage, allowing us to inspect the snapshot without further manipulations:

ᐅ /opt/toit-sdk/tools/toitp /tmp/ai.snapshot
snapshot: 527204 bytes (/tmp/ai.snapshot)
 - program: 93965 bytes
 - method_table: #1674, 62970 bytes
 - class_table: #141, 2461 bytes
 - primitives: #42, 1258 bytes
 - selector_names_table: #276, 1094 bytes

We can now disassemble the bytecodes:

ᐅ /opt/toit-sdk/tools/toitp -bc /tmp/ai.snapshot main
Bytecodes for methods[4]: (only printing elements matching: "main")1414: main cli-secret.toit:3:1
  0/1418 [016] - load local 2
  1/1419 [020] - load literal sk-FB0EA...<MY OPENAI-TOKEN>
  3/1421 [053] - invoke static main cli.toit:16:1
  6/1424 [089] - return null S1 1
1427: main cli.toit:16:1
  0/1431 [052] - load local, as class, pop 3 - List_(112 - 116)
  2/1433 [052] - load local, as class, pop 2 - StringSlice_(84 - 86)
  4/1435 [020] - load literal gpt-3.5-turbo
  6/1437 [053] - invoke static program-name <sdk>/system/system.toit:222:1
  9/1440 [020] - load literal ai4
 11/1442 [062] - invoke eq
 12/1443 [082] - branch if false T18
 15/1446 [041] - pop 1
 16/1447 [020] - load literal gpt-4-turbo-preview
 18/1449 [022] - load null
 19/1450 [019] - load local 5
 20/1451 [053] - invoke static CollectionBase.is-empty <sdk>/core/collections.toit:116:3
 ...

And these few lines of disassembly already reveal the bug in my code: the comparison invoke eq at offset 11/1442 compares the result of calling program-name with “ai4”. However, program-name returns the path of how the program was invoked (similar to argv[0]) and not just the name of the executable. The fix is as simple as calling basename on the result of program-name first. This also explains why I hadn’t noticed the bug earlier: I always called the program with ai4 and not with ~/local/bin/ai4.

It’s important to note that not every Toit executable has disassembly output that is that easy to understand. If we had compiled the executable with the --strip flag all debug information would have been removed. Given that the program is open-source and I only shared it with my siblings there was no need to use that flag, and keeping debug-information yields nicer stack traces in case of issues.

Conclusion

Toit executables are just small virtual machines with a snapshot bolted onto them. By knowing a bit of how snapshots are stored in vessels we can easily extract these snapshots again. We can then use all the standard tools, like toitp to inspect the extracted snapshots.