Parsing an Undocumented File Format

I'm reminded of the comment in XeePhotoshopLoader.m:

  // At this point, I'd like to take a moment to speak to you about the Adobe PSD format.
  // PSD is not a good format. PSD is not even a bad format. Calling it such would be an
  // insult to other bad formats, such as PCX or JPEG. No, PSD is an abysmal format. Having
  // worked on this code for several weeks now, my hate for PSD has grown to a raging fire
  // that burns with the fierce passion of a million suns.
  // If there are two different ways of doing something, PSD will do both, in different
  // places. It will then make up three more ways no sane human would think of, and do those
  // too. PSD makes inconsistency an art form. Why, for instance, did it suddenly decide
  // that *these* particular chunks should be aligned to four bytes, and that this alignement
  // should *not* be included in the size? Other chunks in other places are either unaligned,
  // or aligned with the alignment included in the size. Here, though, it is not included.
  // Either one of these three behaviours would be fine. A sane format would pick one. PSD,
  // of course, uses all three, and more.
  // Trying to get data out of a PSD file is like trying to find something in the attic of
  // your eccentric old uncle who died in a freak freshwater shark attack on his 58th
  // birthday. That last detail may not be important for the purposes of the simile, but
  // at this point I am spending a lot of time imagining amusing fates for the people
  // responsible for this Rube Goldberg of a file format.
  // Earlier, I tried to get a hold of the latest specs for the PSD file format. To do this,
  // I had to apply to them for permission to apply to them to have them consider sending
  // me this sacred tome. This would have involved faxing them a copy of some document or
  // other, probably signed in blood. I can only imagine that they make this process so
  // difficult because they are intensely ashamed of having created this abomination. I
  // was naturally not gullible enough to go through with this procedure, but if I had done
  // so, I would have printed out every single page of the spec, and set them all on fire.
  // Were it within my power, I would gather every single copy of those specs, and launch
  // them on a spaceship directly into the sun.
  //
  // PSD is not my favourite file format.

This "like standard protocol/format X, but strangely invalid" is a thing I've seen many times.

I speculate that one of the ways this happens is that someone decides or is told to use format Foo. Then they and possible collaborators implement both the writer and the reader for their idea of Foo from scratch, never testing with an off-the-shelf standard parser.

You'd think that doing XML like this is unlikely, given how easily available correct and validating parsers have been. But I've nevertheless seen this with XML too. I speculate that sometimes the programmer is on a platform that doesn't have an easily available off-the-shelf parser/writer, or they simply don't know about it.

I've also seen a variation of this, in half-butted "integrations", like to have a sales check-off feature of "we can generate X". These are sometimes tested only lightly, and sometimes not at all (such as when they don't have access to the tool that uses that format, and they were just working from poor documentation or an example). It's a thing.

You are not alone. I did a project to upgrade "sort of XML" to standard XML. Your example content gave me flashback shivers.

I have seen code that produces output like this first hand. Instead of doing proper serialization, they were using string templating to construct the response and never bothered to validate the output. Laziness and stupidity basically.

Some of the worst code of my life, created 20 years ago when I was a teenager, today posted openly on my GitHub, was reverse engineered custom chat server protocol, as I wrote my own client to replace the Java applet. And to have logs.

The catch is... I didn't have any Internet connection. I was going to an internet cafe, logging onto the chat server, and chatting, while recording the connection with Wireshark.

At home, I'd print the hex + ASCII connection dump on my dot matrix printer, and used a highlighter and ballpoint pen to mark the fields of the message packet.

Then I'd code something around it, planned new tests, compiled a new version of the app and.... took it with me on a hard drive to the internet cafe to test next day, or next weekend.

I think I was way smarter and goal oriented than I am today.

>is almost a rite of passage for programmers, it is incredibly fun and rewarding,

Only when you are doing it for yourself or when it's a known undertaking. It can be very frustrating when you are integrating with some hardware and you are 99% complete and you've told everyone you are ready to ship and the last 1% is a surprise reverse protocol engineering project.

That’s what I was gonna say. Reverse engineering was the only way in the 90’s as documentation was scarce. I had to reverse anything if I wanted to understand how it worked.

Here is an extractor I wrote for Westwood PAK and Lucasarts LFD files when I was 16: https://gist.github.com/ssg/e3e9654612be916336c01e104b10ddc7

Agreed, reverse engineering a job submission and validation protocol that ran over a Unix socket was some of the most fun I ever had on the clock.

I wound up basically re-implementing the software that was listening to create a test suite by the time it was all said and done.

> We were not supposed to write parsers, but instead manually write out a 3D model vertex by vertex and then texture map them.

Good grief. It's like writing assembly: a good exercise, but only for trivial or particularly tricky parts of a program. For everything else, proper tooling (a compiler or a 3D modeler) is the way to go. I fully agree that the best learning experience is to build tools to automate away the annoying parts :-D

Just in case anyone sees this and finds themselves curious, I actually just shipped the fix: https://ezkeys.kabaragoya.com/

Followed, hopefully, by publishing that info so it's not lost to the dustbin of time

Where I throw stones at myself is that I have a "perfect is the enemy of good" problem in that I don't publish writeups that are incomplete, because I keep hoping that if I just spend more holidays banging on it then I'll figure out why the parser works most of the time

Reverse-engineering a Postman backup file was fun as well. There are OpenAPI schemas, but I found many undocumented fields, and it was not at all obvious when default values for settings were set and when not. Probably a mix of glitches from schema evolution and fields that Postman uses internally.

There was a story about reverse engineering highspeed Broadcom networkcard firmware on HN last week. That included a custom compression if I remember correctly.

This is the post: https://news.ycombinator.com/item?id=38772862

They are relatively easier to detect though. Compressed file formats or virtual machine codes in disguise tend to be more annoying, especially since they can look like somehow structured and waste your time.

While I don't have any handy link, I did reverse-engineered several file formats without any further information and I can give some points.

First, make sure that you know what the format is actually supposed to encode. For example, if some file weighs (say) 40 KB then it is unlikely to be a raster image. The file name, if any, helps a lot to narrow the scope.

Second, you should have some understanding of similar file formats. I generally recommend to study PNG first because it gives an example of typical structured file formats and raster image formats. (Don't delve into the compression though---bitwise analysis is much harder.) This is also why you needed to know what the format is for, many formats with the same goal tend to have similar structures.

Third, collect as many examples as possible. You can line them up to see commonalities and differences and spot patterns. Even better if you can actively generate different files. This is generally the last hope when you are run out of reasonable hypotheses.

Fourth, optimize the feedback loop. You will have to do a lot of hypothesization, validation and automation. You can't really optimize the number of iterations, but you can optimize the time for a single iteration. Use a comfortable scripting language with good binary operation. I tend to use a vanilla Python with struct and make everything else by my own, but there are several libraries that greatly help you if you don't feel like doing so.

I had reversed engineered some ASCII file formats. While probably overkill, my background parsing simple programming languages (for which there are many good educational resources) was really helpful (in the approach I use). I tokenize, and try figuring out syntax structures from the order of token types, then from there, extract the information I need into my program's data representation. I'm not sure if this is the approach used by everyone else, but it seems plausible for someone with a CS/PL implementation background.

But first, it helps to have sample files to see recurring structures. Ideally, you also have access to software that generates these files. This allows you to deal with simpler files containing less information to reason about, make small changes within the program and compare the corresponding change(s) in the file.

oh god, that sounds like a painful thing to try to do. there's an Access redistributable at https://www.microsoft.com/en-us/download/details.aspx?id=549...

i've never attempted JS interop, so i don't know how much it sucks, but it definitely seems doable. it'll likely be an adventure on it's own, but it's gotta be 1000x less fuckery than trying to reverse the binary format of Access

the saner option would probably just be a small .Net program that creates an endpoint for your JS

unless i'm assuming too much, and you're just doing it for sheer masochistic pleasure, and in that case: i salute you