This will hopefully be a short series on the old-timey tools around C6.0, C/C++ 7.0 and Visual C++ 1.0. I’m not sure how long it will take me to write the other parts…
For context, this is what was going on in the OS world at that time…
Table of Microsoft Operating System Releases
Aug 1981 MS-DOS 1.0 Single-directory FAT12 file system, 8088 real mode, command-line shell for IBM PC
Mar 1983 MS-DOS 2.0 Subdirectories, hard disk support, installable device drivers, UNIX-inspired I/O redirection & pipes
Aug 1984 MS-DOS 3.0 1.2MB floppy support, larger hard disks (FAT16), network support hooks
Mar 1985 MS-DOS 3.1 Network redirection support (MS-NET compatible)
Nov 1985 Windows 1.0 Tiled windows GUI, mouse support, cooperative multitasking, bundled apps (Paint, Write, Calculator)
1986 MS-DOS 3.2 3.5" 720KB floppy support
Apr 1987 MS-DOS 3.3 Logical partitions >32MB, national language support, 1.44MB 3.5" floppies
Dec 1987 OS/2 1.0 Protected mode (286), preemptive multitasking, up to 16MB RAM, named pipes/IPC — text mode only
Dec 1987 Windows 2.0 Overlapping windows, keyboard shortcuts, expanded/extended memory support
May 1988 Windows 2.1 Separate 286 and 386 editions; 386 version adds virtual 8086 mode multitasking
Jul 1988 MS-DOS 4.0 DOS Shell (graphical file manager), volumes >32MB, EMS support
Oct 1988 OS/2 1.1 Presentation Manager GUI, graphical windowing system, VIO/AVIO subsystems
Oct 1989 OS/2 1.2 HPFS (high-performance file system), installable file systems, improved Presentation Manager
May 1990 Windows 3.0 Protected mode (286/386), virtual memory, redesigned UI with Program Manager, TrueType-ready
1991 OS/2 1.3 Improved performance, smaller memory footprint — last version with Microsoft involvement
Jun 1991 MS-DOS 5.0 HMA/UMB memory management (load high), full-screen editor, UNDELETE/UNFORMAT, QBasic
Apr 1992 Windows 3.1 TrueType fonts, multimedia (MCI), OLE 1.0, improved stability (protected mode only)
Oct 1992 Windows for Workgroups 3.1 Built-in peer-to-peer networking, file/printer sharing, NetBEUI/IPX
Mar 1993 MS-DOS 6.0 DoubleSpace disk compression, MEMMAKER, multi-boot configurations, defrag, antivirus
Jul 1993 Win NT 3.1 32-bit preemptive multitasking, NTFS, hardware abstraction layer, Win32 API, multiprocessor support, security subsystem (C2-level)
Nov 1993 MS-DOS 6.2 DriveSpace replaces DoubleSpace (legal), SCANDISK, safer disk operations
Nov 1993 Windows for Workgroups 3.11 32-bit file access, TCP/IP stack option, faster networking
1994 MS-DOS 6.22 Last standalone DOS release, refined DriveSpace
Sep 1994 Win NT 3.5 Smaller footprint, TCP/IP as default protocol, long filename support, improved NTFS performance
Jun 1995 Win NT 3.51 PowerPC support, PCMCIA, Windows 95–style common controls, compressed NTFS
Aug 24, 1995 Win 95 Start menu/taskbar, 32-bit apps with Win32 API, Plug and Play, long filenames on FAT (VFAT), integrated TCP/IP, preemptive multitasking for 32-bit appsA few days ago, the Microsoft Voluntary Retirement Program had its first actual separation day, and so many people that I had known for a very long time — including some comparatively old-timers — decided this was a good time to retire. I’m very happy for them. I think of all the ways to part with the company, this method is pretty gentle. And you get some nice perks on the way out, which is very nice.
One of the fellows told me that I was now maybe the last person still at the company who had any memory of the early product days that we worked on together, back when Languages was much younger. Well, Languages has always been much older. The first products were languages, so even when I joined in 1988, Languages was already comparatively old at 13 years. Many businesses would have to go a lot longer before they reach the 13-year milestone. But now 13 years is less remarkable. The company’s been around 50 years. Lots of businesses seem comparatively older. And the difference between Languages and, say, the Applications business is much less than it was in the middle ’80s. OK, fair enough.
I wanted to talk a little bit about what was going on then and what the languages products looked like. I can speak to the native languages. There are other languages products that I had nothing to do with, but I’ll try to talk to the C compiler and its friends. Many of the other systems that produced native binaries used the same back end, and so a lot of the tooling situation for them was basically identical. If you look at the native output of, say, BASIC or Fortran or some of the other languages that were available, you’ll find that it’s the same from an ecosystem perspective.
You can’t really talk about languages and tooling without talking about the operating systems. And to give you a sense of what operating systems were happening when, I made a table above and it illustrates the top operating systems that were going on from 1980 to 1995, which is a good 15-year interval. A lot of it is covered by DOS, and DOS 3.x basically covers the early history of Languages from a customer perspective. By the time I arrived, we were using OS/2 for development. Actually, that’s not quite true. I arrived in 1988 and for the first several months I was doing development under DOS. I had a 386/16 — that is, a 386 processor running at 16 megahertz. And if I recall correctly, I had two megabytes of memory, one of which was normal memory and one was sitting on a Cheetah card. The Cheetah card sat on the ISA bus, which was not so bad at the time because the ISA bus still ran at 8 megahertz and the processor was only running at 16 megahertz. So putting memory on that bus wasn’t completely stupid. But it wasn’t as good, and only the first megabyte was well-addressable by the processor. That meant anything above that, on say a Cheetah card, was accessed via one of the various standards like the Lotus-Intel-Microsoft standard for accessing upper bank memory, and you’d have to swap it in and see it in 64K chunks — I think four of them. Anyway, the systems were not that big. While I enjoyed a processor with a full one megabyte of memory of which 640K was available to DOS, many of our customers didn’t have anything like that. It was quite normal to think that some of our customers might have as little as 128K. 256K, 512K — they were all quite common. I think 128K was not so common among people using tools, but certainly the bulk of them were using DOS. And that remained the case for a long time.
I wrote elsewhere that I was hired to work on C# — and I don’t mean the regular C# that you think of now that comes with .NET. I mean the one that was being built in 1988 as incremental compilation technology. I’ve written about that elsewhere, so I’m not going to cover it here. I worked on C# ’88 for about two months before it was basically canceled. It might have been three months.
As soon as that happened, I was put on the C6 project. We used C 5.1 to build C6, so I was using the C 5.1 compiler all the time, and I was largely using the C5 tool chain to do the building and debugging. Later on, the system got stable enough that we started using the C6 tools to develop C6 — a sort of normal thing in these kinds of projects — and that made things a little bit better. Now by the time I was working on C6, we had cut over to developing on OS/2. Even when we were targeting DOS, most of us ran OS/2 and we used the so-called 3X box. Some people called it the Chernobyl box because it crashed a lot. In short, we’d do as much DOS development as possible on OS/2. The good news is, by then we had computers with more memory, which helped a ton because we had space for a disk cache and whatnot. That made a big difference in development. But runtime was still very tight; it was important that the system stay small. We were targeting systems that maybe had 256K or 512K, and we needed to run pretty well on those systems.
So I was working on tooling, and in particular I was working on PWB — the Programmer’s Workbench. Specifically, I was working on the browser in the Programmer’s Workbench, although PWB was in a lot of trouble at the time and it was important that everyone sort of pitch in everywhere in the product. There were bugs all over the place and we just all helped wherever we needed help. But nominally, I was working on the source browser, and the source browser had a three major components.
One thing is that the compiler had a flag where it would emit, into a .SBR file, something that you could reasonably see as being comparable to the preprocessed token stream. So you could see what macros expanded to, you could see what symbols there were, you could see the file names, the line numbers — all that stuff. It’s kind of like the preprocessed input, except tokenized. The file didn’t duplicate things like symbol names, but otherwise it was a set of records that told you which symbols appeared where and in what context. So when you saw the start of a function, a function block would open, then you’d see variable declarations, then variable usages. You’d see the line numbers marching up and down, you’d see the function close, and then you’d see the next one open, and so forth like that. The SBR file format is documented somewhere else, I’m sure. It’s very unremarkable — it’s a variable-length record thing that corresponds pretty directly to what’s in the preprocessed stream token by token. Notably, you don’t see the #ifdef parts that weren't processed. And if there are errors, you can't get an SBR file — you have to have a compilable file to get an SBR. Which meant we could only browse programs that had compiled. If you were browsing a program that hadn't compiled, you were using stale info, which is not that different than the situation with a tags files and such that similarly get stale.
Once you have these files, you have to process them into something that is suitably sorted and indexed. In particular, there was one SBR file for every compilation unit. They were peers to the .OBJ file, if you like. And you didn’t want to search through hundreds of SBR files to find what you were looking for, and the SBR files tended to be highly redundant. This is because the bulk of an SBR file actually recapitulated the header files again and again and again. This is a common problem with tooling in the C universe. The #include directive is just no fun for anyone, and it causes all kinds of fits for the tool chain because you may or may not get exactly the same thing each time you include your header files. Other languages don't have this problem. Modern languages import library context rather than recompiling it; and what you import is exactly the same every time, so it can be cached. You can't do that with C header files because of preprocessing and other things — the files could be totally different when you read them from different translation units. In fact, you can include the file more than once and it could have different consequences on the second, third, fourth, and etc. inclusions, although we did try to do some things to make that better over the course of the years.
To do this processing there was a tool called — are you ready for this — PWBRMAKE. Which stands for Programmer’s Workbench Browser Make. It makes .BSC files, which stands for Browser Symbol Cache. In C7 it wa s renamed to BSCMAKE (by me).
Now the reason I was working on any of this is because I had worked on the browser system for the C# system that I mentioned earlier, and it was actually pretty cool. Someone saw it — actually, several someones saw it — and they decided that we should take some of those ideas and use them in the actual shipping product. And then when that was starting to fall behind, it was natural for me to kind of help with the shipping version of this browser technology, although the details were totally different. In the C# system, everything lived in what we called the Global Symbol Cache, or the GSC file. So BSC is basically the browser version of the GSC. It’s not coincidence that they have similar names, because the fellow that made that name had just talked to me.
OK, this program is pretty complicated because it has to read all kinds of SBR files, collate them all into memory, deduplicate all that stuff, and then write it out into a single file with multiple streams for things like: here are the definitions in order of occurrence, here are the files, here are their contents, here are the functions, here’s what’s in them, here they are alphabetically, stuff like that. It’s not too different from a relational database with many implicit keys based on the record index of fixed size records. You can join between the tables, and one of the tables is the files, and one of the tables is the functions, and then there’s a table between the functions and the usages of symbols, and so forth. Typical queries basically joined between a few of these tables. Now I didn’t think of them as tables or relational things at the time, but that’s pretty much what they are. The only way to change the database is to rebuild it pretty much from scratch. There was no need for transactions or stuff like that. You get a new file or you don’t. Even when I did incremental building of databases, it was done by reading the old database (much faster than reading the SBRs) except the parts that were changing then adding the changed bits as usual.
OK, so this was hard because some programs were big and the amount of data that you needed to hold in memory in order to do these operations was significantly greater than 640K. On OS/2, where you had buckets of memory — where “buckets” was like 2 megabytes (!) or 4 megabytes (crazy!!) — OK, but our customers weren’t going to have 4 megabytes, and our customers were going to have to run this tool under DOS and there’s no virtual memory. If you want to virtualize the memory, you have to do it yourself. So that’s basically exactly what I did. The tool would page out chunks of memory to disk, and it had a little system where it could keep 16K chunks of simple data for any one of the data types in memory. It would “fault” in what it needed when it needed it. And all of this was done with a little macro system that checked to see if the page you needed was resident, and if it wasn’t, it would swap out something and away you go.
By itself, that would have been horrible. Actually, it was horrible. While it only took me a few days to implement it because, fortunately, PWBRMAKE was small enough that it wasn’t so crazy to find all the places where the browser information was being read and virtualize them all manually. But the performance was terrible. To make this better, a ton of work went into density, but actually a lot more work went into locality.
The thing that saved it was the same thing that was causing it such consternation. We always recap the headers. And the headers are the same every time, or nearly the same every time. That’s the bane of our existence because we have to read that stuff and look for anything that might be new. But it was also going to be our savior. It turned that all the work is basically deduplicating these symbols — i.e. Do I already have this reference? Is this a new call? Have I already seen this function? Is this a new reference in this function? — the tool was always in this mode of collating things. And in the bulk of the cases it turns out the header file always replays in the same order. The system was very good at recording things in the order that they had been seen; and we could do all the sorting we needed after we were finished collating.
The epiphany was that if I just remembered where the last search for duplicates left off, and started the next search looking forward from the previous position, everything would go super fast. Because almost always, the next symbol you found would be the next symbol in the chain, because the headers really were the same every time. So no matter how many times you pulled in windows.h or whatever, it would look the same every time, or nearly the same — at least the same enough that from a performance perspective, you got huge wins. Typical dedupe time went from O(N) to O(1) and on that day, the thing got maybe 20 times faster. And that's what made it even possible to do the job on DOS.
But how do you even run this thing inside of the IDE and get any memory? PWB was not small. As editors go, it was pretty big — a few hundred K, which was a lot at the time, especially on a machine that might only about that much total memory. This is a character-mode editor, but it used that peculiar beast called Character Oriented Windows, or just COW. In particular, it used a variation that was swapped, the name of that library was COWS, that name stuck even though it was supposed to be Swapped COW.
Now, COWS is a great little beast because it’s a lot more flexible than typical linker overlays, and indeed it shares all the same techniques that real-mode Windows used to do its own code virtualization. It doesn’t come with any data virtualization — if you want to do that, you have to do it yourself. But at least it uses the usual sort of thunking mechanisms to ensure that the code you need is swapped in before you use it. You always call through a thunk that does the paging in if you need it. This is not very different from linker overlays; it’s just that it could have several overlays. And as a consequence, you didn’t have to try quite so hard to swap-tune it. Between the overlay magic of COWS and the editors data virtualization for editor buffers it all fit pretty good.
A quick detour to swap tuning is probalby in order here: Swap tuning is itself a black art where you try to make sure that the right stuff is in the right segments so that you’re not constantly thrashing. If you put code for unrelated stuff in the same segment, you’re constantly swapping in the middle of typical operations. So you wanted to basically swap in a bunch of stuff, do whatever you needed to do, then swap that out and go back to what you were doing. For instance, say you’re building Find in Files. Well, you want to make sure that all the code you need to do Find in Files lives in a couple of segments that can stay resident. Then when the Find in Files is done, they can be swapped out and your regular editing functions can be swapped back in, and that just makes life much better. If you were constantly swapping between many different segments, then Find in Files would be horribly slow because of the disk load while trying to search. And of course this is true of every feature, not just Find in Files.
Notably, Office used to spend a whole lot of time on this — and I say Office, but I mean Word and Excel and PowerPoint and so forth, because there was no such thing as Office. Actually, we hadn’t even bought PowerPoint yet at this time, so there wasn’t even a PowerPoint. And Access was under development.
OK, so while you were building the database, you could reasonably expect to use pretty much all of the memory that the computer had available. And that was fine — just like you could expect to use all the memory while you were compiling or linking. But while you were editing and browsing, you couldn’t, right? Because the user needs memory to actually store their files. The browser had to be small. So the indexes had to be good. In fact, they were very good. We were able to browse all of Excel with just 14K of browser code and less than 20K of data. The Excel database was north of a megabyte, which was a lot at the time. So importantly, you didn’t really have to have very much of that megabyte resident at any given time in order to answer the kinds of questions a developer was likely to ask. Like: Where’s this symbol? Where’s this reference to? What functions use this? Show me all the locals in this function. Show me the types I use. Show me the types in this file. Stuff like that. All of these things are comparatively local queries you can get from the clustered versions of the browser information stored in the BSC file. As a consequence, the browser was fairly popular. It was a good little productivity tool and it could give you lots of code intelligence. This kind of code intelligence was quite rare in 1990.
OK, but what about the rest of the system? Well, I did mention that it was swapped, but even that wasn’t enough to facilitate builds. There was no way that we could keep the editor loaded while we were building. It would take way too much memory. So we did some pretty tricky sleight of hand. Now on OS/2 you could just do normal process launches the normal sort of way and it just worked. The operating system would swap out stuff and that would be that. But on DOS you could not count on any of this.
So what PWB did was: when it started a build, it would exit. It would leave a file that included in it the set of commands that you were supposed to run, including how to restart PWB. And there was a little stub to run all this — the same stub that had launched PWB itself — that stub would find the build file and then run those commands. One of those might be actual builder, like NMAKE. We would cleverly set up the output system so that the character output of DOS would be “windowed”, if you will, to write only within a portion of the screen. Importantly, we wouldn’t clear the screen — we would just exit the program. It would look like PWB was still running, but it wasn’t. And then we would cause the screen output to go into that section of the screen. Meanwhile, the little stub was also capturing the output so we could do error analysis and stuff like that after the fact. But importantly, it was writing to the correct parts of the screen so that it looked like the output was going into the output window, which you could not scroll or anything like that. You could just watch it go by. But it was still a decent illusion.
And the thing is, doing that for just the editor wouldn’t have been enough. NMAKE had to do the same thing. When you were running like this, the make tool would write into that same file that had all the commands it needed to run in order to successfully complete the build. It would think about the dependencies, look at the recipes, figure out all the things that had to happen, write them all to a file, and then exit. And then that same stub would run those things. So the only thing that stayed resident was the little stub. That meant that the compiler got almost all of the memory, the linker got almost all of the memory, CVPACK got almost all of the memory. Basically whatever you were doing from inside of NMAKE would get almost all the memory because by the time your commands were running, NMAKE had exited and the editor had exited.
When all of this was done and the build succeeded or failed, we would relaunch the editor with some sneaky little flags. That would cause it to slurp up the build output, put it into a real window (and by window I mean a character-oriented window), do the display, and continue editing like nothing had happened. But actually we did a full relaunch. We just didn’t display the splash screen and all the other jazz like that. We’d basically swap the editor back in.
This actually worked remarkably well considering the number of steps involved. As a consequence, we could do meaningful builds and mostly stay out of the way. I think the only tool that didn’t take this kind of approach was the debugger — CodeView. At the time we still had a separate debugger. Integrated debuggers were way off. And CodeView basically tried to stay out of the way of the program by hiding in the upper chunks of memory as much as possible. So it had some amount in the regular area — maybe a few tens of K — and then it would try to keep as much of itself as it could in the high-memory chunks. Assuming any of that was available, but often it was. This was important because of course CodeView couldn’t exit — it had to be there to allow you to debug.
Now at the time we were doing C6, CodeView was being reimagined to use Character Oriented Windows itself. Previously it had its own sort of ad hoc UI. But we were switching to COW across the board, and we were getting a pretty nice-looking debugger. I got to use that as the months wore on. But sometimes I was using none of this. There was a character-mode symbolic debugger — something more like GDB, or LLDB or something like that.
Debugging in this situation was kind of a nightmare. Because in DOS, there’s no memory protection, and so problems from one part of the program can easily leak into another part of the program, or even leak into future programs. And the nastiest bug I ever had to fix was of this type.
The symptom was that after a build, PWB would crash while it was being reloaded. Now, this is already sort of weird because there’s basically nothing there at the time that you reload, right? We’re starting from scratch. All we have to do is fire up the thing, start clean, load up the one file that has the build output, queue up the other windows to virtually restore whatever files you thought you had open before, and away you go. But the program would crash long before it got through any of that. This is very puzzling, because of course that path is not really any different than the normal startup path, except in really tiny ways — most of which hadn’t even happened yet because it would crash comparatively early. And this of course only happened under DOS. So this is a bit of a nightmare.
It took me many, many days — probably a whole week — to get something that could reproduce this problem consistently, because while everyone had seen it happen, but it was quite challenging to make it happen on demand. And I wanted not only to just make it happen, I wanted to make it happen while I was using PWB on OS/2 inside of the DOS emulator. The reason I wanted to do that was because if I was running on OS/2, I could use the OS/2 kernel debugger, and it had access to the i386 debug registers, and I could use those to figure out who was writing to where and what was getting corrupted. Because I was pretty sure this was a memory corruption issue.
After several days of sweating this, I was able to find an exact repro that would crash every time. In order to be sure that it was going to crash every time, I started my repro from a power-on reset. So you can imagine this took many, many iterations, and every iteration required a reset. I had to get exactly the right program of exactly the right details to reproduce this problem.
Once I had it, we were actually in a pretty good place because we had two paths forward. One was we could try to use the i386 debug registers and the kernel debugger like I planned. And the other was we had an in-circuit emulator in one of the labs down the hall — it was cool because it gave you history. You could look at what had happened leading up to a crash, basically looking backwards in time. Often that was just what you needed to figure out who had corrupted things. But I didn’t know how to do that. So we broke up into two teams with my repro: me and a friend in my office going at it with the kernel debugger, and two people going after it with the in-circuit emulator.
It was a beast. I mean, once you get to a repro where you literally have complete reproducibility — like byte for byte address reproducibility, where the exact same memory gets written in the same way, the same addresses every time — a bug is really on borrowed time. But it could still be a big loan.
It was not hard to determine that the thing was crashing because it was trying to indirect through a bad pointer, and that was causing all kinds of problems. Remember, it’s DOS, so it’s not like you’re going to get an access violation. But you can get to the point where things are so seriously bad that you’re going to run bad instructions — you jump through something horrible.
OK, but then you’re left with: how did that get bad? So you set some hardware breakpoints. Well, no — first you reboot, attach the kernel debugger (which is one of the steps), then set memory-write breakpoints for the address that you think is corrupted. Zoom forward to the point where you think the corruption is going to happen soon, turn those on, and let ‘er rip. And that tells you where the bad stuff was written. Great — that moves you one step closer to the bug.
Soon we found out that it was a bad allocation, and as a consequence, memory was getting stomped. The bad allocation happened because the DOS Arena headers had been corrupted, and they are in in a linked list. So we found the first memory corruption. We found the allocation that caused it. Then we went back to the Arena header that was problematic, and it had gotten hammered. But it had gotten hammered because of a previous allocation, and that had gotten hammered because of a previous allocation. So each one of these — winding the clock back to the previous pointer that had been corrupted, that caused the next pointer to be corrupted — was a full reboot cycle. Run the thing, get the pointers, verify that your repro is working exactly the way you expect and you didn’t make any mistakes. And sometimes you did, of course, and then you’re like, well, throw that one away, repeat with the next address to check.
Something like 20 or 30 reboots later, we had rewound all the DOS Arena headers and we were back inside of NMAKE — near the start of the build, not long after PWB had exited. And there it was. There was a string copy that was missing the allocation for the terminating null, and so the memory that was allocated for the string was one byte too small. Most times this didn’t matter, because the way the heap lined up, that extra byte was there anyway. Or alternatively, the extra byte clobbered something that didn’t really matter. And NMAKE was going to exit soon anyway. So it never seemed to cause NMAKE to crash, but it did sometimes corrupt the DOS Arena header. And when it did that, it caused a chain of events during the next parts of the build, which would ultimately result in PWB itself getting a bad allocation at the start of its startup, which then turned into a bad write, which then turned into a jump through a pointer that was bogus, which turned into a crash.
Ironically, one of the guys trying to debug with the in-circuit emulator owned NMAKE. And they too were closing in on this conclusion. Within 30 minutes of each other, we had both confirmed the same root cause, and you could find that root cause in the source. One + 1 later, the mysterious bug that had plagued us for something like a month — and had been actively worked on by at least two engineers for at least a week — was gone. Of course, these days Address Sanitizer would catch this kind of bug in five seconds flat. And it certainly would never have been allowed to spread from one process to another.
Now, this isn’t actually the weirdest bug I ever worked on. I can fast-forward to a time when I was working on Visual C++ 1.0. And remember, there are lots of operating systems in flight here and lots of tool combinations. This is a great example of how all these things really make your life complicated when you’re a developer working on tools.
I was working on the Visual C++ 1.0 debugger, and a bug came in that said when you try to put a breakpoint on a certain symbol, the breakpoint goes in the wrong place. And sure enough, it did in fact go in the wrong place. But it didn’t look like there was a problem in Visual C++ — I soon realized that the debug information was wrong and Visual C++ was putting the breakpoint exactly where the info said it belonged. And the debug information came from the linker.
OK, so now let’s remember that this is Visual C++ 1.0. It’s debugging 16-bit binaries. And so this is a linker that produced 16-bit binaries. But the linker that produced 16-bit binaries was itself a 32-bit process. To pull off that miracle, we were using a tricky DOS extender — you may remember it, it was called TNT. It was pretty cool in that it let you build and run things that were just like Windows NT character-mode executables under DOS. It was quite useful because the same binary could run either under Windows NT or under DOS, and therefore under Win16. OK, so it’s a 32-bit linker, producing 16 bit excutables (hence the name link3216.exe) . And it’s producing a bad debug file.
But I knew the linker — I had worked on it in the past — so I looked at its input and sure enough, the input was correct. The linker’s output was incorrect. Great, so that looks like a linker bug for sure. Turns out it’s not a linker bug. When I looked at the code for the linker, I saw that it was correct. And when I looked at the codegen — because that’s what you did at the time, you single-stepped the codegen — the computation of the symbol address, had incorrect codegen. There was a problem in the 32-bit compiler! It was producing incorrect code, and that was causing the linker to produce an incorrect PDB file, which was causing Visual C++ 1.0 to put the breakpoint in the wrong place.
No problem, I know the compiler guys. I can go check with the compiler guys, it’ll be swell. So we went off to see the compiler guys. At this point I’ve pulled in the linker guy and we’re off to see the compiler guy. And sure enough, he looked at it and said yeah, that code looks wrong. But he looked at the .OBJ files that were going into the 32-bit linker and they were correct. So the compiler was off the hook. The problem was in the 32-bit linker! That is, the linker that produces 32 bit binaries (which I think was just link.exe even thought maybe it should have been link3232.exe)
OK, so now we definitely wanted to know what was going on. The crew plus one more person marched into the office of the 32-bit linker guy and went through the same exercise. And we once again discovered that it was not in fact a linker bug — there was a codegen problem. The linker’s source code was nominally correct, but the code generated was incorrect. We’re back to the 32 bit compiler!
So fortunately the compiler guy was now with us — and we’re back in his office. And he’s looking again. And once again he looks at the code and it looks right. So he single-steps through the code. And son of a gun if he doesn’t get the wrong result. Well, it turns out the problem was that one of the library calls — a memory copy — had a bug. As he single-stepped through his algorithm, he discovered that not all of the memory that he needed copied was getting copied. There was a runtime problem.
So all of us marched over to the guy who owned the memory copy function in question. And by this point, it’s quite a parade because we’re attracting an audience. And the beauty of this is that this whole thing took less than 15 minutes. We were literally going from one person to the next to the next. This was a case where the local expert knew exactly where to look and exactly what breadcrumbs they needed, and all the information was present. So unlike the previous bug, this one was like, “Well, give me the thing and I can see the code” or “It should be right here” and “Here’s what’s going on.” In a few minutes you could know exactly what was happening.
So we marched over to the guy who owned the memory copy function on the runtime. And this time we knew that’s where the bug was because we could see the bug in the source. We said, “Hi, we want to report a bug.” And he says, “Well, what are the symptoms?” And I said, “Oh, Visual C++ sets a breakpoint in the wrong place.” And he said, “What does that have to do with me?” And so we told him the long story, and sure enough — looking at that memory function in two seconds, he made the necessary fix. About five days later, all of those binaries had been rebuilt enough times that the linker used on the day to link binaries for Visual C++ had the correction. And the bug was fixed. When I resolved the bug, I noted that it had been fixed in the 32-bit runtimes and that the tester should wait five to seven days in order to actually see the fix in the 16-bit binaries.
The C6 system powered our development for a long time. Almost all of the development of the C compiler happened with that system in mind, and later I worked on an IDE that was also built in C using a variation of cfront. We aimed the cfront output — which was C, of course — into the C6 compiler, and that was our ecosystem for the next two years.
For our customers, it was the best optimizing compiler we had ever shipped, although by today’s standards I don’t think it would be considered very good. But considering how much memory it was allowed to use, how fast the processors were, and how much think time you could reasonably expect to have, I think at the time it was pretty darn good. Certainly it was state-of-the-art. Although if I recall correctly, both Borland and Watcom had really good offerings that were very competitive. Still, the combination of a high-quality editor in an integrated editing environment with nice mouse support, a windowed environment, pull-down menus, good-quality debugger, good-quality browsing — it was very compelling. It was the first major product that I ever shipped.