Reflections on My Tech Career – Part 1

14 min read Original article ↗

I’ve been lucky enough to have had a successful career as a software developer. Spanning six companies and thirty-seven years I’ve had the opportunity to work on Elastic Reality, Xbox, Windows, Steam, Internet Explorer, dozens of games, and Chrome, and create a blog whose investigative reporting and ETW tutorial articles have been read over five million times.

I was recently explaining to a friend how I got my start in this industry and I decided that I should write it down, if only to clarify in my mind how it all happened. I don’t necessarily recommend my career path, but there may be other people out there for whom it is the right way to go. Or, more likely, perhaps there will be a few ideas in here that could inspire somebody’s career path or help them avoid mistakes.

Here goes.

This is first part of the story of how my career as a software developer unfolded (part 2 is here). In this half I learn my craft, work at my first couple of tech jobs, and travel.

Learning to Program – 1983 to present

The most important initial detail is that I am a university dropout. During second-year university I stopped going to school. I lacked a support network of friends and was probably depressed and lonely and the more classes I skipped the harder it got to resume attending. I was also busying myself with disassembling the Apple ][ ROMs and other experimental programming projects. At the end of the year UBC politely told me that I was not permitted to continue attending. Oops. My parents were horrified.

Luckily through some convenient nepotism (thanks Lance!) I got an unskilled union desk job in a time when living in Vancouver was affordable. I made friends at work and got out of my lonely and depressed stage. The job paid well enough that I didn’t need to work a full 40 hours so I had lots of spare time.

I bought an Amiga computer and spent all my free time reading the Amiga, 68000, and 68881 reference manuals, C language tutorials, and whatever other sources of information there were in the pre-internet mid-80s. My dream was to become a video-game programmer. I spent hours on my Amiga and also continued programming on my 6502-based Apple ][ and at some point around 1986 I interviewed for a job at a local game-programming company called Distinctive Software.

They turned me down. I didn’t have enough experience or education.

At this time I discovered the Mandelbrot set. This is an infinitely detailed mathematical object that can be fascinating and beautiful to explore, but imagedoing so was very compute intensive, especially on circa-1986 hardware. It was particularly slow if you used software floating-point to do the calculations, which all of the explorer programs were doing. I immediately knew that by discarding the flexibility of a floating-point exponent and using fixed-point numbers (exactly three bits to the left of the binary point) and by coding in assembly language I could make things run at least an order of magnitude faster. Note that this project and my subsequent one were both done in collaboration with a friend but I’ve omitted any other mention of him to simplify the story – sorry Steve!

Estimating

At this point I leveraged a skill that would serve me well throughout my career. I wrote up what the basic assembly-language calculation would look like, precisely estimated how many clock cycles it would take for that loop to execute (easy in those in-order processing days), estimated how many loops would need to execute to render an initial image, and on the back of an envelope calculated that the first image should take about fifteen seconds. That was enough faster than the many minutes for other programs that I knew that I had a viable product before I typed in the first line of code. Here is the critical inner loop for calculating the Mandelbrot set to 16 bits of precision, with cycle timings as comments:

loop16
         LSL.L   #4,zi                   ;4
         SWAP    zi                      ;4
         ADD.W   ci,zi                   ;2
         SUB.L   isquared,zr             ;2
         LSL.L   #3,zr                   ;4
         SWAP    zr                      ;4
         ADD.W   cr,zr                   ;2
entrypoint16
         MOVE.W  zi,isquared             ;2
         MULS    isquared,isquared       ;28
         MULS    zr,zi                   ;28
         MULS    zr,zr                   ;28
         MOVE.L  isquared,temp           ;2
         ADD.L   zr,temp                 ;2
         CMP.L   four,temp               ;2
;;;;    DBHI    counter,loop16          ;6/10/6
; Technically should be DBHI, thus keeping 2.0 in the loop,
; but +2.0^2 doesn’t quite fit.
         DBCC    counter,loop16          ;6/10/6
;;;;    —————                 ———-
;;;;    13 instructions                 116 cycles, 84 from three multiplies

My original cycle timings would have been for the 68000 processor but all the source-code copies I can find have updated timings that appear to be for the 68020. The multiplies, in particular, would have been 38+2*n cycles (where ‘n’ depends on the bit patterns) and the total time would have been somewhere around 200 clock cycles. 68000 instruction timings can be found here.

When I got an initial version of my program running and the first image took over a minute to render I trusted my estimates and knew that there must be something wrong with my code. I quickly found that the WritePixel() calls were consuming the extra 45 seconds so I learned how to batch those up and I got the predicted performance. This served as a powerful lesson in the value of knowing how fast something could theoretically run. Only then can you know when no further speedups are possible.

Arthur C ClarkeAs an aside, Arthur C. Clarke briefly became obsessed with fractals so I have a letter from him asking for a copy of MandFXP. He then mentioned it in the sources and acknowledgments section of his odd Fractal/Titanic cross-over book, which led to a few extra sales.

Arthur C Clarke MandFXP Mention - croppedHere’s the mention from Clarke’s The Ghost From The Grand Banks. Don’t try contacting me at that address. I gave up that PO box 30 years ago.

Clarke’s letter included his phone number so I called him once on a dare at 2 am (mid-afternoon in Sri Lanka) and we briefly spoke, talking about fractals and 2001: A Space Odyssey.

I also wrote high-precision math routines to allow zooming arbitrarily deeply (more or less) into the Mandelbrot set.

Aside: the 68000 version of my high-precision code could accumulate a 16×16 multiply result every 80 clock cycles. In 2011 an x64 processor could accumulate a 64×64 multiply result every 3 clock cycles. That’s 16x as much work per block, running 26.67x as fast per clock, running at 446x as high a clock speed, for a throughput increase of over 190,000 times. Plus more speedup from multiple cores, so call it a million to one. Ain’t progress grand?

1987_06 Info mergedOnce I had a Minimum Viable Product called MandFXP (Mandelbrot FiXed Point) I took it to the local Amiga User’s Group meeting to demonstrate it. The reaction told me that I had a hit on my hands, and I did indeed turn the MVP into a shareware product that I sold many copies of. But that wasn’t the most important outcome.

A game programmer from Distinctive Software (hi Mike!) was at the meeting. He saw my demonstration and recognised that there was some raw talent behind this. I was now a much safer bet than when I had interviewed a year or so earlier and a job offer was quickly made. This was a good thing because at this point in my life I was unemployed.

CygnusEd

Around the same time I had written a text editor, CygnusEd, for Amiga programmers. This text editor had blazing fast text rendering, smooth scrolling, and some user-interface improvements that I felt were important. I’d been so confident in its success that I’d quit my job to focus fulltime on promoting it, and it turned out I was a terrible promoter. Although CygnusEd eventually developed a cult following and strong sales, in 1987 it was a commercial flop. It sold poorly so when this job offer showed up it was particularly fortuitous.

Cirque de SoleilBut first I had a choice to make. In addition to being a software developer I was an amateur juggler and unicyclist. I harboured dreams of turning pro and I had applied to go to the circus school in Montreal. Around the same time that I got the job offer I also got a circus-school acceptance letter.

I spoke no French, I had no savings, and (in hindsight) I was a strictly mediocre juggler while I was a fairly promising programmer. I chose the game-programming offer, and thank God. I would have been a failed circus performer, so programming was definitely the correct choice.

Distinctive Software – 1987 to 1990

undefinedAt Distinctive Software I worked on Test Drive, Test Drive II, and Grand Prix Cycles. The Amiga was powerful enough (compared to the PC, Apple ][ and Commodore 64 which also ran these games) that I could code mostly in C and still end up with games that were more colourful and ran faster than the other platforms. I even played around with the Amiga’s custom Copper chip to program the video display to do beautiful 60 fps transitions between menu screens while using almost no CPU time – the closest I ever came to programming a classic game console.

While working at Distinctive Software I continued to develop CygnusEd. I used it at work, as did many of my coworkers, and it slowly attracted more attention. Eventually it caught the attention of ASDG, a publisher of Amiga software. They worked with me to create a higher-quality manual and got CygnusEd in stores. They raised the price and I started getting royalty cheques. I wasn’t making a huge amount of money at Distinctive Software (about the same as my union job) and sometimes the monthly royalty cheques from CygnusEd were larger than my monthly salary.

Travel – 1990 to 1992

SC2_250I’d been intending for a long time to try my hand at travelling the world, inspired by my ex-girlfriend (hi Heather!). My goal was to backpack around the world for two years. In early 1990 I suddenly realised that I’d saved up enough money to make this dream a reality. The only problem was that I’d just met a woman who I kinda sorta liked, and I didn’t like the idea of choosing between this trip and her. And so it was that about four months after we had met that I asked Helen to quit her job, sell her car, give up her apartment, and put all of her stuff in storage. Totally normal stuff. Amazingly enough she agreed and by the end of the year we were in Sydney, Australia.

ASDG continued sending me royalty cheques while we were traveling. This was quite frustrating for them because it meant that they were funding my continued travels while what they really wanted was for me to come home and release a new version of CygnusEd.

We traveled for two years. I mention this because it’s important to remember that a successful career isn’t all there is to life. I’m sure that dropping out of the job market for two years cost us a fortune but it also gave us amazing memories which sustained us for decades. We saw natural and human wonders that fed our souls and after two years of traveling – as planned – we came home to resume normal life (including getting married and having two kids).

ASDG/Elastic Reality – 1993 to 1997

When we returned to Vancouver we could have gotten our old jobs back but that didn’t seem particularly adventurous so at this point I did the world’s worst job search. I contacted the only other software company that I knew – ASDG – and asked for a job. They said yes, they helped me with the immigration paperwork, and we moved to Madison, Wisconsin. I call this the world’s worst job search because it apparently never occurred to us to look for other companies and do a proper series of interviews. I should have talked to Microsoft, for instance, but I did not. Maybe I wasn’t ready, maybe things would have gone worse, but it was pretty sloppy to not even try.

At ASDG I learned Windows, C++, and version control – until then I had no experience with any of these things. Yes, in the late 80s it was possible to ship five different software products without using version control. It was a crazy time.

1995_12 Amazing Computing Amiga 14During this time I also created another fractal program, because why not? Actually two others. The first was called Mand2000 and it was for the Amiga. I wanted to integrate one of the key ideas from CygnusEd – smooth transitions between states – into fractal exploration. Roughly speaking I felt it was important that when you zoomed in on the image it should animate the zooming of the image and then recalculate. This was a radical idea at the time, and very difficult to make run fast enough on the bit-plane graphics architecture of the Amiga – the blitter chip couldn’t scale graphics. I ended up writing C/C++ code that on-the-fly generated machine code for each magnification level. Fun stuff. I actually roughed out the idea while traveling. Apparently I sent a nerdy friend (hi David!) a letter that included a description of the idea, some hand-written 68000 assembly language, and (of course) some timing estimates. Always be coding.

I also did progressive rendering with 8×8 blocks on the first pass to give a low-res overview of the fractal images far faster than the total rendering time. Again, a fairly unusual idea at the time.

1997_06 PC Format 14 - croppedThe other fractal program was called Fractal eXtreme and it was my first solo project for Windows. It had all the same bells and whistles of the previous versions, plus the ability to create “zoom movies” (key frames that a custom player could interpolate, with speed adjustable after rendering) to make it more efficient than ever before to create long fractal movies. My two most popular YouTube videos are both Fractal eXtreme zoom movies. I just watched the 4K version and it still blows my mind that the ridiculous complexity of this video comes from an algorithm that can be expressed in two lines of code.

Amazingly enough Fractal eXtreme (with updates for multi-core and 64-bit) still sells about a dozen copies a year, almost thirty years later.

While still at ASDG I also created a new version of CygnusEd. In total these four independently created programs (MandFXP, Mand2000, Fractal eXtreme, and CygnusEd) probably brought in about $200,000, spread out over decades. This was particularly vital money when I wasn’t making much, but the experience I gained by hacking away on these home projects, and the reputation, were even more valuable.

r/vintagecgi - Avid Elastic Reality (Warping and Morphing Software), Avid Media Illusion (Digital Nonlinear Compositing Software) and Avid Matador (Paint/Rotoscoping/Motion Tracking & Image Stabilization) [Late 90s - Early 2000s]While at ASDG the main product that I worked on was a morphing program called Elastic Reality. This product was so important to ASDG that they eventually renamed the company for it, and then sold the company to AVID. It was used in many movies and TV shows and won a technical Academy Award. The Windows version (not including the underlying morphing engine that was shared between platforms) was created by two of us and, though I didn’t know it at the time, it was basically the last time that I wrote significant amounts of new code to implement a consumer product. From then on my career moved towards developer tools and fixing of code rather than writing it.

For most of my time at ASDG/Elastic Reality I wasn’t making much more than I’d made at Distinctive Software, which had been about the same as at my unskilled union job.The fractal and CygnusEd income helped, but I definitely wasn’t making the Silicon-Valley stock-option money that I’d heard rumours about. This is because employers are happy to pay you as little as possible if you don’t force their hand with competitive interviewing. Well, that and the fact that I was still an inexperienced software developer.

This post is already quite long so I’m going to do the rest of my career as part 2.

Hacker news discussion is here.

Blue sky discussion is here.