You would think that groundbreaking technology would be easy to see, at least if you were somewhat close to the ground being broken. But that hasn’t been my experience. Twice in the last decade I have completely missed a major breakthrough, despite being seemingly “close enough” to see it coming. There were also a few breakthroughs that I did see coming, too, and the differentiating factors are illuminating.
I was organizing a machine learning seminar in 2018 and completely missed transformers
I started studying machine learning the summer after my first year of graduate school, when a friend mentioned that a research problem I was working on (recognizing bouts of running using data from a wearable accelerometer) was a great fit for machine learning algorithms.
This was circa 2017, so my on-ramp was pretty typical for the data-science era:
- Andrew Ng’s MATLAB-based machine learning course
- Trevor Hastie’s Elements of Statistical Learning and its R-based companion, Introduction to Statistical Learning
- Andrew Ng’s Deep Learning Specialization
- Karpathy’s CS231n
- Plus some more specialized statistical coursework in semiparametric regression and functional data analysis, which made it into my dissertation
After a few years of intensive study I was basically up-to-speed with machine learning circa 2017. Having made some friends with PhD students in the math department, I got involved with a machine learning seminar they were organizing, and taught a few of those seminars on basis functions and the mathematics of (vanilla) neural networks. That group morphed into a “machine learning for research” club that worked on various image-based, graph-based, and tabular data problems.
I was involved with that group through 2022. I was even on Andrew Ng’s “The Batch” newsletter, which had regular updates on progress in computer vision, robotics, and NLP. So, how the heck did I miss transformers?
I had heard of transformers, and seen a few Kaggle notebooks using SWIN transformers for images, but since none of my work (and none of my hobby projects) involved natural language, I more or less ignored the parade of Sesame Street-themed models coming out in 2018–2022, for the same reason I mostly ignored all the incremental tweaks on ConvNets and YOLO that bumped up the SOTA by a few percentage points.
It wasn’t until ChatGPT in November of 2022 that I realized that there was something “there” with the transformer, and I spent several years playing catch-up!
Since my PhD research straddled the intersections of biomechanics, physiology, and physical activity epidemiology, I wound up contributing articles every week from 2018–2023 to Obesity and Energetics Offerings, an email newsletter with research highlights divided by area (children and adolescents, pharmaceuticals, surgery, etc.). The new papers I highlighted went into the “Exercise and Physical Activity” section, three sections below the “Drugs” section. And yet I found out about GLP-1s the same way almost everyone else did: newspaper headlines.
So, as a contributor to a research highlights paper managed by one of the top obesity researchers in the world…how did I not anticipate the rise of the biggest blockbuster drugs of the last 50 years? I’m sure the STEP Trial made it into OEO in 2021, but I never registered what it was.
Seeing breakthroughs requires extreme proximity, or extremely good taste
The reasons I missed these two breakthroughs illustrate something important about what it takes to see something new on the horizon.
I missed transformers because I wasn’t quite close enough to the field—temporally, in this case. The key transformers paper came out in 2017, but Karpathy’s CS231n lectures were recorded in 2016. There were even a few slides at the very end about “this new attention idea, which is kind of cool for image captioning.” Ditto for Andrew Ng’s deep learning course: 2017. François Chollet’s deep learning book? 2017.
If I’d been getting into deep learning a few years later, I would’ve been flooded with info on transformers, but given the timeframe, I made it right up through CNNs and RNNs, but no further.
In the case of GLP-1s, I missed it because obesity pharmaceuticals were far enough outside my own discipline that I didn’t have a well-attuned sense of taste. I couldn’t tell the difference between some generic clinical trial and one that would kickstart a multi-billion-dollar drug (and the OEO newsletter is pretty devoid of commentary or opinion: it’s more or less just a list of new papers).
Taste matters because every research field—even small ones like biomechanics—is flooded with new ideas. Every biomechanics conference I went to had a dizzying maze of posters with preliminary data, new takes on old ideas, and wild, strange gambles. And if we’re being honest, most of these new ideas are going nowhere.
Because of the gamesmanship of academia, everyone (including me) wants to pretend like their research line is going to be the next big thing. For a while, the American Society of Biomechanics advertised the field as “the breakthrough field of the 21st century” which seems wildly ambitious, even setting AI aside (CRISPR gene editing anyone?).
I saw the same phenomenon getting into machine learning. I read hundreds of papers on physical activity recognition (which used to be an entire sub-field of the computer science literature), reading through laborious comparisons of C4.5 decision trees, Hidden Markov Models, and various SVM kernels on classic benchmark problems. It took months until I could see the big picture: it was less about the precise features you used and the exact ML algorithm, and more about the amount, quality, and diversity of data you could collect. But it took getting close to the frontier to appreciate this phenomenon.
Proximity has more to do with doing than reading
To sift through the firehose of new papers and new ideas in any field, you need an extremely well-attuned sense of what really matters in a given field—or you need to trust someone else who has that highly attuned sense of taste.
And how do you develop taste? Plowing through hundreds of papers is surprisingly unhelpful. By the time I started graduate school I’d already read hundreds of papers on exercise physiology and biomechanics, but didn’t yet have a strong sense of taste—what made a paper good, bad, or forgettable.
I developed taste in biomechanics much more quickly by doing biomechanics research: seeing what worked and what didn’t, visualizing the actual data coming off the equipment, and so on. This proximity to the field (and the accompanying sense of taste) allowed me to spot some things quite early that were big breakthroughs, at least within the field of gait mechanics.
I did recognize the value of direct collocation
When you walk down the street, what muscle activations produce the motion of your body? Answering that kind of question is the core motivation behind musculoskeletal modeling. It’s a notoriously hard question to answer because the problem is underdetermined: there are many possible sets of muscle activations that could produce a given gait pattern.
Circa 2017 there were two ways to approach the problem: static optimization, which was fast but kind of a hack (it ignored many of the physiological limits on muscles, and was severely limited in its optimization objectives) and forward dynamics, which was unbelievably slow (as in, months of CPU time on a supercomputer to simulate a single step of walking).
The modern solution is direct collocation, a trajectory optimization approach first developed for aerospace problems (e.g. planning minimum-fuel paths for planes and rockets). Direct collocation is fast, flexible, and physiologically realistic. But unless you were very close to the frontier in computational biomechanics, it seemed like it came out of nowhere.
But this was a breakthrough I didn’t miss. I was reading the early forum posts that debated implicit versus explicit dynamics, and I was at the small side-room sessions on computational methods at the American Society of Biomechanics meetings. I had implemented static optimization (the hack solution) by hand myself, and I’d read the early forward dynamics papers and understood why the computational requirements were so prohibitive. And crucially, I understood enough about optimization to know why automatic differentiation was so valuable.
So, when I saw the OpenSim Moco project in 2019, I was already playing around with the example code. I’d seen the same approach being workshopped at conferences and over coffee in the preceding years, and I knew it was going to be a big deal.
Proximity is contingent and taste is subjective
If you miss a big technological breakthrough you shouldn’t be too hard on yourself. Much of whether you “see it coming” has to do with contingent factors that are outside of your control: the exact time and place you got into a field, the timing of the resources you happen to stumble across, and who you happen to be plugged into in the field.
There’s also an element of luck when it comes to what breakthroughs really break through—some genuinely good ideas never really go anywhere, or linger on the margins for decades before being rediscovered. Neural nets looked like an outdated flash in the pan during the rise of support vector machines in the late ‘90s, and in biomechanics, reinforcement learning and evolutionary methods still feel that way. But maybe they’ll turn out to be a great idea!
Calibrating taste requires a lot of trial-and-error: you have to go down some wrong-headed rabbit holes before you learn why they were a bad idea in the first place. You just have to be careful not to develop priors that are too rigid: sometimes “beginner’s mind” actually beats out calibrated taste.
People who are extremely close to a field and who do have lots of hands-on experience still make wrong calls in both directions—betting big on a flash in a pan that goes nowhere, or completely dismissing something that turns out to be a big breakthrough.