Most if not all theories of science are unable to explain the following phenomena: that good non-empirical papers routinely receive just as many if not more citations than empirical papers which document new findings.
This can be counterintuitive because when we think of “big” scientific developments, we tend to think about empirical discoveries. Examples include Galileo’s observation of the moons of Jupiter; Crick, Watson, and Franklin’s discovery of the structure of DNA; Hubble’s observation that the universe is expanding in all directions; Marie and Pierre Curie’s discovery of radium and other elements; Young’s demonstration of the fact that atomic particles are somehow both like waves and small billiard balls; Fleming’s discovery of penicillin; and John Snow’s work on the germ theory of disease, to name a few.
However, across scientific fields, review papers receive ~3x as many citations on average compared to empirical papers (Miranda & Garcia-Carpintero, 2018, analyzing 14.2 million scientific papers.) Note that a review comments on the current state of the scientific literature, and is a type of non-empirical work (other types include theory/framework papers, and methodological critiques.) Two famous examples are Einstein’s relativity papers, or Darwin’s On the Origin of Species, both of which are primarily theoretical.
If Miranda & Garcia-Carpintero’s paper wasn’t proof enough — or if you don’t have access to their paper behind the paywall — take, for example, Terrie Moffitt, widely considered by insiders to be one of the best (and most cited) living psychological researchers. Her most cited paper is a non-empirical paper. Four of the top five most cited papers in psychology are, by my count, non-empirical papers. The journals with the highest “impact factors” (average accrued citations over a 2-year period) in psychology exclusively publish reviews. Across a variety of fields (like physics and medicine), journals that publish reviews are either at the top or in the top handful of journals. Perhaps most strikingly, ten of the top fifteen journals by impact factor primarily publish reviews and non-empirical work.
You’ll notice that the existence of non-empirical papers has little to do with what you learned as “the scientific method” in school. Yet while your educators were well intentioned, and despite the importance of the scientific method in creating our modern society, it is likely that your understanding of the scientific method is incomplete. Especially if you don’t have any formal scientific training.
Let’s call the account of “The Scientific Method” we all learned in grade school the elementary version. It consists of the following steps:
Choose a question
Make a hypothesis
Run an experiment capable of proving your hypothesis wrong
Analyze the data
Draw a conclusion
On the surface, this seems like a reasonable account of how new knowledge is produced. Intellectually bougie types may offer nuances to this account by quoting Roger and Francis Bacon (who respectively argued for experimentation and inductive reasoning as paths to knowledge), Karl Popper (who pointed out the importance of having hypotheses that can be proved false), and Thomas Kuhn (who said that most scientific progress is incremental but occasionally punctuated by paradigm shifts).1 These are all important points. But most of these theories of science boil down to either theories of ideas, or theories of historical scientific developments. Most theories of science can neither predict nor explain basic meta-scientific phenomena like the presence of review papers and the success of non-empirical work. Given that scientific theories can be judged by their ability to explain and predict phenomena, not only is the elementary school version of the scientific method incomplete, formal theories of science are too.
To offer a theory that can explain the success of non-empirical work, let’s start by recognizing that one thing the elementary school version overlooks (but which the sociological approaches from Robert Merton to Bruno Latour and Donna Haraway get right) is that scientists and researchers exist within an ecosystem of knowledge production. Real science is created by communities of researchers having conversations through the admittedly slow medium of publishing peer-reviewed papers. Thus what makes for good science has much more to do with identifying and highlighting what needs to be added or discarded from the communal understanding shared among researchers.
Specifically, a good review paper (or other non-empirical work) attempts to shift the conversation by identifying that which is missing from it. This is a difficult skill to master, as it requires a kind of imaginative capacity to see what isn’t there that is only implicitly taught in graduate school.
Why does this matter? Metaphorically speaking, just as glass lenses are tools for focusing light, scientific papers are conceptual tools for focusing attention on and directing thoughts about some phenomenon in the world. And so it is with this interpretation of what papers do (a “lens” lens if you will) that we can explain the puzzle of non-empirical papers: when done well, they can change researchers’ minds about what entire lines of pre-existing empirical research mean.
To put it another way, roughly what happens is that when you write a non-empirical paper, you say “hey, everyone, here’s a new window through which you can see these mountains we’ve been studying. Don’t they look different from this angle?” If the paper seems right, everyone else who wants to get a good view of any aspect of those mountains will incorporate your new vantage point and cite your work in the process.
With this understanding in hand, here’s an updated proposal for what The Scientific Method consists of. (One caveat: I obviously don’t have access to how every piece of science around the world is done, and this is based on my own observations. Exceptions likely exist! ) Let’s assume you have a topic you are interested in. If you want to contribute to the research conversation, you’ll need to follow the following steps in more or less the following order.
Step 1a: Observe and understand recent “conversational” trends in the scientific literature as it pertains to your topic. It is highly likely, given the relative “old growth” status of the scientific ecosystem, that other people have already thought about the thing you are interested in, unless you are already part of that particular research community. The tricky part is that as with all specialized communities, the topic you are interested might be disguised in obscure language and concepts, phrases you aren’t familiar with, or it may have been studied with specialized techniques unknown to you. Other smart and curious people, who have also been around in the world, have likely been discussing something relevant to your topic of interest. Science does not happen in vacuums, at least not without highly specialized equipment.
Step 1b: Notice a specific phenomenon in the world as it pertains to your topic and what you read about in Step 1a. (I’m going to call this “The Phenomenon” for the rest of this post.) This may be easier for people like myself in the social sciences, as we have the option of drawing from our life experience. I imagine that if you do bio-chemistry, the time when you could use what happens in your kitchen as the basis for your hypotheses is mostly gone.2
Step 2: Iterate on Steps 1a and 1b, until you can imagine “what’s missing.” By going back and forth between a) reading and understanding what the scientific record has to say about The Phenomenon, and b) what you think about it (informed by observations, etc.), you develop the ability to think about The Phenomenon in the way that would allow you to have real conversations with other experts who also study the topic. This is essentially what happens during the first few years of getting a research PhD: you learn enough new conceptual structures that you start to think like a researcher.
To imagine “what’s missing” well, you have to be able to see what everyone else is thinking, and then figure out how to think about The Phenomenon in a way that a) no else is and b) advances the research conversation.
Crucially, “what’s missing” is sometimes described as either “a gap” or like “a piece of a larger puzzle.” I think these are incomplete metaphors. For even the tiniest gap is still a gap, and the “gap” framing implies a small and bounded negative space to be filled in. More importantly, scientific contributions don’t really fill gaps as much as they change how the landscape of existing knowledge is seen. Thus the creation of knowledge is not really like solving a jigsaw puzzle, because new puzzle pieces do not cause the shape, size, and image depicted by the extant puzzle to change (nor do new pieces cause “old pieces” to be discarded as theories go out of date).3
I think more interesting research is done when instead of thinking of your project as something akin to spackle that fills in negative space, you add something positive. The best way I can put it is that the goal is to create something through which most future thinking on your topic of interest will have to flow through. While this may end up filling a gap, ideally you want your contribution to surpass the known contours of the problem.
Step 3: Think of a way to address “what’s missing” such that other researchers in the community will pay attention. In order to do this well, you have to know what other researchers in your community will pay attention to. If you are working on an empirical project, this could take the form of a falsifiable way to test your phenomenon. If you are working on a non-empirical piece of commentary (i.e., “a review”), you’ll want to aim for creating a backbone that links other pieces of research together. In either case, the goal is that people who want to create future work on the same topic will have to funnel their ideas through the new lens your paper provides.
There’s a sense in which it is helpful to think that there is “too much” scientific literature out there. (This isn’t true, but bear with me for a second.) By publishing a paper, you’ll be incrementally contributing to the problem. So the least you can do is a) push the conversation as far as possible, and b) provide some way to focus future thought.
For example, virtually all work on child development mentions either Bronfenbrenner’s Ecological Systems Theory model or Belsky’s Process Model of Parenting. Both frameworks stand in for having to synthesize a bunch of other work together to make the same point. Both frameworks were responses to what the authors thought was missing at the time (and have since turned into decent primers on their specific subject, as they influenced the communal understanding of their subjects when they were published.) Also, given that we don’t know everything about child development, we can be sure that these models are incomplete. They both extend and can limit our thinking on the subject.
Empirical work obviously also guides future thought, and the difference between empirical work and non-empirical papers in this “papers as tools for focusing thoughts” metaphor is that the “aperture size” for the former is often narrower than the “aperture size” for the latter.
Step 4: Form a hypothesis or set of hypotheses. (Steps 3 and 4 can be swapped and are probably best done in an iterative loop.) For empirical work, hypotheses are sets of expectations about what will happen in your carefully controlled test. For non-empirical work, the “hypotheses” take the shape of expectations for many future papers instead of one experiment.
If you are working on an empirical paper:
Step 5a: Conduct your test, gather data. As Feynman says, “The first rule is that you must not fool yourself. And you are a very easy person to fool.”4 It pays to be wary of how you experience your tendencies of identity protective cognition, motivated social cognition / reasoning, the egoic tendency to want to be correct, expert overconfidence, and our general discomfort with uncertainty (as data is often a little messier than we’d like).
Step 5b: Interpret your results. Here you have to figure out what your data means. At the risk of repeating myself, if you’ve asked the right question, you are implicitly creating a lens on, or view of, the world that future research won’t be able to ignore.
If you are working on a non-empirical paper or review:
Step 5: Synthesize a new lens. It’s true that some non-empirical papers are relatively rote and just try to summarize prior literature. This can be quite valuable, as meta-analyses form one of the core pillars of medical science. But if you notice an assumption you are making as you think about The Phenomenon, or if you see other people making an assumption, your final paper will be the better for it.
Step 6: Get your work published by going through the peer review process. (This is a larger topic for future posts.)
In other words, you want to be able to accurately say “It’s common for researchers to collectively think X about the effects of Y on Z. However we who are thinking about the Y=>Z relationship are missing something.” Or in other words, because we want to fully understand The Phenomenon, we want to understand how we might be collectively wrong.
If you do this right, people who read your work will not be able to go back to the old way of thinking, or what was “the standard way” before your paper arrived in front of their eyeballs. Please note this is very far from saying that everyone else is stupid. Far from it! But if you think about it, how else would you influence the research conversation? You’ll also notice I did an admittedly aggressive form of this with the title for this post, where I suggested that most people’s notion of the scientific method wasn’t the fully story.
With all of this in mind, if a scientific theory is one that can explain some real-world phenomena (like why there are rainbow patterns on soap bubbles), a theory of science is technically part of the science of science, or what people call “metascience.” But what I find so curious is that the theories of science I’ve mentioned above (and in the first footnote below) are only semi-oriented to really explaining, predicting, or even describing many key parts of the research ecosystem, which would seem to be a pre-requisite for trying to improve it.
I’ve attempted to demonstrate here that classical theories of science have a blind spot and can’t explain things like why review papers are cited at ~3x the rate of empirical papers. In part because classical theories of science are more focused on ideas (“epistemology”) there are plenty of other aspects of research that a) cannot be explained (or aren’t even mentioned) by classical theories of science, but b) are well within the remit of metascience. Consider these meta-scientific phenomena and questions:
The age distributions of scientific conferences. Why is it the way it is and not some other way?
For geographically distributed labs that study similar topics… how much variation is there in the day to day lab experience? Does this variation differ between fields and subfields? Does this have an effect on the quantity and quality of new knowledge that is produced?
On the flip side, why is there so little variation in the organizational structure of most labs? (i.e. why are the org-charts in most labs likely to have a similar number layers?)
In other words, many day-to-day organizational processes and systematic constraints are largely unmentioned and unacknowledged by metascientific theories, at least from my point of view as someone who has worked in both the “private information economy” of Tech and the “public information economy” of academia. I aim to explore more of these topics in future posts.5