by Luna Fazio
An anecdote
In the course of my research, I will occasionally indulge in a session of academic archaeology and follow a trail of references back in time to see if I can find the source of some particular concept or idea. I would like to think that acquiring this kind of historical perspective can help me be a better scholar, and sure, it might, but I also have to admit that I enjoy doing this for a far more mundane reason: it’s fun to observe the way academic communication has evolved over time.
One particular example that really stuck with me is a footnote in the first page of a landmark paper by Neyman and Scott 1948
I found it simultaneously amusing and sobering to realize that the humble square bracket, at one point in the not-so-distant past, constituted a novel technology that warranted explanation. With the awareness that even such a small thing, today a completely ordinary standard, was subject to processes of iteration and innovation, I felt my assumptions about the various forms scientific publishing might take drastically broaden.
I don’t see much value in making speculative forecasts, but I think it can still be informative to consider which dimensions of change appear to be both feasible (technically, socially) and desirable. My thoughts on these points draw from three places:
1. Text reuse across papers
When going over the work of a specific author, something I’ve often noticed is that certain sections of the introduction are very clearly re-used across multiple papers. Nothing wrong with that! Of course, you should update the text when new studies come in, but it seems otherwise pointless to spend time finding new ways to re-express the same core information.
Now, while that’s a very straightforward solution, I would argue we can probably do better than copy-pasting the same chunk of information across different documents. What I’m essentially invoking here is the DRY principle (Do not Repeat Yourself), a common notion in software development where, if a piece of code is seen to appear multiple times in the codebase, one should instead try to replace that with a reference to a single, shared definition. This brings me to my next point…
2. Science could learn more from software development
In a talk titled Science as Amateur Software Development*, Prof. Richard McElreath puts forward the thesis that the workflows of science and of software development aren’t all that different, and that certain good practices of the latter still have much to offer in improving the former. The thought is most explicitly stated around 39:07 in the video:
“[software engineering] has sociological things to contribute as well, because it has a set of professional standards that allow people to work in distributed teams, often internationally, and continuously integrate their contributions, test for code before it’s deployed, [etc.]”
Just before that quote, McElreath makes it clear that software development already has a direct technical role in modern science in the form of the various tools we use to gather, process, analyze and store our data. His statement then goes one step beyond, to suggest that the process of knowledge integration itself can also incorporate these workflows.
This all sounds very nice in theory, but do we have any indication that research can be conducted in such a decentralized and simultaneously well-integrated manner? I believe we do!
3. Mathematicians have been doing it for a while
In A New Paradigm for Mathematical Proof?, Prof. Emily Riehl asks a question (12:43) that I think should be of interest for researchers in general: “why has mathematics largely avoided the replication crisis that has confronted other fields?” There’s not a concise, nicely-quotable answer to this, but a minute later she makes a comment that I think strongly hints at the reason: “one thing that is remarkable about the mathematical community is the wide degree of consensus; people all around the world who I’ve never met and I have a very similar worldview about a whole bunch of different things.”
To me, this sounds like something quite close to the ideal McElreath points us towards. Even when mathematics has historically followed a similar publication-based model for dissemination of knowledge as other sciences, I believe that the structured nature of that knowledge and the value the discipline places on tracing a clear provenance of their results has more or less resulted in similar dynamics as those seen in software development. And if you continue watching Riehl’s talk, you will see that this is a direction that is now being taken deliberately, by bringing in systems for computer proof verification that essentially require that mathematical knowledge be represented as a computer program.
In light of the above developments, I think it is not completely farfetched to think that researchers in other disciplines will eventually converge to the same notion that expressing their claims in more formal, computer-representable terms will facilitate the construction of a common body of knowledge with potentially many benefits in the form of increased efficiency and trustworthiness of results.
What, then, will the paper of the future look like? Perhaps it won’t look like a paper at all. Instead, next time you have some new result you would like to contribute to your discipline, your contribution may be commit into a repository holding the knowledge that has been so far accumulated by your peers.
Epilogue
I think it might be natural to react with some pushback to the idea that all forms of research are amenable to a formalized treatment to the degree that math and computer programs are. In this regard, I feel compelled to share the thoughts expressed in Yarkoni 2022, which I think can be crudely summarized in this way:
Yes, some research questions can seem very hard to formalize. But if we don’t try, how can we even hope to extract some form of usable, shared understanding from our results?
That may sound a bit harsh, but I actually think that recent developments in the text-computer interface have a huge potential to facilitate this task (yes, I’m talking about LLMs). Again, all of this is just me enunciating what I think is possible in principle and things may evolve in a very different manner. Be as it may, you can be sure my colleagues at the I4R and yours truly will be eagerly observing, and perhaps even facilitating, any developments that can improve the way research comes together to build a trustworthy, collective body of knowledge.
*: There’s a more recent version of that talk, but it’s a bit shorter and leaves out a more substantial discussion of an already-working implementation of such a workflow (starting at 43:35 in the first link).