It is unsettling to read prose that appears straight from an LLM. Sometimes, it is word-choice — “load-bearing”, “genuinely,” “smoking gun” — sometimes, it is sentence structure — “it’s not X, but Y” — sometimes, it’s formatting — bold areas in places that look unusual — sometimes, it’s just the overall feel.
But it’s hard to know with certainty if something is AI, and likewise it’s hard to know with certainty if it’s not. This is a classification problem — any set of text has a ground-truth label as AI or not, and we attempt to predict the label.1 Classification errors became familiar to the general public during COVID-19 as trade-offs between false positives and false negatives turned into every day policy conversations. Now the conversation, especially in academia, turns to the question of “is this writing AI?”
I’m interested in this question for economics research. In a previous post, I looked at whether AI was causing a boom in the number of economics papers and found little evidence. But economists could be producing roughly the same number of papers while changing how they write them. Paper counts wouldn’t tell us much about that, but classifying papers’ writing would.
Using the commercial detector model Pangram, I find a couple pretty remarkable facts about the growth in AI writing in economics papers. First, almost none of the NBER and arXiv economics papers in 2022 are classified as AI. Second, almost 50% of arXiv papers have at least one 300 word window classified as AI, and roughly 25% of NBER working papers. Third, detected share of AI writing really takes off in 2025, with the advent of agentic AI.
Before getting to what I found, though, it’s worth understanding how AI detection works in practice. How much better can we do than reading a paragraph and thinking, “That sounds like AI”?
There are many potential ways to go about determining AI text, including open-source methods and commercial methods, such as GPTZero, Turnitin, and Pangram. These methods were evaluated in a paper by Jabarian and Imas (nicely summarized in this Chicago Booth Review article), where they (among other things) put a number of pre-2020 articles to the test to evaluate the false positive rates. They found that Pangram was one of the few models that was tuned sufficiently conservatively to have a low false positive rate — it did not find AI writing in the pile of writing that was impossible to write with AI.
Full disclosure: I asked for a grant from Pangram for API tokens to do some of the data analysis below. They gave me five thousand dollars in tokens which I then spent in the process of doing this blog post. I believe anyone interested in doing research with Pangram’s models could do the same. I was not compensated or connected to Pangram in any other way.
So how does a model like Pangram work, then? If we are trying to write a classification model, we need a lot of training data — text labeled not-AI and AI. We then can put the text in as features (or covariates, in econometrics lingo), and estimate our favorite machine learning models. The dataset for this exercise is generated using synthetic mirroring — take text that we know is not AI (e.g. the pre-2020 text in the Jabarian and Imas (2025) paper), and ask an LLM to mimic that chunk of text.
To make this concrete, let me show an example of how this mirroring would work using a large corpus of around 45,000 economics papers. The prompt to the LLM gives the title of the paper, the section being drafted, and the first 60 words of the human chunk. Then, the prompt asks the model to continue “in the same scholarly register” for roughly 550 more words.
For example, a chunk of Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China states for sixty words:
Except for rare exceptions (e.g., Crépon et al., 2013; Egger et al., 2019), randomized control trials produce causal partial equilibrium estimates of an intervention but cannot provide evidence on the intervention’s general equilibrium or indirect consequences. This setting is one where knowledge of the general equilibrium consequences may be especially important. This is because it is at least plausible that…
Now contrast two continuations (one human, one AI (Qwen 3.6)):
[This is because it is at least plausible that] … the response to citizen complaints triggers strategic adjustments by local officials and polluting firms that extend far beyond the specific cases being addressed. In the context of China’s environmental governance, where performance evaluations are heavily weighted toward social stability and ecological targets, the allocation of regulatory resources is not static. If the government increases enforcement or provides subsidies in response to a surge in complaints, it may inadvertently distort the marginal cost of pollution for other firms that have not yet voiced grievances. This creates a potential “crowding out” effect, where compliant firms or those lacking the social capital to articulate complaints face a relative disadvantage.
and:
[This is because it is at least plausible that] … regulators responded to public and private appeals by shifting inspections and other regulatory effort between firms, allowing untreated firms to increase their emissions due to the reduced regulatory scrutiny. If appeals only shift enforcement, it is possible that they had little or even zero impact on total emissions and ambient pollution concentrations. We designed the experiment to learn about the general equilibrium consequences of appeals by cross-randomizing treatment intensity across regions. Specifically, in 60% of the prefectural cities, 95% of the CEMS firms were assigned to the treatment groups, while in the other 40% of the prefectural cities, 70% of the CEMS firms were assigned to the treatment groups.
Feel free to respond in the comments which you believe is AI, and which is human.
With these two data sources, we can estimate a model that uses features of text of the text to predict the classification (Pangram uses an attention-based neural net). However, this is hard to do well even with a lot of data! So a key innovation for Pangram is to iterate with the model — use a leave-out sample of human text, predict labels using the first version of the model, and add the writing examples that were misclassified into the original dataset. This effectively forces the algorithm to improve on texts where it has done the worst.
The Pangram approach is pretty straightforward: generate a lot of synthetic data, train your algorithm and reinforce on places where the model is making errors, and keep your cutoff very conservative to avoid false positives.
Papers have pointed to Pangram has the most successful model at detecting AI while avoid false positives. Despite the conservative bent of Pangram, the evidence using Pangram to measure the growth of AI writing is pretty staggering. According to Dolezal et al. (2026), “by the first half of 2025, as much as 35% of websites uploaded to the internet in a given month were AI-generated or AI-assisted [up from ~0% in 2022].” Pew research finds the same.
Now let’s look at the actual results in more detail. I spent roughly three thousand dollars in API credits on Pangram’s model to classify every NBER working paper and economics ArXiv paper from January 2022 to mid-August 2026 — 2022 capturing the pre-ChatGPT launch as a placebo, and 2023 onwards in the new AI world. I chunk the text and focus on the introduction and a randomly selected section from the body of the paper. Pangram classifies text in sliding windows, each roughly about 250-350 words, and I cap chunks at around 2500 words (no ten page intros for me). The median paper has 5 windows per chunk, and I consider two chunks (the intro and the randomly selected section), so there are roughly 10-11 windows per paper.
I find a couple pretty remarkable facts. First, almost none of the NBER and arXiv economics papers in 2022 are classified as AI. This has really shifted my priors on how well we can avoid false positives. Second, almost 50% of arXiv papers have at least one 300 word window classified as AI, and roughly 25% of NBER working papers. Finally, the acceleration really only started occurring in 2025. This acceleration coincides with the growth of agentic AI (Claude Code) rather than just web-based AI platforms like ChatGPT.
The intensity of the adoption is also interesting. Only a tiny share of the sample in the NBER has a full-throated “dark factory” vibe, with all windows marked as AI. In contrast, 6% of the ArXiv papers have all windows in my sample marked as AI.
We already see a significant difference between the NBER working papers and arXiv working papers in their use of AI in writing. There are many hypotheses one could form for why — for example, the relatively low cost of submission to arXiv vs. high reputational costs at the NBER for AI writing. An alternative answer might simply be topics. The NBER tends to be empirical work, while arXiv includes more theory (both economic and econometric). I group the NBER working groups together to make it legible:2
In the end, while there’s clearly some variation (with productivity and innovation leading the pack at the NBER, and general econ at ArXiv), there isn’t an obvious story about fields. Instead, there does appear to be differences based on venue.
What about published papers? Over a sample of 200 published papers per year from top econ and finance journals, I found only one paper in 2025 that had a window marked as AI. But given the timeline between working papers and publication, this is perhaps unsurprising. A recent blog post in law journals found evidence of use of AI in law review journal articles growing, suggesting that working papers vs. publication is not much of a gap.
Pangram is not a perfect detector — in part because it is calibrated to have high specificity (low false positive rates), it may not be sensitive enough to pick up all AI writing (almost certainly not). Additionally, AI writing isn’t a binary — there can be AI writing that is edited, or AI writing that comes verbatim from an LLM. Frontier models continue to improve, and it’s possible that better quality models will grow harder to detect (or change their key signatures).
In fact, Pangram has released a new model (Pangram 4) which is far more sensitive and attempts to identify AI-assistance vs. AI writing. Moreover, it’s better tuned to the frontier models. It is also quite a bit more expensive, so I was not able to use it for the full census. When I run Pangram 4 on a sample of the most recent papers, I find far higher shares of AI writing — the model detects that over 50% of papers in the NBER working papers have at least one window that is AI written or AI assisted. Moreover, the gap in detection between the old model and new model has grown (despite neither having false positives in 2022).
Imagine you had the divine power to see through space-time and identify if a writer had truly written the text, or had AI do it for them. Holding fixed the content of the writing, would permuting the origin of the writing affect your evaluation of the text?
In the case of an academic paper, we assert that Jane and John Doe are responsible for the content and we rely on them to have spent the costly verification effort on the results. The fear is that when something is written by AI, that same content is less vetted. When forced to exert the effort to write out the argument, the author is much more likely to verify the work themselves simulatenously. When an LLM does this for you, the costly verification effort must happen afterwards, and it’s an outstanding question on whether we will exert as much effort.
Second, there is a related concern about credit. Often, papers and writing are a proxy or signal of the authors’ understanding and insight. This signal may now be worse for predicting their ability to reason and argue in other non-AI settings. In other words — did you write the paper, or did the AI? Should the hiring committee want to hire you?
Hence, if you could divinely identify AI writing even for identical writing, you might use it as a signal. But, if a classification algorithm can detect AI writing, this suggests there are also differences in the style and content of the written text. So, how does AI-detected writing differ from human prose in economics papers?
Detected AI prose scores as harder to read and uses longer words
Readability scores are metrics that quantify if writing is at the college-level, high-school level, etc. One example is the Flesch-Kincaid index, which trades off between total words per sentence (more is considered “harder” to read) and total syllables per word (more is considered “harder” to read). Coleman-Liau is similar. There are a number of comparisons one can make — the text flagged by Pangram vs. the text within the same paper that is not AI, or the overall papers flagged with any AI vs. those without AI, or the papers flagged with AI vs without within authors (e.g. exploiting our panel). A useful contrast is to compare the human papers vs. their raw AI mirrors (where the text has directly come from Qwen 3.6).3
In almost all cases, I find that the papers are viewed as harder to read. However, this readability effect is attenuated relative to mirrored Qwen 3.6 LLM text. This suggests that either the models used by researchers have less of an effect than Qwen, or authors work to reduce the effect.
These metrics are quite simple (linear combinations of word length and sentence length), so what drives these effects? It turns out that the economics writing flagged as AI uses longer words, but not necessarily longer sentences. It’s hard to pin down exactly why, but it appears that longer and rarer words are used by the AI models.
AI papers are similar length and have similar numbers of exhibits
The structure of AI-detected papers appears relatively similar compared to non-AI papers. They’re similar length or slightly shorter, with fewer exhibits, but only marginally so. Which suggests that so far, AI is not being used to create beastly papers (or we already had beastly papers to begin with).
Which words show up more often?
Pangram uses a complex attention model to identify AI, so word choice is not an ideal indicator for AI. But it is still fun to see the strange words chosen by AI. Some of the most common words for AI relative to non-AI detected papers include clarifies, capability (and capabilities), merely, and institutional. Clarifies is 12x relative to baseline, and institutional is 6x (20% of AI-detected papers have institutional relative to 3.5% of baseline).
The key ingredient to Pangram’s AI classification algorithm is access to (1) many human written texts and (2) mirroring this text using an LLM engine. I happen to have a large corpus of economics texts and an open-source LLM. Consequentially, I trained my own economics Pangram model. In a future column, I’ll walk through this model. For now, here’s the model if you’re interested in playing around with it.
I am very bullish on AI capabilities, but I am not quite sure how the roll out of AI will influence our approach to research. I still do not think AI will cause a dramatic increase in the number of papers written, but it’s clear that the number of AI-written (or AI-influenced) papers has grown. We need develop important norms to avoid outsourcing our thinking to these models.
Despite these concerns, it seems like a mistake to use something like Pangram in our publication process. Readers, editors and referees should be able to identify whether they think writing is good or bad, and there is sufficient heterogeneity on writing quality even among human writers that AI may be a boost for some writers and a negative for others (an overall negative effect does not mean everyone loses!). Moreover, an AI detection score is a crude proxy for the veracity of an argument, especially as LLM models grow in sophistication. An author could thoroughly verify an AI-assisted draft, and a fully human-written draft does not guarantee that you have sufficiently verified the results.
For now, I am trying very hard to avoid using AI to do much of my own writing — I find that writing down my arguments is an easier way to verify arguments. Instead, I keep my LLM agent focused on many other useful tasks, such as coding, reading, ideation, and search. Writing the argument myself is still the most reliable way I know to discover what I haven’t thought through. But, the benefits of having quick AI write-ups forces me to think about I can best develop methods and rhythms for my writing process. For my economics research, perhaps I want to write everything myself, but does documentation for coding packages require it? What about short blog posts? I’m still thinking through it (by writing).