I have been following the AI debate from the sidelines for quite some time now. I wanted to stay out of it for several reasons: I am not an expert, in fact (except for the most basic applications) I am not an actual user, and there are way too many other people involved in the discussions already.
Yet I’m concerned. Worried in fact. And it is significantly affecting my mood. Which is why I’m writing this: to face this ‘monster’ and try to make sense of it. Mostly for myself. But perhaps it is useful for others too. Which is why am sharing this publicly.
On AI
Artificial intelligence means different things to different people, ranging from neural networks used for image recognition to algorithms that generate an image, a video or a text when given a prompt describing the desired output. Text generating algorithms are commonly known as Large Language Models, LLMs for short. And this blog mostly concerns this type of AI.
As Bender puts it, artificial intelligence has always a been a marketing term. (Oh! The marketing term has changed: now we are supposed to call it Super Intelligence (SI) apparently.) And unfortunately, the discussion whether computers can or cannot be intelligent has turned into a bitter argument between haters and believers that is totally counterproductive.
AI scepticism
There are many reasons to be sceptical and reserved about AI and its use. And so I am, even though I do see its potential (as discussed further on).
My primary concern is the extractive nature of AI. The models are trained on data they scrape, without permission, from all over the web. This is a both a massive copyright violation as well as a horrendous privacy infringement. Even people within Microsoft called AI scraping ‘the largest theft of labor in human history’.
Another big concern for me is a loss of autonomy, in three different ways. The first way is by depending on a tool that runs in some data centre that is not under my control. Several things could happen: access could be revoked, conditions of use could change, functionality updated, or cost of use increased. (So far I managed to be pretty independent of cloud based solutions for the same reason.) The second way is more subtle (and to which I will return later on): the fear of deskilling: to depend so much on a tool that I cannot do the things I do now without the tool anymore. And finally, and related to the previous point, I fear that using AI will subtly influence my work and private life, and that I will stop thinking (critically) for myself. And perhaps that is the whole point: as Ali Alkjatib puts it:
AI is an ideological project to shift authority and autonomy away from individuals, towards centralized structures of power.
Also concerning is the bias exhibited by AI because of the bias in its training data. Moreover, and this is especially the case for generative AI, its output can be wrong. Which is exactly why critical AI researchers refer to LLMs as “stochastic parrots”: they randomly select the most plausible response based on the huge amount of training data used to build it. The response of such a parrot is restricted to what it was told about (hence the bias), and selected randomly from the most likely responses (hence not necessarily being correct all the time).
Finally, generative AI models are resource intensive and demand large amounts of electricity, water, and hardware, and produce a significant amount of carbon dioxide, both when training the model and when using it. This is a serious concern given climate change, which means we have to carefully consider for which applications to use AI.
AI capabilities
But there are also many reasons to admire the capabilities of AI. As Sanders and Schneier put it:
AI represents the first time we humans can do cognitive work outside of our bodies at scale.
On top of that, agentic AI systems can do so autonomously, without human interaction. AI may turn out to be just as disruptive as the industrial revolution, that provided us the tools to do mechanical work outside of our bodies at scale.
I simply have to admit AI is doing amazing things. The fact that it can transcribe audio, recognise even my poor handwriting, make summaries, play Go, write a convincing argument, help construct a mathematical proof, predict protein structures, generate ever more realistic images or videos, write software, or find bugs in them: all these things are true advancements compared to what a computer could (help us) do ten to twenty years ago. It may be far from perfect, and it may require quite a bit of proper prompting to get there. Still, it is quite remarkable. And it is already having a significant impact.
Impact at work
My mood is most directly affected by the impact generative AI already has on my work as an associate professor in computer science.
As a researcher
As a researcher I have to compete with LLM generated papers, or (even worse) have to review them.
The use of generative AI in research has exploded. Two weeks ago I heard the USENIX Security Symposium received a whopping 2000 submissions in the first cycle alone, many of which were written by a single author. It scrambled to deal with this by extended the Program Committee to 800 people. This is unsustainable. (And while writing this blog I learned about the curious case of a professor publishing 200 papers and 14 books this year.)
Earlier this year I was engaged in a discussion on the update of the policy of the Privacy Enhancing Technologies Symposium regarding the use of (generative) AI. Back then (only half a year ago!) the PETS leadership believed it could manage an increase in submissions, and was permitting the use of (third party) AI tools, without even requiring that the substance of the paper must originate from the authors themselves. At the same time, reviewers were not allowed to upload submissions or any parts of them to any third-party service, be it AI-related or not, and were required to do the reviews themselves, not using (local) AI tools or sub-reviewers. I understand that USENIX is now considering to also allow reviewers to use AI. But where does this lead us? AI generated papers being reviewed by AI reviewers, being published in conferences and journals that universities pay to get access to?
The mathematics community has been relatively quick to engage with the concerns that AI raises in the sciences. The Leiden Declaration on Artificial Intelligence and Mathematics published June 2026 points out several threats:
Results may look plausible but could be incorrect; it is difficult to distinguish them from correct proofs.
If results are not published as a research paper but instead as a press release or blog post, independent evaluation is impossible.
The models are trained on previously published works. They frequently return outputs that do not properly cite this human work, thus undermining the traditional system of attribution.
Researchers without access to the latest models, cannot afford the significant costs, or those who have moral objections to their use, will be severely disadvantaged.
The use of AI may lead to a strong focus on studying problems that are amenable to automated mathematics. Other deeply significant problems relevant for the broader understanding of the field may be left underexplored.
This month Fields medallists published a follow-up declaration on A Severe Misalignment of AI in Mathematics, stressing that the point of solving hard mathematical problems is not the answer itself, but properly understanding the mathematical landscape underpinning these problems. This understanding is necessary for mathematics to progress, to study new ideas.
To that I would add that depending on models hosted by large American or Chinese corporations creates a vulnerability for scientist in Europe in terms of strategic autonomy. And as the recent turmoil around OpenAI’s claim that it had solved the Navier-Stokes problem reveals, the interactions with such models can and will be used to improve them, to the point that the intermediate results found by human researchers are co-opted by the model (or perhaps unwittingly by other scientists using the same model) to publish the final results first.
With the current publish-or-perish policy, researchers are incentivised to increase their output at all costs: whether to publish in predatory open access journals, or using an LLM to write your paper and stick your name to it.
(Oh, and I am not even willing to think about the apply-for-funding hell. If LLMs are indeed bullshit machines, then proposal writing is their perfect use case.)
I must honestly say that I am extremely happy that I have had the privilege of doing research before the LLMs took over. It was fun, stimulating, enriching, maddening, frustrating, … I will try to do my best to retain that fun for the few more years ahead (before my retirement). But I do not look forward to having to engage with the incoming tsunami of obviously LLM assisted CVs, papers, and project proposals, as I fear it will wash us all away completely.
As a teacher
As a teacher and supervisor of students, I increasingly encounter LLM generated content in student papers. Sometimes they contain hallucinated references. Others read like glossy marketing brochures. Let me tell you: computer science students usually do not write like that.
Over the decade(s) that I taught the Privacy Seminar I have noticed that students got better at presenting, but worse at writing. Which means students are even more incentivised to use this amazing tool to help them write their assignments, better and much quicker.
So far, I have not been affected that much by generative AI in my seminar. (Or, at lest that’s what I think…. There is not much more than gut feeling to guide me.) At the start of the course I make very clear that the use of AI is not permitted, and (more importantly) that the whole point of the course is for students to learn how to write and to discover their own voice. (And that I really enjoy reading those voices.) I also happily make use of an analogy that I heard from a colleague of mine:
Using generative AI to write a paper or solve an assignment is like bringing a fork-lift truck to the gym to lift your weights. The point is not to lift the weight, but to train your muscles. Similarly, the point of writing a paper or solving an assignment is to learn and exercise your brain.
Moreover the format of the course, which forces students to also present about their topic, ensures that they at least have to understand what they are talking and writing about. Other colleagues have not been so lucky: they have had to redesign their courses as unattended assignments where massively completed with the help of LLMs.
Nevertheless I have encountered cases where I suspected a student to have used an LLM. To confirm a suspicion is not easy though. Tools like Pangram (though heavily used) are not perfect and may falsely accuse a student. Even in the obvious cases (hallucinated references) the exam board of my faculty failed to uphold my judgement (a fail) when the student complained.
The Radboud University (where I work) does not have a university wide policy for the use of AI in education yet. There are some general guidelines though, that in my view do not really engage with the issue in a serious manner:
Ultimately, the responsibility for the use of tools such as ChatGPT lies with the student themselves, but lecturers do have the responsibility to help students deal with this in a responsible manner and to point out the risks. The most important thing is that students learn about the limitations and reservations of AI
This is in stark contrast with the recommendations in the MIT AI and Education report,
Rather than simply “AI-proof” current methods of assessment, instructors need to revisit what they really want students to know and devise assessments that foster, or even include, the kind of productive struggle that builds durable understanding and mastery.
We urge instructors to consider forms of assessment that are less vulnerable to AI, and more valuable for learning, such as oral exams, semester portfolios, and out-of-class assignments paired with in-class conversations. This likely means resources such as TAs and class time will become more central to evaluation.
This latter approach aligns well with concerns that carelessly relying on AI too much and can have negative impact on critical thinking and developing necessary job-specific skills, while structured, deliberate use may actually boost critical thinking and creativity. This is not a problem specific to AI though (as this Nature paper also points out).
The Dutch laissez-faire approach, or worse, actively stimulating the use of AI in education without clear guidance and boundaries, is deeply concerning.
Impact in society
More broadly, the impact of AI in society at large scares me too.
The use of algorithms in decision making can lead to unfairness, unaccountability, and lack of transparency. In a way this is nothing new. We already know this for quite some time. But LLMs add to this even more, at scale.
Over thirty percent of the web pages published after the release of ChatGPT in late 2022 appear to be AI generated. This has also made searching the web much less powerful, as the returned result are increasingly less useful. As described above, the use of LLMs as writing tools (also) has a significant negative impact on teaching and academic publishing.
Generative AI is being used to create deep fakes, fake porn, and misinformation online. In this sense generative AI exacerbates the problem of algorithm driven social networks that threaten to destabilise our society. Not only by the spread of provocative content, but also by depriving independent media of a stable revenue stream (summarising the news and thus completely severing the ties between the actual news outlets and their audience).
People rely on AI a lot/too much in their daily lives, to get medical advice, plan a trip, make flyer or a menu, to book a restaurant, make music, or for counselling. Sometimes with devastating consequences. We are given the tools to compose an email with an LLM, or to let an LLM write a nice reply. Soon the bots will be talking with each other, on our behalf. Where does all this leave us?
Politicians - including our prime minister - use AI to generate speeches, or social media posts. Rather than being embarrassed about it, our PM counterargued that everybody was using AI so why shouldn’t he? But how sincere is an AI generated post about the horrors of the Holocaust or the wrongs done during colonial times? If a politician does not want to engage with citizens directly and sincerely, then don’t. Stop pretending.
More generally it appears the main driving force for using AI is efficiency: doing things faster, at lower cost, with less (personal) hassle and friction. Whether it is writing code, doing research, making music, making a movie, writing a document, etc. I am therefore not so sure that we can actually separate AI’s technological problems from its capitalism problems. Even more so as it appears to monopolise all available resources to create a means of production in the hands of a few big corporations.
AGI and human extinction?
The final trigger that got me started writing this every growing post was the recent flurry of media coverage of the threat of imminent Artificial General Intelligence (AGI) and the risk of human extinction, while others were busy debunking the hype.
In particular the Hugging Face incident (where OpenAI agents hacked into the systems of Hugging Face) was framed as an example of AI agents ‘going rogue’. As this independent analysis of the incident explains, “this wasn’t a rogue agent disobeying orders. It was a system built to hunt for exploits, doing exactly that, just against the wrong target.”
Others also are playing down the significance of the Hugging Face incident, especially the narrative in the media that the agents ‘went rogue’ and ‘broke free’. However, we should acknowledge that the agents showed remarkable flexibility and perseverance in pursuing the goals set to it, especially with all guardrails apparently off.
I would say this has very little to do with the agents being intelligent or going rogue. But quite a lot to do with agents being agentic, the agents being general purpose and able to respond to challenges flexibly and ably, the sheer compute power thrown at it, and (let’s not forget) a lack of proper oversight. Being general purpose, flexible and powerful implies that (unlike traditional, pre-programmed agents) there are no prior guarantees on the limits of their behaviour. An agent specifically programmed to make a reservation at a gym will not be able to hack the reservation database of the gym in order to achieve its goals. However, a general purpose AI agent with the latent potential to hack might be incentivised to use that power to achieve its goals. A larger organisation using agentic AI without oversight might similarly be seen to use unexpected approaches to achieve its goals, possibly with much more severe consequences.
Another issue is human gullibility, and the apparent sycophantic nature of LLMs. As a result, people tend to believe their output at face value, opening avenues for all kinds of nefarious purposes, from elaborate automated phishing scams to subtle influencing through chatbots. I sketched one such scenario last year. Social networks are already used to spread misinformation and as medium for destabilisation campaigns by malicious actors. Adding AI chatbots amplifies their power.
So intelligence, let alone artificial general intelligence, is not the issue here, but the unrestrained and unsupervised deployment of AI, and its potential for malicious use, are cause for great concern.
Conclusion
AI is already having a significant impact. There are benefits, but also downsides. The uncritical adoption of AI for work and private lives, without thinking the consequences through worries me. The focus is doing things more efficiently, not doing them better in a much broader sense - and without any regard for the environmental and societal cost. What is especially missing is any sense of responsibility when deploying AI. This needs to change. For example by holding those responsible to account - and make them liable for the costs (including the societal costs that are less easy to quantify).
We are also not putting any restraints on the development of AI. Even though there are clear reasons to do so: not the supposed threat of AI ‘going rogue’, but rather the theft of intellectual property, the privacy violations, and the concentration of power that lie at its core. We should not turn a blind eye to these trespasses, simply because we believe AI is useful.
For me the discussion whether AI is actually (super)intelligent or will ‘go rogue’ is a red herring, that stops us from seeing the actual disruptive effect it is already having, and to hold those that are responsible to account.1
I was interested in the topic very early on in life. I devoured Hofstadter’s Gödel, Escher, Bach when I was a teenager and wrote a thesis on AI for a philosophy class in high school, for which I read Weizenbaum’s ‘Computer Power and Human Reason’ as well as Boden’s ‘Artificial Intelligence and Natural Man’. Can’t remember much of the thesis (I cannot seem to find it anymore), but I do recall that the main conclusion was rather shallow but expressed with the typical boldness of adolescence: why even care about artificial intelligence if you can get normal, human, intelligence essentially for free ;-).↩︎
In case you spot any errors on this page, please notify me!