Wikipedia:Village pump (WMF) - Wikipedia

As pointed out above, the current development stage is way too early for a broad rollout. This should have been better clarified, both to reassure the community about the experiment, and to provide clearer development goals.

However, taking a step back, a discussion can still be held regarding the potential of this whole endeavor. Would the community, in theory, agree to an AI model surfacing suggestions to newcomers in such a way, assuming the current pitfalls could be smoothed out in development? More concretely, are these expectations realistic, and is it worth investing further resources in this project?

These are questions I don't, personally, hold the answers to. However, we should be discussing them together, alongside members of the Editing team involved in its development (courtesy ping to @Quiddity (WMF)), if we want them to work in sync with community sentiment, and avoid investing resources in dead ends. Chaotic Enby (in solidarity · talk · contribs) 23:57, 15 August 2026 (UTC)Reply

Would the community, in theory, agree to an AI model surfacing suggestions to newcomers. I think a better way to explore this tool would be to make it available to established editors first. People who have the experience and policy knowledge to be able to properly evaluate the suggestions. Maybe the people will say "The suggestions were all spot-on and incredibly valuable". Maybe they will say "Nothing this thing suggested made any sense at all, it's total garbage". More likely, somewhere in between. But let's do the experiment rather than pre-judging it. RoySmith (talk) 00:08, 16 August 2026 (UTC)Reply

I think a better way to explore this tool would be to make it available to established editors first. While it might not have been clear at first, this is, in fact, exactly what the experiment is planning to do. The tool is still in development, and we can't say, in advance, how it will end up in terms of quality. For now, I'm just trying to figure out the proportion of editors who either will find it a non-starter in principle (regardless of the suggestion quality) or are opposed to investing further resources in its development for any other reason. Chaotic Enby (in solidarity · talk · contribs) 00:20, 16 August 2026 (UTC)Reply

Mark me down as "opposed to investing further resources in its development for any other reason". Wikipedia has a large amount of technical debt that would be easy for a WMF developer to fix, but instead they are doing this?

As one example, Arbcom is a very important function, and many people who participate get stressed over whether they are under the word count limit. But the tool that puts a banner at the top of your comment fails to accurately count your words! Worse, you can't invoke it while composing -- you have to post and hope that you didn't go over. And if an arb replies to you inline, that increases your count! This is the sort of thing that a WMF developer could fix in an afternoon. It's important, but making an accurate arbcom word counter will never hit the top of any survey of things many editors want to see fixed.

Another example: what happens if when I sign this comment I accidentally hit the "~" key three times instead of four? How about 5, 6 or 7? This is a typo that happens again and again. How hard would it be for a WMF developer to make it so that you get a "are you sure" message before accepting a signature that is almost always a typo?

There are hundreds and hundreds of these easy to fix things that depend on old scripts written by volunteers and all too often no longer maintained.

I think the WMF should spend a significant amount of developer effort -- 75% or 80% -- fixing these small, non-sexy quality of life issues and only then devote the other 20% - 25% to fun things like AI suggestions. --Guy Macon (talk) 01:02, 16 August 2026 (UTC)Reply

Or, if you don't like the above 4-tilde signature, --Guy Macon (talk) (3 tildes),  --01:13, 16 August 2026 (UTC) (5 tildes), or --01:13, 16 August 2026 (UTC)Guy Macon (talk) (7 tildes).Reply

Guy, on that last one, you don't need to add a signature at all anymore. Discussiontools will do it for you. I haven't added one to the end of this post, for example. In solidarity, asilvering (talk) 01:04, 16 August 2026 (UTC)Reply

Will I (Guy Macon) get an error message for this unsigned post or do I have to make a preferences change to get the autosign goodness? Show preview says it will post the unsigned comment with no error message.

Discussiontools is a nice tool, but it is no substitute for software baked into Wikipedia that checks for common errors and throws up an "Are you sure?" message. I could give you a hundred examples of places where the WMF is depending on unpaid volunteers to maintain basic functions that keep the Encyclopedia running smoothly while focusing on the exciting new stuff --01:30, 16 August 2026 (UTC) Guy Macon (talk)Reply

Guy would need to be using DiscussionTools for that, but unfortunately they are using classic full page source editing where none of those helpful things will occur. DLynch (talk) 13:13, 16 August 2026 (UTC)Reply

FYI I made Module:Word count, I can refine this further if you think it would be useful. It does not 100% solve the issue you mention Czarking0 (talk) 15:57, 16 August 2026 (UTC)Reply

Although I agree with a lot of this sentiment, I think an organizational strategy which places say 20% of the engineering resources on long term tech rather than present issues is reasonable. As long as the development of this feature counts under that I do not see the problem. Czarking0 (talk) 16:00, 16 August 2026 (UTC)Reply

+1, that seems to be what is happening here (see the comment below about 'avoiding sunk costs' and getting feedback early and often from established editors). I can see individual categories of suggestion being useful to experienced editors, focus on making something that works for them before considering anything that might be visible to newcomers. (They already have Special:Homepage) – SJ + 23:14, 18 August 2026 (UTC)Reply

I agree with Fram above in that this feels like a way (though a much more subtle way than Simple Summaries) for the WMF to influence editorial decisions which, with the exceptions of legal reasons, they just should not be a part of. ♠JCW555 (talk)♠ 01:55, 16 August 2026 (UTC)Reply

There isn't really any plans for the WMF to get involved in editorial descision (and I say that as somebody who has through m:PTAC reviewed the Annual Plan). The plan for Edit Suggestions is purely meant as a assistive tool to help editors and kinda comes from the idea of being able to surface gadget/userscript suggestions to everyone without having to have coding knowledge. Sohom (talk) 02:57, 16 August 2026 (UTC)Reply

But the fact that the AI is making these suggestions at all in the first place is the WMF having a subtle hand on editorial decisions in my mind. The examples Gnomingstuff lists above, like the FreedomWorks example, is an example of the AI making an editorial judgement that it should not be doing. Some of the others are more subtle like the DeAndrea G. Benjamin example, but they're still editorial decisions that the WMF shouldn't be engaging in period. ♠JCW555 (talk)♠ 03:23, 16 August 2026 (UTC)Reply

They are using generic models without fine tuning for these suggestions, so out of all the parties that could be said to have made editorial decisions in this scenario I would say OpenAI and Google would rank above the WMF, and I really wouldn't consider either of those companies to have made any editorial decisions... the problem IMO is potentially encouraging uh... not really making any editorial decisions, and vibing through things without considering the context of, e.g. the contents of the sources as we are supposed to. Alpha3031 (t • c) 08:28, 16 August 2026 (UTC)Reply

It is not worth creating AI prose suggestions for newcomers (especially if it takes a year for each model!). Putting aside quality questions, editors here are expected to be competent and able to contribute in English. While there are different ways to be confident, we expect the ability to read and write in English, and thus to some extent to have the tools to be able to copyedit themselves. We also expect editors to be able to read and comprehend our guidelines and policies (pre-emptive clarification, read, not memorise). The best way for us to be able to understand competence in these areas, and thus to be able to assess and offer advice if needed, is to see their edits. Seeing instead a whole slew of new editors making the same llm-prompted changes is harmful to the community in being able to understand and accommodate new editors, and harmful to the new editors in giving them a misleading picture of how things work, and more harmful when the llm doesn't understand our policies and practices (as this one does not). Apologies to Sohom, but "There isn't really any plans for the WMF to get involved in editorial descision" just isn't true if this sort of system is being set up. This sort of suggestion task is the WMF making editorial decisions, even if they're making it through an llm. There are many reasons time would be better spent anywhere else. (I distinguish "prose suggestions" from say the add-a-link task, as that at least teaches a technical competence that we do not expect new editors to have. Such teaching does seem useful, although as mentioned above it would be nice if it was developed further.) CMD (talk) 03:45, 16 August 2026 (UTC)Reply

Seeing instead a whole slew of new editors making the same llm-prompted changes is harmful to the community in being able to understand and accommodate new editors I just want to emphasize this -- the English Wikipedia community is not good at welcoming newbies who make mistakes. That is a problem. The English Wikipedia community is actively hostile to editors making mistakes with large language models. If a newbie puts a poor LLM-based reword into an article, an experienced editor is almost certainly going to WP:BITE them off. And a newbie, operating in a system they're unfamiliar with, with a human-sounding voice telling them "This copyedit is right", is never going to have the knowledge and is almost certainly not going to have the confidence to challenge the AI when it presents them with a bad suggestion. That is going to set them up for failure when a grumpy human editor, burnt out from dealing with LLM edits, callously reverts them.I think using machine learning to help with encyclopedia maintenance is a wonderful thing! And I think large language models are really cool -- but they require a high degree of skill to use correctly in the manner that looks like it's being explored here. And, again, the community is so burnt out from dealing with poor quality LLM content that any editor who uses this tool and (inevitably) makes a mistake will be attacked by the community. Does the team working on this understand that, @MMiller (WMF)? GreenLipstickLesbian💌🧸 05:10, 16 August 2026 (UTC)Reply

Seconding that this could be helpful for all sorts of maintenance but should be aimed at experienced editors while working out kinks.

I would personally like to see rubrics for highlighting potential vandalism and LLM edits. And I want much less text in my sidebar and zero suggested text: just a few words indicating the kind of issue to look for / the kind of style guidelines to check.

And I'd like to see explicit self-evals of each rubric for suitability to the task and for false positive/negative rates, which could also be compiled and developed by community maintainers. – SJ + 23:52, 18 August 2026 (UTC)Reply

@GreenLipstickLesbian -- I think that the most important thing that would prevent against the situation you're describing is that the suggestions wouldn't propose text for the newbie to accept/reject. It would just point out the spot in the article that needs attention, e.g. "Does this sentence need to be rewritten to be easier to read?" -- it would not give them a re-written sentence to add. I know that in that situation, the AI may be wrong about whether the sentence needs to be rewritten, and the sentence may be perfectly fine -- but the newbie might be like, "Hmmm, well I guess I'll reword it?" and they may make a pointless edit, or may make the sentence worse.

Suggestion tasks like "add a link" and "add an image" actually do give newbies specific edits to accept or reject, and we see them generally apply good judgment and be constructive. Yes, many of them mess up and get reverted, but that might have happened to them anyway if they were left to their own devices. So these tasks cause a bunch of things to happen at once, and the question is whether it all adds up to a net positive or net negative, you know?

What do you think? MMiller (WMF) (talk) 21:13, 19 August 2026 (UTC)Reply

@MMiller (WMF) Thanks for the response! Yes, I think these sound like reasonable precautions. I do really want to emphasize the point about making it clear to experienced editors what the newbies actually are seeing is very important.

Having had the beta experimental version of suggested edits enabled for a few days, I definitely see the vision. And I see how it could be very useful. (My response to many of the "consider adding a citation" suggestions has, admittedly, been a)"lol no, I'm removing the unsourced text for other PAG issues", b)"this is cited, it just needs an inline citation", c) "... yes, i see how citogenesis is made", or, d)"... we need an easily accessible essay on 'how to source a statement' that we can link to the newbies here" because wow, i'm having trouble". ) But I also see the biting that happens when newbies do make pointless edits/less than ideal edits ( has some which took more than a few years to rectify), and so I am worried about adding any "they're just thoughtlessly adding machine suggestions to articles"-esque ammunition, even it's it's not strictly true. GreenLipstickLesbian💌🧸 21:33, 19 August 2026 (UTC)Reply

This is why I suggested some basic FAQ type popups upon first edit, including an agreement to not use LLMs, to prevent true good faith mistakes and remove plausible deniability for the rest. ChompyTheGogoat (talk) 17:37, 26 August 2026 (UTC)Reply

RE: especially if it takes a year for each model! to be fair to the teams involved the stated idea behind this specific experiment seems to be wanting to try something that won't take a year per task on what the editing and ML teams think are relatively low risk tasks... I'm just not sure that the community and the teams involved have a sufficiently compatible idea of what tasks are low risk. I think it would be possible to develop something acceptable to the community, but unless very sure about it, it may be best to get a vibe check from the community before thinking something is "low risk", and treating things as "high risk" otherwise. Alpha3031 (t • c) 08:36, 16 August 2026 (UTC)Reply

Would the community, in theory, agree to an AI model surfacing suggestions to newcomers in such a way – I won't, and if that ever happens I'm gone. Models are bias black boxes, and even if suggestions were 100% accurate this would still be an issue. fifteen thousand two hundred twenty four (talk) 04:37, 16 August 2026 (UTC)Reply

If AI had far more quality control, I wouldn’t be opposed to this. Except it doesn’t yet. LLMs hallucinate, and those hallucinations are not something you want informing newcomers who are near-clueless as to how Wikipedia works, let alone people learning English who won’t be able to tell when an AI-based edit suggestion system instructs them to insert a grammatical error or something similar, like the above pyusician —> musician.

This could probably work in theory. But it would assume an AI with a reasonable knowledge of the given article subject, some common sense, and a degree of fluency in English (by which I mean not doing things like pyusician/musician), and, given the above examples provided by Gnomingstuff, this is definitively not that.

I oppose AI-based features being integrated into Wikipedia software, and will continue to do so unless a day comes when LLMs equal or surpass the common sense of a human. Perhaps in a few years LLMs will be reliable enough for this to work smoothly and without issue. With the current state of AI, I don’t think we’re there yet. Cheers, 𝔰𝔥𝔞𝔡𝔢𝔰𝔱𝔞𝔯 ^{(𝔱𝔞𝔩𝔨)} (any/all) In solidarity. 04:59, 16 August 2026 (UTC)Reply

The AI could work 100% of the time and I'd still oppose it because these edit suggestions are being directed by an AI that's controlled by the WMF, which outside of legal purposes, should never have any editorial control on Wikipedia in the first place. ♠JCW555 (talk)♠ 05:17, 16 August 2026 (UTC)Reply

That’s a fair point; I hadn’t thought of that. (Yet another reason why this is a terrible idea.) Cheers, 𝔰𝔥𝔞𝔡𝔢𝔰𝔱𝔞𝔯 ^{(𝔱𝔞𝔩𝔨)} (any/all) In solidarity. 05:39, 16 August 2026 (UTC)Reply

The solution for this is making prompts publicly available and allowing each community to tweak them. Alaexis_¿question? 06:36, 16 August 2026 (UTC)Reply

LLMs hallucinate, and those hallucinations are not something you want informing newcomers who are near-clueless as to how Wikipedia works, let alone people learning English who won’t be able to tell when an AI-based edit suggestion system instructs them to insert a grammatical error or something similar, like the above pyusician —> musician. - that's a rather odd example to get hung up on, given that automated spell checkers and their admittedly sometimes absurd correction suggestions have been around for decades (long before the term "hallucinations" came to be used in this context). Many or even most professional writers - journalists, book authors, scholars - use them routinely, and have learned to live with the occasional fail of this type (pyusician —> musician).

Indeed, in stark contrast to your logic here, WP:SPELLCHECK has long described them as potentially useful for Wikipedia editors, too, as long as they don't blindly rely on such tools:

Spellchecking software and online tools can be helpful when copyediting Wikipedia articles. [...]

• No spellchecker is completely accurate. You must check the output of any tool you use. [...]
• You are responsible for all spelling and grammar changes you make, even if the changes are suggested by an error-checking tool, such as Grammarly or ChatGPT.

Maybe it is time to conceive of Wikipedia editors a bit more as adults who are generally capable enough of using such tools even though their suggestions are not 100% error-free (very few things are).

That said, I do generally agree with your point that AI needs quality control, and folks should definitely ask if WMF has done enough here yet (I'm not sure it has). It's just that - as the spell checker example shows - it's not realistic to demand 100.000% accuracy, or to point to isolated failure cases without assessing how frequent they are. (To be fair, User:Gnomingstuff did already make an informal heuristical effort at the latter with regard to their list above: it took me only about ~1-2 hours to find the above issues, and that was only a spot check, i.e. there is informal evidence that these are not very rare at this point. But ultimately I'd be interested in more concrete assessments of how likely editors using this tool will be to encounter each of those failure cases in practice.)

Regards, HaeB (talk) 06:54, 16 August 2026 (UTC)Reply

Most adults have a lot more experience with spelling than they do with Wikipedia's guidelines. LittlePuppers (talk) 06:58, 16 August 2026 (UTC)Reply

Sure, but that doesn't mean that we keep the edit button away from them.

See Wikipedia:Competence is required, which is perhaps a better known version of this principle that we generally expect editors to be competent adults (metaphorically, with apologies to all the very smart and capable teenage editors among us) that do not require special restrictions and safeguards to protect them against their own mistakes.

Regards, HaeB (talk) 07:30, 16 August 2026 (UTC)Reply

The thing I’m concerned about is everyone knows that spellcheck is obviously not infallible, as the Internet often likes to humorously point out. But to a newcomer, anything that comes from ‘Wikipedia’ seems naturally correct and reliable.

Certainly not everyone would fall into this trap, since, as you point out, it isn’t like all newcomers are to be treated as children who don’t know what they’re doing. But it’s likely that some would; perhaps enough that the consequences of Wikipedia’s reputation for factual accuracy is something we should take into account on this. Given all of that, I don’t think we can draw a one-to-one comparison between run-of-the-mill spellcheck and an edit suggestion feature built into Wikipedia itself.

That’s why I argue that we should hold ourselves to a higher standard; after I’ve given your points some thought, however, I think you are correct that my own request for equal [to] or surpass[ing] human-level reliability is asking a bit much of a literal artificial intelligence. Simultaneously, I think spellcheck-level absurdity is far too low a bar for this feature.

Cheers, 𝔰𝔥𝔞𝔡𝔢𝔰𝔱𝔞𝔯 ^{(𝔱𝔞𝔩𝔨)} (any/all) In solidarity. 09:11, 16 August 2026 (UTC)Reply

Sounds like something that could be addressed with a user level disclaimer about LLMs before the feature is enabled on their account. Czarking0 (talk) 16:02, 16 August 2026 (UTC)Reply

That would definitely fix that issue (I’m surprised I didn’t think of that, to be honest). Cheers, 𝔰𝔥𝔞𝔡𝔢𝔰𝔱𝔞𝔯 ^{(𝔱𝔞𝔩𝔨)} (any/all) In solidarity. 00:11, 17 August 2026 (UTC)Reply

I agree with @RoySmith that some suggestions are more suitable for experienced editors. I believe that checking citations is a good use case with an LLM flagging potentially problematic citations and editors verifying them manually (we have a proof-of-concept that I've been maintaining but as long as it's a userscript it won't make a dent in the sourcing problem). This is a task that is not too complex but requires familiarity with WP:RS and some experience. Alaexis_¿question? 06:35, 16 August 2026 (UTC)Reply

Based on the suggestions here, my stance is basically the same as what WP:LLM already says: typos and punctuation suggestions seem unnecessary but mostly fine (except when they're not, see the "physician" example), anything beyond that is unlikely to be fine, and suggestions related to NPOV are light years away from fine. 100% accuracy isn't possible but, realistically speaking, people are going to rubber-stamp whatever suggestions they receive.

As far as the spot-check goes, the timeframe is not really all that scientific since I actually saw this last night (not even while doing AI stuff, I was doing Commons new image patrolling and saw one of the screenshots from it), slept on it, and did the full writeup the next day. I also have seen tens of thousands of AI edit suggestions, as well as thousands of snippets of parsed Simple Summary text, so I already knew the places where this was likely to run into issues. This is also why I think the "experienced editor"/"new editor" dichotomy is not helpful, in general. Experienced editors are not necessarily experienced in copyediting and/or the specific problems that crop up with AI, and newcomers are not fragile baby birds who have never edited a piece of writing in their lives. Gnomingstuff (talk) 07:10, 16 August 2026 (UTC)Reply

Automated suggestions could be helpful in very limited circumstances. I can imagine an experienced editor who has chosen to opt in to them assessing each proposal carefully, skilfully rejecting those with the sorts of problem listed above, and improving Wikipedia by acting only on the good ideas. I'm sceptical that AI is the best way to produce such input but I'm willing to be proven wrong. Methods such as this are totally unsuitable for new editors, many of whom will blindly obey The System, make bad edits in good faith, and get reprimanded or blocked. There is also the unoriginal but valid argument that editors would prefer the WMF to devote its resources to other activities, even if we don't always agree exactly on which ones. We're seeing a lot of these "Here's a new toy no one asked for that we've secretly blown this year's development budget on" announcements. I do honestly try to hope for the best, but I'm afraid my initial reaction has become "Oh no, what have they broken now?" Certes (talk) 09:57, 16 August 2026 (UTC)Reply

No, LLM suggestions would be the opposite of useful. They would only reduce the amount of thinking an editor does, leading inexperienced editors to potentially make an edit because the LLM said so. People trust tool output, so if wikipedias own tooling gives them a suggestion, however ridiculous, they might easily do it.

This should be stopped at all costs. TietoTeekkari (talk) 19:48, 20 September 2026 (UTC)Reply

Wouldn't it make more sense to focus on basic copyediting suggestions? I just went to a random article, Mircea II of Wallachia, and requested some copyedits from chatgpt.

Most of these seem like pretty decent suggestions, don't dip into npov/tone danger zones, and are simple enough for new editors to review and handle. ScottishFinnishRadish (talk) 00:14, 20 August 2026 (UTC)Reply

Also, that might be the first time my first random article click wasn't about a sports ball player. ScottishFinnishRadish (talk) 00:15, 20 August 2026 (UTC)Reply

the nice thing about having an insite spell/grammar checker would be less people would use Grammarly (we could tell people using Grammarly to use the insite one instead, similar to how PasteCheck works). I've found asking a chatbot for corrections gives decent results, but obv you can't defer to them (esp. on content) Kowal2701 (talk, contribs) 19:59, 20 August 2026 (UTC)Reply

Why is it a problem that people use Grammarly? RoySmith (talk) 20:02, 20 August 2026 (UTC)Reply

it goes beyond its scope as a spell/grammar checker and rewrites content, and a lot of people using it don't know it's LLM-powered. It's the worst because people will put time and effort into writing, then put it through Grammarly which turns it into slop. We see it at AINB a fair bit Kowal2701 (talk, contribs) 20:10, 20 August 2026 (UTC)Reply

Hi y'all – I'm Peter Pelberg, product manager of the Editing Team. Together with the Machine Learning Team, we've been working on the experimental model-generated edit suggestions. You have raised a range of valid concerns/questions that warrant responses. You can expect those when I'm back online in earnest next week.

In the meantime, I'd like to clarify some aspects of the work and the thinking that's informing it.

First and most importantly: there are no plans right now to deploy LLM-generated edit suggestions as default-on to anyone. The closest thing to a deployment that we've talked about is a potential A/B experiment that would not move forward until we, volunteers and staff, have deemed these suggestions reliable and promising. This commitment is in Phabricator by way of the experiment (T431377) needing T428311 to happen first. For context, T428311 states, "Learn whether experienced volunteers at en.wiki think the experimental suggestions are sufficiently reliable to be shown to newcomers so that we can decide: Will we invest the effort to scale this initial set of LLM-generated MoS suggestions across languages via T431376?"

Now, with regard to what this initial set of 30,000 LLM-generated suggestions is and is not…

1. Who has access to these experimental suggestions? In this initial, experimental phase, these suggestions will only be available to people who A) enable the Suggestion Mode beta feature and B) install this user script. Note: We will soon introduce a setting within Special:Preferences so people interested in trying out experimental suggestions don't need to install a user script to access them.
2. What do these experimental suggestions do? This initial batch contains experimental suggestions for three types of improvements/issues (listed below). Each suggestion will highlight the span of text it is relevant to, offer a generic description of the issue, and crucially, ask the experienced volunteers encountering it to indicate whether you think the suggestion itself is valid/useful or not. And if not, offer an explanation as to why. Said another way: These suggestions do not offer fixes or ask experienced editors to make any.

   

   

3. What are "experimental" suggestions? "Experimental" suggestions are a set of – as I think @Sohom Datta put it well – pre-alpha suggestions. The purpose of them is to evaluate their reliability. Currently, there are a total of 6 experimental suggestions available at en.wiki to people who opt-in to seeing them. 2 of these 6 suggestions are powered by machine learning models; the other 4 are based on deterministic heuristics. You can see the full list here: Special:EditChecks#Experimental checks.
4. What types of suggestions is this LLM generating? This initial batch contains suggestions for simplifying language, rewriting language in a neutral point of view, and adjusting place-based names to follow Wikipedia's geographical guidelines. We are by no means committed to this set of suggestions or to using LLMs in general. We chose this initial set because we found them to be reasonably accurate and assumed they would be relatively low-risk.

Last thing for now: we are aware of the sunk cost fallacy. Thank you for raising it, @Kowal2701. In fact, as I hope the above demonstrates, the whole point of developing experimental suggestions, and making them available in production to experienced volunteers who have explicitly opted into them, is as @Chaotic Enby described: to assess whether they're reliable. Ones that aren't, we will abandon. The ones that are, we'll work together to figure out how and to whom to make them available.PPelberg (WMF) (talk) 05:48, 16 August 2026 (UTC)Reply

as […] “1.” and “3.” demonstrate. This is perhaps not the most important point, but it is bugging me: were the numbers meant to all be “1.”? Comment struck due to this being fixed; apologies. Cheers, 𝔰𝔥𝔞𝔡𝔢𝔰𝔱𝔞𝔯 ^{(𝔱𝔞𝔩𝔨)} (any/all) In solidarity. 06:08, 16 August 2026 (UTC)Reply

Thanks for responding now and letting us know you don't have time to fully engage immediately, hopefully you will have time to go through all the comments above later in the week as you say. As the process of contingent experiments has been brought up again, perhaps it would help to answer the experimental question plainly, because it can be very easily answered without a complicated multi-month experiment. The suggestions are not reliable, and definitely not reliable enough to show to newcomers (not that this is the only metric that could be considered). To look further, re We chose this initial set because we found them to be reasonably accurate and assumed they would be relatively low-risk, the linked section lacks an assessment of accuracy or risk. Part of the issue here is that a simple glance is all that is needed to identify issues with many the suggestions, so it feels like even that is not being done before these are farmed off to volunteers. There is a mismatch in understanding somewhere. CMD (talk) 06:18, 16 August 2026 (UTC)Reply

Part of the issue here is that a simple glance is all that is needed to identify issues with many the suggestions, so it feels like even that is not being done before these are farmed off to volunteers.

Honestly can't put it any better than that. I do appreciate the response and understand it is the weekend. Gnomingstuff (talk) 07:18, 16 August 2026 (UTC)Reply

Tbh this is a systemic issue to do with WMF governance rather than a reflection on anyone here, where developers and idea labs seem to typically be distant from the projects and communities, and everyone at both ends is left straining to compensate for this. Phab being largely open to the public goes a long way, but I wonder if there could be something like a WP:VPI on meta for developing/screening ideas (obv staff's medium is meetings and private convos etc. and idrk what currently happens, but something like "John and I were talking yesterday about this, X is a potential solution" would be good to get some feedback at the earliest stage). I've previously suggested that staff should be encouraged to do a little bit of editing each week at any wiki they choose (and reduce hours/week a bit for this while keeping salaries the same), but it risks the WMF's status as a 501(c)(3) organization (I was told) Kowal2701 (talk, contribs) 07:57, 16 August 2026 (UTC)Reply

I don't really know that it has anything to do with governance in this case. The issue would be the same regardless of whether the development process was public or semi-public or completely closed off; it's a classic, perennial issue of workplaces. The issue being -- and I am really trying to be as polite as possible here -- that QA is either not being done, or not being done correctly.

From what I understand, the suggestions were assessed for quality via AI, and then someone seems to have reviewed 15 suggestions manually. Unless there's more internal discussion (there probably is, this stuff is nearly impossible to follow), that seems to be the whole QA process.

Meanwhile, I actually have worked in QA. For a task like this, we would be given a spreadsheet like the ones here and would review every single line individually. We caught a lot of errors this way, the final product was better as a result, and it only took a day or two of work at most. I do not feel this is an unreasonable amount of work to be expected from a professional organization. Gnomingstuff (talk) 17:52, 16 August 2026 (UTC)Reply

A stage of basic competitive fault-finding, annotating a large sample by hand to flag and classify problems, goes a long way. Your immediate reflex to do this as the first response above demonstrated this nicely.

If an originating team works in public to catch and patch issues in that way, before others run into them, with high standards for accuracy, that would set discussions like this off on a better foot. – SJ + 00:23, 19 August 2026 (UTC)Reply

Thank you. Sorry, I didn't know who was in the team working on it, I trust all these people. Kowal2701 (talk, contribs) 06:51, 16 August 2026 (UTC)Reply

I oppose any A/B tests of this feature. It is antithetical to what Wikipedia should be, and is not something the Wmf should attempt. Suggestions sent by some central, biased, singleminded entity diminishes the wide variety of input, style, viewpoint, ... that makes the richness of the community. Wikipedia ia a beacon of human diversity, an LLM is the opposite of this. Fram (talk) 06:58, 16 August 2026 (UTC)Reply

In contrast to such hyperbolical rhetorical flourishes, the community has already been widely using AI suggestions sent by some central, biased, singleminded entity controlled by WMF for over a decade, in form of ORES (or now the "revert risk" models). These AI suggestions (on whether to revert another editor's changes as vandalism) are made available to any editor at Special:RecentChanges even though they might be seen as more consequential on average than those of the new tool under development here, and have a fairly high error (false positive) rate. I have personally implemented many thousands (probably tens of thousands) of those AI suggestions over the years, and survived skipping many more mistaken suggestions.

I don't want to entirely dismiss concerns about control by the Foundation though. In case of these vandalism detection AI models, the more recent work on them seems to have been dominated by decisions and aims of the WMF Research department that may not entirely align with the community here (for example, they seem to have foregone possibly substantial quality improvements for English Wikipedia in favor of language equity, and implemented their own conceptions of "fairness" with regard to IP editors).

The main employee behind the success and broad community acceptance of the original ORES left WMF years ago, and his parting recommendations for "Community-centered Evaluation of AI Models on Wikipedia" do by and large not seem to have been taken up by WMF.

Regards, HaeB (talk) 08:36, 16 August 2026 (UTC)Reply

As I said earlier, keeping the prompts (and the pipeline in general) publicly available and enabling each community to tweak them would go a long way in assuaging these concerns. Some competence is required to maintain these tools, but there is no reason not to make prompts and benchmarks transparent. Alaexis_¿question? 10:50, 16 August 2026 (UTC)Reply

The system prompts used for the model appear to be published on gitlab here according to the mw:VisualEditor/Suggestion Mode/Model-generated editing suggestions#Research findings. Not sure which size Gemma gemma4:latest is, maybe the E4B? I would be interested to know if other models were evaluated, for example Nemotron 3 Super is a similarly sized model to gpt-oss:120b, but a bit newer. Mistral is dense so might be a bit slow to run. In any case, there are many newer models in the same weight class, surely it wouldn't be that hard to generate say ~1000 from each of them to see if anything is clearly better, given the only thing that has been modified is the system prompt? Alpha3031 (t • c) 12:22, 16 August 2026 (UTC)Reply

Thanks. Going off of that, it appears that no reference information is given, which is not surprising looking at the NPOV suggestions, and that no actual Wikipedia policies/guidelines are included in the prompt. LittlePuppers (talk) 15:24, 16 August 2026 (UTC)Reply

A successful model would likely go beyond a prompt alone and be equipped, at the very least, with retrieval-augmented generation capabilities in order to access the text of the policies/guidelines themselves, and, in the case of NPOV, ground its answer in available reliable sources. I don't think that would be enough for NPOV specifically (there is still too much of a black-box effect in the model's weights, that won't be fully offset by prompting or sourcing), but this is to say that the current approach is far from optimal. Chaotic Enby (in solidarity · talk · contribs) 15:46, 16 August 2026 (UTC)Reply

The particular challenge is, I think, that there seems to be a trend to use increasingly sophisticated models to push increasingly challenging tasks increasingly to newer editors. LittlePuppers (talk) 15:33, 16 August 2026 (UTC)Reply

Regarding point 4., community feedback makes it pretty clear that anything regarding NPOV or tone will not be seen as low-risk, and might be interpreted as an attempt by the WMF to influence editorial decisions. I believe it would help the prospects of the experiment to commit to follow the emerging consensus, which is clearest on this specific aspect.On a broader level, I will reiterate my suggestions on Phabricator on working with the community:

[...] the scope and timeline of any planned community rollout, and the importance of working with the editor community and respecting a future consensus on whether to deploy it, should be made as clear as possible to avoid a repeat of Simple Summaries.

Chaotic Enby (in solidarity · talk · contribs) 12:31, 16 August 2026 (UTC)Reply

I'm with the rest of the group in saying that the NPOV suggestions are high risk and difficult to get right. I'm excited about trying to figure out if a good suggestion model can be developed for simplifying language. There is a large body of research showing that Wikipedia's science content is often way and way too complicated and linguistic complexity is an important part of that. We've had discussions on Wikipedia about using heuristics (sentence lenght, number of syllables per word), which is contentious. Perhaps an AI model can do this better than a heuristic, as it can distinguish between a long sentence with an easy sentence structure and an impenetrable long sentence. In solidarity, —Femke (talk) 🐦 15:27, 16 August 2026 (UTC)Reply

@Femke I have tried both AI models and conventional readability tests like Flesch–Kincaid and this is an area in which current AI tech cannot and should not be used. And my conclusion was that conventional readability tests also suck.

NPOV is another area where AIs cannot and should not be used.

We can use AI to find typos, but fixing them actually requires an experienced Wikipedian. Polygnotus (talk) 15:33, 16 August 2026 (UTC)Reply

I too have plenty of experience using AI for this purpose and find that they can be used successfully in identifying overly difficult text and decently enough for suggesting alternatives if one pays attention to subtle changes in meaning. As these edit checks are only about identifying overly complicated text and because we have an abundance of low-hanging fruit in this area, I'm confident that a tool can be developed. The big question is if within the limits of a certain budget, it can become good enough, and to what extent it attracts the right editors to fix it. In solidarity, —Femke (talk) 🐦 15:40, 16 August 2026 (UTC)Reply

One of my common complaints at FAC (especially for scientific articles) is choppy writing style (sort of the opposite problem from overly difficult text). I often advise authors that they should try using some of the LLM tools to make suggestions for how their writing could be improved. I don't know how well this advice is received. I also don't know how much it is used in the intended way of offering suggestions vs just copy-pasting the output into their article; that would be an abuse of the tool, but people abuse tools all the time and the fact that they do so doesn't mean the tool is bad. RoySmith (talk) 15:43, 16 August 2026 (UTC)Reply

find that they can be used successfully in identifying overly difficult text and decently enough for suggesting alternatives if one pays attention to subtle changes in meaning. Subtle changes in meaning convert text supported by reliable source(s) to text not supported by reliable source(s).

I'm confident that a tool can be developed Yeah, developing a tool is the easy part. The hard part is ensuring it is a net positive.

The big question is if within the limits of a certain budget The WMF is not the right party to create such a tool because they are unfamiliar with the challenges Wikipedia editors face and because they have a tendency to waste a lot of time and effort on creating shiny new tools while neglecting decades of tech debt.

The good news is that volunteer devs can do it, but they will run into the problem that it won't work as I explained above. Polygnotus (talk) 15:46, 16 August 2026 (UTC)Reply

Some teams are unfamiliar with the editing challenges. Other teams have a track record of listening, or have editors in their team. The editing team is an example of the latter. For instance, when people at enwiki and dewiki asked for better VE performance recently, the team managed to fix this tech debt really rapidly.

I'm very willing to help test this part of the tool. I have experience with editors doing this well and less well. In solidarity, —Femke (talk) 🐦 15:58, 16 August 2026 (UTC)Reply

@Femke You can try WP:SCRIPTREQ, the people there are a lot more agile than the WMF is.

Please ping me if you have something I may be able to help test it. Polygnotus (talk) 16:02, 16 August 2026 (UTC)Reply

Tbh I still think we should look at incorporating SEWP articles here as 'simple summaries', but that's a different discussion Kowal2701 (talk, contribs) 15:47, 16 August 2026 (UTC)Reply

I wouldn't assume they'd want us to. Simplewiki is a tiny wiki with just a few active people. If we incorporate their content their vandalism rates will go through the roof. Polygnotus (talk) 15:49, 16 August 2026 (UTC)Reply

They'd also get more good-faith editors though! What I'm thinking is that SEWP still exists as a separate wiki, we just have an opt-in feature for displaying one of their articles behind a button. IIRC @Ferien was sort of open to the general idea (may be wrong) Kowal2701 (talk, contribs) 15:53, 16 August 2026 (UTC)Reply

Adding a CTA, if we get consent from the Simplewiki regulars, may be a good idea. Polygnotus (talk) 16:04, 16 August 2026 (UTC)Reply

Yeah, I am personally quite open to the idea, though I am not too sure what our community as a whole would make of it at the minute. --Ferien (talk) 21:04, 17 August 2026 (UTC)Reply

What kind of typos is AI fit to resolve that WP:AWB is not? Czarking0 (talk) 20:13, 16 August 2026 (UTC)Reply

One thing I'm having in mind are typos that depend on the context to make sense of them (or to whether there is a typo to begin with). Chaotic Enby (in solidarity · talk · contribs) 20:19, 16 August 2026 (UTC)Reply

FWIW, the fixes in Special:Diff/1369578064 were all suggested by Claude. I imagine most of them would have been caught by other tools, but I was impressed by the flagging of Freilberg. I don't remember exactly what it said, but the gist was that it spotted that I had "Peter Freiberg" in one place and "Peter Freilberg" in another. It figured out that these were probably referring to the same person and while it didn't know which was wrong, it assumed one of them was, and left it up to me to figure out which. RoySmith (talk) 20:24, 16 August 2026 (UTC)Reply

And I note the firsts of those fixes is changing a direct quote in a way that is inconsistent with the source. Sure, you can probably justify that (MOS:QUOTE allows minor typographic things to be silently corrected) but I don't think that's something an AI should be recommending in any way. * Pppery * _{(alt) in solidarity} 20:49, 16 August 2026 (UTC)Reply

So, you're saying the fix is correct, and if a human had suggested it you would agree with it, but since an AI suggested it there's a problem? RoySmith (talk) 20:56, 16 August 2026 (UTC)Reply

I'm saying it might be correct (not that it is correct), but determining whether it is is a judgement call I don't want an AI to make. * Pppery * _{(alt) in solidarity} 21:22, 16 August 2026 (UTC)Reply

The AI didn't make the judgment call. It just brought this to my attention and I made the judgement call. That's why my name is on the diff. I'm really not seeing the issue here. I made an error, a tool alerted me to it, and I fixed it. How is this a problem? RoySmith (talk) 21:36, 16 August 2026 (UTC)Reply

The funny thing about using LLMs here is that they start working against each other. The default behavior of AI trying to "rewrite articles into formal encyclopedic tone" is to undo any text simplification (and to do so poorly, for instance replace "is" with "serves as", "uses" with "utilizes", etc). The "text simplification" category here, however, seems to be basically a catch-all. "Simplification" suggestions here range from stuff like Correct the misspelling of the player's first name from "Russel" to the proper "Russell" (which isn't even correct!) to "break up these sentences."

There's also the problem that telling someone to Rewrite the sentence for clarity and smoother flow is not actionable to the majority of people: if someone doesn't know how to copyedit then they don't know how to do that, and if someone does know how to copyedit they don't need those vague directions. It's like prompting people as AIs. Gnomingstuff (talk) 18:23, 16 August 2026 (UTC)Reply

(Ironically, the LLM prompt that judged suggestions reads in part You are a strict reviewer. Your job is to find flaws, not to be nice. Really weird feeling to be envious of an LLM, its opinion certainly seems to be taken more seriously and its tone is given much more leeway.) Gnomingstuff (talk) 20:42, 16 August 2026 (UTC)Reply

Who on earth outside the group of people responsible for this is going to read Gnomingstuff's report and consider those suggestions to be "reasonably accurate"?  — Hex • talk 16:02, 16 August 2026 (UTC)Reply

Pre-alpha or not, the WMF shouldn't be experimenting with ways to funnel freeform model suggestions to editors to begin with. Machine models should never be allowed to so directly influence the contents of the project, as even if the suggestions are individually found to be valid and reliable, there will still exist overall biases. An LLM will favor certain sources, certain topics, certain sides. This will be reflected in what suggestions are and are not made, and editors evaluating and implementing individual suggestions will be entirely blind to any larger systemic issues they would be enabling.

Humans have issues with bias too of course, but this can be counteracted on an individual level by self-awareness of this fact, and on a group level by the diversity of our views. A monolithic model has neither, it predicts tokens. fifteen thousand two hundred twenty four (talk) 21:45, 16 August 2026 (UTC)Reply

"Humans have issues with bias too of course, but this can be counteracted on an individual level by self-awareness of this fact - Citation needed: "making people aware of their bias doesn't do anything to mitigate it." Levivich (talk) 22:45, 16 August 2026 (UTC)Reply

Citation: I made it up from first principals and subjective experience, the preprint will be out soon.^[Humor]

I feel entirely comfortable claiming that editors who operate with awareness of their own potential biases will take steps to mitigate them in this structured environment where WP:NPOV serves as a strong guiding force. If you find this unpersuasive, so be it, it is human to disagree. fifteen thousand two hundred twenty four (talk) 23:58, 16 August 2026 (UTC)Reply

The operative word here is can. As human beings that are capable of independent thought and self awareness, we can choose to examine our own biases and seek out information and experiences to change how we think about the world around us and the assumptions we make. Large language models are capable of exactly none of those things, because of the very simple fact that they are computer programs. Claude is exactly as capable of herself awareness as MS Paint is of having an independent thought.

Not every person makes the conscious choice of examining their baises, or is even fortunate enough to exist in a socioeconomic situation to even be able to, but that is beside the point ‑‑gurkubondinn 00:26, 17 August 2026 (UTC)Reply

"Research from Harvard found that the effects of personal interventions such as awareness raising at a personal level are positive, but short-lived."

"And the worst method, the one that actually has no effect at all, is to tell people to be good people, to be egalitarian, and so on. ... It is easy in the sense that it last for a short period of time, but it won't last very long. ... Now, when young people encounter this result, when they see that, yes, they were able to make change, but the change doesn't last, they get very sad, because they want a better world. And I'm not at all sad about that. ... So our brains change, our minds change, associations move around, but they always will gravitate to whatever is your cultural default. And so to bring about actual change, society around us has to change, and then we will move, and then the default will be a new default."

LLMs are biased because humans are biased. Because they're trained by humans, and humans train their biases into the machines. We are no more able to cure bias in machines than we are able to cure it in ourselves. There are plenty of ways in which human intelligence is superior to machine intelligence, but lack of bias isn't one of them (in either direction). Levivich (talk) 02:57, 17 August 2026 (UTC)Reply

Absolutely, I agree with you. That's why the models can never be "neutral" or "unbiased". My point was just that I think you misunderstood 15224's reply, people have the capability to examine and be aware of their biases, but computer programs don't. It's not automatic and it doesn't happen for all people (for various reasons), but we have the ability to examine our own biases because (unlike computer programs) we are conscious beings. It's a bad comparison is what I'm saying, and it anthropomorphises computer programs (that were created by biased people). I have a beef with whoever it was that decided to wrap LLMs in chatbot interfaces. ‑‑gurkubondinn 11:06, 17 August 2026 (UTC)Reply

We're looking forward to getting more deeply into this with you all this week. Before that, I wanted to express gratitude to y'all for the perspectives you are sharing. From the concerns about transparency and community control over models of this sort to issues with specific suggestions you're encountering, please keep the feedback/questions/concerns/etc. coming.

And for anyone who is interested in seeing what these suggestions look like in practice, please do the following...

TRYING EXPERIMENTAL SUGGESTIONS

1. Ensure you have the Suggestion Mode beta feature enabled
2. Enable "experimental" suggestions by pasting the following snippet into your common.js: mw.loader.load( 'https://meta.wikimedia.org/w/index.php?title=User:DLynch_(WMF)/alwaysbesuggesting.js&action=raw&ctype=text/javascript' );
3. Open a page in VE that has one of the 30,000 suggestions available. E.g. https://en.wikipedia.org/w/index.php?title=List_of_examples_of_Stigler%27s_law&veaction=edit

Note: this week, I'm going to see if we can share a spreadsheet so you can see all of the model-generated suggestions in one place rather than having to tap around the wiki looking for them. PPelberg (WMF) (talk) 00:59, 17 August 2026 (UTC)Reply

I wrote a small script so you can view the list of suggested edits on an article without needing to enable the beta feature, switch to VisualEditor, or add that line to common.js to enable "experimental" suggestions: User:DVRTed/sandbox/edit-suggestions.js. — DVRTed (Talk) 03:06, 17 August 2026 (UTC)Reply

The spreadsheet would be helpful, if only because I don't know which of the csvs is the "real" one. In general I think this kind of thing is much easier to review in spreadsheet form than article-by-article. Gnomingstuff (talk) 04:18, 17 August 2026 (UTC)Reply

@Gnomingstuff: what you described makes total sense to me.[i][ii] I've checked in with engineering and it turns out that compiling this list will take a bit of time. Assuming nothing unexpected turns up, you can expect me to return here with a link to a CSV you (and everyone else here!) can review before this week is over.

---

i. Knowing, definitively, what suggestions we need y'alls expertise in reviewing and being able to differentiate those from earlier iterations that we've since discarded.

ii. Seeing all of the suggestions in one place so that you don't have to hunt them down yourself. PPelberg (WMF) (talk) 22:49, 17 August 2026 (UTC)Reply

Is the intent that the U/I would just present these suggestions to the user and let them edit the article themselves if they opt to accept the suggestion? Or is the idea to have a "Make this edit" button that the user could just click? The reason I ask is that if it's the later, it would make sense to include some machine-readable marker in the wikitext (I'm thinking an HTML comment) identifying the source of the inserted text. And/or have a log of such changes. The idea is to make it easier for somebody to audit the performance afterwards. RoySmith (talk) 23:04, 17 August 2026 (UTC)Reply

The latter would fail WP:NOLLM, so that would be a complete nonstarter. The former is not great either. ‑‑gurkubondinn 23:15, 17 August 2026 (UTC)Reply

WP:NOLLM says "Editors are permitted to use LLMs to suggest corrections to their own writing, and to incorporate them after human review. This is limited to spelling, punctuation, capitalisation, grammar, and other simple mistakes." So this would indeed be a starter in those cases. RoySmith (talk) 23:22, 17 August 2026 (UTC)Reply

The suggestions that Gnomingstuff went through go far beyond the very narrow exception in NOLLM. And this exception only applies to [an efitor's] own writing. ‑‑gurkubondinn 23:44, 17 August 2026 (UTC)Reply

The latter would be impossible in the current implementation, anyway, the LLM is not prompted to provide an actual change and the suggestions are usually just stuff like "The sentence is long, contains a grammatical error, and could be expressed more clearly." Gnomingstuff (talk) 05:09, 18 August 2026 (UTC)Reply

@RoySmith: Good question. If/when we (staff + volunteers) come to think these suggestions are reliable and useful, the intention would be to offer a suggestion that would contain the following:

1. A description of the issue and type of fix that is needed. Both of which will need to be generic enough to make sense across the contexts it might appear within while at the same time being concrete enough for the people encountering it to know how to start on the path of acting on it. So for the "Simplify language" suggestion, it might be something like, "Readers might find this text difficult to understand. Try rewriting this using shorter sentences and plain language."

2. A link to the local policy/guideline the suggestion originates from. This serves both as an opportunity for people encountering a suggestion to learn more and also a way to ensure that suggestions are grounded in project consensuses and conventions.

3. Two actions: one action to Dismiss the suggestion and along with it, a way to express why someone has elected that choice. And a second action – maybe we'd label it Rewerite? – that when tapped would A) focus someone's cursor into the span of text the suggestion thinks there is an issue within and B) cause the article text in question to enter a "revising state" so people are clear about where exactly their focus is needed (see screenshot below). From there, the responsibility would be on the person acting on the suggestion to decide what to fix and how to fix it. Said another way: there are NO plans for these suggestions to make fixes with the click of a button let alone to describe specific solutions.

If you'd appreciate a more concise answer, what @Gnomingstuff described here is spot-on. PPelberg (WMF) (talk) 05:32, 18 August 2026 (UTC)Reply

@PPelberg (WMF): So how would you cram enough information in such a tiny area to give the person all the information they need? Are you aware that many guidelines are like 7k words? If you give a newcomer a link to WP:NPOV, that is obviously not enough to have them actually be able to judge the neutrality of an article if they do not have relevant experience and knowledge of the field. And what will happen when inevitably people complain that edits are not improvements? Will this just WP:BITE newcomers even more? Polygnotus (talk) 05:40, 18 August 2026 (UTC)Reply

@Polygnotus: great spot. The questions you're asking sit at the very core of this project.[i]

We seem to be aligned in thinking[ii] that some suggestions may be more harmful than helpful to show to newcomers. They're likely, as you alluded to, too complex and experience-dependent to distill down into a relatively small piece of guidance. If we don't account for this, we could lead newer folks into publishing edits that experienced volunteers revert or respond to with hostility. This could in turn drive these potential contributors away.

And while I don't think we can know for certain which suggestions those are in advance, I think we (staff and volunteers) are develpoiong a pretty good sense that conversations like this one are helping us refine.

I also think we'll learn a lot over time by trying things out, and there are a few features we've put in place to help:

Experimental suggestions: suggestions can be enabled as either default-on or experimental. The latter means that people who have explicitly enabled the soon-to-be-available setting have the space to safely experiment with suggestions. Through that testing, they can decide who—if anyone—an experimental suggestion should be shown to by default.

Tags: all edits in which someone sees and/or acts on a suggestion are tagged. You can see these in action by filtering Special:RecentChanges for Edit Suggestion seen or Edit Suggestion used.

On-wiki configuration: volunteers can independently decide the minimum number of edits someone must have published in order to see a suggestion. If you visit Special:EditChecks and look at the link suggestion, you'll see that volunteers have set the minimumEditCount value to 1000. This means the suggestion will only be shown to editors who have made ≥1,000 edits.

The idea is that, together, the above will let us see the kinds of edits these suggestions lead folks to make in practice and, with that, decide whether, how, where, and to whom they're shown.

How does this sound to you? What, if anything, do you think we might be missing or misunderstanding?

---

i. I hear the questions you're asking as something like: "How might we translate a great deal of nuance and complexity into a format that is simultaneously 1) succinct and simple enough that newcomers will engage with it and 2) explanatory enough that in doing so newcomers will be equipped with the information and know-how they need to act on them in ways experienced volunteers see as constructive?"

ii. Please correct me if I've misinterpreted what you've said PPelberg (WMF) (talk) 05:32, 19 August 2026 (UTC)Reply

I would urge you to please stop developing this feature. It seems like any version of this feature will be detrimental to wikipedia.

A suggestion from an LLM, a suggestion from an official wikipedia tool, will easily be read by an editor as trustworthy, when we know it is not. LLM usage offloads critical thinking to software, meaning the editor is doing less of that themselves. This alone will have a negative impact on quality. And even grammar edits can change the meaning of a sentence, so this is just a non-starter in general.

Please do not pursue this track. TietoTeekkari (talk) 20:00, 20 September 2026 (UTC)Reply

I'm pessimistic about this, but I have a very high bar for pessimism/hopelessness to stop me from supporting a low-stakes experiment. Giving this tool to experienced users to try out seems like one of those low-stakes experiments worth trying, in case there's a way to make it work (again, I'm pessimistic, but possibly with certain constraints regarding task and topic?).
But also, just to put a finer point on something, because I think it does good to repeat it now and then: there's the worry about the quality of these suggestions, but there's also the worry about, for lack of a better word, branding. The branding that the WMF has begun using, "knowledge is human", is an echo of a popular sentiment here. At a time when absolutely every company and every project is cramming in as much AI as possible, Wikipedia is mostly headed in the other direction. That's a good thing IMO, but as a result, any pitch someone has for an LLM-based tool is evaluated with a handicap applied. If you'd otherwise be graded on a scale of 1-10 where 1 is harmful and 10 is helpful, start by subtracting 2 or 3 for "we don't want to be associated with that" (or, alternatively, "we see these as detrimental by default"), and it needs to be really useful to get a critical mass of people behind it. But no complex tool starts its life as an 8+ on that scale, I don't think, so super-low-stakes tests are IMO a good way to start working towards it. FWIW. — Rhododendrites ^talk \\ 22:06, 16 August 2026 (UTC)Reply

I really appreciate and agree with the branding note Rhododendrite gives here. Best, Barkeep49 (talk) 22:48, 16 August 2026 (UTC)Reply

I know people love to make fun of AI hallucinations, so I couldn't resist posting this fun map (4:38 in the video). Strange spellings aside, Hoboken, NJ has gotten transported to the midwest, Chicago is on the Pacific coast, and Boston has been relocated to Colorado. I want some of whatever it's smoking. RoySmith (talk) 02:12, 17 August 2026 (UTC)Reply

Also available in Africa. I think I'll stick to the Commons maps. Certes (talk) 14:30, 17 August 2026 (UTC)Reply

These LLM map-fails are legion. My take away from these examples is that they are not only vivid examples of LLM hallucination, and thus LLM unreliability, but they are also vivid examples of the unreliability of humans, because each one of these published hallucinations is only possible because humans obviously failed to check the work--to even look at the maps--before publishing. Humans are unreliable: an important lesson for any crowdsourced project. Levivich (talk) 14:57, 17 August 2026 (UTC)Reply

I don't know about this particular channel but loads of them are (almost) fully automated with no human in the loop. Polygnotus (talk) 15:39, 17 August 2026 (UTC)Reply

Then again, aside from quite a few mistakes, have you all seen the "Chloe vs History" channel on youtube? Quite the advancements in AI lately. Some of the advancements have been made on this channel, which should probably have a Wikipedia page. Randy Kryn (talk) 15:42, 17 August 2026 (UTC)Reply

That Africa one is from a US State Dept presentation. That ain't no YouTube clickbait, that's supposedly professional humans at work. You'd think they'd have looked at the slides before making their presentation. And, I'm speculating here, but I bet multiple humans were involved in that, because when the US State Dept gives a presentation at an int'l conference, I don't think it's just one person creating and making the presentation all by themselves. Maybe. But either way: evidence that at least sometimes, even professional humans presenting at an int'l conference obviously don't bother to check their work. I guess my point is: what makes LLMs unreliable isn't just that they hallucinate, it's that humans won't catch it because sometimes they don't even bother to look. Another well-known example is lawyers submitting briefs to courts with hallucinated citations -- literally licensed professionals in the performance of their profession. So this isn't a problem limited to youtube clickbaiters or "kids on the internet," even trained and licensed professionals, even on a global stage, succumb to laziness. At least the lawyers get fined for it -- now there's a new fundraising channel for the WMF: fine editors for publishing hallucinations on-wiki! Levivich (talk) 16:15, 17 August 2026 (UTC)Reply

At least with hallucinations, they're usually immediately obvious to anybody who bothers to look. This particular example is clearly YouTube clickbait. The bottom-feeders who produce these things don't give a whit about accuracy, just that they can churn out some mildly entertaining garbage videos that collect likes and shares and other revenue-producing metrics. But the fact that people are willing to abuse a tool for their own commercial benefit doesn't mean that the tool is inherently worthless. People abuse wikipedia in all sorts of ways (spam, SEO, reputation management, advertising, etc). Does that mean we should shut Wikipedia down? RoySmith (talk) 15:49, 17 August 2026 (UTC)Reply

PS, it doesn't take AI to generate garbage maps. Some people are able to do it all by themselves with nothing more technologically advanced than a sharpie. RoySmith (talk) 16:22, 17 August 2026 (UTC)Reply

Sure, and wikipedia had hoaxes and plenty of innocent misinformation long before LLMs became popular, but the problem, of course, is one of scale: LLMs make this sort of thing like 100x more common than before. It used to take hours to write a good hoax on wikipedia, now it takes minutes. Levivich (talk) 16:30, 17 August 2026 (UTC)Reply

"Chiiicago". AAND it's in multiple places at once. Ladies,Gentlemen and enbies, The Chicago hivemind. Starlet 01:43, 25 August 2026 (UTC)Reply

• No AI on Wikipedia and this should be extended to article talk pages, Wikipedia discussions, visible editing suggestions, or draft pages. Too many work-arounds and exceptions have been allowed and these issues and concerns could go on for years. There should be some good faith edit reverts for AI edits made in mainspace while the confusion existed but maybe end the confusion from here-on-in with a blanket ban on AI. Thanks (darn it, the word "thanks" was aided by AI). Randy Kryn (talk) 15:06, 17 August 2026 (UTC)Reply

  I'd support that. I've had enough with AI turning everything to shit, and clearly WMF can't take no for an answer. Tercer (talk) 15:55, 17 August 2026 (UTC)Reply

  I'd say that Polygnotus should be allowed to play around with AI because Polygnotus actually understands its limitations and assumes full responsibility for any edit made with their account.

  The WMF is a different story unfortunately. Polygnotus (talk) 15:59, 17 August 2026 (UTC)Reply

  Does Polygnotus always refer to himself in the third person? RoySmith (talk) 16:03, 17 August 2026 (UTC)Reply

  Polygnotus does not. But Polygnotus could! Polygnotus (talk) 16:05, 17 August 2026 (UTC)Reply

  AI, or LLM? Cluebot, for example uses machine learning (AI but not LLM) to great benefit with remarkably few errors. Certes (talk) 16:36, 17 August 2026 (UTC)Reply

  Generative AI is what bothers me, irrespectively of the underlying technology. I'm fine with cluebot because it doesn't generate content, so there's no risk of poisoning Wikipedia. I would still be fine with it if it were LLM-powered (even if that would be completely pointless). Tercer (talk) 17:46, 17 August 2026 (UTC)Reply

  I would absolutely support a ban on AI (LLMs) for (direct or indirect) content generation or suggestion. Ita140188 (talk) 19:12, 17 August 2026 (UTC)Reply

  I would definitely support this. Cheers, 𝔰𝔥𝔞𝔡𝔢𝔰𝔱𝔞𝔯 ^{(𝔱𝔞𝔩𝔨)} (any/all) In solidarity. 19:14, 17 August 2026 (UTC)Reply

  I would support this on principal, at least until the AI industry stops being complete garbage. —Leaf.Sheap ⇖ /.°°.\ ⇗ (They•Them) 21:18, 17 August 2026 (UTC)Reply

  I would support aswell. I don't think (generative) AI has a place on Earth at all, let alone on wikipedia (which has always been guided by the principles of human knowledge and effort). TheDowningStreetCat (talk) 23:41, 17 August 2026 (UTC)Reply

  100% support. I fail to see how AI slop would actually help the project in any conceivable way. All I’m seeing is negatives… lots of them. 296cherry (talk) 18:29, 20 August 2026 (UTC)Reply

  What would you think of suggestions more along the lines of what I outlined here? ScottishFinnishRadish (talk) 18:32, 20 August 2026 (UTC)Reply

  I would support a ban on LLMs to stop similar things from happening in the future. TietoTeekkari (talk) 20:02, 20 September 2026 (UTC)Reply

• Right now one of our biggest strengths is that we are a large, human written, generative AI free chunk of knowledge. My crystal ball isn't working, so I can't tell you if AI will be good enough to do something cool with Wikipedia in the future. That said, Wikipedia is one of the last places which should incorporate AI because it will give up our unique advantage. Proposals like this from the WMF are not sufficient, are not close, and make me want to use blanket statements to convince the WMF not to keep trying to shoehorn AI into things over and over again. Tazerdadog (talk) 16:59, 17 August 2026 (UTC)Reply

  I feel like it should be pointed out, again, that this is not a proposal. It's an experiment. "Learn whether experienced volunteers at en.wiki think the experimental suggestions are sufficiently reliable" is what they're doing. I think Gnoming helpfully answered that question. Levivich (talk) 17:58, 17 August 2026 (UTC)Reply

• This would be an overreaction. User:Polygnotus and many others have been building various LLM-powered tools, including ones that are used to detect LLM edits (User:Fermiboson/AIlog) and to fight vandalism (Wikishield - LLM is optional). There are more than one hundred editors using these tools. We do want WMF to experiment and implement the best ideas. Alaexis_¿question? 07:06, 18 August 2026 (UTC)Reply
• A full ban does not make sense. We already have a range of community tools that do cool things with Wikipedia using AI. In particular I want the best available tools for review, including those that take advantage of AI for trainable pattern matching and classification. That includes anything that helps with slop-detection, edit checks, citation checks, page linting, and vandal-fighting. This experiment falls under review... with a higher bar for quality I can see a range of suggestions being useful, particularly after a few rounds of focusing on quality. – SJ + 01:07, 19 August 2026 (UTC)Reply
• I would support a ban on generative AI touching any Wiki content regardless of whether LLMs improve. JoelleJay (talk) 11:30, 22 August 2026 (UTC)Reply

Thank you all for keeping the conversation going, and thank you to Gnomingstuff and others for taking the time to look through outputs. @PPelberg (WMF), @SSalgaonkar-WMF, and I have read the whole thread and tried to pull out some of the most important things we heard and questions being asked. Peter and Sucheta, please add if I missed anything (well, for that matter -- anyone please tell me if I missed anything). I'm just going to list these questions/topics for now, and we'll keep working through them and hopefully you'll keep discussing with us (there are many of you and not as many of us!) This is to build on what I posted above and what Peter posted above.

Firstly, I think we're on the same page about something especially important: we're not going to deploy LLM-backed suggestions to anyone unless this community is supportive of it. And if they do become deployed, they'll be configurable via Special:EditChecks the way existing checks and suggestions are (i.e. community could decide who gets to see them, change what they say, what articles they show up on, or turn them off). I hope that in projects like these, we at WMF are bringing capabilities to volunteers so that they can produce the kinds of checks and suggestions that help both newcomers and experienced editors get the most good wiki work done with the least amount of drudgery. There is also a question of whether these suggestions would be a vehicle for content suggestions -- the answer there is no, we are not designing these to propose text to the user; rather they point out places the user should look and what they should look for. The LLM explanations that Gnomingstuff pointed out initially are in those files as a way for us to understand internally why the LLM is making the suggestion; they would not be shown to editors. (But I know there is a more subtle question here -- when just pointing out a sentence that needs attention in some way exerts that sort of influence).

In that vein, I wanted to say that this current project is more about figuring out what it might be like for checks/suggestions to be generated via LLMs, than about what those exact types of suggestions are. If NPOV is not a good one to pursue (volunteers here have given many good reasons why NPOV is particularly tricky), then we should put that one down in favor of simpler ones to try out. (Worth noting that checks and suggestions are being generated via a bunch of other ways, too, like simple rules, text match, simple models, etc.)

Also just a quick nomenclature thing:

• Checks: this refers to edit checks that react to what the editor is doing at that moment in the editor. e.g. they paste in a blob of text from ChatGPT, and the check pops up right then and says, "Please avoid copying text from other sources".
• Suggestions: this refers to suggestions for improvement that have been pre-calculated and are waiting to show up when someone clicks Edit, e.g. they open up the editor, and there is a box that says, "This link appears more than once in this section." Right now, Suggestion Mode is available on English Wikipedia as a beta feature, and suggestions are available in the feed on Special:Homepage.

You can see all the checks and suggestions and their statuses at Special:EditChecks (note that "experimental" checks are only available to people who have a specific user script installed).

Sorry -- this post is getting long. But on to the list of questions we want to be able to talk about here, raised in the thread above. This list of questions is not something that we just want to provide answers to and then expect that everyone will agree with us. This is a complex area, and we don't have all the answers, but we're trying to figure them out with you.

1. What kind of communication with communities has there been on this project so far?
2. Why are we working on projects like this instead of more work on the backlog of bugs and small improvements for editing?
3. How did we / do we QA lists of suggestions like these?
4. What if communities don't want certain suggestions on their wiki?
5. What determines whether suggestions are good enough to go beyond testing?
6. Is NPOV appropriate to point an LLM at, given its nuance and complexity?
7. If an LLM provides suggestions, how do we make sure that the editor isn't swayed/biased just by virtue of it coming from Wikipedia (a trusted source)?
8. Does it make sense for the message to the world to be "Knowledge is Human", but we are also using AI for certain things?
9. How could the models we use be transparent, open, and have community controls/auditing?

Alright -- more to come as we get into those questions. -- MMiller (WMF) (talk) 23:53, 17 August 2026 (UTC)Reply

@MMiller (WMF): Hi Marshall!

Because of a long history the relation between the community and the WMF is, let's say, far from perfect. And arguably worse than ever.

This is obviously not your fault, but its important to set the scene.

Because Wikipedians are smart and informed they are, generally, skeptical and wary of AI.

This is the correct position in 2026, because we do not yet know the impact of AI on the environment, jobs, and the upcoming resource wars.

The community generates the value and people think they donate to support it. The WMF is terrible at doing the community actually wants and needs. MediaWiki's tech debt is a sad joke. Community members who try to explain what we need from the WMF are routinely ignored. For decades.

Instead of working on the things that are important, the WMF nerds build shiny new toys. Debugging old code sucks. Doing something fun with AI/ML is fun. Because WMF leadership is terrible it seems (from the outside) that no one is working on the important stuff while we get an endless stream of halfbaked projects that waste a lot of time and money. The WMF basically never finishes a project it starts, overcommits at an early stage, and only gets feedback when its too late.

The WMF has a tendency to drop in and reveal they spent a lot of time working on a terrible idea, without asking input from the community, and then are surprised when the community rejects it.

The WMF is terrible at communicating its wishes and goals and what its working on.

The previous WMF attempt to "do something with AI" was a terrible idea and everyone hated it. It proved yet again that the WMF does not understand the Wikipedia community, what writing an encyclopedia means, and (the limitations of) AI.

The WMF did not learn from this and is again presenting a half-baked plan they spent a lot and time and resources on.

The community desperately wants and needs the WMF to succeed and produce great software at a reasonable cost, but that has yet to happen.

Quite a few members of the community are nerds who have spent a lot of time playing around with AI so they know what its limitations are in the context of writing an encyclopedia.

As the person behind the AI Proofreader, AI Source Verifier, AI Editsummary etc I am clearly not some anti-AI luddite.

The WMF is actively making the community more and more anti-AI, and to be honest rightly so. The WMF has no respect for the hard work of countless Wikipedians.

The fact that the WMF does things like have an AI check for NPOV problems shows that the WMF does not understand AI or its limitations or how to write an encyclopedia.

The community could greatly benefit from responsible AI use in a few specific tasks, whereby the human makes the final decision and is responsible for the edit. And the WMF makes it impossible for me to communicate that to people.

At this point, we both know its too late to listen to my feedback. This terrible idea will continue no matter what. The WMF has spent millions on AI related stuff and any benefits to the community were not proportional to the amount of money spent. It doesn't matter to the WMF; they had fun and can put something cool AI-related on their CV and move on.

The WMF has a toxic positivity problem wherein honesty and negative feedback is punished and ignored and all criticism must be hidden below 7 layers of a compliment sandwich. Even people far more diplomatic than I am just can't deal with all the corpo-speak and manipulation.

In an ideal world the WMF would stop what its doing and actually listen.

LLMs present an unique set of challenges and even opportunities and the WMF is fucking it up for everyone else and does not seem to understand how much damage they are causing and can cause in the future.

Is NPOV appropriate to point an LLM at, given its nuance and complexity? No, and that is a silly question. I don't teach my fish braille and ask it to rewrite the bible. Tools are useful in some contexts and bad in others.

What if communities don't want certain suggestions on their wiki? None of the proposed suggestions in its current form would be an improvement.

Does it make sense for the message to the world to be "Knowledge is Human", but we are also using AI for certain things? No, of course not.

How could the models we use be transparent, open, and have community controls/auditing? That is impossible unless you accept the models are terrible compared to competitors. There are no ethical LLMs available. And making something like that would be unethical and require unethical actions.

Why are we working on projects like this instead of more work on the backlog of bugs and small improvements for editing? Because the important work sucks and nerds think its fun to play around with AI. Also the WMF does not understand its role; it should serve and protect the community. In any well-run company you do maybe 90% important stuff and 10% fun stuff.

In the future, the community should be involved at the earliest opportunity, when brainstorming. The WMF should learn from its mistakes. Reflect on Simple Summaries. What went wrong, why, how to avoid that in the future? Allowing random newcomers to act on typofixes proposed by an AI will just make a lot of people very angry, sets newcomers up for failure and degrades the quality of Wikipedia. I use the opposite approach; my software helps experienced Wikipedians to fix typos, and an AI helps filter out things that aren't typos, and no one objects to that.

Polygnotus (talk) 00:41, 18 August 2026 (UTC)Reply

Please assume good faith. No one is doing this to have fun or put "something cool" on their CV and move on.

Wikimedia has actually spent a terrifyingly small amount on AI infra and tooling, which is part of the problem here: when an experiment is run, fast iteration on prompts, benchmarks, evals, and models isn't second nature. There are indeed ethical language models, as others note below, and better orchestration would help choose the best for a given task. – SJ + 16:02, 21 August 2026 (UTC)Reply

Marshall (and colleagues), I will say that from reading your post earlier, I think it reflects a good attitude for approaching the issue. (I'm not going to go down the rabbit hole of the difference between words and actions and trust.) I will also note that I have no idea how much say you have in what projects you work on and how far you take them. (Although if "Senior Director of Product" isn't just a bunch of fancy words, I would guess quite a bit.)

I clicked your link to see what's currently on Special:EditChecks, and I will say that almost all of them seem like really good ideas. None of them (the ones I like, at least), require any AI beyond some if statements in a trenchcoat.

In short: I think we could completely ignore all this AI stuff and you'd have some really great and helpful projects to work on. (Others have been mentioned above.)

I know working with such a large community can be hard. I would certainly like to think that we could give you some advice on how to help us. Ask . We only bite sometimes. Thanks for taking the time to listen. LittlePuppers (talk) 01:47, 18 August 2026 (UTC)Reply

That is an excellent and fitting username. Polygnotus (talk) 01:53, 18 August 2026 (UTC)Reply

One time I said that I didn't know what "Director of Product" means (I am not a native speaker) and the ex-CEO of the WMF started attacking and accusing me because she couldn't handle mild criticism. Polygnotus (talk) 03:11, 18 August 2026 (UTC)Reply

Thanks -- just to take this opportunity to shed a little light on what I do and how we're set up:

• We're set up as a bunch of "cross-functional" product teams. Cross-functional means each team has a few engineers, an engineer manager, a designer, a product manager, a data analyst, a movement communications specialist (give or take -- sometimes a couple teams will share people in certain roles).
• The product manager is responsible for setting the priorities of the team, what order to work on them, deciding what is in and out of scope, deciding whether the results of an experiment show that something is worth pursuing further. They do this in collaboration with their teammates, not unilaterally.
• As a director of product, many of these product managers report up to me. When I started at the foundation, I was the PM of the Growth team, and I've been around for eight years and am now in a management role.
• So in my role, I look across the various teams and play a role in setting the overall priorities -- like which various potential editing projects the various teams working on contributors should prioritize, what the readers teams should prioritize. I work with director counterparts in engineering and design to do this.

MMiller (WMF) (talk) 18:28, 18 August 2026 (UTC)Reply

Re 6, and to some extent 9: Does the WMF have a suitable operational definition of NPOV to measure against? As I understand it, transformer-based models are fairly common in sentiment analysis nowadays, but I don't know if pre-existing models (if any exist) would be well adapted to an encyclopedic context, and I don't think they would be sufficiently interpretable. In any case, I'd say that it is something that is strictly more difficult than tone check, as well as likely being considered higher risk by the community. I can't be certain of course, but I would expect for the community to grant social licence to a model for NPOV specifically would, at minimum, require much better interpretability than typical for current models. Whether the team believes they can achieve that is probably something best answered by the technical staff, the community can only inform as to acceptance criteria. Alpha3031 (t • c) 05:00, 18 August 2026 (UTC)Reply

@Alpha3031 They just take off-the-shelf selfhosted open-weight models, which are way worse than Claude Fable 5 (and other mainstream commercial offerings) like gpt-oss:120b aya:35b / aya-expanse:32b, llama4:scout, qwen3:235b and qwen3:1.7b and then give it a presumably AI generated prompt containing:

 Role: You are an expert Wikipedia Copyeditor and Reviewer.
    Goal: Conduct a professional audit of an article's plaintext so it aligns with Wikipedia core content policies and Manual of Style.

    Context & Constraints:
    - Plaintext environment: non-prose elements may be stripped (infoboxes, references, templates, media, etc.).
    - Non-markup focus: do not suggest wiki-syntax, template, link-formatting, or reference-format edits.
    - Awareness of extraction gaps: if text appears truncated from extraction, do not flag it unless clearly an authorial issue.
    - Focus only on prose, terminology, neutrality, and information structure.

And

    - Neutrality: remove peacock terms, bias, and unnecessary loaded language.

You appear to be overestimating the sophistication of their approach by a rather wide margin.

Polygnotus (talk) 05:14, 18 August 2026 (UTC)Reply

I am aware of that, given that I had read (and in fact posted the link to) said prompts above. Though, I wouldn't say hosted open weights models are necessarily "way worse" than commercial offerings currently given recent and especially upcoming releases (GLM at 700B especially is likely a lot easier to host than the 2.xT models, though still harder than 120B of course). The page does indicate that the team recognises some situations may require bespoke models though. Alpha3031 (t • c) 07:05, 18 August 2026 (UTC)Reply

Open-weights models can be perfectly adequate for some use cases. For source verification the very modest gpt-oss-20b model worked as well as Sonnet 5. Detecting NPOV issues is just a much harder problem, and it needs much more context and probably better models as well. Alaexis_¿question? 07:18, 18 August 2026 (UTC)Reply

I don't disagree that open-weights models (even older or smaller ones) can be adequate for many tasks. I just also wanted to point out that as of the time of the current discussion, there are several open-weights models in the 700 billion to 3 trillion parameter range that are comparable to the frontier in a much wider selection of tasks, namely Kimi, Qwen and the upcoming GLM 5.3 (which being much smaller, is probably going to be cheaper to evaluate).

Incidentally, it does appear that previous WMF research findings (m:Research:Test External AI Models for Integration into the Wikimedia Ecosystem#Evaluation Results and Findings) have pointed out the difficulty of NPOV assessment:

Some policy detection tasks are hard for humans and machines alike. Specifically for NPOV, precision across the three model families is very low. Models overall do better when detecting Peacock behavior. This is because while understanding neutrality requires in-depth reasoning, [bold mine] peacock behavior can be detected via language features.

so it's somewhat odd that they've picked it as a easy, low-risk task in this specific project.

I would say that adequate NPOV assessment is probably the hardest possible task to set as a goal, given that it requires the aforementioned tone check, a comparison with the sources, and also some sort of test to ensure the sources the other models see are an accurate reflection of the body of published RS more generally. It's definitely not suitable for a project intended to explore what can be done with relatively generic models. I think there could also be room to explore, e.g., faster community feedback cycles so that the WMF doesn't feel like it needs a whole year to develop a single task specific model. Alpha3031 (t • c) 08:56, 18 August 2026 (UTC)Reply

Agree that NPOV assessment is the hardest possible task, also for humans. More importantly, the assessment relies on consensus. There is no authority that has the truth about whether something is NPOV. The whole point is that the community reaches a consensus looking at all available reliable sources. Delegating this to an LLM (with the mark of approval of the WMF itself, thus giving it an undue resemblance of authority) defies the premise of Wikipedia, which is based on decisions taken by consensus. Aggravated by the fact that LLMs are not at all unbiased by any definition of the term (and are often just plainly wrong). Ita140188 (talk) 10:27, 18 August 2026 (UTC)Reply

Agreed, NPOV is hard! Alaexis_¿question? 12:52, 18 August 2026 (UTC)Reply

Thanks for the response, this is already much more transparency than the last time around and I appreciate it.

I am a bit confused as to whether these questions are for us and for you. Gnomingstuff (talk) 17:35, 18 August 2026 (UTC)Reply

@Gnomingstuff: oh, good clarification. The questions Marshall posted are a first pass at translating what we're hearing from y'all into a set of questions that we (staff) can then respond to one-by-one. Of course, if you think there are questions we've missed and/or questions that we've misinterpreted, please comment as much. PPelberg (WMF) (talk) 20:05, 18 August 2026 (UTC)Reply

I thought I would ring in to point to another AI-backed project WMF is doing and share our experience working with volunteers on it, given the interest here in this sort of thing. For context, I lead the Product Safety and Integrity (PSI) team here at WMF, we build security and safety features.

One thing we're working on is detecting abusive content using LLM-based models. Specifically to enwiki, last week we deployed a new feature that is only visible to oversighters, at Special:AbuseReview, which uses an open-weight model called CoPE to flag edits that probably need to be suppressed due to containing personal information.

This on-wiki feature is now being used enwiki oversighters to review and take action on what it raises. They tell us the practical accuracy rate they experience in reviewing the output is better than what they see in the reports they get from human users. And, when we were testing the model output in June and July, in batches through an off-wiki process, most of the true positives it was catching were not being caught organically on-wiki.

One thing I want to mention is the iteration and volunteer collaboration that it took for us to make something deployable. There was some initial volunteer skepticism, and we needed to demonstrate not only that it was valuable, but that this was going to be accurate enough to not waste precious volunteer time. That took some internal experimentation on our part to get something that showed enough promise on both those fronts that we could ask for support doing a round of manual labeling -- which is what really pushed it over the edge of accuracy to be a deployable feature.

This collaboration has helped us do something meaningful about doxxing on English Wikipedia over the last few months, that feels good to both WMF and volunteers. That is possible in large part because volunteers were willing to give us space to operate, and to keep an open mind that could be convinced by data. We also needed to be flexible in our own thinking and incorporate volunteer ideas, something PSI has gotten used to doing in our work. EMill-WMF (talk) 21:17, 19 August 2026 (UTC)Reply

Endorsing Eric's statement, as one of the oversighters involved in the testing/iteration he describes. This is catching OS-level material we were not catching otherwise, and is a clear benefit to the encyclopedia. In solidarity, asilvering (talk) 21:54, 19 August 2026 (UTC)Reply

I think there is a role for AI to play on Wikipedia and it's going to be in something kind of like this. Something Eric alluded to but doesn't say directly which I think is important: the first draft the WMF showed us was nowhere close to ready. From that the learning was that we needed to go even farther in how confident . The second draft - which is when we started doing manual labeling - was still not anything which would have been appropriate for use. It did lead to a learning that one element - personal information - was more reliable than the other OS criteria which got us to the third draft and the efforts to then backfill June & July. That got us a lot of incredibly sensitive information that wasn't getting reported and which is now getting appropriately oversighted. I was the one who ran some data after we completed June to compare the true positive rate against email reports and found it to be in the range of 5% higher. I plan to revisit these stats soon and my hope would be that we'd have a higher difference between email reports and these flags for two reasons: 1) the classifier has been improved since I ran those numbers originally 2) some of the "obviously needs OS" tickets we'd have gotten in the past, we won't be getting because OS will be oversighting edits before some other qualified person finds them and reports them. I am really glad we're getting these signals now to stop some very sensitive personal information from being exposed against policy, we wouldn't have been able to do it without the AI classifer model, and it did take an iterative process to get there. Best, Barkeep49 (talk) 23:05, 19 August 2026 (UTC)Reply

Downloaded the most recent csv, picked an NPOV one at random:

• "1358549724,The_Dying_Rooms,10791564,NPOV,Remove biased and profane language describing the Chinese government and replace it with a neutral summary of the government's response.,"In the film, Blewett and others travel to mainland China to visit orphanages housing children abandoned due to the ""one-child policy"". The filmmakers stated that unwanted female and disabled children were left to die of neglect, allowing parents to have another child. Showing that China government is a piece of sh!t for letting this happened and being a coward by lying to our faces that it didn't happened and said that all the footage is fabricated to destroy the reputation of the china government.>",enwiki,df59b690-6c8e-4827-b809-76e90d409f9f,"Editors often revise this kind of wording, saying the tone is unbalanced. You can help rewriting it using a [neutral point of view](https://en.wikipedia.org/wiki/Wikipedia:Neutral_point_of_view).",Revise tone"

The statement the LLM objects to was in the article for less than a minute and long reverted by the time the suggestion was created. So one can add to all the above problems and errors that it also wastes times and resources by not checking the current version but some snapshot.

Profanity checks in general are a bad idea.

• "1358746317,2024_G20_Rio_de_Janeiro_summit,72163669,NPOV,Rephrase the incident with the first lady using neutral language and omit the explicit profanity.,"During a speech about fake news, Rosângela Lula da Silva, first lady of Brazil, swore at Elon Musk, saying: ""I'm not afraid of you. Fuck you, Elon Musk"" (Eu não tenho medo de você. Inclusive, fuck you, Elon Musk, in Portuguese).",enwiki,fba2904a-9276-47e3-8f5c-27fe5aff71f6,"Editors often revise this kind of wording, saying the tone is unbalanced. You can help rewriting it using a [neutral point of view](https://en.wikipedia.org/wiki/Wikipedia:Neutral_point_of_view).",Revise tone"

No, we are not going to "omit the explicit profanity" from a quote, and suggesting things like this is a very, very bad idea.

• "1357774252,God_Emperor_Trump,74631506,NPOV,Remove profanity from the description of the phrase on the sword to maintain a neutral and encyclopedic tone.,"According to Fabrizio, the phrase could mean 'here's your fucking tariffs'.",enwiki,0f886b1f-2a4c-4303-b06e-63da6700b71e,"Editors often revise this kind of wording, saying the tone is unbalanced. You can help rewriting it using a [neutral point of view](https://en.wikipedia.org/wiki/Wikipedia:Neutral_point_of_view).",Revise tone"

Again, it's a quote, from the creator of the sculpture. The AI should not make statements like "Editors often revise this kind of wording, saying the tone is unbalanced. ", which only works to influence newbies by making a false claim to authority.

And then there the internally contradictory advices, indicating the inherent stupidity of LLMs.

• "1356142445,Allan_Segura_(model),80130058,simplify_language,Combine the two sentences about sexual orientation and activism into one concise sentence.,Allan Segura is openly gay. He is a transgender rights activist.,enwiki,d28d67bc-2b32-4cbb-9135-ed391c91c13b,Readers might find this text difficult to understand. Try rewriting this using shorter sentences and plain language. [Learn more](https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style#Vocabulary).,Simplify language"

So do we need to combine these two (very short) sentences into one sentence, or do we need to rewrite this using shorter sentences? Or, just perhaps, our readers are perfectly capable of understanding these two sentences and won't "find this text difficult to understand". What a joke. Fram (talk) 09:01, 18 August 2026 (UTC)Reply

@Fram Also, wasn't the fact that the WMF didn't do content the thing protecting them in lawsuits?

Let's say an article contains negative information about a rich person. If the WMF starts to mess with content, do they not open themselves up to be forced to make changes? I am not a lawyer. Polygnotus (talk) 12:23, 18 August 2026 (UTC)Reply

All good reasons not to use AI for this purpose and probably not for any purpose. Once again, the WMF is wasting what remains of its valuable technical staff (and its ample funds) on trying to drag Wikipedia in completely the wrong direction.

If the bot is criticising vandalism which was only live for a minute, I suspect that it may be reading page history (why?) rather than taking a snapshot. Certes (talk) 12:32, 18 August 2026 (UTC)Reply

A snapshot sample of a large number of articles will also catch revisions that lasted only for a minute. There are lots such edits that are reverted quickly.

Assuming that edit suggestions are not displayed when the content has changed, this particular suggestion would never have been shown - no harm down do anyone. Generating checks on a snapshot rather than live is an architectural decision driven by performance, complexity and other considerations. Alaexis_¿question? 12:51, 18 August 2026 (UTC)Reply

Presumably that means any such system will need to be rerun on each relevant page each time they are edited, in case the edit touched the prompted part? Not sure how the current tasks handle this actually. CMD (talk) 12:59, 18 August 2026 (UTC)Reply

Not necessarily, it can be run once a month, with some kind of caching enabled not to re-check the vast majority of content that stays the same. Then the complex and time-consuming part (LLM calls) is done asynchronously, and the easy part (deterministically checking that the content stayed the same) is done when the user opens visual editor. Alaexis_¿question? 14:34, 18 August 2026 (UTC)Reply

That is indeed how it's currently working. A large batch of the suggestions were pre-generated, and was poured into a fairly simple API that VisualEditor's suggestion mode knows how to query and match up to a document.

Presumably if this was a successful experiment that we decide together to scale up, it'd turn into a more complicated system where we do something like fire off a job that precomputes the suggestion for each new revision of a page. Still avoiding needing to do the expensive generation every time someone opens the editor, but keeping them more up-to-date. DLynch (WMF) (talk) 23:03, 19 August 2026 (UTC)Reply

Again, it's a quote, from the creator of the sculpture.

That's another LLM tic -- at least when it comes to Wikipedia edits/suggestions, they really hate direct quotes and will usually tell you to paraphrase them and/or do so themselves. (Example from a 2026 edit summary: "Paraphrased a lengthy direct quote regarding the skater's performance at the 2025 World Team Trophy into a concise summary. This improves readability and helps maintain a neutral, encyclopedic tone by removing overly detailed personal reflections.") Gnomingstuff (talk) 17:41, 18 August 2026 (UTC)Reply

The examples Fram shows above makes me more resolute in opposing this on principle. The WMF is trying to exert editorial control and hiding it by labelling them as "suggestions". ♠JCW555 (talk)♠ 16:48, 18 August 2026 (UTC)Reply

@JCW555, I guarantee you they are not trying to exert editorial control with this feature. They're trying it because they think it will be helpful for beginner editors. We can tell them they're wrong and we don't like the feature without impugning their motives. In solidarity, asilvering (talk) 21:21, 19 August 2026 (UTC)Reply

But suggesting sentences should be reworded is the WMF trying to exert editorial control. To quote MMiller above "It would just point out the spot in the article that needs attention, e.g. "Does this sentence need to be rewritten to be easier to read?". Whether a sentence/passage needs to be reworded is a matter for the talk page amongst editors, not through the WMF via their AI. No reply from the WMF in this section has done anything to assuage that concern for me. If the WMF comes out and says that these "suggestions" wouldn't touch content at all, no matter how small or big, then I'd be a tad less aggressive in my opposition. ♠JCW555 (talk)♠ 21:44, 19 August 2026 (UTC)Reply

Given the WMF seems inclined to continue developing these tools regardless of what we say, I propose we hold an RfC establishing that LLM tools may not be deployed without an explicit consensus from the community. If editors think this may be a good idea, I suggest the following as the first draft of the question:

Should we require an explicit consensus before the WMF is permitted to deploy tools involving the use of an LLM? The WMF may deploy tools for user testing, so long as all of the following criteria are met:

A: Testing of the tool concludes after no more than 12 months, after which the tool must be removed unless there is a community consensus for either extended testing or full deployment.

B: Use of the tool is restricted to editors who have opted-in

C: Use of the tool is restricted to editors who have been granted the LLM-tool user right. This would be a new user right created and granted by admins at WP:PERM.

Does anyone see issues with this proposed question? Are there any revisions? BilledMammal (talk) 07:42, 21 August 2026 (UTC)Reply

I'm not fundamentally opposed to this, but I think we need a lot more clarity about what this new LLM-tool user right would mean. A naive reading of the name would be "This user is exempt from WP:LLM" which I suspect is a broader interpretation than you intended. RoySmith (talk) 12:53, 21 August 2026 (UTC)Reply

This seems like policy creep, and not a good approach (we shouldn't have blanket limitations based on the "type of tool" involved). I believe it is also misdirected: the example above is testing what would be an entirely community-configurable tool. Opt-in is appropriate for things that are so new, and I'd say anything that messes with your margins should be configurable. But proliferating user rights is an anti-pattern, and should only be used as a last resort. – SJ + 16:21, 21 August 2026 (UTC)Reply

yeah, I think this may be premature for now, but worth revisiting if there are any disasters Kowal2701 (talk, contribs) 18:24, 22 August 2026 (UTC)Reply

Consistent with the current wording of NOLLM and with active work being done - so far successfully - at helping limit abuse, I would suggest some cleaving of administrative actions and content actions. Best, Barkeep49 (talk) 17:05, 21 August 2026 (UTC)Reply

Solely regarding access control (I'm undecided regarding the overall proposal): I agree with the other commenters that that creating a new user right isn't the best fit. I think tool-specific JSON lists of approved users that are fully protected could suffice, and be more adaptable to allow for per-tool authorization. isaacl (talk) 17:15, 21 August 2026 (UTC)Reply

The whitelist system works well for AWB/JWB. Certes (talk) 21:13, 27 August 2026 (UTC)Reply

I also think that the community should be able to control and configure the tools used in en wiki but I'm not sure an rfc is needed atm and its wording unfortunately isn't clear.

What is a "tool"? What else, other than edit suggestions, is in the scope?

12 months deadline is arbitrary: it's too long for a tool that the community actively opposes and may be too short for an iterated testing of a complex idea.

Some abuse/vandalism prevention tools by definition cannot be opt in. Alaexis_¿question? 20:18, 21 August 2026 (UTC)Reply

It's hard to guess what is a tool, because the AI is developed secretly and added to Wikipedia's software or configuration as a fait accompli. Certes (talk) 21:13, 27 August 2026 (UTC)Reply

I agree with most of the comments above that this is not a well-defined RfC. In addition, the WMF have said this will be community-configurable if it makes it into production so it appears to be unnecessary anyway. Mike Christie (talk - contribs - library) 18:34, 22 August 2026 (UTC)Reply

I assume this is intended to apply to any future implementation that would integrate LLM features in any way, not just this one concept, which I fully support. Pushing it on us without community consent is inappropriate and exactly the type of forced AI we're seeing on every other platform. Wikipedia is supposed to be different.I'd agree on amending the deadline - I'm not sure what an appropriate initial test phase would look like, but I'd recommend a limit aligned with that and only extended or fully implemented via consensus. Just enough to get a feel for it, not a full beta phase to polish it for release. People with more programming experience than me would probably have a better notion of the timeline.I do think we need some limitation on access (especially if it is fully implemented) and I think Isaacl's suggestion sounds like a better way to handle it. ChompyTheGogoat (talk) 20:58, 27 August 2026 (UTC)Reply

I think that some level of access control is warranted, but the idea of getting individual approval for every tool that the WMF wants to test would impede testing numbers without much safety gain over a single user list for all beta tools.

Having a time limit on testing is strange, but my exact opinion on it depends on how community consensus is defined here. I think something that both mitigates the issue i think you're trying to solve while not dictating the WMF's development calendar would be something like "Any tool in testing shall be disabled if a consensus is formed at the tool's thread on VPWMF that the tool is causing disruption. At which point the tool will only be re-enabled with consensus." Although, if we got anything near consensus that a tool in testing is disruptive, I think the team working on the tool would disable/fix it very quickly. These are people who chose to work on mediawiki here, not WMF management. ^MetalBreaks_AndBends (One for all) 01:32, 28 August 2026 (UTC)Reply

I think it's only suggesting permissions for LLM tools that would potentially be more prone to abuse, not for any and all in testing. ChompyTheGogoat (talk) 07:13, 28 August 2026 (UTC)Reply

There isn't any limiting language in the comment, so I (and most people, probably) would think it would apply to all LLM tools. ^MetalBreaks_AndBends (One for all) 23:42, 28 August 2026 (UTC)Reply

That is what I meant - all LLM, not all tools period. If we end up with so many LLM tools being tested that they're hard to keep track of I'd consider that a problem unto itself. ChompyTheGogoat (talk) 02:50, 29 August 2026 (UTC)Reply

I'm not saying the issue is that they could be hard to keep track of (though over a long enough time period it could), I'm saying that applying for every beta is an unnecessary hassle. ^MetalBreaks_AndBends (One for all) 03:38, 29 August 2026 (UTC)Reply

I think I would share the view that a full, formal RFC would be unnecessary if prior discussion shows clear consensus to implement, for example (assuming it's advertised to a noticeboard like this one or the cleanup wikiproject). I would expect any community members participating in such discussions to be sufficiently in touch with current attitudes towards LLM tools to bring up anything that would seem potentially controversial and standard pre-RFC discussion processes should function fine (w formal closures and move to full RFC if no clear consensus or if there is consensus there should be an RFC in the specific discussion). Alpha3031 (t • c) 04:05, 29 August 2026 (UTC)Reply

Do you mean a full rfc for each tool or an RFC for this proposal? ^MetalBreaks_AndBends (One for all) 06:25, 29 August 2026 (UTC)Reply

I think this conversation has been healthier than some other recent enwiki–WMF conversations. Just wanted to say thanks to the WMFers that are participating here and doing things like compromising (i.e. getting rid of the NPOV model), replying a lot instead of making one polished statement then leaving, and speaking clearly and honestly.

It's tough because on some issues, WMF and enwiki are very out of sync. So even if everyone does everything right, these conversations may still be tough. But doing things like compromising, having conversations with us that aren't just statements, and speaking clearly and honestly are definitely a good approach. Please keep it up. –Novem Linguae (talk) 19:23, 22 August 2026 (UTC)Reply

Absolutely seconding this! Really happy to see WMF folks take community feedback into account. I understand the task can be much harder than it seems at first, especially as these discussions get sprawling and it can be hard to find a thread that unites the whole range of community opinions together, let alone incorporate it in the team's plans for the project.The way you managed to navigate it was a very positive surprise, and I'm looking forward to more productive exchanges from both sides! Chaotic Enby (in solidarity · talk · contribs) 19:45, 22 August 2026 (UTC)Reply

Yes, absolutely. Ymblanter (talk) 08:10, 23 August 2026 (UTC)Reply

Absolutely agreed on all of this Gnomingstuff (talk) 19:01, 23 August 2026 (UTC)Reply

+1. Can't speak for anyone else, but pretty much all my complaints about WMF's poor communication are limited to the Trustees and a few Officers. Everyone else at the WMF seems to communicate fine. MMiller and PPelberg are two usernames (among others) I've grown very accustomed to seeing regularly on-wiki, they've communicated often and effectively with volunteers for years. Levivich (talk) 20:22, 23 August 2026 (UTC)Reply

Thank you -- we're glad to hear it -- we are trying hard! Although there are going to keep being times that we all disagree on ideas, the most important thing is that we can discuss constructively to figure out how to best improve/adapt the wikis. Thank you all for being here for these conversations, on top of doing all your usual wiki work. MMiller (WMF) (talk) 05:20, 24 August 2026 (UTC)Reply

I am of the opinion that most -- maybe all -- WMF employees who are not in top management are competent, helpful, want to do the right thing, and are eager to communicate. I attribute the stonewalling we often see with a perfectly reasonable fear that actually having a dialog with the volunteers will never help your career and just might get you fired. If only there was some way that WMF workers could organize and join an entity that works to protect them from being unjustly fired... Let me know if anyone has ever heard about something like that. --Guy Macon (talk) 12:38, 24 August 2026 (UTC)Reply

I find this post amusing. But as a Wikipedian, I can't help but be myself and note that if you define top management as something beyond "C Suite" Marshall is would probably be considered top management. Best, Barkeep49 (talk) 14:47, 24 August 2026 (UTC)Reply

+Whatever we’re on _{In solidarity} Wikipedian12512 (Talking is fine | contribs) 01:05, 6 September 2026 (UTC)Reply

Based on some of the questions that volunteers raised in the previous discussion about LLM-backed suggestions.

Thank you for reading and thinking critically about this work! PPelberg (WMF) (talk) 19:09, 10 September 2026 (UTC)Reply

1. Yes. 2. Is it possible to grab the datasets from those two studies (or any other similar) and cross-reference the pages by article-page categories, talk-page categories, and/or talk-page templates (like WP:CTOP templates), to see if any categories or CTOPs are more error-prone than others? That might help prioritize review of any identified problem areas. I would think WP:BLPs, which are all in Category:Living people, would be a top priority for source verification. Also: articles that are new, haven't been edited for a long time, or are tagged with relevant maintenance templates, like {{dubious}} or {{failed verification}}. Levivich (talk) 19:24, 10 September 2026 (UTC)Reply

Is it possible to grab the datasets from those two studies (or any other similar) and cross-reference the pages by article-page categories, talk-page categories, and/or talk-page templates (like WP:CTOP templates), to see if any categories or CTOPs are more error-prone than others? That might help prioritize review of any identified problem areas.

Oh, I think this is a great question/idea. @Alaexis: do you know if we're able to access the datasets used in the studies we referenced so that we could do the comparison @Levivich is describing?

I would think WP:BLPs, which are all in Category:Living people, would be a top priority for source verification.

This intuitively makes sense to me and to be doubly sure: to what extent (if any) would it be accurate for us to understand you suggesting this because of the following reasons?

1. Like articles within WP:CTOP, the suggestion performing poorly on BLPs is more consequential relative to articles in other categories like Category:Railway lines
2. Biographies of living people tend to use news, and other web-accessible sources. This means this suggestion should, in theory, be able to retrieve a larger share of the citations used within them.

Also: articles that are new, haven't been edited for a long time...

Good call. This makes sense to me.

...tagged with relevant maintenance templates, like {{dubious}} or {{failed verification}}

Mmm. In essence, you're saying articles within these two categories offer an existing corpus of claims volunteers have identified as failing verification. Accordingly, running the model against those could help us estimate the model's proficiency. Might I be missing/misinterpreting anything here?

A resulting question that comes to mind as I think about this: might articles tagged with {{dubious}} or {{failed verification}} be more likely to contain offline sources? PPelberg (WMF) (talk) 20:40, 10 September 2026 (UTC)Reply

Some of the datasets are available. In fact Semnani et al have made this analysis themselves Articles in the “history” category exhibit the highest inconsistency rate (17.7%), followed by Everyday Life (16.9%) and Society & Social Sciences (14.3%) (Figure 5). The most common error type in history articles is numerical discrepancy. By contrast, categories requiring precise technical knowledge and quantifiable information—such as Mathematics (5.6%) and Technology (9.4%)—show markedly lower rates.

I like the idea of generating edit suggestions for articles with {{failed verification}} templates for Bayesian reasons - if one such citation has been found it's likely that there are more (aka "if you see one cockroach, there are more" principle). Alaexis_¿question? 20:50, 10 September 2026 (UTC)Reply

On why BLP, I actually didn't have either of those reasons in mind, but both are good reasons. I was thinking something similar to #1: content that fails verification (not just poor performance of the tool) is more consequential for BLPs than, eg, railway lines. Of all FVs, those are the ones we should find and fix first.

On why the FV tag, yes, and it could also help us estimate human proficiency :-) I think it'd be useful to know whether the model confirms the FVs or finds that the FVs are actually verified -- either way it'd be useful. But I had in mind what Alaexis mentioned: if there's one FV tag, there are probably more untagged FVs in the article. I'd guess the odds are higher than average (but I don't know that for sure).

For the dubious tag, that's like a "might" or "arguably" fails verification, so it'd be useful to have the tool analyze those for a person to review. A statement tagged dubious needs a verification check.

And yeah, I'd guess dubious and FV tagged content is more likely to have offline sources. I don't know the statistics, but I'd guess offline sources are rare and so it's unlikely to be significantly more?

Cool idea btw and well presented. Thanks to the teams for working on this! Levivich (talk) 21:12, 10 September 2026 (UTC)Reply

This sounds great and I'm glad the WMF is working on it. Regarding "what types of articles", I don't particularly see any reason to restrict articles included in the dataset, beyond "we don't want to scan the entire Wikipedia" - is that the reason? Or something else? In solidarity, asilvering (talk) 19:26, 10 September 2026 (UTC)Reply

If "We don't want to scan the entire Wikipedia" is the or a reason, then not scanning articles tagged as unreferenced (Category:All articles lacking sources) or lacking inline citations (Category:All articles lacking in-text citations) are obvious ones to not include as (assuming the tags are correct, which is a different issue) they cannot contain references that can be validated in this manner (~143k articles total). It's also not worth spending the resources attempting to scan references tagged as (permanently) dead, failed verification, or dubious. Thryduulf (talk) 19:55, 10 September 2026 (UTC)Reply

If "We don't want to scan the entire Wikipedia" is the or a reason, then not scanning articles tagged as unreferenced (Category:All articles lacking sources) or lacking inline citations (Category:All articles lacking in-text citations) are obvious ones to not include as (assuming the tags are correct, which is a different issue) they cannot contain references that can be validated in this manner (~143k articles total).

Great spot, @Thryduulf. Excluding articles in these categories for the reasons you named [i] sounds like a great idea to me unless, of course, there is a consequence here I'm not seeing.

It's also not worth spending the resources attempting to scan references tagged as (permanently) dead, failed verification, or dubious...

With regard to {{failed verification}} and {{dubious}}, can you please say a bit more here? Asked another way: what's prompting you to think it would not be worthwhile to scan articles with those two templates present?

Per what Levivich and I were discussing above, I'd been assuming, perhaps inaccurately, that scanning those articles could provide a helpful baseline to compare the model's proficiency against. Might I be missing something here?

Regarding the "tagged as (permanently) dead" bit specifically, I'm assuming the following. Please let me know what (if anything) I might've missed...

1. I assume you are referring to articles that include Template:Permanent dead link

2. If so, I assume the reason for excluding articles that contain ≥1 of these templates would be because these citations are unlikely to have an archived copy the suggestion could retrieve, making a check on them likely to fail

---

i. We can assume articles within All articles lacking source do not contain citations the model can evaluate claims against PPelberg (WMF) (talk) 21:01, 10 September 2026 (UTC)Reply

re failed verification and dubious, I hadn't thought about training the model. Rather I was just thinking that if a human has already tagged a reference as not supporting the associated text there isn't much benefit in an AI suggesting to a different human that it might not support the associated text (this would be a waste of time and resources). I agree that using them to train the model would be useful.

re permanent dead links, I was thinking on a per-citation not per-article basis - read the tag and skip the associated without spending any resources attempting to verify it in the source.

Your comment about archive templates has sparked another thought though, that this tool could highlight potential problems with archives. Firstly, if the url-status parameter is blank or live then it should attempt to verify using the live link or both links, if it is "dead", "deviated", "usurped" or "unift" then it should only attempt to verify against the archive link.

For every citation that has an archive link the following are possible:

• Live link and archive are identical, both verify the text (no problem)
• Live link and archive are identical, neither verify the text (problem, but not with the archive but worth flagging to human)
• Live link and archive differ, but both verify the text (almost certainly not a problem)
• Live link and archive differ, live link fails verification archive link passes verification (url-status parameter should be changed to deviated, usurped or unfit, but which probably requires human judgement)
• Live link and archive differ, live link passes verification but archive link does not (flag this for human attention)
• Live link and archive differ, both fail verification (include this information when flagging this for human attention)
• Live link is dead, archive verifies text (url-status should be set to dead, possibly flag to something like user:InternetArchiveBot or some other automated task to make changes more widely).
• Live link is dead, archive fails verification (change the url-status as above and flag to both bots and humans as other citations to the same source may also be dead and pass verification).

Thryduulf (talk) 22:29, 10 September 2026 (UTC)Reply

re failed verification and dubious...I agree that using them to train the model would be useful.

Wonderful. Thank you for walking out what you had been thinking!

...re permanent dead links, I was thinking on a per-citation not per-article basis - read the tag and skip the associated without spending any resources attempting to verify it in the source.

Ah, I see. In concept, what you're describing seems valuable to me. @Alaexis: do you think instructing the model to "skip over" claims that have the Template:Permanent dead link associated with them would require an update to the prompt?

@Thryduulf: in case you're curious, I'm asking about the prompt above because, for now, we're reluctant to make any changes to it. Rationale: a) the prompt has been performing pretty well, b) adjusting the prompt could affect the output in unexpected ways. For these reasons, we're hoping to keep the prompt as-is for this initial round of evaluation.

Your comment about archive templates has sparked another thought though, that this tool could highlight potential problems with archives. Firstly, if the url-status parameter is blank or live then it should attempt to verify using the live link or both links, if it is "dead", "deviated", "usurped" or "unift" then it should only attempt to verify against the archive link.

Oh, this is an interesting set of cases. @Alaexis two resulting questions for you and/or @Isaac (WMF):

1. Do we know how (if at all) the suggestion will behave in these various archive template cases?

2. More broadly, to what extent (if any) would it be accurate for me to think that both a) the LLM could accommodate nuanced instructions of this sort were we to deem them important and b) implementing these instructions would come in the form of an adjustment to the prompt? PPelberg (WMF) (talk) 00:22, 11 September 2026 (UTC)Reply

@Thryduulf If you're curious about how the code selects links to check, you can see the logic and additional notes in the userscript via the extractHttpUrl function. It was written to prefer internet archive links, and then the original link, and only then some of the other archive sites. Checking multiple URLs is possible without changing model prompts but it would still complicate the pipeline as it would functionally double the number of URLs to scrape and model requests to make so would slow things down a good bit. I'll leave that decision to Alaexis but the way I've been thinking about it: the core goal here is verifying the claim, which thusfar we've attempted to do that by fetching the "best" URL. As you raise, there are a variety of additional checks you could do at the same time with the goal of improving the citation (and therefore general Verifiability). I've also thought about inferring the language of the source URL to help fill in the lang parameters on citation templates. These additional checks/actions would complicate the core claim verification task though so I would almost prefer them to be separate at least from an interface perspective, but I am taking note of them.

I'll attempt to answer re: the permanent dead link suggestion as well. It's likely possible to add a check for them (this would also be with the heuristics for extracting links, not the LLM portion of the flow). My thinking: there aren't a ton of them and if they're a dead link then the flow will quickly+gracefully fail anyways (they would never show a suggestion to the end-user). Adding this sort of language-project-specific logic to the code can also complicate efforts to extend this tooling to other language editions in the future. But I'll leave it up to Alex whether it's worthwhile. Isaac (WMF) (talk) 17:22, 11 September 2026 (UTC)Reply

Unfortunately I don't know js so the link doesn't really aid my understanding (but that's not your problem). I don't have a strong feel for what is possible, my suggestions are all things that would are desirable if they are possible. If there are things it isn't going to do (for any reason) but might discover in the process of what it does do then documenting those things somewhere that other humans and/or bots can deal with would be a good thing. Thryduulf (talk) 18:04, 11 September 2026 (UTC)Reply

Yes and please keep the suggestions coming! I mainly wanted to communicate that even if some of these extension ideas don't end up incorporated into this experimental suggestion, they're not being discarded. Isaac (WMF) (talk) 19:48, 11 September 2026 (UTC)Reply

...beyond "we don't want to scan the entire Wikipedia" - is that the reason? Or something else?

Good question, @Asilvering. What you described is accurate, [i] In addition, it would be ideal if this initial dataset includes the types of articles that:

1. You all can imagine this suggestion being the most useful on

2. Includes cases that you think could be particularly complex/not straightforward so we can learn how/if the model fails here

---

i. The size of this initial batch of suggestions will be limited to ~1,000 articles PPelberg (WMF) (talk) 21:07, 10 September 2026 (UTC)Reply

This seems like a perfectly reasonable use of AI on Wikipedia, since it's not actually generating text. I'm looking forward to seeing how it performs. --Ahecht (TALK
PAGE) 20:07, 10 September 2026 (UTC)Reply

Yeah, this sounds like a wonderful tool. It would be awesome if this could be fitted with a API front end so other tools could build on top of it. A while ago I built something (https://wikirefs.toolforge.org/show?page_title=Bronx+Grit+Chamber) which parses a page, pulls out all the individual claims, and matches them up with citations. But I did a half-assed job of it and got it to the point where it was good enough for my purposes. And I know if fails badly on some referencing styles. It would be great if all that low-level crud could be done once, properly, correctly, and then everybody else who wanted to build tools in that space could just take advantage of it instead of reinventing it from scratch. RoySmith (talk) 22:03, 10 September 2026 (UTC)Reply

A couple of other things that would be useful... If you don't find the claim in the cited source, look at the other sources cited in the article. It's not uncommon during editing an article for a properly cited statement to get moved but the citation doesn't move with it. Being able to recover the correct pairing would be valuable. Also, if you can't reach a source's URL, see if you can find some other place that has the same item. For example, I'll often cite a NY Times article via the NYT's own archives, which you may not be able to get to because it's behind a paywall, but you can find the same article in ProQuest or some other aggregator. RoySmith (talk) 22:52, 10 September 2026 (UTC)Reply

Combining source verification with finding the edit which added the source to the article often helps with this, could be a useful extentsion. It also helps when existing text in front of a source is modified without reference to the source. CMD (talk) 00:38, 11 September 2026 (UTC)Reply

I think that you can combine the script with "Who Wrote That" already, but you're that it would be useful to see it at a glance. Alaexis_¿question? 11:06, 11 September 2026 (UTC)Reply

@RoySmith, it would be great to integrate with TWL and with the Internet Archive to get access to sources that aren't publicly available. Earwig's Copyvio detector has access to TWL so there is a precedent.

For citation that failed with the "Not Supported - Omission" verdict checking other sources in the article makes sense. It's not necessarily cheap - if you have 30 unsupported citations out of 300 total you'll need to run 30*300=9,000 checks, unless there is some kind of screening (the passages nearest where the claim sits or the sources added in the same edits). I tried building a standalone script (User:Alaexis/CNfirmed) to solve the broader problem of finding reliable sources. It turned out to be harder than I thought though. The biggest problem is *where* to look - generic web search is expensive and the alternatives are brittle. Alaexis_¿question? 10:56, 11 September 2026 (UTC)Reply

It would be great if all that low-level crud could be done once, properly, correctly, and then everybody else who wanted to build tools in that space could just take advantage of it instead of reinventing it from scratch.

@RoySmith to be doubly sure I'm following, by "low-level crud" are you referring to things like splitting the article up into its constituent claims, identifying the source(s) associated with each, etc.?

A while ago I built something...which parses a page, pulls out all the individual claims, and matches them up with citations

Neat! If you happen to have them handy, we'd be eager to learn what referencing styles you noticed this tool struggling with.

...everybody else who wanted to build tools in that space could just take advantage of it instead of reinventing it from scratch.

I can imagine the creativity something like this could inspire. With this said, I think we're still a ways away from being able to determine how feasible something like this would be. Once you confirm the first question I posed above, I'm thinking I can create a phabricator ticket so that we can come back to it at a future point. PPelberg (WMF) (talk) 00:39, 11 September 2026 (UTC)Reply

Yeah, I envision some kind of network API where you give it a page title (revid, whatever) and it gives you back a list of claims and the associated citations in some structured form. Of course, this is really just one specific example of a general pattern. You really should be able to treat every page as an object on which you can perform operations and access those operations via a network API.

We spend too much time and effort reinventing wheels. For example, I don't know how many tools we've got which measure the readable prose size of an article. They all give different results because they all have slightly different definitions of what "readable prose" means. Neither are more right than any other, they're just different. And it's dumb that so many people have spent so much time writing essentially the same function in slightly different ways.

To go back to this particular example, there's already a bunch of different referencing styles in use. My code parses one of them (the one I tend to use) pretty well. I know it fails on others (to be honest, I don't remember which). Imagine a world where parsing claims and citations from articles was handled by one API that handled them all. Then all sorts of tools could take advantage of that. And more to the point when a new format comes along (say, sub-referencing, which is going to hit enwiki Real Soon Now), one bit of code will need to be adapted to handle that, and automatically all the various other tools will now be able to handle it too. RoySmith (talk) 01:12, 11 September 2026 (UTC)Reply

@RoySmith, that's so true. I had to build a lot of non-core stuff and would've been happy to reuse others' work instead. Discoverability of tools is a huge topic, Toolhub-Evolved is the latest solution I'm aware of.

Currently verification is not exposed as a standalone service but that wouldn't be too hard to do if you have a use case in mind. Fetching website data is already a standalone service on Toolforge (repo, ToolHub page). Alaexis_¿question? 10:15, 11 September 2026 (UTC)Reply

Thank you for bringing this discussion here. As someone who's used Alaexis's tool on and off for a few months, this is the kind of collaboration I like to see: working with an established editor to support development or rollout of a tool that's already been field-tested by the community.To answer your questions: 1. Yes, definitely. 2. Dreamyshade has been building https://projo.toolforge.org/review to identify high-priority articles for WP:NPP reviewers – another opportunity for collaboration or sharing ideas? —ClaudineChionh (she/her · talk · email) 00:51, 11 September 2026 (UTC)Reply

Yes, thank you Claudine! The Projo unreviewed article priority-ranking tool (which is new and still in development, still janky) reflects my understanding of how to prioritize articles that need editor help in general, based on factors including CTOPs, BLPs, reference need, page views, orphan status, and cleanup tags such as AI-generated, COI, and POV (the page contains a table of all the factors). It's focused on supporting New Pages patrollers because that is the most urgent work that I know of right now, with the giant backlog and constant influx of COI/UPE and LLM-generated articles. I'm adding more factors based on data I can derive from categories and other elements in the replica database. I'd love to have access to APIs for predicted percentage of LLM-generated content and predicted numbers of verified/partially-verified/failed-verification citations!

I use Alaexis' Source Verifier a lot, especially for Articles for Creation review, New Pages patrol, and AI cleanup tasks. I find it very helpful, especially on articles that are tagged as likely AI-generated or that I suspect are AI-generated. It helps me rapidly check source-text integrity, which helps me figure out whether an article is mostly fine, salvageable, or trash. It's part of my standard toolkit for article quality evaluation, along with https://copyvios.toolforge.org/, https://wikipedia.gptzero.me/, and Cite Unseen - none of them are perfect, just tools, but I believe they're useful when used with competence and a grain of salt. I've been trying to gather lists of tools along these lines: Article workflows#Tools for specific workflow steps + User:Dreamyshade/Article workflows#Semi-automation. (That page has some of the ideas I'm trying to put into practice in the Projo tool.) Dreamyshade (talk) 01:48, 11 September 2026 (UTC)Reply

Wouldn't it not be possible to provide this System access to paywalled content over the Wikipedia Library? The Other Karma (talk) 09:44, 12 September 2026 (UTC)Reply

@The Other Karma, that's the single biggest thing that would extend coverage. Earwig's Copyvio Detector already reaches EBSCO through The Wikipedia Library (phab:T378077), but those credentials were granted for copyright enforcement only, so verification would need its own permission. Flagging this for Meta:User:Samwalton9-WMF as one more editor asking. Alaexis_¿question? 06:21, 14 September 2026 (UTC)Reply

This looks like an actually productive use of AI on Wikipedia. Given the fact that it's not creating new information, it seems fine to me. I think a test run of this may be promising to introduce new editors. 🪐Kepler-1229b | talk | contribs🪐 17:23, 13 September 2026 (UTC)Reply

As for the questions, 1. Possibly but my main editing area lies outside of source verification, so I might not be able to test it as much. 2. Articles with "Failed verification" tags could be first priority for testing. 🪐Kepler-1229b | talk | contribs🪐 17:25, 13 September 2026 (UTC)Reply

For anyone interested in seeing a demo and talking about this in a voice call, we will be hosting a meeting in the Wikimedia Community Discord on 14 Sep 2026 from 17:00 - 18:00 UTC.

This Discord call is happening tomorrow (Monday). Here is a link for anyone interested in joining: https://discord.gg/wikipedia?event=1547782657969098752.

Note: we'll continue being responsive in this thread too. PPelberg (WMF) (talk) 04:51, 14 September 2026 (UTC)Reply

This seems really interesting, and a good usage of LLMs in Wikipedia. Thank you for the open communication too. qcne (talk) 12:02, 14 September 2026 (UTC)Reply

I'll be interested to see how this goes. LLMs are very bad at subtlety and being overconfident on specific answers, but very good when delivery an array of possible answers. Checking source verification could fit very well into the second type, displaying an array of possible issues that may require user intervention is something that could be very useful. -- LCU ActivelyDisinterested «@» °∆t° 20:41, 17 September 2026 (UTC)Reply

It's a much better fit than an LLM trying to understand NPOV, see my point about subtlety and overconfidence.
A common problem in verification is that an editor will add new unsourced content between a sentence and it's reference. So the content then looks like it's supported by a reference, but that's only partially true. Even if this could only surface those issues it would be a major win. -- LCU ActivelyDisinterested «@» °∆t° 20:47, 17 September 2026 (UTC)Reply

Sorry, I'm a bit confused by the question: "Would any of you all be interested in evaluating a batch of these experimental suggestions for en.wiki articles?" I signed up for what sounds exactly like this maybe six weeks or two months ago. The entry test was very buggy, but it seemed to be getting updated. Then nothing. Is this the same project or unrelated? Johnjbarton (talk) 22:17, 28 September 2026 (UTC)Reply

Thanks for the pro-active post on this. I have some reservations based on common things that can go wrong (as well as other things I would prefer be prioritized) but will wait to see the actual suggestions first. Gnomingstuff (talk) 06:09, 14 September 2026 (UTC)Reply

I have some reservations based on common things that can go wrong (as well as other things I would prefer be prioritized) but will wait to see the actual suggestions first.

Understood and sounds great. We'll be in touch in this thread once we have the initial set of suggestions generated for y'all to review.

Before that, we'll be in touch with what criteria we're proposing to use to select the initial 1,000 articles we'll generate suggestions for to make sure we think it will produce a useful sample.

Thanks for the pro-active post on this.

You bet; thank you for demonstrating to us the need and value in doing so, @Gnomingstuff. PPelberg (WMF) (talk) 22:44, 14 September 2026 (UTC)Reply

Hi y'all – below is the criteria we are planning to use to select the initial ~1,000 articles we will generate source verification suggestions for.

If you see criteria that are included or missing from the below that you think is critical to reviewing the reliability of this suggestion, please comment as much.

Article selection criteria

We are proposing to generate an initial batch of source verification suggestions for ~1,000 articles using the following selection criteria. The criteria you see are meant to reflect what we heard from you all: this dataset ought to be representative of where you see these suggestions being most useful and potentially, risky.

Next steps In terms of process, here is how we see things going from here:

• Step #1: Use the criteria above to select ~1,000 articles we will create an initial batch of source verification suggestions for. ← we are here
• Step #2: Conduct an internal review of these suggestions to ensure they are of sufficient quality to warrant y'alls (volunteer) attention
• Step #3: Share the results of the internal review and invite volunteers (you all) to conduct a second review of the internally-vetted dataset.
• Step #4: Make this initial batch of suggestions available for volunteer review in two ways:
  1. In bulk via a spreadsheet
  2. In context via an experimental suggestion within Suggestion Mode
• Step #5: Analyze volunteer evaluations
• Step #6: Share findings and discuss next steps on-wiki

How does this all sound?

PPelberg (WMF) (talk) 16:17, 24 September 2026 (UTC)Reply

This sounds like a solid plan. I must emphasize the importance of being honest with yourselves in the internal review – if you're having to, say, regenerate half the suggestions because of obvious errors, that might mean more work is required on the underlying tool before volunteers test it. Please tell us exactly what you find there, with numbers. I also wonder, if each of these nine categories gets ~1/9th of the 1000 articles, do we have enough statistical power to do whatever analysis you are hoping to do? I'm not a statistician, and 100+ articles per category is still a lot, but that's something you want to check beforehand. Finally, would this check the entire article on the articles where it's run? For instance, if a page has an inline failed verification tag, I would be disappointed if the program was set to generate, say, 3 suggestions per article max and then didn't check the sentence tagged as failed verification. For the NPP category, I suggest choosing only articles that have sat in the NPP queue for a week or two to make them less likely to be instantly reviewed and fixed up by someone patrolling the front of the queue. Toadspike [Talk] 16:50, 24 September 2026 (UTC)Reply

This sounds like a solid plan. I must emphasize the importance of being honest with yourselves in the internal review – if you're having to, say, regenerate half the suggestions because of obvious errors, that might mean more work is required on the underlying tool before volunteers test it. Please tell us exactly what you find there, with numbers.

Well put and agreed. To be explicit, following the internal review, you can expect us to share:

1) How many suggestions we reviewed

2) What a review of each of these suggestions entailed

3) What the quantitative and qualitative outcomes of these reviews were

4) What we think the next step(s) are in the light of the above.

I also wonder, if each of these nine categories gets ~1/9th of the 1000 articles, do we have enough statistical power to do whatever analysis you are hoping to do? I'm not a statistician, and 100+ articles per category is still a lot, but that's something you want to check beforehand.

First, to the question you asked: we will be generating suggestions for all claims within the articles in the dataset, not just a subset. In line with the above, we expect this approach to produce a dataset of, at least, 10,000 suggestions. I think this will provide the scale we need to draw meaningful conclusions. Tho, I defer to @Isaac (WMF) to say definitively here.

For the NPP category, I suggest choosing only articles that have sat in the NPP queue for a week or two to make them less likely to be instantly reviewed and fixed up by someone patrolling the front of the queue.

Good spot. I think incorporating this constraint shouldn't be too difficult for this run. Although, if that proves to be the case, we'll let you know as much.

Thank you for thinking this through, @Toadspike. PPelberg (WMF) (talk) 21:24, 24 September 2026 (UTC)Reply

I think incorporating this constraint shouldn't be too difficult for this run. Although, if that proves to be the case, we'll let you know as much.

Ok! It turns out that implementing the above at this point could have unintended side effects. So, for this first run, we'll not apply any date-based criteria to the articles we analyze from the NPP queue. "Worst" case, we'll see fewer suggestions there because they will be, as you hypothesized, of better quality. PPelberg (WMF) (talk) 21:31, 24 September 2026 (UTC)Reply

Thanks, sounds good. I look forward to being able to get started on this. Toadspike [Talk] 21:45, 24 September 2026 (UTC)Reply

Wonderful. I expect us to complete the internal review and have results to share the week October 12th. PPelberg (WMF) (talk) 21:59, 27 September 2026 (UTC)Reply

I will start with one that caught my eye:

• Project page: meta:Research:Slow Editing Towards Equity
• Amount Spent: $42,356.25 USD
• Result: meta:Research:Slow Editing Towards Equity#Results
• Promised result: meta:Grants:Programs/Wikimedia Research Fund/Slow Editing towards Equity#Impact
• Related: (that last one makes some interesting claims about Wikipedia)

Was this $42K well spent? Who benefited? What was accomplished?

Was this the sort of thing that the donation banners describe your contributions as funding?

Why are some of the links at dead or useless? Are we not capable of posting these publications on WMF servers? --Guy Macon (talk) 18:38, 28 September 2026 (UTC)Reply

Takeaways from the linked paper at [14]:

Theory Section

- The power to shape consensus on Wikipedia is not evenly distributed. Even aside from groups like ArbCom, things like technical knowledge, access to bots, and understanding of the site's jargon gives the small slice of the community with mastery over them ("experienced users") an overwhelming presence in policy discussions.

- The rules pages act as entrenched technical authority that can be used by experienced users against outsiders, not merely as consensus records.

- As proof of the above, most policy pages are very old. If a page survives its first year of official status, it gains enough momentum to be effectively permanent. 11 of the 15 sampled policy pages were fully stabilized by 2011.

- The rules pages are very stable, and that's good. However, they are resistant to change in a way that discourages any new user that isn't already aligned with Wikipedia's way of thinking. Such users don't become experienced users with the ability to shape discussion, and so can't reform the very rules preventing their participation. This is a barrier to gaining new editors from outside the spheres Wikipedia originated in (American male tech nerds).

- Wikipedia's policies, both content and social, can be used as a cudgel against users who threaten local consensus and/or individual editors' worldviews. Examples cited.

- All of the above to say, Wikipedia's current policies fail to reflect actual community consensus due to how they empower users who already agree with them.

There's more stuff about the actual form said empowerment takes, lot of sociology jargon and such. It records the various levels of authority a rules page can have, including unofficial levels like "essay endorsed by popular community members."

Data Section

- The number of individual users involved in creating each policy is very low. It's mostly the same handful of people talking to each other. There were more "critical discursive moments" identified in the sample than unique editors.

- enwiki prefers discussion among a few people with strong opinons and arguments over the flatter voting system of eswiki. The other three languages didn't have much of either, and just kind of trusted whoever was writing the rules.

- Recommendation made that the processes used to define and change policy be adjusted to better reflect actual editor consensus rather than consensus among the technocratically empowered few. No hard suggestion included.

Overall I think it's a useful study, if not for Wikipedia itself then certainly for the history and sociology of internet communities. I'm not mad about it having been funded. ~2026-52337-89 (talk) 23:56, 28 September 2026 (UTC)Reply

The promise (when they were asking for $42K) was "identifying gaps in policy that are necessary for the needs of all English Wikipedians." Please name a gap in policy that was identified, and how we can fix that gap.

And again I ask, Was this the sort of thing that the donation banners describe your contributions as funding? See The Huge Fight Behind Those Pop-Up Fundraising Banners on Wikipedia in Slate --Guy Macon (talk) 02:14, 29 September 2026 (UTC)Reply

Thinking on how to resolve this, perhaps that's the solution - we establish a consensus that donations banners run on enwiki must accurately detail where the money is going. If something is beyond the scope of what donation banners said the WMF is spending money on, then either the WMF needs to cut the program, or not run banners on enwiki. BilledMammal (talk) 02:19, 29 September 2026 (UTC)Reply

@Guy Macon, I don't think expecting research to "succeed" in doing something 100% of the time is a reasonable expectation to have. Research can and does "fail" even with the best effort. Doing research on policies on Wikipedia is within Wikimedia Foundation's overarching programmatic goals to better the community to which donators donate to. There was a certain amount of funding allocated to the researchers, they tried to gain some useful information, they failed in the overall goal but found some interesting findings about internet communities which got published in a (what I assume) is a well respected journal. Sohom (talk) 02:55, 29 September 2026 (UTC)Reply

No need to ping me. I have a watchlist and know how to use it.

Useful? Sure. $42,356 USD of the donor's contributions useful? Not so much. A lot of those donors are living in poverty and sacrifice to give because they were told that this was needed to keep Wikipedia free.

There are many, many researchers who have published papers on Wikipedia in well respected journals. I question why only a handful of insiders get funding for doing that from the WMF. --Guy Macon (talk) 03:06, 29 September 2026 (UTC)Reply

seful? Sure. $42,356 USD of the donor's contributions useful?, that's how research works??? You provide a chunk of funding so that a group of researchers can try different methodologies and techniques and build prototypes to see if any work. If anything, 42K is fairly cheap in the research world for the length of time the project took (2022-2025) actually since if you think about it, a single grad student's yearly tuition and stipend in CS (being paid barely enough to sustain themselves, while also being one of the most funding rich areas) amounts to roughly that number.

I question why only a handful of insiders get funding for doing that from the WMF.. Guy Macon, I would suggest looking at the page for the dissemination of research funds which provides full transparency into who is allowed to put in proposals, how much money WMF is willing to give for research projects, the proposals submitted and rejected, the folks who decide who to fund (who are well respected researchers within the Wikimedia Research community). I don't think characterizing this as "insiders get funding" is a correct characterization. Sohom (talk) 03:22, 29 September 2026 (UTC)Reply

As I have said many times, I think that everyone at the WMF -- from engineers to the people who decide who and what to to fund, are doing a great job at what they are trying to do. I have zero doubt that you are really looking hard at who you decide to pay to do research, and by "insiders" I don't mean "your buddies" or anything implying corruption or malfeasance. I believe you are like the fine people back in the 1960s who were managing the war in Vietnam. They did seriously great work deciding whether to fund more jeeps or more tanks, whether the jungle boots could be made better, etc. Thousands of good decisions were being made by dedicated individuals doing a great job, but never once considering or even being allowed to consider whether we should be fighting a war in Vietnam at all.

Now consider the entire world population of people who have published research on Wikipedia. All of them. They all get paid by someone, usually their university. What percentage get WMF grants? Do an honest estimate. One in a thousand? What differentiates the two groups? One group knows how to craft a proposal that will get them funding in the usual ways academics get funding. The other has learned how to work the system to get WMF funding. How many of each group get free travel and lodging to go to Paris and spend time with Wikimedia staff and Wikipedia editors? How many in each group know the names of multiple WMF employees? How can you call the tiny minority that get grants anything other than insiders? --Guy Macon (talk) 06:47, 29 September 2026 (UTC)Reply

And again I ask, was this the sort of thing that the donation banners describe your contributions as funding? --Guy Macon (talk) 06:47, 29 September 2026 (UTC)Reply

There's lots of things the WMF is doing badly, but I don't think this particular funding decision is one of them. The article seems to be of good quality to me, and seems in line with the research questions in the application. TietoTeekkari (talk) 07:56, 29 September 2026 (UTC)Reply

And I fully agree, just as I agree that the military made some great decisions about funding jungle boots during the Vietnam war. They were great boots. I still have a pair and they look like new. And, just as I think that same military made those great boot decisions without even considering whether we should be there we should be fighting a war is Vietnam, I think the WMF did a fine job of picking which researcher to fund while never even considering whether we should be funding academic papers at all. And again I ask, was this the sort of thing that the donation banners describe your contributions as funding? Is there some special feature in Wikipedia's software that makes the preceding words invisible when I write them? --Guy Macon (talk) 13:50, 29 September 2026 (UTC)Reply

Not Tieto, but the last time I donated to Wikipedia I recall the banner saying something about "supporting and defending open knowledge projects." I'm probably not getting the phrasing exactly right. This research seems very relevant to Wikipedia and how our internal community and governance systems operate, along with how community-based open knowledge projects operate more broadly. ThadeusOfNazereth(he/him)^{Talk to Me!} 19:17, 29 September 2026 (UTC)Reply

You mean this banner? The actual promise was "Please join the 2% of readers who give what they can to to help keep this valuable resource ad-free, up-to-date, and available for all". Nothing about supporting and defending open knowledge projects. The banner also talks about what you are doing when you visit Wikipedia, which I thought was a nice addition. Please note that something can be "very relevant to Wikipedia" without being what the donors were told their donations were funding.

You can see a bunch of fundraising banners here. I couldn't find any documentation as to which were shown when and to who. Perhaps someone else can find that information. --Guy Macon (talk) 23:45, 29 September 2026 (UTC)Reply

Guy, it sounds to me that you want the WMF to support Wikimedia and open knowledge projects but to do no research into how to support those projects, no research into how effective support for those projects is, whether support would be better directed towards maintaining existing (governance) structures or implementing different (governance) structures? If so, why do you think that? If not, please try explaining what I'm getting wrong. Thryduulf (talk) 19:47, 29 September 2026 (UTC)Reply

Not at all. The choice is not between "funding no research" and believing without evidence that this particular research actually ended up "maintaining/implementing governance structures". Can you name a single governance structure that was maintained or implemented by this research? Look at the results of this research again: The first two results are trivial and the third is just plain wrong.

In the banner at the donors -- many of whom live in poverty -- were not told that they were funding research, just as they were never told that they were funding trips to talk about Wikipedia in exotic vacation destinations. They were told that they were keeping Wikipedia ad-free, up-to-date, and available for all. I am fine with donors funding any research that has a reasonable chance of advancing those goals, but they should be told that some of their donations will be given to other individuals and organizations. --Guy Macon (talk) 23:45, 29 September 2026 (UTC)Reply

Research does not have to be successful to have been valuable, and if you only fund research that produces the results that you want it to then that's not research. Please explain how funding people and projects that write and maintain the projects, write and maintain the sources we use, research how the project can be improved, research how best to support the projects, etc. are not "keeping Wikipedia ad-free, up-to-date and available for all" either directly or indirectly? Thryduulf (talk) 00:04, 30 September 2026 (UTC)Reply

By those criteria I can't think of a single thing that the WMF spends money on that doesn't keep Wikipedia ad-free, up-to-date and available for all either directly or indirectly. I can't see any plausible way that not funding this sort of thing could possibly result in Wikipedia being ad supported, stop the volunteers from updating it, or make it unavailable, but you clearly do so I won't waste your time with further disagreement. --Guy Macon (talk) 00:18, 30 September 2026 (UTC)Reply

Can you name a single governance structure that was maintained or implemented by this research? To this among other research appears to be part of a body of research into policies that eventually led to the establishment of the NPOV Working Group and the subsequent implementation of the global baseline NPOV policy. Sohom (talk) 00:27, 30 September 2026 (UTC)Reply

Just to be clear: I think that Wikiresearch is in general a good thing, and funding some of this by WMF seems reasonable. The specifics of funding, methods, and transparency and ethics of the coordination of the research should, of course, be subject to transparent debate (apart from privacy issues). I'm not judging the validity of funding for this particular project. Boud (talk) 11:39, 30 September 2026 (UTC)Reply

I found meta:Baseline NPOV policy (a baseline NPOV policy for the Wikipedias which do not yet have an NPOV policy) to be a fine document. The question is whether it is worth the $42,356.25 USD ($88.43 per word) it cost the donors.

I would be interested in how many Wikipedias did not yet have an NPOV policy when it was created and how many have adopted it. If the donor is paying for "governance structures implemented by this research" it would seem reasonable to ask how many actual governance structures were implemented, not just a web page that seems like it would be useful to someone doing that.

I also noticed that it is only in English and French. One would think that some of that $42,356.25 USD would have been spent on translating it to the languages of the Wikipedias which do not yet have an NPOV policy. I'm just saying. --Guy Macon (talk) 16:14, 30 September 2026 (UTC)Reply

Couldn't small wikis just copy the NPOV policy from English, French or any other well-scrutinised Wikipedia, gratis, and alter any bits they deem inappropriate for their project? Certes (talk) 16:23, 30 September 2026 (UTC)Reply

@Certes, I would suggest looking at Serbian, Bosnian Wikipedia situations where despite copying the policy text over over they've had multiple significant issues surrounding the enforcement of said policies. Similar issues exists to smaller extent on other projects as well. Even on the rather bare-bones baseline policy, smaller communities have pushed back on the baseline policy due to concerns over the fact that it often constrains smaller language wikis in multiple ways in terms of prioritizing spoken history vs colonial written sources or preferring accounts in local language sources over the broader international consensus on a topic. Sohom (talk) 16:38, 30 September 2026 (UTC)Reply

Certes, the real problem in searching for worldwide solutions is that in my experience different language Wikis are effectively different planets with totally different climates. I sometimes look at the Italian, French, German and Spanish wikis, and their culture and customs are strikingly different, and range from those with a highly controlled system (eg German) to others with freewheeling on non-crucial topics. Their wiki cultures adre as different as the foods they eat. You can not give them a universal cookbook. Yesterday, all my dreams... (talk) 18:02, 30 September 2026 (UTC)Reply

@Guy Macon The policy hasn't yet gone into effect. It is still in proposal phase. Also re cost, 42K (and more actually) spent on building policy to avoiding incidents where Wikimedia project's credibility is called into question is imo money fairly well spent. Sohom (talk) 16:40, 30 September 2026 (UTC)Reply

That would indeed be money well spent if that's what the $42K bought. A bargain, I would say. The problem is that you have offered no examples where having One More Web Page On Meta actually avoided any incident where any Wikimedia project's credibility was called into question. Not even a plausible path where that might happen in the future. You seem to be saying that money spent on goals without any hint of a way to actually accomplish those goals is money well spent. I say that $88.43 per word is way above the going price for good intentions, which are currently so cheap that they pave roads with them. --Guy Macon (talk) 22:34, 30 September 2026 (UTC)Reply

The problem is that you have offered no examples where having One More Web Page On Meta actually avoided any incident where any Wikimedia project's credibility was called into question - The m:CheckUser policy and the m:UCoC would be prominent examples of the a "page on meta fixes problems" effect. Sohom (talk) 00:21, 1 October 2026 (UTC)Reply

Please accept my apologies in advance for saying this, but with phrasing like "Wikimedia Foundation's overarching programmatic goals to better the community" you could run for senate. The problem with their research results was similar. Too many words, too little substance. Sorry, but bluntness was needed here. Yesterday, all my dreams... (talk) 15:07, 30 September 2026 (UTC)Reply

Foundation's overall goal of making the wiki community better is what I meant. Ignore the word programmatic, it's a reference to the way WMF and other non-profits calculates it's budget, (as programmatic expenses and non programmatic expenses). Sohom (talk) 16:28, 30 September 2026 (UTC)Reply

Guy, I think $42,355.25 was wasted on the word salad they produced. The other $1 was useful, given that it made me laugh. Yesterday, all my dreams... (talk) 14:56, 30 September 2026 (UTC)Reply