Intro
An investigation into whether its possible to detect if a user has changed their writing style from looking at their comments. Could this be a potential method to detect users using LLMs? Or maybe just useful way to gain insight into how users have changed how they write and interact with the HackerNews site?
Given enough words a body of text can be “fingerprinted” – in better detail this is also known as “stylometry” – the measurement of a writing style. Can we detect if fingerprints change over time, and what would that tell us?
You can skip to the discussion, but overall, it looks like most active users are not using LLMs but there are patterns of normal usage that could be seen.
Try it: https://timwaters.github.io/HNfinger/
Project Home: https://github.com/timwaters/HNfinger
About
Like many on HackerNews (HN) I’m worried about AI slop masquerading as human, and wondered if some users were using LLMs in their comments. Detecting LLM writing is a kind of pattern recognition for me, I can smell when something seems off although its not really something I can set into a set of clear rules but if you are interested in that Wikipeda has a page about these signs.
I wanted to look at two things: 1) has a user changed their writing style and 2) could this new style be identified as LLM text? This project can only really answer the first question. The assumption is that if a user changed after the date of mass LLM adoption it might be possible that the change was due to them using LLM generated text.
Try it with your own username: https://timwaters.github.io/HNfinger/
This project shows how a user has changed their style over time, and shows the top 100 of users who may have changed the most after 2023. It uses the Clickhouse Playground to query Hacker News comments.
This project was inspired by this post about Antirez’s project Reproducing Hacker News writing style fingerprinting and by the comment underneath this submission by user zX41ZdbW https://news.ycombinator.com/item?id=43734314 the query of which was adapted in this project. Antirez’s tools and investigations looked at the stylometric fingerprint signature of users and compared it to users with similar writing styles.
The project was also inspired by https://news.ycombinator.com/item?id=46829029 which finally encouraged me to try this and put it behind a little interactive web page which queries the ClickHouse Playground: https://timwaters.github.io/HNfinger/
The dataset on ClickHouse Playground, added by zX41ZdbW `hackernews_history` is used. Free users of the playground cannot create tables, so we have to change the example query to use sub queries but substantially it’s similar.
The `hackernews_history` table may contain multiple versions of a comment. A comment may have been flag killed or deleted or otherwise edited. So there’s some deduplication so that only one comment text per comment row is recorded. There is a `hackernews` table on clickhouse playground without the history but this hasn’t been updated since 2023.
The method is a simple hashed-word cosine similarity. Words within a comment are split, hashed and put into one of 512 bins.
Only users with comments before Jan 2023 (mass adoption of LLMs) are counted.
Users must be somewhat active users.
It counts how many words users make in blocks of every half year (6 months). Users should have written more than 500 words in each half year block.
There are two queries: for a specific user, and for a batch of users. There is a python script that performs batches and then merges together the results, giving a final top_100.csv file of the users most likely to have changed style from before and after 2023.
For a specific user, a cumulative score of before 2023 is calculated and the output shows each active 6 month block before and after and gives similarity to that.
For the batch of users, a slightly different calculation is given in which users must have at least three 6 month blocks before and after 2023 where they have written more than 500 words.
The thumbnail / stylometric method used here is simple. Better methods could and should be used, for example, Burrows Delta calculates a reference text distribution. For example we could create a baseline distribution from all comments from all users and then that method would be good to see which users are most similar to other users. Ideally we want to determine two things, did the user change their writing and is the new writing identified as LLM text, so any similarity method can only really help with the former.
Results
For a user, the table will give a similarity score and difference to a baseline of “before 2023”. The larger the difference and the lower the similarity, the bigger the difference in the type of words used before and after. There are scores for each first and last half year periods too, as well as the number of tokens where 1 token is pretty much equal to 1 word. My own score was quite low because I’ve not posted much in recent years.
| Period | Tokens | Similarity / 100 | Difference from before |
|---|
For the top 100 users, it shows the number of first or last half year periods, the number of tokens pre and post 2023 and the similarity and difference compared to the 2023 baseline
| Number of pre-23 periods | Number of post-23 periods | before ’23 words | after ’23 words | raw distance | after 2023 similarity score | difference from before 2023 |
|---|
CAVEAT: This data is almost probably wrong and should not imply anything about the users in general or any specific user.
CAVEAT: People should adapt and investigate themselves if they want to take this further. I am no expert in stylometry or LLM detection, or even performant SQL for that matter.
Overall, it looks like most active users are not using LLMs.
From the top 100 users, the range was 88 out of 100 to, at the top, 54 out of 100. The general picture with some closer examination of these accounts is that they do not seem to show LLM usage, but more like a change in how they comment and use the site (see below)
Discussion
There was more variation when the number of words in a time period was lowered, and this also lead to more “active” periods. These additional time periods had less activity and so the words written in these times would carry more weight. I decided 500 words in 6 months was a good compromise, but it might be smoothing over actual differences.
Probably more change will be calculated if there are less sample buckets – if the user signed up in 2022 and only posted a few times before and since, for example. Maybe instead of 6 months periods, we could look at 1 year?
The limit of 500 words per 6 months may filter out some LLM usage or small amount of text in many comments.
Note that users are not allowed to use AI generated text in comments, but this no LLM rule was introduced in 2026 (6ish months ago https://news.ycombinator.com/item?id=47340079). Before that some users may have used it, or may have mixed in with their normal writing from time to time. Determining that might be tricky.
Logically and generally most older users are not going to be changing their usage of social media substantially. While they may be early adopters generally I think they would also be most invested and care about the site.
Baseline calculation. While Jan 2023 was set as the time of mass LLM adoption as this was when the chatGPT was released into the wild (An AI Eternal September), for normal users it’s probably only in more recent years i.e. 2025 – 2026 that more and more people are actually using it for more things.
Change in topics. Since 2023 was a watershed moment, the topics that were discussed on HN also changed. The vocabulary we use when talking about LLMs have changed the overall vocabulary. It could be that any changes before and after may reflect these changes in topics.
Change in usage:
* Some types of comments, like the replies under “Ask HN: Who Wants To Be Hired” posts are formatted differently than usual comments and people will use different words. So if a user talked normally before and is now just using it to find jobs, the results would probably show this change.
* Show HNs also seem to be formatted differently from regular comments.
* As users grow and change their vocabulary changes in how they talk about things. (For me, depending on my reading habits I notice my own style changing to sometimes reflect my reading!)
* Many of the top users are changed their usage into being helpful users such as:
* Showing duplication submissions and past comments.
* Posting links to archival alternatives to submissions
* Comments about the guidelines, helping new users etc.
Improvements
Some things to do to make it more useful:
* Allow users to change the cutoff date for the queries.
* Automatically detect the dates of any major changes in writing styles.
* Use a better fingerprinting / stylometric algorithm.
My own LLM usage for this project
I used gtp-6.1-sol to help write the sql queries, the python batch script and the javascript in the html files. Totalling about 30 requests and 290K tokens spent