Last week, as part of our Outdated Answers project, we ran a test on Stack Overflow to see what would happen if we stopped pinning the accepted answer to the top of the list of answers. As expected, there were no negative repercussions when we made this change. So, as promised, today we are making this change permanent.
By default, answers are now sorted strictly by Votes (descending order by highest score), and the accepted answer's order in the list is based on its score. If you prefer to sort by Active (descending order by answer's created or edited timestamp) or Oldest (ascending order by answer's created timestamp), the accepted answer is also unpinned.
Read on if you're interested in details on what we learned from the experiment.
How we conducted the test
For questions where the highest scored answer was not the accepted answer and the user's preference was the default Votes sort, 50 percent of users saw the accepted answer pinned to the top and the other 50% saw answers sorted by the highest score with the accepted answer unpinned. For both sorts, our success metric was the rate at which users either upvoted or copied.
Findings
We analyzed the data in a number of ways.
Users copied or voted on any answer. We looked at users who copied all or part of any answer, or users who took any voting action (upvote, downvote, etc.) on any answer. Our hypothesis was that there would be no statistically significant difference between the two sorts, but we found that when the accepted answer was unpinned there was a 4% increase in copying and voting. Success rate was 20.3% when the accepted answer was pinned, and 21.1% when it was unpinned. (In case you are wondering, upvotes far outnumber downvotes, so we weren't terribly concerned that the two types of votes were mixed together in this analysis.)
Users copied the first answer. There was a whopping 61.6% increase in users copying from the top answer when the accepted answer was unpinned and the highest-scoring answer was first in the list of answers. Success rate was 6.9% when the accepted answer was pinned, and 11.2% when it was unpinned.
Users upvoted the first answer. We did not have enough sample size to draw statistically significant conclusions, but there was a 90.5% increase in users upvoting the top answer when the highest-scored answer was shown first. Upvote rate was 0.5% when the accepted answer was pinned, and 1.0% when it was unpinned.
Users copied the accepted answer. There was a 43.6% decrease in users copying from the accepted answer when the highest-scored answer was shown first. Copy rate was 6.9% when the accepted answer was pinned, and 3.9% when it was unpinned.
Users upvoted the accepted answer. We did not have enough sample size to draw statistically significant conclusions, but there was a 39% decrease in users upvoting the accepted answer when the highest-scored answer was shown first. Upvote rate was 0.5% when the accepted answer was pinned, and 0.3% when it was unpinned.
Users copied an answer that was neither the accepted nor highest scored answer. There was an 8% increase in users copying from an answer that was lower down in the list of answers when the highest scored answer was shown first. Copy rate was 5.4% when the accepted answer was pinned, and 5.8% when it was unpinned.
Users upvoted an answer that was neither the accepted nor highest scored answer. We did not have enough sample size to draw statistically significant conclusions, but there was a 3.5% decrease in users upvoting an answer that was lower down in the list of answers when the highest scored answer was shown first. Upvote rate was 0.48% when the accepted answer was pinned, and 0.46% when it was unpinned.
Feedback
Please leave any bugs related to unpinning the accepted answer as answers below this post. We will monitor this post until Wednesday, September 15. Report any further issues after September 15 as new questions on Meta.
MachavityMod
31.8k38 gold badges289 silver badges390 bronze badges
asked Sep 8, 2021 at 15:49
63
As an almost exclusively necro-poster, this change removes the only potential foothold that was available when I post something valuable on an old page with hugely upvoted answers.
If a question from 2010 has 15 answers, and I post a modern, superior answer AND the OP accepts it, then researchers didn't have to journey to the next page/tab to find it because it was pinned to the top of the first page.
Should the green tick at least ensure that the accepted answer is always on the first page?
While I am generally in favor of this new feature, it is bad when there is a provably incorrect answer with loads of upvotes.
On the other hand, it is super good when the accepted answer is significantly downvoted.
25
Thank you for the team for doing this! I think it's a very positive step for the community.
We'll need to remember to make appropriate updates in the Help Center, such as the help answer on What does it mean when an answer is "accepted"? (emphasis changed):
If you accept:
- someone else's answer: You gain +2 reputation and the author of the accepted answer earns +15 reputation.
- your own answer: No reputation is awarded, and the answer does not float to the top of the list. You must wait 48 hours to accept your own answer.
- a community-wiki answer: No reputation is awarded.
It looks like most of the other references to "float" or "pin" are gone, so that's probably just a single missing reference. There are a few other CW answers on meta (like this one and this one) that might need updates too, though those might depend on whether the unpinning will happen across other SE sites as well.
Speaking of that, as this becomes the norm across the site, I am very curious if it has any effect on the acceptance rate on answers, particularly among new users. If there is any motivation in acceptance of "this answer was most useful to me and I'd like to pin it to the top", that motivation may disappear. I don't think that's a likely outcome—the checkmark stays, the +2 rep stays, and the social grace of "my question has been answered" stays as well—but I could see that as a long-term effect based on the wide variety of motivations to accept an answer.
Ryan MMod
20.6k12 gold badges196 silver badges223 bronze badges
4
This is an excellent change. No more scrolling way down the page to see someone crying out that the accepted answer is bad/broken/misleading (I hope)!
answered Sep 8, 2021 at 18:58
8
Unpinning the accepted answer is a good change.
However, I would still very much like a clear indication (near the top of the question) that an answer has been accepted and offer a link to jump to it. As it is now, when viewing the question, there is no indication that an answer has been accepted unless you scroll through the answers (if there are other answers that have more votes).
Perhaps a link next to the sorting buttons, something like:
2
I agree with the change, thanks for doing this.
For users who want to keep the old behaviour, I've written a small userscript, Accepted Answer Pinner (direct installation link – Stack Apps post). It pins the accepted answer back on top, unless it's a self-answer that wouldn't qualify for pinning in the old situation.
When the accepted answer has the same number of votes as other answers, I would expect the accepted answer to be listed first (e.g. here: How to convert string to float in Python? Why is it a float when directly assigned but a string when a user inputs it?)
However in the link above, the accepted answer is (currently, for me at least) third in the list (presumably some other secondary sort criteria is applied? It doesn't appear to be timestamp or user rep, but I could be wrong)
I am interested if this was a conscious decision (in which case, fine, although I'd be interested in the reasoning); or just an artefact of unpinning the accepted answer (in which case, should we consider adjusting the sort order for this scenario?)
answered Sep 9, 2021 at 8:13
11
I'm loving this change! There's just one small point of feedback I have:
Can we also have this on Meta?
In the past, there have been heavily downvoted answers that got accepted because they basically tell the OP exactly what they want to hear.
I think it'd be better not to pin those.
4
How to counter the "Early Bird Effect"?
The change could be a way to cater for the outdated solutions but the change also applies to all posts, new or old.
Would this change also makes it difficult to find a solution that gone through very detailed consideration by the answerer and that really fit the requirement of OP but just answered it later than the other people who post answers quicker and earlier?
From my observation, at least in the tag that I focus mostly in, people who post answers early are more often to get more upvotes than an answer who post later. I would call this Early Bird Effect because new posts are more readily attracting people viewing the posts and more people to vote if an answer is good (but not necessarily the best respective to OP's requirements).
People answers late may take extended time in crafting a comprehensive solution that fit well the requirements but then also got the penality? that there are less people to see their answers and thus got less chance for upvotes.
Pinning the accepted solution at the top could reduce this kind of Early Bird Effect. Ultimately, it is the OP that have chosen this solution that at least the OP think is best for their requirement. Should we respect their choice that the solution they picked is the best? Understand that sometimes for some OP new to the field, they may not be able to pick the best solution. But I more often see other people who upvote really don't go into the very details of the requirements and think clearly enough for them to judge what's the best solution.
So, any remedy for this situation after we unpin the accepted solution permanently?
One last comment is that in the old days, even the accepted solution is not the best and/or is outdated, people can still easily check the highest vote answer by scrolling a little bit down to the second answer listed. Now, we are more difficult to find the accepted solution (that's the best at least according to OP) somewhere down in the list of answers (could be the 4th, 5th, or even 8th answer down in the list, if this accepted answer only got one upvote from OP, while several other quickly posted answers could have 2 or 3 votes and listed higher).
11
There was a suggestion from user31389 in the comments on another answer that I agree with strongly and that I think deserves to be an answer itself (emphasis mine):
How about showing the accepted answer first if it's more than a week newer than all answers with higher score? Or showing it before all the answers older than it? This way if OP accepted a new answer, it will be highly visible. This is especially useful if the old answers stopped being valid or a new, superior approach has become possible.
The problem with unpinning the accepted answer entirely and sorting only by score is that it suffers from largely the same problem that it was meant to solve: older, outdated answers being shown ahead of newer, more helpful ones. With this current change, even if the OP accepts a new helpful answer that is objectively better than the existing ones, that more useful answer may remain buried beneath older ones and never see the light of day by anyone but the OP who would get a notification. It would be much less likely to eventually attract enough votes to move ahead of the older, potentially outdated answers.
The suggestion to pin the accepted answer above older answers, but not above newer ones, seems like it would resolve this problem in both cases. Newer answers that get more votes still rise over the accepted answer, but a newer answer that the OP accepts as better than the existing ones would rise above the existing answers, even if they had already been highly upvoted.
For what it's worth, it's been my observation on every SE site that I frequent that answers that are posted sooner get more upvotes, even when there are objectively better answers posted later, simply by virtue of the fact that they're listed first until/unless the OP accepts an answer. Having the accepted answer rise above those has historically been one of the best counters against the problem of outdated answers (as well as just a counter against quick, so-so answers being displayed above slower, but more useful ones, even if the earlier ones aren't technically outdated.)
4
It's a very good step and will surely improve how SO delivers knowledge and the experiment and conclusion were done quite fast. Full marks on that.
However, it took extremely long (6-8 years) to even start an experiment about it. Zero marks on that.
I think this delay is really bad in general. It shows that there are still huge gains to win by relatively simple adjustments. My conclusion is that by far not enough experiments are done. I hope that this bottleneck can be overcome in the future, although judging from the past I'm not overly optimistic about that.
I could imagine at least 10 such experiments per year would be a good number. You would learn more from them.
Need an example? What about a copy button on the corner of code snippets. The metric (number of copies) is already in place, the button itself is fairly standard to implement (and standard on many other sites too, so the idea is not really new). An experiment about that could be conducted at extremely low costs. Maybe it would show that people copy even more if there is a button (or maybe not). And it would even save time for the users.
There are probably many similar experiments that would not require much work to implement and are often asked for and could be done right now. Please consider doing them soon.
answered Sep 12, 2021 at 10:39
1
Some thought needs to be given into how this interacts with deleted answers.
Out of curiosity, I went to check out what this did with my most upvoted answer. What I saw was the question at the top (obviously), then all the deleted junk (mostly comments posted as answers by low-rep users), and finally, all the way below that, my answer.
From the perspective of someone looking for answers, this is just horrible UI.
From the perspective of the person who spent couple of days putting that answer together... well do the math.
For those that can't see deleted posts, here's an illustration:
13
feature-request If the rationale is that the best answers "rise to the top" so there's no need to pin an accepted answer, please re-pin it on the views where that doesn't happen: "Active" and "Oldest". The "Votes" view is the only one where the most upvoted one appears first.
It happens I use the "Oldest" view a lot. Often, the oldest answer is not the best by whatever measure you want to apply, whether it's votes or the question poster's opinion (accepted answer), or objective reality. ;-)
8
Are there plans to carry this over to other Stack Exchange network sites? This is probably also useful on non-coding based posts.
answered Sep 10, 2021 at 19:20
1
Finally!
This pinning of accepted answer gives OP way too much power, and I stopped accepting answers for that single reason.
Now I will consider going back to accept answers, even if I still think it's a weird feature. What if there are two or three answers that I think are equally good? What if it comes a better answer later? It serves virtually no point, since the only thing it says is that "OP thought this was the best answer". That has absolutely no use for me when I'm reading a question later.
What's even weirder is the culture that it's rude to not single out one answer to accept.
But anyway, I'm happy that it finally had its most fatal flaw corrected.
5
Maybe you could sort the answers by the rep what the OP got with them. So: accept equals 1.5 upvote, a downvote equals -0.2 upvote, and bounties as their rep/10. It would be also a way to use bounties to move forward the exceptionally useful answers.
The rationale: the indicator of how useful are our posts on the site, is the reputation we got for them. There is no obvious reason to use a different indicator (like the difference of the ups/downs) to determine the default answer order.
answered Sep 8, 2021 at 17:11
18
You must log in to answer this question.
Explore related questions
See similar questions with these tags.