Ask HN/PG: Why are comments being paginated?
Is this a new experiment or is it a new permanent thing to reduce load, I don't think many people are going to be looking at the lower end of the spectrum (clicking more) which may detract from interesting comments and conversations.
Examples: look on the front page with comment threads >~30 I agree, I think this basically disincentivizes commenting when you know you're going to end up "below the fold". I went into the feature request thread to see if anyone had requested the ability to turn this off, but gave up on looking after rapid-fire clicking "More" 25+ times. Content more than a page or two back might as well not exist. The Reddit community actually speaks of this phenomenon in threads addressed to new members ("things you should know about Reddit", etc). They say that once there's several long threads with a lot of upvotes, you're better off not commenting as your comment will most likely languish without any attention whatsoever. I'd say that's partially due to scrolling, but you're correct - that More button is so non-obvious that I didn't even see it until I read this thread. Those comments below More might as well not even exist. The most common retort to that viewpoint that I saw was "well, so you won't get karma. Bummer. Your opinion is out there anyway," which is missing the point, I think. It's not about the karma, it's about contributing to the conversation. Now, contributing to the conversation is a competition based on time and popularity. This change will probably do two things: hide a lot of good comments below page flips, and cause people to quickly comment on stories to fill above the fold. The more link has the rel="nofollow" attribute, so anything but the top comments will (in theory) not be indexed by search engines either. Given that some functions of the site are based on karma, I hope it's just a stopgap measure. I believe rel=nofollow only means that the link carries no weight for PageRank and other signals that may affect the target URLs ranking, not that it won't be indexed. To block indexing, you need a <meta name=robots content=noindex> tag on the target page itself. On the plus side - this encourages people to check out the New page and upvote stories they find interesting there, so that they can get in on the comment threads before they've become hot. It could be counterbalanced by some sort of notification (on the Threads page, especially) for replies. I don't care if it's fallen off the front page, if somebody is carrying an interesting discussion in a thread in which I've commented, I may still like to know. Even if there are several days between each comment. The site's incentives to only comment on the hot threads are probably detrimental. You can get reply notifications (email or mobile) using Notifo (YC W10)[1]. I think it's only for replies to your comments, not for all activity in a thread. Maybe we need something like twitter implemented... when you scroll below certain number of comments they load n comments more automagically. That kills the scrollbar. A better way to do it would be to show far fewer comments per page, so that you get approximately a screenful at a time, then make the "next" and "previous" links huge and blindingly obvious. I think you should scroll or navigate, not both. If performance is an issue, some sort of lazy load would be better, so just before you reach the bottom of the page more comments are loaded. Funny, I find that I can reply to a comment, even when it's childrens' comments number in the hundreds, and it still gets attention. Naab it's because of shitty posts like yours than HN is slow! New users should not be able to post. Abusing the system is a problem. [Edit: The comment I replied to here originally had very different content about threading and scoring. I don't know what it is now, so I've removed the reply.] I don't know how reddit works exactly, but I notice that they generally have a few "hidden" comments in the tree. I assume this is to prevent this phenomenon. If your comment is significantly worse than the rest of an active thread, it shouldn't be prominently displayed. The 'hidden' comments are based on a per-user threshold. My brief Reddit experience improved significantly when I raised said threshold. I agree that there's a lot of fluff on Reddit, but I don't think I agree with your solution. I find a lot of good, technical comments end up quite a way down the page in more detailed discussions between users who are more interested in the subject matter than having a conversation (or having their joke voted up the page.) Really, I wish Reddit had something closer to Slashdot's moderation categorisation to filter on. Filtering out "funny" comments would be nice. Ignoring votes cast by people who just want to register their agreement or disagreement would be an incredible, impossible goal. Well, jed has removed it, but his original comment here implied that links|comments are ordered solely on the score via voting... While that is a larger driver in the "topics" that get posted, it certainly is NOT the algorithm used for determining comment ordering. I noted more in my later comment responding to your (in my opinion) naive cheapshot on reddit, but the "best" algorithm, that is the default, does not simply rank by the number of votes. And even then, the nature of subreddits and people opting to subscribe to that which they're interested in goes a long way to putting their "upvotes" or "downvotes" into the context of the users in that subreddit. Your summary of my comment is nowhere near what I wrote. I'm paraphrasing from memory, but I believe my comment was something like this: > That's a problem with any site that threads comments and sorts by score. The parent to my comment pointed out a corner case with scoring that he felt was specific to Reddit, if I recall, and I replied to indicate that such a problem is inherent to sorting/scoring at all and not Reddit itself. Since I made that reply (which actually gives Reddit credit as saying said assertion isn't specific to them), the parent edited to say something else, and I'm not sure how to grok it. Rather than delete and look extra shady, I decided to leave that there. I'm glad I did, because your edit here in response to that speaks volumes as to why you chose to respond as you have. First, I'm well aware of how sorting works on Reddit. I've deployed my own copy of Reddit's code and successfully brought it up, and I've manipulated the code and looked around in it (I'm a Python developer by trade). Second, one of the problems inherent to text is that meaning can be lost, very easily. I wrote the comment to mean "that's inherent to the style of commenting Reddit chose, not Reddit per se". That's a comment regarding a concept far abstract from specifics of Reddit's voting constraints - it's a comment addressing the concepts of threading and scoring themselves. You took my statement as "Reddit has that problem, and it sucks because". I apologize that I wasn't completely crystal clear about what I meant. Even after rereading what I wrote, I cannot see how the way you've taken it could be considered what I intended -- you're really reading a meaning that you want to read, because you want to pick a fight over someone disparaging Reddit. I was a regular contributor to Reddit for several months and realize its value. I think it's a great community. You're not being a great representative of it at the moment, but that's your choice, not theirs. Since you edited out 'slander', I'll put it back: > naive cheapshot slander on Reddit Even with the aforementioned misunderstanding, which is somewhat forgiveable, jumping to slander is a bit much. You know this because you edited it out. I'd ask the courtesy in the future of assuming the best in people, and not accusing them of torts when there's any ambiguity to meaning that you might have missed. Well, as you have removed it, I can only speak to my interpretation, which as you imply, may have been off-base, and if so, I do apologize. Please don't take me as some white knight of reddit, I think that we may agree more than I originally understood. I took your comment as a dismissal of reddit purely on the basis of voting. I apologize as it seems that may not have been your intention. I merely sought to explain the other mechanisms that reddit uses, in addition to the automatic (imo) value of subreddits, to determine the value of content. >I'd ask the courtesy in the future of assuming the best in people, and not accusing them of torts when there's any ambiguity to meaning that you might have missed. You're of course correct. As you noted, I editted my comment as I tried to take a more understanding ground. I'm not used to people reading so quickly. I have a tendency to rephrase and uh, un-embelish, after a second or third evaluation of the thread. I do apologize. Though, I suppose in the end, I still simply don't understand the alternative to voting on content and I feel like the directed subreddits are add relevancy to those votes. The alternative is all moderated or selected content? A community where comments can be killed (ahem)? I guess I like the anarchist communal democracy, even if it ends up that a bunch of Internet kiddies want revenge for a tortured cat. In my mind, at the very least ideologically, it's always superior to the alternative. I'd love another shot at this conversation. If you grant that there are other factors at play for determining comment ordering, and you still insist that a site is "broken" because it utilizes voting... what is the alternative? Or are you suggesting that all sites are in some fashion "broken" and that it's a common problem with any site powered by user content? Reddit solves this by hiding poorly ranked comments. Not quite. That filters out the bad ones, but the underlying problem still remains: people reply to highly-ranked comments with something only tangentially related, or not related at all, in order to put it in front of a wider audience. People then upvote it even though it isn't related because it might be insightful. > People then upvote it even though it isn't related because it might be insightful. What if we, as a community, decided that deliberately trying to float with highly rated comments by posting completely unrelated "responses" is not something we'll tolerate, and downvoted violators, regardless of the insightfulness of the comment? Tangential responses are something more of a gray area, but I'm willing to let them be voted upon on their own merits. Please don't just kill this again -- I cleaned this post up as my wording the first go around was unnecessary, but I think there is a valuable point here. I don't find this to be the case at all... though I do find it to be the assumption made by many who don't actually use reddit. I just checked the top few threads to ensure I don't have confirmation bias (heh, though I understand the futility of such an exercise). Reddit is VERY good about not letting people piggy back on top posts and their "best" algorithm does not sort according to score. It sorts according to how those who voted generally vote, and based on how the majority has voted, both up and down, regardless of overall positive or negative score for the post. The notion that replies are contextual relevant to their parent comment on reddit is a TESTAMENT to my original claim:: That reddit does a good job of exposing, even new posts, in inundated threads. (As usual, it surprises me how much if the "our community is better that there" rhetoric goes on here, while simultaneously, we seem to mock that idea of communities making that judgemental statement.) edit: And even so, is it a bad thing if non-relevant intriguing content is upvoted? Just click the [-] if you don't want to see it... but that's the very nature of discussion. Things go off topic and humanity learns as a result. I meant on reddit. Is your reply satire or am I missing something? I'm not a new user and my comments generally fair well here. I (was, but am not now) downvoted on HN for saying that my posts on reddit get well received attention even with lots of noise... Seems ironic... Ironic that PG's definitive reply is now below the fold: Double ironic that his post is below the fold - but this one isn't. That's deliberate gaming of the system by me, as discussed here: I feel like it would be more tolerable if it loaded the new comments inline a la reddit: at least you wouldn't lose the context/continuity of the conversation that way. Unfortunately it appears that its actually spreading the comments across multiple pages so that when you click "more" you lose everything you were just looking at. Perhaps more importantly, it seriously skews the karma that accrues to responses---those that are above the fold will continue to get upvotes while those below the fold get none, so the disparity grows. I've seen this effect on other sites; one gaming site[0] shows two views of the comment list, "recent" and "top-voted" (both with a "more" button that few people seem to click)---the "recent" comments rarely get more than two or three votes before scrolling off, while the difference between the lowest upvote count in the "top-voted" count and the next-highest comment can be in the hundreds or thousands once a game has been up for more than a day or two. [0]Kongregate.com, if anyone's curious. I think this basically disincentivizes commenting when you know you're going to end up "below the fold". And incentivises replying to comments above the fold. This feature is actually causing damage. Several times yesterday I found myself opening a seemingly interesting discussion, reading the comments, then wondering why so few people were talking about it. The link that says "Hey, there's actually more discussion that we're hiding. Click here to see it" is tiny (and unexpected) so I just plain missed it. I even missed it on this thread until I read a comment talking about other comments that had scrolled off the 1st page, thus demonstrating that there must indeed be a 2nd page and that I should look harder for a way to find it. Had I been able to find (and therefore read) the whole discussion on those topics yesterday, I might have had interesting things to add. So might all the other people who missed them for the same reason. I suspect that the overall quality of discussion has taken a dip since this feature went live. This feature makes me want reply to the comment at the top page. Even if my comment has nothing to do with the parent comment, at least people will read it. I don't even hit the "more" button on the homepage to see older stories; there is no way I am going to hit the more comments. The new system can easily be "gamed," as has been discussed already several times. Here's a specific example. I wanted to observe the irony that PG's definitive answer is now below the fold, but I wanted my comment to be "above the fold" so that people could find PG's comment. Realising that if I simply commented in the appropriate place - as a reply to the original submission, as I have done with this comment - then my comment would repidly disappear even further below the fold, I added it as a reply to the unshakeably top reply: http://news.ycombinator.com/item?id=2119471 Thus my comment is technically mis-placed, but guaranteed to be on the first page of replies and discussion. There has to be a better solution to the problem of load. It depends on the cause, of course, but in the absence of profiling information (always the first step) I would investigate more cacheing to make the system less dynamic. ADDED IN EDIT: I love the way this comment has attracted down-votes - I've been watching it bounce up and down for a bit now. It's clearly alright to discuss the merits and otherwise, but for some people, clearly not alright to demonstrate the effect. On a forum for hackers, I find that delightful! You know, this isn't a problem specific to the new system. It's always been better, karma-gaining wise, to reply to a high-scoring comment, instead of adding a brand new reply. This just makes that (annoying) behavior even more impactful. I'd rather the page take 10 minutes to load than have it paginated. I know PG is just trying to implement a quick temporary fix, but this one is so annoying it made me go back to work (gasp!). Yeah, horrible feature. I get lost where I'm at. I'd love to see it back the way it was please. :) To cut load. I'm not sure it helped much though, so I may not keep it. Er, that doesn't seem like a good reason to downgrade the UI of something important. I don't know how much money it takes to keep HN running, but would just buying ten times as much computing power really be a significant expense for YCombinator, compared to the dealflow from HN? As far as I know, the current server has close to the fastest processor available. RAM might matter more for news. or rewrite it in a more efficient language/runtime would likely have more effect. I suspect the bottleneck isn't cost so much as the mental bandwidth to switch hosts, etc. It would be a context switch for PG/RTM/bringing someone else up to speed on arc. HN's just naturally lagging other priorities. --- Farther in the future: HN's arc webserver uses simple flat files for storage, so moving from one host to two is probably a big context switch as well. I wonder what the current server's specs are, and how long we have until that kinda-sorta-Y2K-like barrier. How about displaying the comments "shallower": if some discussion ("a branch in the comments") goes too deep, maybe that's what doesn't have to be on the same page, as it's obviously something where a lot of arguments and counterarguments occur, and not something where the biggest insight is to be expected. Such comment handling would however certainly motivate people to "post without the parent." I'd very much welcome that. My gut feeling is that on quite a few posts, discussion seems to stick to the first comment. I'm assuming that starts to happen as soon as this top-level comment has enough upvotes to quickly displace any new postings. I think preventing such "sticky" threads from clobbering the first pages of comments may add some decent value to the discussion. Especially in brining in different aspects and povs, perhaps sometimes yielding a more balanced discussion. I almost missed this comment because it was just barely above the fold. It's actually below the fold now. Good. Your comment is the most valuable answer in this thread and it was the last one to show on first page. Indeed, I had to go below the fold. Is there some way the HN community could help with the load, or some pathological behavior that a lot of us are doing that could be replaced by behavior that's easier on the server(s)? If you decide to keep it, please keep displaying the header on subsequent pages. Ajax loading the rest on the first page is better, though. it's fun that this answer is below the fold.
I wouldn't keep it. It would be interesting to see an auto-extend feature like Facebook does with the timeline. Keep scrolling? We will AJAX load them into the page in batches. That should do ya. No buttons necessary, full comment history the further you go, and you don't show more information than you originally ask for (just a couple of comments). I've been using this for some time (though on HN it seems to choke on author-comment pages, I haven't taken the time to update the rule): https://addons.mozilla.org/en-US/firefox/addon/autopager/ It's such a better experience on every site when things auto-load as you scroll down. chrome version: https://chrome.google.com/extensions/detail/mmgagnmbebdebebb... I very much hope pg doesn't do this; I find it very annoying. As some other commenter already noted, this breaks browser text-search. In case you want to quickly checked whether what you want to say has been said already (and upvoted that comment instead or add something as a reply). As long as it doesn't grow at the same rate as Twitter's, this should scale. There's a really easy fix to this: Load more comments on scroll-to-bottom. No button, no bandwidth used, but those who make it to the bottom of a thread don't have to do anything to see new comments. I agree with OP; the current system is suboptimal, to say the least. Ugh, I hope this does not happen, I hate the scroll-to-the-bottom-and-get-more thing. Particularly if the server is under heavy load already, you never know if there is more or if so how much, and it takes the scrollbar away from your mouse pointer once it's loaded another chunk due to the relative offset. Maybe I'm missing something because I've never seen it before, but has anyone ever tried leaving blank space at the bottom of the first page relative to the total number of comments? It may work better on items of fixed length than comments, but even on comments it would at least make the scrolling a little less jumpy. I'm confused by your reasons. With scroll for more, you would know if there is more- the end is the end, just like with pagination. And it's no worse than pagination in the "you don't know how much more" department, either. The pagination is a "More" link, it doesn't indicate how much is on the next page. One would think it would be relatively trivial for "More" to become "256 More" or similar... NOoooo!!! Anything but the Scrollbar of Sisyphus! I'd not heard this term for it before, and it sums up the horror accurately. You have enriched my life. The Scrollbar of Sisyphus breaks the back button (click "Back", browser remembers where you had scrolled to, can't go that far down, lands at the bottom, triggers second "page" of results) jumping to the bottom of the page, occasionally searching, and forces you to jump to the end of the page repeatedly to figure out where you were. It breaks nearly every UI expectation one has about scrolling (especially on a phone) and several that your browser has about caching. Those problems you listed are implementation-specific, and not necessary conditions of the feature. This still would prevent being able to CTRL/CMD-F for a particular term to see if anyone has mentioned it already, something I do a lot, and I imagine others do too. This is true, but a little bit of js would take care of that. Infinite scrolling screws with scrollbar behavior though :( Not until you hit the bottom of the first "page", at which point you've resigned yourself to more content. That would be the first instance of an ajaxy thing on HN I believe. The upvote/downvote on comments and stories uses ajax i believe. yes/no: it uses simple assignations to (new Image()).src. Not sure why you got hit with the down votes, I always assumed it used "real" ajax. I like plain text emails, terminals, emacs, and I really prefer un-paginated html pages. Often I read HN with Lynx. The plain HTML here works much better than the markup at most other sites, including Reddit. I didn't even notice that - that's incredibly annoying. I'd be fine with it, if the More returned the rest of the comments, not the x next comments. I don't really see the point of this either; it can hardly be that big of a resource hog on either ends. It's harder to see whether someone has already made the same point that you want to make, so it could lead to redundant commentary. I often do a quick keyword search of the page to see if someone has already made a related comment. With pagination, you can't easily do that. Until it gets reverted (hopefully) -- AutoPagerize [1] ---- [1]: https://chrome.google.com/extensions/detail/igiofjhpmpihnifd... I'm curious: Why does this plug-in need access to my browsing history? It's probably requesting "tabs" permissions. The Chrome security permissions aren't fine grained enough so often you need to request permission for a lot more than you actually need. For some reason requesting "tabs" shows a scary warning about having access to all your history. You'll find that a ton of extensions need access to your history for this reason, it's actually really annoying as an extension developer. Cool. Thanks very much. That's the priveledge that Chrome extensions require in order to inspect URLs as you load them, so they can conditionally apply themselves to the appropriate site(s). As far as I know, there isn't a more granular priveledge to allow access only to URLs as they're loaded (which an extension could still use to track your browsing habits by storing them in its localStorage). That's a bit bizarre to comment when the primary company that developed Chrome is known for keeping tabs on users' browsing history. wat do you mean? If you type into Chrome's URL/browser bar, it sends the query to google's servers which in turn returns a list of possible search suggestions (autocomplete style). Google likely retains the query for commercialisation purposes. So the user being worried about a Chrome plugin sending browsing data seems bizarre. So, when I click a link on HN, using Chrome, Chrome is sending that information to Google? Likewise when I type http://somethingorother, that's going through Google first? Worst HN feature. Please revert. Couldn't people take advantage of this by only commenting on high-karma comments, regardless of whether or not it's relevant to the parent-comment, but instead just piggy-backing to stay on the first page? That is what happens on slashdot. Let's wait for the official answer, but I also include my take. Maybe because of the community growth. When I began to hang out around here, 6 months before I created an account I believe, 40 upvotes was a huge amount for a post or a comment. Today it's common to see posts with more than 100 in the front page and comments receiving 60 or so. EDIT: The number of comments in each posts also exploded, 20 comments in a thread used to make it very active. EDIT 2: For clarification. Your more likely to upvote to get comments that you like higher. I typically do not upvote much but now it seems important. I noticed the same, and assumed that it was a measure designed to improve site performance. Recently, I've found the HN site has become pretty unresponsive at times - I imagine limiting the number of comments on each page is going to reduce the burden on the server. There should at least be a feature within our profile settings to disable this. Using Ajax for "load on scroll" may be a solution. Or go the reddit way, don't paginate but don't load full sub-threads. I don't understand the rationale for this feature. Here's a better way to reduce load: make commenting not require a reload. Same for editing comments, deleting them, etc. Of all the features we need on HN, this is not one of them. I just noticed the hilarious(ly atrocious) redundancy this causes: Go to the [W3C HTML5 logo thread](http://news.ycombinator.com/item?id=2115551) and see how many times the "Autobot" joke is made. I've counted at least four instances. (I'm even a part of one of the branches, as oblivious as I was to the new system and similar discussions.) I wonder if this comment will show above or below the fold. Flip a coin, I guess. Is there anything around on the HN site architecture? A quickly google didn't turn up anything specific. http://www.arclanguage.org -> install -> arc3.0.tar/news.arc. There is no meaningful information in this comment because nobody is going to see it anyway. When did this start? I read almost exclusively through the ihackernews.com mobile site. I just checked and it does not implement the more button and only shows the first 40 or so comments. It looks like I have been missing out on the end comments of popular threads without realizing it. :( the last few hours Everyone should be using AutoPagerize[1] (a Sarari extension) by now, or at least something similar. It makes your life a lot easier on most websites, one of which is now HN. [1] - http://autopagerize.net/ I, for one, tend not to even read comment threads larger than a certain size, unless I want to see specific reactions to an article. I think jedsmith has it partially right up above when he says "it's about contributing to the conversation", but for me it's also about reading conversations that I can hold in my head. Beyond about 40 comments, I'm not sure I can do this at all. Yet another reason to use Auto Patch Work - https://chrome.google.com/extensions/detail/aeolcjbaammbkgai... I didn't even notice the pages were paginated thanks to the extension (it auto appends page 2... page 3... to the bottom of paginated content) Ironically, if PG does respond to this, his answer might end up unseen below the fold :) I'd rather see a hard limit on the number of top-level threads allowed than a "More." Quoth RiderOfGiraffes 34 minutes ago, Ironic that PG's definitive reply is now below the fold: Is the feature really saving that significant of load time? I mean HN is as minimal as it gets, I'd rather all the comments load and take 3-4 seconds more than have to manually go through and hit more several times. As pointed out by pg above, it seems not to. An educated guess (read: wild speculation on my part) indicates that, due to the ranking algorithm of HN (versus Twitter's simple sort, where new things show up first and order never changes) rendering and bandwith time are probably dwarfed by the time required to load and sort all of the comments on all pages to determine what comments show up on the first page. It might be a good idea for PG to charge a small monthly fee if the issue is server strength. I'd rather they look over the architecture and possible changes to the codebase instead of resorting to charging the users. update: no more the case: see http://news.ycombinator.com/item?id=2121727
[1]: http://news.ycombinator.com/item?id=1720763