Stack Exchange Machine Learning Contest
blog.stackoverflow.comIf you're someone who's interested in ML/datamining but haven't had a chance to put your ideas to any hard/interesting problems I strongly recommend a kaggle contest. It's one thing to plug some data into a random forest and go "oh cool, I guess that did okay" and entirely another to see how other competitors are comparing.
one of the biggest challenges I've found in implementing ML projects is I don't have a great sense of when I've really gotten the most info out of the data. I'm not particularly competitive but the contest format is great for this. When you see that a solution you'd normally be happy with ranks in the lower half of the answer you're really pushed to improve your solution.
This is leads you to learn your tools and algorithms better. For a couple of contests I took seriously I ended up learning tons about R, spent most of my nights reading academic papers on various newer techniques, and also read through a few books. On top of all that you really should spend time reading up on how past winner have won which gives a bunch of practical insight into approaching different ML problems.
In one contest I tried the hardest in I actually placed terribly after the final results were calculated, but looking over what went wrong I was amazed to see that I actually did progress really far with my understanding of ml. I'd say a month of seriously competing is easily worth a semester long grad class.
I am really interesting in ML, but have only recently been diving into it. I have watched all of the videos for Andrew Ng's Coursera course (and most of the programming exercises), but just looking over some of the Kaggle contests I think I would be quickly out of my depth.
Would I be wasting my time attempting these with such a basic level of knowledge?
If you read through the past winners you'll find that in many cases a very simple model will win. I believe one of the winners that posted a blog post had pretty much the background you describe.
When I started I was in a similar position to you and just wanted to see if I could even tread water with some of the really knowledgeable members of the community. I ended up placing in the top 5 for one of the contests I was in (with btw a really simple model).
They usually give you some starter code in either R or Python which will give you the results for a benchmark, start there and then use cross-validaton to see if you can beat that bench mark, and if you do submit. It's very addictive and you'll come away knowing a lot more than you started with.
Awesome, thanks for the info. I am checking out some of the benchmarks now.
Why do you think it is that simple models often win? Is it due to the experts no participating or is there a lot more low-hanging fruit than I previously thought? Or just that simple models are easier to use and reason with for humans and thus easier to get right.
If you look at some of the bios of the top 50 kaggler's there's some pretty impressive backgrounds there, and they participate heavily. So I don't think that's the reason.
I know for my own beginner mistakes, it's a big error to try something out-of-the-box and immediately try to get better cv scores by creating much more complicated solutions.
The truth is a lot of work has been put into any standard implementation of an SVM, RandomForest etc (and even more work has been put into the theory behind those algorithms). Since I haven't come in 1st in any competition and am not a ML expert I don't think I can give you the correct strategy to win. But I can say as a general trend, all of my attempts to create non-standard complicated models did terribly, and many of the decisions I made based on research into fixing a particular problem in a known solution seemed to be better performing (i.e. "How to deal with imbalanced classification problems with a RF?" type questions)
I work at Kaggle.
In many cases where simple models win, there's some insight into the data that the winner found - engineered a new feature, or noticed a pattern and appropriately tuned a particular method. Where those insights exist, they often overshadow any gains by super-sophisticated ML techniques.
Makes sense. Thanks!
You might not do particularly well (or you might!), but diving into a Kaggle competition is one of the better ways to learn a lot about machine learning very quickly.
This is a great idea! I just signed up.
So let me ask you, where do you read up on how past winners have won?
How did you decide on algorithms to try out on a contest? How did you find promising academic papers?
Kaggle has blog posts of many of the past winners: http://blog.kaggle.com/category/dojo/
For algorithms, just try whatever you know best/is fastest to implement. If you're using R I highly recommend the Caret package.
For papers: the best place to get started is to begin browsing the forums or any similar contests, the community there is actually pretty awesome and will frequently post papers. After that google scholar (or even just google) for particular problems will yield nice results.
Also checkout the wiki: http://www.kaggle.com/wiki/Home
Looks like a cool contest, I may check it out. What bothers me about modeling contests (I've taken part in several, it's my field) is they often reward putting 90% of your effort into extracting relatively small performance gains. For one thing it's not a realistic operating environment, there are usually many other factors more important than pure performance like upkeep, cost, speed, etc. This is why the netflix contest winning models couldn't go into production. The other issue I have is that people with other commitments (like a job) don't really stand a chance, it's usually very time-consuming to go from fifth place to first.
As someone who went from top 5 to somewhere in the 60s in one contest, and reviewing results of past contests, I believe a lot of those small tweaks for slight gains in leader board scores end up penalizing the contestant for over-fitting. I saw a similar complaint to yours in a couple of forums but I do believe more often than not those small performance gains in the leader board actually hurt final scores.
Additionally for contests like Heritage Health [0], I believe the necessary goal of RMSLE of less than 0.4 is not considered possible (I came across this in the forums but never verified), so even if the contestants just inch past 0.4 it would still be something impressive.
It's not about small tweaks, it can be substantial additions to a model that improve its actual, out-of-sample performance. A popular method in these contests is ensembling, which involves building many sub-models and combining their scores into a single ensemble model. The netflix winner used ~100 sub-models in their ensemble, but the vast majority of the predictive power came from just three of those sub-models (can't find the source now).
Ah, I think I see what you are saying: essentially that the time it takes to build and tune the blending method and model selection for a 100+ ensemble gives you only a slightly better prediction than an appropriately choosen reasonably performant model at both a large computation and human labor cost?
What I was addressing was the issue that some users on Kaggle seemed frustrated that people were essentially submitting models with small parameter tweaks in order to marginally boost leader board scores. To these complaints I would argue that over-fitting is it's own punishment.
Thanks for the clarification!
Does any other profession have a Kaggle? Imagine a more general contest: build my company a tool that increases our market value by X%; we'll give the winners $Y and a job interview. The expected value of participating is $Y/n, where n is the number of participants.
It's like the opposite of a professional organization. I suppose the libertarians approve. It drives down the cost of labor and therefore might make the market more efficient. Yet I'm suspicious.
I'd like to propose a counter-organization. Analysts can band together and offer a contest. We collaborate to create a tool that gives your company an X% increase in value. Companies bid for the rights to that tool. I'd expect that the value to the laborer would be greater than $Y/n. I guess that just described a consulting company.
Perhaps the situation is not so unique. Art also provides much value in the act of production and many organizations hold art contests similar in design to Kaggle competitions. Open-source software often doesn't even have a competition sponsor.
It'd be ludicrous to imagine holding a contest to offer the best legal advice or diagnosis. I'm not saying that I agree with the restrictions that the American Medical Association has placed over the ability to attend medical school, but the free market is harsh enough competition.
Kaggle does promote the value of the field as a whole. I worry that it commoditizes rather than professionalizes.
It's nice to see this kind of contest but the topic just sets me off on a much-needed rant.
The moderator situation on Stackoverflow is getting out of control. I see a Q&A site as having three main groups:
1. People who ask questions;
2. People who answer questions; and
3. People who edit/moderate questions.
Even 2+ years ago there was a lot of lip service paid to the value of (3). I disagreed then and it's only been reaffirmed by subsequent events. To be clear: it's not that I think these functions have no value, it's that they are, at best, secondary to content creation.
The problem is that these roles without diligent oversight attract the wrong kinds of people (eg [1] [2] and a scandal a few years about an admin black list that I can't seem to find right now).
Take this question from Stackoverflow: Database development mistakes made by application developers [3], a question I spent some time answering and that people seemed to appreciate the answer to (based on comments and 1000+ upvotes). It is closed as "not constructive". This is hardly a unique phenomenon. We've all seen many interesting questions posted here that are now closed or locked and who knows how many have been deleted.
The kind of person you end up is overly pedantic and a real stickler for an arbitrary set of rules.
Editors/moderators are the bureaucrats of the Internet.
As Oscar Wilde said, “The bureaucracy is expanding to meet the needs of the expanding bureaucracy.” [4]. These sorts of people just invent work for themselves in the absence of anything to do.
Joel needs to make some changes to Stackoverflow. It's rapidly going the way of the old Usenet days when anything interesting gets shot down and anything else gets closed and the OP lambasted for not having found the 17 previous duplicates. Not good.
The biggest problem I see is an extreme interpretation of what is "subjective". "What language should I learn?" is an obviously subjective question. In the absence of any concrete criteria, it's hard to give a useful answer.
But consider a question like "What are the pros and cons of Sinatra vs Rails?" This sort of question (IMHO) absolutely has value as someone experienced with both could enumerate the relative merits of each in a pretty objective fashion without making an absolute determination. This is something that absolutely could have value to anyone evaluating Ruby Web frameworks.
So, back to this post, what are the odds of any particular question being closed? it seems to be positively correlated with how much time has passed (since SO's inception) and how interesting the question is.
[1]: http://www.nbcnews.com/technology/technolog/wikipedia-admins...
[2]: http://www.searchenginepeople.com/blog/most-notorious-wikipe...
[3]: http://stackoverflow.com/questions/621884/database-developme...
[4]: http://www.goodreads.com/quotes/130452-the-bureaucracy-is-ex...
> But consider a question like "What are the pros and cons of Sinatra vs Rails?" This sort of question (IMHO) absolutely has value as someone experienced with both could enumerate the relative merits of each in a pretty objective fashion without making an absolute determination. This is something that absolutely could have value to anyone evaluating Ruby Web frameworks.
I guess, but Zookeepers could also potentially talk about "What are the pros and cons of Gorillas vs Sharks?"
http://blog.stackoverflow.com/2011/08/gorilla-vs-shark/
> Database development mistakes made by application developers
This is a discussion, not a question. The entire text of said "question" is, quite literally, "What are common database development mistakes made by application developers?" If it can have infinite answers, is it really a question?
http://stackoverflow.com/questions/621884/database-developme...
Great post, indeed, but it belongs on your blog.
One of the biggest misconceptions about Stack Exchange is this idea that discussion is, in and of itself, a net good to the world -- and therefore we are monsters for not allowing discussion. I do not believe this to be true. There is, and will always be, an infinity of discussion. Like Jay Leno once said about Doritos, "type all you want, we'll make more". If something can be had in infinite amounts, what is its value?
Stack Exchange supports only the minimal subset of discussion necessary to get practical, useful answers to specific questions. The goal is not discussion, but science-in-the-small. Back up your claims. Show us references. Show us data. Share your specific experiences.
Otherwise you end up with Quora, a system where everything is a discussion, and all answers are opinions. Thus they can only be evaluated based on how famous the poster is, or how compelling a yarn they can spin.
Nothing against Robert Scoble and great storytelling (I used to work with Joel Spolsky, after all), but I've seen where that system leads. Given a choice, I'll always take tiny science. You should too.
What are common database development mistakes made by application developers?
It is a question, an open question intended to provoke debate and teach about a subject, in effect it's a request for an FAQ. Now perhaps SO is not intended to be for that sort of question, and that is of course for SO to decide.
I suppose the reason many people come to SO to read questions is that they'd like to learn about a subject area, and the reason many people come is write answers is that they'd like to teach a little about a subject, and this sort of open-ended questions offer the opportunity for someone to answer questions the asker didn't even know they had - like should I add an index to my db, if so when? Should I use natural keys? etc. To in effect tell them to unask all those questions they would otherwise have asked in groping their way to familiarity with the subject. It functions as an FAQ for that particular subject, to prevent beginners from making the same mistakes/asking the same questions over and over.
So that sort of question can be very useful for someone starting out, for the kind of person your site targets. Maybe that sort of question belongs on some other site though, a sort of training site rather than a question/answer site, or maybe SO should just expand to encompass that sort of FAQ function?
I'm not so convinced that for this category of question there is a clear line between 'db x breaks when I do y, what should I do?', 'Do I need to use db transactions in db x?', and 'What are the common db mistakes?', and that one sort of question/answer is rational, and another narrative - is the division really that clear? Are there not many many borderline questions which solicit opinion (of someone who knows more about the subject), and yet are useful for others too? Are not many of these smaller 'science-in-the-small' questions actually answerable in many different ways, each of which may be somewhat valid and none of which is actually 'right' in some categorical way?
For example this question, which is equally open-ended, remains open (rightly so I think as it could be a useful discussion :)
http://stackoverflow.com/questions/327199/what-will-we-do-af...
> or maybe SO should just expand to encompass that sort of FAQ function?
Some of the tag wikis do this:
http://stackoverflow.com/tags/java/info
http://stackoverflow.com/tags/c%23/info
So Cletus' post might make more sense as the tag wiki for, say:
Yes it could although the wikis don't contain the other alternative questions or the dialogue which make Q&A sessions involving several people who know the subject well useful for other readers. For many questions (particularly open ones) there are many answers, and no right answer for all circumstances, so there is no clear division between this question/answers and other longer more specific ones as to being opinion or fact - they're mostly a mix of both.
Because people will inevitably continually ask/answer these questions and many similar more specific and yet still open ones and see the resulting debate as useful on any QA site, it might be good to have a more structured way of moving them to an FAQ section on SO, without destroying all the ad-hoc relations and rewards that users have built up using your QA format (i.e. not turning them into a wiki, which doesn't really suit them and loses all the attribution, comments, discussion etc).
It feels a little draconian at present sometimes when useful answers are marked as 'trivial' or 'not constructive' when they clearly are constructive, but are constructive in a direction SO didn't anticipate.
I have to wonder if the original plan was to leverage the question answering as a wiki generator, but if so they should put pointers on closed questions. I'm not sure whether I have a problem with how SO is conducting themselves, since it could simply be that they're opinionated about what constitutes valid content and aren't afraid to leave the other stuff to other sites. That is, I'm not sure Zawinsky's Law can be extended to GYOFB situations.
> I guess, but Zookeepers could also potentially talk about "What are the pros and cons of Gorillas vs Sharks?"
And on an appropriate forum with as many readers as SO/SE you'd likely get well-thought out responses going far into detail regarding purchase cost, habitat maintenance, prevalence of skilled keepers, etc. Someone who actually was deciding whether to add a Gorilla enclosure or a shark aquarium would find it a very enlightening post.
I understand you created SO/SE and want to see it move in a certain direction, but closing and deleting questions you don't like because it threatens your "science-in-the-small" goal is, I think, a terrible way to go about it.
(FTR, I understand Jeff is not personally going around deleting stuff on the site)
You hit the nail on the head... "an appropriate forum with as many readers as SO/SE"
Stack overflow on the other hand is for people to:
[A]sk practical, answerable questions based on actual problems that you face. Chatty, open-ended questions diminish the usefulness of our site and push other questions off the front page.
Thanks for the reply, Stefan, but I think you misunderstood what I meant by "appropriate" (thought I understand your use); I meant the zoological equivalent of SO/SE (Zoooverflow?)
* Chatty, open-ended questions diminish the usefulness of our site*
How, pray tell?
Because they scare off the experts who post the really useful (i.e. practical) answers.
If too many of them leave, the site becomes useless.
The people who answer (and upvote) questions like "What's your favourite programmer cartoon" do not think this through - they just see something funny and click the upvote button. People have argued "but this got upvoted so it must be good" since the site started, but it just isn't true.
There's a world of difference between "I want to learn Perl or Python, what are the pros and cons of each?" and "What's your favorite programming cartoon?"
I think moderation efforts would be much better spent directing the answers to the open-ended questions away from monoglotist flame wars and toward an honest discussion (there's that word again) of something.
I don't think anyone would complain about aggressively closing questions like the one you posited.
You start attracting the people more interested in discussion than actually Getting Things Done. Bikeshed discussions become the order of the day, and why not, because they are so fun and easy to answer -- everyone's got an opinion to share. And they generate huge amounts of reputation because they're easy to process. "Yup, GOTO sure is terrible, preach it brother, {upvote}"
> Database development mistakes made by application developers
This is a discussion, not a question. The entire text of said "question" is, quite literally, "What are common database development mistakes made by application developers?" If it can have infinite answers, is it really a question?
http://stackoverflow.com/questions/621884/database-developme....
Yes, it really is a question. I don't mean to come off as sarcastic, but it's got a question mark at the end of it - one that you, yourself, put there.
You're the one that's imposing esoteric semantics and restrictions on this.
Great post, indeed, but it belongs on your blog.
Except that:
1) no blogs have the visibility and user base that SO has - not even yours or Joel's. 2) a blog post isn't crowdsourced - at least not to the extent that SO is
One of the biggest misconceptions about Stack Exchange is this idea that discussion is, in and of itself, a net good to the world -- and therefore we are monsters for not allowing discussion. I do not believe this to be true. There is, and will always be, an infinity of discussion. Like Jay Leno once said about Doritos, "type all you want, we'll make more". If something can be had in infinite amounts, what is its value?
Nutpicking and a false dichotomy. Ease up on the defensiveness and try to see it from the point of view of the many people that want/need to know the answer to that question.
For Pete's sake, at least 568 people upvoted the question, and at least 1004 people upvoted the first answer alone.
The community has spoken - they see this as valuable content.
Stack Exchange supports only the minimal subset of discussion necessary to get practical, useful answers to specific questions. The goal is not discussion, but science-in-the-small. Back up your claims. Show us references. Show us data. Share your specific experiences.
This is an overly narrow, baffling and frustrating definition of "question"
(And the goal may not be discussion, but discussion is a characteristic of most answers. It's a community, after all.)
For Pete's sake, at least 568 people upvoted the question, and at least 1004 people upvoted the first answer alone.
The community has spoken - they see this as valuable content.
Popular is not necessarily the same thing as valuable. This is a problem that reddit's been hitting a lot lately (e.g. image macro posts everywhere). The most easily-digested simple things get lots of upvotes because everybody can relate, or at least understand. The more complex, useful answers to more specific problems get a lot less attention because a lot fewer people can recognize the value, or even understand the question.
I have to agree with Jeff Atwood, these types of discussion questions shouldn't be on SO. They're just fluff questions that drown out the important work of solving actual problems.
Popular is not necessarily the same thing as valuable
Fair enough, but we're not talking about "Why does Vista suck?"
Maybe he chose the wrong example to make his point. But the only other example he chose was Python vs Perl, for which he cited his own blog post in which he hyperbolically characterized that question as "Sharks vs Gorillas".
That's not a rational argument. That's refusing to reconsider your position and engaging in fallacies to justify your position.
They're just fluff questions that drown out the important work of solving actual problems.
Like. maybe, avoiding database development mistakes made by application developers?
I'm not sure what your point is. What's the fallacy? I haven't seen any examples of this sort of discussion question that are good arguments for allowing them on SO.
Like. maybe, avoiding database development mistakes made by application developers?
Nobody's saying don't post that information. Just, don't post it on Stack Overflow. There's plenty of other places to post long-form pieces about general discussion questions. The best of which is probably a blog. If I want to avoid database development mistakes I can google for that question pretty easily. I don't need to go trawling through Stack Overflow to find that sort of generic advice, and I don't need to drown out other people's legitimate questions with my own fluff question asking for a rehash of widely-known database development practices which I could have found elsewhere.
"Take this question from Stackoverflow: Database development mistakes made by application developers [3], a question I spent some time answering and that people seemed to appreciate the answer to (based on comments and 1000+ upvotes). It is closed as "not constructive". This is hardly a unique phenomenon. We've all seen many interesting questions posted here that are now closed or locked and who knows how many have been deleted."
This is a rant that's needed and one that SO needs to open themselves to receiving. Their stance on this is just wrong.
I'd actually bookmarked several of your posts (among others) because they were so valuable. So it enrages me to no end to click those bookmarks and find that the entire discussion is simply gone.
To recap - a high quality contribution whose value was validated by dozens of individuals (or perhaps even more) was simply deleted.
The problem starts at the top. As you can see from Jeff Atwood's post below (codinghorror) , even the founders of Stack Overflow don't understand the value of their own platform to their customers. They had a preconceived notion of what StackOverflow Is and they are going to stick to it, users be damned.
I don't mean the bash them, but if nerds want to see a prime example of why they take orders from non-nerds (whom we like to think of as "less intelligent" than us), this is exhibit "A". And if you want to know why StackExchange will fail everywhere else, this is exhibit "A". No other group of users will put up with that crap.
"They had a preconceived notion of what StackOverflow Is and they are going to stick to it, users be damned."
Which is why the site is hugely successful. If you want discussion then do it here or on Reddit. I don't want the questions I ask about why something doesn't work in jQuery or C# or C++ or whatever drowned out by Ruby Vs Python posts or stuff like the question cletus mentioned above, or "what have I got in my pocket" mysteries.
The brilliance of Stack Overflow is how quickly one can get answers to "specific programming problems" because everyone on the site is focused on answering these types of questions, not participating in discussion and navel gazing.
"Which is why the site is hugely successful."
You and Jeff are confused about what makes stackoverflow.com successful.
The reason stackoverflow is successful is because it's a clone of another successful tech Q&A site (experts-exchange.com) minus the scummy answer cloaking plus some social voting.
That's it.
All this stuff about the "right kinds" of questions being "key to its success" is based on absolutely nothing. There is nothing to support that. The one thing we do know is that deleting high quality user-generated content pisses people off.
Look at Reddit vs HN, do you think the difference is down technology ?
On multi-sided market websites (marketplaces, forums, ugc sites, etc) community management is generally a far bigger differentiator than technology. Quora isn't better than Yahoo Answers because of it's technology, but because of it's community.
My guess is that SO wants to steer away from subjective questions which can lead to downward spiralling in quality (i.e. language/tool-chain wars) which can then have a knock-on effect on what type of behaviour users think is acceptable across the rest of the site (i.e broken window syndrome).
Does every Stack Exchange post need to be hijacked by yet another whine about site policy and moderation activities?
No, but it's particularly apposite on this post, as SE is asking for a machine learning algorithm is to assist in moderating questions:
an algorithm that predicts whether (and for what reason) a question will be closed.
which raises all sorts of questions as to what the reasons are for which questions [should][1] be closed - this is a grey area, and there is much argument on SE itself over what sort of questions are considered acceptable, to the extent that they have started retroactively disabling lots of content which doesn't fit with an arbitrary set of rules about what a Q&A site should have on it. Without some moderation obviously the site would descend into chaos, but with very heavy handed and arbitrary moderation it will atrophy and the very people creating the quality content they want will leave, leaving them with moderators (robot or not), new users and trolls, and not much else.
Personally I think it's a huge mistake for SE to start banning questions on the basis of them being not constructive. All their other categories of problem questions make sense, but trying to ban questions that are too controversial or involve opinions is IMHO unwise - that's exactly the sort of question which leads to engaging content on SO, even if some of it verges on a troll. This has been the problematic area for them and has lead to them marking lots of useful posts as not constructive even though they clearly are constructive and informative, just because they fall on the wrong side of a line decided retrospectively to declare certain questions unquestions, and others valuable.
It will be very interesting to see if any of their robot moderators are useful in delineating this more problematic area of questions which are controversial or involve opinions - potentially that is every question/answer set more complex than 'what is 2 + 2?', and that line can vary dramatically depending on the moderator, and their opinions.
So I think this raises an interesting (though perennial) question about how heavy handed community moderation should be; in some ways related to those questions raised recently about quality and moderation on HN.
Worth a try in the name of science, but I'm pessimistic about the ability of machine learning to measure subjective "quality" of a question.
In the best case you'll end up with a txtspeak and Proper English detector. But I knew that already.
I'd be interested to see if there's any predictors for "good quality" questions vs anything else. Some research project for students, perhaps.
I remember reading something about telling the difference between truthful and fake reviews.
(http://cornellsun.com/section/news/content/2011/09/23/cornel...)
There's more literature in this paper.
(http://www.cs.uic.edu/~liub/publications/WWW-2012-group-spam...)
I think a useful metric will be the time-to-first-answer and rate of submitted answers. These will be the answers to badly-researched questions á la "How do I get all the buttons on this page?[jQuery]".
Unfortunately, that is a post-question signal, and won't be able to be used in the context of the competition.
It's in the terms of service.
But seriously, Stack Exchange thrives because of its focus (which is maintained in large part by its culture of moderation, though only a small part of that is done by actual "moderators"; the community self-polices pretty well).
This contest is aimed at catching posts that would be closed before they're posted. Ideally we'll get a classifier that lets us give guidance to the askers. Better education of new/confused/troublesome users, leading to better posts, and fewer closed questions overall.
(Disclaimer: SE Inc. employee, yadda yadda yadda)
Yeah, I figured this would be used to possibly provide some Just In Time help. Can't say I'm surprised people read that as "people posting questions detected to be closed will be shot in the face and fed to the SE Inc pet Unicorn"
Good luck with that. Even your current human moderators can be extremely frustrating to deal with. Can't wait until they have the force of an algorithm behind them.
This particular post is about replacing human moderators with bots, whining about SE moderation policy is actually quite related to the post.
Replacing moderators with bots would be a terrible idea and hopefully will never happen. Kevin Montrose clarified it in a comment:
> To be clear, we’re not intending to have an algorithm automatically close posts. This is aimed at improving our quality metrics (which guide users before posting) and auto-flagging facilities (which help focus moderation attention).
There is a 0% chance of replacing moderators with bots.
I don't even know how you could read that blog post and come to that conclusion...
This: Your goal is to build a classifier that predicts whether or not a question will be closed given the question as submitted.
Everyone seems to have skipped right over this:
"In a perfect world, we'd be able to offer specific, targetted guidance for authors whose posts were likely to be shot down, before they ever showed up on the site, and without requiring as much up-front effort from our community."
Which is waaaaay before the classification line.
So you're wanting to use an automated system (which one might loosely term a bot) to prevent content from being displayed without having to have that content moderated by humans.
How is that not replacing moderators [in part] with "bots"?
Because it's not making the final decision on weather a question will be allowed or not. It just aids in providing the user advise about how to improve the question even before the user has asked it.
The problem right now is that the user mostly has very little opportunity to improve his question, because most of the time it has already been closed, and has very little chance of getting opened again (unless the question is flagged for moderator attention).
I understand what you're saying but this particularly issue is a prime example of the problem so I think a discussion of it is very salient
Honestly, I agree. I just use SO more as a destination from Google; I don't get involved much anymore. It's a great place. But, it's just gotten so pompous I just can't stop rolling my eyes whenever I see many of the moderator discussions or see much on their blog. The "niceness" graph or whatever it was, really just took it such a strange level for me. I feel like Joel is much smarter than this. It's almost like they took Atwood's unabashed overly aggressive moderation stance and made a religion out of it. Actually, a lot of the groupthink over at SO is kind of preachy.
But it's OK because they say "we're making the Internet a better place," :) That's kind of an annoying trope at this point too but that's for another discussion.
That's why this has to happen on Kaggle. If they asked it on SE there was the danger of one of their mods closing the question. There is the claim that people will have a worse experience if generic like comparative questions are allowed. A machine learning contest attempting to verify this claim would be more important. Or they could simply ask this on stackoverflow? Maybe there will be enough answers before somebody sends it to oblivion (meta stackoverflow site) or closes the question.
How do I contact you? The Kontactr form at http://www.cforcoding.com/ is broken..
probabilityOfClosing = (question) ->
text = question.text.toLowerCase()
return (text.length / (text.indexOf('jquery') + 2)) / 100I realize this was in jest, but your algorithm says that the shorter a post, the less likely it is to be closed. A post of zero length has a 0% chance of being closed.
Also, the result is not bound to 0..1
Amusing side note: I clicked on their job ad, apparently they score 10/12 on the "Joel Test", which according to the link indicates "serious problems".
Sounds easier than winning the Loebner Prize, and yet there is more cash on offer.
I just hope the winning entry will prompt the developers to remove that stupid filter[1] that prevents you from referring to the Halting Problem in question titles.
[1]: http://meta.stackoverflow.com/questions/107989/using-the-wor...
All those people, who helped in generating data for machine learning algos, now may feel fooled, like suckers.
why is the prize so small when the economic benefit could be much larger?