Settings

Theme

Analyst, measure thyself

blog.noblemail.ca

46 points by wvanbergen · 4 comments

Reader

1 thread
msellout

Why not use logistic regression to estimate likelihood of signup rather than clustering the population into 5 groups? I'd imagine the resulting predictive model would not only have more direct methods for measuring prediction error, but also provide more business insight.

  • carbocation

    Absent underlying qualitative differences, there is rarely a good reason to break a continuous distribution into discrete groups for modeling purposes or for model performance evaluation[1].

    But for human consumption (and I believe this article is an example of that), it can help. It's basically an ocular Hosmer-Lemeshow; not a rigorous or even consistent approach to model performance evaluation, but often interesting to those consuming the model's output. For example, we do it here to give students a sense of what their chances have meant historically: https://www.parchment.com/c/college/college-1404-University-...

    [1] See the Hosmer-Lemeshow test, now uncommonly used.

  • snoble

    Oh, I'm a big fan of logistic regressions. And this model actually has some logistic regression in the mix. But I just used this break down of 5 groups to easily report the performance of the model. I can also report a gini but it is way more visceral to see "this is who I predicted to be my top performing group, this is how big that group was, and this is how they actually performed."

    In general your work may be complicated/sophisticated. But your results need to be simple.

  • noelwelsh

    Note that probabilities estimates from logistic regression are often not well calibrated. This typically doesn't matter if you're using logistic regression to create a classifier. However, if you are making cost-sensitive decisions based on probabilities predicted by logistic regression you make run into trouble. There are a number of publications on calibrating probabilities that suggest fixes.

    All that said, I doubt it makes a difference in this case.

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection