Applying principles of data science to your own thinking

Philosophy

I’m currently reading the book ”The psychology of money” (highly recommended) and the author makes the point about the impact of chance and risk. He states that the more extreme the outcome, the less likely you can apply its lessons to your own life. Therefore, you should look at broad patterns and not individual cases.

As a data scientist, this immediately got me thinking about outliers in statistics and machine learning. Just like machine learning models, we are swayed by outliers. To showcase the impact of outliers in statistics/machine learning, a linear regression model fit using ordinary least squares (OLS) can be used. Below we have a clear linear relationship between x and y:

Now, let’s visualize the effect of adding outliers of different distances on the resulting slope/model: