Local Minima, Saddle Points, and Plateaus
theorangeduck.comThis made me think about code. I've long thought of incremental refactoring as "following the gradient" of good design principles, but this essay opened up new connections for me.
For example, I've heard elsewhere that great programmers possess the skill of being able to decompose large changes into smaller commits, making the code no worse as they go. Having the ability to see the small changes that get you where you want to go corresponds to finding the direction of change that gets you past the saddle point.
Random restart is a standard tool in the kit of stochastic gradient descent. But if local minima are rare, it may be overused. You don't have to rewrite your code if you can see the refactors and gradual improvements that get you where you need to go.
I enjoyed this article. It got me thinking that one of the problems in life is that the loss function is unknown: we don't always know beforehand what would make us happy or satisfied. However, it's a lot easier to gauge one's current gradient: whether things are getting better or worse, or stagnant, as in plateau.
Life is reinforcement learning, the reward signal is dopamine :)
Side effects may include an opioid epidemic :(
I just wish they would say "minimum" when they mean minimum.
In real high-dimensional spaces, like life, exploring away from local minima and plateaus ("plateaux"?) is expensive. Figuring in the expense of exploration distorts the reward function intriguingly.
This translates well into advice for startups: Don't obsess over finding the perfect solution, but focus on getting something then making constant improvements to what you have. Don't try to jump to a global optimum straight away. There are always a huge number of options to choose from, and you can always improve.
I love this analogy. There's a great lesson here in assuming the saddle point case and trying to eke out the additional gains you know could be close by.
The tricky part in real life is figuring out the dimensionality and then reasoning about how changes in the gradient of one dimension appear to affect changes in the gradient of another.
While I believe to understand the general idea behind the theory as applied to real life, I'm struggling with a few concepts.
What are local minima, in relation to the real life? Are those the "places", mentally speaking, where one feels like they're failing?
What, exactly, are saddle points in this context?
>What are local minima, in relation to the real life? Are those the "places", mentally speaking, where one feels like they're failing?
They are places where you're doing OK but can't seem to make progress to "better than OK" by moving a little away from them (e.g. for many steps away from that situation, things are getting worse -- you need to travel a lot to get out of the "pit" you're in).
Consider someone obese. They feel worse when doing exercize/diet (compared to e.g. munching a pizza), but the feelings they have are not the best they could have related to their weight situation -- they'd feel best if they could persevere and eventually feel fit/healthy.
Saddle points are those where you can't seem to make easy progress on one direction (one aspect, e.g. how you feel about your health) but you can on another (e.g. how much money you make).
It's a little more involved (because if you're obese you're always best to lose, whereas a function can try both increasing and decreasing values to get out of a local minimum) but that's the gist I think.
Thank you.
I'm still having trouble visualizing it, though. I reckon it's the kind of an idea where, if you don't give me a clear-enough image, I'm stuck no matter how well you pick your words others.
Not to put any kind of pressure on you, of course. I'm grateful for the explanation. It's my brain that does its tricks.