We just finished a quick “risk in optimization series.” Here at Win Vector we think this is a neglected topic that can easily be adapted to improve and even save optimization projects.
Our series was:
- Modeling Uncertainty For Better Outcomes: here we argue for adopting “expectation minus k standard deviations” as your objective function to improve project outcomes.
- Reducing Optimization Risk through Portfolio Methods: here we adapt the previous method to constraints by encoding diversity of solution as an anti-fragile reducer of result risk.
- Save Your Optimization Project From Overestimation Bias. The concept we were working up to (with Nina Zumel). We demonstrate an “over-tuning trap” in some optimization set ups, that looks a lot like over-fitting. We show how to mitigate the issue with out of sample early stopping ideas.
We think the issues illustrated above already have hurt many practical optimization projects. These examples may make the issues easier to visualize and identify in your own projects. We also feel that once you know to look for these issues, they can be efficiently mitigated using the methods we outlined above.
Video intro:
Categories: Exciting Techniques Tutorials
Tagged as: operations research Optimization overfit
