Ask HN: Is Synthetic Data Just Repackaged Knowledge?

4 points by pal9000 a year ago · 1 comment · 1 min read

I often see synthetic data conflated as if it were equivalent to new real-world samples. In reality, when a model generates synthetic data from its own learned distribution, isn’t it just rearranging the information it already captured? Could an expert explain—using principles of information theory—why synthetic data might still improve a model’s performance despite not providing genuinely ‘novel’ information