Rohan Pandey (@khoomeik) on X

Post

Post

• 

  📢 Excited to finally be releasing my NeurIPS 2024 submission! Is Chinchilla universal? No! We find that: 1. language model scaling laws depend on data complexity 2. gzip effectively predicts scaling properties from training data As compressibility 📉, data preference 📈. 🧵⬇️

  

  

  Say your training compute budget = ~1.5e13 FLOPs If your dataset has a gzip compressibility ratio of 0.14, you should *max out your param count* and skimp on dataset size But if your dataset is less compressible (gzip=0.61), *keep your model small* and train it on a ton of data

  

  

• 

  Chinchilla claims their 1-to-1 parameter-data scaling law is agnostic to the type of textual training data used 🤨 But

  @ArmenAgha@AIatMeta

  find code-gen scaling prefers parameters 😳

  @deepseek_ai

  team further noticed that scaling with cleaner data also prefers parameters 🤔

  

  How could we control data complexity to explore data-dependent scaling properties? We modulate the syntactic🌳 properties of Probabilistic Context-free Grammars (PCFGs) to generate 6 synthetic datasets. Their gzip-compressibility is a good proxy for their syntactic complexity✅

  

  

  Now we can train models (4.4M - 1.4B params) on portions (100K - 100M tokens) of these datasets (0.12 - 0.60 gzip) and fit a scaling law for each dataset. But first, observe that regardless of model size, harder to compress datasets (e.g. yellow) require more compute to learn.

  

  

  From each dataset's fitted scaling law, we can derive its compute-optimal frontier (best parameter-data tradeoff ⚖️) and plot it beside Chinchilla's stationary 🏔️ frontier. We see that as data becomes less compressible, the frontier shallows out—a stronger parameter preference‼️

  

  

  We then fit linear regressions to predict scaling parameters from a dataset's gzip-compressibility. The regressions hold strongly across all parameters (except E) and are not confounded by the PCFGs' underlying syntactic parameters 🤩

  

  Finally, we present a simple re-parameterization of the Chinchilla scaling law to allow it to depend on training data's gzip-compressibility H. Since our experiments are relatively small, we also offer a formulation that adjusts the Chinchilla parameters by some ε.

  

  

  We're excited about 4 future directions 🔭: - information-theoretic investigation of why scaling parameters emerge - training compute-optimal code-gen models - using gzip in data filtering & curriculum learning - auto-allocating compute for each collected dataset's training runs

  

  

  All the code to generate PCFG datasets, measure gzip-compressibility, and run the training jobs is available in our GitHub repo! Would appreciate if you drop a ⭐️ too 🫡 github.com/KhoomeiK/compl… And of course here's our Arxiv link: arxiv.org/abs/2405.16684

  

• 

  Really impressive work! So what should we do if the data is a mixture of different compression ratio?