published length869 words read time5 min
First, a modeling progress update
Since my last post about interesting machine learning challenges in Magic: The Gathering, I’ve made some really promising progress on training a modified permutation-invariant transformer from Lee et al. that is worth pursuing fully.
To give a rough sense of how well the model can choose cards that fit a Commander
deck, here are results from my latest benchmark. I held out a subset of decks I
have played against and/or own, then had the model iteratively propose K
recommendations, scoring only cards from the original source decklist as relevant:
| Model | Parameters | K | Macro Recall | Macro NDCG | Macro MRR |
|---|---|---|---|---|---|
| Standard | 5,338,624 | 5 | 3.73% | 3.32% | 4.20% |
| Standard | 5,338,624 | 10 | 5.18% | 5.02% | 8.03% |
| Standard | 5,338,624 | 25 | 13.80% | 11.55% | 18.19% |
| Standard | 5,338,624 | 50 | 17.96% | 15.16% | 26.66% |
| Small | 1,453,568 | 5 | 3.12% | 2.83% | 4.82% |
| Small | 1,453,568 | 10 | 5.53% | 4.92% | 8.44% |
| Small | 1,453,568 | 25 | 11.31% | 9.56% | 18.10% |
| Small | 1,453,568 | 50 | 14.73% | 12.61% | 25.53% |
Translating from Results-Table to English:
- At an absolutely puny 1M and 5M parameter count, these models are both absolutely tiny compared to what’s whipped together by people competing at frontier labs. I think we just don’t need that scale. My caveman approach to preprocessing with a custom vocabulary size of 8,192 gets us 98.44% coverage across the 32,000+ cards in my decklist corpus. My current training runs expose the models to a rough equivalent of ~30B tokens, but it’s not super useful to think in pre-training terms here.
- A macro MRR of 18–25% suggests that useful cards tend to appear near the top of the recommendations.
- Macro recall of 17% impressed me given that the candidate pool includes cards from every set released from the ’90s to the present. On a random sampling of my not-so-competitive Commander decks, these models are starting to understand archetypes! There is nothing truly groundbreaking yet; another eval on cEDH decks had macro recall scoring 30%, which makes sense as competitive cEDH decks are likely to rely on the same pool of expensive cEDH staples.
- Macro NDCG below 20% across the board shows that we’re gonna have a bad time using the same cauldron of weights for both candidate selection and ranking. That’s fine. I think a two-stage system similar to the now classic YouTube paper will work. Throwing a reranker on top of the candidate selector gives us a lot more control over the nuance and vibe of our recommendations, but I’ll need good negative mining strategies to get us off the ground without user data.
None of these numbers fit my definition of “amazing”, it’s still worse than EDHRec I think. But my preliminary testing suggests that the model is good enough to throw into the increasingly popular agentic ablation pit.
The ferocious appetite of a million tiny experiments
I ran a training run or two on Modal and realized I was gonna be dropping rent check money if I wanted to go through every ablation under the sun. It’s so weird to me that “every ablation under the sun” is a handful of prompts away now. The bot is more diligent about using MLflow than I ever was. The bot can take my meager list of hyperparameter sweeps and come up with over 100 experiments. This is where the appetite for compute comes from. Am I part of the problem?
For better or worse, I have been a bit of a homelabber since before the current supply crunch. Pictured below is an old picture of my dual system setup with a Turing Pi 2 (product page only points to the revised 2.5 board these days), which I had originally backed several years ago. Until now, it has mostly run underwhelming quantized 8B models using now-outdated versions of Ollama that are compatible with my Jetson Xavier NX, which I originally acquired while working in computer vision so I could practice outside of work.
Thanks to the supply deficit of good machine learning hardware, I’ve decided to go the budget route and finally let my Turing Pi evolve into its final (budget) form instead of throwing used car money at a workstation GPU. These days, used Jetson Xavier NX modules can be had for pretty cheap! 64GB of memory sharded across 4 devices for a few hundred bucks should get me pretty far before I go back to doing training runs on Modal.
How I’ll actually use this hardware
The plan is to write experiment jobs that test whether a neat arXiv paper or hyperparameter tweak improves recall on my evals. Each job will run from a custom container image pushed to a local registry, with configuration supplied through command-line args or config files. When a job finishes, it will log its results and corresponding PyTorch checkpoints to MLflow. If there’s a better way to go about this, please let me know in the HN comments or by shooting me an email at [email protected].
Tentatively, I’m thinking of running K3s on each node, with my main Debian homelab box serving as the controller. That Debian box currently hosts game servers and a few other nifty services. Ideally, I’ll “just migrate everything,” but we’ll see how much I can get done. For now, I am waiting for the new hardware to arrive.