Web applications like EDHRec and TappedOut’s Deck Space visualizer (unfortunately an account is needed to use it) are really cool examples of applying machine learning to have more fun in the real world. For many interesting applications of machine learning to my daily life, the bottleneck is getting access to good data, which I am often far too lazy or understimulated to actually get around to collecting. Thankfully, we can get an up-to-date corpus of individual card metadata from Scryfall, and there are lots of ways you can collect decklists from the web, you can start with this dataset on Kaggle. Here are some problems I have been working on.
If you are interested in hearing more, and potentially getting early access to the product I am building in this space, send me an email at mtg-ml@derekrodriguez.dev!
Challenge 1: Given a partially complete Commander deck, recommend “good” cards to complete the deck
My current attempt at tackling this is hosted here. I didn’t get as far as I would have liked, but I made a lot of progress this year thanks to frequently leaning on LLMs for assistance with much of the infrastructural work. I am currently approaching this with a simple NMF model, due to my familiarity with that algorithm from prior work. It’s okay, but still much worse than just picking popular synergies off of EDHRec. Many thanks to my friends for testing this and helping out with evals. If you can’t tell, this blog post is also my first trailer for what I have been working on over the past few years.
I think a “good” recommender system in this space needs to be very steerable, as cards synergize in very interesting but complex ways. I think it is also essential for the system to be somewhat cost conscious, as it is quite easy for a 100-card commander deck to cost more than $450 when purchasing the deck as singles.
I am currently experimenting with using a ~100M parameter Set Transformer, but it’s still struggling with odd behavior, which is much more difficult to debug than the simple NMF model. The new model is obsessed with cards without oracle text, which literally never appear in the training set, I might cull cards which have never appeared in my decklists entirely.
Challenge 2: Scanning and automating card inventory with a smartphone camera
There has been a lot of interesting work in this space, so I don’t know if there is much to add here which isn’t just focusing on cost optimization. ManaBox and DragonShield’s app both offer impressive implementations of efficient and fast card scanning. That being said, I think iOS in particular has an enormous trove of impressive features which can be run entirely on the phone, and are currently massively underutilized by the existing crop of card flipping / deck management apps. Scanning cards using the photogrammetry tooling provided by RealityKit could perhaps one day be a cheaper and more immediate (albeit less thorough) alternative to getting a card graded by PSA or TAG.
The existing crop of mobile apps appear to do much better than my multi-day effort tweaking Tesseract, but I wonder if this isn’t a case where it could be very easy to synthesize some data by building some really simple scenes in Blender, and then swapping a texture mapped to a card object in that scene. While my efforts with Tesseract where less than fruitful, The Vertex AI OCR API did seem to work quite well out of the box, and I have not tested Apple’s RecognizedDocumentsRequest function provisioned through Vision (WWDC video here). I think the GCP API will be a solid baseline, although for pretty evident reasons it will suffer from not being able to tell you which set/printing the card from just the text, since I doubt you can actually rely on scanning the set and card number at the very bottom. Here’s an example image from a relatively recent iPhone for testing, in case you’d like to see for yourself!
Challenge 3: Given a deck of non-land cards, optimize a deck’s land curve given a cost constraint
Tweaking a deck’s land curve under a cost constraint is often an interesting exercise in probabilistic programming. Most of the more casual magic players I talk to have a default starting heuristic which is something along the lines of “lands = median color pips times 11 minus mana dorks”, whereas the more competitive players will use spreadsheets to simulate their curve precisely. I think there are plenty of tools out there which should let me (and a chatbot) provide a pretty good solution to this, we’re mostly just playing around with a hypergeometric distribution.