Earlier this year I made The Long Walk (of paintings). I had collected some 30,000 paintings from various museums for it. I was inspired to make that after seeing this by John Muyskens. I was looking back it that last week and was reminded of Aleksey Tikhonov's Average Art where he used over 18,000 portraits to construct an average face from them. I wanted to do some analysis over the dataset I had amassed for Long Walk. I ended up down a rabbit hole and expanding my initial dataset to include artworks from Wikidata, Wikimedia Commons, WikiArt, Museum open data, and Zenodo to have over 370,000 paintings. Here is a sliver of analysis I have completed over the course of about a week that I thought was interesting to share.
First, the data.
Then, using similar methods as Tikhonov, I created an average face. You can scroll through time and see how it changes.
Looking at the data, the number of faces in paintings decreases over time.
I thought that was interesting (since the trend also started before the invention of photography). What about people in general? Do people and figures follow the same trend?
They do. As I was pouring through samples of different figures, I noticed their poses changed over time. And that's where this rabbit hole starts.
What about poses?
To find the poses I ran a pose model (ViTPose) over every person detected in the paintings. It places 17 points on a body: the nose, eyes and ears, shoulders, elbows, wrists, hips, knees and ankles.
Running that over the course of a full day on my laptop gave me 278,164 figures, and 261,750 of them were clear enough to sort into one of twelve kinds of pose, from standing and facing out to lying down. Grouped together they make five families: standing still, seated, leaning or bending, an arm raised, and crouching, fallen or lying.
Standing still is the most common pose by far, but not by the same amount in every century. In the 1400s, 61% of figures are standing still. By around 1580 that falls to about 42% while leaning and bending peak, and by 1900 it's back near 60%.
It should be said that this pose model is not perfect! Especially when it comes to detecting poses in paintings of all things. But looking through samples myself I considered it good enough.
I then applied the same averaging to poses.
On the left, every figure in that pose from the fifty years around the year is warped onto the same skeleton and averaged together. On the right are the same figures drawn as stick figures.
With this skeleton pose dataset, we can now search for a specific pose and get figures that kind of match it. What if you created an animation of poses and created an average figure for each frame?
That's pretty cool in my opinion. CMU motion capture database has recordings of people walking, bowing, waving, sitting down, dancing, boxing and doing cartwheels. I applied them to the paintings as well. Some of these looked a bit cursed, but still, I think this is interesting.
Of course, if you can animate it you can control it. I cut walk, jump and punch from the motion capture into short loops to create spritesheets for a small game. Here's a simple beat 'em up game:
If you're interested in the data, you can email me and I can figure out a way to get you access.