Where is the Love in Renoir?

Renoir has an image problem. He painted rosy cheeks, picnic lunches and mothers with babies, and for a century that has been enough for some people to file him under sweet and move on. As the London exhibition opens, the critics of two British papers start by defending him. Alastair Sooke’s review in The Telegraph is headlined with the claim that Renoir is not kitsch nonsense. Jonathan Jones, in The Guardian, goes further and calls him the most radical of the Impressionists.

The exhibition is Renoir and Love, and it opens at the National Gallery on 3 October. It began at the Musée d’Orsay in Paris under the title Renoir et l’amour. La modernité heureuse, and it will end its tour at the Museum of Fine Arts in Boston in 2027. Curators from all three museums worked on it. The London press release promises more than fifty works from the mid 1860s to the mid 1880s, and the first visit to the UK of Bal au Moulin de la Galette.

The thesis is easy to state and harder to argue. Love, for these curators, is wider than romance. It covers flirting, conversation, friends at a café, men together in a boat, parents with children. Christopher Riopelle, the London curator, says Renoir chronicled love and friendship as “keys to modern life”. The Apollo review of the Paris version reports the same argument: these pictures of relationships are more modern than their reputation suggests.

Renoir showed La Loge, a couple in a theatre box, at the first Impressionist exhibition in 1874. He was already painting people who sit close and look at something else, which is close to what the curators mean by love as a way of seeing modern life. It is also something a computer can measure, because it is a relation between figures: two bodies, how near, turned which way.

So I asked a narrow question. An image model has seen millions of pictures with captions, so it must hold some idea of what love looks like. Can I extract that idea, and then ask the model to point at the part of a painting where it sees it?

A direction called love

The model is CLIP, released by OpenAI and trained on about 400 million pairs of images and captions collected from the web. It turns an image into a list of 512 numbers, and turns a sentence into a list of 512 numbers in the same space, so a photo of a dog and the words “a dog” land close together. Those lists are called embeddings. I used the ViT-B/16 version, which runs comfortably on a laptop.

A space has directions. In the word embeddings of 2013, king minus man plus woman landed near queen. The same arithmetic works here. Take the average embedding of sentences like “a couple in a tender embrace”, subtract the average of sentences like “a still life of flowers”, and what is left is an arrow that points from flowers toward couples. I treat that arrow as the model’s idea of love. A painting’s love score is how far its embedding leans along it.

I use sentences because the word “love” alone drags in valentines and hearts, while a sentence about two people in an embrace pulls toward what I am after.

pos = embed(["a couple in a tender embrace",
             "people in love looking at each other",
             "an intimate loving moment"])       # three of six
neg = embed(["a still life of flowers",
             "a painting of a building",
             "an abstract pattern"])             # three of six
love = F.normalize(pos.mean(0).sub(neg.mean(0)), dim=0)
score = image_embedding @ love

The same recipe gives three more arrows. Tenderness uses sentences about a mother holding her child. Festivity uses friends dancing and laughing at a party. Loneliness is a control: sentences about a person alone in an empty room. Every score is then converted to a z-score across the whole collection, so +4 means four standard deviations above the typical Renoir in my set. Raw cosine similarities are small and tell you little on their own, while a z-score tells you where a painting stands among the others.

None of my sentences mentions a book, a reader, a theatre or a lamp. Keep that in mind for later.

Gathering the paintings

I downloaded about a thousand images of Renoir’s paintings from Wikimedia Commons, dated 1860 to 1900 and compared their embeddings to remove near duplicates. The collection I kept runs from 1860 to 1893, and most of it, 614 paintings, falls in the years the exhibition covers.

The set is messy in the way real collections are. The same painting often exists in three photographs of different quality, some files are details of larger canvases, and a few were scanned from books. The duplicate filter catches most of these and not all. I kept the mess on purpose, because a cleaner set would have been a smaller one.

Three kinds of picture

This piece shows several kinds of image, and they answer different questions, so it is worth separating them.

The first is the score: one number per painting, the lean of the whole image along an arrow. It says how much, and it says nothing about where.

The second is the hiding map. Cut the painting into an 8 by 8 grid, paint a 3 by 3 block grey, and score the picture again. Do this in all 36 positions. If the score falls when a block is hidden, that block was doing work for the model. Average the drops over the cells each block covers and you have a map. It answers a causal question: what does the model need in order to see love here?

def occlusion_map(image, love, grid=8, win=3):
    base = score(image, love)
    total = np.zeros((grid, grid))
    count = np.zeros((grid, grid))
    steps = len(range(win, grid + 1))
    for i in range(steps):
        for j in range(steps):
            hidden = paint_grey(image, i, j, win, grid)
            drop = np.subtract(base, score(hidden, love))
            total[i:i + win, j:j + win] += drop
            count[i:i + win, j:j + win] += 1
    return total / count

The figure shows the test on a single painting. On the left is Dans la loge as it is, with a love score of +2.5. In the middle a grey square covers the block whose loss hurts the score most, and the score falls to zero, an average Renoir. On the right the square sits on the block that matters least, over his beard, and the score stays at +2.1.

So hiding the woman’s face, her high hair and the gap beside the man’s head costs the whole score, while hiding his beard costs almost nothing. A full map is the result of repeating this in every position.

The third kind is the patch map. Instead of hiding anything, it reads every small square of the image straight from the model and scores each one on its own. The method is called MaskCLIP. It takes the last layer of the image encoder, keeps only the path that carries each patch’s own content, and projects it into the same space as the sentences.

y = last_block.ln_1(tokens)                   # tokens of the last transformer block
v = y @ W_v.T + b_v                           # keep only the value path
patch = F.normalize(ln_post(out_proj(v)) @ proj, dim=-1)
patch_score = patch @ love                    # one number per patch

The two maps do not say the same thing, and the difference is useful. The hiding map says what must be present. The patch map says where a small piece already looks like love by itself.

In Confidences the hiding map gives a wide glow around both heads, because the blocks are large: each one covers more than a third of the canvas on each side. The patch map lands on a much smaller spot, the cheek where one woman’s mouth meets the other’s face. They agree on the general area and disagree on the exact spot. Across the collection their median rank correlation is 0.37, which I count as moderate agreement.

The cover uses the patch map. Every other map in this piece is a hiding map, except the right panel of the Confidences figure. Two rules apply to all of them. Each one is scaled inside its own painting, so the brightest spot is always that painting’s own maximum, and a bright map on a low scoring canvas shows only where a small amount sits. The amount is in the score. And to keep the pictures readable, I paint only the upper part of each map’s range in orange.

What the model likes

I expected kisses at the top of the ranking. There are none. The highest scores go to people bent over a page or a whisper: La lecture from 1890 (+4.9), Confidences (+4.2), a double portrait of Edmond Renoir and Marguerite Legrand reading together (+3.9), La leçon d’écriture (+3.7).

Remember that no sentence of mine mentions a book. The model found reading on its own.

Dante would not be surprised. In the fifth canto of the Inferno, Paolo and Francesca explain how they fell in love: they were reading about Lancelot, alone, and the book did the rest. Galeotto fu ‘l libro e chi lo scrisse: the book was the go-between, and so was its author. CLIP seems to share the association. Two heads over one page means something is happening.

Not all of these pairs are lovers. Several are sisters, friends or a woman and a girl, and the model treats them all alike. That is also how the exhibition uses the word: love as a wide family of closeness, from the confidences of friends to courtship. I do not believe the model knows this. It may only be noticing two faces very near each other.

Trying to catch it cheating

Renoir’s warm palette is the first suspect. I correlated the love score with how red each painting is (red minus blue) and got 0.06. I also scored grayscale copies: the ranking stays close, with a correlation of 0.80. Pink cheeks do not explain the score.

My first arrow had a worse problem. Its score for loneliness correlated 0.58 with its score for love, because both mostly detected people up close in an interior. So I built a second arrow for “people are present”, removed it from the love arrow, and ran everything again. The ranking barely moved (0.92 against the first version), but the link to loneliness fell to 0.42. That is the arrow used throughout this piece.

There is a fairer objection, and I raised it against myself: my sentences describe couples and embraces, so a model that rewards couples proves little. What was not in the sentences is what rose to the top. None of them mentions a book, a whisper or a lamp, and the best scores still go to people reading and confiding. To see how much depends on my wording, I rebuilt the arrow three more times: with paraphrases that avoid embraces, with sentences that mention no people at all (“a love story”, “romantic atmosphere”), and with six abstract words such as love and devotion. The rankings agree with the original at 0.93, 0.92 and 0.80, and La lecture comes first every time. The arrow is a probe, and the honest claim concerns what the model puts inside the word love.

The maps also have a habit I should own up to. In 80 percent of the paintings the hottest cell falls in one of the two central columns of the grid. Renoir painted his people near the middle, so part of this is real, but I cannot rule out that the method itself pulls toward the center.

Some limits remain. Love and tenderness still correlate at 0.72, so Renoir’s mothers and children score much like lovers. The arrow itself shifts with the wording, since two arrows built from disjoint halves of my prompts have a cosine similarity of only 0.47, though the ranking survives, as the check above shows. And CLIP learned from captions on the web. I suspect its idea of love has been shaped by wedding photos and stock images of couples.

Love and festivity overlap

I built an arrow for festivity to see whether the model can tell a love scene from a party. It can, but only in part. Across the collection the two scores travel together, with a correlation of 0.61, which makes sense: a love scene and a party both contain people being close, and for this model closeness is most of what love is made of. A painting full of people near each other tends to score high on both.

The exceptions are the interesting part. The garden parties sit well above the line. Sous la tonnelle du Moulin de la Galette scores +3.5 for festivity and only +0.6 for love. Luncheon of the boating party has the highest festivity score in the set, +4.2, and the same +0.6 for love. The file I have of that one is a photograph of a copy taken in Cromer, not a museum scan, so its point is noisy. At the other end, Patineurs from 1868 sits at the far left with a love score of minus 2.3.

Put Sous la tonnelle next to Confidences and the distinction the model makes becomes visible: a whispering pair scores high on love, a table of friends scores high on festivity. The maps would add little to this comparison, because the two arrows share the same ingredient, people close together, and covering the people lowers both. Across the collection the hiding maps for love and festivity have a median rank correlation of 0.76, and for Sous la tonnelle it is 0.93. Both arrows look at the same table and weigh what they see differently.

So the model separates a love scene from a party in the clear cases and mixes them up in the rest.

What a map shows when the score is low

Every painting gets a map, including the ones that score low or around the average. Here are three of them: a crowd of skaters on the ice, two people in a boat on the Seine, and a snowy corner of a village.

In each one the glow sits on something: the thickest knot of skaters, the boat and its two passengers, a roofed hut. Since every map is scaled inside its own painting, the brightest spot is always a maximum, even when the maximum is tiny. The skaters score minus 2.3, so their map tells you where the model looked and says nothing about love. Read the map together with the score, never alone.

Where the critics and the model part ways

Jones praises the early Renoir for a joy that refuses cynicism, and Sooke admires the same years while dismissing The Great Bathers of the mid 1880s as soft pornography. The exhibition itself stops in 1885.

The model has another calendar. Averaging love scores by period, the 1860s sit lowest, at 0.6 standard deviations below the collection’s mean. The score climbs through the 1870s, dips a little in the early 1880s, and reaches its highest average, +0.25, in the early 1890s. The 614 paintings inside the exhibition window average slightly below the 315 outside it, which are mostly later works.

I have not checked how much of this comes from what Renoir chose to paint in each decade. A run of outdoor scenes can pull an average down as easily as a change of heart. As a place to start an argument it is good: by this measure the model finds more love in late Renoir, the soft domestic scenes of the late 1880s and early 1890s, than in the years the curators picked. Two paintings I would most like to add to the set are Bal au Moulin de la Galette and Les Parapluies, and neither is in it yet.

What the model shows about us

So where is the love in Renoir? The model gives an address: the narrow space between two heads, a shared page, a whisper. The address says as much about us as about the paintings. The model learned from captions written by millions of people, and when it glows in that place, it reports what we usually point at when we say a picture is about love. Dante wrote that habit into the fifth canto of the Inferno seven centuries ago, and Renoir painted two people bent over one sheet in 1877. The map is a portrait of how we recognise a feeling, drawn by a machine that has never had it.

When the exhibition opens, try the test without the map. Stand in front of a canvas, let your eye settle, and notice where it goes. That is the model’s answer too, and it was ours first.