Jan Van Balen

Infinite Oblique Strategies

2020-07-14

TL;DR I made a bot that tweets machine-generated "oblique strategies."
Go follow it here.

A 2013 edition of Oblique Strategies
CC-BY-SA Cory Doctorow

Wouldn't it be nice to have an excuse to jump on the 🤗 Train and try out their pretrained language models?

Here's how I used GPT-2 to generate infinite Brian Eno wisdom.

The data

In 1975, Brian Eno and Peter Schmidt published a set of cards designed to help overcome roadblocks in a creative process, and generally stimulate lateral thinking. Each card presents a suggestion or strategy. Here are some examples:

Brian Eno is a producer, so quite a few of the strategies relate to music-making:

There's only about 100 of these in total. That's almost nothing in the world of machine learning datasets. Still, I thought it would be nice to see if we can use a recent language model to generate more of these.

I started by getting the list of original strategies here.

The model

As a little proof of concept, I used Talk to Transformer, prompting it with 10 real strategies at a time (with a dash in front to tell the model it's a list):

- Look at a very small object, look at its centre
- Use an unacceptable color
- Don't be frightened of cliches
- Imagine the music as a set of disconnected events
- Shut the door and listen from outside
- Use an old idea
- Reverse
- Repetition is a form of change
- Use fewer notes
- Think of the radio

These are the best examples, others didn't make much sense.

I then installed Huggingface's Transformers library, and tried to do the same with their language modeling example script. This allowed for experiments with longer prompts and more output. I used the pretrained GPT-2 model, though to be honest, I didn't really compare the results to any other models (I hear you, reviewers...)

Results were initially less interesting but, after trying a few different settings of the temperature parameter, still OK:

High-quality environmental music

So I decided to steer it a bit more towards music, and other topics related to Eno's work.

First, I prepended, to the prompt, the liner notes from Eno's 1978 classic album Ambient 1: Music for Airports. It's a short piece of text that captures the ideas behind Ambient Music, one of the things Eno is famous for.

I have begun using the term Ambient Music. An ambience is defined as an atmosphere, or a surrounding influence: a tint. My intention is to produce original pieces ostensibly (but not exclusively) for particular times and situations with a view to building up a small but versatile catalogue of environmental music suited to a wide variety of moods and atmospheres. (...) Ambient Music must be able to accomodate many levels of listening attention without enforcing one in particular; it must be as ignorable as it is interesting.

Note that, while some of the statements above are simply metaphorical, they can be used to create high-quality environmental music to meet all the demands that our tastes allow us to (...)

Generally though this didn't happen too often as long as the number of strategies shown in the prompt was large enough (30–50).

The downside of adding more context to prompt was that I wasn't sure anymore whether the model's 1024-token attention span even allowed it to see any of it, so I started looking at the other natural way to instill some more Eno into my transformer: fine-tuning it.

Fine-tuning

Mixed results. I could tell that it picked up on the format (unconditionally generated text tends to have dates in it as little section headings, like the diary), and it picked up on the content. However, a lot of the training data consisted of excerpts from Eno's 1995 diary, which turns out to talk about all kinds of topics—letters, performance ideas, airports, charity work—and only a little bit about music. And so the model produces sentences like: "This project has been difficult, but it was something that I have been wanting for quite some time". Good for you, bot, but not too helpful for me.

Still, some good things came out!

An infinite Eno

To my surprise, some of these generated strategies look like they could be just as helpful as the original, or at least to a sufficiently open-minded person. Perhaps current language models generate stuff that's just the right kind of weird? In general, I find generating 'creative material' at this level rather more promising (for fruitful human-computer collaboration) than the more common approach where entire pieces of music or audiofiles are generated. Something to think about.

In any case, I thought some of the results were good to enough to invest a bit of time figuring out how to share these in a more systematic way, and ended up putting together a Twitter bot. I also added in a bit of image composition to make the result more lively. (It struck me that Oblique Strategies–owners really like to photograph their cards on beautiful wooden surfaces. I guess I would do the same? I don't have a set.) Anyway, follow @infinite_eno on Twitter for hourly obliqueness.

For now, it's tweeting both fine-tuned and non-fine-tuned strategies. Many of them are still bad. But hidden between the bad ones shines the occasional true Enoism:

Not a real card

Some observations

I mostly learned that you can go a long way without having to do any fine-tuning, you just need to get the prompts right. On the other hand, this didn't suprise me too much, having seen some of the things other people have been able to do. In particular, there's the lists Janelle Shane has been coming up with. They are as uncanny as they are hilarious. Definitely go follow her experiments if you don't already.

Some weeks after I started this project, OpenAI published GPT-3. One of their main findings is that a large pre-trained model can effectively adapt to all kinds of unseen contexts and tasks, without fine-tuning. Google's T5 was first hint of this trend a little before GPT-3. With the little bit of experience I now have in not fine-tuning models, I can definitely see this develop further still.

Finally, I can now also confirm that, yes, the 🤗 Transformers library is super helpful, at least for this kind of project. They did a great job making some of the most complex and powerful machine learning technologies accessible with very few headaches.

All this time, I'm aware that I might not be the first to try and generate Oblique Strategies. On this, however, I can only agree with GPT-2 Eno:

Don't count on it being original