All the information (theory) is on the task

We're currently in the middle of the 22^nd series of Taskmaster¹ here in the UK which - for the uninitiated - involves a group of (pseudo-) celebrities competing to perform a variety of deeply silly tasks. The contestant lineups vary but are similar in spirit - usually a combination of comedians, TV presenters, and increasingly a token member from across the pond.

The contestant that I've been thinking about (and there's always one!) is the absolutely awful one - failing almost every task thrown at them through incompetence, indifference or outright malice. Miraculously though, they always seem to manage to win at least one episode, no matter how hard they try to avoid doing so.

The questions I need to answer, nay, the nation demands answers to, are the following:

• Is this a miracle?
• If so, is $P(x\text{ wins episode } | \text{ }x\text{ formed of pure chaos})$ statistically surprising?
• Can I jam some physics into the answer regardless? (I think we know this one. Is it foreshadowing given it's been fore-foreshadowed in the post description? Is this a post-foreshadowing?)

Let's start, as all good things do, with some data.

Fetching the data

Fortunately the UK is blessed with an abundance of nerds eccentrics, who have gathered every possible detail about the aired episodes of Taskmaster, no matter how small. In this case I used:

• silverdavis GitHub repo² of per-task scores...
• ...compiled from the Taskmaster Wiki³...
• ...cross-checked with the TdlM database⁴...
• ...with support from TaskMaster.Info⁵ for confirmation of the task ordering and formats.

The cleaned CSV file contains 182 episodes from series 1-20, including 940 tasks and 4,700 scores. (Special and tiebreak tasks are not included.) You can view and download the dataset here:

To compare contestants across different series, let's define a 'win credit':

• If you win an episode, you get 1 credit
• If you draw with, say, 2 other contestants, you get $1/3$ credits

Then, per series, we rank contestants from strongest to weakest using 'task credits' - a normalised score per-task, assuming $N$ tasks in the series:

Sorting contestants this way, then summing up all win credits from all series, we get the following:

So far, so unsurprising - the strongest contestant in each series wins the most episodes. No miracles just yet. The question we need to ask ourselves though, is whether this distribution is what we'd expect?

Taskmaster is filmed in a particular fashion - the contestants perform their tasks in quick succession, usually independently. Later on, all contestants are brought together to film in-studio segments where the task performances are reviewed and scored. This approach gives the nefarious task scheduler the chance to allocate tasks to episodes in any order they like - they might choose to pick all of the strong task performances for the weakest performer and place them in one episode, just to give them an (undeserved) chance to win an episode.

(Very much like gerrymandering⁶, though with more rubber ducks involved and - probably - a lower chance of triggering the inexorable descent into autocracy.)

We can perform a numerical experiment to investigate this possibility: shuffle the tasks amongst episodes while retaining the per-task scores. The per-series strong/weak classification of contestants isn't affected, but the win credits are. After performing many shufflings and averaging the results, here's what we see:

Ah-hah! I don't need to be no stinkin' statistician to spot that those bars are different heights. As the foreshadowing above predicted, the weakest contestants are observed to perform more strongly than average, and everyone else more weakly.

This is certainly suggestive, but we'll need to be more confident than that to convince the nerds eccentrics of the distributed Taskmaster fandom - let's continue down the rabbit hole.

Choosing a model

Let's get clear with some definitions before continuing:

• There are a huge number of valid Taskmaster schedules across 20 series - together they form a set $\Omega$
• A particular valid schedule is $x \in \Omega$.
• The weakest contestant scores win credits $W(x)$ under the schedule $x$.
• The observed weakest-contestant-win-credit is $W_{\text{obs}} \approx 23$, and the shuffled average is $W_0 \approx 17$.

Let's assume at zero^th order that the probability of seeing any given schedule amongst all possibilities is uniform, i.e.

This is the model that the 'Shuffled average' bars above represent - when shuffling, we just pick schedules uniformly at random.

Clearly though in reality this model is incorrect, otherwise the bars above would match. We need to deform $P_0$ in a sensible way, then measure how much deformation is required to match the observation (and whether this deformation is significant!)

To pick a 'good' deformation I'd usually reach for physical intuition, which is sadly not possible here. (Or is it? Post-post-foreshadowing count: 2.) Instead, we can peer across the fence to our friends in the information theory department, and borrow their concept of relative entropy: a metric to compare probability distributions. The deformed distribution with the lowest relative entropy is the 'closest' to $P_0$, and therefore requires the fewest additional assumptions to reach.

We'll use the Kullback-Leibler divergence⁷ to measure how much a new distribution $Q(x)$ differs, defined as

We'd like to minimise this quantity, while also ensuring $\sum_xQ(x) = 1$, and $\sum_xQ(x)W(x) = W_{\text{obs}}$. We therefore have a constrained optimisation problem, so turn to the tool of Lagrange multipliers⁸, resulting in the Lagrangian

where I've simplified some notation and added arbitrary constants $c_1$ and $c_2$. Proceeding by differentiating with respect to a particular $Q(x)$,

and re-arranging and renaming to different arbitrary constants we have

We know that $Q(x)$ is normalised, so we can write the normalisation constant out the front as

To get a feel for what $\lambda$ represents, consider two schedules $x_1$ and $x_2$ that have $W(x_1) = W_1$ and $W(x_2) = W_2$. Then

i.e. for every unit increase in $W$, the probability of that schedule occurring is boosted by $e^\lambda$. Positive $\lambda$ therefore increases the probability of schedules favouring the weakest contestant by boosting their average score $W$.

The difference between $P(x)$ and $P(W = w)$

Before we continue, it's useful to spend a little time discussing how the distribution of schedules ($x$) is related to that of win credits ($W$) - the former can be uniform while the latter is not. Define the distribution over $W$ like

i.e. the probability of a particular $w$ is the sum of probabilities over schedules that produce this score.

Consider a toy example with three schedules $x_{1-3}$, each with uniform probability $P_0(x_i) = 1/3$. Each schedule has a different score: $W(x_1)=0$, $W(x_2)=1/3$ and $W(x_3)=2/3$.

The toy partition function is therefore

and the probabilities for $W$ look like:

We can plot how these probabilities change as we vary $\lambda$, and how those changes affect the expected value of $W$ as it varies: . $\mathbb{E}_{\lambda}(W) = \sum_w wQ(W=w)$.

As expected, higher $\lambda$ means that stronger schedules have a higher probability of appearing, and so the expected score $\mathbb{E}_{\lambda}(W)$ increases.

It's also helpful to express $Q$ and $\mathcal{Z}$ in terms of $W$ rather than $x$:

Turning back to the observed Taskmaster scores, we can plot our estimate of $P_0(W=w)$ alongside an estimate of $Q_{\lambda}(W=w)$ (constructed by drawing $x$ from $P_0(x)$ with altered probabilities $Q_{\lambda}(x)$ via inverse-transform sampling). A few things stand out:

• $P_0(W=w)$ is centred around the shuffled average $W_0$ as expected.
• $W_{\mathrm{obs}}$ lies quite far from the centre of $P_0$ - this is helpful to know, as it quantifies the surprise we should feel at seeing a discrepancy this large.
• To get the peak of $Q_{\lambda}$ to match $W_{\mathrm{obs}}$, we need to increase $\lambda$ to ~0.65

This is helpful! We now have a rough estimate $\hat{\lambda} \approx 0.65$. The next questions to ask: how sure are we of this value? Are the observed scores also compatible with $\lambda = 0$? But first, a quick hop back over the fence into the physics department. (They've got better coffee there anyway.)

Why $\mathcal{Z}(\lambda)$

We choose to write $\mathcal{Z}$ here not because it's cool to use \mathcal{Z} all over the place (it is), but because this looks very much like a partition function⁹ from statistical mechanics. Continuing the analogy, we can identify $x$ as a microstate of the system, and $W(x)$ as a measurable macroscopic property of the system. You might like to think of $x$ as the configuration of atomic spins in a magnet¹⁰ - completely unmeasurable by themselves. However the overall magnetisation is measurable, corresponding to our $W(x)$. By applying an external magnetic field, you can bias the probability distribution of the microscopic atomic spins towards particular configurations, manifesting in a macroscopic change to the measurable magnetisation.

In this sense, we can identify $\lambda$ as a kind of editorial field - an external pressure applied to the system (task schedule) to modify an observable property (scores of weakest contestants). It has units of $1/($Taskmaster points$)$, known as the Horne (Hn). At this point in our analysis we can conclude a rough measurement of a field strength of $\lambda \approx$ 0.65$\,$Hn, to my knowledge the strongest editorial field strength yet reported in the literature. An Ignoble is surely on the way!

Using $\mathcal{Z}(\lambda)$

Let's look now at actually using $\mathcal{Z}$ to do something useful - finding which values of $\lambda$ are most likely. We know the probability for $w$ given $\lambda$ already, so we can write down the likelihood for $\lambda$ given the observed $W_{\mathrm{obs}}$:

Defining

then the log-likelihood looks like

Differentiating with respect to $\lambda$ the first term disappears and so

Computing the second term we get the nice result that

i.e. the derivative of the log partition function is just the expected value of $W$! (This is why physicists love exponentials so much.) At maximum likelihood

i.e. at the optimal value of $\lambda = \hat{\lambda}$ ($\approx$ 0.65), the likelihood is maximised and the expected value of $W$ under this model is the one which we observe. This is exactly the procedure for the interactive plot above - we varied $\lambda$ until $\mathbb{E}_{\lambda}(W)=W_{\mathrm{obs}}$.

To see that this is a maximum, we can differentiate again to uncover another delightful identity

which is always below zero since variance is a positive quantity.

Approximating $P_0(w)$

To better illustrate the discussion above, let's choose an explicit form for $P_0(W = w)$ - a normal distribution with mean $W_0$ and variance $\sigma^2$:

The partition function is then

and hence the following identities for this model:

The plot below compares the normal approxmations for $P_0$ and $Q_{\lambda}$ - the former is pretty good, the latter starts to fail for $\lambda$ far from zero as the assumption of infinite support for $w$ starts to fail:

There's one final point to note that brings us neatly back round to physics again. Note that

The left-hand side is the 'response' of the system - how far the macroscopic property $W$ moves as a result of the application of a force $\lambda$. The right-hand side is the natural fluctuation of the macroscopic property $W$ in the absence of any force. This is then a manifestation of the fluctuation-response relation¹¹ for this system - a relation observed in thermodynamic systems.

Does this observation ultimately justify the increasingly strained analogy of Taskmaster scheduling with a physical system? That you'll have to decide for yourself, reader.

Answering the question

Let's - finally - answer the question we posed ourselves long above: is it reasonable to conclude that a non-zero editorial force was applied to the Taskmaster schedules, gerrymandering the task/episode distribution?

To calculate a probability distribution for $\lambda$, we need to combine the computed likelihood from our data points with some assumed prior distribution $\pi(\lambda)$. Once we know this, we can compute

You can try different $\pi(\lambda)$ below and see what probability we assign to $\lambda > 0$ - for the flat prior we get 97%.

To capture the spread of predictions of these models, the following plots the area of the shaded region above ($\mathrm{Pr}(\lambda > 0)$) and also the Bayes factor $BF_{10}$¹² for each of the priors, parameterised by their relative 'spread' parameter $\sigma$ and keeping the mean prior value of $\lambda$ at zero. Loosely speaking, $BF_{10} > 1$ implies that the alternative model explains the data better than the null $\lambda = 0$ model.

Here we define the Bayes factor as the ratio of probabilities of observing $W_{\text{obs}}$ under the different models (or hypotheses, hence the $H$):

(If the lack of integral in the denominator bothers you, you can imagine that we're integrating over the null model that there is no field, i.e. $\delta(\lambda)$.)

Ultimately we observe some weak evidence under some priors that the editorial field is non-zero - $BF_{10} > 2$ - and reasonable evidence that if it does exist, it points in the positive direction.

So, success? 5 points for this analysis? Or would the Taskmaster take his usual dim view of over-eager contributions to the show and reward us with a swift nul points? I'm not sure we can derive that answer from observed data, so instead we'll have to wait until the inevitable series 23 and hope that the Horne field directs it's scheduling force towards tasks of a more statistically involved nature. Here's hoping!