The Memory Decoding Challenge
Experiments that could win the prize
Concrete experimental proposals that, if successful, would decode a non-trivial memory from a static map of synaptic connectivity.
Version 0.9 — a work in progress. Spotted an error or have a suggestion? Email [email protected].
$100,000
The Memory Decoding Challenge is a $100,000 prize, offered by Aspirational Neuroscience, for the first team to decode a non-trivial memory (or other learned function) from a static map of synaptic connectivity. This page explains what we mean by a memory and by non-trivial, and then sketches five experiments that would be strong contenders to win the prize.
Eligibility
What counts as memory
In the broadest sense relevant to the challenge, a memory is something that specifies a particular animal’s lived experiences out of the much larger set of experiences creatures like it could conceivably have had. An experiment that could only decode behaviours that are generic to the species — such as reflexes, instincts, and innate sensory preferences — would not be eligible.
Additionally, not everything which is neural, influences behaviour, and is idiosyncratic to a particular animal counts as memory in the sense we care about. Brain damage from physical trauma is idiosyncratic to the injured animal, may change its behaviour, and is a record of its previous experiences — much as a scar is a record of a previous injury. Nonetheless, reading out the existence of a lesion from a connectome would not be a meaningful demonstration of memory decoding. The spirit of the challenge is closer to uncovering neural records of past experiences that guide future behaviour in an ethologically adaptive manner and that are specific to that animal.
The bar
What counts as non-trivial
One useful measure is how much information can be recovered about the animal’s specific history, quantified in bits. A decoding that simply revealed an animal had been exposed to a particular odor at some point in its life is less compelling than one that distinguished which of several similar odors it had learned to associate with a reward. An experiment that could reliably specify which odor out of thirty-two possibilities (five bits) or more would be clearly non-trivial.
Examples at either end of the memory-decoding spectrum can bound triviality. At one end, a complete, high-fidelity emulation of a fly that perfectly reproduced its learned preferences and aversions would definitively win the prize. At the other end, observing that a rodent had a typical visual cortex because it was reared in an environment with light — as contrasted against the abnormal cortex of another animal reared solely in the dark — would not win. Similarly, nor would a structural version of Xiong et al. (2015), who showed electrophysiologically (not structurally) that learning a tone discrimination selectively potentiates different corticostriatal synapses; recovering that one bit from synaptic structure alone would be impressive, but would not by itself win the prize.
Between these extremes lies a large space of experiments that may or may not qualify, and reasonable people will disagree about where exactly the line should fall. Rather than attempt a tighter formal definition, the proposals below sketch what a winning attempt might look like.
01
Decoding an odor association from a Drosophila mushroom body connectome
The fruit fly Drosophila melanogaster can learn to associate odors with reward or punishment, and the circuit responsible is among the best characterized learning circuits in any animal. Odors are represented in the mushroom body by sparse, combinatorial activity in Kenyon cells (KCs). Each KC receives input from several antennal lobe projection neurons and responds to a small subset of odors. During conditioning, dopaminergic neurons that innervate particular mushroom body compartments depress the synapses that active KCs make onto mushroom body output neurons (MBONs), shifting the balance of downstream approach and avoidance circuits. The learned association is therefore stored largely as compartment-specific depression of identifiable KC→MBON synapses.
A decoding experiment could train individual flies to associate one of about 32 discriminable odors with shock and then image its antennal lobes and mushroom bodies by electron microscopy (EM) or expansion microscopy (ExM). The decoder needs to know which KCs each candidate odor would activate. This cannot be looked up in a reference brain, because the wiring from projection neurons to KCs is effectively random and differs from fly to fly. However, what is the same in every fly is the identity of each antennal lobe glomerulus and its odor tuning. The decoder could therefore identify the glomerulus of origin of every projection neuron in the volume, assign each glomerulus its known odor responses, and trace the projection neuron inputs to every KC. From this it would predict the KCs activated by each odor, then look for depression at KC→MBON synapses in aversive compartments that is concentrated among the KCs of one odor’s predicted ensemble. The odor with the strongest signature is the answer, and a clean result would be worth about 5 bits.
Two things have to hold for this to work. First, the learned weight change must leave a detectable structural trace readable in a single fly, which has not yet been shown at KC→MBON synapses: light microscopy shows learning-induced changes in active-zone protein and active-zone number that fade within hours, yet flies still remember at 24 hours. Second, the KC ensembles predicted from wiring must be accurate enough, which can be checked in separate flies by imaging KC odor responses before EM.
As a warning though, not every odor-specific trace would be enough to win. Baltruschat et al. (2021) found that appetitive long-term memory increases the number of calyx microglomeruli formed by the conditioned odor’s projection neurons. While impressive work, those neurons are directly identifiable by glomerulus, so reading their structural changes only shows which input channel’s gain was raised. Randomizing valence along with odor — sugar for some flies, shock for others — would force the decoder to find the trace in the output compartments, adding the extra component that makes this non-trivial. It is additionally helpful that a wiring diagram of the entire adult fly brain is now available.
Figure 1. Decoding an odor association from a Drosophila mushroom body connectome. (A) A fly is trained to associate one of 32 candidate odors (here benzaldehyde) with shock in an iDOC chamber (adapted from Ortega et al. 2024), then tested for odor versus air preference; the performance index (PI) runs from +1 (toward the odor) to −1 (toward air). The boxed fly is imaged. (B) Top, fly head showing the mushroom body, antennal lobe and antenna (Heisenberg 2003). Bottom, benzaldehyde-responsive glomeruli (orange) drive a sparse ensemble of Kenyon cells (KCs) through wiring that differs between flies; shock-driven dopaminergic neurons (DAN) depress this ensemble’s synapses onto the output neuron (MBON) in compartment α2. (C) Hypothesized structural trace of this depression at a KC→MBON synapse. (D) Aggregate strength of each candidate odor’s predicted KC ensemble onto the MBON; the most depressed ensemble identifies the conditioned odor. Data in A and D are simulated.
Related work
Lappalainen et al. 2024
A network model given the measured connectivity of 64 cell types in the fly visual system, with unknown cellular parameters fitted to a task, predicted recorded neural activity. Support for the step from connectome to activity.
Shiu et al. 2024
A whole-brain model built from the fly connectome and used to study sensorimotor processing.
Baltruschat et al. 2021
Using EM, appetitive long-term memory was found to increase the number of calyx microglomeruli responding to the conditioned odor. A second structural trace of odor learning, upstream of the KC→MBON synapses.
02
Decoding spatial environment features from rodent CA3
Spatial memory is among the most intensively studied forms of learning in mammals, and hippocampal area CA3 is thought to play a central part in storing spatial representations. CA3 place cells fire when the animal occupies particular locations in an environment, and different environments recruit largely separate subsets of cells, so the active ensemble amounts to a sparse map of each environment. These maps are thought to be stored by potentiation at recurrent CA3→CA3 synapses, and at entorhinal inputs, between cells that are active together. If so, the pattern of strong recurrent connections should carry an imprint of the environments the animal has known.
A decoding experiment could expose rodents to one of several environments that differ in shape, boundary geometry or local cues, over enough sessions for place fields to stabilize, and then reconstruct CA3 by EM or ExM to estimate recurrent synaptic weights across a large population of pyramidal cells. Success depends on whether cells with overlapping place fields become more strongly connected. Attractor models predict that they should, and recent plasticity data are consistent with this (Li et al. 2024; Mishra et al. 2016), but it has not been tested directly. If it holds, the cells could be laid out in an abstract two-dimensional space so that strongly connected cells sit close together. The result would be a predicted map of place fields, and its geometry — a ring for a circular track, say, or an open sheet for a square arena — could be compared with each candidate environment. Even a binary distinction between two very different environments would be a useful proof of principle, and distinguishing among 32 would reach 5 bits.
The central assumption is shared with the other experiments: that the structural correlates of potentiation, such as larger spine heads or PSDs, are visible in the reconstruction and large enough relative to background variation to reveal the assemblies. CA3 additionally brings its own difficulties: a behaviorally meaningful recurrent network occupies a far larger volume than the fly mushroom body, and recurrent connectivity is denser than once assumed, so separating learning-related potentiation from baseline variation will need careful statistics. On the other hand, the reconstruction and analysis tools developed for cortical EM in the MICrONS project should transfer to hippocampal tissue, and recordings from the same animal before fixation could supply the place fields needed to validate any inference from connectivity.
Related work
Sammons et al. 2024
3D EM combined with paired recordings found CA3 connectivity several times higher than previously assumed, sufficient in models for pattern completion and sequence replay.
Uytiepo et al. 2025
EM after tagging the cells active during learning showed that these cells did not preferentially connect to one another, though they gained multisynaptic boutons. A challenge to the assumption that co-active cells become more strongly connected.
03
Decoding a fear-conditioned tone from rodent lateral amygdala
Along with spatial navigation, auditory fear conditioning is among the best mapped learning paradigms in mammals. Pairing a tone (the conditioned stimulus, CS) with foot shock potentiates the auditory inputs that carry that tone onto principal neurons of the lateral amygdala (LA), and also the connections between LA neurons recruited during learning. Serial EM shows that fear conditioning enlarges LA synapses, so at least part of this potentiation persists as a change in structure, presumably at the synapses that carry the CS.
A decoding experiment could condition a rodent to one tone drawn from a set of candidates, then acquire EM of the LA together with its inputs from auditory cortex. The frequency preference of each cortical input neuron projecting from auditory cortex to lateral amygdala could be estimated from its position on the tonotopic map. The decoder would then ask which tone, had it been conditioned, would best explain the pattern of potentiated cortico-amygdala and amygdala-amygdala synapses. Presenting the other candidate tones during training without shock would give within-animal negative controls, as mice can learn to freeze to one tone and not to another. The candidate tones would need to be well separated in frequency, and the number of bits would be set by how many such tones an animal can reliably discriminate.
The logic is the same as in the mushroom body experiment, but in mammalian tissue and with a much larger volume to image, since the relevant inputs arrive from distant structures. The central assumption, again, is that conditioning-induced strengthening is visible in EM. Identifying which engram in the connectome corresponds to which tone would be a non-trivial result.
Figure 2. Decoding a fear-conditioned tone from lateral amygdala connectivity. (A) A mouse is trained with a 30 kHz tone paired with foot shock (CS+) and a 5 kHz tone without shock (CS−). (B) Beforehand, distinct anterograde labels are injected into high-frequency (40 kHz) and low-frequency (5 kHz) sites of auditory cortex (AC), chosen from its tonotopic map (after Stiebler et al. 1997). (C) Labelled AC axons terminate in the lateral amygdala (LA), which is then reconstructed by EM or ExM. (D) LA→LA connection strengths, shown as a neuron-by-neuron matrix, are clustered into ensembles, revealing one strengthened ensemble (orange); the side strip marks each neuron’s labelled AC input. (E) Each ensemble’s mean connection strength plotted against its input frequency index; dot area shows synapse count. The strengthened ensemble receives mostly high-frequency input, a read-out of about 26 kHz that identifies the 30 kHz tone as the CS+. Data in D and E are simulated.
Related work
Choi et al. 2021
After tone fear conditioning, synapses from auditory cortex onto LA between activated ensembles had larger spines; extinction reversed the change. This is the tone-linked structural trace the experiment would try to read.
Abatis et al. 2024
Patch-clamp recordings from labelled lateral-amygdala engram cells after fear learning found more — and stronger — recurrent connections among them, suggesting each fear memory is stored as an autoassociative attractor network in the LA. This is the amygdala-to-amygdala strengthening the experiment would read.
Abdou et al. 2018
Two tone fear memories shared LA neurons, but manipulating the input synapses of one memory affected only that memory — suggesting the identity of each memory is carried by its input synapses.
Kim & Cho 2017
Mice froze to the conditioned tone but not to a control tone, and LTP appeared only in the auditory pathway carrying the conditioned tone.
04
Decoding visual experience from mouse V1 connectivity
The primary visual cortex (V1), where signals from the eyes enter the cortex, is a region capable of learning. Repeated passive exposure to oriented gratings induces stimulus-specific response plasticity (SRP) in adult mouse V1, with persistent enhancement of responses to the experienced orientation that is thought to involve LTP-like changes at thalamocortical and intracortical synapses serving that orientation.
A decoding experiment could begin by imaging the orientation tuning of neurons in a block of V1, presenting gratings at each of eight orientations equally often. One orientation would then be drawn at random and shown to the mouse in daily sessions over several days to weeks. Finally the imaged block — about 1 mm³ or less, and so comparable to the MICrONS volume — would be reconstructed by EM or ExM and registered to the functional data.
With no access to the draw, or to any recordings made after exposure, a decoding team could then try to recover the orientation from structural signs of orientation-specific potentiation. These might be found at synapses between neurons tuned to the trained orientation, or at thalamocortical inputs onto those neurons, with the tuning of unimaged neurons inferred from like-to-like connectivity. Measuring tuning before exposure matters because SRP is itself an enhanced response to the trained orientation, so any functional map recorded afterwards would give the answer away.
Eight orientations give 3 bits, short of the 5 bits mentioned above, so this version would be a proof of principle. Spatial frequency could perhaps be added as a second stimulus dimension to raise the count. The experiment may also be harder than the mushroom body or amygdala cases, if, as seems likely, SRP leaves smaller structural changes than associative conditioning does. An easier first step would compare mice whose visual experience was biased toward one orientation — restricting juvenile mice to contours of a single orientation, using cylinder-lens goggles, over-represents that orientation in V1. Telling vertical from horizontal would not by itself be enough to win, but it would be a valuable early test of the method.
Related work
Ding et al. 2025
Describes the like-to-like wiring rule that this experiment would use to infer tuning from connectivity.
El-Boustani et al. 2018
Receptive-field plasticity in mouse visual cortex was anchored by potentiation of identified spines, accompanied by depression of their neighbors. A spine-level signature of altered visual input.
Dorkenwald et al. 2022
In mouse V1, synapse size behaves like the sum of a binary and an analog component, and the binary component is shared by synapses between the same pair of cells. A candidate potentiated state to look for.
05
Playback of a zebra finch song from the HVC and RA connectome
Each adult male zebra finch sings a single stereotyped song, learned during a developmental critical period and unique to that bird. The motor circuit behind it is well mapped. Projection neurons in premotor nucleus HVC, the HVC(RA) neurons, each fire a single brief burst at a precise moment in the song motif. They drive RA, whose neurons fire several stereotyped bursts per motif, and RA in turn drives the brainstem motor pools controlling the syrinx and breathing.
The leading account of HVC’s sparse sequence is the synaptic chain model, in which each HVC(RA) neuron is driven to burst by those that burst just before it. Population imaging and recordings in singing birds (Picardo et al. 2016; Lynch et al. 2016) and a partial EM reconstruction of HVC (Kornfeld et al. 2017) are consistent with this picture. If the model is right, the temporal order of the song should be recoverable from the connections among HVC(RA) neurons alone, as Seung (2009) pointed out, and the connections from HVC to RA should encode how this particular bird converts that timing signal into motor commands.
A decoding experiment could use a single adult bird with a stable, crystallised song. First the song would be recorded. Then calcium activity would be imaged during singing: HVC(RA) neurons by two-photon imaging — and more ambitiously, also RA, which lies ~2 mm deep, by three-photon imaging or through an implanted microprism. After that, all of HVC (and possibly RA) would be acquired with EM or ExM (hundreds of teravoxels, comparable to MICrONS) and co-registered to the live imaging. From synapse size, contact number and dendritic location, the decoder would estimate a weight matrix among HVC(RA) neurons. A graph-layout method would then order the neurons in time without reference to the imaging, and that order would be checked against the imaged burst sequence.
A more ambitious version would include RA. The decoder would predict the activity of each RA neuron from its HVC inputs, and use the known dorsoventral topography of RA to group neurons by whether they drive syringeal or respiratory motor pools; the predicted drive could then be fed to a low-dimensional biomechanical model of the syrinx to synthesize a spectrogram. The experiment would likely succeed in stages. Recovering the burst sequence would show that the bird’s song timing is written in its wiring, and recovering the transformation from HVC to RA would show that its individual motor program is written there too. The strongest result — a synthesized audio track that resembles this bird’s own song more closely than the songs of other zebra finches — would clearly win the prize.
Related work
Fee & Scharff 2010
A review of the HVC-to-RA sequence-generating circuit and of song learning.
Conclusion
The experiments sketched above are illustrative rather than exhaustive. They share a common structure: identify a circuit where learning produces detectable structural changes, choose a stimulus space large enough for the decoding to be non-trivial, and use the connectome plus minimal functional ground truth to recover which stimulus the animal experienced. The space of viable experiments is likely substantially larger than the five covered here.
We encourage anyone considering work in this area to get in touch — particularly if you have an experiment in mind and aren’t sure whether a successful result would qualify, if you want clarification on the rules, or if you want to contribute funding, serve as a judge or advisor, or collaborate on an attempt.