New transformer arch emulates imagination and higher-level human mental states

"Inspired by recent cellular neurobiological evidence linking neocortical pyramidal cells to distinct mental states, this work shows how models (e.g., transformers) can emulate high-level perceptual processing and awake thought (imagination) states to pre-select relevant information before applying attention."

"This model works by pre-selecting relevant information and identifying the most salient parts of it before applying its full attention to it."