🚨This week's top AI/ML research papers: - Model Spec Midtraining - Sparser, Faster, Lighter Transformer LMs - Continuous Latent Diffusion LM - Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting - GLM-5V-Turbo - Nonsense Helps - TIDE - Manifold Steering Reveals the
Model Spec Midtraining: Improving How Alignment Training Generalizes Overview: Model spec midtraining (MSM) injects synthetic documents detailing desired behavioral specifications into LLMs after pre-training to constrain how they generalize from subsequent alignment
Sparser, Faster, Lighter Transformer Language Models Author's Explanation: x.com/SakanaAILabs/s… Overview: Sparser, Faster, Lighter Transformer Language Models introduces a sparse packing format and custom CUDA kernels to execute unstructured sparsity within LLM feedforward
Continuous Latent Diffusion Language Model Overview: Cola DLM utilizes a hierarchical latent diffusion framework to decouple global semantic organization from local token realization. By employing a Text VAE for latent mapping and a block-causal DiT for prior modeling, this
Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting Overview: Sharpness-Aware Minimization, large learning rates, and shortened annealing periods bias LLM pretraining toward flatter minima to preserve capabilities during subsequent parameter updates. These
GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents Overview: GLM-5V-Turbo integrates multimodal perception as a core component of reasoning and planning to improve agentic capabilities across heterogeneous contexts like images, videos, and GUIs. This
Nonsense Helps: Prompt Space Perturbation Broadens Reasoning Exploration Overview: LoPE addresses the zero-advantage problem in LLMs trained via GRPO by injecting task-irrelevant Lorem Ipsum prompt perturbations to force the exploration of orthogonal reasoning pathways. This
TIDE: Every Layer Knows the Token Beneath the Context Overview: TIDE replaces the standard single-injection embedding design with EmbeddingMemory, an ensemble of independent MemoryBlocks that inject context-free semantic vectors into every layer via a depth-conditioned softmax
Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior Author's Explanation: x.com/GoodfireAI/sta… Overview: Manifold steering investigates the causal link between internal representations and behavioral outcomes by intervening along activation manifolds rather than assuming Euclidean geometry. This approach demonstrates that steering along the activation manifold produces behavioral trajectories that align with output geometry, whereas linear interventions fail to recover natural dynamics. By verifying this bidirectional relationship across LLMs and video world models, this work identifies geometric structure as the primary mechanism for principled control over neural model behavior. Paper: arxiv.org/abs/2605.05115
Neural networks might speak English, but they think in shapes. Understanding their rich *neural geometry* is key to understanding how they work – and to debugging and controlling them with precision. Starting today, we’re releasing a series of posts on this research agenda. 🧵
That's a wrap for last week, thanks for reading. You can see in-depth explanations on my newsletter too, stay tuned here: mail.bycloud.ai and check out our latest learning materials, where you can learn LLMs intuitively: intuitiveai.academy
Don't miss what's happening
People on X are the first to know.
