Introducing π‘¨π’•π’•π’†π’π’•π’Šπ’π’ π‘Ήπ’†π’”π’Šπ’…π’–π’‚π’π’”: Rethinking depth-wise aggregation. Residual connections have long relied on fixed, uniform accumulation. Inspired by the duality of time and depth, we introduce Attention Residuals, replacing standard depth-wise recurrence with https://t.co/gcWyzhZVc0

1 min read Original article β†—

Introducing π‘¨π’•π’•π’†π’π’•π’Šπ’π’ π‘Ήπ’†π’”π’Šπ’…π’–π’‚π’π’”: Rethinking depth-wise aggregation. Residual connections have long relied on fixed, uniform accumulation. Inspired by the duality of time and depth, we introduce Attention Residuals, replacing standard depth-wise recurrence with learned, input-dependent attention over preceding layers. πŸ”Ή Enables networks to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth. πŸ”Ή Introduces Block AttnRes, partitioning layers into compressed blocks to make cross-layer attention practical at scale. πŸ”Ή Serves as an efficient drop-in replacement, demonstrating a 1.25x compute advantage with negligible (<2%) inference latency overhead. πŸ”Ή Validated on the Kimi Linear architecture (48B total, 3B activated parameters), delivering consistent downstream performance gains. πŸ”—Full report: github.com/MoonshotAI/Att…