Congrats to the team
Congrats to the @Kimi_Moonshot team! This is awesome.
Great to see this level of research coming from open-source frontier model labs.
I liked the paper so much I built a Rust implementation of it ;)
Full AttnRes + Block AttnRes with two-phase inference, built using Burn
Kimi.ai @Kimi_Moonshot Introducing 𝑨𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏 𝑹𝒆𝒔𝒊𝒅𝒖𝒂𝒍𝒔: Rethinking depth-wise aggregation.
Residual connections have long relied on fixed, uniform accumulation. Inspired by the duality of time and depth, we introduce Attention Residuals, replacing standard depth-wise recurrence with
