Nous Research unveils Lighthouse Attention, boosting training speed by 1.4-1.7× at 98K context
Today, researchers announced the release of Lighthouse Attention, a new hierarchical attention mechanism designed for long-context pre-training, which achieves a significant speedup—1.4 to 1.7 times faster wall-clock pre-training at 98K…
