A significant achievement in AI development has been realized as a frontier model was trained across three continents, showcasing the effectiveness of distributed AI training. This training involved generating data in separate countries and updating it asynchronously via cloud technology. Notably, the model integrated compressed weight diffs to sync updates efficiently across inference clusters, allowing for distributed rollout without the need for constant communication between the engines, as emphasized by the team at @cognition.

Cognition: Cognition is an applied AI lab building end-to-end software agents, best known for its Devin AI software engineer. The company focuses on reinforcement learning techniques to advance autonomous software development capabilities. In the news, it achieved a distributed RL training run spanning multiple datacenters and continents by combining its GPUs with external inference providers.
Fireworks AI: Fireworks AI operates a frontier platform for training and inference on open-weights models at scale. It specializes in high-performance serving infrastructure that supports large-scale model deployments. The company supplied additional compute resources for the rollout inference phase of the multi-continent training described in the news.

Distributed Training: Frontier models can leverage asynchronous cloud-based updates to incorporate training data generated across different countries.
Inference Integration: Compressed weight diffs enable efficient syncing of model updates between training and distributed inference engines without requiring tight collective communication for rollouts.