DeepSpeed

Content sourced from Wikipedia, licensed under CC BY-SA 3.0.

DeepSpeed is an open-source optimization library for PyTorch. It helps reduce compute power and memory use so you can train very large distributed models on standard hardware, with low latency and high throughput.

A key feature is the Zero Redundancy Optimizer (ZeRO), which makes it possible to train models with trillions of parameters. It also supports mixed-precision training and scales from a single GPU to many GPUs across multiple machines, with options for custom model parallelism.

DeepSpeed was first released in May 2020, with a stable release in March 2025. The project is available on GitHub as an open-source package. Users report performance gains such as up to 6.2x higher throughput, 2.8x faster convergence, and 4.6x less communication.