Let's make that more clear... ;)
Megatron (an NVidia-developed framework that excels at multi-GPU AI Acceleration), coupled with DeepSpeed, a Microsoft-written library were used.
The framework ensures many GPU's can be used at once to train the model.
The library itself helps by accelerating...