Remove deprecated Habana mixed precision from gaudi config
Mixed precision was turned off in this model due to the usage of Deepspeed
@olszd
Could you add "use_torch_autocast": true
in the Gaudi config please? Actually HMP was enabled since we have "use_habana_mixed_precision": true
.
Besides, regression tests didn't pass because there is an issue with autocast when doing gradient checkpointing. So let's wait for this to be solved before merging this PR.
I updated the Gaudi config with custom bf16/fp32 op lists that give better throughput and similar accuracy, closing this one: https://huggingface.co./Habana/gpt2/blob/main/gaudi_config.json
I'm getting better results keeping the same custom ops (the current ones with HMP) for autocast than default autocast, so I'm going to update that directly.