Fix Trainer when model is loaded on a different GPU
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
index 590c5da195351..59ac8029e54f3 100755
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -377,12 +377,12 @@ def __init__(
         else:
             self.is_model_parallel = False
 
-        if (
-            getattr(model, "hf_device_map", None) is not None
-            and len([device for device in set(model.hf_device_map.values()) if device not in ["cpu", "disk"]]) > 1
-            and not self.is_model_parallel
-        ):
-            self.is_model_parallel = True
+        if getattr(model, "hf_device_map", None) is not None:
+            devices = [device for device in set(model.hf_device_map.values()) if device not in ["cpu", "disk"]]
+            if len(devices) > 1:
+                self.is_model_parallel = True
+            else:
+                self.is_model_parallel = self.args.device != torch.device(devices[0])
 
             # warn users
             logger.info(

# What does this PR do?

When a small model is loaded with `device_map="auto"` it might end up all on GPU 1, so currently `is_model_parallel` is set to `False` (cause one device) and later on the Trainer moves the model to GPU 0 which fails the execution of all the Accelerate hooks.

This PR fixes this by making sure `is_model_parallel` is set to `True` when there is one device but it's not GPU 0.