Include cublas error details when getting cublas handle fails
diff --git a/csrc/includes/context.h b/csrc/includes/context.h
index 3a9067dc3b9f..cd80f8fbeebe 100644
--- a/csrc/includes/context.h
+++ b/csrc/includes/context.h
@@ -50,8 +50,12 @@ class TrainingContext {
     {
         curandCreateGenerator(&_gen, CURAND_RNG_PSEUDO_DEFAULT);
         curandSetPseudoRandomGeneratorSeed(_gen, 123);
-        if (cublasCreate(&_cublasHandle) != CUBLAS_STATUS_SUCCESS) {
-            auto message = std::string("Fail to create cublas handle.");
+        cublasStatus_t stat = cublasCreate(&_cublasHandle);
+        if (stat != CUBLAS_STATUS_SUCCESS) {
+            // It would be nice to use cublasGetStatusName and
+            // cublasGetStatusString, but they were only added in CUDA 11.4.2.
+            auto message = std::string("Failed to create cublas handle: cublasStatus_t was ") +
+                           std::to_string(stat);
             std::cerr << message << std::endl;
             throw std::runtime_error(message);
         }
diff --git a/csrc/transformer/inference/includes/inference_context.h b/csrc/transformer/inference/includes/inference_context.h
index 66596c9bfb82..aaf568553560 100644
--- a/csrc/transformer/inference/includes/inference_context.h
+++ b/csrc/transformer/inference/includes/inference_context.h
@@ -60,8 +60,13 @@ class InferenceContext {
     {
         _workSpaceSize = 0;
         _workspace = 0;
-        if (cublasCreate(&_cublasHandle) != CUBLAS_STATUS_SUCCESS) {
-            auto message = std::string("Fail to create cublas handle.");
+
+        cublasStatus_t stat = cublasCreate(&_cublasHandle);
+        if (stat != CUBLAS_STATUS_SUCCESS) {
+            // It would be nice to use cublasGetStatusName and
+            // cublasGetStatusString, but they were only added in CUDA 11.4.2.
+            auto message = std::string("Failed to create cublas handle: cublasStatus_t was ") +
+                           std::to_string(stat);
             std::cerr << message << std::endl;
             throw std::runtime_error(message);
         }

I've been getting hard-to-debug errors in some DeepSpeed runs. During initialization, one of the worker processes raises `RuntimeError: Fail to create cublas handle.` with no further details, which feels pretty mysterious.

This change includes details of the failure status by using https://docs.nvidia.com/cuda/cublas/#cublasgetstatusname and https://docs.nvidia.com/cuda/cublas/#cublasgetstatusstring

---

**original error message** (using deepspeed 0.9.2): `RuntimeError: Fail to create cublas handle.`
<img width="1135" alt="image" src="https://github.com/microsoft/DeepSpeed/assets/133466/f3c4f14b-b820-463b-bf1b-a85b1e0b2399">


**new error message** with this change: `RuntimeError: Failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED the library was not initialized`
<img width="874" alt="image" src="https://github.com/microsoft/DeepSpeed/assets/133466/3f04bec0-5922-44aa-a309-7bf4e0429c1d">

This is still not a great error message, but it has better search results (most results suggest that it's due to running out of GPU memory; bizarrely [some people also report removing `~/.nv` fixes it...](https://github.com/tensorflow/tensorflow/issues/9489#issuecomment-405213443)).