Fix rnn flop profiler to compute flops instead of macs
diff --git a/deepspeed/profiling/flops_profiler/profiler.py b/deepspeed/profiling/flops_profiler/profiler.py
index 29d74c617ac9..38ab92ed8284 100644
--- a/deepspeed/profiling/flops_profiler/profiler.py
+++ b/deepspeed/profiling/flops_profiler/profiler.py
@@ -960,10 +960,11 @@ def _reload_tensor_methods():
 
 
 def _rnn_flops(flops, rnn_module, w_ih, w_hh, input_size):
+    input_size, hidden_size = w_ih.shape
     # matrix matrix mult ih state and internal state
-    flops += w_ih.shape[0] * w_ih.shape[1]
+    flops += 2 * input_size * hidden_size - hidden_size
     # matrix matrix mult hh state and internal state
-    flops += w_hh.shape[0] * w_hh.shape[1]
+    flops += 2 * hidden_size * hidden_size - hidden_size
     if isinstance(rnn_module, (nn.RNN, nn.RNNCell)):
         # add both operations
         flops += rnn_module.hidden_size

This MR is related to the issue I recently opened: https://github.com/microsoft/DeepSpeed/issues/3816

It implements the simple fix I proposed there, which simply computes FLOPs instead of MACs.