H100 on-premise for occasional LLM usage, how do you handle GPU idle time when deploying at a client site?

Intermediate
1

Hi,

I’m a software publisher and I need to deploy my solution on-premise at a client’s site. Based on our tests, the workload requires an H100. However, the usage will be occasional and not continuous, so the GPU will be idle most of the time.

I’m not comfortable recommending a client invest in an H100 that will sit unused 80% of the time, but I don’t have a choice on the hardware side.

My questions:

  • Do other ISVs face the same issue when deploying on-premise?

  • How do you justify this to the client?

  • Does the client typically share the GPU with other internal workloads to spread the cost?

  • Are there any common organizational or contractual patterns to handle this?

This is as much a business/organizational question as a technical one. Any real-world feedback welcome.

Thanks

2 Likes