# Llama-3.1-8b performance on AWS Inferentia2 (Latency & Throughput)

How fast is Llama-3.1-8b on Inferentia2?  Let's figure out!

For this benchmark we will use the following configurations:

| Model type       | batch_size | sequence_length |
|------------------|------------|-----------------|
| Llama3.1 8b BS1  | 1          | 4096            |
| Llama3.1 8b BS4  | 4          | 4096            |
| Llama3.1 8b BS8  | 8          | 4096            |
| Llama3.1 8b BS16 | 16         | 4096            |
| Llama3.1 8b BS32 | 32         | 4096            |
| Llama3.1 8b BS48 | 48         | 4096            |

*Note: all models are compiled to use 4 devices corresponding to 8 cores on the `inf2.48xlarge` instance.*

*Note: please refer to the [inferentia2 product page](https://aws.amazon.com/ec2/instance-types/inf2/) for details on the available instances.*

## Time to first token

The time to first token is the time required to process the input tokens and generate the first output token.
It is a very important metric, as it corresponds to the latency directly perceived by the user when streaming generated tokens.

We test the time to first token for increasing context sizes, from a typical Q/A usage, to heavy Retrieval Augmented Generation (RAG) use-cases.

Time to first token is expressed in **seconds**.

![Llama3.1 8b inferentia2 TTFT](https://raw.githubusercontent.com/huggingface/optimum-neuron/main/docs/assets/benchmarks/inferentia-llama3.1-8b/ttft.png "Time to first token")

## Inter-token Latency

The inter-token latency corresponds to the average time elapsed between two generated tokens.

It is expressed in **milliseconds**.

![Llama3.1 8b inferentia2 inter-token latency](https://raw.githubusercontent.com/huggingface/optimum-neuron/main/docs/assets/benchmarks/inferentia-llama3.1-8b/latency.png "Inter-token latency")

### Throughput

Unlike some other benchmarks, we evaluate the throughput using generated tokens only, by dividing their number
by the end-to-end latency.

Throughput is expressed in **tokens/second**.

![Llama3.1 8b inferentia2 throughput](https://raw.githubusercontent.com/huggingface/optimum-neuron/main/docs/assets/benchmarks/inferentia-llama3.1-8b/throughput.png "Throughput")