synth Throughput calculator

Hardware & Model

GPU Type

3026 TFLOPS350W

LLM Model

Context: 16,384 tokens0.4 TFLOPS/token

Inference Engine

Number of GPUs1

164

Advanced Settings

Throughput Results

Tokens per second

8799.3

8799.3 per GPU

Tokens per minute

527,957

Tokens per hour

31677.4K

Cost per million tokens

$0.00

Cloud + power costs

Tokens per watt

25.141

Energy efficiency

Time for 1K tokens

0.1s

Time for 10K tokens

1.1s

Model fits in GPU memory

20% used

Weights: 16.0 GB

KV Cache: 0.0 GB

Total: 16.0 GB

VRAM: 80 GB

Model Size

8.0B

parameters

Context Length

16,384

tokens

Power Draw

350W

total

Hourly Cost

$N/A

cloud pricing