Skip to content

Files

Failed to load latest commit information.

Latest commit

 Cannot retrieve latest commit at this time.

History

History
 
 

benchmarks

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
cpp
cpp
 
 
python
python
 
 
README.md
README.md
 
 

README.md

TensorRT-LLM Benchmarks

Overview

There are currently three workflows to benchmark TensorRT-LLM:

  • C++ benchmarks
    • The recommended workflow that uses TensorRT-LLM C++ API and can take advantage of the latest features of TensorRT-LLM.
  • Python benchmarks
    • The Python benchmarking scripts can only benchmark the Python runtime, which do not support the latest features, such as in-flight batching.
  • The Python benchmarking suite
    • This benchmarker is native to TensorRT-LLM and is a Python benchmarker for reproducing and testing the performance of TensorRT-LLM.
    • NOTE: This benchmarking suite is a current work in progress and is prone to large changes.