Triton Model Analyzer is a CLI tool which can help you find a more optimal configuration, on a given piece of hardware, for single, multiple, ensemble, or BLS models running on a Triton Inference Server. Model Analyzer will also generate reports to help you better understand the trade-offs of the different configurations along with their compute and memory requirements.
Optuna Search -ALPHA RELEASE- allows you to search for every parameter that can be specified in the model configuration, using a hyperparameter optimization framework. Please see the Optuna website if you are interested in specific details on how the algorithm functions.
Quick Search will sparsely search the Max Batch Size, Dynamic Batching, and Instance Group spaces by utilizing a heuristic hill-climbing algorithm to help you quickly find a more optimal configuration
Automatic Brute Search will exhaustively search the Max Batch Size, Dynamic Batching, and Instance Group parameters of your model configuration
Manual Brute Search allows you to create manual sweeps for every parameter that can be specified in the model configuration
Ensemble: Model Analyzer can help you find the optimal settings when profiling an ensemble model
BLS: Model Analyzer can help you find the optimal settings when profiling a BLS model
Multi-Model: Model Analyzer can help you find the optimal settings when profiling multiple concurrent models
LLM: Model Analyzer can help you find the optimal settings when profiling Large Language Models
Detailed and summary reports: Model Analyzer is able to generate summarized and detailed reports that can help you better understand the trade-offs between different model configurations that can be used for your model.
QoS Constraints: Constraints can help you filter out the Model Analyzer results based on your QoS requirements. For example, you can specify a latency budget to filter out model configurations that do not satisfy the specified latency threshold.
See the Single Model Quick Start for a guide on how to use Model Analyzer to profile, analyze and report on a simple PyTorch model.
See the Multi-model Quick Start for a guide on how to use Model Analyzer to profile, analyze and report on two models running concurrently on the same GPU.
See the Ensemble Model Quick Start for a guide on how to use Model Analyzer to profile, analyze and report on a simple Ensemble model.
See the BLS Model Quick Start for a guide on how to use Model Analyzer to profile, analyze and report on a simple BLS model.
Below are definitions of some commonly used terms in Model Analyzer:
We appreciate any feedback, questions or bug reporting regarding this project. When help with code is needed, follow the process outlined in the Stack Overflow (https://stackoverflow.com/help/mcve) document. Ensure posted examples are:
minimal – use as little code as possible that still produces the same problem
complete – provide all parts needed to reproduce the problem. Check if you can strip external dependency and still show the problem. The less time we spend on reproducing problems the more time we have to fix it
verifiable – test the code you're about to provide to make sure it reproduces the problem. Remove all other problems that are not related to your request/question.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4