Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/NVIDIA/TensorRT-LLM/issues/2121 below:

Llama3.1 support removed? · Issue #2121 · NVIDIA/TensorRT-LLM · GitHub

A100

@byshiue

• The official example scripts
• My own modified scripts

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Build from source https://github.com/triton-inference-server/tensorrtllm_backend
Install the latest transformers version supported by tensorrt-llm

transformers 4.43.1 should be supported by tensorrt_llm, hence supporting Llama3.1.
This is as per https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct which says transformers version should be > 4.43

Llama3.1 is not supported because transformers needs to be <=4.42.4,>=4.38.2 for this tensorrt_llm

As per #2008, transformers 4.43.1 was supported, allowing use of Llama3.1
However, in the latest Tensort-llm, this transformers version is not supported anymore #2094

Why is the support for Llama3.1 removed? Was there a bug?

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4