A RetroSearch Logo

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Search Query:

Showing content from https://github.com/mit-han-lab/TinyChatEngine below:

mit-han-lab/TinyChatEngine: TinyChatEngine: On-Device LLM Inference Library

TinyChatEngine: On-Device LLM/VLM Inference Library

Running large language models (LLMs) and visual language models (VLMs) on the edge is useful: copilot services (coding, office, smart reply) on laptops, cars, robots, and more. Users can get instant responses with better privacy, as the data is local.

This is enabled by LLM model compression technique: SmoothQuant and AWQ (Activation-aware Weight Quantization), co-designed with TinyChatEngine that implements the compressed low-precision model.

Feel free to check out our slides for more details!

Code LLaMA Demo on NVIDIA GeForce RTX 4070 laptop:

VILA Demo on Apple MacBook M1 Pro:

LLaMA Chat Demo on Apple MacBook M1 Pro:

LLM Compression: SmoothQuant and AWQ

SmoothQuant: Smooth the activation outliers by migrating the quantization difficulty from activations to weights, with a mathematically equal transformation (100*1 = 10*10).

AWQ (Activation-aware Weight Quantization): Protect salient weight channels by analyzing activation magnitude as opposed to the weights.

LLM Inference Engine: TinyChatEngine

For MacOS, install boost and llvm by

brew install boost
brew install llvm

For M1/M2 users, install Xcode from AppStore to enable the metal compiler for GPU support.

For Windows, download and install the GCC compiler with MSYS2. Follow this tutorial: https://code.visualstudio.com/docs/cpp/config-mingw for installation.

pacman -S --needed base-devel mingw-w64-x86_64-toolchain make unzip git
Windows with Nvidia GPU (Experimental) Step-by-step to Deploy Llama-3-8B-Instruct with TinyChatEngine

Here, we provide step-by-step instructions to deploy Llama-3-8B-Instruct with TinyChatEngine from scratch.

Deploy vision language model (VLM) chatbot with TinyChatEngine

TinyChatEngine supports not only LLM but also VLM. We introduce a sophisticated chatbot for VLM. Here, we provide easy-to-follow instructions to deploy vision language model chatbot (VILA-7B) with TinyChatEngine. We recommend using M1/M2 MacBooks for this VLM feature.

Precision x86
(Intel/AMD CPU) ARM
(Apple M1/M2 & RPi) Nvidia GPU FP32 ✅ ✅ W4A16 ✅ W4A32 ✅ ✅ W4A8 ✅ ✅ W8A8 ✅ ✅ Quantization and Model Support

The goal of TinyChatEngine is to support various quantization methods on various devices. For example, At present, it supports the quantized weights for int8 opt models that originate from smoothquant using the provided conversion script opt_smooth_exporter.py. For LLaMA models, scripts are available for converting Huggingface format checkpoints to our int4 wegiht format, and for quantizing them to specific methods based on your device. Before converting and quantizing your models, it is recommended to apply the fake quantization from AWQ to achieve better accuracy. We are currently working on supporting more models, please stay tuned!

Device-specific int4 Weight Reordering

To mitigate the runtime overheads associated with weight reordering, TinyChatEngine conducts this process offline during model conversion. In this section, we will explore the weight layouts of QM_ARM and QM_x86. These layouts are tailored for ARM and x86 CPUs, supporting 128-bit SIMD and 256-bit SIMD operations, respectively. We also support QM_CUDA for Nvidia GPUs, including server and edge GPUs.

Platforms ISA Quantization methods Intel & AMD x86-64 QM_x86 Apple M1/M2 Mac & Raspberry Pi ARM QM_ARM Nvidia GPU CUDA QM_CUDA

We offer a selection of models that have been tested with TinyChatEngine. These models can be readily downloaded and deployed on your device. To download a model, locate the target model's ID in the table below and use the associated script. Check out our model zoo here.

Models Precisions ID x86 backend ARM backend CUDA backend LLaMA_3_8B_Instruct fp32 LLaMA_3_8B_Instruct_fp32 ✅ ✅ int4 LLaMA_3_8B_Instruct_awq_int4 ✅ ✅ LLaMA2_13B_chat fp32 LLaMA2_13B_chat_fp32 ✅ ✅ int4 LLaMA2_13B_chat_awq_int4 ✅ ✅ ✅ LLaMA2_7B_chat fp32 LLaMA2_7B_chat_fp32 ✅ ✅ int4 LLaMA2_7B_chat_awq_int4 ✅ ✅ ✅ LLaMA_7B fp32 LLaMA_7B_fp32 ✅ ✅ int4 LLaMA_7B_awq_int4 ✅ ✅ ✅ CodeLLaMA_13B_Instruct fp32 CodeLLaMA_13B_Instruct_fp32 ✅ ✅ int4 CodeLLaMA_13B_Instruct_awq_int4 ✅ ✅ ✅ CodeLLaMA_7B_Instruct fp32 CodeLLaMA_7B_Instruct_fp32 ✅ ✅ int4 CodeLLaMA_7B_Instruct_awq_int4 ✅ ✅ ✅ Mistral-7B-Instruct-v0.2 fp32 Mistral_7B_v0.2_Instruct_fp32 ✅ ✅ int4 Mistral_7B_v0.2_Instruct_awq_int4 ✅ ✅ VILA-7B fp32 VILA_7B_CLIP_ViT-L_fp32 ✅ ✅ int4 VILA_7B_awq_int4_CLIP_ViT-L ✅ ✅ LLaVA-v1.5-13B fp32 LLaVA_13B_CLIP_ViT-L_fp32 ✅ ✅ int4 LLaVA_13B_awq_int4_CLIP_ViT-L ✅ ✅ LLaVA-v1.5-7B fp32 LLaVA_7B_CLIP_ViT-L_fp32 ✅ ✅ int4 LLaVA_7B_awq_int4_CLIP_ViT-L ✅ ✅ StarCoder fp32 StarCoder_15.5B_fp32 ✅ ✅ int4 StarCoder_15.5B_awq_int4 ✅ ✅ opt-6.7B fp32 opt_6.7B_fp32 ✅ ✅ int8 opt_6.7B_smooth_int8 ✅ ✅ int4 opt_6.7B_awq_int4 ✅ ✅ opt-1.3B fp32 opt_1.3B_fp32 ✅ ✅ int8 opt_1.3B_smooth_int8 ✅ ✅ int4 opt_1.3B_awq_int4 ✅ ✅ opt-125m fp32 opt_125m_fp32 ✅ ✅ int8 opt_125m_smooth_int8 ✅ ✅ int4 opt_125m_awq_int4 ✅ ✅

For instance, to download the quantized LLaMA-2-7B-chat model: (for int4 models, use --QM to choose the quantized model for your device)

To deploy a quantized model with TinyChatEngine, compile and run the chat program.

make chat -j
# ./chat <model_name> <precision> <num_threads>
./chat LLaMA2_7B_chat INT4 8
make chat -j
# ./chat <model_name> <precision>
./chat LLaMA2_7B_chat INT4

TinyEngine: Memory-efficient and High-performance Neural Network Library for Microcontrollers

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

llama.cpp

whisper.cpp

transformers

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4