Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/sgl-project/sglang/issues/8180 below:

[Roadmap] Quantization Support · Issue #8180 · sgl-project/sglang · GitHub

Objective: Refactor the code to enhance the maintainability and extensibility of the quantization module.

• Weight Only Methods
  • GPTQ [1/n] chore: decouple quantization implementation from vLLM dependency #7992 [2/n]decouple quantization implementation from vLLM dependency #8112 [4/n]decouple quantization implementation from vLLM dependency #9191
  • AWQ [3/n] chore: decouple AWQ implementation from vLLM dependency #8113
  • Compressed Tensors
• MoE Quantization
• Kernel Optimization
• fbgemm fp8

Objective: Extend sglang's efficient inference capabilities to a broader range of hardware.

• Ascend NPUs
  • W8A8 INT8 [feature]Ascend quantization support #7791
• Intel Xeon CPUs
  • W8A8

Objective: Optimize components beyond standard linear layers to further improve performance.

• Attention
  • MLA Quantization
  • GQA/MHA Quantization
• Improved KV Cache Quantization @Wilbolu
• Communication Quantization

Objective: Stay current with cutting-edge quantization techniques and data formats.

• MXFP4 Quantization

Swipe4057, Hongbosherlock, lambert0312, yuan-luo, xu-yfei and 4 more

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4