Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/vllm-project/vllm/issues/11539 below:

[Model] DeepSeek-V3 Enhancements · Issue #11539 · vllm-project/vllm · GitHub

This issue tracks follow up enhancements after initial support for the Deepseek V3 model. Please feel free to chime in and contribute!

• Follow up [Model] [Quantization] Support deepseek_v3 w8a8 fp8 block-wise quantization #11523: enhance testing with shapes of production models and run it regularly on H100.
  • Solving via cutlas blockwise quantization kernels.
• Follow up Deepseek v3 #11502:
  • Test and enable torch.compile
  • Refactor MoEMethodBase to unify and clean up the extra arguments of scoring_func and e_correction_bias
  • Kernel tuning for 8xH200, MI300x, H100 (TP16 and TP8PP2 case)
    • Use https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/benchmark_moe.py, but adapt it for the w8a8 fused moe kernel.
  • CUDA Graph support
• MLA [WIP] Deepseek V2 MLA #10927 @simon-mo
• Support nextn prediction heads (EAGLE style prediction heads)
  • Original PR for EAGLE support [Speculative Decoding] EAGLE Implementation with Top-1 proposer #6830 Perf [Performance]: vllm Eagle performance is worse than expected #9565 Discussion [Usage]: how to use EAGLE on vLLM? #11126 Docs [Doc]Add documentation for using EAGLE in vLLM #11417
• Support expert parallelism for MoE.
• Support data parallelism for MLA.

youkaichao, zhouyuan, HuaizhengZhang, robertgshaw2-redhat, Isotr0py and 39 morezengqingfu1442

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4