Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/sgl-project/sglang/issues/7736 below:

Development Roadmap (2025 H2) · Issue #7736 · sgl-project/sglang · GitHub

Here is the development roadmap for 2025 H2. Contributions and feedback are welcome (Join Bi-weekly Development Meeting). The previous 2025 H1 roadmap can be found in #4042

• Feature compatibility and reliability: Make all advanced features fully compatible with each other and achieve production-level reliability, such as P/D disaggregation, all parallelisms, speculative decoding, and load balancing.
• Usability: easy installation on all backends; simple launch scripts for large-scale deployments.
• Kernel optimizations for new generations of hardware (Blackwell, MI350, TPU, etc).
• Reinforcement learning training framework integration.

• FP4 moe kernels [POC] enable trtllm fp4 from trtllm wheel #7711
• More attention kernel variants from flashinfer TRTLLM Gen MLA Decode Kernel Integration #7938
• CuDNN attention backend [Draft] Implement CuDNN Attention Backend with Graph Caching #5505 Integrates cudnn_batch_prefill_with_kv_cache into flashinfer attention backend #7841
• Torch FlexAttention backend [Kernel] feat: add flexprefill for long-context #6867
• Improve FP8 gemm kernels on hopper and blackwell (try torch built-in kernels)
• Overlap allreduce in tensor parallelism
• Faster multi-node allreduce (infiniband, mnnvl)

• HiCache with storage layer enhancement Hicache Storage Layer Prototype #7704
  • Mooncake integration: Support l3 cache (mooncake store) for hiradix cache #7211
  • 3FS integration: Add hf3fs support for hicache storage (based on #7704) #7280
  • More backends including https://github.com/ai-dynamo/nixl, etc.
  • Alternatively, directly mapping storage device or remote memory pool as host memory pool if hardware permits: Add GDS alternative for hierarchical kv cache #7896
• Holistic scheduling with a global view of KV cache, more can be found: [Roadmap] Distributed Serving Enhancement on 2025 H2 #8210
• C++ implementation of radix and hiradix tree to mitigate scheduling overhead [Refactor] Rewrite Hierarchical Prefix Cache in C++ #7194 [Feature] Radix Tree in C++ #7369 Minor Optimizations in Radix Cache and Schedule Batch #6907
• Sliding window attention memory pool optimizations (dynamic ratio)

• Support arbitrary combination of DP attention + TP + EP [1/N] MoE Refactor: refactor select_experts #7966
• Async schedule for pipeline parallelism
• Support all parallelism + speculative decoding
• Support all parallelism + PD disaggregation

• CPU transfer backend [Roadmap] Distributed Serving Enhancement on 2025 H2 #8210
• Auto scaling in OME
• Fault tolerance & reliability & health check
• Elastic expert parallelism

• Major roadmap: [Roadmap] Quantization Support #8180
• ModelOpt integration to support all models
• Communication quantization (fp8 allreduce)
• MXFP4 support

• AREAL, slime, veRL integration (sorted alphabetically)
• Faster weight sync
• Reproduce Deepseek/Kimi + GRPO training

• Simplify overlap scheduler
• Remove deadcode (e.g, double sparsity, flashinfer lora backend)
• Modularize each component
• Document major parts

• Reference-based speculative decoding
• Make speculative decoding compatible with all other features
• Fix all corner cases in structured output + speculative decoding + reasoning/function call parsing

• Major Roadmap: [Roadmap] Lora Support #2929
• Asynchronous lora updates: Support overlapped lora updates  #8213
• Support Radix Attention: Support radix cache for Lora feature #7216
• GPU pinning: Support GPU pinning for LoRA #8697
• LRU cache
• Performance improvements Improve LoRA Perf by Deprecating FlashInfer and Eliminating Redundant Tensor Ops #8940
• Support embedding layers Feat: support LoRA for embedding layer #8222
• Serving lora adaptors with different ranks using unified paging

• Better CI coverage of AMD
• Support Ascend NPU [Roadmap] Supporting Ascend NPU on 2025 H2 #8004
• H20-specific optimizations [Roadmap] Kimi-K2 performance enhancement on H20 GPU #8151
• Support TPU
• Support intel XPU
• Support more hardware with multi-framework [Roadmap] Supporting multi frameworks on 2025 H2 #8199

• Day 0 support for all upcoming OSS models
• Multi-modal models
• Language models
  • Mamba models

• Provide an gRPC interface
• Rewrite api layer (fastapi, tokenizer manager) in sgl-router
• Support all advanced apis (e.g. OpenAI response API, MCP integration)

zhyncs, lambert0312, Swipe4057, ZelinMa557, hzh0425 and 54 morezhyncs, ispobock, KaiyuZhang001, Swipe4057, slin1237 and 20 morezhyncs, KaiyuZhang001, Swipe4057, slin1237, b8zhong and 21 morelkm2835, xwuShirley, Swipe4057, b8zhong, JustinTong0323 and 13 more

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4