RetroSearch Browse

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/sgl-project/sglang/releases/tag/v0.4.8 below:

Release Release v0.4.8 · sgl-project/sglang · GitHub

Highlights OpenAI-Compatible Server Refactor

Re-structured the OpenAI-compatible server to support production and enterprise environments. Key improvements include:

Consistent metrics and logging for better observability and debugging.
Unified error handling, request validation, and processing logic for improved reliability and maintainability.
Improved request tracking across sessions and components.
Fixed bugs in embedding requests and reasoning parsers.

This work was a collaborative effort involving engineers from academic and industry institutions. Special thanks to the Oracle Cloud team and the SGLang team and community — including @slin1237, @CatherineSue, @key4ng, @JustinTong0323, @jhinpan, @yhyang201, @woodx9 and @whybeyoung — for their invaluable contributions.

DeepSeek R1 FP4 on Blackwell GPU

Added support for DeepSeek R1 with FP4 and MTP on NVIDIA Blackwell GPU.

Integrated FlashInfer NVFP4 MoE, supporting TP, EP, and DP.
Supported 2-stream shared expert execution.
Achieved up to 90 TPS per user at isl/osl/bs = 1k/1k/16 on B200.

Further optimization in progress. Special thanks to the FlashInfer, NVIDIA Enterprise Products, Novita AI, DataCrunch, Google Cloud, and SGLang teams — especially @Alcanderian and @pyc96 — for their critical contributions.

Breaking Change: OpenAI-Compatible API Module Moved

The sglang/srt/openai_api directory has been removed and replaced with sglang/srt/entrypoints/openai.

Update your imports to the new module path. For example:

- from sglang.srt.openai_api.protocol import Tool
+ from sglang.srt.entrypoints.openai.protocol import Tool

What's Changed

Update README.md by @merrymercy in #7040
[Docker] Upgrading base image from 24.04 to 24.12 by @Swipe4057 in #7043
fix 24.12 docker by @zhyncs in #7045
Minor cleanup of fa3 backend by @merrymercy in #6999
Fix eagle on AMD by @merrymercy in #7051
Clean up server_args.py by @merrymercy in #7037
Minor style fix in cuda_graph_runner.py by @merrymercy in #7053
[WA] fix output data is nan in CI test "test_moe_eval_accuracy_large.py" by @kkHuang-amd in #7021
[fix] libmlx5.so already in base image by @HanHan009527 in #7060
Fix test_lora.py CI by @Fridge003 in #7061
Tiny fix cutlass_mla_get_workspace_size stub incorrect signature by @fzyzcjy in #7057
Add sanity checks when a test file is not added to CI by @fzyzcjy in #6947
Revert "Add sanity checks when a test file is not added to CI (#6947)" by @zhyncs in #7063
Fix missing tool call id if tool call index >0 in streaming tool call output. by @Xu-Wenqing in #7049
chore: update dev docker by @zhyncs in #7064
Open AI API hidden states by @kyle-pena-kuzco in #6716
fix arm sgl-kernel link issue by @zhyncs in #7066
[Feature] Add Logit Bias by @b8zhong in #6579
Improve perf tuning docs by @merrymercy in #7071
Frontend language separate reasoning support by @binarycrayon in #6031
Do not run frontend_reasoning.ipynb to reduce the CI load by @merrymercy in #7073
Simplify the heuristics for setting --mem-fraction-static by @merrymercy in #7054
update doc by @Ximingwang-09 in #7046
Clean up docs for server args and sampling parameters (generated by grok) by @merrymercy in #7076
Fix GGuf and add back test_gguf.py by @Fridge003 in #7067
vlm: adapt internvl to VisionAttention by @mickqian in #6870
Fix circular import in test_prefix_chunk_info.py by @Fridge003 in #7097
Fix misusing the "_is_cuda". by @sogalin in #7091
Support VILA models by @futrime in #6106
[FIX]remove redundant code in logits_processor.py by @pc-neo in #7079
[feat]: Emit fixed-size KV blocks events by @faradawn in #6824
[Perf] Refactor LoRAManager to eliminate stream syncs and redundant computations by @lifuhuang in #6994
Fix positional argument by @liquanfeng in #7093
[sgl-kernel] Add cuda kernel for moe_ep_silu_and_mul by @yuan-luo in #6919
Improve log status by @hnyls2002 in #7115
feat: update blackwell setup by @zhyncs in #7119
Update CODEOWNERS by @merrymercy in #7126
Add gfx950 support for sgl-kernel. by @sogalin in #7092
[Fix] Reduce busy polling when scheduler is idle by @p12tic in #6026
Minor add utility to read expert distribution recorder output by @fzyzcjy in #7134
Remove unnecessary metadata_expand.max_seq_len_k operations in fa3 to… by @byjiang1996 in #7140
Minor speedup topk postprocessing by @fzyzcjy in #7058
filter by num_hidden_layers by @pansicheng in #7056
Remove 200us slow concat kernel (part 1: kernel) by @fzyzcjy in #7145
Support new DeepGEMM format in per token group quant by @fzyzcjy in #7146
chore: bump v0.1.8.post1 by @zhyncs in #7152
Support new DeepGEMM format in per token group quant (part 2: srt) by @fzyzcjy in #7155
Fix DeepEP error in some environments by @fzyzcjy in #7154
Minor speed up block_quant_dequant by @fzyzcjy in #6814
Tiny add sanity checks for DeepGEMM inputs by @fzyzcjy in #7157
Remove 200us slow concat kernel (part 2: srt) by @fzyzcjy in #7020
Re-quantize DeepSeek model weights to support DeepGEMM new input format by @fzyzcjy in #7156
Minor style change of triton backend by @merrymercy in #7165
Split the eagle test into two files by @merrymercy in #7170
Support new DeepGEMM input format in silu_and_mul_masked_post_quant_fwd by @fzyzcjy in #7153
Refactor DeepGEMM integration by @fzyzcjy in #7150
Add test for refactored openai server by @jhinpan in #7161
Improve test cases for eagle infer by @merrymercy in #7173
Support new DeepGEMM by @fzyzcjy in #7172
Increase timeout in test/srt/test_disaggregation.py by @merrymercy in #7175
Add Phi-4-mm to supported VLM supported model list. by @lifuhuang in #7178
Fix shared experts fusion + weight requant by @fzyzcjy in #7177
[fix] fix dsv3 weight loader tqdm and simplify shared experts fusion by @Alcanderian in #7181
[fix] fix cutlass_mla_backend with cuda_graph and add sm_scale for sgl-kernel cutlass_mla by @Alcanderian in #7184
[PD] Update prefill.py by @ByronHsu in #7190
Fix a minor bug related to DeepGEMM upgrade by @zhijian-liu in #7191
chore: bump v0.1.8.post2 by @zhyncs in #7189
[fix] fix determine_num_fused_shared_experts by @Alcanderian in #7180
chore: upgrade sgl-kernel v0.1.8.post2 by @Alcanderian in #7186
Fix NCCL 2.27.3 not in docker image by @fzyzcjy in #7195
Fix error when disabling new DeepGEMM by @fzyzcjy in #7198
[PD] Support decode retract and update decode.py by @ByronHsu in #7196
Move host memory pools into a separate file by @merrymercy in #7200
Lianmin/simplify memory pool by @merrymercy in #7202
Fix grammar abort & Minor style fixes by @merrymercy in #7204
feat: use zstd for docker by @zhyncs in #7205
[EAGLE] Refactor code for page size > 1 & more simplifications by @merrymercy in #7163
Revert "[EAGLE] Refactor code for page size > 1 & more simplifications" by @merrymercy in #7210
[PD] use int32 for kv indices & get num_reserved_decode_tokens from server_args by @ByronHsu in #7214
Minor PD style fix by @ByronHsu in #7215
Fix ChunkCache object has no attribute 'disable' by @Fridge003 in #7217
Implement gather before attn by @ch-wan in #6378
Support LoRA in MMMU benchmark script. by @lifuhuang in #7218
refine fused_moe benchmark by @BBuf in #7221
Minor style and doc fix by @merrymercy in #7228
[EAGLE] Refactor code for page size > 1 & more simplifications by @merrymercy in #7213
Fix sampling for speculative decoding & simplify kernels by @merrymercy in #7207
Release sgl-kernel 0.1.9 by @merrymercy in #7232
[EAGLE] Fix draft kv cache layout for fa3 and topk > 1 by @merrymercy in #7239
[Eagle] Fix kernel call after updating speculative sampling kernels by @merrymercy in #7231
minor fix by @hnyls2002 in #7245
Tiny remove comments about DeepEP on H20 by @fzyzcjy in #7234
Feat/support rerank by @woodx9 in #6058
[fix] fix DeepGEMM blackwell input quant & ut & fix style and log by @Alcanderian in #7247
Update CI flakes. by @saienduri in #7244
chore: bump v0.4.7.post1 by @zhyncs in #7248
fix amd EP MoE FP8 issue by @alexsun07 in #7125
Use seq_len_fill_value in the cuda graph runners by @merrymercy in #7233
support custom weight loader for model runner by @yukavio in #7122
Fix AMD speculative decoding by @merrymercy in #7252
[Refactor] OAI Server components by @JustinTong0323 in #7167
OAI Server Skeleton & Core Utility Endpoints by @yhyang201 in #7179
[amd] Opt dsv3 moe by @kkHuang-amd in #7160
update ci node for xeon by @DiweiSun in #7265
feat: mtp support dp-attention by @u4lr451 in #6081
support qwen2 running on ascend npu device by @zhuyijie88 in #7022
Fix Deepseek R1 0528 FP4 tensor name mismatch issue during weights loading. by @pyc96 in #7164
bugfix(tool call ebnf): Fix EBNF generation for optional function parameters by @CatherineSue in #7283
Fix AWQ Dequant and Weight Loading of deepseek v2 by @AniZpZ in #6842
fix: resolve b200 dsv3 mtp issue by @zhyncs in #7286
ci: Fix test_ebnf_generate_all_optional_function_params by @CatherineSue in #7288
fix: only enable flash_attn test on sm80 sm90 by @zhyncs in #7289
[PD] Support get local ip from NIC for PD disaggregation by @ShangmingCai in #7237
[PD] Add custom memory pool option to support Mooncake PD with NVLink by @ShangmingCai in #7264
Upstreaming hicache bug fixes by @xiezhq-hermann in #7267
Update python API of activation, topk, norm and rope and remove vllm dependency by @yanbing-j in #6614
Fix hicache benchmark script bug - some sampled input_request is [] by @byjiang1996 in #7300
chore: change logs fromINFO to DEBUG for dp and add force quit for tokenizer manager by @ishandhanani in #7251
update invalid link in doc by @habaohaba in #7297
Fix mini_lb for PD with long output: limit chunk size of decode response by @ch-tiger1 in #7301
Fix profiler error when there are idle passes by @fzyzcjy in #7003
[pd] optimize dockerfile for pd disaggregation by @whybeyoung in #7319
Merge PDLB (Prefill-Decode Load Balancer) into SGLang Router by @slin1237 in #7096
Add more refactored openai test & in CI by @jhinpan in #7284
fix: resolve blackwell deepep image issue by @zhyncs in #7331
add seed in CPU UTs to avoid flaky failure by @chunyuan-w in #7333
Multi-Stage Awake: Support Resume and Pause KV Cache and Weights separately by @hebiao064 in #7099
Reintroduce tiny fix sampler error when prob is not contiguous by @fzyzcjy in #7354
[Refactor] Clean up radix cache related API by @DarkSharpness in #7303
Put _normalize_rid before other normalization in io_struct by @CatherineSue in #7363
[PD] Transfer hidden states for mtp when disaggregation by @Atream in #7242
[Bugfix][PD] Set conclude state before clear when failure happens by @ShangmingCai in #7362
docs: update installation by @zhyncs in #7366
[Docker] optimize dockerfile remove deepep and blackwell merge it to… by @whybeyoung in #7343
Clean unused import for mimo mtp model by @lambert0312 in #7370
[Bugfix]Fix hang bug using dp attention with HiRadixCache by @LLLL114 in #7159
[Doc] add embedding rerank doc by @woodx9 in #7364
Fix judgment condition for enabling Deepseek V3/R1 shared expert fusion optimization by @lambert0312 in #7371
Feat/refactor embedding server by @woodx9 in #7322
Purge VerlEngine by @MrAta in #7326
support return logprobs for pipeline by @strgrb in #7356
[PD] Optimize custom mem pool usage and bump mooncake version by @ShangmingCai in #7393
Support THUDM/GLM-4-0414 (GLM-Z1) Glm4ForCausalLM architecture. by @solrex in #5485
Refine OpenAI serving entrypoint to remove batch requests by @JustinTong0323 in #7372
[Feature] Comprehensive Hybrid Parallelism Support by @ch-wan in #6389
[DeepSeekNextN] fix: residual of head norm can be None by @ch-wan in #7398
[OAI refactor] Add rerank and score serving by @woodx9 in #7399
[OAI Server Refactor] [ChatCompletions & Completions] Implement UsageInfo Processor by @yhyang201 in #7360
Fix All-Gather under world size one by @ch-wan in #7219
Optimize DP attn scheduling for speculative decoding by @ch-wan in #7285
Update usage_processor.py by @ch-wan in #7402
Fix 7285 Merge Conflicts by @ch-wan in #7403
chore: upgrade mooncake-transfer-engine 0.3.4 by @zhyncs in #7401
[OAI Server Refactor] [ChatCompletions & Completions] Support Return Hidden State by @key4ng in #7329
Remove batches api in docs & example by @jhinpan in #7400
[BugFix]: fix EmbeddingReqInput single input error by @woodx9 in #7396
[BugFix]fix qwen25 invoke function call streaming responses with curly braces as the starting indicator by @ehuaa in #7394
fix overlap pagecount by @pansicheng in #6984
fix: Fix CI test_function_call_parser.py by @CatherineSue in #7425
Fix CPU offloading for MLA memory pool by @hnyls2002 in #7409
[fix] PD disaggregation when enable mtp and tp!=dp by @Atream in #7420
feat(oai refactor): Replace openai_api with entrypoints/openai by @CatherineSue in #7351
Refactor LoRAManager and LoRAMemoryPool state management logic for dynamic LoRA loading support by @lifuhuang in #7412
refactor(test): reorganize OpenAI test file structure by @CatherineSue in #7408
[minor] simplify the TokenToKVPoolAllocator by @hnyls2002 in #7414
Tiny add logging for GC by @fzyzcjy in #7406
FlashInfer NVFP4 MoE with EP & 2-stream shared expert by @trevor-m in #7327
Remove copy after bmm by @ispobock in #7441
Fix torch compile run by @kkHuang-amd in #7391
[misc] Add PD service discovery support in router by @slin1237 in #7361
add fused moe config for qwen3 in triton3.3.1 by @yizhang2077 in #7445
Fix CUDA Graph Check under Deepep with DP FFN by @ch-wan in #7451
Update hyperparameter_tuning.md by @merrymercy in #7454
feat: integrate deepgemm into EPMoE by @xutizhou in #6821
Solve docker build failed in the virtual machine by @kkHuang-amd in #7290
Fix a bug in BatchTokenIDOut & Misc style and dependency updates by @merrymercy in #7457
[CI] Upgrade mooncake to 0.3.4.post1 to fix 8 gpu tests by @ShangmingCai in #7472
Fix prefill OOM due to wrong token calculation when page > 1 by @hnyls2002 in #7397
feat(func_call): Add more check in BaseFormatDetector.parse_streaming_increment by @CatherineSue in #7479
Fix dtype for idle input in spec decoding by @ch-wan in #7456
update mooncake in dockerfile by @hnyls2002 in #7480
kvcache io kernels and test case by @xiezhq-hermann in #7382
[perf] slightly imporve DeepSeek-R1-FP4 TP8 by @Alcanderian in #7481
Quick fix for DeepGemm requant to also cover MTP. by @pyc96 in #7378
Support weight loading without mmap by @guoyuhong in #7469
ci: Revert openai_server related tests in AMD suites by @CatherineSue in #7449
Perormance: Enable cuda graph for dp idle batch by @u4lr451 in #7269
bugfix: Prevent global mutation of conv.stop_str across requests by @huangtingwei9988 in #7347
Fix RequestValidationError response format by @CatherineSue in #7487
Fix MTP with Deepseek R1 Fp4 by @pyc96 in #7376
chore: bump sgl-kernel v0.2.0 by @zhyncs in #7490
chore: bump v0.4.8 by @zhyncs in #7493

New Contributors

@futrime made their first contribution in #6106
@faradawn made their first contribution in #6824
@liquanfeng made their first contribution in #7093
@p12tic made their first contribution in #6026
@byjiang1996 made their first contribution in #7140
@zhijian-liu made their first contribution in #7191
@DiweiSun made their first contribution in #7265
@zhuyijie88 made their first contribution in #7022
@pyc96 made their first contribution in #7164
@ch-tiger1 made their first contribution in #7301
@Atream made their first contribution in #7242
@LLLL114 made their first contribution in #7159
@key4ng made their first contribution in #7329
@ehuaa made their first contribution in #7394

Full Changelog: v0.4.7...v0.4.8

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4