{ "hf_id": "MiniMaxAI/MiniMax-M3.1-Flash", "meta": { "title": "MiniMax-M3.1-Flash", "slug": "minimax-m3.1-flash", "provider": "MiniMax", "description": "MiniMax M3.1-Flash vision-language MoE (~428B total / 23B active) with MSA sparse attention throughout and a built-in DSpark draft; quantization-aware trained, MXFP8 dense layers and NVFP4 experts. Served TP4 on NVIDIA Blackwell or AMD MI355X, prefill/decode disaggregated as TP4 + TP4 on MI355X or DEP2 + DEP4 on GB200/GB300, and TP8 on NVIDIA Hopper.", "date_added": "2026-10-23", "date_updated": "2026-10-30", "tasks": [ "text", "multimodal" ], "performance_headline": "MSA sparse attention, 1M context, DSpark speculative decoding built into the checkpoint", "related_recipes": [ "MiniMaxAI/MiniMax-M3" ], "default_hardware": "b300", "hardware": { "mi355x": "verified", "b300": "verified", "gb200": "verified", "gb300": "verified" } }, "recommended_command": { "hardware": "b300", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_FLOAT32_MATMUL_PRECISION": "high" }, "docker_image": "vllm/vllm-openai:minimax-m31-flash", "command": "vllm serve MiniMaxAI/MiniMax-M3.1-Flash \\\n --block-size 128 \\\n --disable-hybrid-kv-cache-manager \\\n --tensor-parallel-size 8 \\\n --kv-cache-dtype nvfp4 \\\n --attention-config '{\"indexer_kv_dtype\": \"nvfp4\", \"use_prefill_query_quantization\": true}' \\\n --compilation-config '{\"cudagraph_mode\": \"FULL_AND_PIECEWISE\"}' \\\n --tool-call-parser minimax_m3 \\\n --enable-auto-tool-choice \\\n --reasoning-parser minimax_m3 \\\n --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"attention_backend\":\"FLASHINFER_TS\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}'", "argv": [ "vllm", "serve", "MiniMaxAI/MiniMax-M3.1-Flash", "--block-size", "128", "--disable-hybrid-kv-cache-manager", "--tensor-parallel-size", "8", "--kv-cache-dtype", "nvfp4", "--attention-config", "{\"indexer_kv_dtype\": \"nvfp4\", \"use_prefill_query_quantization\": true}", "--compilation-config", "{\"cudagraph_mode\": \"FULL_AND_PIECEWISE\"}", "--tool-call-parser", "minimax_m3", "--enable-auto-tool-choice", "--reasoning-parser", "minimax_m3", "--speculative-config", "{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"attention_backend\":\"FLASHINFER_TS\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_FLOAT32_MATMUL_PRECISION=high \\\n vllm/vllm-openai:minimax-m31-flash MiniMaxAI/MiniMax-M3.1-Flash \\\n --block-size 128 \\\n --disable-hybrid-kv-cache-manager \\\n --tensor-parallel-size 8 \\\n --kv-cache-dtype nvfp4 \\\n --attention-config '{\"indexer_kv_dtype\": \"nvfp4\", \"use_prefill_query_quantization\": true}' \\\n --compilation-config '{\"cudagraph_mode\": \"FULL_AND_PIECEWISE\"}' \\\n --tool-call-parser minimax_m3 \\\n --enable-auto-tool-choice \\\n --reasoning-parser minimax_m3 \\\n --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"attention_backend\":\"FLASHINFER_TS\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}'", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_FLOAT32_MATMUL_PRECISION=high", "vllm/vllm-openai:minimax-m31-flash", "MiniMaxAI/MiniMax-M3.1-Flash", "--block-size", "128", "--disable-hybrid-kv-cache-manager", "--tensor-parallel-size", "8", "--kv-cache-dtype", "nvfp4", "--attention-config", "{\"indexer_kv_dtype\": \"nvfp4\", \"use_prefill_query_quantization\": true}", "--compilation-config", "{\"cudagraph_mode\": \"FULL_AND_PIECEWISE\"}", "--tool-call-parser", "minimax_m3", "--enable-auto-tool-choice", "--reasoning-parser", "minimax_m3", "--speculative-config", "{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"attention_backend\":\"FLASHINFER_TS\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "blackwell", "compute_arch": "sm103", "display_name": "B300", "description": "NVIDIA B300 SXM 268 GB HBM3e · 8-GPU HGX B300 node (Blackwell Ultra)", "gpu_count": 8, "vram_gb": 2144, "multi_node": false }, "alternatives": { "single_node_tep": "/MiniMaxAI/MiniMax-M3.1-Flash/strategies/single_node_tep.json", "single_node_dep": "/MiniMaxAI/MiniMax-M3.1-Flash/strategies/single_node_dep.json", "multi_node_tp": "/MiniMaxAI/MiniMax-M3.1-Flash/strategies/multi_node_tp.json", "multi_node_dep": "/MiniMaxAI/MiniMax-M3.1-Flash/strategies/multi_node_dep.json", "multi_node_tep": "/MiniMaxAI/MiniMax-M3.1-Flash/strategies/multi_node_tep.json", "pd_cluster": "/MiniMaxAI/MiniMax-M3.1-Flash/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/MiniMaxAI/MiniMax-M3.1-Flash/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/MiniMaxAI/MiniMax-M3.1-Flash/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "b300": "/MiniMaxAI/MiniMax-M3.1-Flash/hw/b300.json", "b200": "/MiniMaxAI/MiniMax-M3.1-Flash/hw/b200.json", "gb200": "/MiniMaxAI/MiniMax-M3.1-Flash/hw/gb200.json", "gb300": "/MiniMaxAI/MiniMax-M3.1-Flash/hw/gb300.json", "mi355x": "/MiniMaxAI/MiniMax-M3.1-Flash/hw/mi355x.json" } }, "model": { "model_id": "MiniMaxAI/MiniMax-M3.1-Flash", "min_vllm_version": "0.33.0", "nightly_required": true, "docker_image": { "nvidia": "vllm/vllm-openai:minimax-m31-flash", "amd": "vllm/vllm-openai-rocm:minimax-m31-flash" }, "architecture": "moe", "parameter_count": "428B", "active_parameters": "23B", "context_length": 1048576, "base_args": [ "--block-size", "128", "--disable-hybrid-kv-cache-manager" ], "base_env": {}, "install": { "docker": { "command": "docker pull vllm/vllm-openai:minimax-m31-flash" } } }, "features": { "tool_calling": { "description": "MiniMax M3 tool call parser with automatic tool choice", "args": [ "--tool-call-parser", "minimax_m3", "--enable-auto-tool-choice" ] }, "reasoning": { "description": "MiniMax M3 reasoning parser", "args": [ "--reasoning-parser", "minimax_m3" ] }, "spec_decoding": { "description": "DSpark speculative decoding with the draft shipped in the checkpoint (`dspark/`), FP8 draft KV cache. Per brand: seven speculative tokens with FLASHINFER_TS draft attention on NVIDIA, three with TRITON_ATTN on AMD.", "hardware_overrides": { "hopper": { "args": [] }, "nvidia": { "args": [ "--speculative-config", "{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"attention_backend\":\"FLASHINFER_TS\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}" ] }, "amd": { "args": [ "--speculative-config", "{\"method\":\"dspark\",\"num_speculative_tokens\":3,\"attention_backend\":\"TRITON_ATTN\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\"}" ] } } }, "text_only": { "description": "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache.", "args": [ "--language-model-only" ] }, "prefill_dbo_sp": { "label": "DBO / SP prefill", "description": "AMD MI355X single node: faster long prefill steps. DBO (dual-batch overlap) splits a prefill step into two microbatches and runs one microbatch's all-reduce under the other's compute; decode steps are never split. SP (sequence-parallel prefill) keeps the residual stream sharded by tokens on steps of 4096+ tokens, so each layer's two all-reduces become an all-to-all and a reduce-scatter; it keeps a full o_proj copy per GPU (about 3.1 GB less KV cache).", "strategies": [ "single_node_tp" ], "default_mode": "sp_dbo", "modes": { "sp_dbo": { "variants": [ "default" ], "label": "SP + DBO", "description": "SP prefill, and DBO on prefill steps of 16384+ tokens: each microbatch then has 8192+ tokens and stays on the SP path.", "args": [ "--enable-dbo", "--dbo-prefill-token-threshold", "16384", "--dbo-decode-token-threshold", "32768" ], "env": { "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_ROCM_MINIMAX_M3_SP_PREFILL": "1" }, "hardware": { "hopper": "unsupported", "blackwell": "unsupported" } }, "dbo": { "label": "DBO", "description": "DBO on prefill steps of 8192+ tokens, no SP.", "args": [ "--enable-dbo", "--dbo-prefill-token-threshold", "8192", "--dbo-decode-token-threshold", "32768" ], "env": { "VLLM_USE_V2_MODEL_RUNNER": "1" }, "hardware": { "hopper": "unsupported", "blackwell": "unsupported" } }, "sp": { "variants": [ "default" ], "label": "SP", "description": "SP prefill on steps of 4096+ tokens, no DBO.", "args": [], "env": { "VLLM_ROCM_MINIMAX_M3_SP_PREFILL": "1" }, "hardware": { "hopper": "unsupported", "blackwell": "unsupported" } } } }, "mono_decode": { "label": "Mono decode", "description": "AMD MI355X single node, MXFP4 (Quark) checkpoint, low concurrency: a decode step of up to 16 tokens (4 requests with DSpark 3) runs each layer as one fused kernel; larger steps and steps with prefill run on the normal path. Uses FP8 KV and indexer K caches (fewer KV tokens than NVFP4) and FULL_DECODE_ONLY graphs, so prefill and mixed steps run eager.", "strategies": [ "single_node_tp" ], "modes": { "mono": { "variants": [ "mxfp4" ], "args": [ "--kv-cache-dtype", "fp8", "--attention-config", "{\"indexer_kv_dtype\": \"fp8\"}", "--compilation-config", "{\"cudagraph_mode\": \"FULL_DECODE_ONLY\"}", "--gpu-memory-utilization", "0.85" ], "env": { "VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT": "1", "VLLM_ROCM_MINIMAX_M3_MONO": "1" }, "hardware": { "hopper": "unsupported", "blackwell": "unsupported", "mi300x": "unsupported", "mi325x": "unsupported", "mi350x": "unsupported" } }, "none": { "variants": [ "default", "fp8" ], "args": [] } } } }, "opt_in_features": [ "text_only", "prefill_dbo_sp", "mono_decode" ], "variants": { "default": { "precision": "nvfp4", "precision_hardware": {}, "supported_hardware": [ "mi355x", "gb200", "gb300", "b200", "b300" ], "vram_minimum_gb": 285, "description": "MXFP8 dense layers + NVFP4 experts with NVFP4 KV and indexer K caches. Blackwell runs the NVFP4 experts natively; MI355X re-quantizes them to MXFP8 for the AITER MoE.", "hardware_overrides": { "mi355x": { "vram_minimum_gb": 533 }, "amd": { "extra_args": [ "--quantization-config.moe", "mxfp8" ] } } }, "mxfp4": { "model_id": "amd/MiniMax-M3.1-Flash-Quark-MXFP4", "precision": "mxfp4", "label": "MXFP4 (Quark)", "supported_hardware": [ "mi355x" ], "vram_minimum_gb": 293, "description": "AMD Quark export: routed experts stored in OCP MXFP4 and run as W4A4 on the AITER MoE, with NVFP4 KV and indexer K caches. MI355X only.", "hardware_overrides": { "amd": { "extra_args": [ "--moe-backend", "aiter" ], "extra_env": { "VLLM_ROCM_USE_AITER_MOE": "1", "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS": "0" } } }, "json": "/amd/MiniMax-M3.1-Flash-Quark-MXFP4.json" }, "fp8": { "precision": "fp8", "supported_hardware": [ "arch:sm90" ], "vram_minimum_gb": 533, "tp": 8, "description": "Hopper: FP8 KV and indexer K caches, NVFP4 experts on the Marlin MoE, 256K context, video inputs off.", "extra_args": [ "--kv-cache-dtype", "fp8", "--attention-config", "{\"indexer_kv_dtype\": \"fp8\"}", "--compilation-config", "{\"cudagraph_mode\": \"FULL_AND_PIECEWISE\"}" ] } }, "hardware_overrides": { "blackwell": { "extra_args": [ "--kv-cache-dtype", "nvfp4", "--attention-config", "{\"indexer_kv_dtype\": \"nvfp4\", \"use_prefill_query_quantization\": true}", "--compilation-config", "{\"cudagraph_mode\": \"FULL_AND_PIECEWISE\"}" ], "extra_env": { "VLLM_FLOAT32_MATMUL_PRECISION": "high" } }, "amd": { "extra_args": [ "--kv-cache-dtype", "nvfp4", "--attention-config", "{\"indexer_kv_dtype\": \"nvfp4\", \"indexer_decode_cp\": true}" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS": "1" } } }, "presets": { "gb300_agentx_agg": { "label": "AgentX TP4 (Dynamo)", "hardware": "gb300", "strategy": "single_node_tp", "variants": [ "default" ], "workload": "Agentic (InferenceX AgentX)", "description": "Aggregated TP4 vLLM workers, one per 4-GPU GB300 tray, behind the Dynamo KV router with session affinity. Seven-token DSpark draft on FLASH_ATTN and prefill CUDA graphs up to 8192 tokens. KV stays in HBM, or spills to host DRAM through SimpleCPUOffloadConnector.", "source": { "label": "InferenceX #4631", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4631", "key": "minimaxm31flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg", "revision": "5d3aa21b680eb8fb66a72386bd8e85e435e15a8c" }, "image": { "vllm": "vllm/vllm-openai:minimax-m31-flash-arm64-cu134", "dynamo": "1.6.0.dev20261023", "pip": [ "ai-dynamo-runtime==1.6.0.dev20261023", "ai-dynamo==1.6.0.dev20261023" ] }, "tp": 4, "select": { "features": { "text_only": true } }, "args": [ "--enable-prefix-caching", "--max-model-len", "1048576", "--max-num-batched-tokens", "32768" ], "env": { "PYTHONHASHSEED": "0", "VLLM_ENGINE_READY_TIMEOUT_S": "7200", "VLLM_HTTP_TIMEOUT_KEEP_ALIVE": "600", "VLLM_CHAT_TEMPLATE_RENDER_TIMEOUT": "600" }, "json_args": { "--compilation-config": { "cudagraph_capture_sizes": [ 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 320, 384, 448, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1536, 2048, 3072, 4096, 6144, 8192 ] }, "--speculative-config": { "attention_backend": "FLASH_ATTN" } }, "points": { "gpu_kv": { "label": "GPU-resident KV", "description": "KV cache in HBM only: --max-num-seqs 32 for about 16 sessions per worker, 64 for up to 32.", "default": true, "args": [ "--max-num-seqs", "32", "--gpu-memory-utilization", "0.9" ], "select": { "kv_offload": "off" } }, "dram_kv": { "label": "Host-DRAM KV offload", "description": "Adds a host-DRAM KV tier (860 GB per worker) for about 40-56 sessions per worker.", "args": [ "--max-num-seqs", "96", "--gpu-memory-utilization", "0.86" ], "env": { "VLLM_USE_SIMPLE_KV_OFFLOAD": "1" }, "select": { "kv_offload": "simple" }, "json_args": { "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": null, "cpu_bytes_to_use": 860000000000, "lazy_offload": true } } } } }, "runbook": "The GB300 aggregated AgentX lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`minimaxm31flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg`, [SemiAnalysisAI/InferenceX#4631](https://github.com/SemiAnalysisAI/InferenceX/pull/4631)) serves the default checkpoint text-only, with one TP4 `dynamo.vllm` worker per 4-GPU tray behind the Dynamo frontend. Add trays to scale out: the KV router spreads conversations across them. The lane measured 29.4k tok/s/GPU at 417 tok/s/user with 16 sessions per tray (GPU-resident KV, `--max-num-seqs 32`), and 96.6k tok/s/GPU at 185 tok/s/user with 56 sessions per tray (host-DRAM KV offload, `--max-num-seqs 96`).\n\n**Image and versions.** The lane used an arm64 CUDA 13 build of the `vllm/vllm-openai:minimax-m31-flash` image (tag `minimax-m31-flash-arm64-cu134`); any build of that image family for GB300 will do. Dynamo `1.6.0.dev20261023` was installed on top: `pip install --extra-index-url https://pypi.nvidia.com ai-dynamo-runtime==1.6.0.dev20261023 ai-dynamo==1.6.0.dev20261023`.\n\n**Tuning rules.**\n\n- `--max-num-seqs` is about twice the sessions one worker serves: 32 up to 16 sessions, 64 at 32-48, 96 at 56.\n- **GPU-resident KV** (`--gpu-memory-utilization 0.9`) is enough up to about 32 sessions per tray. Beyond that, multi-turn contexts outgrow HBM and evicted prefixes are recomputed.\n- **Host-DRAM KV offload** adds `SimpleCPUOffloadConnector` with lazy offload (`VLLM_USE_SIMPLE_KV_OFFLOAD=1`). `cpu_bytes_to_use` is the budget for the whole worker, split across its four ranks: 860 GB pins 215 GB per rank. Size it from the host DRAM that is free after the engine processes are up. Lower `--gpu-memory-utilization` to 0.86 to leave room for the transfer buffers.\n- The CUDA graph sizes go up to 8192 tokens, so prefill chunks of up to 8192 tokens also replay graphs. `--max-num-batched-tokens 32768` still allows larger mixed steps, which run piecewise.\n\n**Running under Dynamo.** Start etcd and NATS (32 MB max payload) once:\n\n```bash\netcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://:2379\nprintf 'max_payload: 33554432\\n' > nats.conf && nats-server -c nats.conf\n```\n\nOn each tray, export `ETCD_ENDPOINTS=http://:2379` and `NATS_SERVER=nats://:4222` together with the generated environment and `DYN_TCP_CONNECT_TIMEOUT=120`, `DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT=1024`, `DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT=1024` and `DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT=2048`. Then run the generated flags with `python3 -m dynamo.vllm --model MiniMaxAI/MiniMax-M3.1-Flash --served-model-name MiniMaxAI/MiniMax-M3.1-Flash --trust-remote-code` in place of `vllm serve MiniMaxAI/MiniMax-M3.1-Flash`. `dynamo.vllm` rejects vLLM's API-server flags, so leave out `--tool-call-parser minimax_m3 --enable-auto-tool-choice` and add:\n\n```bash\n--dyn-tool-call-parser minimax_m3 --dyn-reasoning-parser minimax_m3 \\\n --kv-events-config '{\"publisher\":\"zmq\",\"topic\":\"kv-events\",\"enable_kv_cache_events\":true,\"endpoint\":\"tcp://*:5200\"}'\n```\n\nStart the frontend:\n\n```bash\nexport ETCD_ENDPOINTS=http://:2379 NATS_SERVER=nats://:4222\nexport DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 DYN_TCP_CONNECT_TIMEOUT=120 DYN_TOKENIZER=fastokens\npython3 -m dynamo.frontend --http-port 8000 --router-mode kv --router-session-affinity-ttl-secs 14400\n```\n\n`DYN_TOKENIZER=fastokens` switches the frontend to the fastokens tokenizer backend that ships with Dynamo. With 100K+ token agentic prompts, the default tokenizer spends about 150 ms per request on the frontend (up to 600 ms at p90), all of it added to TTFT; fastokens takes about 1 ms.\n\nThe KV router uses the workers' KV events to send each request to the worker that holds its prefix. Session affinity keeps a conversation on that worker for up to four hours; send a stable conversation ID in the `X-Dynamo-Session-ID` request header.\n\nThe InferenceX throughput points replaced block verification with synthetic rejection at the committed golden DSpark acceptance length (4.04), so every point is measured at the same acceptance rate. Production serving keeps the default rejection sampler (`rejection_sample_method: block`), as in the command above.\n", "json": "/MiniMaxAI/MiniMax-M3.1-Flash/presets/gb300_agentx_agg.json" }, "gb300_agentx_pd": { "label": "AgentX PD (Dynamo)", "hardware": "gb300", "strategy": "pd_cluster", "variants": [ "default" ], "workload": "Agentic (InferenceX AgentX)", "description": "DEP2 prefill workers, two per GB300 tray, feed one DEP8 decode worker (throughput) or DEP16 (lower latency) over NIXL. Both roles share a Mooncake KV store in host DRAM. The Dynamo KV router routes with session affinity. Seven-token DSpark draft on FLASHINFER_TS, DeepGEMM MegaMoE on both roles.", "source": { "label": "InferenceX #4631", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4631", "key": "minimaxm31flash-fp4-gb300-dynamo-vllm-agentic-dspark-disagg", "revision": "5d3aa21b680eb8fb66a72386bd8e85e435e15a8c" }, "image": { "vllm": "vllm/vllm-openai:minimax-m31-flash-arm64-cu134", "dynamo": "1.6.0.dev20261023", "pip": [ "ai-dynamo-runtime==1.6.0.dev20261023", "ai-dynamo==1.6.0.dev20261023", "mooncake-transfer-engine-cuda13==0.3.13.post1" ] }, "select": { "pd_router": "dynamo", "features": { "text_only": true, "tool_calling": false } }, "args": [ "--enable-expert-parallel", "--moe-backend", "deep_gemm_mega_moe", "--numa-bind", "--enable-prefix-caching", "--max-model-len", "1048576" ], "env": { "PYTHONHASHSEED": "0", "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_ENGINE_READY_TIMEOUT_S": "7200", "VLLM_HTTP_TIMEOUT_KEEP_ALIVE": "600", "NCCL_NET_GDR_C2C": "1", "NCCL_IB_HCA": "mlx5_0,mlx5_1,mlx5_2,mlx5_3", "NVSHMEM_REMOTE_TRANSPORT": "none", "NVSHMEM_ENABLE_NIC_PE_MAPPING": "1", "NVSHMEM_HCA_PE_MAPPING": "mlx5_0:1:1,mlx5_1:1:1,mlx5_2:1:1,mlx5_3:1:1", "UCX_TLS": "rc,cuda_copy", "UCX_NET_DEVICES": "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1", "UCX_CUDA_IPC_ENABLE_MNNVL": "n", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n", "UCX_RNDV_PIPELINE_ERROR_HANDLING": "y", "NIXL_UCX_SGL_ENABLE": "1", "WITH_NVIDIA_PEERMEM": "0", "MC_SLICE_SIZE": "1048576", "MC_MAX_MR_SIZE": "4294967296", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "DYN_TCP_CONNECT_TIMEOUT": "120", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT": "1024", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT": "1024", "DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT": "2048" }, "roles": { "prefill": { "nodes": 1, "parallelism": "dep", "args": [ "--data-parallel-size", "2", "--data-parallel-size-local", "2", "--max-num-seqs", "64", "--max-num-batched-tokens", "32768", "--long-prefill-token-threshold", "4096", "--gpu-memory-utilization", "0.85", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}", "--max-cudagraph-capture-size", "2048", "--kv-events-config", "{\"publisher\":\"zmq\",\"topic\":\"kv-events\",\"enable_kv_cache_events\":true,\"endpoint\":\"tcp://*:5200\"}", "--kv-transfer-config", "{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"backends\":[\"UCX\"],\"enforce_handshake_compat\":false,\"kv_lease_duration\":120}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_offload\":false}}]}}" ], "env": { "VLLM_MOONCAKE_LOAD_RECV_THREADS": "8" } }, "decode": { "nodes": 2, "parallelism": "dep", "remove_args": [ "--no-enable-prefix-caching" ], "args": [ "--max-num-seqs", "48", "--max-num-batched-tokens", "8192", "--watermark", "0.05", "--gpu-memory-utilization", "0.95", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}", "--max-cudagraph-capture-size", "384", "--numa-bind-nodes", "0", "0", "1", "1", "--kv-transfer-config", "{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"backends\":[\"UCX\"],\"enforce_handshake_compat\":false,\"kv_lease_duration\":120}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_offload\":false,\"save_decode_cache\":true}}]}}" ], "env": { "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "VLLM_MAX_ASYNC_KV_ADMITS_PER_STEP": "4", "VLLM_RANDOMIZE_DP_DUMMY_INPUTS": "1", "VLLM_ALLREDUCE_USE_SYMM_MEM": "0", "NCCL_CUMEM_ENABLE": "1", "NCCL_MNNVL_ENABLE": "1", "NCCL_NVLS_ENABLE": "1" } } }, "points": { "dep8": { "label": "DEP8 decode (throughput)", "description": "One DEP8 decode worker across two trays.", "default": true }, "dep16": { "label": "DEP16 decode (lower latency)", "description": "One DEP16 decode worker across four trays.", "roles": { "decode": { "nodes": 4 } } } }, "runbook": "The GB300 disaggregated AgentX lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`minimaxm31flash-fp4-gb300-dynamo-vllm-agentic-dspark-disagg`, [SemiAnalysisAI/InferenceX#4631](https://github.com/SemiAnalysisAI/InferenceX/pull/4631)) serves the default checkpoint text-only with Dynamo and vLLM. Six prefill workers, each DEP2 on two GPUs (two per 4-GPU tray, three trays), feed one decode worker that is DEP8 across two trays or DEP16 across four. KV moves from prefill to decode over NIXL (UCX over RDMA), and every rank contributes host DRAM to a shared Mooncake KV store, so a returning conversation's prefix is loaded from the store instead of recomputed. Both roles use expert parallelism with DeepGEMM MegaMoE, attention data parallelism, the V2 model runner and the seven-token DSpark draft on `FLASHINFER_TS`.\n\n| Prefill workers (DEP2) | Decode | GPUs | Sessions | tok/s/GPU | tok/s/user | TTFT p90 | E2E latency p90 |\n|---:|---|---:|---:|---:|---:|---:|---:|\n| 6 | DEP8 | 20 | 280 | 93.1k | 240 | 2.7 s | 12.2 s |\n| 6 | DEP8 | 20 | 320 | 100.5k | 229 | 3.3 s | 13.8 s |\n| 6 | DEP8 | 20 | 360 | 107.9k | 224 | 4.2 s | 15.9 s |\n| 6 | DEP8 | 20 | 400 | 114.5k | 219 | 5.9 s | 18.4 s |\n| 6 | DEP16 | 28 | 300 | 75.0k | 296 | 1.8 s | 9.3 s |\n| 6 | DEP16 | 28 | 360 | 85.8k | 285 | 2.4 s | 10.3 s |\n\nThe preset renders one prefill worker and the decode worker. Run more prefill workers with the same command. Two workers share a tray on GPUs 0-1 and 2-3 (`--device-ids 0,1` and `--device-ids 2,3` on `dynamo.vllm`). Give each its own `--data-parallel-rpc-port`, `VLLM_NIXL_SIDE_CHANNEL_PORT` and `--kv-events-config` endpoint port.\n\n**Image and versions.** The lane used an arm64 CUDA 13 build of the `vllm/vllm-openai:minimax-m31-flash` image (tag `minimax-m31-flash-arm64-cu134`), with Dynamo `1.6.0.dev20261023` and Mooncake `mooncake-transfer-engine-cuda13==0.3.13.post1` installed on every worker and on the Mooncake master host: `pip install --extra-index-url https://pypi.nvidia.com ai-dynamo-runtime==1.6.0.dev20261023 ai-dynamo==1.6.0.dev20261023 mooncake-transfer-engine-cuda13==0.3.13.post1`.\n\n### Tuning rules\n\n- **Prefill pool.** The prefill pool sets TTFT, so scale prefill workers with load. The lane's points kept TTFT p90 between 1.8 and 5.9 s with one DEP2 prefill worker per 47-67 concurrent agentic sessions, so 93-133 sessions per prefill tray (six workers at 280-400 sessions). Add prefill workers when TTFT p90 climbs while decode still has room. Two DEP2 workers per tray give the router twice as many prefill targets as one DEP4 worker.\n- **DEP8 or DEP16 decode.** DEP8 gives the most tokens per GPU: up to 50 sessions per decode GPU (400 sessions) at 219 tok/s/user. DEP16 halves the batch per rank and gives about 23% more tok/s/user (296 at 300 sessions, against 240 at 280 on DEP8) at about 19% fewer tokens per GPU, and keeps E2E latency p90 near 10 s up to 360 sessions. `--max-num-seqs 48` per DP rank is a cap, not the operating point. The decode graph cap is `--max-num-seqs` x (1 + 7 draft tokens) = 384. `--watermark 0.05` keeps 5% of the decode KV cache free, so extra requests wait in the queue instead of preempting running ones. Keep decode `--max-num-batched-tokens` at 8192: a preempted request recomputes its context in chunks of that size.\n- **Prefill batching.** `--long-prefill-token-threshold 4096` chunks long prompts into 4096-token pieces. Short prompts then share each step with them instead of waiting behind a 100K-token context, which lowers the TTFT tail. The recipe's 1P1D section above uses 8192. `--max-num-batched-tokens 32768` bounds a prefill step. Prefill captures CUDA graphs up to 2048 tokens.\n- **Mooncake pool size.** `global_segment_size` is registered by every DP rank, prefill and decode alike, so a tray pins four times that value. The lane used 180 GB per rank (720 GB per tray), which leaves 80-100 GiB of a GB300 tray's host DRAM free with the engines up; 200 GB per rank ran trays out of host memory. A larger pool raises the hit rate at high session counts: going from 160 to 180 GB per rank, with the services moved off the engine trays, lowered TTFT p90 by 0.4-0.8 s at 280-360 sessions. Size it as (host DRAM free per tray after the engines are up - headroom for NIXL and the 4 GB local buffers) / ranks per tray. The decode ranks also write the KV they generate into the store (`save_decode_cache: true`), so the next turn's prefill can load the previous answer instead of recomputing it.\n- **Routing.** The KV router (`--router-mode kv`) reads the prefill workers' KV events and sends each request to the worker that already holds its prefix. Session affinity (`--router-session-affinity-ttl-secs 14400`) keeps a conversation on that worker for up to four hours. Send a stable conversation ID in the `X-Dynamo-Session-ID` header. The Mooncake store catches the prefixes that miss after a reroute.\n- **Transport.** Cross-tray NIXL needs `UCX_TLS=rc,cuda_copy` with `UCX_NET_DEVICES` set to the tray's RDMA ports. Without them, UCX can fall back to TCP and transfers stall. Point `UCX_MODULE_DIR` at the UCX bundled with the NIXL wheel (`nixl_cu13.libs/ucx` in the image's `dist-packages`). `kv_lease_duration: 120` holds a prefill's blocks for up to 120 s until decode reads them (the default is 30 s). `VLLM_MAX_ASYNC_KV_ADMITS_PER_STEP=4` caps how many requests with finished KV loads decode admits per step.\n- **NUMA.** `--numa-bind` pins each rank to its GPU's NUMA node. Decode passes the tray's map, `--numa-bind-nodes 0 0 1 1`, explicitly.\n- `--moe-backend deep_gemm_mega_moe` is the same as the lane's `--kernel-config '{\"moe_backend\":\"deep_gemm_mega_moe\"}'`. `--prefill-schedule-interval 1`, also set by the lane, is vLLM's default.\n\n### Services\n\nStart etcd, NATS and the Mooncake master once, on a host that runs no engine workers, together with the frontend. The engine trays' host DRAM then goes to the Mooncake pool.\n\n```bash\netcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://:2379\nprintf 'max_payload: 33554432\\n' > nats.conf && nats-server -c nats.conf\nmooncake_master --rpc_port=50051 \\\n --default_kv_lease_ttl=60000 --eviction_high_watermark_ratio=0.95 --eviction_ratio=0.10\n```\n\nEvery worker reads the store configuration from the JSON file that `MOONCAKE_CONFIG_PATH` points to (export it with the rest of the environment):\n\n```json\n{\"master_server_address\": \":50051\", \"metadata_server\": \"P2PHANDSHAKE\",\n \"global_segment_size\": \"180GB\", \"local_buffer_size\": \"4GB\", \"protocol\": \"rdma\",\n \"device_name\": \"mlx5_0,mlx5_1,mlx5_2,mlx5_3\", \"enable_offload\": false}\n```\n\n### Workers\n\nExport `ETCD_ENDPOINTS=http://:2379`, `NATS_SERVER=nats://:4222` and the generated environment on every tray, then run the generated `python3 -m dynamo.vllm` commands. Add `--served-model-name MiniMaxAI/MiniMax-M3.1-Flash --trust-remote-code --dyn-tool-call-parser minimax_m3 --dyn-reasoning-parser minimax_m3` to both roles. `dynamo.vllm` rejects vLLM's API-server flags `--tool-call-parser` and `--enable-auto-tool-choice`. Multi-node DEP decode runs one process per tray with `--data-parallel-start-rank` set to 0, 4, 8 and so on, and a shared `--data-parallel-address` and `--data-parallel-rpc-port`. `--data-parallel-hybrid-lb` lets every tray's Dynamo runtime register its own DP ranks with the frontend.\n\n### Frontend\n\n```bash\nexport ETCD_ENDPOINTS=http://:2379 NATS_SERVER=nats://:4222\nexport DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 DYN_TCP_CONNECT_TIMEOUT=120 DYN_TOKENIZER=fastokens\npython3 -m dynamo.frontend --http-port 8000 --router-mode kv --router-session-affinity-ttl-secs 14400\n```\n\n`DYN_TOKENIZER=fastokens` switches the frontend to the fastokens tokenizer backend that ships with Dynamo. With 100K+ token agentic prompts, the default tokenizer spends about 150 ms per request on the frontend (up to 600 ms at p90), all of it added to TTFT; fastokens takes about 1 ms.\n\nThe InferenceX throughput points replaced block verification with synthetic rejection at the committed golden DSpark acceptance length (4.04), so every point is measured at the same acceptance rate. Production serving keeps the default rejection sampler (`rejection_sample_method: block`), as in the command above.\n", "json": "/MiniMaxAI/MiniMax-M3.1-Flash/presets/gb300_agentx_pd.json" }, "gb300_agentx_tp4": { "label": "AgentX TP4", "hardware": "gb300", "strategy": "single_node_tp", "variants": [ "default" ], "workload": "Agentic (InferenceX AgentX)", "description": "One TP4 engine on a 4-GPU GB300 tray with the seven-token DSpark draft on FLASHINFER_TS; GPU-resident KV, with per-concurrency scheduling and CUDA graph settings.", "source": { "label": "InferenceX #4662", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4662", "key": "minimaxm31flash-fp4-gb300-vllm-agentic-dspark", "revision": "85956a00e240c2b7428e1e9406f25c731ddb0470" }, "image": "vllm/vllm-openai:minimax-m31-flash-arm64-cu134@sha256:5d76347628973fb80519a919a2f2740b6645b62cba3fe2bd87b25f111bd0e948", "tp": 4, "select": { "features": { "text_only": true } }, "args": [ "--enable-prefix-caching", "--max-model-len", "1048576", "--max-num-batched-tokens", "8192" ], "env": { "VLLM_ENGINE_READY_TIMEOUT_S": "7200", "VLLM_CHAT_TEMPLATE_RENDER_TIMEOUT": "600" }, "points": { "c1": { "label": "1", "concurrency": 1, "args": [ "--max-num-seqs", "2" ] }, "c4": { "label": "4", "concurrency": 4, "args": [ "--max-num-seqs", "8" ] }, "c8": { "label": "8", "concurrency": 8, "args": [ "--max-num-seqs", "16" ], "json_args": { "--compilation-config": { "cudagraph_capture_sizes": [ 1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 264, 272, 280, 288, 296, 304, 312, 320, 328, 336, 344, 352, 360, 368, 376, 384, 392, 400, 408, 416, 424, 432, 440, 448, 456, 464, 472, 480, 488, 496, 504, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096, 4608, 5120, 5632, 6144, 6656, 7168, 7680, 8192 ] } } }, "c16": { "label": "16", "concurrency": 16, "args": [ "--max-num-seqs", "64" ], "json_args": { "--compilation-config": { "cudagraph_capture_sizes": [ 1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 264, 272, 280, 288, 296, 304, 312, 320, 328, 336, 344, 352, 360, 368, 376, 384, 392, 400, 408, 416, 424, 432, 440, 448, 456, 464, 472, 480, 488, 496, 504, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096, 4608, 5120, 5632, 6144, 6656, 7168, 7680, 8192 ] } } }, "c24": { "label": "24", "concurrency": 24, "args": [ "--max-num-seqs", "64", "--max-num-batched-tokens", "32768", "--long-prefill-token-threshold", "8192", "--prefill-schedule-interval", "16", "--cudagraph-capture-sizes", "1", "2", "4", "8", "16", "24", "32", "40", "48", "56", "64", "72", "80", "88", "96", "104", "112", "120", "128", "136", "144", "152", "160", "168", "176", "184", "192", "200", "208", "216", "224", "232", "240", "248", "256", "272", "288", "304", "320", "336", "352", "368", "384", "400", "416", "432", "448", "464", "480", "496", "512", "528", "544", "560", "576", "592", "608", "624", "640", "656", "672", "688", "704", "720", "736", "752", "768", "784", "800", "816", "832", "848", "864", "880", "896", "912", "928", "944", "960", "976", "992", "1008", "1024", "1280", "1536", "1792", "2048", "2304", "2560", "2816", "3072", "3328", "3584", "3840", "4096", "4608", "5120", "5632", "6144", "6656", "7168", "7680", "8192" ], "env": { "VLLM_HTTP_TIMEOUT_KEEP_ALIVE": "600" } }, "c32": { "label": "32", "concurrency": 32, "args": [ "--max-num-seqs", "64" ] }, "c40": { "label": "40", "concurrency": 40, "args": [ "--max-num-seqs", "64", "--long-prefill-token-threshold", "8192", "--gpu-memory-utilization", "0.95", "--watermark", "0.05", "--cudagraph-capture-sizes", "1", "2", "4", "8", "16", "24", "32", "40", "48", "56", "64", "72", "80", "88", "96", "104", "112", "120", "128", "136", "144", "152", "160", "168", "176", "184", "192", "200", "208", "216", "224", "232", "240", "248", "256", "272", "288", "304", "320", "336", "352", "368", "384", "400", "416", "432", "448", "464", "480", "496", "512", "528", "544", "560", "576", "592", "608", "624", "640", "656", "672", "688", "704", "720", "736", "752", "768", "784", "800", "816", "832", "848", "864", "880", "896", "912", "928", "944", "960", "976", "992", "1008", "1024", "1280", "1536", "1792", "2048", "2304", "2560", "2816", "3072", "3328", "3584", "3840", "4096", "4608", "5120", "5632", "6144", "6656", "7168", "7680", "8192" ], "env": { "VLLM_HTTP_TIMEOUT_KEEP_ALIVE": "600" } } }, "runbook": "The GB300 TP4 points of the AgentX lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`minimaxm31flash-fp4-gb300-vllm-agentic-dspark`, [SemiAnalysisAI/InferenceX#4662](https://github.com/SemiAnalysisAI/InferenceX/pull/4662)) serve the default checkpoint text-only on one TP4 engine per 4-GPU tray, with `--max-num-seqs` set per concurrency.\n\n- **Concurrency 1, 4 and 32** use the default CUDA graph capture sizes.\n- **Concurrency 8 and 16** capture CUDA graphs for every size up to 8192 tokens.\n- **Concurrency 24** raises `--max-num-batched-tokens` to 32768 and schedules prefill every 16 steps; **concurrency 40** keeps 8192 and raises `--gpu-memory-utilization` to 0.95. Both cap prefill chunks at 8192 tokens and capture graphs to 8192.\n", "json": "/MiniMaxAI/MiniMax-M3.1-Flash/presets/gb300_agentx_tp4.json" }, "gb300_agentx_dep4": { "label": "AgentX DEP4", "hardware": "gb300", "strategy": "single_node_dep", "variants": [ "default" ], "workload": "Agentic (InferenceX AgentX)", "description": "One data-parallel, expert-parallel engine across the four GPUs of a GB300 tray behind vllm-router with session affinity. Three-token DSpark draft on FLASHINFER_TS.", "source": { "label": "InferenceX #4662", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4662", "key": "minimaxm31flash-fp4-gb300-vllm-agentic-dspark", "revision": "85956a00e240c2b7428e1e9406f25c731ddb0470" }, "image": "vllm/vllm-openai:minimax-m31-flash-arm64-cu134@sha256:5d76347628973fb80519a919a2f2740b6645b62cba3fe2bd87b25f111bd0e948", "select": { "features": { "text_only": true } }, "args": [ "--enable-prefix-caching", "--max-model-len", "1048576", "--max-num-batched-tokens", "8192" ], "env": { "VLLM_ENGINE_READY_TIMEOUT_S": "7200", "VLLM_CHAT_TEMPLATE_RENDER_TIMEOUT": "600" }, "json_args": { "--speculative-config": { "num_speculative_tokens": 3 } }, "points": { "c64": { "label": "64", "concurrency": 64, "args": [ "--max-num-seqs", "32" ] }, "c96": { "label": "96", "concurrency": 96, "args": [ "--max-num-seqs", "48" ] }, "c128": { "label": "128", "concurrency": 128, "args": [ "--max-num-seqs", "64" ] } }, "runbook": "The DEP4 points of the GB300 AgentX lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`minimaxm31flash-fp4-gb300-vllm-agentic-dspark`, [SemiAnalysisAI/InferenceX#4662](https://github.com/SemiAnalysisAI/InferenceX/pull/4662)) run one engine with `--data-parallel-size 4` and `--enable-expert-parallel`; `--max-num-seqs` is the per-rank limit.\n\nThe router uses the `consistent_hash` policy so a conversation stays on the rank that holds its prefix. The lane also runs it with `--request-timeout-secs 14400` and `--disable-retries`.\n", "json": "/MiniMaxAI/MiniMax-M3.1-Flash/presets/gb300_agentx_dep4.json" }, "gb300_agentx_tp2": { "label": "AgentX TP2 + KV offload", "hardware": "gb300", "strategy": "single_node_tp", "variants": [ "default" ], "workload": "Agentic (InferenceX AgentX)", "description": "One TP2 engine on two of the tray's four GB300 GPUs with the seven-token DSpark draft on FLASHINFER_TS and host-DRAM KV offload (482 GB per engine).", "source": { "label": "InferenceX #4662", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4662", "key": "minimaxm31flash-fp4-gb300-vllm-agentic-dspark", "revision": "85956a00e240c2b7428e1e9406f25c731ddb0470" }, "image": "vllm/vllm-openai:minimax-m31-flash-arm64-cu134@sha256:5d76347628973fb80519a919a2f2740b6645b62cba3fe2bd87b25f111bd0e948", "tp": 2, "select": { "features": { "text_only": true }, "kv_offload": "simple" }, "args": [ "--enable-prefix-caching", "--max-model-len", "1048576", "--max-num-batched-tokens", "32768" ], "env": { "VLLM_ENGINE_READY_TIMEOUT_S": "7200", "VLLM_CHAT_TEMPLATE_RENDER_TIMEOUT": "600", "VLLM_HTTP_TIMEOUT_KEEP_ALIVE": "600", "VLLM_USE_SIMPLE_KV_OFFLOAD": "1" }, "json_args": { "--compilation-config": { "cudagraph_capture_sizes": [ 1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 264, 272, 280, 288, 296, 304, 312, 320, 328, 336, 344, 352, 360, 368, 376, 384, 392, 400, 408, 416, 424, 432, 440, 448, 456, 464, 472, 480, 488, 496, 504, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096, 4608, 5120, 5632, 6144, 6656, 7168, 7680, 8192 ] }, "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": null, "cpu_bytes_to_use": 482000000000, "lazy_offload": true } } }, "points": { "c28": { "label": "28", "concurrency": 28, "args": [ "--max-num-seqs", "42" ] }, "c32": { "label": "32", "concurrency": 32, "args": [ "--max-num-seqs", "48" ] }, "c36": { "label": "36", "concurrency": 36, "args": [ "--max-num-seqs", "54" ] } }, "runbook": "The TP2 points of the GB300 AgentX lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`minimaxm31flash-fp4-gb300-vllm-agentic-dspark`, [SemiAnalysisAI/InferenceX#4662](https://github.com/SemiAnalysisAI/InferenceX/pull/4662)) add a host-DRAM KV tier through `SimpleCPUOffloadConnector` with lazy offload. `cpu_bytes_to_use` (482 GB) is the budget for the whole engine, about half of the tray's host DRAM, so the engine needs that much free host RAM. `--max-num-seqs` is 1.5 times the concurrency.\n", "json": "/MiniMaxAI/MiniMax-M3.1-Flash/presets/gb300_agentx_tp2.json" }, "b300_agentx_tp4": { "label": "AgentX TP4", "hardware": "b300", "strategy": "single_node_tp", "variants": [ "default" ], "workload": "Agentic (InferenceX AgentX)", "description": "One TP4 engine on four of the node's eight B300 GPUs with the seven-token DSpark draft on FLASH_ATTN; GPU-resident KV up to concurrency 32, host-DRAM KV offload (about 1.5 TB per engine) at 24 and 40-56.", "source": { "label": "InferenceX #4467", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4467", "key": "minimaxm31flash-fp4-b300-vllm-agentic-dspark", "revision": "615ee422fa655b90a0551e7fe80230f543f57a10" }, "image": "vllm/vllm-openai:minimax-m31-flash@sha256:86f582dae25ea92802353727164ae4eb03b37b18baf9119b805ffaec2a7ef07c", "tp": 4, "select": { "features": { "text_only": true } }, "args": [ "--enable-prefix-caching", "--gpu-memory-utilization", "0.92", "--max-num-batched-tokens", "32768", "--stream-interval", "20" ], "env": { "VLLM_HTTP_TIMEOUT_KEEP_ALIVE": "600", "VLLM_CHAT_TEMPLATE_RENDER_TIMEOUT": "600" }, "json_args": { "--compilation-config": { "cudagraph_capture_sizes": [ 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 320, 384, 448, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1536, 2048, 3072, 4096, 6144, 8192 ] }, "--speculative-config": { "attention_backend": "FLASH_ATTN" } }, "points": { "c1": { "label": "1", "concurrency": 1, "args": [ "--max-num-seqs", "2" ], "select": { "kv_offload": "off" } }, "c4": { "label": "4", "concurrency": 4, "args": [ "--max-num-seqs", "8" ], "select": { "kv_offload": "off" } }, "c8": { "label": "8", "concurrency": 8, "args": [ "--max-num-seqs", "16" ], "select": { "kv_offload": "off" } }, "c16": { "label": "16", "concurrency": 16, "args": [ "--max-num-seqs", "32" ], "select": { "kv_offload": "off" } }, "c24": { "label": "24 (KV offload)", "concurrency": 24, "args": [ "--max-num-seqs", "48" ], "select": { "kv_offload": "simple" }, "json_args": { "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": null, "cpu_bytes_to_use": 1499000000000, "lazy_offload": true } } } }, "c32": { "label": "32", "concurrency": 32, "args": [ "--max-num-seqs", "64" ], "select": { "kv_offload": "off" } }, "c40": { "label": "40 (KV offload)", "concurrency": 40, "args": [ "--max-num-seqs", "64" ], "select": { "kv_offload": "simple" }, "json_args": { "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": null, "cpu_bytes_to_use": 1499000000000, "lazy_offload": true } } } }, "c48": { "label": "48 (KV offload)", "concurrency": 48, "args": [ "--max-num-seqs", "64" ], "select": { "kv_offload": "simple" }, "json_args": { "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": null, "cpu_bytes_to_use": 1499000000000, "lazy_offload": true } } } }, "c56": { "label": "56 (KV offload)", "concurrency": 56, "args": [ "--max-num-seqs", "96" ], "select": { "kv_offload": "simple" }, "json_args": { "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": null, "cpu_bytes_to_use": 1499000000000, "lazy_offload": true } } } } }, "runbook": "The B300 AgentX lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`minimaxm31flash-fp4-b300-vllm-agentic-dspark`, [SemiAnalysisAI/InferenceX#4467](https://github.com/SemiAnalysisAI/InferenceX/pull/4467), run [40340506758](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40340506758)) serves the default checkpoint (MXFP8 dense layers, NVFP4 experts, NVFP4 KV and indexer K caches) as one aggregated text-only vLLM engine on a single 8-GPU B300 node. Its TP4 points run the seven-token DSpark draft with `FLASH_ATTN` draft attention; the recipe's NVIDIA default, `FLASHINFER_TS`, is what the [AgentX TP2](#preset-b300_agentx_tp2) points run.\n\n- **GPU-resident points** (concurrency 1-16 and 32) keep the whole KV cache in HBM, with `--max-num-seqs` at twice the concurrency (64 at 32).\n- **KV offload points** (24 and 40-56) add a host-DRAM KV tier through `SimpleCPUOffloadConnector` with lazy offload. `cpu_bytes_to_use` (1,499 GB) is the budget for the whole engine: each of the four ranks pins 349 GiB of host memory, so the engine needs about 1.5 TB of free host RAM. Pinning takes a couple of minutes at startup.\n- The CUDA graph capture sizes go up to 8192 tokens, so prefill chunks of up to 8192 tokens also replay graphs.\n\nThe InferenceX throughput points replaced block verification with synthetic rejection at the committed golden DSpark acceptance length (4.04) so every point is measured at the same acceptance rate. The command above keeps block verification.\n", "json": "/MiniMaxAI/MiniMax-M3.1-Flash/presets/b300_agentx_tp4.json" }, "b300_agentx_tp2": { "label": "AgentX TP2 + KV offload", "hardware": "b300", "strategy": "single_node_tp", "variants": [ "default" ], "workload": "Agentic (InferenceX AgentX)", "description": "One TP2 engine on two of the node's eight B300 GPUs with the seven-token DSpark draft on FLASHINFER_TS and host-DRAM KV offload (about 750 GB per engine).", "source": { "label": "InferenceX #4467", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4467", "key": "minimaxm31flash-fp4-b300-vllm-agentic-dspark", "revision": "615ee422fa655b90a0551e7fe80230f543f57a10" }, "image": "vllm/vllm-openai:minimax-m31-flash@sha256:86f582dae25ea92802353727164ae4eb03b37b18baf9119b805ffaec2a7ef07c", "tp": 2, "select": { "features": { "text_only": true }, "kv_offload": "simple" }, "args": [ "--enable-prefix-caching", "--gpu-memory-utilization", "0.9", "--max-num-batched-tokens", "32768", "--stream-interval", "20" ], "env": { "VLLM_HTTP_TIMEOUT_KEEP_ALIVE": "600", "VLLM_CHAT_TEMPLATE_RENDER_TIMEOUT": "600" }, "json_args": { "--compilation-config": { "cudagraph_capture_sizes": [ 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 320, 384, 448, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1536, 2048, 3072, 4096, 6144, 8192 ] }, "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": null, "cpu_bytes_to_use": 749000000000, "lazy_offload": true } } }, "points": { "c24": { "label": "24", "concurrency": 24, "args": [ "--max-num-seqs", "48" ] }, "c32": { "label": "32", "concurrency": 32, "args": [ "--max-num-seqs", "48" ] }, "c40": { "label": "40", "concurrency": 40, "args": [ "--max-num-seqs", "60" ] }, "c48": { "label": "48", "concurrency": 48, "args": [ "--max-num-seqs", "72" ] } }, "runbook": "The TP2 points of the B300 AgentX lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`minimaxm31flash-fp4-b300-vllm-agentic-dspark`, [SemiAnalysisAI/InferenceX#4467](https://github.com/SemiAnalysisAI/InferenceX/pull/4467), run [40340506758](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40340506758)) serve the default checkpoint as one aggregated text-only vLLM engine on two GPUs of an 8-GPU B300 node, with the recipe's seven-token DSpark draft on `FLASHINFER_TS` (the [AgentX TP4](#preset-b300_agentx_tp4) points use `FLASH_ATTN`).\n\nEvery point adds a host-DRAM KV tier through `SimpleCPUOffloadConnector` with lazy offload. `cpu_bytes_to_use` (749 GB) is the budget for the whole engine: each of the two ranks pins about 349 GiB of host memory. The lane ran one engine per node; several TP2 engines on one node each need their own share of host RAM.\n\nThe InferenceX throughput points replaced block verification with synthetic rejection at the committed golden DSpark acceptance length (4.04) so every point is measured at the same acceptance rate. The command above keeps block verification.\n", "json": "/MiniMaxAI/MiniMax-M3.1-Flash/presets/b300_agentx_tp2.json" }, "gb200_agentx_agg": { "label": "AgentX TP4 (Dynamo)", "hardware": "gb200", "strategy": "single_node_tp", "variants": [ "default" ], "workload": "Agentic (InferenceX AgentX)", "description": "Aggregated TP4 vLLM workers, one per 4-GPU GB200 tray, behind the Dynamo KV router with session affinity. Seven-token DSpark draft on FLASH_ATTN and prefill CUDA graphs up to 8192 tokens. KV stays in HBM, or spills to host DRAM through SimpleCPUOffloadConnector.", "source": { "label": "InferenceX #4632", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4632", "key": "minimaxm31flash-fp4-gb200-dynamo-vllm-agentic-dspark-agg", "revision": "936213146b79b55e8a11ffbc1d6010145995c6a6" }, "image": { "vllm": "vllm/vllm-openai:minimax-m31-flash-arm64-cu134", "dynamo": "1.6.0.dev20261023", "pip": [ "ai-dynamo-runtime==1.6.0.dev20261023", "ai-dynamo==1.6.0.dev20261023" ] }, "tp": 4, "select": { "features": { "text_only": true } }, "args": [ "--enable-prefix-caching", "--max-model-len", "1048576", "--max-num-batched-tokens", "32768" ], "env": { "PYTHONHASHSEED": "0", "VLLM_ENGINE_READY_TIMEOUT_S": "7200" }, "json_args": { "--compilation-config": { "cudagraph_capture_sizes": [ 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 320, 384, 448, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1536, 2048, 3072, 4096, 6144, 8192 ] }, "--speculative-config": { "attention_backend": "FLASH_ATTN" } }, "points": { "gpu_kv": { "label": "GPU-resident KV", "description": "KV cache in HBM only, for up to about 16 sessions per worker.", "default": true, "args": [ "--max-num-seqs", "32", "--gpu-memory-utilization", "0.88" ], "select": { "kv_offload": "off" } }, "dram_kv": { "label": "Host-DRAM KV offload", "description": "Adds a host-DRAM KV tier (360 GB per worker) for about 32-40 sessions per worker.", "args": [ "--max-num-seqs", "64", "--gpu-memory-utilization", "0.82", "--kv-cache-metrics" ], "select": { "kv_offload": "simple" }, "json_args": { "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": null, "cpu_bytes_to_use": 360000000000, "lazy_offload": true } } } } }, "runbook": "The GB200 aggregated AgentX lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`minimaxm31flash-fp4-gb200-dynamo-vllm-agentic-dspark-agg`, [SemiAnalysisAI/InferenceX#4632](https://github.com/SemiAnalysisAI/InferenceX/pull/4632)) serves the default checkpoint text-only, with one TP4 `dynamo.vllm` worker per 4-GPU tray behind the Dynamo frontend. Add trays to scale out: the KV router spreads conversations across them. At 16 sessions per tray (GPU-resident KV) the lane measured 29.3k tok/s/GPU at 402 tok/s/user.\n\n**Image and versions.** The lane used an arm64 CUDA 13 build of the `vllm/vllm-openai:minimax-m31-flash` image (tag `minimax-m31-flash-arm64-cu134`); any build of that image family for GB200 will do. Dynamo `1.6.0.dev20261023` was installed on top: `pip install --extra-index-url https://pypi.nvidia.com ai-dynamo-runtime==1.6.0.dev20261023 ai-dynamo==1.6.0.dev20261023`.\n\n**Tuning rules.**\n\n- `--max-num-seqs` is about twice the sessions one worker serves: 32 up to 16 sessions, 64 at 32-40.\n- **GPU-resident KV** (`--gpu-memory-utilization 0.88`) is enough up to about 16 sessions per tray. Beyond that, multi-turn contexts outgrow HBM and evicted prefixes are recomputed.\n- **Host-DRAM KV offload** adds `SimpleCPUOffloadConnector` with lazy offload. `cpu_bytes_to_use` is the budget for the whole worker, split across its four ranks: 360 GB pins 90 GB per rank. Size it from the host DRAM that is free after the engine processes are up. Lower `--gpu-memory-utilization` to 0.82 to leave room for the transfer buffers. `--kv-cache-metrics` exports KV residency metrics, which help with that sizing; it can be left out.\n- The CUDA graph sizes go up to 8192 tokens, so prefill chunks of up to 8192 tokens also replay graphs. `--max-num-batched-tokens 32768` still allows larger mixed steps, which run piecewise.\n\n**Running under Dynamo.** Start etcd and NATS (32 MB max payload) once:\n\n```bash\netcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://:2379\nprintf 'max_payload: 33554432\\n' > nats.conf && nats-server -c nats.conf\n```\n\nOn each tray, export `ETCD_ENDPOINTS=http://:2379` and `NATS_SERVER=nats://:4222` together with the generated environment and `DYN_TCP_CONNECT_TIMEOUT=120`, `DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT=1024`, `DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT=1024` and `DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT=2048`. Then run the generated flags with `python3 -m dynamo.vllm --model MiniMaxAI/MiniMax-M3.1-Flash --served-model-name MiniMaxAI/MiniMax-M3.1-Flash --trust-remote-code` in place of `vllm serve MiniMaxAI/MiniMax-M3.1-Flash`. `dynamo.vllm` rejects vLLM's API-server flags, so leave out `--tool-call-parser minimax_m3 --enable-auto-tool-choice` and add:\n\n```bash\n--dyn-tool-call-parser minimax_m3 --dyn-reasoning-parser minimax_m3 \\\n --kv-events-config '{\"publisher\":\"zmq\",\"topic\":\"kv-events\",\"enable_kv_cache_events\":true,\"endpoint\":\"tcp://*:5200\"}'\n```\n\nStart the frontend:\n\n```bash\nexport ETCD_ENDPOINTS=http://:2379 NATS_SERVER=nats://:4222\nexport DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 DYN_TCP_CONNECT_TIMEOUT=120 DYN_TOKENIZER=fastokens\npython3 -m dynamo.frontend --http-port 8000 --router-mode kv --router-session-affinity-ttl-secs 14400\n```\n\n`DYN_TOKENIZER=fastokens` switches the frontend to the fastokens tokenizer backend that ships with Dynamo. With 100K+ token agentic prompts, the default tokenizer spends about 150 ms per request on the frontend (up to 600 ms at p90), all of it added to TTFT; fastokens takes about 1 ms.\n\nThe KV router uses the workers' KV events to send each request to the worker that holds its prefix. Session affinity keeps a conversation on that worker for up to four hours; send a stable conversation ID in the `X-Dynamo-Session-ID` request header.\n\nThe InferenceX throughput points replaced block verification with synthetic rejection at the committed golden DSpark acceptance length (4.04), so every point is measured at the same acceptance rate. Production serving keeps the default rejection sampler (`rejection_sample_method: block`), as in the command above.\n", "json": "/MiniMaxAI/MiniMax-M3.1-Flash/presets/gb200_agentx_agg.json" }, "gb200_agentx_pd": { "label": "AgentX PD (Dynamo)", "hardware": "gb200", "strategy": "pd_cluster", "variants": [ "default" ], "workload": "Agentic (InferenceX AgentX)", "description": "DEP4 prefill workers, one per GB200 tray, feed one DEP8 decode worker (throughput) or DEP16 (lower latency) over NIXL. Both roles share a Mooncake KV store in host DRAM. The Dynamo KV router routes with session affinity. Seven-token DSpark draft on FLASHINFER_TS, DeepGEMM MegaMoE on both roles.", "source": { "label": "InferenceX #4632", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4632", "key": "minimaxm31flash-fp4-gb200-dynamo-vllm-agentic-dspark-disagg", "revision": "936213146b79b55e8a11ffbc1d6010145995c6a6" }, "image": { "vllm": "vllm/vllm-openai:minimax-m31-flash-arm64-cu134", "dynamo": "1.6.0.dev20261023", "pip": [ "ai-dynamo-runtime==1.6.0.dev20261023", "ai-dynamo==1.6.0.dev20261023", "mooncake-transfer-engine-cuda13==0.3.13.post1" ] }, "select": { "pd_router": "dynamo", "features": { "text_only": true, "tool_calling": false } }, "args": [ "--enable-expert-parallel", "--moe-backend", "deep_gemm_mega_moe", "--numa-bind", "--enable-prefix-caching", "--max-model-len", "1048576", "--load-format", "fastsafetensors" ], "env": { "PYTHONHASHSEED": "0", "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_ENGINE_READY_TIMEOUT_S": "7200", "VLLM_HTTP_TIMEOUT_KEEP_ALIVE": "600", "VLLM_RANDOMIZE_DP_DUMMY_INPUTS": "1", "NCCL_NET_GDR_C2C": "1", "NCCL_IB_HCA": "mlx5_0,mlx5_1,mlx5_2,mlx5_3", "NVSHMEM_REMOTE_TRANSPORT": "none", "NVSHMEM_ENABLE_NIC_PE_MAPPING": "1", "NVSHMEM_HCA_PE_MAPPING": "mlx5_0:1:1,mlx5_1:1:1,mlx5_2:1:1,mlx5_3:1:1", "UCX_TLS": "rc,cuda_copy", "UCX_NET_DEVICES": "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1", "UCX_CUDA_IPC_ENABLE_MNNVL": "n", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n", "UCX_RNDV_PIPELINE_ERROR_HANDLING": "y", "NIXL_UCX_SGL_ENABLE": "1", "WITH_NVIDIA_PEERMEM": "0", "MC_SLICE_SIZE": "1048576", "MC_MAX_MR_SIZE": "4294967296", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "DYN_TCP_CONNECT_TIMEOUT": "120", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT": "1024", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT": "1024", "DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT": "2048" }, "roles": { "prefill": { "nodes": 1, "parallelism": "dep", "args": [ "--data-parallel-size", "4", "--data-parallel-size-local", "4", "--max-num-seqs", "64", "--max-num-batched-tokens", "16384", "--long-prefill-token-threshold", "4096", "--gpu-memory-utilization", "0.92", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}", "--max-cudagraph-capture-size", "2048", "--kv-events-config", "{\"publisher\":\"zmq\",\"topic\":\"kv-events\",\"enable_kv_cache_events\":true,\"endpoint\":\"tcp://*:5200\"}", "--kv-transfer-config", "{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"backends\":[\"UCX\"],\"enforce_handshake_compat\":false,\"kv_lease_duration\":120}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_offload\":false}}]}}" ], "env": { "VLLM_MOONCAKE_LOAD_RECV_THREADS": "8" } }, "decode": { "nodes": 2, "parallelism": "dep", "remove_args": [ "--no-enable-prefix-caching" ], "args": [ "--max-num-seqs", "64", "--max-num-batched-tokens", "8192", "--watermark", "0.05", "--gpu-memory-utilization", "0.88", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}", "--max-cudagraph-capture-size", "512", "--numa-bind-nodes", "0", "0", "1", "1", "--kv-transfer-config", "{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"backends\":[\"UCX\"],\"enforce_handshake_compat\":false,\"kv_lease_duration\":120}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_consumer\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_offload\":false,\"save_decode_cache\":false,\"enable_lookup\":false}}]}}" ], "env": { "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "VLLM_ALLREDUCE_USE_SYMM_MEM": "0", "NCCL_CUMEM_ENABLE": "1", "NCCL_MNNVL_ENABLE": "1", "NCCL_NVLS_ENABLE": "1" } } }, "points": { "dep8": { "label": "DEP8 decode (throughput)", "description": "One DEP8 decode worker across two trays.", "default": true }, "dep16": { "label": "DEP16 decode (lower latency)", "description": "One DEP16 decode worker across four trays.", "roles": { "decode": { "nodes": 4 } } } }, "runbook": "The GB200 disaggregated AgentX lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`minimaxm31flash-fp4-gb200-dynamo-vllm-agentic-dspark-disagg`, [SemiAnalysisAI/InferenceX#4632](https://github.com/SemiAnalysisAI/InferenceX/pull/4632)) serves the default checkpoint text-only with Dynamo and vLLM. Several prefill workers, each DEP4 on one 4-GPU tray, feed one decode worker that is DEP8 across two trays or DEP16 across four. KV moves from prefill to decode over NIXL (UCX over RDMA), and every rank contributes host DRAM to a shared Mooncake KV store, so a returning conversation's prefix is loaded from the store instead of recomputed. Both roles use expert parallelism with DeepGEMM MegaMoE, attention data parallelism, the V2 model runner and the seven-token DSpark draft on `FLASHINFER_TS`.\n\n| Prefill workers (DEP4) | Decode | GPUs | Sessions | tok/s/GPU | tok/s/user | TTFT p90 | E2E latency p90 |\n|---:|---|---:|---:|---:|---:|---:|---:|\n| 2 | DEP8 | 16 | 224 | 94.2k | 250 | 3.0 s | 12.2 s |\n| 2 | DEP8 | 16 | 256 | 98.8k | 244 | 5.3 s | 15.7 s |\n| 3 | DEP8 | 20 | 256 | 86.4k | 240 | 2.4 s | 12.1 s |\n| 3 | DEP8 | 20 | 288 | 93.2k | 230 | 3.5 s | 13.3 s |\n| 3 | DEP16 | 28 | 280 | 70.4k | 302 | 1.9 s | 9.4 s |\n| 3 | DEP16 | 28 | 320 | 77.3k | 291 | 2.2 s | 10.0 s |\n| 3 | DEP16 | 28 | 360 | 85.4k | 284 | 2.5 s | 10.6 s |\n| 4 | DEP16 | 32 | 352 | 74.5k | 284 | 1.8 s | 9.9 s |\n\nThe preset renders one prefill worker and the decode worker. Run more prefill workers with the same command, one per tray.\n\n**Image and versions.** The lane used an arm64 CUDA 13 build of the `vllm/vllm-openai:minimax-m31-flash` image (tag `minimax-m31-flash-arm64-cu134`), with Dynamo `1.6.0.dev20261023` and Mooncake `mooncake-transfer-engine-cuda13==0.3.13.post1` installed on every worker and on the Mooncake master host: `pip install --extra-index-url https://pypi.nvidia.com ai-dynamo-runtime==1.6.0.dev20261023 ai-dynamo==1.6.0.dev20261023 mooncake-transfer-engine-cuda13==0.3.13.post1`.\n\n### Tuning rules\n\n- **Prefill pool.** The prefill pool sets TTFT, so scale prefill workers with load. The lane's points kept TTFT p90 between 1.8 and 5.3 s with one DEP4 prefill tray per 85-128 concurrent agentic sessions (2 trays at 224-256 sessions, 3 at 256-360, 4 at 352). Two prefill trays per DEP8 decode give the most tokens per GPU; a third keeps TTFT p90 near 3.5 s up to 288 sessions. Add a prefill tray when TTFT p90 climbs while decode still has room.\n- **DEP8 or DEP16 decode.** DEP8 gives the most tokens per GPU: 28-36 sessions per decode GPU at 230-250 tok/s/user. DEP16 halves the batch per rank and gives about 25% more tok/s/user (302 at 280 sessions) and holds E2E latency p90 near 10 s up to 360 sessions, at about 10-20% fewer tokens per GPU. `--max-num-seqs 64` per DP rank is a cap, not the operating point. The decode graph cap is `--max-num-seqs` x (1 + 7 draft tokens) = 512. `--watermark 0.05` keeps 5% of the decode KV cache free, so extra requests wait in the queue instead of preempting running ones. Keep decode `--max-num-batched-tokens` at 8192: a preempted request recomputes its context in chunks of that size.\n- **Prefill batching.** `--long-prefill-token-threshold 4096` chunks long prompts into 4096-token pieces. Short prompts then share each step with them instead of waiting behind a 100K-token context, which lowers the TTFT tail. The recipe's 1P1D section above uses 8192. `--max-num-batched-tokens 16384` bounds a prefill step. Prefill captures CUDA graphs up to 2048 tokens. Prefill runs at `--gpu-memory-utilization 0.92`, decode at 0.88.\n- **Mooncake pool size.** `global_segment_size` is registered by every DP rank, prefill and decode alike, so a tray pins four times that value. The lane used 190 GB per rank (760 GB per tray), which lets three DEP4 prefill trays serve up to 288 sessions with DEP8 and 360 with DEP16; 210 GB per rank ran the prefill trays out of host memory. Size it as (host DRAM free per tray after the engines are up - headroom for NIXL and the 4 GB local buffers) / ranks per tray. The decode ranks only add capacity to the pool (`kv_consumer` with `enable_lookup: false` and `save_decode_cache: false`): the prefill workers write and read every block.\n- **Routing.** The KV router (`--router-mode kv`) reads the prefill workers' KV events and sends each request to the worker that already holds its prefix. Session affinity (`--router-session-affinity-ttl-secs 14400`) keeps a conversation on that worker for up to four hours. Send a stable conversation ID in the `X-Dynamo-Session-ID` header. The Mooncake store catches the prefixes that miss after a reroute.\n- **Transport.** Cross-tray NIXL needs `UCX_TLS=rc,cuda_copy` with `UCX_NET_DEVICES` set to the tray's RDMA ports. Without them, UCX can fall back to TCP and transfers stall. Point `UCX_MODULE_DIR` at the UCX bundled with the NIXL wheel (`nixl_cu13.libs/ucx` in the image's `dist-packages`). `kv_lease_duration: 120` holds a prefill's blocks for up to 120 s until decode reads them (the default is 30 s).\n- **NUMA.** `--numa-bind` pins each rank to its GPU's NUMA node. Decode passes the tray's map, `--numa-bind-nodes 0 0 1 1`, explicitly.\n- `--moe-backend deep_gemm_mega_moe` is the same as the lane's `--kernel-config '{\"moe_backend\":\"deep_gemm_mega_moe\"}'`. `--prefill-schedule-interval 1`, also set by the lane, is vLLM's default.\n\n### Services\n\nStart etcd, NATS and the Mooncake master once, on a host that runs no engine workers, together with the frontend. The engine trays' host DRAM then goes to the Mooncake pool.\n\n```bash\netcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://:2379\nprintf 'max_payload: 33554432\\n' > nats.conf && nats-server -c nats.conf\nmooncake_master --rpc_port=50051 \\\n --default_kv_lease_ttl=60000 --eviction_high_watermark_ratio=0.95 --eviction_ratio=0.10\n```\n\nEvery worker reads the store configuration from the JSON file that `MOONCAKE_CONFIG_PATH` points to (export it with the rest of the environment):\n\n```json\n{\"master_server_address\": \":50051\", \"metadata_server\": \"P2PHANDSHAKE\",\n \"global_segment_size\": \"190GB\", \"local_buffer_size\": \"4GB\", \"protocol\": \"rdma\",\n \"device_name\": \"mlx5_0,mlx5_1,mlx5_2,mlx5_3\", \"enable_offload\": false}\n```\n\n### Workers\n\nExport `ETCD_ENDPOINTS=http://:2379`, `NATS_SERVER=nats://:4222` and the generated environment on every tray, then run the generated `python3 -m dynamo.vllm` commands. Add `--served-model-name MiniMaxAI/MiniMax-M3.1-Flash --trust-remote-code --dyn-tool-call-parser minimax_m3 --dyn-reasoning-parser minimax_m3` to both roles. `dynamo.vllm` rejects vLLM's API-server flags `--tool-call-parser` and `--enable-auto-tool-choice`. Multi-node DEP decode runs one process per tray with `--data-parallel-start-rank` set to 0, 4, 8 and so on, and a shared `--data-parallel-address` and `--data-parallel-rpc-port`. `--data-parallel-hybrid-lb` lets every tray's Dynamo runtime register its own DP ranks with the frontend.\n\n### Frontend\n\n```bash\nexport ETCD_ENDPOINTS=http://:2379 NATS_SERVER=nats://:4222\nexport DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 DYN_TCP_CONNECT_TIMEOUT=120 DYN_TOKENIZER=fastokens\npython3 -m dynamo.frontend --http-port 8000 --router-mode kv --router-session-affinity-ttl-secs 14400\n```\n\n`DYN_TOKENIZER=fastokens` switches the frontend to the fastokens tokenizer backend that ships with Dynamo. With 100K+ token agentic prompts, the default tokenizer spends about 150 ms per request on the frontend (up to 600 ms at p90), all of it added to TTFT; fastokens takes about 1 ms.\n\nThe InferenceX throughput points replaced block verification with synthetic rejection at the committed golden DSpark acceptance length (4.04), so every point is measured at the same acceptance rate. Production serving keeps the default rejection sampler (`rejection_sample_method: block`), as in the command above.\n", "json": "/MiniMaxAI/MiniMax-M3.1-Flash/presets/gb200_agentx_pd.json" }, "mi355x_agentx": { "label": "AgentX TP4", "hardware": "mi355x", "strategy": "single_node_tp", "variants": [ "mxfp4" ], "workload": "Agentic (InferenceX AgentX)", "description": "One TP4 engine on four of the node's eight MI355X GPUs, serving the Quark MXFP4 checkpoint: seven-token DSpark with mono decode and an FP8 KV cache up to concurrency 8, seven-token DSpark with DBO and an NVFP4 KV cache at 16 and 32, and three-token DSpark with DBO and a host-DRAM KV tier at 64.", "source": { "label": "InferenceX #4671", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4671", "key": "minimaxm31flash-fp4-mi355x-vllm-agentic-dspark", "revision": "a328b443582c45e45eef7f0c4f133f3e173f0211" }, "image": "vllm/vllm-openai-rocm:minimax-m31-flash@sha256:ae8681212aa5291d8c27f06e90016435e3fca144a68fd5c439231f6475f74569", "tp": 4, "select": { "features": { "text_only": true, "spec_decoding": true } }, "args": [ "--enable-prefix-caching" ], "env": { "VLLM_HTTP_TIMEOUT_KEEP_ALIVE": "900" }, "json_args": { "--speculative-config": { "num_speculative_tokens": 7, "rejection_sample_method": "block" } }, "points": { "c1": { "label": "1 (mono)", "concurrency": 1, "select": { "features": { "mono_decode": "mono" } }, "args": [ "--max-num-seqs", "2", "--max-num-batched-tokens", "8192" ] }, "c2": { "label": "2 (mono)", "concurrency": 2, "select": { "features": { "mono_decode": "mono" } }, "args": [ "--max-num-seqs", "4", "--max-num-batched-tokens", "8192" ] }, "c4": { "label": "4 (mono)", "concurrency": 4, "select": { "features": { "mono_decode": "mono" } }, "args": [ "--max-num-seqs", "8", "--max-num-batched-tokens", "8192" ] }, "c8": { "label": "8 (mono)", "concurrency": 8, "select": { "features": { "mono_decode": "mono" } }, "args": [ "--max-num-seqs", "16", "--max-num-batched-tokens", "8192" ] }, "c16": { "label": "16 (DBO)", "concurrency": 16, "args": [ "--max-num-seqs", "32", "--max-num-batched-tokens", "32768", "--gpu-memory-utilization", "0.85", "--enable-dbo", "--dbo-prefill-token-threshold", "16384", "--dbo-decode-token-threshold", "32768" ], "env": { "VLLM_USE_V2_MODEL_RUNNER": "1", "DEBUG_HIP_DYNAMIC_QUEUES": "0" } }, "c32": { "label": "32 (DBO)", "concurrency": 32, "args": [ "--max-num-seqs", "64", "--max-num-batched-tokens", "32768", "--gpu-memory-utilization", "0.85", "--enable-dbo", "--dbo-prefill-token-threshold", "16384", "--dbo-decode-token-threshold", "32768" ], "env": { "VLLM_USE_V2_MODEL_RUNNER": "1", "DEBUG_HIP_DYNAMIC_QUEUES": "0" } }, "c64": { "label": "64 (DBO, KV offload)", "concurrency": 64, "select": { "kv_offload": "simple" }, "args": [ "--max-num-seqs", "128", "--max-num-batched-tokens", "32768", "--gpu-memory-utilization", "0.9", "--enable-dbo", "--dbo-prefill-token-threshold", "16384", "--dbo-decode-token-threshold", "32768" ], "env": { "VLLM_USE_V2_MODEL_RUNNER": "1", "DEBUG_HIP_DYNAMIC_QUEUES": "0", "PYTHONHASHSEED": "42" }, "json_args": { "--speculative-config": { "num_speculative_tokens": 3 }, "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": null, "cpu_bytes_to_use": 1349000000000, "lazy_offload": false } } } } }, "runbook": "The MI355X AgentX lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/)\n(`minimaxm31flash-fp4-mi355x-vllm-agentic-dspark`,\n[SemiAnalysisAI/InferenceX#4671](https://github.com/SemiAnalysisAI/InferenceX/pull/4671)),\non the Quark MXFP4 checkpoint with `--max-num-seqs` at twice the concurrency\n(c64 admits 128). Throughput runs replaced block verification with synthetic\nrejection at the committed golden DSpark acceptance length (4.04 for seven\ntokens, 2.88 for three); the commands above keep block verification.\n\n- **Concurrency 1-8** run mono decode with an FP8 KV cache and no DBO: a decode\n step of up to 16 tokens runs each layer as one fused kernel, and prefill steps\n stay at 8192 tokens.\n- **Concurrency 16 and 32** run seven-token DSpark with DBO and an NVFP4 KV cache\n at 0.85 memory utilization, leaving headroom for DBO prefill.\n- **Concurrency 64** runs three-token DSpark with DBO at 0.90 and adds a host-DRAM\n KV tier through `SimpleCPUOffloadConnector`, so cached prefixes are not lost\n once the working set outgrows the GPU KV cache (14.6M tokens at 0.90).\n `cpu_bytes_to_use` (1,349 GB) is the budget for the whole engine.\n\n[InferenceX run 41003224750](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/41003224750)\npassed all seven performance points and GSM8K at concurrency 8, 32 and 64 at\n`a328b4435` (single run):\n\n| Concurrency | Interactivity p50 / p90 (tokens/s/user) | Total tokens/s per GPU |\n| --- | ---: | ---: |\n| 1 | 551.0 / 515.7 | 6,766 |\n| 2 | 548.9 / 410.4 | 6,918 |\n| 4 | 546.3 / 395.6 | 9,494 |\n| 8 | 407.4 / 240.0 | 15,411 |\n| 16 | 228.6 / 168.3 | 27,275 |\n| 32 | 172.3 / 115.1 | 54,158 |\n| 64 | 71.4 / 47.3 | 75,434 |\n\nGSM8K exact match (strict) over 1319 questions was 0.964 at concurrency 8, 0.959 at\n32 and 0.957 at 64.\n\n- **Concurrency 64 needed a rerun.** The first attempt hung at its first load from\n the host KV tier (23 fetches in progress, none completing; `sample_tokens` RPC\n timeout) and was rerun once, which passed. Treat the offload point as less\n proven than the others.\n- **Sequence-parallel prefill does not apply to this checkpoint.** InferenceX also\n set `VLLM_ROCM_MINIMAX_M3_SP_PREFILL=1` at concurrency 64, and the server logged\n sequence-parallel prefill as off; it is omitted here. SP prefill is available on\n the default checkpoint through the DBO / SP prefill feature.\n", "json": "/MiniMaxAI/MiniMax-M3.1-Flash/presets/mi355x_agentx.json" } }, "guide": "## Overview\n\n[MiniMax-M3.1-Flash](https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash) is a\nvision-language MoE model from [MiniMax](https://www.minimax.io/) that keeps\nthe MoE architecture of [MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3):\nabout 428B total / 23B active parameters and a 1M-token context, with MSA\nsparse attention in every layer.\n\n- **Quantization-aware training** — the checkpoint stores MXFP8 dense layers\n and NVFP4 experts.\n- **Built-in DSpark draft** — the `dspark/` directory of the checkpoint;\n `--speculative-config` needs no draft model path.\n- **Native multimodal** — image and video inputs.\n\n## Prerequisites\n\n- **AMD:** MI355X (gfx950), ROCm 7.2+, TP4.\n- **NVIDIA Blackwell:** GB200, GB300, B200 or B300, TP4; NVFP4 MoE runs\n natively on the FP4 tensor cores.\n- **NVIDIA Hopper:** H100 or H200, TP8 on a full node, with the FP8\n variant: FP8 KV and indexer caches, NVFP4 MoE on Marlin.\n- **`--block-size 128` is mandatory** (MSA sparse/index cache).\n\n## Launching the Server\n\n### Single node (TP4, AMD MI355X)\n\n```bash\nVLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \\\nvllm serve MiniMaxAI/MiniMax-M3.1-Flash \\\n -tp 4 \\\n --reasoning-parser minimax_m3 \\\n --block-size 128 \\\n --kv-cache-dtype nvfp4 \\\n --attention-config '{\"indexer_kv_dtype\": \"nvfp4\", \"indexer_decode_cp\": true}' \\\n --quantization-config.moe mxfp8 \\\n --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":3,\"attention_backend\":\"TRITON_ATTN\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\"}' \\\n --disable-hybrid-kv-cache-manager\n```\n\n- `--kv-cache-dtype nvfp4` and `indexer_kv_dtype: nvfp4` store the KV cache\n and the sparse-attention indexer's K cache in NVFP4.\n- `indexer_decode_cp` splits the indexer's decode scoring across the TP ranks\n by context instead of by index head.\n- `--quantization-config.moe mxfp8` re-quantizes the NVFP4 experts to MXFP8 at\n load for the AITER MoE.\n\n### DBO and SP prefill (TP4, AMD MI355X)\n\nTwo opt-in switches make long prefill steps faster on the single-node MI355X\nengine (the **DBO / SP prefill** feature in the command builder):\n\n- **DBO** (dual-batch overlap): `--enable-dbo` splits a prefill step into two\n microbatches and runs one microbatch's all-reduce under the other's compute.\n `--dbo-decode-token-threshold 32768` keeps decode steps whole. DBO needs\n `VLLM_USE_V2_MODEL_RUNNER=1`: without it, `--enable-dbo` selects the V1\n runner, which refuses the DSpark draft.\n- **SP** (sequence-parallel prefill): `VLLM_ROCM_MINIMAX_M3_SP_PREFILL=1`\n keeps the residual stream sharded by tokens on steps of 4096+ tokens, so\n each layer's two all-reduces become an all-to-all and a reduce-scatter. It\n keeps a full `o_proj` copy per GPU, about 3.1 GB less KV cache.\n- **SP + DBO**: set both, with `--dbo-prefill-token-threshold 16384`, so each\n DBO microbatch has 8192+ tokens and stays on the SP path.\n\nOn the MXFP8 MoE, one 32K-token prefill takes 1035 ms plain, 655 ms with DBO\nand 484 ms with SP + DBO. The server log confirms SP with\n`MiniMax-M3 sequence-parallel prefill (from 4096 tokens, DBO microbatches from 8192): on`.\n\n```bash\nVLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \\\nVLLM_USE_V2_MODEL_RUNNER=1 VLLM_ROCM_MINIMAX_M3_SP_PREFILL=1 \\\nvllm serve MiniMaxAI/MiniMax-M3.1-Flash \\\n -tp 4 \\\n --reasoning-parser minimax_m3 \\\n --block-size 128 \\\n --kv-cache-dtype nvfp4 \\\n --attention-config '{\"indexer_kv_dtype\": \"nvfp4\", \"indexer_decode_cp\": true}' \\\n --quantization-config.moe mxfp8 \\\n --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":3,\"attention_backend\":\"TRITON_ATTN\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\"}' \\\n --enable-dbo \\\n --dbo-prefill-token-threshold 16384 \\\n --dbo-decode-token-threshold 32768 \\\n --disable-hybrid-kv-cache-manager\n```\n\n### Single node (TP4, NVIDIA GB200)\n\n```bash\nVLLM_FLOAT32_MATMUL_PRECISION=high \\\nvllm serve MiniMaxAI/MiniMax-M3.1-Flash \\\n -tp 4 \\\n --reasoning-parser minimax_m3 \\\n --block-size 128 \\\n --kv-cache-dtype nvfp4 \\\n --attention-config '{\"indexer_kv_dtype\": \"nvfp4\", \"use_prefill_query_quantization\": true}' \\\n --compilation-config '{\"cudagraph_mode\": \"FULL_AND_PIECEWISE\"}' \\\n --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"attention_backend\":\"FLASHINFER_TS\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}' \\\n --max-num-seqs 256 \\\n --disable-hybrid-kv-cache-manager\n```\n\n- Blackwell runs the NVFP4 experts natively, so no\n `--quantization-config.moe` override is needed.\n- `use_prefill_query_quantization` quantizes the prefill query on the MSA\n indexer path (Q8KV4).\n\n### Single node (TP8, NVIDIA H100 / H200)\n\n```bash\nvllm serve MiniMaxAI/MiniMax-M3.1-Flash \\\n --tensor-parallel-size 8 \\\n --reasoning-parser minimax_m3 \\\n --block-size 128 \\\n --kv-cache-dtype fp8 \\\n --attention-config '{\"indexer_kv_dtype\": \"fp8\"}' \\\n --compilation-config '{\"cudagraph_mode\": \"FULL_AND_PIECEWISE\"}' \\\n --max-model-len 262144 \\\n --max-num-seqs 80 \\\n --limit-mm-per-prompt '{\"video\": 0}' \\\n --gpu-memory-utilization 0.9 \\\n --no-enable-log-requests \\\n --disable-hybrid-kv-cache-manager \\\n --moe-backend marlin \\\n --enable-auto-tool-choice \\\n --tool-call-parser minimax_m3\n```\n\n- Hopper has no FP4 tensor cores: the KV cache and the indexer K cache use\n FP8, and the NVFP4 experts run on the Marlin MoE kernel.\n- `--limit-mm-per-prompt '{\"video\": 0}'` turns off video inputs; images\n still work.\n- DSpark speculative decoding is not enabled on Hopper.\n\n### Prefill/decode disaggregation (1P1D, TP4 + TP4, AMD MI355X)\n\nPrefill node:\n\n```bash\nVLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \\\nVLLM_KV_CACHE_LAYOUT=BLHNC VLLM_NIXL_SIDE_CHANNEL_HOST= \\\nvllm serve MiniMaxAI/MiniMax-M3.1-Flash \\\n -tp 4 \\\n --reasoning-parser minimax_m3 \\\n --block-size 128 \\\n --kv-cache-dtype nvfp4 \\\n --attention-config '{\"indexer_kv_dtype\": \"nvfp4\"}' \\\n --quantization-config.moe mxfp8 \\\n --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":3,\"attention_backend\":\"TRITON_ATTN\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\"}' \\\n --disable-hybrid-kv-cache-manager \\\n --max-num-batched-tokens 32768 \\\n --port 8100 \\\n --kv-transfer-config '{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"connectors\":[\n {\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_producer\",\"kv_connector_extra_config\":{\"num_threads\":4}},\n {\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use_per_rank\":601295421440,\"cpu_hugepage_block_size\":\"1GB\",\"lazy_offload\":false}}]}}'\n```\n\nDecode node:\n\n```bash\nVLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \\\nVLLM_KV_CACHE_LAYOUT=BLHNC VLLM_NIXL_SIDE_CHANNEL_HOST= \\\nvllm serve MiniMaxAI/MiniMax-M3.1-Flash \\\n -tp 4 \\\n --reasoning-parser minimax_m3 \\\n --block-size 128 \\\n --kv-cache-dtype nvfp4 \\\n --attention-config '{\"indexer_kv_dtype\": \"nvfp4\", \"indexer_decode_cp\": true}' \\\n --quantization-config.moe mxfp8 \\\n --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":3,\"attention_backend\":\"TRITON_ATTN\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\"}' \\\n --disable-hybrid-kv-cache-manager \\\n --port 8200 \\\n --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_consumer\",\"kv_connector_extra_config\":{\"num_threads\":4}}'\n```\n\n- `` and `` are each node's own address, reachable from the\n other node (the NIXL side channel).\n- `VLLM_KV_CACHE_LAYOUT=BLHNC` goes on both nodes. It keeps all layers of a\n block contiguous, so NIXL reads one region per block instead of one per\n layer.\n- The prefill node never decodes, so it leaves `indexer_decode_cp` off.\n- The prefill node also offloads KV to host memory\n (`SimpleCPUOffloadConnector`): 560 GiB per rank in 1 GiB HugeTLB pages, so\n 2240 pages for the four ranks. Reserve them on the host before the start\n (and allow them in the container's hugetlb limit), or the server fails at\n startup with `Cannot map 560.00 GiB of 1024 MiB HugeTLB pages`:\n\n ```bash\n echo 2240 | sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages\n ```\n\n### Prefill/decode disaggregation (1P1D, DEP2 + DEP4, NVIDIA GB200 / GB300)\n\nA DEP2 prefill on one 4-GPU tray feeds a DEP4 decode on a second tray over\nNIXL. Both roles run attention data parallel with the NVFP4 experts\non the DeepGEMM MegaMoE kernel over expert parallelism, the V2 model runner,\nNUMA binding, and a seven-token DSpark draft on the FlashInfer PrimiTS\n(`FLASHINFER_TS`) backend. The settings were tuned on GB300; GB200 uses the\nsame commands.\n\nPrefill node:\n\n```bash\nVLLM_FLOAT32_MATMUL_PRECISION=high VLLM_USE_V2_MODEL_RUNNER=1 \\\nVLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_NIXL_SIDE_CHANNEL_HOST= \\\nvllm serve MiniMaxAI/MiniMax-M3.1-Flash \\\n --data-parallel-size 2 \\\n --enable-expert-parallel \\\n --moe-backend deep_gemm_mega_moe \\\n --numa-bind \\\n --reasoning-parser minimax_m3 \\\n --block-size 128 \\\n --kv-cache-dtype nvfp4 \\\n --attention-config '{\"indexer_kv_dtype\": \"nvfp4\", \"use_prefill_query_quantization\": true}' \\\n --compilation-config '{\"cudagraph_mode\": \"FULL_AND_PIECEWISE\"}' \\\n --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"attention_backend\":\"FLASHINFER_TS\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}' \\\n --max-num-seqs 64 \\\n --max-num-batched-tokens 32768 \\\n --long-prefill-token-threshold 8192 \\\n --gpu-memory-utilization 0.9 \\\n --enable-prefix-caching \\\n --disable-hybrid-kv-cache-manager \\\n --port 8100 \\\n --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_producer\",\"kv_load_failure_policy\":\"fail\",\"kv_connector_extra_config\":{\"engine_ttl\":0}}'\n```\n\nDecode node:\n\n```bash\nVLLM_FLOAT32_MATMUL_PRECISION=high VLLM_USE_V2_MODEL_RUNNER=1 \\\nVLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_NIXL_SIDE_CHANNEL_HOST= \\\nvllm serve MiniMaxAI/MiniMax-M3.1-Flash \\\n --data-parallel-size 4 \\\n --enable-expert-parallel \\\n --moe-backend deep_gemm_mega_moe \\\n --numa-bind \\\n --reasoning-parser minimax_m3 \\\n --block-size 128 \\\n --kv-cache-dtype nvfp4 \\\n --attention-config '{\"indexer_kv_dtype\": \"nvfp4\", \"use_prefill_query_quantization\": true}' \\\n --compilation-config '{\"cudagraph_mode\": \"FULL_DECODE_ONLY\", \"max_cudagraph_capture_size\": 224}' \\\n --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"attention_backend\":\"FLASHINFER_TS\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}' \\\n --max-num-seqs 28 \\\n --max-num-batched-tokens 8192 \\\n --gpu-memory-utilization 0.95 \\\n --watermark 0.05 \\\n --no-enable-prefix-caching \\\n --disable-hybrid-kv-cache-manager \\\n --port 8200 \\\n --kv-transfer-config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_consumer\",\"kv_load_failure_policy\":\"fail\",\"kv_connector_extra_config\":{\"engine_ttl\":0}}'\n```\n\n- `VLLM_ENGINE_READY_TIMEOUT_S=3600`: with data parallel, each API server\n waits this long for its engines to start (default 600 s). A cold prefill\n start (JIT builds, CUDA graph capture, FlashInfer autotune) takes longer\n than 10 minutes.\n- `--max-num-seqs` is per DP rank. The decode CUDA graphs cover 28 requests\n of 1 + 7 tokens (224).\n- `--watermark 0.05` keeps 5% of the decode KV cache free when admitting\n requests, so extra requests wait in the queue instead of preempting\n running ones. On GB300 (about 4.5M KV tokens per decode rank) at a\n 200K-token mean context, expect about 19-21 running requests per rank.\n- Keep the decode `--max-num-batched-tokens` at 8192: a preempted request\n recomputes its context in chunks of that size, and every DP rank runs\n eager steps until it finishes.\n- `engine_ttl: 0` turns off NIXL's cleanup of idle remote engines; with the\n default one-hour TTL, a decode crashed an hour after its prefill went away.\n- Prefix caching stays on the prefill node only. For multi-turn traffic, the\n prefill node can also offload KV to host memory, 180 GiB per rank:\n `--kv-transfer-config '{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_producer\",\"kv_connector_extra_config\":{\"engine_ttl\":0}},{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use_per_rank\":193273528320,\"lazy_offload\":true}}]}}'`.\n\nFor a quick test, put the toy NIXL proxy from the vLLM repository in front of\nthe two nodes and send requests to port 8000:\n\n```bash\npython tests/v1/kv_connector/nixl_integration/toy_proxy_server.py --port 8000 \\\n --prefiller-host --prefiller-port 8100 \\\n --decoder-host --decoder-port 8200\n```\n\n## Client Usage\n\nRecommended sampling parameters (from the model card): `temperature = 1.0`,\n`top_p = 0.95`. Set the reasoning effort per request with\n`chat_template_kwargs: {\"reasoning_effort\": \"\"}`: `low`, `medium`,\n`high`, `xhigh` or `max`.\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"MiniMaxAI/MiniMax-M3.1-Flash\",\n \"temperature\": 1.0,\n \"top_p\": 0.95,\n \"messages\": [\n {\"role\": \"user\", \"content\": \"Explain MSA sparse attention in 3 bullets.\"}\n ]\n }'\n```\n\n## Benchmarking\n\n```bash\nvllm bench serve \\\n --backend vllm \\\n --model MiniMaxAI/MiniMax-M3.1-Flash \\\n --endpoint /v1/completions \\\n --dataset-name random \\\n --random-input 8192 \\\n --random-output 1024 \\\n --max-concurrency 32 \\\n --num-prompts 320\n```\n\n## Troubleshooting\n\n- **`--block-size` mismatch.** MSA's sparse block size is 128; the vLLM KV\n cache block size must match (otherwise `No common block size for 16`).\n- **Parsers.** `--tool-call-parser` and `--reasoning-parser` both use\n `minimax_m3`.\n- **HugeTLB pages (PD prefill node).** See the AMD PD section above.\n\n## References\n\n- [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash)\n- [MiniMax-M3 recipe](https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3)\n- [MiniMax](https://www.minimax.io/)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "a39b1759dba64342a9262a640c46f63abc0e2d6c", "path": "recipes/models/MiniMaxAI/MiniMax-M3.1-Flash.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/MiniMaxAI/MiniMax-M3.1-Flash.yaml", "overlays": [] } }