Make local and production model inference predictable across speed, memory, and compatibility

“### Summary Serving `Qwen3.8-Flash-Next` with `VLLM_PLE_CPU_OFFLOAD=1` at `tensor_parallel_size=1` hangs permanently during startup. Engine init completes fully — weights load, KV cache is allocated, FlashInfer autotune finishes — and then the process goes silent before `init engine took Xs`. The API server never starts. No exception, no CUDA OOM, no kernel OOM-kill. Reproduced 3/3 times, waiting 25, 60 and 60…”

“### Name and Version Reproduced identically on three builds (CPU-only, `cmake -DCMAKE_BUILD_TYPE=Release`, Linux x86_64): - `f280b26983` (current at time of testing) - `4d19b28` (master, includes #27679) - `60addddf3c` (about a week older) ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell ./build/bin/llama-server -m…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### Name and Version llama-b10639-bin-win-rocm-7.14-x64 build: 5e6a37cb1 (10639) built with Clang 20.1.8 for Windows x86_64 ### Operating systems Windows ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell llama-server.exe -m Qwen3.8-27B-UD-Q4_K_XL.gguf -ctk q8_0 -ctv q5_0 or llama-bench.exe -m Qwen3.8-27B-UD-Q4_K_XL.gguf -ctk q8_0 -ctv q5_0 ``` ### Problem description &…”

“**Summary.** `Qwen/Qwen3.8-Flash-Next-FP8` ships `quantization_config.modules_to_not_convert` with multimodal-wrapper paths (`model.language_model.layers...`). When the checkpoint is loaded through `AutoModelForCausalLM` the text stack is named `model.layers...`, so none of the exclusion patterns match. The FineGrainedFP8 quantizer then converts modules the vendor intended to keep in bf16 (e.g. the PLE…”

“### 🚀 The feature, motivation and pitch ## Motivation Qwen3.8-Flash-Next introduces a very large N-gram / PLE embedding table. This part of the model is particularly suitable for offloading because it is fundamentally a lookup workload rather than a large GEMM-heavy compute workload. A useful deployment topology would be: ```text Primary GPU - Main model - Attention / GDN - MoE computation - KV cache - CUDA…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“_This is a proposal for contribution. This feature request comes with the corresponding proposed patches_ The target scope is: - Apple Silicon hardware - The MLX Metal backend - Qwen3.8 27B dense model with NVFP4 quantization The objective is to accelerate prefill and decode speed via FP16 execution paths on Apple Silicon. I know the target is narrow but I believe running Qwen on Macs still reaches a wide…”

“## Environment - vLLM 0.27.1, `vllm/vllm-openai` image digest `sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967` (created 2026-08-11) - Model: `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` (`NemotronHForCausalLM`), served weight-only FP4 via the Marlin fallback on SM89 - `--tensor-parallel-size 2` on 2x RTX 4090, `--enable-lora --max-lora-rank 16 --lora-modules` (one adapter),…”

“### Your current environment Environment ```text vLLM: 0.27.1 (pip, isolated venv) torch: 2.13.0+cu130, CUDA 13.0 GPU: NVIDIA GeForce RTX 3090, 24 GiB (23.56 GiB usable), sm_86 Driver: 610.43.02 OS: Debian 13, kernel 6.12.101+deb13-amd64 Python: 3.13.5 Model: RedHatAI/Qwen3.8-27B-INT4 (compressed-tensors, ships model_mtp.safetensors) model_type qwen3_5, vocab_size 248320, hidden_size 5120, tie_word_embeddings…”

“# Vulkan: ~78% decode throughput cliff at 131072 context window — default ~1 GiB suballocation fragmentation; fixed via GGML_VK_SUBALLOCATION_BLOCK_SIZE=4 GiB ## Environment | | | |---|---| | GPU | AMD Radeon RX 7900 XTX (24 GB, RDNA3, gfx1100) | | OS / Driver | Windows 11, AMD proprietary driver | | llama.cpp | 0.2.0-dev (build **10603**, commit c060ca974), Vulkan backend (ggml-vulkan.dll) | | Model |…”

“## DFlash2 on Ampere: online W8/FP8 of the draft + `hf_overrides` callable **Hardware:** 2× RTX 3090 (SM86), vLLM 0.27.1 **Target:** Qwen3.8-27B AWQ (`Qwen3_5ForConditionalGeneration`) **Draft:** `Qwen3.8-27B-DFlash2` BF16, `method=dflash`, n=7 **Goal:** 4 concurrent 128k sessions. KV fp8_e4m3 pool is 428 331 tokens = **3.35× at 128k**. Need ~2.1 GiB/GPU more. The BF16 draft is 3.58 GiB; INT8 of the 34 large…”

“## Environment - llama.cpp: upstream master as of 2026-08-26 (build **b10643**), self-built Release / CUDA sm_86. The only local patches are unrelated GGML kernel optimizations; `common/chat.cpp` and the server chat pipeline are vanilla. - Model: Qwen3.8-27B-Q4_K_M GGUF with embedded official jinja template (+mmproj) - Server flags: `-c 120000 -fa on -ctk q4_0 -ctv q4_0 --split-mode layer -ts 1,1,1 --spec-type…”

“### Name and Version build : b10549-b2e5e9b28 windows vulkan 64bit ### Operating systems Windows ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell Server (no speedup from MTP): llama-server.exe ^ -m "Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced.Q4_0.gguf" ^ --model-draft "mtp-gemma-4-26B-A4B-it.gguf" ^ --spec-type draft-mtp ^ --spec-draft-n-max 2 ^ -np 1 ^ -ngl 99 ^…”

“### Name and Version version: 0.3.0-dev (build 1, commit 5d5cb4c) built with Clang 23.0.0 for Linux x86_64 ### Operating systems Linux ### GGML backends HIP ### Hardware AMD Ryzen AI 9 HX 370 + Radeon 890M, i.e. gfx1150. ### Models Repo: `bartowski/huihui-ai_Qwen3-Coder-Next-abliterated-GGUF`. File: `huihui-ai_Qwen3-Coder-Next-abliterated-Q6_K/huihui-ai_Qwen3-Coder-Next-abliterated-Q6_K-00001-of-00002.gguf`. ###…”

“### Name and Version ## llama.cpp Version Information ``` Git commit: 47c786924 (HEAD, tag: b9999) kleidiai : add SME2 f32 kernel (#24414) Build version: version: 9999 (47c786924) built with GNU 11.4.0 for Linux aarch64 ``` ### Operating systems Linux ### GGML backends CUDA ### Hardware ## Hardware Information ``` CPU: CPU(s): 6 On-line CPU(s) list: 0-5 Model name: Cortex-A78AE Thread(s) per core: 1 NUMA node0…”

“## Proposal to improve performance ### Summary Model Runner V2 pipeline parallelism sends sampled results from the last PP rank back to the non-last ranks. The CUDA path uses a sibling process group and a side stream, as introduced by [#42187](https://github.com/vllm-project/vllm/pull/42187). However, each non-last rank posts its receive in local feedback step `T`, while that result is not consumed until `T +…”

“### Motivation. **TL;DR** 1. We propose to use Helion by default for some CustomOps. 2. We show that Helion kernels have superior performance compared to the CUDA kernels in vLLM. Across broad shape sweeps, the three Helion kernels we benchmarked achieve 1.382–1.785x geomean speedups on H100, 1.556–1.804x on B200, and 1.022–1.427x on MI350X. On Qwen3-8B-FP8, this translates to 2.59%–3.49% higher end-to-end…”

“It would be great if the backend can support [llama.cpp](https://github.com/ggml-org/llama.cpp) inference engines which would enable rapid expansion to new quantized model. For intance, unsloth released a new dynamic quantization method that works great and still gives you a usable model for a fraction of the resources the full one needs. Read [here](https://unsloth.ai/docs/basics/dynamic-3.0-ggufs) Which also…”

“Hello, I am trying to understand how to get a clean installation without unneccessary packages into my virtualenv. I have a setup with 2 servers: A is with AMD CPU to run docling B is an Nvidia Jetson ORIN running llama.cpp connected via LAN. During the installation of docling, pip installs a lot of dependencies including some packages that will never be able to run onto server A: pip install docling ``` ...…”