Make local and production model inference predictable across speed, memory, and compatibility

“### Model description hey there, i saw transformers still havent support kimi k3 architecture yet, while llama.cpp for inference already. i have trained new tiny model from scratch using the kimi k3 architecture (0.5B & 1.35B), it also fully compatible with llama.cpp gguf. but it has no Vision, no MTP, and its full dense. so, i wonder when will transformers finally add support kimi k3 architecture, also i saw…”

“The idea is to implement/perform this speculative-free post-training method for popular ggml-optimized LLM models (e.g. Qwen3.8-27B). Alternatively, when such post-trained models become available, implement support for their diffusion-augmented variants in llama.cpp / ggml. This will possibly accelerate inference with a small memory footprint, enabling inference of popular LLM models in smaller GPUs.”

“### Name and Version version: 0.4.1-dev (build 10991, commit 930e2fa59) built with IntelLLVM 2026.1.1 for Linux x86_64 ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? Other (Please specify in the next section) ### Command line ```shell grep or rg ggml_backend_buffer_type_t grep -F ggml_backend_buffer_type_t ggml/include/ggml-backend.h grep -F ggml_backend_sycl_split_buffer_type…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### Name and Version version: 0.4.1-dev (build 10989, commit 38a5b42d9) built with GNU 13.3.0 for Linux x86_64 ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? Other (Please specify in the next section) ### Command line ```shell ./build/bin/llama-batched-bench -m models/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf -npl 1,2,4,8 -lv 4 -c 120000 -fit on ``` ### Problem description & steps to…”

“### 🚀 The feature, motivation and pitch ## Summary Integrate the CFT counted-write MoE all-to-all kernel introduced in [TensorRT-LLM PR #17647](https://github.com/NVIDIA/TensorRT-LLM/pull/17647) into vLLM through FlashInfer. ## Background The kernel adds a Compute Fabric Transport (CFT) path for MoE dispatch and combine communication. It uses CUDA Logical Endpoints and `fabric.try_put.counted` operations to…”

“## Summary In PD-disaggregated serving on ROCm/MI300X (MoRIIO KV connector, WideEP DP8/EP8), the **decode** engine emits degenerate repeated-token garbage (`locklocklock…` / `!!!`) for **remote-prefilled** requests whenever decode runs under **any CUDA-graph capture** (`FULL`, `PIECEWISE`, or `FULL_AND_PIECEWISE`). With `DECODE_CUDAGRAPH_MODE=NONE` (eager) the identical stack decodes **coherently**. Colocated…”

“### Name and Version commit 96ffdc41c (master, 2026-09-14), built locally with -DGGML_VULKAN=ON, GNU 16.2.1, Linux x86_64 ggml_vulkan: 0 = AMD Radeon RX 9070 XT (RADV GFX1201) (radv) | uma: 0 | fp16: dot2 | bf16: 1 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? Other (Please specify in the…”

“### Name and Version Using latest docker images in docker compose ``` docker compose exec llama.cpp /app/llama --version WARN[0000] The "HF_TOKEN" variable is not set. Defaulting to a blank string. version: 0.4.1-dev (build 10975, commit 4c9233c03) built with GNU 14.2.0 for Linux x86_64 ``` ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? _No response_ ### Command line ```shell…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### Your current environment - vLLM nightly `0.28.1rc1.dev681+ge7edf17ce` (2026-09-11), image `vllm/vllm-openai` nightly aarch64, torch 2.13 + cu130 - 1 tray of GB200 NVL72 (4× GB200, SM100), `zai-org/GLM-5.3-Flash` FP8, recipe `single_node_tp` flags minus parsers: `--tensor-parallel-size 4 --kv-cache-dtype fp8 --max-model-len 16384 --no-enable-prefix-caching` - Load: vllm-bench (Rust) random 8192-token prompts /…”

“### Name and Version Reproduced only through Ollama 0.33.3 on NVIDIA DGX Spark (GB10, aarch64, CUDA SM121). Ollama’s bundled llama.cpp runner; I do not have a standalone llama-cli --version for this run. Have not yet reproduced with llama-server / llama-completion from this repo. ### Operating systems Linux ### GGML backends CUDA ### Hardware NVIDIA DGX Spark (GB10 Grace Blackwell), aarch64, 128 GB unified…”

“**Environment** - 2x AMD Strix Halo, 10 GbE link. llama.cpp master `8ea29024` (2026-09-11), identical build on the llama-server (router) leader and the ggml-rpc-server worker. Reproduced identically on `18443257a3` (2026-09). - Leader runs llama-server in router mode: each model child opens its own long-lived RPC connection to the worker (`--rpc :50052`). **Repro** Load a second model while a first model's RPC…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### Motivation. The disaggregated serving path in vLLM is render → generate → derender. `/render` turns an OpenAI request into a `GenerateRequest` (#36166). `/inference/v1/generate` runs it on the GPU tier and returns token IDs (#22817, #24261). `/v1/chat/completions/derender` and `/v1/completions/derender` turn the result back into an OpenAI response (#42729). This works well for batch and RL traffic where the…”

“### 🚀 The feature, motivation and pitch In a single-node scenario involving concurrent prefill and decode requests, the decode process remains in a phase of low generation speed for an extended period. This phenomenon is exacerbated when KV cache capacity is limited. ### Alternatives _No response_ ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for…”

“We designed and implemented KVMem for Qwen3.8-27B: a new way for llama.cpp to manage conversation memory. The basic idea is simple. A 16GB card cannot keep a full 256K KV cache in VRAM, so we do not try. Finished history lives in regular RAM; for each new question the GPU only loads a limited window of the pieces that matter. We checked this at 256K on LongMemEval-S and AgentLongBench: a 32K GPU window is…”

“## Summary On a hybrid GDN model (`qwen3_5` architecture: 48 gated-delta-net linear-attention layers + 16 full-attention layers; model is Qwen3.8-27B), `llama-perplexity` results are **not monotonic in weight precision**: a full **F16** model scores *worse* than its own **Q4_K_M** quantization on 7 of 8 chunks, and the ranking among quants is inverted vs their bit width. Since full precision cannot legitimately…”

“### Motivation. vLLM already supports surfacing MoE routing data to clients via the `routed_experts` field when the server is started with `--enable-return-routed-experts`. This is useful for MoE research, routing analysis, expert-load/utilization studies, distillation, and debugging expert selection. Today this only works for **non-streaming** responses: - `CompletionResponseChoice.routed_experts`…”

“AI-authored (disclosed): produced and verified on our live CMP 170HX cluster; the account owner is a user, not a vLLM dev. ### Environment - Model: Qwen3.8-27B (hybrid: 48 GDN linear-attention layers + 16 full-attention layers), AWQ int4 with eq8emb embeddings, `max_model_len 262144`. - Draft: DFlash2-W4A16, `num_speculative_tokens=7`. - Engine: v0.13.0 line via the `wtdcode/vllm-backport` port image…”

“Adreno libOpenCL_system.so (Android /system_ext/lib64) is not ICD-loader compatible — heads-up for anyone chasing this path Device: RedMagic 10 Pro, Snapdragon 8 Elite (SM8750), Adreno 830, Android 16, no root. Running llama.cpp under Termux (Android userspace terminal, no NDK/system-app context). Why I went looking The usual vendor OpenCL blob (/vendor/lib64/libOpenCL_adreno.so) is blocked for non-system apps by…”

“## Summary Cross-posting field data from a 3-day bring-up of **DeepSeek-V4.1-Flash** (769B total, MXFP4 experts, Engram memory) on **8x RTX PRO 6000 (SM120, consumer Blackwell)**, since SM120 users currently have to rediscover all of this from scratch. Related: #54929 (Triton sparse-MLA fallback — we left perf data there), #5135-tracked flashinfer instantiation gaps. ## What works (final configuration) - vLLM…”

“### Your current environment ### current environment vLLM version: 0.29.0 (official `vllm/vllm-openai:latest` image) Also reproduced on a custom fork/dev build (version reports as `0.1.dev20051+g487ecf187`) Deployment: Kubernetes, single GPU (Qwen2.5-7B-Instruct) and 8x GPU tensor-parallel (custom GLM5Next model) opentelemetry-sdk: 1.41.0 / 1.40.0 (confirmed on both builds) opentelemetry-exporter-otlp-proto-http:…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### Misc discussion on performance I ran `vllm bench throughput` on `deepseek-ai/DeepSeek-V4.1-Flash` (`--tensor-parallel-size 8`, ShareGPT, 1000 prompts, seed 42) on 8x B200 with the precompiled-wheel editable install from the CUDA install doc, and got 24.6 s for the 1000 requests in a clean run but 27.6 to 27.9 s in about one run out of three. A per-step timer showed one prefill step taking 3.4 s instead of…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### Git commit There is an issue where after the frontend stops inference, the CPP backend continues computing. The frontend (multiple IDEs including CodeBuddy) has had its Stop button clicked, and the frontend shows that inference has stopped, yet the GPU utilization on the backend remains at 90%. I asked an AI about the possible cause. The AI's response is as follows: ## llama.cpp server Stop on Streaming…”

“### Your current environment - OS: Linux container; two-node deployment (private host details omitted) - Python: 3.12.13 - vLLM runtime: 0.28.0+cu129 - PyTorch: 2.13.0+cu129 (CUDA 12.9) - Ray: 2.48.0 - Model: Kimi K3 - Source inspection: vLLM main snapshot d86257e (2026-09-12) The benchmark output and logs are available on request; no private host addresses, credentials, or raw service logs are included here. ###…”

“Hello, I was experimenting with a very old instance on some low-end hardware (AMD HX470) and found this strange self-referential logic in the logs. It seems the system is maintaining a kind of internal consistency by actively creating deviations from the global optimum. Is this a known edge case in low-precision quantization, or could it be some form of emergent behavior in specific hardware constraints? Here is…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“# [MSVC] `\uHHHH` escapes in BPE pre-tokenizer patterns abort model load (`regex_error`) ## Problem Any BPE pre-tokenizer pattern containing `\uHHHH` escapes (e.g. K2-Horizon `\u200C`/`\u200D`) aborts model loading on native Windows builds: ```text Failed to process regex: '(?:...|\u200C|\u200D)...' Regex error: regex_error(error_escape): The expression contained an invalid escaped character, or a trailing…”

“### What is the issue? ## Jetson Orin Nano 8GB: Gemma 4 E4B multimodal projector causes host OOM, despite successful CPU-projector configuration ### Description On an NVIDIA Jetson Orin Nano Super 8GB with unified CPU/GPU memory, Ollama 0.34.0 cannot load a Gemma 4 E4B multimodal model because the multimodal projector is offloaded to CUDA. The underlying `llama-server` can successfully run the exact same model on…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“I ported DeepSeek-V4.1-Flash to a llama.cpp fork and ran it on a single RTX 5090 (31.8 GiB of VRAM) with 125.7 GiB of RAM, with the routed experts streaming from NVMe through a VRAM cache and a pinned host tier. It runs exactly — logits within the reference implementation's own fp8 rounding floor — at 5.1 tokens/s on new content and 21 tokens/s on content already resident in the cache. Report, tools and raw…”

“### Name and Version `llama-cpp-python` 0.3.35 (prebuilt CUDA wheel from `https://abetlen.github.io/llama-cpp-python/whl/cu124`, installed via `pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124`) CPU capability banner printed by the loaded backend: ``` CUDA : ARCHS = 600,610,700,750,800,860,890,900 | FORCE_MMQ = 1 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 |…”

“### Name and Version ❯ llama-server --version version: 0.3.0-dev (build 10781, commit unknown) built with GNU 16.2.1 for Linux x86_64 ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell llama-server --device Vulkan0 -b 2048 -ub 1024 --port 10000 --slot-save-path /home/llamacpp/slots/ -cram 32768 -bs --models-max 1 -np 1 --sleep-idle-seconds…”

“### Name and Version version: b10900 (win-cuda-13.3-x64) Also reproduced on b10472 (win-cuda-13.3-x64). ### Operating systems Windows ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell llama-server -m Qwen3.8-27B-Q4_K_M.gguf --mmproj mmproj-Qwen3.8-27B-f16.gguf ^ -ngl 99 -fa on -ctk q4_0 -ctv q4_0 --jinja -c 131072 -np 1 ^ --image-min-tokens 1024 --mcp-servers-config…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### Name and Version marco@marco-desktop:~/ai/llama.cpp/build/bin$ ./llama-cli --version version: 0.4.0-dev (build 10867, commit f3f1a8f27) built with GNU 13.3.0 for Linux x86_64 ### Operating systems Linux ### GGML backends HIP ### Hardware i5-9600k + RX 9070 XT ### Models In this case, I reproduced with the following models, but I think it will work with any models: - Ministral-3-14B-Reasoning-2512-Q4_K_M -…”

“### Name and Version bin/llama-cli --version version: 0.4.0-dev (build 10868, commit 304665fe7) built with IntelLLVM 2026.1.1 for Linux x86_64 Environment: Ubuntu 24.04, kernel 6.17.0-42-generic ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? Other:: SYCL ### Command line ```shell $ . /opt/intel/oneapi/2026.1/oneapi-vars.sh $ cmake -B build -DGGML_CUDA=OFF \ -DGGML_SYCL=ON \…”

“### Name and Version version: 0.3.0-dev (build 10621, commit c1d0e7a00) built with Clang 20.1.8 for Windows x86_64 ### Operating systems Windows ### GGML backends CUDA ### Hardware 7950x + RTX3090 ### Models https://huggingface.co/mradermacher/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU-GGUF Q4_K_M This was used for testing but I have tried many other Qwen3.5/3.6/3.8 models and most of…”

“### Your current environment import os import sys import time import signal import subprocess import json # ================= 双卡配置 ================= GPU_DEVICES = "0,1" os.environ["CUDA_VISIBLE_DEVICES"] = GPU_DEVICES # 修复 libstdc++ 版本兼容性问题 conda_lib_path = os.path.join(sys.prefix, "lib") if os.path.exists(conda_lib_path): old_ld = os.environ.get("LD_LIBRARY_PATH", "").strip(":") os.environ["LD_LIBRARY_PATH"] = (…”

“## Problem I want to run Qwen3.8-Flash-Next (NVFP4, 48 layers x 512 experts) on an RTX 6000 Ada (48 GB) at a usable speed. The expert weights do not fit in GPU memory, so part of them has to live on the CPU side, but with the existing CPU offload (UVA) decode stays at about 7 tok/s. Development and measurement were done on an RTX PRO 6000 (96 GiB) with a separate process holding GPU memory so that vLLM had 48 GiB…”

“### 🚀 The feature, motivation and pitch Problem Statement In the current vLLM architecture, the render API (online_renderer.py -> renderer.render_chat_async -> _process_multimodal -> mm_processor.apply) handles everything from prompt template rendering to heavy multimodal processing. The execution pipeline currently performs the following steps inside apply(): | Step | What happens | Notes | | --- | --- | --- |…”

“### Name and Version ``` $ ./build_cuda/bin/llama-server --version version: 0.4.0-dev (build 200, commit 9cf3bf256) built with GNU 13.3.0 for Linux aarch64 ``` Build: `-DGGML_CUDA=ON`, CUDA 13, sm_121 (GB10). Also reproduced on a CPU-only build of the same commit, where it does **not** occur (see scope). ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? llama-server ### Command…”

“### Name and Version version: 0.4.0-dev (build 0) + built with GNU 13.3.0 for Linux x86_64 reproduced at master @ e2d2c0d6aa9b996d5d3a3c1d5e24c8c19728bb3d（2026-09-09） real-model A/B：master @ 4d9176092d00586775af140581bb0b558ddc4389（2026-09-04） ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? libllama (core library) ### Command line ```shell ``` ### Problem description & steps to…”

“### Name and Version ~~~~ version: 0.4.0-dev (build 0, commit unknown) built with Clang 17.0.6 for Linux x86_64 ~~~~ I've used a self-compiled Llama.cpp based on `b10819`. ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? Other (Please specify in the next section) ### Command line ```shell ./bin/llama-batched-bench \ \ --threads 8 \ --cpu-mask 0xaaaa \ --cpu-strict 1 \ \…”

“## Summary `vkCreateComputePipelines` fails with `VK_ERROR_UNKNOWN` for the `mul_mat_vec_q4_k_f32_f32` compute shader on Qualcomm's proprietary Adreno Vulkan driver (not Mesa/Turnip), on a real device. I isolated this to a minimal, standalone reproducer (no llama.cpp, no model, just the raw SPIR-V + a bare `vkCreateComputePipelines` call). The interesting part: **the same GLSL source, compiled by two different…”

“### Summary In `vllm/multimodal/audio.py`, `normalize_audio` ignores the configured `channel_reduction` whenever `spec.target_channels > 1`: it silently slices the first N channels instead of reducing. The code comment already flags this as provisional. ### Details ```python # vllm/multimodal/audio.py (normalize_audio) if spec.target_channels == 1: # Reduce to mono if spec.channel_reduction ==…”

“### Name and Version ``` $ llama-server --version version: 0.3.0-dev (build 1, commit 0f3a71b) built with GNU 15.2.0 for Linux x86_64 ``` Reproduced against `ggml-org/llama.cpp` commit `0f3a71be15af836d277c9f918adfafb45732677e`. ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ``` llama-server \ -m Qwen3.8-27B-UD-Q4_K_XL.gguf \ --model-draft…”

“we need lots of patience while running these models at local if the hardware configuration doesn't match. 😀”

Comment, author, and video data come from YouTube API Services. NeedRadar generated the classification, summary, AI path, counterevidence, and unknowns; YouTube did not provide or endorse them.

“Optimizations for Qwen/Qwen3.8-Flash-Next ## Kernels [#55272：Remove torch.compile for NVIDIA implementation](https://github.com/vllm-project/vllm/pull/55272) [#54517：Fuse Qwen4Exp PLE kernels](https://github.com/vllm-project/vllm/pull/54517) [#54513：Separate prefill and decode paths for QSA indexer](https://github.com/vllm-project/vllm/pull/54513) [#54560：Add Hopper LL-GEMM tuning…”

“## Motivation RDNA4 PCIe systems currently rely on AITER/PyNCCL all-reduce paths. On 4× AMD Radeon AI PRO R9700, a vLLM-owned FlyDSL implementation can reduce launch overhead for graph-replayed tensor-parallel collectives and sustain about 41–45 GB/s bus bandwidth at large messages, while retaining PyNCCL where a custom route does not meet the performance or integration-safety gate. The design goals are: - keep…”

“### Feature request `transformers` has no low-precision attention path today: the KV cache is always FP16/BF16 by the time it reaches flash-attention, even when FA3/FA4 is installed and the kernel itself accepts `q_descale` / `k_descale` / `v_descale`. `grep -r descale src/transformers/` returns nothing, and `QuantizedCache` doesn't cover this — it's int-only, and it dequantizes to full-length BF16 before…”

“### Name and Version $ ./build_vk_valid/bin/llama-cli --version version: 0.4.0-dev (build 10852, commit 050dde50c) built with GNU 15.2.0 for Linux x86_64 ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? libllama (core library) ### Command line ```shell test-backend-ops -o "MUL_MAT(type_a=f32,type_b=f32,m=16,n=4,k=256,bs=[2,3],nr=[1,1],per=[0,3,2,1],k_v=0,o=1,src_overlap=0)" -b…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“**[ROCm] Sparse-MLA decode top-k falls back to the vLLM kernel above 64k because of an aiter 0.1.19 limitation** `vllm/v1/attention/ops/rocm_aiter_mla_sparse.py` disables the AITER top-k kernel for decode: ```python if not is_prefill: # AITER v0.1.19 decode is one-block only. This measured gfx950 # FP32/k=1024 compressed-row boundary is independent of the native # split-count boundary in sampler.cu. if ( num_rows…”

“### Name and Version ``` $ ./llama-cli --version version: 0.3.0-dev (build 10798, commit c390d0abb) built with GNU 13.3.0 for Linux x86_64 ``` Also reproduced on the `b10819` release tarball. `src/llama-sampler.cpp` is byte-identical between `b10798` and master. ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? libllama (core library), Test code ### Command line ```shell ./repro…”

“### Name and Version version: 0.4.0-dev (build 10825, commit 9e0e22059) built with GNU 13.3.0 for Linux x86_64 ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell llama-server --host 0.0.0.0 --port 11080 \ -dev CUDA0,CUDA1 \ -m unsloth-qwen3.8-27b/Qwen3.8-27B-Q5_K_M.gguf \ -ngl 999 \ -sm tensor \ -ts 0.49,0.51 \ -c 190000 \ --cache-type-k q8_0…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### The model to consider **[IFM/K2-Horizon-MoVA-36B-A4B](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B)** — a 36B MoE (~4B active) with the novel **MoVA (Mixture-of-Values) attention**, where the value projection is a router over `mova_num_experts` value experts instead of a plain `v_proj`. ### Current state vLLM's Transformers backend can't serve MoVA: the model's custom value-expert routing inside…”

“### Name and Version ``` $ llama-server --version version: 0.3.0-dev (build 10791, commit 95ef7fc16) built with GNU 13.3.0 for Linux x86_64 ``` Reproduced on this build; **also verified still present on latest master `6a1a922d2` (b10819)** — `ggml_cuda_fattn_kv_type_supported()` and the default FA kernel set in `ggml/src/ggml-cuda/fattn.cu` are unchanged. ### Operating systems - Linux (Ubuntu 24.04) ### Which…”

“### Summary On a DeepSeek-V4-Flash model (DSV4, `LLM_ARCH_DEEPSEEK4`), using a quantized KV cache (`--cache-type-k q8_0 --cache-type-v q8_0`) causes a ~2.8x decode slowdown versus f16 KV, because the sparse-fa fused kernel (added in #27970 / 8e93a97) is only reachable through the `mma_f16` path, which requires **non-quantized** K/V. ### Environment - llama.cpp: `67a17c1` (build 203) - GPU: 4x RTX A6000 (Ampere…”

“### Name and Version (llama) mrs@huginn:~/workspace/llama.cpp$ build/bin/llama-cli --version version: 0.4.0-dev (build 10819, commit 6a1a922d2) built with GNU 13.3.0 for Linux x86_64 ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? llama-quantize ### Command line ```shell python ~/workspace/llama.cpp/convert_hf_to_gguf.py Abliterated/ --outfile Abliterated/glm5.3-Flash-q8_0.gguf…”

“### 🚀 The feature, motivation and pitch ### Motivation & Problem In long-context LLM serving (32k to 128k+ tokens), dynamic KV-cache eviction/compaction algorithms (e.g., H2O, Scissorhands, StreamingLLM, or heavy-hitter pruning) are essential to prevent out-of-memory (OOM) faults. However, standard runtimes execute compaction using **out-of-place gathering** (`torch.gather` / dynamic tensor allocation): 1.…”

“### Proposal to improve performance ### Description While investigating potential instruction-level optimizations for the AWQ CUDA GEMM kernel (`gemm_forward_4bit_cuda_m16nXk32` in csrc/libtorch_stable/quantization/awq/gemm_kernels.cu), I ran an Nsight Compute (`ncu`) profile to evaluate the bottlenecks. The profile reveals that the hand-written CUDA kernel is heavily memory-bound, specifically at the L1/TEX…”

“### Name and Version ``` $ llama-server --version version: b10752 (b96806d96) ``` Container image `ghcr.io/ggml-org/llama.cpp:server-vulkan`, run in router mode (`--models-preset`), one child `llama-server` per model. ### Operating systems Linux (Ubuntu, kernel 7.0.0-31-generic) ### GGML backends Vulkan (Mesa RADV) ### Hardware AMD Ryzen AI Max+ 395 — Radeon 8060S Graphics (RADV STRIX_HALO, gfx1151), 120 GiB GTT,…”

“The official library artifacts for Qwen2.5-Coder-3B-Instruct at four low-bit quantization levels produce fluent-looking but entirely non-functional output. On a 15-task functional smoke suite (easy HumanEval+ tasks that every other Qwen2.5-Coder configuration passes at 87-100%), all four score 0/15: | tag | smoke pass | failure signature | |---|---|---| | `qwen2.5-coder:3b-instruct-q2_K` | 0/15 | runaway…”

“### Name and Version llama.cpp at commit `1863ac033` (0.4.0-dev line). Two builds were made from the **identical source tree**, differing **only in the CUDA toolkit** used to compile them: - Build A: CUDA toolkit **12.9.86** (`nvcc --version` = release 12.9, V12.9.86) - Build B: CUDA toolkit **12.8.93** (`nvcc --version` = release 12.8, V12.8.93) ### Operating systems Linux ### GGML backends CUDA ### Hardware -…”

“### Name and Version ``` build 10750 (b356fa262) with Clang 20.1.8 for Windows x86_64 ``` Official Windows CUDA 13.3 x64 release binary (`llama-server.exe`, sha256 `E1EB18C4F13AB9DB48DBED4CD9BD3DC4F94F1E97E4CF4EA53D0FEB2A0E24B26E`). ### Operating systems Windows ### GGML backends CUDA ### Hardware | | | |---|---| | GPU0 / GPU1 | 2× NVIDIA GeForce RTX 5060 Ti, 16311 MiB each, sm_120 (Blackwell) | | Driver | 591.86…”

“### Name and Version build: b10808-64a155d24 llama-cli --version version: 0.3.0-dev (build 10808, commit 64a155d24) built with GNU 15.2.0 for Linux x86_64 ### Operating systems Linux lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 26.04 LTS Release: 26.04 Codename: resolute ### Which llama.cpp modules do you know to be affected? llama-server, llama-cli ### GGML backends…”

“### Name and Version ``` $ llama-server --version version: 0.3.0-dev (build 10797, commit 235f4e8) ``` That commit string is the bug: `235f4e8` is not a llama.cpp commit. It belongs to an unrelated repository. ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? Build system (`cmake/build-info.cmake`) — affects every binary's `--version` output. ### Problem description & steps to…”

“### Your current environment ## Summary Kimi-K3 fails to start on H100 when the low-M TP8 KDA projection-overlap path introduced by #54697 is enabled. During CUDA-graph profiling, `kda_skinny_gemm.run_n()` fails in the CUTLASS DSL compiler with `NVVM_ERROR_COMPILATION` for `sm_90a`. The failure has reproduced in two independent four-node launches. The same deployment works on an older vLLM revision that predates…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“**llama.cpp version tested**: ed262173 (also reproduced on master 9a4843cf) **Hardware**: - RTX 2000 Ada (PCH Gen 1 x4, 16 GB) - RTX 3080 (Slot 2, CPU Gen 3 x16, 20 GB) - RTX 3080 (Slot 1, OCuLink Gen 3 x4, 20 GB) - 128 GB RAM, CUDA 13.2, driver 595 - Ubuntu 24.04 **Model**: - Main: unsloth/Qwen3.8-27B-GGUF UD-Q4_K_M (16.5 GB) - Drafter: incoai/Qwen3.8-27B-DFlash2-Q4_K_M - spec-type: draft-dflash,…”

“### Name and Version 10665 ### Operating systems Windows ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell “Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf --host 188.0.0.1 --port 5000 -c 262144 -ngl 99 -t 16 -b 2048 -ub 512 -np 2 -mg 2 --tensor-split 0.4,1.6 --no-mmap --temp 1.00 --top-k 20 --min-p 0.00 --repeat-penalty 1.00 -n 131072 --chat-template-kwargs…”

“### 🚀 The feature, motivation and pitch ## Motivation CUTLASS provides an SM100 [Lamport fused GEMM + AllReduce kernel](https://github.com/NVIDIA/cutlass/blob/main/examples/python/CuTeDSL/cute/blackwell/kernel/distributed/distributed_gemm_all_reduce_lamport_blackwell.py). vLLM’s existing [GEMM-RS/AR…”

“## Motivation. Deployment planners, autoscalers, and inference optimizers need a small empirical performance model for the exact model, quantization, parallelism, CUDA Graph configuration, and GPU type being served. Static benchmark databases do not cover every combination, and external load generation cannot reliably construct exact internal scheduler coordinates such as: - total uncached prefill tokens in one…”

“`ggml_vk_load_shaders` creates `mul_mat_vec_bf16_f32_f32`, `mul_mat_vec_bf16_f16_f32` and `mul_mat_vec_id_bf16_f32` unconditionally: https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-vulkan/ggml-vulkan.cpp#L5374 Every other bf16 pipeline in that function is behind a support check. `pipeline_matmul_bf16` is guarded by `device->coopmat_bf16_support` at :4870 and :4916, and the non-coopmat creates at…”

“### Name and Version I generated with llama-server Browser UI an svg image, it was incomplete first i thought that it is a bug in the llm model. But then I discovered if I copy the image the missing parts are displayed. If I download it it has the missing parts. It displays the Same image as the downloaded Svg Image. Copied Image Downloaded Image ### Operating systems Windows ### Which llama.cpp modules do you…”

“`mtmd_decode_use_non_causal()` returns true for `PROJECTOR_TYPE_GEMMA4V` and `PROJECTOR_TYPE_GEMMA4UV` for every model size, so image chunks are decoded with `llama_set_causal_attn(lctx, false)`: https://github.com/ggml-org/llama.cpp/blob/0df974d77/tools/mtmd/mtmd.cpp#L2170-L2185 The reference uses bidirectional attention over image tokens only when `text_config.use_bidirectional_attention == "vision"`, and then…”

“Failed to load model: This model is not supported yet. Try a different model. (Original error: llama.cpp does not support this GGUF's model architecture ('spark2_5'). The file is valid, but this model type cannot be run with llama-server.)”

“### Name and Version $ llama-cli --version version: 0.3.0-dev (build 10775, commit 67a17c17c) built with GNU 13.3.0 for Linux aarch64 ### Operating systems _No response_ ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell llama-server --host 0.0.0.0 --port 8080 --ssl-key-file ~/sslkeys/key.pem --ssl-cert-file ~/sslkeys/cert.pem --metrics --no-mmap --prio -1 --threads 10…”

“### Name and Version Present on current `master` in `ggml/src/ggml-cpu/arch/s390/quants.c`. Found while working on a z/OS port of llama.cpp; the file is s390x-only so this affects IBM Z builds. ### Operating systems Other? (z/OS, and any Linux on IBM Z) ### Which llama.cpp modules do you know to be affected? libggml (CPU backend, s390x) ### Problem description `ggml_vec_dot_q5_1_q8_1()` reads `v_acc` in its own…”

“### Name and Version [b10766](https://github.com/ggml-org/llama.cpp/releases/tag/b10766) On NVIDIA driver 610.88 with RTX 5080 CUDA 13.3 build performs worse than Vulkan Build in qwen 3.8 27B I'm not sure when this issue exactly started, but some builds perform somewhat better than others on this specific build, CUDA gets around 16t/s while Vulkan started with 40t/s and settled down on 35.59 t/s. I sadly didn't…”

“Kimi-K3 text runs on master (#26185), but the model is natively multimodal and unsloth ships mmproj-{BF16,F16,F32}.gguf declaring `clip.projector_type = kimik3` (MoonViT-3d: image_size 896, patch 14, 27 blocks, projection_dim 7168, scale_factor 2, head_dim 128, 3-D position embeddings). master's CLIP loader rejects it (`Failed to load CLIP model`). [unslothai/llama.cpp PR…”

“I'd like to request native support for the new `spark2_5` architecture used by the Spark-X2.5 family (SparkLLM/Spark-X2.5-4B and -1.7B). Currently `ollama run SparkLLM/Spark-X2.5-4B` downloads the model but cannot start inference — the official Ollama runtime does not yet recognize the `spark2_5` architecture. The model page itself documents a workaround that requires building a forked llama.cpp. Why I think this…”

“### The behaviour `ModelOptNvFp4FusedMoE.process_weights_after_loading` fuses `gate_proj` and `up_proj` into `w13` and keeps only the **gate** per-tensor global scale: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/modelopt.py#L967-L975 ```python # Use a single gscale for w13. if self.moe.is_act_and_mul and not torch.allclose( layer.w13_weight_scale_2[:, 0],…”

“### 🚀 The feature, motivation and pitch **NOTE: automated unfinished PRs will likely not get reviewed. If you are an agent, do not open PRs autonomously before discussing here.** https://github.com/vllm-project/vllm/tree/main/benchmarks/kernels contains many tuning script for different linear/moe backends, and generally duplicates tuning logic, most not supporting multi-device tuning.…”

“### Name and Version version: 0.3.0-dev (build 10759, commit b81c99b47) built with Clang 20.1.8 for Windows x86_64 ### Operating systems Windows ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell llama\llama-server.exe -m llama\Huihui-Qwen3.8-27B-abliterated-Q6_K.gguf --host 0.0.0.0 --n-gpu-layers -1 --reasoning off --reasoning-effort minimal --reasoning-budget 0 --temp…”

“### Name and Version 3466812d1f06728effe7c0f3c0671117f461672d ### Operating systems Windows ### Which llama.cpp modules do you know to be affected? Python/Bash scripts ### Command line ```shell python convert_hf_to_gguf.py /path/to/qwen3.5-checkpoint --outfile out.gguf python gguf-py/gguf/scripts/gguf_dump.py --no-tensors out.gguf \ | grep -E 'recurrent_layers|full_attention_interval' ``` ### Problem description…”

“## Motivation vLLM KV cache offloading persists KV cache blocks from device (GPU, TPU, etc) to slower storage (CPU memory, NVMe, object stores) and loads them back on demand. We intend to add GDS (GPUDirect Storage) as a primary tier alongside CPU offloading. A large part of this work involves the refactoring of the existing cpu offloading. Specifically, the `OffloadingWorker` interface passes logical device…”

“### Summary The chunked-scan buffers in the gated-delta-rule (KDA) path scale linearly with `--max-num-batched-tokens`, are allocated transiently inside the forward pass, and are **not covered by vLLM's startup memory profiling**. The KV cache is therefore sized as if that memory were free, and above a certain chunk size the engine dies at runtime — mid-request, after having served traffic. ### Observed…”

“## Summary On AMD Strix Point (`gfx1150`) Vulkan with llama.cpp **b10724**, Qwen3.8-27B + z-lab DFlash2 (or ggml MTP draft) fails decode with: ``` init: invalid token[1] = 248320 decode: failed to initialize batch ``` **248320 equals `tokenizer.ggml.tokens` array length** — valid token ids are `0..248319`. This looks like a classic off-by-one in draft sampling / logits indexing. ## Environment - GPU: AMD Strix…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### 🚀 The feature, motivation and pitch vLLM main already contains the optional direct **B12X** NVFP4 linear *and* MoE backend merged in #52018. Its support predicate explicitly covers **SM120 and SM121**, NVFP4 W4A16, supported shapes, tensor-parallel constraints and package availability. However, the automatic NVFP4 MoE selector in `vllm/model_executor/layers/fused_moe/oracle/nvfp4.py` never considers…”

“## Description Serving a DeepSeek-V4-Flash GGUF on GB10 (sm_121) produces **deterministic, character-level output corruption** when the model reproduces text from its own context. It is 100% reproducible at temperature 0, survives two llama.cpp builds, two different quantizations (MXFP4 and Q2_K_S), and is absent when the same model runs hosted. Output is fluent and plausible — the defect only becomes visible…”

“### Name and Version ./llama-cli --version version: (2d8d612e4) built with ROCm 7.14 for x86_64-linux-gnu ### Operating systems Linux ### GGML backends HIP ### Hardware AMD Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S iGPU, gfx1151 / RDNA3.5, 128 GB unified memory. Two nodes, the second attached via `rpc-server`. ### Models A 35B-A3B MoE at Q8_0. Any MoE model should do — the fault is in the `MUL_MAT_ID` path.…”

“When a backend fails during llama-bench, the only output at default verbosity is e.g.: ``` test_prompt: failed to decode prompt batch, res = -3 llama_bench: error: failed to run prompt ``` The backend's actual error - in our case: ``` ggml_metal_synchronize: error: command buffer 0 failed with status 5 error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory) ``` is filtered out unless…”

“### Name and Version version: 0.3.0-dev (build 10678, commit 6fe749801) built with GNU 11.4.0 for Linux x86_64 ### Operating systems Linux, Windows ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell "$DIR/llama/vulkan/llama-b10678/llama-server" \ -m "$MODEL_PATH" \ Logs ```console ```”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“**Problem** ROCm/HIP users currently have no official way to get a prebuilt `llama-server`: - GitHub Releases provide prebuilt binaries for CUDA / Vulkan / Intel / MUSA only — no Linux ROCm asset. - `ghcr.io/ggml-org/llama.cpp` tags (`full-*`, `server-*`, `light-*`) cover cuda/vulkan/intel/musa — no `rocm` tag. - Building from source requires a full ROCm SDK (hipcc, hipblaslt, etc.). On **immutable distros**…”

“**Authors:** [@new-TonyWang](https://github.com/new-TonyWang), [@aoshen524](https://github.com/aoshen524) **Organization:** inferact.ai **Based on:** Source-code audit of 88 PWAL implementations, 181 `replace_parameter` call sites, and 13 online quantization implementations. --- ## Motivation vLLM's current weight hot-reload for RL training uses **layerwise reload**: move all params to meta device →…”

“## Summary On integrated GPUs (tested: AMD gfx1151 / Strix Halo APU, ROCm/HIP backend), `llama-server` can leak one request's conversation content into a later, unrelated request's responses. The KV cache gets corrupted during chunked prefill: K/V rows are scattered to wrong cells, and stale K/V from a previous request remains resident under cells the new request's attention mask marks visible. We hit this in…”

“## Summary Following analysis of 1Cat-vLLM's MTP-based speculative decoding for Qwen3.6-35B-A3B, this issue proposes adding MTP (Multi-Token Prediction) speculative decoding support f...[truncated]”

“### Name and Version version: 0.3.0-dev (build 252, commit 6b39dd5d5) built with GNU 16.1.1 for Linux x86_64 ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? libllama (core library), llama-server ### Command line ```shell llama-server --alias qwen38 -m Qwen3.8-27B-UD-Q4_K_XL.gguf --mmproj mmproj-F16.gguf \ -ngl 999 -fa on -ub 2048 -b 2048 -c 204800 -np 1 -cram 32768 \…”

“## Summary This issue documents a systematic investigation into Volta-class GPU (SM70/SM72) flash attention performance in llama.cpp, motivated by 1Cat-vLLM's FLASH_ATTN_V100 backend which achieves significant improvements on V100 via TurboMind SM70 WMMA kernels and split-D attention design. Benchmarking was performed on Jetson AGX Xavier (SM72, aarch64, CUDA 11.4) with Qwen3.6-35B-A3B MoE D=256 GGUF models. ##…”

“### Name and Version $ llama-cli --version version: 0.3.0-dev (build 10701, commit cc231cb0d) built with GNU 16.2.1 for Linux aarch64 ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? libllama (core library) ### Command line ```shell env HK_SYSMEM=$((20 * 1 Full output (click to open) ```log USER@fedora:~$ env HK_SYSMEM=$((20 * 1 = 1024 MiB of free device memory, no changes…”

“## TL;DR The QSA path of `Qwen3.8-Flash-Next` refuses any KV dtype other than bf16. Most of the machinery for fp8 is already in vLLM — only the **read** side of the QSA Triton kernels was missing. Wiring it up roughly doubles the KV pool. Measured on **one** GB10 (DGX Spark, sm_121): | | bf16 | fp8_e4m3 | | |---|---|---|---| | KV pool | 780 638 tokens | **1 399 848** | ×1.79 | | Max concurrency @ 262144 tok/req |…”

“### Your current environment Field deployment: - 4× NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs (97,887 MiB each) - NVIDIA driver 595.58.03, CUDA 13.2 - TP4 / DCP4, V2 model runner, FlashInfer sampler enabled - Full GLM-5.3 checkpoint on a downstream vLLM/B12X image The fatal field reproduction was on a downstream build, not stock vLLM `main`. I am filing this upstream because the same dispatch invariant is…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### Name and Version c841aeeb8 (release b10687) ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell /home/llama.cpp/build/bin/llama-server -m /home/gguf/DeepSeek-V4-Flash-3.86bpw-Q3_K.gguf --chat-template-file /home/jinja/dsv4_0731.jinja --reasoning-preserve -lv 4 -lm mmap --spec-type none --cors-origins "*" --ui-mcp-proxy -cms 6144 --jinja…”

“### Summary On ROCm with GLM-5.3-Flash, vLLM logs at startup: ``` Fused multi-step draft decode is not supported by attention backend(s) DEEPSEEK_V32_INDEXER, KPOOL_TAIL, ROCM_AITER_MLA_SPARSE; falling back to rebuilding attention metadata between draft steps. ``` Because the metadata is rebuilt between every draft step, the per-step cost of MTP speculation is high enough that deeper speculation stops paying for…”

“### Name and Version version: 0.3.0-dev (build 200, commit 17252c7) built with GNU 13.3.0 for Linux x86_64 ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? libllama (core library), llama-quantize ### Command line ```shell llama-quantize --prune-layers 3 LFM2.5-230M-BF16.gguf pruned.gguf Q8_0 llama-cli -m pruned.gguf -ngl 0 -n 8 -p hi ``` ### Problem description & steps to…”

“mlxrunner: `num_ctx` from Modelfile is not enforced, allowing prompts up to architecture max — causes Metal watchdog panic on long prefill ### Environment - Ollama version: 0.33.2 (also reproduced on the version prior to upgrading) - OS: macOS (Darwin 27.0.0), Apple M2 Max, 64 GB unified memory - Model: custom import, `qwen3_5` architecture, 27.8B params, `nvfp4` quantization, Modelfile sets `num_ctx 65536`,…”

“## TL;DR - One model served with `--data-parallel-size 3` across 3 GPUs; a second, much smaller vLLM server shares GPU 0 with it. - The budget vLLM sizes from is a fraction of the **whole** card and **one value for every rank**, so the shared card caps the group: GPU 1 and 2 hold ~17 GiB each that nothing can use, while GPU 0 is left at ~0.15 GiB free and dies with `torch.OutOfMemoryError` under load. - CUDA…”

“> **Disclaimer:** I used AI to assist with the investigation, but it did not generate the full issue. I manually reviewed this report before opening it. ### Environment - llama.cpp: `0.3.0`, build `40`, commit `c1d0e7a004015f23bc0233470b747b596f29b264` - Build: Linux `x86_64`, GCC `16.2.1`, Vulkan backend (`AMD Radeon AI PRO R9700`, RADV) - Model: `Qwen3.8-27B-UD-Q6_K_M.gguf` (`Qwen3.8`, 27B, Q6_K) - Client…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“`llama-tts` doesn't set a default `n_ctx`, so for qwen3-tts the full 32768-cell KV cache is allocated — that's **3584 MiB** with 28 layers / f16. On CUDA, together with weights and compute buffers, total VRAM usage is **~7.1 GB**: ``` load_tensors: CUDA0 model buffer size = 1434.85 MiB (talker, Q8_0) llama_kv_cache: CUDA0 KV buffer size = 3584.00 MiB (32768 cells, 28 layers, f16) sched_reserve: CUDA0 compute…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“## Description Running Qwen3.8-27B Q6_K through llama.cpp on an RTX 5090 can cause the entire graphical display to go black after several normal inference requests. Recovery requires a reboot. The failure reproduces through Codex and previously also through DeepSeek Harness, both using the same local llama.cpp server. This is not a normal server-only crash: the NVIDIA graphical stack loses the GPU and the desktop…”

“### Summary On macOS / Apple Silicon, `llama-server`'s **CPU-side malloc heap** grows with the number of requests served and is not released when the workload stops. It is not the KV cache, and it is not visible in `/api/ps`. Under a sustained batch workload the process footprint climbed **8.25 GB → 13.74 GB in 120 seconds** and drove **~23 GB/hour of swap writes** on a 24 GB machine, while `/api/ps` reported the…”

“### Name and Version Both machines: version: 0.3.0-dev (build 10675, commit 90c26fcd4) built with Clang 20.1.8 for Windows x86_64 NVIDIA Game Ready Driver 610.88, CUDA 13.3 ### Operating systems Windows ### GGML backends CUDA ### Hardware Client: Ryzen 7 9700X, 5070ti, 5060ti (16gb) Host: Ryzen 7 8845HS, 4060 laptop ### Models Qwen3.5-9B-Q4_K_M…”

“### `rms_norm_f32` exceeds the CUDA `gridDim.y` limit at n_ctx 262,144 Running `qwen4exp` (Qwen3.8-Flash-Next, added in #27742) at the model's full native context aborts on CUDA. **This is not OOM** — VRAM sits at 68% when it happens. ``` CUDA error: invalid argument current device: 0, in function ggml_cuda_kernel_launch at ggml/src/ggml-cuda/common.cuh:1674 cudaGetLastError() ggml/src/ggml-cuda/ggml-cuda.cu:107:…”

“## Summary Several k-quant variants of a `laguna`-arch model (poolside Laguna-S-2.1 based, 48 layers, 256-expert MoE, 3072 hidden) produce NaN logits when evaluated with the CUDA backend, while the **same GGUF files evaluate cleanly on CPU** (`-dev none`). The failure is content-dependent (plain English wikitext triggers it; a C++ code corpus often does not) and rounding-dependent (requantizing the same recipe…”

“### 🚀 The feature, motivation and pitch There are already several caching mechanisms in vLLM, including multimodal processing/encoder caching and prefix caching. It would be useful to expose an explicit, application-level cache_id (or similar handle) for reusable cached state. The idea is simple: an application sends a large context or image once, vLLM performs the expensive processing and returns a cache_id.…”

“### What is the issue? I am using an nvidia dgx-spark machine, where I intend to run some models from ollama using the ollama python library. While running in terminal, multi-model concurrent execution in two different terminal sessions is running perfectly. But when I try to run (only inferencing) the two different ollama models in two different python codes using the ollama library, the amount of time each…”

“## Summary `is_shared_expert_quant_fse_compatible` decides whether AITER fused shared experts (FSE) may run by searching the **entire** Quark `exclude` list for the substring `shared_expert`: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/utils/config_utils.py#L106-L111 ```python is_compatible = not any( "shared_expert" in str(entry) for entry in…”

“### Name and Version $ ./bin/llama-cli --version version: 0.3.0-dev (build 10632, commit 11cd98842) built with GNU 13.3.0 for Linux x86_64 ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? Other (Please specify in the next section) ### Command line ```shell ./bin/llama-perplexity -m ~/projects/ds4/gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf -f…”

“### Name and Version llama-server --version version: 0.3.0-dev (build 10665, commit ca3d5a3e10d53f7ea672cb9b6178faca3e2807bc) built with GNU 15.3.0 for Linux x86_64 ### Operating systems Linux ### GGML backends CUDA ### Hardware Ryzen 9700x + rtx 5090 ### Models https://huggingface.co/incoai/Qwen3.8-27B-DFlash2-GGUF/tree/main?show_file_info=Qwen3.8-27B-DFlash2-Q4_K_M.gguf ### Problem description & steps to…”

“### Name and Version **Version** ```bash $ llama-server --version version: 1 (15586e2) built with GNU 13.3.0 for Linux x86_64 ``` ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell $ llama-server -hf evil/repo $ llama-server -hf evil/repo --hf-file model.gguf $ llama-server -m local.gguf -hf evil/preset ``` ### Problem description & steps to…”

“### Name and Version [b10664](https://github.com/ggml-org/llama.cpp/releases/tag/b10664), [Windows x64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/b10664/llama-b10664-bin-win-cuda-13.3-x64.zip) - [CUDA 13.3 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/b10664/cudart-llama-bin-win-cuda-13.3-x64.zip) ### Operating systems Windows ### GGML backends CUDA ### Hardware Ryzen 5950x…”

“## Environment - llama.cpp @ `07822bdd` (the DSpark branch from PR #27383, with the runtime-wiring port of that commit; locally also carries the two small port-fix commits for conv-state slot write / layer-input wiring) - Build: Vulkan, AMD gfx1151 (NUC), `llama-server` - Target model: Qwen3.8-27B (UD-Q4_K_XL GGUF, has MTP layers — `--spec-type draft-mtp,ngram-mod` works on it) - Draft model: Qwen3.6-27B-DFlash…”

“### Your current environment Environment ``` GPU : NVIDIA GB10 (DGX Spark), compute capability 12.1 (sm_121), 128 GB unified vLLM : 0.1.dev20073+g8e685d198 (Qwen3.8-Flash-Next preview build, PR #53896) torch : 2.13.0+cu130 CUDA 13.0 aarch64 / Ubuntu 24.04 model : lovedheart/Qwen3.8-Flash-Next-NVFP4-FP8 (ModelOpt MIXED_PRECISION, FP8_PB_WO blockwise-FP8 attention + GDN projections, NVFP4 experts) ``` ### 🐛…”

“### Your current environment Relevant collect_env.py output ```text OS: Ubuntu 24.04.3 LTS (x86_64) Python: 3.12.3 vLLM: 0.1.dev20073+g8e685d198 (git sha: 8e685d198) Image: vllm/vllm-openai:qwen38-flash-next Image digest: sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8 PyTorch: 2.13.0+cu130 CUDA runtime: 13.0.88 NVIDIA driver: 595.84 GPUs: 4x NVIDIA RTX PRO 6000 Blackwell Workstation…”

“### Name and Version version: 0.3.0-dev (build 10661, commit 32176338a) built with GNU 13.3.0 for Linux aarch64 ### Operating systems Other? (Please let us know in description) ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell llama-server \ --host 0.0.0.0 --port 8848 --threads 16 --warmup --no-mmap \ -ngl all \ --models-max 2 \ --cache-prompt \ --ui-config-file…”

“### Name and Version Current master: b10660 6c84c7d5d model: add Qwen3.8-Flash-Next (qwen4exp) (#27742) This includes the merged DFlash2 implementation from #27342. ### Operating systems Linux ### GGML backends HIP ### Hardware Ryzen 3600 Asus prime x570 pro (8x/8x) 2 x rx 7900 xtx ### Models ### Models Target: ```text Qwen3.8-27B-Q8_0.gguf ``` DFlash2 drafter: ```text Qwen3.8-27B-DFlash2-Q8_0.gguf ``` ###…”

“### Summary On RDNA4 (gfx1201, Radeon AI PRO R9700), every quantized KV cache type decodes **slower** than `f16`, and the deficit grows with the amount of unpacking the type requires. This is the opposite of the expected memory-bandwidth tradeoff. ### Measurements `llama-bench`, single GPU, ROCm backend, `-ngl 99 -p 0 -n 64 -r 1`. Model: `Qwen3.6-27B-UD-Q5_K_XL.gguf` (26.9B, 18.65 GiB, head dim 256). | KV type |…”

“### Summary Serving `Qwen3.8-Flash-Next` with `VLLM_PLE_CPU_OFFLOAD=1` at `tensor_parallel_size=1` hangs permanently during startup. Engine init completes fully — weights load, KV cache is allocated, FlashInfer autotune finishes — and then the process goes silent before `init engine took Xs`. The API server never starts. No exception, no CUDA OOM, no kernel OOM-kill. Reproduced 3/3 times, waiting 25, 60 and 60…”

“### Name and Version Reproduced identically on three builds (CPU-only, `cmake -DCMAKE_BUILD_TYPE=Release`, Linux x86_64): - `f280b26983` (current at time of testing) - `4d19b28` (master, includes #27679) - `60addddf3c` (about a week older) ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell ./build/bin/llama-server -m…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“### Name and Version llama-b10639-bin-win-rocm-7.14-x64 build: 5e6a37cb1 (10639) built with Clang 20.1.8 for Windows x86_64 ### Operating systems Windows ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell llama-server.exe -m Qwen3.8-27B-UD-Q4_K_XL.gguf -ctk q8_0 -ctv q5_0 or llama-bench.exe -m Qwen3.8-27B-UD-Q4_K_XL.gguf -ctk q8_0 -ctv q5_0 ``` ### Problem description &…”

“**Summary.** `Qwen/Qwen3.8-Flash-Next-FP8` ships `quantization_config.modules_to_not_convert` with multimodal-wrapper paths (`model.language_model.layers...`). When the checkpoint is loaded through `AutoModelForCausalLM` the text stack is named `model.layers...`, so none of the exclusion patterns match. The FineGrainedFP8 quantizer then converts modules the vendor intended to keep in bf16 (e.g. the PLE…”

“### 🚀 The feature, motivation and pitch ## Motivation Qwen3.8-Flash-Next introduces a very large N-gram / PLE embedding table. This part of the model is particularly suitable for offloading because it is fundamentally a lookup workload rather than a large GEMM-heavy compute workload. A useful deployment topology would be: ```text Primary GPU - Main model - Attention / GDN - MoE computation - KV cache - CUDA…”

“### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). - [x] I reviewed the…”

“_This is a proposal for contribution. This feature request comes with the corresponding proposed patches_ The target scope is: - Apple Silicon hardware - The MLX Metal backend - Qwen3.8 27B dense model with NVFP4 quantization The objective is to accelerate prefill and decode speed via FP16 execution paths on Apple Silicon. I know the target is narrow but I believe running Qwen on Macs still reaches a wide…”

“## Environment - vLLM 0.27.1, `vllm/vllm-openai` image digest `sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967` (created 2026-08-11) - Model: `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` (`NemotronHForCausalLM`), served weight-only FP4 via the Marlin fallback on SM89 - `--tensor-parallel-size 2` on 2x RTX 4090, `--enable-lora --max-lora-rank 16 --lora-modules` (one adapter),…”

“### Your current environment Environment ```text vLLM: 0.27.1 (pip, isolated venv) torch: 2.13.0+cu130, CUDA 13.0 GPU: NVIDIA GeForce RTX 3090, 24 GiB (23.56 GiB usable), sm_86 Driver: 610.43.02 OS: Debian 13, kernel 6.12.101+deb13-amd64 Python: 3.13.5 Model: RedHatAI/Qwen3.8-27B-INT4 (compressed-tensors, ships model_mtp.safetensors) model_type qwen3_5, vocab_size 248320, hidden_size 5120, tie_word_embeddings…”

“# Vulkan: ~78% decode throughput cliff at 131072 context window — default ~1 GiB suballocation fragmentation; fixed via GGML_VK_SUBALLOCATION_BLOCK_SIZE=4 GiB ## Environment | | | |---|---| | GPU | AMD Radeon RX 7900 XTX (24 GB, RDNA3, gfx1100) | | OS / Driver | Windows 11, AMD proprietary driver | | llama.cpp | 0.2.0-dev (build **10603**, commit c060ca974), Vulkan backend (ggml-vulkan.dll) | | Model |…”

“## DFlash2 on Ampere: online W8/FP8 of the draft + `hf_overrides` callable **Hardware:** 2× RTX 3090 (SM86), vLLM 0.27.1 **Target:** Qwen3.8-27B AWQ (`Qwen3_5ForConditionalGeneration`) **Draft:** `Qwen3.8-27B-DFlash2` BF16, `method=dflash`, n=7 **Goal:** 4 concurrent 128k sessions. KV fp8_e4m3 pool is 428 331 tokens = **3.35× at 128k**. Need ~2.1 GiB/GPU more. The BF16 draft is 3.58 GiB; INT8 of the 34 large…”

“## Environment - llama.cpp: upstream master as of 2026-08-26 (build **b10643**), self-built Release / CUDA sm_86. The only local patches are unrelated GGML kernel optimizations; `common/chat.cpp` and the server chat pipeline are vanilla. - Model: Qwen3.8-27B-Q4_K_M GGUF with embedded official jinja template (+mmproj) - Server flags: `-c 120000 -fa on -ctk q4_0 -ctv q4_0 --split-mode layer -ts 1,1,1 --spec-type…”

“### Name and Version build : b10549-b2e5e9b28 windows vulkan 64bit ### Operating systems Windows ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ```shell Server (no speedup from MTP): llama-server.exe ^ -m "Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced.Q4_0.gguf" ^ --model-draft "mtp-gemma-4-26B-A4B-it.gguf" ^ --spec-type draft-mtp ^ --spec-draft-n-max 2 ^ -np 1 ^ -ngl 99 ^…”

“### Name and Version version: 0.3.0-dev (build 1, commit 5d5cb4c) built with Clang 23.0.0 for Linux x86_64 ### Operating systems Linux ### GGML backends HIP ### Hardware AMD Ryzen AI 9 HX 370 + Radeon 890M, i.e. gfx1150. ### Models Repo: `bartowski/huihui-ai_Qwen3-Coder-Next-abliterated-GGUF`. File: `huihui-ai_Qwen3-Coder-Next-abliterated-Q6_K/huihui-ai_Qwen3-Coder-Next-abliterated-Q6_K-00001-of-00002.gguf`. ###…”

“### Name and Version ## llama.cpp Version Information ``` Git commit: 47c786924 (HEAD, tag: b9999) kleidiai : add SME2 f32 kernel (#24414) Build version: version: 9999 (47c786924) built with GNU 11.4.0 for Linux aarch64 ``` ### Operating systems Linux ### GGML backends CUDA ### Hardware ## Hardware Information ``` CPU: CPU(s): 6 On-line CPU(s) list: 0-5 Model name: Cortex-A78AE Thread(s) per core: 1 NUMA node0…”

“## Proposal to improve performance ### Summary Model Runner V2 pipeline parallelism sends sampled results from the last PP rank back to the non-last ranks. The CUDA path uses a sibling process group and a side stream, as introduced by [#42187](https://github.com/vllm-project/vllm/pull/42187). However, each non-last rank posts its receive in local feedback step `T`, while that result is not consumed until `T +…”

“### Motivation. **TL;DR** 1. We propose to use Helion by default for some CustomOps. 2. We show that Helion kernels have superior performance compared to the CUDA kernels in vLLM. Across broad shape sweeps, the three Helion kernels we benchmarked achieve 1.382–1.785x geomean speedups on H100, 1.556–1.804x on B200, and 1.022–1.427x on MI350X. On Qwen3-8B-FP8, this translates to 2.59%–3.49% higher end-to-end…”

“It would be great if the backend can support [llama.cpp](https://github.com/ggml-org/llama.cpp) inference engines which would enable rapid expansion to new quantized model. For intance, unsloth released a new dynamic quantization method that works great and still gives you a usable model for a fraction of the resources the full one needs. Read [here](https://unsloth.ai/docs/basics/dynamic-3.0-ggufs) Which also…”

“Hello, I am trying to understand how to get a clean installation without unneccessary packages into my virtualenv. I have a setup with 2 servers: A is with AMD CPU to run docling B is an Nvidia Jetson ORIN running llama.cpp connected via LAN. During the installation of docling, pip installs a lot of dependencies including some packages that will never be able to run onto server A: pip install docling ``` ...…”