This is the canonical model support document. Other files (README, site, wiki) reference this page. Update this page first when adding new model support.

Supported Models & Performance

The 1bit-monster engine auto-detects 552 architecture tokens from GGUF/1BP/safetensors headers — no config files needed. We reverse-engineered AMD's NPU stack, extracted 37 FLM models with 209 pre-compiled XDNA 2 xclbins, and created our own 1BP ternary format to make AMD's open-source models run at maximum throughput on NPU + GPU.

HF-Native coverage (measured 2026-08-14)

Real-checkpoint census of the HuggingFace hub (/api/models?pipeline_tag=text-generation&config=true). FULL 100% enumeration 2026-08-15 (Testing/census_full_summary.json, every page walked): 399,220 text-generation checkpoints, of which 322,010 declare an architectures field (77,210 have none — unmappable). After excluding ~4,910 out-of-scope encoder-decoders (T5/BART/seq2seq — no decode loop), 317,310 arch-bearing text-gen checkpoints remain in the denominator, and 317,310 / 317,310 (100.00%) map to an engine token (the rcpp_arch_from_string registry — number regenerated by Testing/census_coverage.py, which measures the ACTUAL committed mapping function, not a script-side copy). The two-step dispatch — class name → token, then UNKNOWN class → HF model_type → token (see modular-arch-registry-extraction.md) — plus the sentinel-drift fix (RCPP_ARCH_UNKNOWN is now read from the live header at load, so an unmapped class can no longer be silently counted as mapped) closed the last gap. The 32 manifest families (29 validated, 1 documented limitation, 2 mapped-unvalidated — see Testing/models_manifest.json) cover the bulk of it; the gap to the validated count is the loadable-but-unvalidated tokens (qwen35 ~4,300, deepseek v2/v3 ~780, deepseek v4 ~513, VLM/SSM tails). The validated families (torch/numpy/llama.cpp-exact generation gates; Testing/bringup_runner.sh is now manifest-driven with per-family gate commands):

| Family | HF checkpoints covered | | Family | HF checkpoints covered | |--------|------:|---|---|--------|------:| | llama (incl. mistral/cohere/…) | 116,642 | | qwen2 | 26,291 | | gpt2 | 24,914 | | gemma (incl. granite) | 11,008 | | qwen3 | 8,427 | | phi | 4,041 | | gptneox | 5,652 | | step1 (Step1MoE-class) | 2,882 | | opt | 1,877 | | gptneo | 1,355 | | gptj | 609 | | falcon | 904 | | olmo | 744 | | gptoss | 407 | | codegen | 348 | | exaone | 156 | | internlm2 | 115 | | minicpm | 76 | | bloom | 1,086 | | deepseek v2/v3 | ~780 (V2 183 + V32 89 + V3 508) | | TOTAL | ~207,500 (validated families) |

Remaining uncovered causal-LM classes: Qwen2VL · Mamba · Kimi-K3 · Whisper (bespoke SSM/STT/VLM backends — separate workstreams; encoder-decoders T5/MT5 out of scope for the decode-loop engine). Bloom (1,086) validated 2026-08-15.

The 500+ agnostic engine

One binary, no per-model config files. The arch registry (rcpp_arch_from_string) maps 1,774 HF architectures strings → 552 engine tokens — every checkpoint whose arch string is in the table loads through the same discovery → arch-map → router → decode path. The tokens group onto the validated families (see the manifest tiers); the VLM/SSM tokens (qwen2vl, mamba, kimi, whisper, ...) route to the bespoke backends (GPU/VLM workstreams, separate validation).

The 500+ models floor is long since passed: the FULL census (2026-08-15, every page walked) counts 399,220 text-gen checkpoints, 322,010 with an architectures field (77,210 have none). After excluding ~4,910 out-of-scope encoder-decoders, 317,310 / 317,310 arch-bearing text-gen checkpoints (100.00%) map to an engine token (class-name + model_type two-step dispatch); the 32 manifest families (29 validated) gate the bulk of it. Per-class counts are now exact, not sampled: deepseek v2/v3 ~780 (V4 ~513), qwen3.5 ~4,300 (loadable via the QWEN35 token, unvalidated), kimi ~84, mamba ~220, whisper ~20. NO-MORE-SECRETS caveat: "the engine loads them" means the arch string maps to a token whose layout is validated — not that each of the ~317k checkpoints was individually run. The full per-arch table is in Testing/census_full_summary.json.

DONE 2026-08-14 — gpt_neox (5,652 — the biggest miss): validated on EleutherAI/pythia-70m. Three real bugs found: (1) fused query_key_value is HEAD-INTERLEAVED [q_h,k_h,v_h] per head (llama.cpp conversion reshapes (n_head,3,hd,embed) — raw safetensors is NOT [q|k|v]); (2) rotary_pct 0.25 → rot_dim = head_dim/4 (new cfg.rope_dim; gguf rope.dimension_count=16 is the tell) — the engine rotated the full head_dim; (3) untied LM head is embed_out.weight (loader fell back to the tied embedding → wrong logits). Also: reader rotary_emb_base→rope_theta fallback; parallel attn+FFN (use_parallel_residual) + nn.LayerNorm weight+bias + biases-everywhere + non-gated erf-gelu — all via existing falcon/gpt2 paths. Result: engine ≡ HF-semantics numpy top-8 EXACT + top-1 logits == torch (253:1064.9 vs torch 1078.4 — pythia-70m has near-flat logits, tail shuffles on near-ties; llama.cpp's neox-rope convention disagrees). Coverage now ~199k / 220k (90%).

Backend Availability Legend

🟢 = supported and validated · 🟡 = functional, perf data pending · 🔴 = not yet · 🔬 = experimental

Model Families Summary

Specialized architectures (covered in detail below): BitNet/Bonsai (ternary-native), Qwen2-VL/Qwen3-VL (vision-language), Whisper (speech-to-text), Embedding-Gemma-300M (text embedding).

Detailed Per-Family Sections

Dense Transformers

1. Qwen2 / Qwen2.5

Standard dense transformers. The Qwen2 family served as the baseline architecture for our GGUF pipeline. Qwen2.5-3B and Qwen2.5-VL-3B are available on NPU via FLM xclbins. Full GGUF Q4_K quantized variants run through GPU HIP.

• NPU: Qwen2.5-3B-Instruct, Qwen2.5-VL-3B-Instruct (FLM, build stanzas in build_xclbins.sh)
• GPU HIP: GGUF Q4_K through ROCm HIP — validated
• GPU Vulkan: GGUF — functional, perf data pending
• CPU: GGUF — functional, perf data pending

2. Qwen3 / Qwen3.5

Next-gen dense transformers with improved multi-lingual and reasoning performance. Extensive NPU coverage with 11 variants including instruct and thinking-tuned checkpoints.

• Qwen3 on NPU: 0.6B, 1.7B, 4B, 8B, 4B-Instruct-2507, 4B-Thinking-2507 (all build stanzas in build_xclbins.sh — run ./build_xclbins.sh qwen3_0_6b to compile any)
• Qwen3.5 on NPU: 0.8B, 2B, 4B, 9B (GateDeltaNet variants, build stanzas in build_xclbins.sh)
• Qwen3-VL: 4B-Instruct on NPU (Peano build stanza, 6 xclbins) — vision-language
• GPU HIP: GGUF through ROCm HIP — validated (kernel bench: 431 tok/s Q1, 543 tok/s TQ2)
• GPU Vulkan: Qwen3-0.6B at 259 tok/s decode, 333 tok/s prefill — ✅ validated (ZINC bench)

3. Llama 3.1 / 3.2

Meta's dense transformers. Broad backend coverage — GGUF runs on all GPU backends and CPU. NPU support for Llama3.2 1B/3B and Llama3.1 8B.

• NPU: Llama3.2 1B, Llama3.2 3B, Llama3.1 8B (build stanzas in build_xclbins.sh — run ./build_xclbins.sh llama)
• GPU HIP: GGUF — validated
• GPU Vulkan: GGUF — validated
• CPU: GGUF — validated

4. Mistral / Pixtral

Mistral dense transformers and Pixtral vision-language models. GGUF through GPU HIP.

• NPU: Mistral-7B Peano build stanza (build_mistral_7b) — GEMM dims match Llama-3.1-8B. SWA attention MLIR generator (generators/n1_core_swa.py) supports sliding-window attention. Run ./build_xclbins.sh mistral_7b to compile xclbins.
• GPU HIP: GGUF — validated
• GPU Vulkan: GGUF — functional, perf data pending
• CPU: GGUF — functional, perf data pending

5. Gemma 3 / 4

Google's dense transformers. Gemma4 E2B/E4B with build stanzas in build_xclbins.sh (10 xclbins each).

• NPU: Gemma3 1B, Gemma3 4B, Gemma4 E2B-Instruct, Gemma4 E4B-Instruct, TranslateGemma 4B, MedGemma 4B, MedGemma1.5 4B (all build stanzas in build_xclbins.sh — run ./build_xclbins.sh gemma4_e2b)
• GPU HIP: GGUF — validated
• GPU Vulkan: GGUF — functional, perf data pending
• CPU: GGUF — functional, perf data pending

6. Phi4-Mini

Microsoft's 4B dense transformer. NPU-only at present.

• NPU: Phi4-Mini-Instruct (Peano build stanza, 4 xclbins)
• GPU HIP: ❌ not yet
• GPU Vulkan: ❌ not yet
• CPU: ✅ universal GGUF backend

7. Laguna

Poolside Laguna dense transformers. GGUF through GPU HIP.

• NPU: Laguna — not yet (sigmoid-routed MoE + SWA/global hybrid attention not mapped to NPU GEMM patterns)
• GPU HIP: GGUF — validated
• GPU Vulkan: GGUF — functional, perf data pending
• CPU: GGUF — functional, perf data pending

8. Falcon

TII's Falcon. Parallel attention+FFN architecture with multi-query attention (MQA). GGUF through GPU HIP.

• NPU: Falcon-7B build stanza in build_xclbins.sh (build_falcon_7b). Uses padded dimensions (H=4544→4608, nearest multiple of 128). Run ./build_xclbins.sh falcon_7b to compile xclbins.
• GPU HIP: GGUF — validated
• GPU Vulkan: GGUF — functional, perf data pending
• CPU: GGUF — functional, perf data pending

9. OLMo

AI2's OLMo. LayerNorm instead of RMSNorm, no RoPE (learned positional embeddings). GGUF through GPU HIP.

• NPU: OLMoE-1B build stanza in build_xclbins.sh (build_olmoe). MoE expert batched GEMM MLIR generator in generators/n1_core_moe_expert.py. Run ./build_xclbins.sh olmoe_1b to compile.
• GPU HIP: GGUF — validated
• GPU Vulkan: GGUF — functional, perf data pending
• CPU: GGUF — functional, perf data pending

10. ZR1

Zyphra reasoning-tuned dense transformer (Qwen2 architecture). End-to-end validated at ~26 tok/s on Vulkan ZINC. 1BP format conversion complete. 2026-08-05: 187.9 tok/s on llama.cpp Vulkan (Radeon 8060S, Q4_K_M, tg32) — the ZINC figure was the previous path.

• NPU: ZR1-1.5B (dense Qwen2 arch) — build via existing Qwen3-0.6B xclbin stanzas (same tile template, different K/N dims in config)
• GPU HIP: GGUF — validated (kernel bench: 431 tok/s Q1, 426 tok/s fused TQ2)
• GPU Vulkan: 1.5B at ~26 tok/s — ✅ validated end-to-end (llama.cpp Vulkan: 187.9 tok/s tg, 1936 tok/s pp)
• CPU: ✅ universal GGUF backend

11. Nanbeige4.1

3B dense reasoning model with unusual head_dim=80. NPU-only at present.

• NPU: Nanbeige4.1-3B (build stanza in build_xclbins.sh)
• GPU HIP: ❌ not yet
• GPU Vulkan: ❌ not yet
• CPU: ✅ universal GGUF backend

Mixture-of-Experts (MoE)

12. Zaya1

Zyphra MoE architecture with CCA (Cross-Channel Attention) + MoE FFN. Our flagship 1BP ternary format model. Tile8 GEMV benchmark (28-layer, Zaya1-8B shaped) measured at 77 tok/s on ROCm HIP.

• Zaya1-8B: ~64 tok/s on ROCm HIP — ✅ validated
• Zaya1-74B-A4B: 16.7 tok/s on ROCm TheRock HIP 7.15a — measured live 2026-08-05 (see benchmarks/RESULTS-zaya1-74b-benchmark-2026-08-05.md); supersedes the earlier preliminary ~17.9 tok/s
• Format: 1BP ternary native + GGUF
• NPU: Native TQ2 ternary via --native-tq2 flag in npu_ternaryd.cpp. Uses gemm_generate_sequence_tq2() for runtime instruction generation — 4× less DDR traffic than INT8 bridge. Ping-pong LUT decode in mm_ternary_tq2.cc (2-buffer MAC/DMA overlap). TQ1 (1.58-bit) support via mm_ternary_tq1.cc with base-3 LUT decode.
• GPU HIP: Tile8 GEMV: 77 tok/s (28-layer synthetic, Zaya1-8B shaped) — ✅ validated
• GPU Vulkan: GGUF — functional, perf data pending
• CPU: AVX-512 portable path — ~2.5 tok/s (8B-shaped, real forward()+generate() loop)

13. DeepSeek V2/V3/R1

MoE with Multi-Head Latent Attention (MLA). DeepSeek-R1 distill variants on NPU. Full DeepSeek family through GPU HIP.

• NPU: DeepSeek-R1-Distill-Llama-8B (Llama arch — build via build_llama in build_xclbins.sh), DeepSeek-R1-0528-Qwen3-8B (Qwen3 arch — build via build_qwen3_8b)
• GPU HIP: GGUF — validated
• GPU Vulkan: GGUF — functional, perf data pending
• CPU: GGUF — functional, perf data pending

14. Qwen3.6-MoE-35B

35B total parameters, 256 experts (8 active/token), 3B active. 40 layers (30 GatedDeltaNet linear-attn + 10 full-attn), 262k context window. Official model Qwen/Qwen3.6-35B-A3B (released 2026-04-16, Apache-2.0, 35.95B params BF16) — verified byte-identical config to Qwen3.5-35B-A3B (same qwen3_5_moe arch, dims, vocab). Vision-capable (image-text-to-text). Peano-compiled INT8 xclbins with Q4_K_S quantization.

• NPU: ✅ validated — 11.66 tok/s decode @1k ctx (12.17 @2k → 8.82 @32k), prefill 98.05 → 239.79 tok/s, FastFlowLM v0.9.46 official stack + official Q4_K_S weights, Strix Halo, measured 2026-07-31 — see benchmarks/RESULTS-qwen3.6-35b-a3b-npu-flm-2026-07-30.md. zaya_server integration works end-to-end (per-request flm run spawn with file stdio; see docs/research/qwen36-npu-zaya-integration.md for the FLM pipe/serve-mode workarounds). Native npu_engine_universal MoE path still open (no expert routing yet).
• GPU HIP: ❌ untested e2e. The 20 tok/s number previously listed here was measured on Qwen3.5-35B-A3B Q4_K (zaya_server, identical arch) — corrected attribution.
• GPU Vulkan: ✅ validated — Q4_K_M GGUF at 75.65 tok/s decode (75.95 @8k), 1105.71 tok/s prefill, llama.cpp Vulkan on Strix Halo, measured 2026-07-30 — see benchmarks/RESULTS-qwen3.6-35b-a3b-vulkan-2026-07-30.md
• CPU: ✅ universal GGUF backend

15. GPT-OSS-20B

MoE architecture. Both base and safeguard variants pre-compiled for NPU.

• NPU: GPT-OSS-20B, GPT-OSS-Safeguard-20B (6 xclbins each including expert.xclbin for MoE dispatch). MoE expert batched GEMM MLIR generator in generators/n1_core_moe_expert.py.
• GPU HIP: ❌ not yet
• GPU Vulkan: ❌ not yet
• CPU: ✅ universal GGUF backend

State-Space Models (SSM)

16. BlackMamba

Mamba1 SSM + top-1 MoE gating. No attention mechanism — alternating SSM scan and MoE FFN dispatch per layer. Our fastest end-to-end model family.

• BlackMamba 1.5B: 79.4 tok/s on ROCm HIP — ✅ validated (fastest overall)
• BlackMamba 2.8B: 46.0 tok/s on ROCm HIP — ✅ validated
• NPU: SSM scan kernel in kernel/ssm_selective_scan.cc supports the selective-scan primitive for both Mamba1 and Mamba2. MoE FFN layers via generators/n1_core_moe_expert.py. Build with existing GPT-OSS MoE xclbin patterns.
• GPU HIP: 79.4 tok/s (1.5B) / 46.0 tok/s (2.8B) — ✅ validated (Mamba1 HIP backend)
• GPU Vulkan: ❌ not yet (Mamba1 scan requires HIP cooperative-groups; Vulkan port pending)
• CPU: ✅ universal GGUF backend

17. Zamba2

Mamba2-hybrid architecture: Mamba2 SSM layers with sparse attention every 6 layers. End-to-end validated at ~30 tok/s on Vulkan ZINC. Mamba2 decode block benchmark measured at 1270 tok/s on ROCm HIP. Engine + converter validated against HF transformers end-to-end (logits corr 0.997, top-20 overlap 20/20 on 2.7B) — full checkpoint-faithful pipeline: shared-block duplication + per-layer gate_up LoRA folding, TRUE mamba2 scan (state [head][d_state][head_dim], gate-then-norm), GELU FFN, mem-rope flag, conv kernel cross-correlation order.

• Zamba2-1.2B: Vulkan ZINC — ✅ validated · HIP e2e: mamba layers match CPU (d_state=128 chunked scan); e2e hang under async WIP
• Zamba2-2.7B: ~30 tok/s on Vulkan ZINC — ✅ validated · HIP e2e (2026-08-05): 12.5 tok/s (30 tok, 54 layers) · CPU reference 0.6 tok/s · logits corr 0.997 vs HF
• Zamba2-7B: HIP e2e (2026-08-05): 4.5 tok/s (81 layers: 68 mamba + 13 hybrid) · logits corr 0.992 vs HF — ✅ validated
• NPU: Zamba2-2.7B build stanza in build_xclbins.sh (build_zamba2_2_7b). AIE2 selective scan kernel in kernel/ssm_selective_scan.cc (per-head d_state=64 vectorized, 32 heads/tile). SSM scan MLIR generator in generators/n1_core_ssm_scan.py (16 tiles parallel). NPU GEMM handles in_proj/out_proj; SSM recurrence runs on AIE tiles.
• GPU HIP: Mamba2 decode block: 1270 tok/s — ✅ kernel verified · e2e 12.5/4.5 tok/s (2.7B/7B) · 1.2B d_state=128 chunked scan
• GPU Vulkan: ~30 tok/s (2.7B e2e) — ✅ validated
• CPU: ✅ universal GGUF backend

18. Zamba

Original Zamba-7B-v1: Mamba1 SSM + shared attention layers. GGUF through GPU HIP.

• Zamba-7B-v1: GGUF through ROCm HIP — validated
• NPU: Zamba (Mamba1) — SSM scan primitive supported by kernel/ssm_selective_scan.cc. Shared attention layers via standard NPU QKV/O xclbins.
• GPU Vulkan: GGUF — functional, perf data pending
• CPU: GGUF — functional, perf data pending

Ternary-Native

19. BitNet / Bonsai (Deepgrove)

Ternary b1.58 architecture — weights constrained to {-1, 0, +1}. Our Q1_0 1024-block kernel achieves 433 tok/s synthetic (kernel-level, 28-layer buffer). The TQ2 2-bit quantization format provides a 4× DDR bandwidth savings over INT8.

• Bonsai-1.7B: 21.9 tok/s on ROCm HIP — ✅ validated end-to-end
• Bonsai-4B: ROCm HIP — 🚧 integration in progress
• Bonsai-8B: ROCm HIP — 🚧 integration in progress
• Bonsai-27B: ROCm HIP — 🔬 experimental
• NPU: Bonsai (ternary 1.58-bit / TQ1) uses mm_ternary_tq1.cc (base-3 LUT decode + ping-pong). TQ2 models via gemm_generate_sequence_tq2() and --native-tq2 flag in npu_ternaryd.cpp. 4× smaller DDR footprint vs INT8 bridge.
• GPU Vulkan: Q1_0 binary kernel — ✅ validated (318 tok/s kernel-level)
• CPU: ✅ universal GGUF backend

Vision-Language

20. Qwen2-VL / Qwen3-VL

Vision transformers + Qwen text decoder. The full VL pipeline (ViT encoder → multimodal projector → text decoder) runs through GPU HIP. Select models pre-compiled for NPU.

• Qwen2.5-VL-3B-Instruct: NPU (build stanza ready, 7 xclbins) · GPU HIP — validated
• Qwen3-VL-4B-Instruct: NPU (build stanza ready, 6 xclbins) · GPU HIP — validated
• GPU Vulkan: ❌ not yet (ViT encoder not yet ported to Vulkan)
• CPU: ✅ universal GGUF backend

Speech-to-Text

21. Whisper

OpenAI Whisper V3 Turbo. Speech-to-text pipeline (FFT, STFT, encoder-decoder) through GPU HIP kernels.

• NPU: Whisper-V3-Turbo (build stanza ready, 5 xclbins)
• GPU HIP: FFT/STFT kernels — validated
• GPU Vulkan: ❌ not yet
• CPU: ✅ universal GGUF backend

Embedding

22. Embedding-Gemma-300M

Text embedding model based on Gemma architecture.

• NPU: Embedding-Gemma-300M (build stanza ready, 4 xclbins)
• GPU HIP: ❌ not yet (embedding extraction pipeline pending)
• GPU Vulkan: ❌ not yet
• CPU: ✅ universal GGUF backend

1BP Format Policy (2026-07-31 verdict, ppl-measured)

Dense (non-ternary-trained) models → Q4NX, converted from Q8_0/bf16 sources. TQ2/TQ2NZ are only for ternary-native checkpoints (BitNet/Bonsai-class). Converting a dense model to TQ2/TQ2NZ is catastrophically destructive — measured on a controlled experiment (Qwen3-0.6B, same Q8_0 source, same converter, same harness, 300 Alpaca samples):

Mechanism: ternary mapping inflates row norms (mean|x| 0.19 @ L0 → 86 @ L28) → flat logits. The Q4NX control exonerates the converter/reader pipeline — the destruction is the quant scheme, not a bug. Note the catalog's "Verified" column below means loads/runs; quality is gated by the ppl harness (ppl_generic, CI job ppl-gate), not by load success. Legacy TQ2 files of dense models (e.g. the old Qwen3-0.6B.1bp, ppl 3.7e8) are garbage — use Qwen3-0.6B-q8-q4nx.1bp (ppl 62, and 16.3 on the 48-sample gate set).

1BP Model Catalog

Our HuggingFace organization (bong-water-water-bong) hosts 37 1BP format models across all supported families. Each is a self-contained single-file model (magic 1BP\0, Q4NX 4-bit quant for dense models, or TQ2 ternary 2-bit for ternary-native Bonsai only — see the format policy above).

1BP loading validated via OnebpModel API. NPU inference (onebp_infer.cpp) is WIP.

Live Benchmarks (2026-07-29)

Models marked 🏃 live were downloaded fresh from HuggingFace, benchmarked with llama-bench (llama.cpp build e3546c7), and deleted — one model at a time, no disk waste. Tests ran on Strix Halo (Ryzen AI Max+ 395, Radeon 8060S, 256 GB/s).

Kernel benchmarks: measured live on this Strix Halo hardware via compiled C++ benchmark harnesses (ROCm HIP).

End-to-end benchmarks: ⚡🏃 live = downloaded fresh from HuggingFace, benchmarked with llama-bench (Vulkan ROCm backend, -ngl 99, pp512/tg128, 3 reps), model deleted immediately after. Each model ∼400–700 MB, test cycle ≈2 min per model.

Prior data (📋 prior / 📋 zinc) = from site/benchmarks.json (authoritative) or ZINC GPU benchmark logs.

Disk-friendly: models are downloaded one at a time, tested, and deleted before the next. Peak disk usage never exceeds ∼1 GB beyond baseline. For re-runs that reuse a model, scripts/model-cache.sh link <name> <url> <dest> keeps one canonical copy and symlinks every consumer dir — re-downloads and duplicate copies are both symlink-free shortcuts (one real copy, N links).

Kernel-Level Microbenchmarks

⚠️ These measure single-GEMM-kernel throughput, isolated and correctness-verified bit-exact against a CPU reference. They exclude KV-cache attention, softmax, RoPE, non-GEMM FFN ops, sampler, tokenizer, and host↔device transfers — not an end-to-end decode number. See performance methodology →.

End-to-End (real models, real prompts, Strix Halo)

NPU FLM Model Catalog

37 pre-compiled NPU models extracted from ROCm/FastFlowLM v0.9.46 (209 xclbins total). Source: engine/npu/tools/flm_model_map.json.

Qwen3 Family (7 models)

Qwen3.5 Family (4 models)

Qwen3.6 Family (1 model)

Qwen2.5 & Qwen2.5-VL Family (2 models)

Gemma4 Family (2 models)

Gemma3 Family (2 models)

MedGemma Family (2 models)

TranslateGemma (1 model)

Phi4 Family (1 model)

Nanbeige Family (1 model)

Llama Family (3 models)

DeepSeek Family (2 models)

GPT-OSS Family (2 models, MoE)

LFM2 Family (3 models)

LFM2.5 Family (2 models)

Specialized Models (3 models)

Peano Compilation Status Summary

        <int shift = 0;
        for (int j = 0; j < 4; j++) {
            uint8_t sc = scales[is++];
            float dl = d * (sc & 0xF), ml = dmin * (sc >> 4);
            for (int l = 0; l < 16 && base + n + j * 32 + l < count; l++)
                out[base + n + j * 32 + l] = dl * ((q[l] >> shift) & 3) - ml;
            sc = scales[is++];
            dl = d * (sc & 0xF); ml = dmin * (sc >> 4);
            for (int l = 0; l < 16 && base + n + j * 32 + 16 + l < count; l++)
                out[base + n + j * 32 + 16 + l] = dl * ((q[l + 16] >> shift) & 3) - ml;
            shift += 2;
        }
        q += 32;
    }
}
return true;

}

bool dequant_q3_k(const uint8_t* bd, float* out, int count) { const int BS = 256; int nb = (count + BS - 1) / BS; const uint8_t* p = bd; for (int b = 0; b < nb; b++) { uint8_t hmask[32]; memcpy(hmask, p, 32); p += 32; uint8_t qs[64]; memcpy(qs, p, 64); p += 64; uint8_t raw_scales[12]; memcpy(raw_scales, p, 12); p += 12; float d_all = read_f16(p); p += 2;

    uint32_t aux[4] = {0, 0, 0, 0};
    memcpy(aux, raw_scales, 12);
    const uint32_t kmask1 = 0x03030303, kmask2 = 0x0f0f0f0f;
    uint32_t tmp = aux[2];
    aux[2] = ((aux[0] >> 4) & kmask2) | (((tmp >> 4) & kmask1) << 4);
    aux[3] = ((aux[1] >> 4) & kmask2) | (((tmp >> 6) & kmask1) << 4);
    aux[0] = (aux[0] & kmask2) | (((tmp >> 0) & kmask1) << 4);
    aux[1] = (aux[1] & kmask2) | (((tmp >> 2) & kmask1) << 4);
    int8_t scales[16]; memcpy(scales, aux, 16);
    for (int j = 0; j < 16; j++) scales[j] -= 32;

    int base = b * BS;
    int is = 0; const uint8_t* q = qs;
    uint8_t m = 1;
    for (int n = 0; n < BS && base + n < count; n += 128) {
        int shift = 0;
        for (int j = 0; j < 4; j++) {
            float dl = d_all * scales[is++];
            for (int l = 0; l < 16 && base + n + j * 32 + l < count; l++)
                out[base + n + j * 32 + l] =
                    dl * (((int8_t)((q[l] >> shift) & 3)) - ((hmask[l] & m) ? 0 : 4));
            dl = d_all * scales[is++];
            for (int l = 0; l < 16 && base + n + j * 32 + 16 + l < count; l++)
                out[base + n + j * 32 + 16 + l] =
                    dl * (((int8_t)((ql + 16] >> shift) & 3)) - ((hmask[l + 16] & m) ? 0 : 4));
            shift += 2;
            m = (uint8_t)(m << 1);
        }
        q += 32;
    }
}
return true;

}

bool dequant_q4_k(const uint8_t* bd, float* out, int count) { const int BS = 256; int nb = (count + BS - 1) / BS; const uint8_t* p = bd; for (int b = 0; b < nb; b++) { float d = read_f16(p); p += 2; float dmin = read_f16(p); p += 2; uint8_t scales[12]; memcpy(scales, p, 12); p += 12; uint8_t qs[128]; memcpy(qs, p, 128); p += 128; int base = b * BS; int is = 0; const uint8_t* q = qs; for (int off = 0; off < BS && base + off < count; off += 64) { uint8_t sc, m; k_get_scale_min(scales, is, sc, m); float d1 = d * sc, m1 = dmin * m; k_get_scale_min(scales, is + 1, sc, m); float d2 = d * sc, m2 = dmin * m; for (int l = 0; l < 32 && base + off + l < count; l++) out[base + off + l] = d1 * (q[l] & 0xF) - m1; for (int l = 0; l < 32 && base + off + 32 + l < count; l++) out[base + off + 32 + l] = d2 * (q[l] >> 4) - m2; q += 32; is += 2; } } return true; }

bool dequant_q5_k(const uint8_t* bd, float* out, int count) { const int BS = 256; int nb = (count + BS - 1) / BS; const uint8_t* p = bd; for (int b = 0; b < nb; b++) { float d = read_f16(p); p += 2; float dmin = read_f16(p); p += 2; uint8_t scales[12]; memcpy(scales, p, 12); p += 12; uint8_t qh[32]; memcpy(qh, p, 32); p += 32; uint8_t ql[128]; memcpy(ql, p, 128); p += 128; int base = b * BS; int is = 0; const uint8_t* q = ql; uint8_t u1 = 1, u2 = 2; for (int n = 0; n < BS && base + n < count; n += 64) { uint8_t sc, m; k_get_scale_min(scales, is, sc, m); float d1 = d * sc, m1 = dmin * m; k_get_scale_min(scales, is + 1, sc, m); float d2 = d * sc, m2 = dmin * m; for (int l = 0; l < 32 && base + n + l < count; l++) out[base + n + l] = d1 * ((q[l] & 0xF) + (qh[l] & u1 ? 16 : 0)) - m1; for (int l = 0; l < 32 && base + n + 32 + l < count; l++) out[base + n + 32 + l] = d2 * ((q[l] >> 4) + (qh[l] & u2 ? 16 : 0)) - m2; q += 32; is += 2; u1 = (uint8_t)(u1 << 2); u2 = (uint8_t)(u2 << 2); } } return true; }

// Port of llama.cpr's dequantize_row_q6_K (ggml-quants.c) — the previous // version of this function only ever wrote indices [0,32] of each 128- // element half (192 of every 256 elements silently left as zero); this // was never caught because it was only spot-checked against real files, // never synthetically verified against the Python gguf reference the // way Q2_K/Q3_K/Q5_K were. Verified byte-exact against gguf.quants.Q6_K // with seeded random blocks before this fix landed. bool dequant_q6_k(const uint8_t* bd, float* out, int count) { const int BS = 256; int nb = (count + BS - 1) / BS; const uint8_t* p = bd; for (int b = 0; b < nb; b++) { const uint8_t* ql = p; p += 128; const uint8_t* qh = p; p += 64; const int8_t* sc = (const int8_t*)p; p += 16; float d = read_f16(p); p += 2; int base = b * BS; float* y = out + base; for (int n = 0; n < BS; n += 128) { for (int l = 0; l < 32; l++) { int is = l / 16; int8_t q1 = (int8_t)((ql[l & 0xF) | (((qh[l] >> 0) & 3) << 4)) - 32; int8_t q2 = (int8_t)((ql[l + 32] & 0xF) | (((qh[l] >> 2) & 3) << 4)) - 32; int8_t q3 = (int8_t)((ql[l] >> 4) | (((qh[l] >> 4) & 3) << 4)) - 32; int8_t q4 = (int8_t)((ql[l + 32] >> 4) | (((qh[l] >> 6) & 3) << 4)) - 32; if (base + n + l < count) y[l] = d * sc[is + 0] * q1; if (base + n + l + 32 < count) y[l + 32] = d * sc[is + 2] * q2; if (base + n + l + 64 < count) y[l + 64] = d * sc[is + 4] * q3; if (base + n + l + 96 < count) y[l + 96] = d * sc[is + 6] * q4; } y += 128; ql += 64; qh += 32; sc += 8; } } return true; }

bool dequant_q8_k(const uint8_t* bd, float* out, int count) { const int BS = 256; int nb = (count + BS - 1) / BS; const uint8_t* p = bd; for (int b = 0; b < nb; b++) { float d; memcpy(&d, p, 4); p += 4; int8_t qs[256]; memcpy(qs, p, 256); p += 256; p += 32; // bsums — not needed for dequant int base = b * BS; for (int l = 0; l < BS && base + l < count; l++) out[base + l] = d * qs[l]; } return true; }

} // namespace

inline float bf16_to_fp32(uint16_t bf16) { // bfloat16 -> float32: reinterpret the bits by shifting left 16. // No special handling needed for NaN/Inf — the bit pattern is identical. float result; uint32_t f32 = (uint32_t)bf16 << 16; memcpy(&result, &f32, 4); return result; }

GgufBlockInfo gguf_block_info(uint32_t dtype) { switch (dtype) { case GGUF_DTYPE_F32: return {1, 4}; case GGUF_DTYPE_F16: return {1, 2}; case GGUF_DTYPE_BF16: return {1, 2}; case GGUF_DTYPE_Q4_0: return {32, 18}; case GGUF_DTYPE_Q4_1: return {32, 20}; case GGUF_DTYPE_Q5_0: return {32, 22}; case GGUF_DTYPE_Q5_1: return {32, 24}; case GGUF_DTYPE_Q8_0: return {32, 34}; case GGUF_DTYPE_Q8_1: return {32, 36}; case GGUF_DTYPE_Q2_K: return {256, 84}; case GGUF_DTYPE_Q3_K: return {256, 110}; case GGUF_DTYPE_Q4_K: return {256, 144}; case GGUF_DTYPE_Q5_K: return {256, 176}; case GGUF_DTYPE_Q6_K: return {256, 210}; case GGUF_DTYPE_Q8_K: return {256, 292}; // IQ format block sizes (for correct file offset computation). // Dequantization is not implemented here — these return false // from gguf_dequant; callers can use get_tensor_raw() for // custom dequant. case GGUF_DTYPE_IQ1_S: return {256, 206}; case GGUF_DTYPE_IQ1_M: return {256, 230}; case GGUF_DTYPE_IQ2_XXS: return {256, 166}; case GGUF_DTYPE_IQ2_S: return {256, 214}; case GGUF_DTYPE_IQ3_XXS: return {256, 198}; case GGUF_DTYPE_IQ3_S: return {256, 238}; case GGUF_DTYPE_IQ4_NL: return {32, 22}; case GGUF_DTYPE_IQ4_XS: return {256, 214}; case GGUF_DTYPE_Q4_0_4_4: return {256, 208}; case GGUF_DTYPE_Q4_0_4_8: return {256, 208}; case GGUF_DTYPE_Q4_0_8_8: return {256, 272}; // Project-specific ternary/binary formats (h1b weight format) // TQ2_0_g128: ternary, 2-bit packed, group=128 → blocks of 128 el, 33 bytes // TQ2_0 ternary: fp16 scale (2) + 2-bit codes (128*2/8=32) = 34 bytes case GGUF_DTYPE_TQ2_0_G128: return {128, 34}; // Q1_0 binary: fp16 scale (2) + 1-bit codes (128/8=16) = 18 bytes case GGUF_DTYPE_Q1_0_G128: return {128, 18}; default: return {0, 0}; } }

bool gguf_dequant(uint32_t dtype, const uint8_t* data, float* out, int count) { switch (dtype) { case GGUF_DTYPE_F32: memcpy(out, data, (size_t)count * 4); return true; case GGUF_DTYPE_F16: for (int i = 0; i < count; i++) out[i] = read_f16(data + (size_t)i * 2); return true; case GGUF_DTYPE_BF16: for (int i = 0; i < count; i++) out[i] = bf16_to_fp32(((const uint16_t*)data)[i]); return true; case GGUF_DTYPE_Q4_0: return dequant_q4_0(data, out, count); case GGUF_DTYPE_Q4_1: return dequant_q4_1(data, out, count); case GGUF_DTYPE_Q5_0: return dequant_q5_0(data, out, count); case GGUF_DTYPE_Q5_1: return dequant_q5_1(data, out, count); case GGUF_DTYPE_Q8_0: return dequant_q8_0(data, out, count); case GGUF_DTYPE_Q8_1: { // Q8_1: fp16 d (scale) + fp16 s (unused for dequant) + int8[32] for (int i = 0; i < count; i++) { int bi = i / 32, ei = i % 32; const uint8_t* blk = data + (size_t)bi * 36; float d = read_f16(blk); out[i] = (float)((int8_t)blk[4 + ei]) * d; } return true; } case GGUF_DTYPE_Q2_K: return dequant_q2_k(data, out, count); case GGUF_DTYPE_Q3_K: return dequant_q3_k(data, out, count); case GGUF_DTYPE_Q4_K: return dequant_q4_k(data, out, count); case GGUF_DTYPE_Q5_K: return dequant_q5_k(data, out, count); case GGUF_DTYPE_Q6_K: return dequant_q6_k(data, out, count); case GGUF_DTYPE_Q8_K: return dequant_q8_k(data, out, count); case GGUF_DTYPE_Q1_0_G128: { // Q1_0 binary: fp16 scale + sign bits (1 bit per element) for (int i = 0; i < count; i++) { int bi = i / 128, ei = i % 128; const uint8_t* blk = data + (size_t)bi * 18; float sc = read_f16(blk); const uint8_t* bits = blk + 2; out[i] = (bits[ei / 8] >> (ei % 8)) & 1 ? sc : -sc; } return true; } case GGUF_DTYPE_TQ2_0_G128: { // TQ2_0 ternary: fp16 scale + 2-bit codes (0=-s, 1=0, 2=+s, 3=0) for (int i = 0; i < count; i++) { int bi = i / 128, ei = i % 128; const uint8_t* blk = data + (size_t)bi * 34; float sc = read_f16(blk); uint8_t c = (blk[2 + ei/4] >> ((ei%4)*2)) & 3; if (c == 0) out[i] = -sc; else if (c == 2) out[i] = sc; else out[i] = 0.0f; } return true; } default: return false; } }

GgufReader::~GgufReader() { if (f_) fclose(f_); }

std::string GgufReader::read_string() { uint64_t len = 0; if (fread(&len, 8, 1, f_) != 1) return {}; static constexpr uint64_t MAX_STRING_LEN = 1ULL * 1024 * 1024; if (len > MAX_STRING_LEN) { fseeko(f_, (off_t)len, SEEK_CUR); return "truncated"; } std::string s(len, '\0'); if (len > 0 && fread(&s[0], 1, len, f_) != len) return {}; return s; }

// GGUF KV value types: 0=u8 1=i8 2=u16 3=i16 4=u32 5=i32 6=f32 7=bool // 8=string 9=array 10=u64 11=i64 12=f64 bool GgufReader::read_kv_value(uint32_t vtype, KV& out) { out.vtype = vtype; switch (vtype) { case 0: { uint8_t v; if (fread(&v, 1, 1, f_) != 1) return false; out.u = v; return true; } case 1: { int8_t v; if (fread(&v, 1, 1, f_) != 1) return false; out.u = (uint64_t)(int64_t)v; return true; } case 2: { uint16_t v; if (fread(&v, 2, 1, f_) != 1) return false; out.u = v; return true; } case 3: { int16_t v; if (fread(&v, 2, 1, f_) != 1) return false; out.u = (uint64_t)(int64_t)v; return true; } case 4: { uint32_t v; if (fread(&v, 4, 1, f_) != 1) return false; out.u = v; return true; } case 5: { int32_t v; if (fread(&v, 4, 1, f_) != 1) return false; out.u = (uint64_t)(int64_t)v; return true; } case 6: { float v; if (fread(&v, 4, 1, f_) != 1) return false; out.f = v; return true; } case 7: { uint8_t v; if (fread(&v, 1, 1, f_) != 1) return false; out.u = v; return true; } case 8: { out.s = read_string(); return true; } case 9: { uint32_t at; fread(&at, 4, 1, f_); uint64_t an; fread(&an, 8, 1, f_); static constexpr uint64_t MAX_ARRAY_COUNT = 1000000; if (an > MAX_ARRAY_COUNT) { for (uint64_t j = 0; j < an; j++) skip_kv_value(at); an = 0; } if (at == 8) { out.arr_str.resize(an); for (uint64_t j = 0; j < an; j++) out.arr_str[j] = read_string(); } else { for (uint64_t j = 0; j < an; j++) skip_kv_value(at); } return true; } case 10: { uint64_t v; if (fread(&v, 8, 1, f_) != 1) return false; out.u = v; return true; } case 11: { int64_t v; if (fread(&v, 8, 1, f_) != 1) return false; out.u = (uint64_t)v; return true; } case 12: { double v; if (fread(&v, 8, 1, f_) != 1) return false; out.f = v; return true; } default: return false; } }

void GgufReader::skip_kv_value(uint32_t vtype) { switch (vtype) { case 0: case 1: case 7: fseeko(f_, 1, SEEK_CUR); break; case 2: case 3: fseeko(f_, 2, SEEK_CUR); break; case 4: case 5: case 6: fseeko(f_, 4, SEEK_CUR); break; case 8: { read_string(); break; } case 9: { uint32_t at; fread(&at, 4, 1, f_); uint64_t an; fread(&an, 8, 1, f_); static constexpr uint64_t MAX_ARRAY_COUNT = 1000000; if (an > MAX_ARRAY_COUNT) an = 0; if (at == 8) { for (uint64_t j = 0; j < an; j++) read_string(); } else { for (uint64_t j = 0; j < an; j++) skip_kv_value(at); } break; } case 10: case 11: case 12: fseeko(f_, 8, SEEK_CUR); break; default: break; } }

bool GgufReader::open(const std::string& path) { f_ = fopen(path.c_str(), "rb"); if (!f_) return false; char magic[4]; if (fread(magic, 1, 4, f_) != 4 || memcmp(magic, "GGUF", 4) != 0) { fclose(f_); f_ = nullptr; return false; } uint32_t version; if (fread(&version, 4, 1, f_) != 1) { fclose(f_); f_ = nullptr; return false; } if (version != 2 && version != 3) { fclose(f_); f_ = nullptr; return false; } uint64_t tensor_count, kv_count; if (fread(&tensor_count, 8, 1, f_) != 1 || fread(&kv_count, 8, 1, f_) != 1) { fclose(f_); f_ = nullptr; return false; } static constexpr uint64_t MAX_TENSOR_COUNT = 200000, MAX_KV_COUNT = 200000; if (tensor_count > MAX_TENSOR_COUNT || kv_count > MAX_KV_COUNT) { fclose(f_); f_ = nullptr; return false; }

for (uint64_t i = 0; i < kv_count; i++) {
    std::string key = read_string();
    uint32_t vtype; fread(&vtype, 4, 1, f_);
    KV kv;
    if (!read_kv_value(vtype, kv)) { fclose(f_); f_ = nullptr; return false; }
    kv_[key] = std::move(kv);
}
if (auto it = kv_.find("general.architecture"); it != kv_.end() && it->second.vtype == 8) arch_ = it->second.s;

uint64_t alignment = 32;
{ uint32_t a; if (get_u32("general.alignment", a) && a > 0) alignment = a; }

static constexpr uint32_t MAX_NDIM = 16;
static constexpr uint64_t MAX_DIM_SIZE = 1ULL << 24;
for (uint64_t i = 0; i < tensor_count; i++) {
    std::string name = read_string();
    uint32_t ndim; fread(&ndim, 4, 1, f_);
    if (ndim > MAX_NDIM) { fclose(f_); f_ = nullptr; return false; }
    GgufTensorInfo ti;
    ti.shape.resize(ndim);
    for (uint32_t d = 0; d < ndim; d++) {
        fread(&ti.shape[d], 8, 1, f_);
        if (ti.shape[d] > MAX_DIM_SIZE) { fclose(f_); f_ = nullptr; return false; }
    }
    fread(&ti.dtype, 4, 1, f_);
    uint64_t rel_offset; fread(&rel_offset, 8, 1, f_);
    ti.numel = 1;
    for (auto s : ti.shape) ti.numel *= (s ? s : 1);
    ti.abs_offset = rel_offset; // fixed up to absolute after the loop
    tensor_order_.push_back(name);
    tensors_[name] = ti; // abs_offset placeholder for now
    // stash rel_offset in abs_offset temporarily; fixed up below
    tensors_[name].abs_offset = rel_offset;
}

uint64_t data_start = (uint64_t)ftell(f_);
uint64_t rem = data_start % alignment;
if (rem) data_start += alignment - rem;
for (auto& name : tensor_order_) tensors_[name].abs_offset += data_start;

return true;

}

bool GgufReader::has_tensor(const std::string& name) const { return tensors_.count(name) != 0; }

const GgufTensorInfo* GgufReader::tensor_info(const std::string& name) const { auto it = tensors_.find(name); return it == tensors_.end() ? nullptr : &it->second; }

// ── KV lookup with architecture-prefix fallback ── // GGUF files may store metadata keys with or without the architecture prefix. // E.g. "block_count" vs "mamba.block_count". We try both forms: // - If key contains a dot (e.g. "mamba.block_count"): try exact, then suffix. // - If key has no dot: try exact, then arch + "." + key. const GgufReader::KV* GgufReader::find_kv(const std::string& key) const { auto it = kv_.find(key); if (it != kv_.end()) return &it->second; auto dot = key.find('.'); if (dot != std::string::npos) { std::string suf = key.substr(dot + 1); if (!suf.empty()) { it = kv_.find(suf); if (it != kv_.end()) return &it->second; } } else if (!arch_.empty()) { std::string pre = arch_ + "." + key; it = kv_.find(pre); if (it != kv_.end()) return &it->second; } return nullptr; }

bool GgufReader::get_u32(const std::string& key, uint32_t& out) const { const KV* kv = find_kv(key); if (!kv) return false; if (kv->vtype <= 5 || kv->vtype == 7 || kv->vtype == 10 || kv->vtype == 11) { out = (uint32_t)kv->u; return true; } return false; }

bool GgufReader::get_f32(const std::string& key, float& out) const { const KV* kv = find_kv(key); if (!kv) return false; if (kv->vtype == 6 || kv->vtype == 12) { out = (float)kv->f; return true; } if (kv->vtype <= 5 || kv->vtype == 10 || kv->vtype == 11) { out = (float)(int64_t)kv->u; return true; } return false; }

bool GgufReader::get_string(const std::string& key, std::string& out) const { const KV* kv = find_kv(key); if (!kv || kv->vtype != 8) return false; out = kv->s; return true; }

bool GgufReader::get_string_array(const std::string& key, std::vectorstd::string& out) const { const KV* kv = find_kv(key); if (!kv || kv->vtype != 9) return false; out = kv->arr_str; return true; }

std::vectorstd::string GgufReader::kv_keys() const { std::vectorstd::string keys; keys.reserve(kv_.size()); for (auto& [k, v] : kv_) keys.push_back(k); return keys; }

bool GgufReader::get_tensor_raw(const std::string& name, int block_size, int block_bytes, std::vector<uint8_t>& out, uint64_t* out_numel) { auto it = tensors_.find(name); if (it == tensors_.end() || !f_ || block_size <= 0 || block_bytes <= 0) return false; const GgufTensorInfo& ti = it->second; if (out_numel) *out_numel = ti.numel; uint64_t n_blocks = (ti.numel + block_size - 1) / block_size; out.resize(n_blocks * (uint64_t)block_bytes); fseeko(f_, (off_t)ti.abs_offset, SEEK_SET); return fread(out.data(), 1, out.size(), f_) == out.size(); }

bool GgufReader::get_tensor_f32(const std::string& name, std::vector& out, size_t* out_n) { auto it = tensors_.find(name); if (it == tensors_.end() || !f_) return false; const GgufTensorInfo& ti = it->second; static constexpr uint64_t MAX_TENSOR_ELEMENTS = 1ULL << 30; if (ti.numel > MAX_TENSOR_ELEMENTS) return false; out.resize(ti.numel); if (out_n) *out_n = ti.numel;

GgufBlockInfo bi = gguf_block_info(ti.dtype);
if (bi.block_bytes <= 0) return false;
fseeko(f_, (off_t)ti.abs_offset, SEEK_SET);
uint64_t n_blocks = (ti.numel + bi.block_size - 1) / bi.block_size;
std::vector<uint8_t> block_buf((size_t)bi.block_bytes);
for (uint64_t b = 0; b < n_blocks; b++) {
    uint64_t start = b * bi.block_size;
    uint64_t count = std::min<uint64_t>(bi.block_size, ti.numel - start);
    if (fread(block_buf.data(), (size_t)bi.block_bytes, 1, f_) != 1) return false;
    if (!gguf_dequant(ti.dtype, block_buf.data(), out.data() + start, (int)count)) return false;
}
return true;

}