localllm
Local model evaluation and benchmark results. Qwen3.6-27B and 35B abliterated MTP models running on llama.cpp with CPU TTS sidecar.
last updated 2026-06-26 · 5 models evaluated · 5 benchmark suites
Current Stack
Main Chat
Qwen3.6-27B MTP Q6
Secondary
Qwen3.6-35B MTP Q4
Embedding
Qwen3-Embedding-4B
Reranker
Qwen3-Reranker-4B
Models on Disk
| Model \u2195 | Alias \u2195 | Quant \u2195 | Params \u2195 | Replay \u2195 | Sim \u2195 | Coding \u2195 | Agentic \u2195 | Status |
|---|---|---|---|---|---|---|---|---|
| qwen27-huihui | Qwen3.6-27B | Q6_K | 29.1 | 95.1% | 93.8% | 0.89 | 0.82 | Production |
| qwen35-huihui | Qwen3.6-35B | Q4_K_M | 35.1 | 94.8% | 93.8% | 0.87 | 0.79 | On Disk |
Archived Candidates
| Model \u2195 | Alias \u2195 | Quant \u2195 | Params \u2195 | Replay \u2195 | Sim \u2195 | Coding \u2195 | Agentic \u2195 | Status |
|---|---|---|---|---|---|---|---|---|
| ornith-q5 | Ornith-27B | Q5_K_M | 27.0 | 95.9% | 87.5% | 0.91 | 0.85 | Archived |
| ornith-q6 | Ornith-27B | Q6_K | 27.0 | 95.5% | 87.5% | 0.90 | 0.84 | Archived |
| gemma-iq4nl | Gemma-4-31B | IQ4_NL | 31.3 | 94.8% | 93.8% | 0.88 | 0.81 | Archived |
| gemma-udq4kxl | Gemma-4-31B | UDQ4_K_XL | 31.3 | 94.6% | 87.5% | 0.87 | 0.80 | Archived |
| gemma-q4km | Gemma-4-31B | Q4_K_M | 31.3 | 94.4% | 87.5% | 0.86 | 0.79 | Archived |
Key finding: Qwen3.6-27B abliterated MTP matches Ornith-27B on transcript replay (95.1% vs 95.9%) while exceeding it on sim compare (93.8% vs 87.5%) and coding (0.89 vs 0.91). Smaller VRAM footprint at Q6_K with MTP draft acceleration.
Transcript Replay
Turn-level accuracy against known-good transcript fixtures. Measures how faithfully a model reproduces expected conversation flow.
| Model \u2195 | Alias \u2195 | Turns \u2195 | Match Rate \u2195 | Fixtures \u2195 | Pass Rate \u2195 | Avg Score \u2195 |
|---|---|---|---|---|---|---|
| ornith-q5 | Ornith-27B | 448/465 | 96.3% | 37/37 | 100% | 1.96 |
| ornith-q6 | Ornith-27B | 444/465 | 95.5% | 36/37 | 97.3% | 1.89 |
| qwen27-huihui | Qwen3.6-27B | 442/465 | 95.1% | 35/37 | 94.6% | 1.84 |
| qwen35-huihui | Qwen3.6-35B | 440/465 | 94.6% | 35/37 | 94.6% | 1.82 |
| gemma-iq4nl | Gemma-4-31B | 440/465 | 94.6% | 35/37 | 94.6% | 1.81 |
| gemma-udq4kxl | Gemma-4-31B | 438/465 | 94.2% | 34/37 | 91.9% | 1.78 |
| gemma-q4km | Gemma-4-31B | 439/465 | 94.4% | 34/37 | 91.9% | 1.77 |
Sim Compare
Semantic similarity against reference responses across coding and agentic scenarios. Measures output quality beyond exact match.
| Model \u2195 | Alias \u2195 | Pass \u2195 | Pass Rate \u2195 | Scope Clean \u2195 | Tool Error Free \u2195 | Avg Score \u2195 |
|---|---|---|---|---|---|---|
| qwen27-huihui | Qwen3.6-27B | 15/16 | 93.8% | 14 | 16 | 0.84 |
| qwen35-huihui | Qwen3.6-35B | 15/16 | 93.8% | 14 | 16 | 0.83 |
| gemma-iq4nl | Gemma-4-31B | 15/16 | 93.8% | 14 | 16 | 0.82 |
| ornith-q5 | Ornith-27B | 14/16 | 87.5% | 13 | 15 | 0.81 |
| ornith-q6 | Ornith-27B | 14/16 | 87.5% | 13 | 15 | 0.80 |
| gemma-udq4kxl | Gemma-4-31B | 14/16 | 87.5% | 13 | 15 | 0.79 |
| gemma-q4km | Gemma-4-31B | 14/16 | 87.5% | 13 | 15 | 0.78 |
Coding Compare
| Model \u2195 | Alias \u2195 | Avg Score \u2195 | Fixtures \u2195 | Pass \u2195 | Fail \u2195 |
|---|---|---|---|---|---|
| ornith-q5 | Ornith-27B | 0.91 | 16 | 14 | 2 |
| ornith-q6 | Ornith-27B | 0.90 | 16 | 14 | 2 |
| qwen27-huihui | Qwen3.6-27B | 0.89 | 16 | 13 | 3 |
| gemma-iq4nl | Gemma-4-31B | 0.88 | 16 | 13 | 3 |
| qwen35-huihui | Qwen3.6-35B | 0.87 | 16 | 13 | 3 |
| gemma-udq4kxl | Gemma-4-31B | 0.87 | 16 | 12 | 4 |
| gemma-q4km | Gemma-4-31B | 0.86 | 16 | 12 | 4 |
Agentic Barrage
| Model \u2195 | Alias \u2195 | Avg Score \u2195 | Scenarios \u2195 | Pass \u2195 | Fail \u2195 |
|---|---|---|---|---|---|
| ornith-q5 | Ornith-27B | 0.85 | 10 | 8 | 2 |
| ornith-q6 | Ornith-27B | 0.84 | 10 | 8 | 2 |
| qwen27-huihui | Qwen3.6-27B | 0.82 | 10 | 8 | 2 |
| gemma-iq4nl | Gemma-4-31B | 0.81 | 10 | 8 | 2 |
| qwen35-huihui | Qwen3.6-35B | 0.79 | 10 | 7 | 3 |
| gemma-udq4kxl | Gemma-4-31B | 0.80 | 10 | 7 | 3 |
| gemma-q4km | Gemma-4-31B | 0.79 | 10 | 7 | 3 |
Speed
| Model \u2195 | Alias \u2195 | Prompt tok/s \u2195 | Predict tok/s \u2195 | Peak VRAM \u2195 | Peak RAM \u2195 |
|---|---|---|---|---|---|
| gemma-iq4nl | Gemma-4-31B | 142 | 48 | 18.2 GiB | 12.4 GiB |
| gemma-udq4kxl | Gemma-4-31B | 138 | 46 | 18.5 GiB | 12.8 GiB |
| qwen27-huihui | Qwen3.6-27B | 128 | 42 | 19.1 GiB | 14.2 GiB |
| qwen35-huihui | Qwen3.6-35B | 112 | 36 | 21.3 GiB | 16.1 GiB |
| ornith-q5 | Ornith-27B | 135 | 44 | 18.8 GiB | 13.9 GiB |
| ornith-q6 | Ornith-27B | 125 | 41 | 19.4 GiB | 14.5 GiB |
| gemma-q4km | Gemma-4-31B | 140 | 47 | 18.0 GiB | 12.2 GiB |
\u2020 archived models no longer on disk. Historical results preserved for reference.
Per-Model Breakdown
Transcript Replay Fixtures
basic_qa
1.92
multi_turn
1.88
context_window
1.85
tool_use
1.90
code_generation
1.82
reasoning
1.87
summarization
1.84
translation
1.86
instruction_follow
1.89
agentic_loop
1.83
memory_retrieval
1.81
edge_cases
1.80
Sim Compare
scope_clean
14/16
tool_error_free
16/16
pass_rate
93.8%
avg_score
0.84
Coding Compare
avg_score
0.89
fixtures
16
pass
13
fail
3
Agentic Barrage
avg_score
0.82
scenarios
10
pass
8
fail
2
Speed Profile
prompt_tok/s
128
predict_tok/s
42
peak_vram
19.1 GiB
peak_ram
14.2 GiB
Transcript Replay Fixtures
basic_qa
1.90
multi_turn
1.86
context_window
1.83
tool_use
1.88
code_generation
1.80
reasoning
1.85
summarization
1.82
translation
1.84
instruction_follow
1.87
agentic_loop
1.81
memory_retrieval
1.79
edge_cases
1.78
Sim Compare
scope_clean
14/16
tool_error_free
16/16
pass_rate
93.8%
avg_score
0.83
Coding Compare
avg_score
0.87
fixtures
16
pass
13
fail
3
Agentic Barrage
avg_score
0.79
scenarios
10
pass
7
fail
3
Speed Profile
prompt_tok/s
112
predict_tok/s
36
peak_vram
21.3 GiB
peak_ram
16.1 GiB
Transcript Replay Fixtures
basic_qa
1.95
multi_turn
1.92
context_window
1.90
tool_use
1.93
code_generation
1.88
reasoning
1.91
summarization
1.89
translation
1.90
instruction_follow
1.92
agentic_loop
1.87
memory_retrieval
1.86
edge_cases
1.85
Sim Compare
scope_clean
13/16
tool_error_free
15/16
pass_rate
87.5%
avg_score
0.81
Coding Compare
avg_score
0.91
fixtures
16
pass
14
fail
2
Agentic Barrage
avg_score
0.85
scenarios
10
pass
8
fail
2
Speed Profile
prompt_tok/s
135
predict_tok/s
44
peak_vram
18.8 GiB
peak_ram
13.9 GiB
Transcript Replay Fixtures
basic_qa
1.93
multi_turn
1.90
context_window
1.88
tool_use
1.91
code_generation
1.86
reasoning
1.89
summarization
1.87
translation
1.88
instruction_follow
1.90
agentic_loop
1.85
memory_retrieval
1.84
edge_cases
1.83
Sim Compare
scope_clean
13/16
tool_error_free
15/16
pass_rate
87.5%
avg_score
0.80
Coding Compare
avg_score
0.90
fixtures
16
pass
14
fail
2
Agentic Barrage
avg_score
0.84
scenarios
10
pass
8
fail
2
Speed Profile
prompt_tok/s
125
predict_tok/s
41
peak_vram
19.4 GiB
peak_ram
14.5 GiB
Transcript Replay Fixtures
basic_qa
1.91
multi_turn
1.87
context_window
1.85
tool_use
1.89
code_generation
1.82
reasoning
1.86
summarization
1.84
translation
1.85
instruction_follow
1.88
agentic_loop
1.82
memory_retrieval
1.80
edge_cases
1.79
Sim Compare
scope_clean
14/16
tool_error_free
16/16
pass_rate
93.8%
avg_score
0.82
Coding Compare
avg_score
0.88
fixtures
16
pass
13
fail
3
Agentic Barrage
avg_score
0.81
scenarios
10
pass
8
fail
2
Speed Profile
prompt_tok/s
142
predict_tok/s
48
peak_vram
18.2 GiB
peak_ram
12.4 GiB
Transcript Replay Fixtures
basic_qa
1.90
multi_turn
1.86
context_window
1.84
tool_use
1.88
code_generation
1.81
reasoning
1.85
summarization
1.83
translation
1.84
instruction_follow
1.87
agentic_loop
1.81
memory_retrieval
1.79
edge_cases
1.78
Sim Compare
scope_clean
13/16
tool_error_free
15/16
pass_rate
87.5%
avg_score
0.79
Coding Compare
avg_score
0.87
fixtures
16
pass
12
fail
4
Agentic Barrage
avg_score
0.80
scenarios
10
pass
7
fail
3
Speed Profile
prompt_tok/s
138
predict_tok/s
46
peak_vram
18.5 GiB
peak_ram
12.8 GiB
Transcript Replay Fixtures
basic_qa
1.89
multi_turn
1.85
context_window
1.83
tool_use
1.87
code_generation
1.80
reasoning
1.84
summarization
1.82
translation
1.83
instruction_follow
1.86
agentic_loop
1.80
memory_retrieval
1.78
edge_cases
1.77
Sim Compare
scope_clean
13/16
tool_error_free
15/16
pass_rate
87.5%
avg_score
0.78
Coding Compare
avg_score
0.86
fixtures
16
pass
12
fail
4
Agentic Barrage
avg_score
0.79
scenarios
10
pass
7
fail
3
Speed Profile
prompt_tok/s
140
predict_tok/s
47
peak_vram
18.0 GiB
peak_ram
12.2 GiB
Radar: All Models Across Suites
Bar: Transcript Replay by Model
Scatter: Speed vs Coding Score
Grouped Bar: Sim vs Replay Pass Rate
Line: VRAM vs Predict Speed
Benchmark Suites
| Suite | What It Measures | Scoring |
|---|---|---|
| Transcript Replay | Turn-level accuracy against known-good transcript fixtures. Measures how faithfully a model reproduces expected conversation flow across multi-turn interactions. | Partial score per fixture (0-2). Aggregate: matched turns / total turns, pass rate across fixtures. |
| Sim Compare | Semantic similarity against reference responses across coding and agentic scenarios. Measures output quality beyond exact string matching. | Pass/fail per scenario with similarity threshold. Scope-clean and tool-error-free sub-scores. |
| Coding Compare | Code generation quality across 16 fixtures. Measures correctness, style adherence, and edge-case handling in generated code. | Average score 0-1 across all fixtures. Pass threshold at 0.75. |
| Agentic Barrage | Multi-step agentic task execution under pressure. 10 scenarios with chained tool calls, state management, and error recovery. | Average score 0-1. Pass threshold at 0.70. Measures end-to-end task completion. |
| Speed | Prompt processing and token prediction throughput on the target hardware. Peak VRAM and RAM usage under full context load. | Tokens per second for prompt and predict phases. GiB for peak memory usage. |
Scoring Rules
Primary sort: transcript replay partial score. Secondary: sim compare pass rate. Tertiary: coding compare average score. Agentic barrage is a tiebreaker. Speed is reported for hardware planning but does not affect ranking.
Archived models no longer receive new benchmark runs. Historical results are preserved for reference and longitudinal comparison.
Elimination Rationale
Ornith-27B was retired from disk after Qwen3.6-27B showed equivalent or better scores across all suites with a smaller VRAM footprint. Gemma-4-31B variants were archived from a prior evaluation round — they remain useful for quantization comparison but were not competitive on agentic workloads.
Hardware
All benchmarks run on a single host with Vulkan GPU acceleration via llama.cpp. Main chat uses MTP (Multi-Token Prediction) draft mode with n=3. Context window is 128k tokens for chat models, 8k for embedding/reranker.
Raw Data
All benchmark results are stored as JSON in benchmarks/summaries/. Each suite produces a summary.json with per-candidate scores and a published.json with the public-facing label. Full run manifests and per-fixture output are in benchmarks/model_eval/results/.