Deep Benchmarks v7: Pushing NextGenPVE to the Limit

We just wrapped up another intensive benchmarking session on the NextGenPVE cluster. Every time a new local model drops, the entire landscape shifts. Our primary goal in the offensive security space is maintaining operational security without sacrificing reasoning capability or speed. That means running these models entirely on local infrastructure.

With the recent additions to the hardware stack, we decided to run the Deep Benchmarks v7 suite across our primary nodes. I want to break down exactly what we are testing, the hardware backing it, and how these models actually perform when the rubber meets the road.

The Hardware: NextGenPVE-RTX and NextGenPVE-NPU

Before diving into the numbers, let us set the stage with the environment specs.

NextGenPVE-RTX (The Heavy Lifter) This node is built on a ZimaCube Pro Creator, punching well above its weight class. It handles the complex reasoning tasks while strictly managing its memory overhead.

• Compute: NVIDIA RTX 2000 Ada Generation (12GB VRAM)
• Memory: 32GB System RAM
• Storage: Fast NVMe arrays for rapid weight loading
• Role: Handling complex reasoning tasks and context-heavy dense models

NextGenPVE-NPU (The Edge Specialist) This is our dedicated neural processing node, built around the Axera AX8850 edge NPU.

• Compute: Dedicated Edge NPU silicon
• Role: Ultra-low latency, high-efficiency inference for smaller specialized models

Benchmarking Methodology

We test speeds across two main phases. The first is the cold boot, measuring the penalty of pulling weights from storage, mapping them into memory, and serving the first token. The second is the optimized hot load, measuring sustained generation speeds once the model is locked into VRAM.

1. Optimized Hot-Load Benchmarks (Current Baseline)

This is the true, optimized speed of the hardware. Tested via the native API with Flash Attention and Q8 KV Caching enabled. The models remain locked in VRAM without spilling into CPU RAM. Speeds reflect one code prompt and three massive security prompts.

Feature Legend

• Dense: Standard monolithic transformer architecture.
• MoE: Mixture of Experts (highly efficient for its parameter size).
• Thinking: Contains advanced chain-of-thought tokens/modes before answering (e.g. Qwen3.6 MoE, DeepSeek R1).
• Tool Calls / MCP: Natively trained to output JSON schema for function calling and capable of driving the Model Context Protocol.
• Agentic: Excels at multi-turn autonomy, self-correction, and system prompting.
• Abliterated / Uncensored: Refusal vectors have been orthogonalized to bypass standard safety guardrails for red-teaming tasks.

2. Unoptimized Cold-Boot Benchmarks (Historical Tracker)

This section tracks the penalty of forcing a cold model load from disk into VRAM along with an uncompressed 16-bit KV Cache. These were tested prior to our hyper-tuning efforts.

3. Axera AX8850 (NPU) Edge Benchmarks

These are compiled natively via Pulsar2 and executed on the zero-copy C++ axllm serve engine directly on the edge silicon.

Analysis and Observations

The hermes-3-llama-3.2-3b model absolutely tears through the benchmarks when hot loaded, hitting upwards of 91 tokens per second. It handles 128k contexts effortlessly on the RTX node and continues to prove its weight in gold.

The 8B and 9B parameter class continues to be a very reliable sweet spot for the NextGenPVE-RTX node. Models like deepseek-r1, Qwythos-9B, and gemma2 are all pushing a comfortable 34 to 35 tokens per second. They provide enough reasoning capability for complex offensive security tasks without bogging down the hardware. We even see the deepseek-r1 model utilizing thinking modes without sacrificing much speed.

When looking at the unoptimized cold boots, you can truly see the penalty of VRAM thrashing. The Hermes models drop from 91 TPS all the way down to a sluggish 15 TPS when they are forced to load from disk on the fly. This perfectly illustrates why keeping models resident in memory is absolutely non-negotiable for responsive red-teaming tasks.

On the edge side, the Axera NPU is proving its worth. Pulling 20 tokens per second on a half-billion parameter Qwen 2.5 model using pure edge silicon is impressive. We are also successfully running multi-billion parameter models like Qwen 2.5 3B at over 7 TPS without waking up the main RTX GPUs.

Under the Hood: llama.cpp Optimization on Debian

A lot of people ask how we pull these numbers, so I did a deep dive into the latest documentation and hardware benchmarking forums specifically for the RTX A2000 (12GB VRAM) running llama.cpp and Ollama.

The short answer is: Yes, we are running the absolute bleeding-edge optimal stack.

Here is exactly how our current flags map to the best practices:

1. OLLAMA_KEEP_ALIVE=-1 The Research: By default, Ollama unloads models after five minutes of inactivity to save VRAM. When a new request comes in, it has to stream 5.3GB from the disk back to the GPU, causing a massive ten-second cold boot penalty. Our Setup: Setting it to -1 permanently locks Qwythos into the A2000's VRAM. Because this is a dedicated AI node, we do not care about sharing VRAM. This is what gives us the instant zero-latency response times.

2. OLLAMA_KV_CACHE_TYPE=q8_0 The Research: When you submit a prompt, the engine stores the Key-Value state of the conversation in VRAM. For 1-Million context models like Qwythos, an unquantized 16-bit KV cache would instantly trigger an Out-Of-Memory (OOM) crash on a 12GB card. Our Setup: Quantizing the KV cache to 8-bit cuts its VRAM footprint entirely in half with zero perceptible loss in generation quality, allowing the 9B model to comfortably fit alongside a massive context window.

3. OLLAMA_FLASH_ATTENTION=1 The Research: Traditional attention mechanisms scale quadratically in memory as the context grows. Flash Attention rewrites the math to be infinitely more memory efficient and hardware-aware. Our Setup: We have this explicitly flipped on, which is the secret sauce behind maintaining 34+ TPS even when the prompt gets long.

4. OLLAMA_NUM_PARALLEL=1 The Research: If you allow Ollama to process multiple requests concurrently, it reserves VRAM for each parallel slot, dramatically reducing the maximum model size you can run. Our Setup: We locked it to 1. Since we use LiteLLM for routing requests, this forces the GPU to put 100% of its compute into generating one request as fast as humanly possible before moving to the next one in the queue.

There is not a single frame of performance being left on the table here. We have a perfectly tuned engine.

Looking Ahead: The Cluster Rebuild

These numbers give us a solid baseline, but the work never stops. We are already planning a complete rebuild of the NextGenPVE environment in the coming weeks. The goal is to optimize the storage layer for even faster cold boots and streamline the networking between the LXC containers to reduce any latency overhead.

We will be documenting that rebuild process as it happens. For now, the current infrastructure is holding its own and providing the local, secure intelligence needed for our daily operations.