Intelligence, Measured.

Build a better AI system for your workload. Better intelligence. Faster responses. Lower costs.

RadianVector measures and optimizes entire AI systems - not just models. We evaluate intelligence, latency, token efficiency, throughput, memory and cost.

See the number of knobs we have identified.

Powered by proprietary optimization technology and original research, RadianVector systematically explores various possibilities to identify and validate the right setup for your workloads - with research-grade rigor.

We test and optimize offline, without requiring sensitive production data or live traces, and without slowing down your production pipeline.

Share only what you choose - not your entire production traffic or live traces.

Frontier Models Open Models
your workload your workload · model APIs your model · hosted GPUs behavior latency cost quality throughput GPU memory Intelligence, Measured AI MODELS AND AGENTS Frontier Models Open Models FRONTIER MODELS Claude · GPT · Gemini · Grok support → use model A claims → use model B regulatory → AI drafts, human checks OPEN MODELS vLLM · SGLang · TensorRT-LLM vllm · fp8 weights fp8 KV cache max_num_seqs 32 illustrative · hover either side

Keep your production traffic private

Teams changing their workflows

Deploying AI is not
deploying software.

Box CEO Aaron Levie put it plainly: changing workflows for AI takes far more work than anyone expects. Workflows need re-engineering for agents, evaluations need to be built and maintained, and the whole system needs updating every time a new model arrives.

If your organization is in the middle of that change, talk to us. Measuring what works, and checking it still works after every model update, is what RadianVector tools are built for.

LinkedIn post by Aaron Levie, CEO of Box: There's a huge opportunity right now in being the deployment layer for AI into the economy. The amount of work it takes to change out workflows in enterprises tends to be far greater than anyone realizes or would prefer.
Aaron Levie, CEO of Box, on LinkedIn ↗
Post on X by Brian Armstrong, CEO of Coinbase: We were able to fine tune an open weight model (Qwen3.5-9B) on proprietary internal data at Coinbase, and it out performed frontier models, at much lower cost and speed, for a specialized task (fraud prevention in this case). I think we'll see more of this pattern inside companies. The frontier models are amazing generalists. But increasingly you'll see specialist models out perform on more narrow tasks, either working on teams, or by themselves. The attached chart shows the fine-tuned model improving fraud detection over each comparison model, including an Opus 4.5 decision agent.
Brian Armstrong, CEO of Coinbase, on X ↗
Teams running their own models

Open models can win.
When they are tuned well.

Coinbase fine-tuned Qwen3.5-9B, an open-weight model, on its own data. For fraud prevention it outperformed frontier models at much lower cost.

Running your own model is where RadianVector tools go furthest. A hosted API exposes a few settings; your own stack exposes every one of them. We measure which combination delivers the answers, speed and cost you need, on your hardware and your workload.

  • vLLM · SGLang · TensorRT-LLM
  • Quantization
  • KV-cache precision
  • Speculative decoding
  • Batching & concurrency
  • KV-cache offload & preemption
  • Temperature · top-k · top-p
  • Reasoning budget

See every setting we tune →   Our Qwen benchmark →

Original RadianVector results

Frontier Model analysis.

We ran models from Anthropic, OpenAI and xAI through each vendor’s own command-line tool, on questions we generate and check by code. Ten findings from October 2026. Some of them:

Findings 02 and 07

Max reasoning is not always the right answer.

Opus 5.5 High2/233,411 and 36,387 tokens
VS
Opus 5.5 Max0/2139,699 and 514,985 tokens

Right answers on two 300-step calculations

GPT‑5.5 Low7/14
VS
GPT‑5.5 High12/14

Right answers on 14 level-10 questions

On the most complex questions, high effort solved them; max effort over-analysed and ran out of room.

At max, Opus spent 503 to 630 tokens a step, so 300 steps need more than one reply holds (128,000).

What it means for you. Do not default to the highest setting. Test effort settings on your own questions. Read finding 2 → Read finding 7 →

Tested: Opus 5.5 max: Claude Code 2.1.292 to 2.1.301, October 8–15; high: Claude Code 2.1.301, October 15 · GPT-5.5: Codex 0.154.0, October 4, 2026

Finding 01

At the output limit, each retry started from the beginning

514,985 output tokens on one question, and no answer ($10.33 at API list price)

At the 128,000-token limit, Claude Code asked Opus to continue up to three times, without the reasoning so far. Each try started over: four replies, no answer.

What it means for you. When a question needs more than one reply can hold, retrying does not help, because each try starts over. Split the work into smaller steps, or try a lower effort setting first (finding 2). Read finding 1 →

Tested: Claude Opus 5.5, max effort · Claude Code 2.1.301 · October 15, 2026 · the API was not tested

Finding 04

A code tool did better than GPT‑6 Astra’s top reasoning setting alone

GPT‑6 Astra + codeRight494 tokens
VS
GPT‑6 Astra, no toolWrong47,931 tokens

One 700-step calculation, at xhigh, its highest setting

With Codex’s shell off, Astra still ran code through code mode; the calls show in the session log, not the event stream. On 28 hard questions: 28 right with the tool, 26 without.

What it means for you. For exact work, a code tool can do more than extra reasoning: on these 28 questions it gave more right answers for fewer tokens. To know when it was used, read the session log as well as the event stream. Read finding 4 →

Tested: GPT-6 Astra, xhigh effort · Codex 0.154.0 (codex exec --json) · October 5, 2026

Read all ten findings → See the results tables →

Original RadianVector results

Open Model analysis.

Measured answer quality, request timings and serving behavior, including a tested vLLM fix.

Answer quality

Open models, with thinking on.

Results at levels 5 and 6 of our difficulty ladder, on the RTX 4090 GPU family.

Swipe to see all columns.

Thinking on · levels 5 and 6 · 28 questions per level
ModelLevel 5Level 6
gpt-oss-20b, high effort25/2822/28
Qwen3.8-27B, 4-bit21/2821/28
Qwen3-8B, 4-bit18/2810/28
GLM-4.7-Flash, 4-bit14/286/28

Explore the task families and difficulty levels →

Response time

What happens as requests increase.

The tool report separates time spent waiting from time spent generating.

Swipe to see all columns.

27B model · vLLM 0.28 · no speculative decoding · all times in seconds
Requests
open		RunningWait for first token
median		Time to completion
mean		Time generating
median
110.054.84.8
220.135.35.2
440.115.55.4
880.115.65.5
16103.18.85.8
321011.816.65.8

At 8 open requests, mean completion time was 5.6 s. At 32, it was 16.6 s, while median generation time changed only from 5.5 to 5.8 s. Only 10 of the 32 requests were running; the others were waiting in line.

See all timing measurements →
Benchmark controls

Same model, three inference engines.

RadianVector’s inferlint measured what each server actually ran under the same benchmark load.

Swipe to see all columns.

Qwen3-8B BF16 · RTX 4090 GPU family · 16,384-token context · at most 32 running · same load from vllm bench serve
MeasurementvLLM 0.30SGLang 0.5.20TensorRT-LLM 1.3.0rc29
KV pool32,336 tokens
47,888 on a second start		32,096 tokens37,760 tokens
Most running at once322932
Output tokens per second1,2747991,389

The benchmark path matters. With vllm bench serve sending the same load to all three engines, SGLang ran up to 29 requests at once and produced 799 output tokens/s. With SGLang’s own client using its native /generate endpoint, the same server ran all 32 requests and produced 1,337–1,342 output tokens/s.

See the full engine comparison and run details →
Answer correctness

A vLLM bug, before and after the fix.

MTP speculative decoding repeated tokens under load. We traced the cause to the model runner, patched it and reran the tests.

Swipe to see all columns.

Qwen3.8-27B W4A16 · RTX 4090 GPU family · WSL2 · V1 model runner · 385 replies per run · 8 concurrent requests · async scheduling and prefix caching on
ConfigurationDifferent replies
before → after		Repeated-token replies
before → after
vLLM 0.28 · MTP-160 → 1129 → 0
vLLM 0.28 · MTP-377 → 626 → 0
vLLM 0.30 · MTP-139 → 813 → 0

Repeated-token replies fell to zero in these patched runs. “Different replies” counts token-for-token differences from the no-speculation run; it is not a correctness score.

Read the fix and full results →
Two paths

Offline AI model evaluation. Hosted and open models.

Offline evaluation compares AI configurations outside your live production pipeline. RadianVector tools support hosted model APIs and self-hosted open models, so you can measure answer quality, response time and cost before deploying a change.

Frontier Models

Compare hosted models and API settings.

Which model and prompt combination meets your requirements? Where does reasoning effort help? What changes when the output format or supplied context changes?

For engineering teams improving hosted-model applications, retrieval workflows and agents.

  • questions
  • reasoning effort
  • prompt sensitivity
  • tool use
  • severe errors
  • latency
  • cost
  • human review
Explore hosted-model evaluation
Open Models

Measure answers and the serving stack.

Where do requests queue? Does quantization change the answers? How do cache settings and concurrency affect latency, throughput and GPU use?

For teams serving open-weight models with vLLM, SGLang, TensorRT-LLM or llama.cpp on their own infrastructure.

  • model
  • runtime
  • quantization
  • KV cache
  • concurrency
  • decoding
  • throughput
  • latency
  • GPU utilization
  • cost
Explore open-model evaluation
What RadianVector measures

See beneath the headline score.

RadianVector compares performance and behavior on the same workload. See which tasks each configuration completes, where it fails and how much time and compute it uses.

Explore the engineering behind the measurements →
Workload-specific evaluation

Which tasks does the system get right?

Compare task results against your requirements. See successful completions, important errors and the tasks each configuration struggles with.

Controlled runs & effort analysis

Where does extra computation help?

Compare models, prompts and supported reasoning settings on the same questions. Measure where a change improves answers and what it costs in time and money.

inferlint · Open source

What happened inside the serving stack?

Use inferlint to inspect request queues, memory use and concurrency, and understand what your inference server actually did.

Inspect inferlint and an example report →
Regression & change analysis

Did the change improve the system?

Test before and after a change. See what improved, what regressed and how response time and cost shifted.

Illustrative study design

One document workflow. 1,000 representative questions.

For a workflow that extracts facts from research documents, a study could screen 10 model and reasoning-level settings with 5 prompt variations, then test output formats and sampling settings on promising candidates.

Baseline

Measure the current setup on representative questions.

Screen

Narrow the search using the quality and cost requirements.

Validate

Test the finalists on held-out cases and repeat runs to check stability.

Deliver

Provide configurations, measured trade-offs and failure examples.

The search is staged; the final validation cases stay separate from the cases used to select candidates. This is an illustrative design; published results are linked above. When a model, runtime or GPU changes, rerun the same tests to check the setup still holds.

Explore offline evaluation for hosted models →
Start a conversation

Want to understand your AI better?
Let’s measure it.

You don’t need to know what to ask. Tell us how your team works and how you use AI today, and we will work out with you what is worth measuring.

[email protected]