← mohithgajjela.com
SYSTEM / 07
MOHITH GAJJELA
RECORDED DEMO
a real session, replayed
SYSTEM 07 / SCANSPEAK

Ask a mouse CT scan a question. A 3-billion-parameter model on a laptop drives the viewer.

Biologists want a number out of a scan, not a picture: how big is the tumor, how much did it grow. ScanSpeak turns a plain-English request into actions in a 3D viewer. The model only picks the actions, and the tools produce every number, so it can't make up a measurement. Then I measured how to make small local models reliable at this.

REPLAY THE SESSIONREAD THE CODE ON GITHUB
01THE REPLAY · 6 REQUESTS · QWEN2.5 3B · NATIVE TOOL CALLING + REPAIR

Step through a real session on mouse M37, one the tumor model never trained on. Each step shows the exact tool calls the model chose, the reply built from the tools' output, and what the viewer looked like right after.

RECORDED DEMO

This is a recording of the real app running on my laptop, with screenshots of each viewer state. The live agent needs Ollama and PyTorch, so it runs locally from GitHub rather than in your browser.

RUN IT YOURSELF
ScanSpeak viewer state
M37 has one tumor, which grows from 382 to 925 mm³ by the experts' outline. The other red patches in the 3D view are the segmenter's false positives, and they inflate the day-0 volume. The agent layer is the point here. A better tumor model drops in without other changes.
02THE BENCHMARK · EVERY OUTPUT, BROWSABLE

Pick a model and a decoder.
See exactly where it breaks.

96 requests a preclinical imaging scientist might type, run through 5 local models and 5 ways of getting structured output. These are the real recorded outputs, scored by replaying both the model's calls and the correct calls on a simulated viewer, then comparing the final state and every number reported.

Set
Model
Decoding
Why requests fail
Click a request to compare the model's calls with the correct ones.
03RESULTS · 72 MAIN + 24 HELD-OUT REQUESTS

Forcing valid JSON
made the answers worse.

Grammar-constrained decoding guarantees every call is valid, and it did: zero invalid calls. It still came out about 11 points behind the models' built-in tool calling, because when the output is a JSON array, models pad it with actions nobody asked for. The built-in format's own failures were mostly type slips, like "False" in quotes. A 30-line schema repair fixes those with no model involved.

REQUESTS HANDLED CORRECTLY · 72 MAIN REQUESTS

One run at temperature 0. With 72 and 24 requests, differences of a few points between models aren't meaningful. The schema repair and a "minimal calls" prompt rule were designed after looking at failures, so both were checked on the 24 held-out requests written beforehand. The repair held up. The prompt rule didn't, and it's reported in the repo as a negative result.

04HOW IT WORKS

The model picks actions.
The tools own the numbers.

01

11 tools

load_scan, segment, measure, compare_timepoints, set_window, set_layout, focus_on… Small enough for a 1.5B model to read the whole list in its prompt.

02

A local model plans

Ollama gets the tool list plus the viewer state, like which scan is open and what's already segmented. It returns tool calls and nothing else.

03

Repair, then validate

Fix types the schema declares, like strings to booleans, numbers or enum spellings. Anything still invalid is rejected, not guessed at.

04

Tools execute

A 3D U-Net segments the tumor, voxel counts times real voxel size give volumes, and the reply is built only from those outputs. The same backend drives NiiVue in a browser or 3D Slicer.

05THE REAL THING · THIS PAGE IS A REPLAY

This page replays it.
The repo runs it.

The repo has the agent, the NiiVue web app, a 3D Slicer module, the 96-request benchmark with its scorer and analysis, and the tumor model, which comes from System 06. The scans are the public TumSeg database (CC-BY).

# a local model with tool calling
ollama pull qwen2.5:3b
# the viewer + agent at http://127.0.0.1:8417
python3 -m scanspeak.server
# the benchmark: needs only Ollama, no scans
python3 -u bench/run_bench.py && python3 bench/analyze.py --repair
GITHUB · MOHITH26/SCANSPEAK← SYSTEM 06 TRAINED THE TUMOR MODEL
06LIMITS · STATED PLAINLY

What this page
cannot claim.

  • Small benchmark, one run. 72 main and 24 held-out requests at temperature 0. On held-out, llama3.2:3b actually did better with strict decoding (87.5% vs 79.2%), so the headline gap is a pattern across models, not a law.
  • The 3D Slicer module hasn't run inside real Slicer. It's written and tested against stubbed Slicer APIs. The browser app was tested end to end on real scans.
  • The tumor model is the weak link. Dice 0.72 on held-out mice, against 0.90 for two experts, with visible false positives. The replay shows the day-0 volume inflated because of them.
  • Latency is pessimistic. It was measured while a training job shared the same CPU. Median responses were 0.4 to 1.6 s per request.
Home