Jiaming Wei
中EN
CV / PDF
FALL 2027 PHD · RESEARCH

Jiaming Wei

Research Engineer — Reliable Agent Evaluation & Adaptive Agentic Systems

What evidence is reliable enough to support an adaptive agent decision?
I study that question across web agents, evaluator reliability, trajectories and post-training.

I study how to evaluate agentic systems when the evidence itself is unreliable: task outcomes vary across reruns, automatic judges make systematic errors, and final success labels can hide failures in trajectories, tools, providers or evaluation infrastructure.

Research & experienceResearch PortfolioResearch CV / PDFGitHubEmail
RESEARCH DIRECTION · FALL 2027

Reliable Agent Evaluation for Adaptive Agentic Systems

What minimum evidence is required before an adaptive agent is justified in changing its behaviour or learning from feedback?

My current direction treats evaluator error and outcome instability as part of the decision problem itself: when should an agent acquire richer perception, switch models, call a verifier, retry, ask a human, spend more test-time compute or abstain?

perception · models · tools · memory · retries · verification · test-time compute

01

Research & Experience

question → evidence → contribution

UCL Computer Science

MSc Artificial Intelligence for Sustainable Development · 2025–2026

Research on representation choice and routing for Web / Computer-Use agents: first measure whether value exists, then ask whether that value is actually learnable.

  • “Routing Is Least Learnable Where It Is Most Valuable” was accepted at the EMNLP 2026 Workshop REALM.
  • The work combines preregistration, controlled representation comparisons, task-level failure analysis, representation probes and a recoverable heterogeneous-compute experiment system.
Web AgentsComputer UseEvaluationRepresentation RoutingResearch Systems

Holistic AI

Research Intern · London · Jun–Sep 2026 · Completed

Treated evaluation as a system rather than a score: from attack generation, target execution and grader auditing through to an executable audit pipeline and live delivery.

  • Owned an end-to-end red-team/evaluation measurement line and separated target behavior, provider-side filtering and grader outcomes instead of collapsing them into one ambiguous binary result.
  • Audited judge/grader FP/FN and failure modes against independent labels, then integrated the measurement logic into the working platform.
  • Later built an NYC Local Law 144 audit pipeline: deterministic Python decision logic, evidence/logging and reviewable reporting, validated on real data and demonstrated live at the end of the internship.
EvaluationRed TeamingJudge ReliabilityAudit SystemsPython

Xi’an Jiaotong University

BEng Automation · 2021–2025

Background in automation and machine learning, later converging on reliable AI evaluation, agent systems and experimental methods.

AutomationMachine Learning
02

Selected Research Systems

5 · public, inspectable evidence
UCL MSc Dissertation / REALM ’26 ACCEPTED

Routing Is Least Learnable Where It Is Most Valuable

Preregistered · OSF3 model families × 6 representationsREALM ’26 accepted

DOM, SoM and vision do contain complementary value. But hindsight value does not imply that a router can predict the right representation before execution. The study separates representation value from routing learnability and turns that negative result into a measurable bound.

  • Controlled comparison of DOM / SoM / vision and phantom representations, preregistered before the main runs.
  • Core result: a routing value–learnability gap — value does not imply learnability.
  • Tracks task-level failures and representation differences, with activation patching / linear probes for diagnosis.
  • Large reproducible research system with heterogeneous-compute orchestration, automatic recovery and 1K+ research tests.

Python · PyTorch · Playwright · VisualWebArena · SGE/HPC

GitHub →

Billed cost across six observation modes / episode

$0.08Classifieds site
$0.064 — $0.073six modes are close in cost

The hard part was not the small cost gap between representations. It was learning when to use which representation — and whether that decision is learnable at all.

PUBLIC RED-TEAM MEASUREMENT STACK / APACHE-2.0

redteam-under-test

Public & auditable103 validated pluginsJudge calibration
redteam-under-test cockpit with risk, attack surface, cost and run trends

The same system inspects the attack surface and the reliability of its own judge.

An attack running is not the same as an attack succeeding.

Puts the target, execution harness, provider behavior, judge and independent gold labels on the same measurement chain. The system asks not only whether the target failed, but whether the evaluator itself is trustworthy.

  • 166 plugins mapped to local generation mechanisms; 103 enabled after end-to-end verification.
  • Target-conditioned attack generation; a zero-egress path can keep generation, execution and judging local.
  • Found an upstream refusal shortcut that missed 8 of 16 breaches on one probe.

JavaScript · Node · Postgres · TanStack Start · SST / Lambda

GitHub →AGENT SECURITY / READ-ONLY MCP

Agent Red-Team Lab

Tool-use securityRead-only MCP574 tests

Stress the agent into crossing a boundary, while making sure any real crossing leaves replayable evidence.

The attack side studies replay, model-hop and call budgets; the defensive side exposes a read-only MCP analyzer that inspects tool-call traces without executing real actions.

  • Detects trust-boundary crossing, sensitive-data egress, destructive actions, confused-deputy behavior and missing authorization.
  • Uses a degrade-not-suppress rule: caller claims of approval cannot erase already-observed risk.

Python · MCP / JSON-RPC · Agent Security

GitHub →
One method / measurement chain

Before trusting a score, I check four layers

model → system → judge → evidence
01→

Target / Agent

What did the model actually do: succeed, refuse, or cross the boundary?

behavior · task success
02→

Execution / Harness

Did the model cause the result, or did observation, tools, provider filters, scaffolding or environment change it?

trajectory · environment
03→

Judge / Grader

Can the evaluator itself be trusted? Rubrics, thresholds, shortcuts and FP/FN can change the conclusion.

FP / FN · calibration
04

Gold / Evidence

What ultimately anchors the claim: independent labels, statistical tests, provenance and replayable evidence?

labels · provenance

Web-Agent routing, Holistic, redteam-under-test, FinQA and the Model Observatory look like different projects, but they ask the same question: what does this result mean, and can the measurement chain that produced it be trusted?

POST-TRAINING ATTRIBUTION / CONTROLLED MATRIX

FinQA

Qwen3 0.6B→14B5×4 controlled matrixSFT / GRPO

A model gained twenty points. That still does not tell you why.

Controls protocol, answer-format SFT, explicit reasoning and RLVR separately to attribute score changes to capability, protocol alignment or reward exploitation.

  • Switching back to the correct native chat protocol alone recovered roughly +20–30 points for some larger base models.
  • 4B answer-only SFT showed no significant net gain; paired McNemar turned an apparent improvement into a testable claim.
  • GRPO/RLVR exposed clear reward exploitation while task accuracy fell.

Python · PyTorch · Qwen3 · TRL · LoRA / SFT / GRPO

GitHub →BENCHMARK PROVENANCE / LIVE

AI Model Observatory

Live data productPer-observation provenanceAutomated drift checks
AI Model Observatory ranking and comparison interface

Live counts stay inside the live product; the portfolio no longer hand-copies volatile observation totals.

The last thing I wanted was another universal score with unclear semantics.

Keeps source, harness and benchmark semantics distinct. Missing evidence stays N/A, agent-system results are not relabelled as pure model ability, and live upstream boards are re-read with automated drift checks.

  • Each observation retains source, version, date, harness, reasoning effort and tool setting.
  • Separates capability, agent systems, coding systems, human preference, speed and price.

TypeScript · Next.js · Data Provenance · EdgeOne Pages

Open observatory →
03

Research Evidence

paper · portfolio · poster · repo
ACCEPTED · EMNLP 2026 WORKSHOP REALM

REALM ’26 paper

Routing Is Least Learnable Where It Is Most Valuable: Bounds on Representation Routing for Web Agents. Accepted at the REALM workshop; not described here as an EMNLP main-conference acceptance or as published.

OpenReview ↗RECRUITER-FACING · WEB AGENT RESEARCH

4-page Research Portfolio

A visual recruiter-facing research portfolio that compresses the question, method, main result and systems evidence into a fast read.

Open 4-page PDF ↗UCL × HOLISTIC AI · 16 SEP 2026

Research Poster & Showcase

The final research poster was presented at the UCL × Holistic AI showcase. This public site exposes only public-safe research context, never private company material or client data.

Open showcase →UCL MSC · PREREGISTERED · PUBLIC REPO

Dissertation & Reproducibility

The full research repository contains preregistration, experiment orchestration, analysis, failure evidence and paper materials. Long-form evidence stays there; the homepage keeps the scan path short.

Research repo ↗
2026-09-20-post-holisticpublic facts verified · 2026-09-21
04

Methods & Engineering

claims backed by artifacts
Evaluation / Research

LLM & Agent evaluation · benchmark design · red teaming · judge / grader calibration · failure taxonomy · paired tests / bootstrap · preregistration

Agents

Web / Computer-Use agents · VisualWebArena · DOM / SoM / vision representations · tool use · MCP · trajectory analysis · recovery

Model / Post-training

PyTorch · Hugging Face · TRL · LoRA / SFT · GRPO / RLVR analysis · protocol attribution · reward-failure analysis

Evaluation Systems

Python · TypeScript · Postgres · Playwright · Docker · GitHub Actions · Linux · AWS / SST · structured logging · data provenance · reproducible harnesses

05

Personal Lab

selected side work
constrained-agent-runner2026An agent control plane spanning ChatGPT Web, durable GitHub transport and a constrained runner: capable of acting while keeping hard capability boundaries.coding-agent-guardrails2026Rules, hooks and skills that keep Claude Code / Codex reliable: git guardrails, proving a check actually ran before claiming a verdict, session handover and cross-model review — each added after a real failure.Autonomous Agent Prediction2026A 60-minute, $2-LLM-budget, CPU-only autonomous ML agent focused on leaderboard stability and guaranteed-valid submissions rather than public-score chasing.
paper-deslop2026A LaTeX-aware academic rewrite pipeline where models propose diffs and deterministic invariant gates protect numbers, citations, equations and terms.
Encode Persona2026Compares natural language, JSON/YAML, semantic labels and opaque-tag persona representations with ablations, cross-model probes and exact provenance.
Spatial Copilot2026Turns first-person narration into a persistent topological map, treating graph topology as truth and geometry as a visual hypothesis.
FitnessOS2026An offline-first SwiftUI fitness system with SQLite as source of truth, HealthKit, deterministic scheduling and a transactional outbox.

I want to make evaluation reliability part of the agent decision problem, not an afterthought.

Current Fall 2027 direction: reliable agent evaluation, evaluator-aware adaptation, trajectory-grounded supervision and resource-aware verification.

Research CV / PDFLinkedIn ↗EmailGitHub ↗