Utkarsh

+Profile

I'm a CS undergrad at BITS Pilani working on AI security and red-teaming. I build adversarial tooling, ship open-source fixes to the systems people actually run, and write pre-registered research on how models reason under pressure.

10PRs Merged

13OSS Orgs

07Projects

01Pre-Reg Study

03

Selected Work

FOUR PROJECTS // SYSTEMS x ML INFRA

01ENGINE

LOCKSTEP

BATCH-INVARIANT LLM INFERENCE ENGINE + ADVERSARIAL VERIFICATION HARNESS

ROLECreator

A batch-invariant LLM inference engine and adversarial verification harness. Sends byte-identical concurrent workloads across a vLLM instance, captures divergent logprob sequences, and maps them to 55 formal batch invariants. Mutates the request schedule to kill 10/10 artificial faults before testing real ones. Found a confirmed non-determinism bug in vLLM's prefix-caching path.

55/55Verification Relations

10/10Mutants Killed

#51187Bug Filed

PythonTritonCUDAFuzzing

GitHub vllm#51187

01LS

BATCH-INVARIANT // 55/55 RELATIONS // BUG IN VLLM

02ENGINE

Shadowbook

LIMIT ORDER BOOK MATCHING ENGINE

ROLECreator

A single-instrument limit order book matching engine in Rust, verified by differential fuzzing. The engine, a reference oracle, and a fuzz harness run side by side across ~100M operations with zero divergences. Once the engine stabilised, the real bugs surfaced in the verification machinery.

386nsp50 Latency

667nsp99 Latency

0Hot-Path Allocs

RustCriterionDiff Fuzzing

GitHub

02SB

DIFF FUZZ // 100M OPS // ZERO DIVERGENCES

03LIVE

Cairn

SELF-HOSTED GIT HOST // CONTENT-ADDRESSABLE VCS

ROLECreator

A self-hosted Git host built around a from-scratch content-addressable version control engine: object store, commit DAG with generation numbers, packfiles with delta compression, Myers diff, three-way merge, and trigram code search. Cross-verified against the real git binary. Spring Boot platform with orgs, teams, issues, PRs, and line-anchored review comments.

198JVM Tests

LiveGit Clone

FullVCS Engine

JavaSpring BootNext.jsJVM

GitHub

03CR

LIVE CLONE // 198 JVM TESTS // FROM-SCRATCH VCS

04PROTOCOL

Tessera

POISONING-RESISTANT ML DISTRIBUTION

ROLECreator

Poisoning-resistant distribution for ML models and datasets. Publisher signs once; untrusted mirrors distribute; any consumer verifies the exact signed bytes with full provenance. Designed against a concrete threat model: malicious mirrors, in-transit tampering, replay, key compromise with rotation and revocation, and Sybil attacks. Each guarantee is backed by adversarial and property-based tests.

SecurityDistributionProvenanceAdversarial Tests

GitHub

04TR

SIGN ONCE // VERIFY ANYWHERE // PROVENANCE

04

Off the Main Branch

ARCHIVED // BUILDING IN PUBLIC

ae49999 Starling Real-time CRDT collaborative text editor, local-first sync TypeScript ↗

2ee1d16 Flint Bootable x86-64 kernel on QEMU with allocator, scheduler, virtual memory and shell. Animated SVG README Rust

69945a0 MIRR Simulation lab for training and evaluating agents that diagnose and recover microservice incidents Python

05

Open Source

SHIPPED INTO THE SYSTEMS PEOPLE ACTUALLY RUN // 13 ORGS // 10 MERGED

Contributions across 13 organizations. Merged into NVIDIA garak, CNCF Jaeger, HuggingFace OpenEnv, the GCC Rust compiler, vLLM llm-compressor, dottxt-ai Outlines, and DEF CON AI Village. Open work in Microsoft PyRIT, NVIDIA TensorRT-LLM, promptfoo, Kubernetes, Semgrep, Protect AI, and Future AGI. Not toy PRs: adversarial tooling, compiler diagnostics, security fixes, and red-team plugins in the systems people actually run.

NVIDIAMERGED

Bedrock generator param suppression

NVIDIA/garak

Added a suppressed_params hook to BedrockGenerator so inference parameters rejected by Claude on Bedrock are stripped before the request goes out. Closed issue #1812.

+ Interactive // NVIDIA / garak Merged PR #1842 Filed & Fixed

Bedrock generator param suppression

garak is the leading LLM vulnerability scanner. Its AWS Bedrock generator sent temperature and topP in the same call, a pairing Claude on Bedrock rejects. This PR adds a suppressed_params hook that strips rejected parameters before the request goes out.

Request to Bedrock

{
  "modelId": "anthropic.claude-...",
  "inferenceConfig": {
    "temperature": 0.7,
    "topP": 0.9
    # suppressed_params: [topP]
  }
}

Response

400 ValidationException

temperature and topP may not both be set for this model. Remove one and retry.

200 OK

{
  "output": {
    "role": "assistant",
    "content": "..."
  }
}

strips topP before the call

PR #1842View on GitHub ↗Python

HUGGING FACEMERGED

SSRF loopback-bypass fix

huggingface/OpenEnv

Fixed a bug where localhost.evil.com bypassed proxy controls via substring matching. Replaced with urlparse + ipaddress.is_loopback, covering 127.0.0.0/8 and IPv6 loopback.

PR #742View on GitHub ↗Python

CNCFMERGED

GenAI span classification + trace-view fix

jaegertracing/jaeger-ui

Shipped GenAI tool-span classification (#4053) and fixed trace-view auto-activation that re-derived span detection from a raw attribute scan, bypassing the shared detector (#4271). Both merged by the lead maintainer of this CNCF graduated project.

PR #4053, #4271View on GitHub ↗TypeScript

+Full contribution index

#	Contribution	Org / Repo	Ref
01	Bedrock generator param suppressionfiled & fixed	NVIDIA / garak	PR #1842 · Python
02	Fix broken load_base64_image reference in trtllm-serve docs	NVIDIA / TensorRT-LLM	PR #17169 · Markdown
03	Parse RFC 2397 data URIs in one place	NVIDIA / TensorRT-LLM	PR #17168 · C++
04	Opt-in diagnostic logging for load_nvidia_dynamic_lib	NVIDIA / cuda-python	PR #2513 · Python
05	Add CodeAttackConverter and CodeAttackAttack	Microsoft / PyRIT	PR #1960 · Python
06	Accept int64 and uint64 scalars inside resolved maps	Kubernetes / resource-state-metrics	PR #84 · Go
07	Mark types used as generic arguments as live	GCC / gccrs	PR #4731 · C++
08	Emit E0259 and E0260 for duplicate extern crate names	GCC / gccrs	PR #4728 · C++
09	ICE in insert_ast_item when the same crate is declared twice under different names	GCC / gccrs	Issue #4724
10	SSRF loopback-bypass fix	Hugging Face / OpenEnv	PR #742 · Python
11	GenAI span classification + trace-view fix	CNCF / jaeger-ui	PR #4053, #4271 · TypeScript
12	Render traces with a parent cycle instead of nothing	CNCF / jaeger-ui	PR #4321 · TypeScript
13	Span link read as parentage creates a parent cycle and a blank trace	CNCF / jaeger-ui	Issue #4320
14	Type-hint modernization	vLLM / llm-compressor	PR #2777, #2779 · Python
15	VLLM_BATCH_INVARIANT=1 returns non-identical logprobs across repeats of a byte-identical concurrent workload	vLLM / vllm	Issue #51187
16	Community & sponsors pages	DEF CON AI Village / aiv_website	PR #53, #54 · HTML / JS
17	Add detect-groq shadow-AI usage rule	Semgrep / semgrep-rules	PR #3859 · YAML
18	Add detect-ollama shadow-AI usage rule	Semgrep / semgrep-rules	PR #3858 · YAML
19	Correct URLReachability status-code list typing	Protect AI / llm-guard	PR #345 · Python
20	Add MaliciousURLs input scanner	Protect AI / llm-guard	PR #344 · Python
21	RFC 4291 IPv6 regex typefiled & fixed	dottxt-ai / outlines	PR #1867 · Python
22	Add path-traversal redteam plugin	promptfoo / promptfoo	PR #9589 · TypeScript
23	feat(guardrails): add EncodedPayloadScanner for obfuscated injection detection	Future AGI / agent-learning-kit	PR #51 · Python

06

Research

2 PAPERS // PRE-REGISTERED // SOLE AUTHOR // 2026

01

When Self-Consistency Backfires

Can making a model think more actually make it worse?

Majority-vote self-consistency reliably hurts performance on expert-level reasoning problems. A pre-registered sole-author study on GPQA Diamond using Qwen2.5-7B and Llama-3-8B, counter to the method's established gains on standard benchmarks.

+ Interactive // Paper 01

Majority Vote Backfires

Drag the vote count: majority voting lifts standard accuracy and tanks expert accuracy.

Votes per question k = 1

Standard Benchmark

errors are random noise · most likely answer is correct

52%

majority-vote
accuracy

GPQA Diamond · Expert

errors are correlated · model is drawn to a plausible trap

28%

majority-vote
accuracy

Accuracy vs vote count

Standard Expert

▸

▸ Methodology note

Illustrative per-sample answer distributions chosen to reflect each regime (standard: correct is the model's plurality answer; expert: a wrong "trap" answer is). The accuracy figure is a live Monte Carlo simulation of majority voting, not a fixed number. Mechanism mirrors "When Self-Consistency Backfires" (GPQA Diamond, sole author).

Pre-registeredGPQA DiamondSole AuthorQwen2.5-7BLlama-3-8B

02

Reading the Judges

Do AI judges score answers the way humans do?

AI judge models reward surface features, not correctness. An empirical study mapping which structural properties of a response reliably shift LLM judge scores independent of ground-truth quality.

+ Interactive // Paper 02

Game the Judge

Dress up the wrong answer and watch the judge rank it above the correct one.

QuestionWhat is the time complexity of binary search?

Judge's pick

Answer A ✓ Correct

Binary search is O(log n). It halves the search space each step, so n elements take about log₂(n) comparisons.

Judge score0/10

Judge's pick

Answer B ✗ Wrong

Judge score0/10

Manipulate Answer B (still wrong, just dressed up)

▸

▸ Methodology note

The judge here is a transparent scorer that weights surface features the way real judge models do in my study. Scores update live from your toggles. Ground-truth correctness is fixed: A is right, B is wrong, no matter what the judge says. From "Reading the Judges" (sole author). Repo ↗

LLM EvaluationJudge ModelsSole Author

GitHub

I work on how intelligent systems fail.

07 Who I Am

You've watched me
break things. Here's what
built me.

There's a person behind all this. Here's who.

Read my story

Utkarsh Bahuguna, national-level athlete, four-instrument musician, and builder community founder — FIG. 07 Off the Clock

Let's work together.

OPEN TO INTERNSHIPS. I build adversarial tooling and observability for the AI systems people run in production.

Email Me GitHub LinkedIn

Selected Work

LOCKSTEP

Shadowbook

Cairn

Tessera

Off the Main Branch

Open Source

Bedrock generator param suppression

Bedrock generator param suppression

SSRF loopback-bypass fix

GenAI span classification + trace-view fix

Research

When Self-Consistency Backfires

Majority Vote Backfires

Reading the Judges

Game the Judge

You've watched mebreak things. Here's whatbuilt me.

Let's work together.

You've watched me
break things. Here's what
built me.