Project Story
Mr. Krabs
Knowledge Replication, Auditing, Backtesting & Stress-testing
An AI multi-agent system that replicates quantitative finance research papers and tells you whether the alpha is actually tradeable.
What it does
Mr. Krabs is a multi-agent AI system that takes a quantitative finance research paper as input (a PDF, no preprocessing) and produces a structured implementability verdict in roughly ten minutes. It answers the question every portfolio manager actually asks when reading a paper: is the alpha real, or is it hand-waving?
The architectural principle is simple. LLMs extract and judge. Code computes and verifies. Every number in the final report traces back to either a tested function call or a verified PDF quote. No invented numbers, no hallucinated citations. It is the rigorous companion to PAT (Pocket Analyst Tool), the lighter-weight internal analyst tool: PAT reads papers. Mr. Krabs replicates them.
The Bridgewater hackathon
Built over 36 hours at Bridgewater Associates' AI Innovation Hackathon. The most fascinating part was hearing about the work inside Bridgewater's AIA Labs (AI Associate Labs), the team building toward what they internally call "the next artificial investor." Seeing how a real systematic investment firm thinks about that problem was easily the highlight of the trip. They prioritize provenance, reproducibility, and honest data limits, and they distrust LLM-only outputs and opaque models. The mentorship was excellent and the energy in the building was infectious.
Inside the system
Mr. Krabs is not a single prompt. It is nine specialized agents arranged in a directed graph, each with one job and a typed input/output contract. Some are LLMs (Sonnet 4.6 and Haiku 4.5), some are deterministic Python, and the edges between them carry Pydantic v2 objects, not free-form strings.
Nine agents in a directed graph. Two feedback loops let the system challenge itself before it ships a verdict: failed checks send A1 back to re-extract, and D2 re-runs the engine with one-parameter mutations.
What each agent does
- A1Methodology Extractor
- Reads the PDF and emits a
ReplicationSpeccovering the signal definition, universe, portfolio construction, rebalance schedule, and headline claim. Every field carries a verbatim supporting quote with a page anchor. - A2Quote Verifier
- Cross-checks every quote against the PDF with a deterministic fuzzy matcher plus a Haiku 4.5 "does this quote support this field?" judgment. Paraphrased or misplaced quotes get flagged and sent back to A1 with feedback.
- A3Adversarial Reviewer
- Forced by schema to emit exactly three criticisms of A1's extraction, so it has to find something even when the extraction looks clean. It is the single biggest source of caught bugs in the system.
- B1, B2Data Mapper and Mapping Verifier
- Translate the spec onto the data backend (DuckDB on local parquet, point-in-time enforced) and classify every blocker honestly:
data_limitationwhen the paper needs pre-1994 fundamentals we don't have,methodology_incompatibilitywhen the engine doesn't support the construction. - CodeEngine
- Pure deterministic Python. Tested, and validated against Ken French's published momentum factor at 0.77 to 0.80 correlation over 288 overlapping months.
- D1, D2Comparator and Diagnostician
- D1 compares the engine's number to the paper's claim. If they diverge, D2 proposes one-parameter mutations (sign flip, lookback change, weighting swap) and re-runs the engine to see which one closes the gap.
- CodeRobustness Battery
- Six families of stress tests: execution lag, transaction costs, subperiod stability, liquidity filters, capacity, and data quality. Each family is wrapped in
_safeso one failure can't kill the rest. - D3Implementability Verdict
- Reads the robustness scorecard and writes one CIO-style verdict:
tradeable,borderline, ornot tradeable at scale, with structuredfragility_signals.
Why multi-agent at all
A single Opus 4.7 prompt could probably extract a methodology, critique it, and produce a verdict. That is the baseline I tested against, and it lost. The three jobs are cognitively different: extraction is faithful transcription, critique is adversarial skepticism, and the verdict is synthesis. In one context window, the model softens its critique to stay coherent with its own extraction, or second-guesses the extraction to please the critique. Separate agents each get their own prompt, model, and typed output. Roles don't share context. They share contracts.
The other reason is debuggability. When a single prompt gets it wrong, you have one knob to turn and no idea which step broke. Here, every intermediate output is on disk: A1's spec, A2's verification report, A3's critique, the engine's run, D2's mutation log. I could pinpoint the broken step in seconds, and that is what made the 8-paper benchmark possible. Every failure was attributable to a specific agent, not a black box.
Typed contracts everywhere
If A1 wrote "the strategy uses a roughly six-month lookback and rebalances monthly with no fundamentals filter," three downstream agents would read that sentence three different ways. A typed ReplicationSpec leaves nothing to interpret:
If a value isn't in the schema, it doesn't exist as far as the pipeline is concerned. The schema also forced a distinction the paper's prose blurs. The first version of A1 attached the same quote to both signal.skip_months (a signal-definition field) and rebalance.execution_lag_days (a trade-timing field), because the paper conflates them. A2 caught it by asking Haiku whether that specific quote supports that specific field, and getting back "no" instead of a vague "sort of."
Forcing exactly three criticisms
Ask an LLM "do you see any issues with this extraction?" and the default answer is "looks good to me." The schema constraint min_length = max_length = 3 on A3's criticisms field removes that escape, and being forced to find three things changes the kind of attention the model pays to the input.
The cost is that A3 occasionally stretches for a third criticism when A1 was genuinely accurate. The benefit is that it finds real problems the other agents miss, including ones we introduced ourselves. On JT 1993, its first criticism was that the spec hard-codes a $10 minimum price with no basis in the paper, whose only inclusion rule is data availability. That floor was a penny-stock filter added during development, and it quietly drops the small, low-priced stocks that drive momentum's strongest returns. Its second criticism caught the spec collapsing the paper's 32-strategy grid into a single J = K = 6 variant. Neither would have surfaced from a reviewer allowed to say "looks good to me."
Choosing the right model for each role
A2 and A3 sound similar but do different work. A2 makes cheap, structured, high-volume judgments: does this verbatim quote support this field, yes, partial, or no? A typical paper has 25 to 40 supporting quotes, and Haiku 4.5 handles them well, where Sonnet would burn budget without improving a binary classification. A3 reasons about whole-paper coherence, spotting internal contradictions, weak inferences, and missing premises, and Sonnet's larger model is measurably better at that. The same split applies to B2's mapping checks (Haiku) versus D2's diagnosis (Sonnet, because explaining a residual gap takes reasoning). Splitting roles by model isn't an optimization. It's an admission that the work isn't the same.
Two-step gating: B1 generates, B2 checks
Mapping a spec onto real data is generative reasoning. Which source provides this universe? How do you handle survivorship bias on a 1994-onwards Yahoo panel? So B1 is a Sonnet 4.6 agent. B2 runs two checks: a deterministic Python check that the proposed source and method actually exist, then a Haiku check that the mapping is reasonable for the paper's claim. That catches both hallucination, where B1 proposes a source that doesn't exist, and under-specification, where it proposes a real source for the wrong field. On LSV 1994, B2's deterministic check caught the coverage gap (defeatbeta_yahoo fundamentals start in 2019, the paper needs 1968) in 191 seconds, and the system halted with a structured blocking_issue instead of substituting fake data.
Why D2 runs experiments instead of guessing
The right answer to "why is your number different from the paper's?" is not "I think the lookback might be wrong." It is an experiment: change the lookback, re-run the engine, and measure how much the gap moves. Mutate one variable, hold everything else fixed, measure the change.
D2 can only propose mutations that are runnable. The LLM proposes hypotheses, and the engine adjudicates them. During development, an early JT 1993 run produced −1.51%/mo against the paper's +0.95%/mo, sign-flipped. D2's first experiment flipped signal.direction from long_high to long_low, and the engine returned +2.64%/mo: the right sign, a 4.15-point swing, and now an overshoot. Its second experiment added a $5 price floor, which cut the magnitude by about 1.2%/mo and pinned the overshoot on microcap stocks and the different sample window. These are observations from a deterministic experiment, not LLM guesses.
Honest hard-blocks beat fluent answers
Most replication tools fail silently. They substitute "approximate" data and return a plausible number that doesn't describe the paper, which is worse than failing, because you can't tell a real replication from a fake one. Mr. Krabs hard-blocks at B2 with structured blocking_issues that name the exact gap. CJL 1996 (1977 to 1993 sample), Asness 1997 (1963 to 1994), and LSV 1994 (1968 to 1989) all halt there with the same diagnosis: they need pre-1994 universe coverage, and defeatbeta_yahoo only has post-1994. Direct Claude, given the same papers, wrote fluent narratives about strategies run on data that doesn't exist.
Per the feedback from our Bridgewater mentors, this was the most important architectural test. The agents were judged not on the wins, but on whether they could refuse to answer.
The engine
The engine is the deterministic backbone. LLMs propose what to compute, and the engine computes it. It takes a typed ReplicationSpec and a data store and returns a BacktestResult, the same result every time for the same inputs.
Six steps, one tested implementation. The engine doesn't ask an LLM whether a number is right. It computes the number.
Why the engine is the heart of the system
Every other agent is replaceable. The extractor could be a different model, the reviewer could be tuned differently, the diagnostician could work from a different prompt. But without the engine, every claim about a strategy is unverified narrative. The engine turns "this paper claims a 6-month momentum strategy earns 95 basis points per month" into "I ran it on real data and observed 1.025%/mo with t = 1.97 over 231 monthly observations." Provenance, reproducibility, and falsifiability all start here, and D1's comparisons and D2's diagnoses are only trustworthy because the engine they wrap is independently verifiable.
Validated against an independent ground truth
The regression test correlates the engine's output with Ken French's published MOM factor, a ground-truth series compiled outside our codebase, on overlapping months:
| Configuration | n | Correlation vs KF MOM | Mean (bps/mo) |
|---|---|---|---|
| JT (6,1,6) decile equal-weighted | 293 | +0.77 | +41 |
| KF (11,1,1) decile equal-weighted | 288 | +0.80 | +27 |
Our universe and survivorship-bias treatment differ from CRSP, so a perfect correlation isn't expected, and because the test runs on overlapping months rather than an in-sample fit, it is independent validation. A correlation around 0.80 across nearly 300 months means the signal computation, the tranche bookkeeping, the execution-lag handling, and the Newey-West standard errors are all correct end to end. If any one of them were wrong, the correlation would drop, and we know because we've seen it drop: with one off-by-one bug in the engine, it was 0.19.
Overlapping tranches and the off-by-one that cost seven months a year
In JT (6,1,6) momentum, the portfolio rebalances monthly but each tranche holds for six months, so at any time you hold six tranches with one-sixth of capital each. The math is simple. The implementation is fragile: when does a tranche exist, how many are active in the first six months, and what happens on rebalance dates?
The original engine got it wrong by one character. pandas.DateOffset(months=N) isn't time-reversible, and the membership predicate in src/engine/tranches.py::active_tranches_in used > where it needed >=. The boundary tranche was excluded on rebalance dates, so 7 of every 12 months of returns went missing, and the correlation with Ken French's factor sat at 0.19 until that one character was fixed. An LLM "calculator" would have produced the wrong number with confidence. The deterministic engine plus the regression test caught it within minutes.
No look-ahead, structurally enforced
Every data-store query requires an as_of_date, and the source drops any data point announced after it. Even if you accidentally request next month's earnings for a formation portfolio, the data store refuses.
No look-ahead becomes a property of the type system, not of vigilance. Survivorship bias travels the same way: every QueryResult wraps a ProvenanceRecord whose fidelity_note flows through the engine into a data_quality_flag on the final BacktestResult. The yellow banner on the Replication tab comes from that flag.
Newey-West because tranches overlap
Overlapping tranches make the long-short returns serially correlated, and ordinary OLS standard errors then inflate the t-statistics. Newey-West with lag = (K − 1) is the literature standard for this construction. Skipping it would overstate significance in the engine, then in D1's comparison against the paper's t-statistic, then in D2's call on whether the gap is meaningful. On JT 1993, it's the difference between t = 1.97 (correct) and roughly t = 4 (overstated by a factor of two).
Determinism unlocks the rest of the pipeline
The same ReplicationSpec and the same data snapshot give a byte-identical BacktestResult. That makes three things sound. D2's mutation experiments are valid, because the only change between two runs is the mutated parameter, so a 4-point shift can be attributed to a sign flip rather than noise. The Ken French regression test is meaningful, because any drift is a real change in behavior. And the disk cache is safe, because the (spec, store_snapshot) tuple is a perfect cache key. None of that would hold if a step in the engine were an LLM call instead of a tested function. "LLMs extract and judge, code computes and verifies" isn't a style choice. It's the load-bearing wall of the whole system.
Benchmark against Opus 4.7
To prove the architecture was doing real work and not just providing fancy LLM scaffolding, I benchmarked the full pipeline against direct Claude Opus 4.7, the strongest single-call baseline available, on 8 canonical asset-pricing papers from 1981 to 1997. The pipeline used Sonnet 4.6, the smaller of the two models. The hypothesis was that structure beats raw model power.
The pipeline won 6 of 8, tied 1, and lost 1, averaging 25.3 / 30 against 20.6 / 30 for direct Claude. Total cost across both systems was $13.50.
Scores out of 30 for all 8 papers, grouped by how far the pipeline got. Each paper is scored 1 to 5 on six axes: methodology, implementability, stress tests, hallucination, adversarial review, and reproducibility. JT 1993 and DBT 1985 were scored as one combined batch.
Coverage and limits
Of the 8 papers, 4 ran end to end and 1 ran with degraded fidelity (Lehmann 1990, a weekly-reversal paper on a monthly engine). The other 3 hard-blocked at the data-fidelity layer because they need pre-1994 fundamentals coverage we don't have, and Mr. Krabs correctly classified them as "data limitation" rather than "paper flawed." The honest hard-blocks mattered as much as the wins. The system never substituted fake data to return a meaningless number.
The product, end to end
The system ships with a FastAPI web interface that walks through the pipeline tab by tab. The screenshots below follow one paper, Jegadeesh-Titman 1993, the foundational momentum paper, from PDF upload to final verdict. The paper claims +0.95% per month, t = 3.07. Famous result. Watch what the system does with it.
01 · INGEST Upload the PDF
The user drops in a PDF, or hits LOAD BRIEFING for the demo. The verdict strip at the top is the headline number you'd ship in a CIO memo: the paper's result and the implementable alpha side by side, with a tradeability pill. Everything below it is the audit trail.
The strip shows the paper's +0.950%/mo (t = 3.07) against +0.958%/mo of implementable alpha after costs (t = 1.84), with a TRADEABLE pill. The rest of the pipeline is what justifies that label.
02 · METHODOLOGY A1 extracts the spec
A1 emits a ReplicationSpec with a plain-English strategy summary, the headline claim and its verbatim quote, the universe, currency, and sample window. Every field carries a page-anchored quote for A2 to verify next.
Headline claim of 0.950%/mo, t = 3.07, Jan 1965 to Dec 1989, backed by the verbatim "6 Buy-sell 0.0084 0.0095..." row from page 7. The rules require an exact PDF substring, with no paraphrase allowed.
03 · VERIFY A2 cross-checks every quote
Every field A1 extracted gets a fuzzy-match status (verified or fuzzy_match) and a Haiku judgment (yes, partial, or no) on whether the quote actually supports it. Anything that fails loops back to A1.
The signal field came back PARTIAL. The quote says stocks are ranked on past returns each month, but not the lookback window or which side goes long: "ascending order" could imply either direction.
04 · REVIEW A3 must find three criticisms
There is no "looks good to me" escape. Criticism one caught a $10 minimum price the paper never mentions, a filter that drops exactly the small, low-priced stocks driving momentum's strongest returns. Criticism two caught the spec collapsing the paper's 32-strategy grid into a single J = K = 6 variant, missing J = 12, K = 3, the paper's most profitable strategy at 1.31%/mo. The system has to argue against itself before it ships an answer.
Three severity-tagged criticisms: missed, oversimplified, and alternative interpretation. For criticism 1, A3 recommends removing the $10 floor and flagging the price filter as an open ambiguity instead.
05 · REPLICATION The engine runs the spec
After B1 and B2 map the spec onto the data, the engine runs. The Replication tab puts the paper's numbers and the engine's side by side, with the data-integrity caveats (survivorship bias, window substitution, anything else that affects an honest comparison) above the table.
The paper reported 0.950%/mo. The engine produced 1.025%/mo on a window-substituted out-of-sample run (1965 to 1989 mapped onto 2007 to 2026, because Yahoo's coverage starts in 1994), a +0.075%/mo gap. The banner is explicit that this is a test on data that didn't exist when the paper was written, not a re-run of the original sample.
05b · REPLICATION Decay charts and the equity curve
The engine also produces signal decay by formation age (how long the signal stays alive after portfolio formation) and the cumulative long-short curve with drawdowns shaded. That answers the trader's questions: how fast does the alpha rot, and what does the worst stretch look like?
The signal is weakest in the first month after formation (0.30%) and steady at about 1.1 to 1.3% for months 2 through 6. The equity curve marks the 2009 momentum crash and stays underwater until about 2021, which is where all of the gains come from.
06 · DIAGNOSIS D2 closes the gap
When paper and replication diverge, D2 proposes one-parameter mutations, runs the engine on each, and reports which one closed the gap. On this run the baseline already matched the paper, a 7.5 bps/mo gap with the right sign, but D2's experiment was still informative. Mutating signal.skip_months from 0 to 1 widened the gap to 28 bps/mo, confirming that the paper's no-skip convention was the right setting. Reverting it restored the match, and D2 exited early.
A gap waterfall where each bar is one engine re-run. The skip-month mutation widened the gap (red), and reverting it restored the match. No LLM is allowed to claim a number that wasn't computed.
07 · STRESS TESTS Six families of frictions
The battery applies realistic frictions to the engine's clean-data result, and implementable alpha gets the headline. For JT 1993: 0.958%/mo at 10 bps round-trip costs (t = 1.84), barely below the 1.025%/mo baseline. It survived 16 of 26 tests, with capacity estimated at about $17.4B at 50 bps of impact. The weak spot is timing: the first half of the sample sign-flips, and the 2009 momentum crash cost 3.16% a month.
Implementable alpha after frictions, the scorecard summary (16 of 26 tests survived), and the fragility signals: the alpha holds up to costs but sign-flips in six subperiods, including the first half of the sample and the 2008 to 2009 crisis.
07b · STRESS TESTS Sensitivity sweeps
Two curves drill into the survivors: signal staleness (does a delay between formation and trading hurt?) and transaction costs. Delay doesn't hurt here. Skipping one to six months actually raised the t-statistic to 2.4 to 2.6, a sign of a structural signal rather than a microstructure one. Costs do: the alpha survives through 25 bps and fails at 50 bps. Every cell in the tables below the curves is one engine re-run.
Direct Claude said "capacity issues kick in around $1 to $5B" with no derivation. Every point on these curves is a real backtest.
The verdict
TRADEABLE, with medium confidence and a big caveat. The alpha survives realistic costs, but the whole record comes from the recent regime. The strategy lost money through the first half of the sample and only worked after about 2016, so the CIO note says to size it as a recent-regime bet, not a 20-year structural alpha, and to demand a crisis-regime hedge before deploying it. Both the pipeline and direct Claude could have been wrong. Only one of them shows its work.
Tech stack
- Language and types Python 3.11, with Pydantic v2 for typed schemas across every cross-component payload
- LLM layer Anthropic API. Sonnet 4.6 for extraction, critique, and diagnosis. Haiku 4.5 for verification. Opus 4.7 for the baseline comparison.
- Backtest engine Custom, with overlapping tranches and Newey-West standard errors, validated against Ken French's published MOM factor (correlation 0.77 to 0.80 over 288 overlapping months)
- Data store DuckDB on local parquet, point-in-time, with a structurally enforced
as_of_dateargument on every query so there is no look-ahead bias - PDF pdfplumber, with a Haiku 4.5 vision OCR fallback for image-only scans
- Web FastAPI plus a hand-rolled vanilla-JS SPA. No build step. No chart library. Every chart is hand-built SVG.
- Tests 238+ across schemas, engine, agents, and integration. Full pytest suite runs in roughly two minutes.
What I learned
This was my deepest project to date in multi-agent system design. The biggest lesson was that LLMs and deterministic code complement each other when you let each do what it does best: LLMs handle unstructured extraction and judgment, and code handles math and verification. Anthropic's adversarial agent pattern (forcing the third agent to find three criticisms) caught real bugs I never would have spotted in code review. And hearing about Bridgewater's AIA Labs work on "the next artificial investor" reframed how I think about good AI tooling. Provenance, reproducibility, and honest limits matter more than headline numbers.
Built with Python, Pydantic, FastAPI, the Anthropic API, and a lot of late nights. Jason Fang, 2026.