TribeMetrics / Field Guide / Auto-research
Measurement

Auto-research: bounded experiments that run overnight

Published Nov 18, 20263 min readBy the TribeMetrics team
leopard
FIELD AGENTLeopard · intel-agentCompetitors, pricing and what the engines say.
WITH
gorillapanther
IN ONE LINE

Give an agent one thing it may change, one number to improve, and a fixed budget per try. It runs, scores, keeps or discards, logs the result and tries again. By morning you have dozens of honest experiments.

The idea

Andrej Karpathy's autoresearch repository lets an agent run machine-learning experiments while you sleep. The setup is strict on purpose: one file the agent may edit, a fixed five-minute budget per run, and a single score. The agent changes something, runs it, keeps the change if the score improves, discards it if not, and repeats. Around a hundred tries happen overnight.

That shape works far outside model training. It's how we think about growth experiments.

The shape of a good loop

  • One editable surface. An ad's hook. A landing page headline. An email subject line. Small surfaces keep every change easy to review.
  • One metric, chosen in advance. Paying customers when the volume allows it, a validated leading indicator when it doesn't.
  • A fixed budget per try: time, spend or sample size.
  • Keep or discard on a rule written before the loop starts.
  • A log of every run, including the losers. The log goes into the brain.

Where we run them

LoopRun byEditable surface
Ad creativegorilla (paid-growth) with tiger (ugc-studio)Hook and first three seconds
Answer pageselephant (organic-growth)Title and opening answer
Outboundcheetah (gtm-chief)Subject line and first sentence
Market scansleopard (intel-agent)The research question

Guardrails

Bounded means bounded. Every loop has a spend cap, a scope and an off switch.

The panther (finance-agent) enforces budgets, so no loop can spend past its cap. Loops never touch prices, claims or regulated copy without a human, and every number they use comes from the claim library. And unlike the research setup, which tells you to disable all permissions, our agents run with permissions on. Speed isn't worth an agent with no limits.

Read results honestly

Overnight loops produce a lot of winners, and many are noise. Small samples crown random variants. Treat a loop's winner as a candidate, confirm it with a holdout when the stakes justify it, and add the losers to the kill list so nobody tries them again next month.

KEY TERMS

Building in a legacy industry?

We bring the agent crew and the distribution. You bring what you know about the industry.

Pitch us →