Discovery Episodes in AI Evaluation
==Discovery episodes== are the paper’s proposed unit for evaluating AI scientists: instead of grading a model on a single question, you evaluate a whole scientific trajectory over time.[:cite[1]{ln=4}][:cite[2]{ln=...
==Discovery episodes== are the paper’s proposed unit for evaluating AI scientists: instead of grading a model on a single question, you evaluate a whole scientific trajectory over time.[:cite[1]{ln=4}][:cite[2]{ln=3}] The paper defines a discovery episode as a record of “the evolving scientific state, the actions taken, the observations produced, the revisions made and the provenance needed to reproduce and audit the process.”[:cite[1]{ln=5}][:cite[3]{ln=5}] More concretely, an episode is structured as a loop from ==state → action → observation → updated state==.[:cite[4]{ln=3}] The state is the current scientific context.[:cite[4]{ln=4}] The action is an intervention such as a tool call, computation, experiment, or analysis.[:cite[4]{ln=4}] The observation is the evidence that comes back.[:cite[4]{ln=5}] The updated state is how the agent’s beliefs, goals, or constraints change before the next step.[:cite[4]{ln=6}] The provenance layer tracks code, instruments, versions, human interventions, and safety constraints so the process can be reproduced and audited.[:cite[4]{ln=7}] The point is that discovery is not a one shot answer. The paper says discovery involves sequences of decisions under uncertainty.[:cite[1]{ln=3}] Each step changes what is known and shapes the next action, so discovery is a sequential decision making process rather than a single act of answering.[:cite[5]{ln=2}] That is why the authors argue AI should be evaluated through discovery episodes rather than isolated problems.[:cite[1]{ln=4}][:cite[3]{ln=4}] This also means a good evaluation should keep the messy parts that papers and benchmarks often throw away. Full episode bundles preserve null results, anomalous outcomes, and failed actions, because those contain boundary conditions and recovery information.[:cite[4]{ln=9}][:cite[4]{ln=11}] The paper also says useful episode records include rejected hypotheses, tool calls, code edits, simulation inputs, instrument settings, intermediate observations, failed attempts, recovery actions, and the evolving rationale connecting decisions.[:cite[6]{ln=1}][:cite[6]{ln=3}] Finally, discovery episodes matter because evaluation has to distinguish ==genuine discovery from rediscovery==.[:cite[7]{ln=6}][:cite[8]{ln=1}] In plain English: an AI system should not get the same credit for re finding something already known as it gets for producing a truly new, evidence supported result.[:cite[10]{ln=2}][:cite[9]{ln=3}] So the authors are pushing for evaluation that scores both the final artifact and the trajectory that produced it, including whether the system handled failure, updated from evidence, and actually contributed something new.[:cite[10]{ln=2}][:cite[11]{ln=4}]