Agents Learning From Experience
==Agents will learn from their own experiences by generating new training data through continual interaction with their environment==, rather than relying mainly on static human created datasets.[:cite[1]{ln=1}], [...
==Agents will learn from their own experiences by generating new training data through continual interaction with their environment==, rather than relying mainly on static human created datasets.[:cite[1]{ln=1}], [:cite[1]{ln=3}] 1) Continual interaction → a growing “stream of experience” Powerful agents are described as having ==“their own stream of experience that progresses… over a long time scale”==, enabling them to take actions for future goals and continuously adapt over time.[:cite[2]{ln=1}], [:cite[2]{ln=2}] Unlike today’s short, mostly stateless chat episodes, an experiential agent can learn “throughout a lifetime,” carrying information across an ongoing stream of actions/observations and adapting from past experiences to self correct and improve.[:cite[3]{ln=1}], [:cite[3]{ln=5}], [:cite[3]{ln=6}] 2) Reinforcement learning: trial and error adaptation from interaction The paper explicitly grounds “learning from experience” in reinforcement learning: ==an RL algorithm learns to achieve a goal “by trial and error… adapting its behaviour from its experience of interacting with its environment.”==[:cite[2]{ln=5}] It also notes that adaptation can happen via multiple mechanisms, including updating neural network weights or adapting in context, based on feedback from the environment.[:cite[2]{ln=6}] 3) Grounded rewards from environment signals (not human prejudgement) To move beyond ceilings imposed by human raters, the paper argues for ==“grounded rewards: signals that arise from the environment itself.”==[:cite[4]{ln=5}] It illustrates rewards defined as functions of real world measurements and outcomes (e.g., wearables signals or exam results) via a reward function that uses the agent’s interactions with the user and environment and outputs a scalar reward.[:cite[5]{ln=2}], [:cite[5]{ln=4}], [:cite[5]{ln=5}] User feedback can also shape learning by fine tuning and adapting the reward function over time, described as a bi level optimization where user feedback is the top level goal and grounded signals are optimized at the low level.[:cite[6]{ln=1}], [:cite[6]{ln=3}] 4) Learning from execution feedback and tool use outcomes The paper highlights a shift from tool use learned primarily from human examples to tool use built on ==“execution feedback… where the agent actually runs code and observes what happens.”==[:cite[7]{ln=2}], [:cite[7]{ln=3}] This is a concrete mechanism for agents to learn from the consequences of their own actions in digital environments.[:cite[7]{ln=3}], [:cite[7]{ln=6}] 5) Updating world models using experience (and planning with them) One proposed way to ground “thinking” is to build a world model that predicts consequences of actions (including reward), then plan directly in terms of actions and their causal effects.[:cite[8]{ln=1}], [:cite[8]{ln=4}] Crucially, ==as the agent continues interacting with the world, its dynamics model is “continually updated” to correct prediction errors==.[:cite[8]{ln=5}] 6) Learning by experimentation: observe results, update understanding The paper frames grounding as a feedback loop: agents test assumptions against reality, actively engage with the world, collect observational data, and iteratively refine understanding.[:cite[9]{ln=2}], [:cite[9]{ln=4}] In science, it forecasts agents that can accelerate discovery ==“by continuously learning from the results of their own experiments.”==[:cite[10]{ln=4}], [:cite[10]{ln=5}] Summary (one sentence) ==Agents learn from their own experiences by acting and observing in long lived streams, using trial and error reinforcement learning with grounded environment derived rewards, leveraging execution feedback, and continually updating models/understanding from the observed consequences of their actions (including experiments).==[:cite[11]{ln=2}], [:cite[11]{ln=3}], [:cite[11]{ln=4}], [:cite[2]{ln=5}], [:cite[4]{ln=5}], [:cite[7]{ln=3}], [:cite[8]{ln=5}], [:cite[9]{ln=4}], [:cite[10]{ln=5}]