Hung-yi Lee ML 2025 Lecture 2: AI Agents - From AlphaGo to LLM-Powered Autonomy

Notes from Professor Hung-yi Lee’s (李宏毅) course “Machine Learning in the Era of Generative AI (2025)” at National Taiwan University.

What is an AI Agent?

Professor Lee starts with a disclaimer: everyone has a different definition of “AI agent.” Some insist only robots with physical bodies count. This lecture focuses on a specific definition.

Traditional AI interaction: Human gives explicit command → AI executes exactly that command → Done.

AI Agent interaction: Human gives a goal → AI figures out how to achieve it through multiple steps, adapting to environmental changes along the way.

An AI agent must:

• Observe the current state (observation)
• Decide what action to take
• Execute the action
• Observe the new state
• Repeat until goal is achieved

From RL Agents to LLM Agents

AlphaGo as an Agent: Goal is winning, observation is the board state, action is choosing where to place a stone. Classic reinforcement learning approach.

The RL limitation: You need to train a separate model for each task. AlphaGo plays Go, but can’t play chess without separate training.

The LLM breakthrough: What if we just use a language model as the agent? Goals become text prompts, observations become text descriptions (or images for multimodal models), and actions are generated as text then translated into executable commands.

AI agents became hot again not because of new agent-specific techniques, but because LLMs became powerful enough to potentially serve as general-purpose agents.

Can LLMs Actually Play Games?

In 2022 (pre-ChatGPT era), researchers tested language models on chess. Result: no model gave correct answers, and weaker models didn’t even understand the rules.

Fast forward to 2025: Someone pitted ChatGPT o1 against DeepSeek-R1 in chess. The result was… chaotic. Both models made illegal moves - using pawns like knights, bishops ignoring obstacles, pieces appearing out of nowhere. DeepSeek eventually ate its own piece with a castle it conjured from thin air, declared victory, and ChatGPT accepted defeat.

Current LLMs still can’t play chess properly, but that doesn’t mean they can’t be useful agents for other tasks.

Three Key Capabilities of AI Agents

1. Learning from Experience (Memory)

The naive approach: feed the agent its entire history before each decision. Problem: by step 10,000, the context is impossibly long.

This is like “hyperthymesia” (超忆症) - people who remember every detail of their lives. Sounds like a superpower, but it’s actually classified as a disorder. These individuals struggle with abstract thinking because their minds are cluttered with trivial details.

Solution: Long-term Memory with RAG

• Write module: Decides what’s worth remembering (not every observation matters)
• Read module: Retrieves relevant memories for the current situation (like RAG)
• Reflection module: Synthesizes higher-level insights from raw memories

ChatGPT already implements this. You can tell it “remember this” and it stores information in its memory. You can even view and manage these memories in settings.

Key finding from StreamBench: Negative examples don’t help much. Telling the model “this was wrong” is less effective than showing it “this is how to do it right.” Don’t tell LLMs what NOT to do - tell them what TO do.

2. Using Tools

LLMs have limitations. They can’t reliably do math, access real-time information, or interact with external systems. Solution: give them tools.

How tool use works:

1. Define tools with text descriptions (function name, parameters, what it does)
2. Model decides when to call a tool
3. Tool returns results
4. Model incorporates results into its response

Example: A speech model that can’t directly process audio can call a speech-to-text tool, then work with the transcript.

The tool selection problem: With hundreds of tools, you can’t describe them all in the prompt. Solution: treat tool descriptions like a memory bank and use RAG to select relevant tools.

LLMs can even create their own tools: They write functions, test them, and if successful, add them to their toolbox for future use.

3. Do LLMs Trust Their Tools Too Much?

Remember Google’s AI Overview suggesting glue for pizza cheese? That came from a Reddit joke that the model took seriously.

But LLMs aren’t completely gullible. Test: tell a model the temperature in Kaohsiung is 100°C - it reports it. Tell it 10,000°C - it recognizes this is impossible and rejects the tool output.

What makes external information persuasive to LLMs?

• Information closer to the model’s existing beliefs is more convincing
• Models with lower confidence in their own answers are easier to sway
• LLMs tend to trust AI-written content over human-written content (even from different AI systems)
• Newer publication dates are more trusted
• Better-designed webpage templates are more persuasive

4. Planning Capabilities

Can LLMs plan? Early tests showed they could decompose tasks like “brush teeth” into steps. But these might just be memorized patterns from training data.

The Mystery Blocksworld Test: Instead of normal block-stacking rules, use bizarre operations like “attack,” “devour,” “conquer.” Models that memorized normal blocksworld fail here.

Results: GPT-4 got only 9% on mystery blocksworld (vs 30%+ on normal). But o1-mini and o1-preview showed real improvement, suggesting actual reasoning rather than pattern matching.

Travel Planning Benchmark: Ask models to plan trips with budget constraints. In early 2024, even GPT-4 Turbo achieved only 0.6% success rate. Common failures: scheduling activities after departure flights, inability to adjust plans to meet budget constraints.

With proper solver tools, success rates jumped to 90%+. The lesson: LLMs can plan, but complex constraint satisfaction needs specialized tools.

Tree Search and World Models

To improve planning, let agents explore possibilities:

1. Tree Search: Try different actions, evaluate outcomes, backtrack from dead ends
2. Problem: Some actions are irreversible (you can’t un-order a pizza)
3. Solution: Simulate in a “dream” - the model imagines outcomes before acting

World Models: The agent predicts what will happen after each action. In chess, this means simulating opponent responses. For web agents, it means imagining what clicking a button will do.

Reasoning Models as Planners

Models like DeepSeek-R1 with “thinking” capabilities naturally perform tree search in their chain-of-thought. Given a blocksworld problem, R1 explores multiple paths in its reasoning, finds an optimal solution, then executes the first step.

The Overthinking Problem: A recent paper “The Danger of Overthinking” found that reasoning models sometimes think too much. They’ll endlessly deliberate about what clicking a button might do, when they could just… click it and see. Some give up without trying, concluding “I probably can’t do this.”

Key Takeaways

1. AI agents use LLMs to pursue goals through multi-step interactions with environments
2. Memory systems (read/write/reflect) prevent context overflow and enable learning
3. Tool use extends LLM capabilities but requires trust calibration
4. Planning is possible but limited - complex constraints need specialized solvers
5. Reasoning models help with planning but can overthink simple problems