François Chollet: How We Get To AGI

François Chollet, creator of Keras and the ARC benchmark, discusses the path to AGI, the fundamental difference between intelligence and skill, and why current AI systems are missing a crucial component.

The Deep Learning Revolution

The Cost of Compute

• Cost of compute has fallen by two orders of magnitude every decade since 1940
• Deep learning started working in the 2010s when GPU compute became available
• Large datasets (ImageNet) combined with compute enabled the breakthrough

The Scaling Paradigm

• Pre-training on massive datasets became the dominant approach
• Scaling laws showed predictable improvements with more compute and data
• But this paradigm has limitations that are becoming apparent

Intelligence vs. Skill

The Key Distinction

François makes a crucial distinction between two concepts:

• Skill: The ability to perform a specific task (static, memorized)
• Intelligence: The ability to handle novel situations efficiently (dynamic, adaptive)

The Road Analogy

• Skill is like a road network - static infrastructure that enables travel
• Intelligence is like a road-building company - the ability to create new roads when needed
• Current AI systems are building massive road networks but lack the road-building capability

Two Definitions of Intelligence

1. Minsky View: Intelligence as task performance (can it do X?)
2. McCarthy View: Intelligence as handling novelty (can it adapt to new situations?)

François argues the McCarthy view is more useful for understanding AGI.

The ARC Benchmark

Why ARC Was Created

• Released in 2019 as an “IQ test for machines”
• Designed to measure fluid intelligence, not memorized skills
• Tasks require understanding abstract patterns with minimal examples

How ARC Works

• Each task shows a few input-output examples
• The system must infer the underlying rule and apply it to a new input
• Tasks are designed to be easy for humans but hard for systems that rely on memorization

ARC-AGI Evolution

• ARC-AGI 1: Original benchmark, now largely solved
• ARC-AGI 2: Released March 2025, harder tasks
• ARC-AGI 3: In development, will assess agency and interactive learning

The 2024 Paradigm Shift

Test-Time Adaptation

• 2024 marked a shift from pre-training scaling to test-time compute
• Systems like OpenAI’s O1 and O3 use more compute at inference time
• O3 achieved human-level performance on ARC-AGI 1

What This Means

• The field is moving toward systems that can adapt on the fly
• But current approaches still have fundamental limitations
• True intelligence requires more than just scaling test-time compute

The Kaleidoscope Hypothesis

Core Idea

• Everything in the universe is a recombination of “atoms of meaning”
• Like a kaleidoscope creating infinite patterns from finite pieces
• Intelligence is the ability to identify and recombine these atoms

Implications

• Learning is about discovering the right atoms of meaning
• Generalization comes from compositional recombination
• This explains why humans can generalize from few examples

Two Types of Abstraction

Type 1: Value-Centric (Continuous)

• Based on perception and intuition
• Handles continuous, fuzzy patterns
• What transformers excel at
• Examples: recognizing faces, understanding language nuance

Type 2: Program-Centric (Discrete)

• Based on reasoning and planning
• Handles discrete, logical structures
• What current AI struggles with
• Examples: mathematical proofs, novel problem-solving

The Missing Piece

• Current AI systems are strong at Type 1 but weak at Type 2
• True AGI requires combining both types
• Need systems that can synthesize programs, not just interpolate patterns

The Path Forward

Program Synthesis

• Future AI needs to be more like programmers
• Synthesizing programs on the fly to solve novel problems
• Discrete search over program space, not just continuous optimization

The Shortcut Rule

• In engineering, the first solution is rarely optimal
• Need to explore the space of possibilities
• Current AI takes shortcuts that limit true generalization

What’s Needed

• Combine deep learning (Type 1) with program synthesis (Type 2)
• Systems that can build new abstractions, not just use existing ones
• True compositional generalization

Key Takeaways

1. Intelligence is not skill - Being good at tasks doesn’t mean being intelligent
2. Novelty is the test - True intelligence handles new situations efficiently
3. Two types of thinking - We need both intuition (Type 1) and reasoning (Type 2)
4. Program synthesis is key - The path to AGI involves synthesizing programs
5. Benchmarks matter - ARC measures what other benchmarks miss

Notable Quotes

“Skill is like a road network. Intelligence is like a road-building company.”

“The question is not ‘can it do X?’ but ‘can it handle novelty?’”

“Current AI is very good at interpolation but struggles with true invention.”