AI Engineering with Chip Huyen – The Pragmatic Engineer

Chip Huyen, author of “Designing Machine Learning Systems” and AI engineering expert, joins Gergely Orosz to discuss the emerging field of AI engineering and how it differs from traditional ML engineering.

What is AI Engineering?

Chip defines the emerging discipline:

AI Engineering vs ML Engineering

• ML Engineering: Training models, feature engineering, model optimization
• AI Engineering: Building applications on top of foundation models
• Key difference: AI engineers often don’t train models, they use them

The New Stack

• Foundation models (GPT, Claude, etc.) as building blocks
• Prompt engineering as a core skill
• RAG, fine-tuning, and orchestration
• Evaluation and monitoring

Building AI Products

The Reality of AI Products

• Most AI products are wrappers around foundation models
• The hard part is product design, not model training
• User experience matters more than model performance

Common Patterns

1. Chat interfaces: Most common but not always best
2. Embedded AI: AI features within existing products
3. Autonomous agents: Emerging but challenging

What Makes AI Products Succeed

• Clear use case
• Appropriate expectations
• Good failure handling
• Continuous improvement

The AI Engineering Skillset

Technical Skills

• Prompt engineering
• RAG implementation
• Fine-tuning (when needed)
• Evaluation design
• System architecture

Non-Technical Skills

• Product thinking
• User empathy
• Communication
• Experimentation mindset

Evaluation: The Hard Problem

Chip emphasizes evaluation challenges:

Why It’s Hard

• No ground truth for many tasks
• Subjective quality judgments
• Edge cases are infinite
• User satisfaction ≠ model metrics

Approaches

• Human evaluation (expensive but necessary)
• LLM-as-judge (useful but imperfect)
• A/B testing in production
• User feedback loops

RAG: The Dominant Pattern

Discussion on Retrieval-Augmented Generation:

Why RAG Works

• Grounds responses in real data
• Reduces hallucinations
• Enables domain-specific knowledge
• More controllable than fine-tuning

RAG Challenges

• Retrieval quality is crucial
• Chunking strategies matter
• Context window limitations
• Latency considerations

Fine-tuning: When and Why

Chip’s perspective on fine-tuning:

When to Fine-tune

• Specific output format needed
• Domain-specific terminology
• Consistent style requirements
• Performance optimization

When NOT to Fine-tune

• RAG can solve the problem
• Data is limited
• Requirements change frequently
• Cost is a concern

The Future of AI Engineering

Short-term Trends

• Better tooling and frameworks
• More specialized models
• Improved evaluation methods
• Cost reduction

Long-term Questions

• Will AI engineering merge back into software engineering?
• How will the role evolve as models improve?
• What skills will remain valuable?

Career Advice

For those entering AI engineering:

1. Build things: Practical experience matters most
2. Understand the stack: From prompts to infrastructure
3. Focus on evaluation: It’s the hardest and most valuable skill
4. Stay curious: The field changes rapidly

On the Hype

Chip’s balanced view:

“There’s real value in AI, but also real limitations. The best AI engineers understand both.”

• Hype attracts investment and talent
• But also creates unrealistic expectations
• Focus on solving real problems