Notes: OpenAI vs. DeepSeek vs. Qwen - Comparing Open Source LLM Architectures

This post summarizes Y Combinator’s technical comparison of three major open source LLM architectures: OpenAI’s GPT OSS, DeepSeek V3, and Alibaba’s Qwen 3.

GPT OSS: OpenAI’s Open Weights Model

OpenAI recently released GPT OSS, its first open weights model since GPT-2 in 2019. It’s one of the highest profile open source model launches since DeepSeek R1 made waves in January.

Architecture Overview

GPT OSS is a Mixture of Experts (MoE) model available in two sizes:

• 120 billion parameters (5.1B activated per token)
• 20 billion parameters (3.6B activated per token)

Each token activates the top 4 experts, meaning only a portion of total parameters are used at any given time. This allows for efficient inference without sacrificing the benefits of a larger model.

Key Technical Features

As a decoder-only transformer, GPT OSS incorporates modern LLM features:

• Grouped Query Attention (GQA): Multiple query heads share the same key-value pairs to reduce memory use and speed up inference
• SwiGLU Activations: More nuanced transformations than simpler activations like ReLU in feed-forward layers
• Rotary Positional Embeddings (RoPE): Encodes token position directly into the attention mechanism to support longer contexts
• RMS Norm with Pre-normalization: Scales inputs by their root mean square for more stable training

Context Window

GPT OSS achieves a 131,000 token context window by applying YaRN scaling during pre-training rather than as an inference-time adjustment. This means the model’s weights have learned to work natively with long contexts.

Tokenizer and Training Data

• Uses OpenAI’s open-source O200K Harmony tokenizer (byte pair encoding with over 200,000 tokens)
• Trained on a text-only corpus in the trillions of tokens
• Focus on STEM, coding, and general knowledge
• Harmful content filtered out for safety

Release Format

• Released in quantized format by default for deployment on modest hardware
• Can run on consumer-grade GPUs and laptops
• No unquantized version available
• Underwent substantial post-training for safety and alignment

Qwen 3: Alibaba’s Model Family

Qwen 3, developed by Alibaba Cloud, dropped in April with benchmark scores rivaling DeepSeek V3 and Llama 4.

Model Variants

Dense Models (7 size classes):

• Range from 0.6 billion to 32 billion parameters
• Activate all parameters for each query

Mixture of Experts Models (2 sizes):

• 30 billion parameters
• 235 billion parameters
• 128 total experts, 8 activated per token
• MoE models match dense model performance with only 1/5 as many active parameters

Architecture

Qwen 3 dense models are similar to Qwen 2.5, incorporating:

• Group Query Attention
• SwiGLU
• RoPE
• RMS Norm

Key Innovation - QK Norm: Replaces QKV bias (static offset) with a normalization step that dynamically rescales query and key vectors to maintain constant magnitudes, keeping attention scores stable at scale.

Training Pipeline

Pre-training (3 stages):

1. General Stage: 30+ trillion tokens covering 119 languages at 4,096 sequence length
2. Reasoning Stage: Additional 5 trillion higher-quality tokens featuring STEM reasoning and coding problems
3. Long Context Stage: Extended to 32,000+ tokens using ABF, YaRN, and dual chunk attention

Total: 36 trillion pre-training tokens (twice as many as Qwen 2.5)

Qwen 3 also uses Qwen 2.5 models to generate trillions of tokens of synthetic data in formats like textbooks, instructions, and code snippets.

Post-Training Pipeline (4 steps)

Goals: Give users control over reasoning depth and enable efficient distillation to smaller models.

1. Long Chain of Thought Cold Start: Curated dataset of challenging reasoning problems from math, logic, and STEM with verifiable answers
2. Reasoning RL Stage: Uses GRPO (originally developed by DeepSeek) on roughly 4,000 query-verifier pairs
3. Thinking Mode Fusion: Key Qwen 3 innovation - integrates reasoning and non-reasoning into a single model, letting users switch modes without changing models
4. General RL: Broadens capabilities in instruction following, formatting, preference alignment, tool use, and specialized scenarios

Strong-to-Weak Distillation: Allows training smaller models from larger ones.

DeepSeek V3: The Efficiency Pioneer

Released in December, DeepSeek V3 was one of the most ambitious open source LLMs from a major lab.

Scale and Architecture

• 671 billion total parameters
• 37 billion activated per token
• Mixture of Experts architecture
• Designed for efficiency as much as capability

Key Innovation: Multi-head Latent Attention (MLA)

Unlike GPT OSS and Qwen 3 which use GQA, DeepSeek V3 uses MLA:

• Compresses keys and values into a smaller latent space before caching
• Decompresses them during inference
• More complex to implement but delivers greater memory savings
• Better modeling performance than GQA, especially in huge long-context models

Training Optimizations

• Native 8-bit training (rather than 16 or 32-bit) - huge unlock for cutting training costs
• Hardware and algorithmic optimizations throughout

DeepSeek V3.1 Update

The newly released V3.1 builds on V3 with:

• Two-phase long context training approach
• Hybrid thinking mode (switch between reasoning-heavy and lightweight inference)
• Improved tool use and agent performance
• Same core architecture, stronger reasoning and smarter tool use

Comparing Context Extension Approaches

YaRN (Yet Another RoPE Extension) stretches rotary positional embeddings to handle longer sequences than originally trained on.

Each model uses YaRN differently:

Model	Approach	Result
GPT OSS	Applied from pre-training	Born with 131K context ability
DeepSeek	Staged fine-tuning (32K → 128K)	Trained into it step by step
Qwen	Fine-tune to 32K, then inference-time YaRN scaling (4x)	Pushes limits without extra retraining

Key Observations

Empirical vs. First Principles

These papers read as empirical findings rather than theoretical derivations:

• Each lab describes what works well for them
• Almost no one gives first-principles justification for why one tool is better
• Very different from math or theoretical physics which derive results from axioms

Surprising Convergence

Despite using often very different techniques:

• Similar topline benchmark statistics
• Broadly same tools (attention mechanisms, activation functions, positional embeddings)
• Different training methods leading to similar results

Reinforcement Learning’s Role

All major models heavily use RL in post-training and reasoning:

• Surprisingly small data requirements (just 4,000 pairs for Qwen’s reasoning RL)
• Fascinating efficiency gains

The Data Set Moat

• Enormous work happening behind the scenes in data set engineering
• Very opaque what differences exist between labs
• Probably a significant aspect of the competitive moat
• Makes replication difficult even with open weights

Takeaways

When reading these papers:

• Don’t focus too much on benchmark performance or topline stats like context size
• Look at the specific methods labs use to achieve results
• Peek under the hood to find nuanced differences
• Use this as a framework for understanding future open source releases

Other notable models not covered: Kimi K2, Google Gemma 3, and many more with their own interesting design choices.