Mining Generalizable Activation Functions via Evolutionary Search

Paper: 2602.05688 Authors: Alex Vitvitskyi, Michael Boratko, Matej Grcic, Razvan Pascanu, Deep Shah, Petar Veličković Affiliations: Google DeepMind Categories: cs.LG, cs.AI

Abstract

This paper investigates evolutionary search as a framework for discovering activation functions. Using frontier LLMs as mutation operators (via AlphaEvolve), the search space expands to all possible Python functions within a given FLOP budget, eliminating manual constraint design. Out-of-distribution performance serves as the fitness metric to discover activation functions that encode specific inductive biases.

Key Contributions

• LLM-guided evolutionary search: Uses frontier LLMs as mutation operators via AlphaEvolve
• Expanded search space: Searches over all possible Python functions within a FLOP budget
• OOD fitness metric: Targets out-of-distribution robustness rather than in-distribution performance
• Practical discoveries: Found GELU-Sinc-Perturbation, GELUSine, GMTU, and Turbulent Activation Function

The Activation Function Design Problem

Activation functions are a fundamental building block of neural networks, yet the design space has been explored mostly through human intuition: ReLU, GELU, SiLU/Swish, and a handful of others. Each was designed by hand with specific properties in mind.

Can we do better by searching systematically?

AlphaEvolve-Powered Search

The key innovation is using frontier LLMs as mutation operators within an evolutionary search framework (AlphaEvolve). This has two advantages:

1. Unconstrained search space: Instead of searching over parameterized function families, the search covers all Python functions within a FLOP budget
2. Semantic mutations: LLMs can propose meaningful modifications to activation functions, not just random perturbations

OOD Fitness: The Right Objective

A critical design choice: using out-of-distribution performance as the fitness metric rather than in-distribution accuracy. This targets activation functions that encode useful inductive biases — functions that help the network generalize, not just memorize.

Discovered Functions

The search yielded several novel activation functions:

• GELU-Sinc-Perturbation: GELU with sinc-function perturbation
• GELUSine: GELU-sine hybrid
• GMTU: Gated multiplicative trigonometric unit
• Turbulent Activation Function: A more complex composition

These outperform ReLU on out-of-distribution benchmarks while maintaining competitive in-distribution performance.

Takeaways

1. LLMs as search operators: Using LLMs for evolutionary search over code is a powerful paradigm
2. Small-scale discovery transfers: Functions found on small synthetic datasets generalize to larger problems
3. OOD as objective: Targeting generalization during search produces more robust components
4. Activation functions are not solved: There’s still room for improvement in this fundamental building block