HypoExplore: Neural Architecture Discovery as Scientific Inquiry

Paper: 2604.12999 Authors: Jaywon Koo, Jefferson Hernandez, Ruozhen He, Hanjie Chen, Chen Wei, Vicente Ordonez Categories: cs.CV

The Gap

Neural Architecture Search (NAS) has evolved from random search to sophisticated optimization methods, but existing approaches treat architecture discovery as a black-box optimization problem. Methods like DARTS, ENAS, and evolutionary algorithms explore design spaces efficiently but don’t accumulate knowledge about why certain design choices work. Each search run starts from scratch, discarding the reasoning behind successful architectures. The field has reached high performance but lacks interpretability and transferability of design principles.

Problem: NAS finds good architectures but not design understanding
    |
    v
Assumption: Architecture discovery can mirror scientific inquiry
    |
    v
Method: Hypothesis-driven evolution with confidence tracking
    |
    v
Evidence: 18.91% -> 94.11% on CIFAR-10, principles transfer across runs
    |
    v
Conclusion: Explicit hypothesis tracking builds reusable knowledge

The Increment

One sentence: Before this paper, NAS found architectures through opaque optimization; after, it generates and tests explicit hypotheses that accumulate into transferable design principles.

Core Mechanism

HypoExplore maintains two core data structures: a Trajectory Tree recording the lineage of all proposed architectures, and a Hypothesis Memory Bank tracking confidence scores for design principles. Each node in the tree represents an architecture with its associated hypothesis (e.g., “adding depthwise separable convolutions improves efficiency”). The system operates in cycles: an LLM selects a parent hypothesis using a dual strategy that balances exploitation (building on validated ideas) and exploration (testing uncertain ones), proposes a modification, implements it as code, trains the architecture, and collects results.

After each experiment, multiple specialized feedback agents analyze results from different angles—one examines accuracy trends, another checks parameter efficiency, a third looks at training stability. These agents consolidate their observations into confidence updates for the hypothesis. If “depthwise separable convolutions improve efficiency” consistently correlates with better results, its confidence score rises. If it fails in certain contexts, the score adjusts accordingly. The confidence scores guide future parent selection, creating a feedback loop where the system learns which principles are robust.

                    Hypothesis Memory Bank
                    [principle_1: conf=0.8]
                    [principle_2: conf=0.3]
                           |
                           v
    LLM selects parent --> Generate new hypothesis
           |                       |
           v                       v
    Implement code <-------- Modify architecture
           |
           v
    Train & Evaluate
           |
           v
    Feedback Agents (accuracy, efficiency, stability)
           |
           v
    Update confidence scores --> Back to Memory Bank
           |
           v
    Add to Trajectory Tree (parent-child link preserved)

Think of HypoExplore as a research lab with a persistent lab notebook. Traditional NAS is like hiring a new researcher for each project who tries random experiments and reports the best result but leaves no notes. HypoExplore is a lab where each experiment is documented with its rationale (hypothesis), results are analyzed by a committee (feedback agents), and findings are recorded in a shared notebook (Memory Bank) with confidence ratings. When starting a new experiment, researchers consult the notebook to see which approaches have strong evidence and which need more testing. The family tree of experiments (Trajectory Tree) shows which ideas descended from which, so you can trace successful architectures back to their conceptual ancestors. Over time, the lab builds institutional knowledge—not just “architecture X works” but “principle Y tends to work because Z.”

Key Concepts

• Hypothesis Confidence Score: Instead of just tracking whether an architecture performed well, HypoExplore tracks confidence in the *principle behind it. Imagine you’re learning to cook. You try adding garlic to three dishes—two turn out great, one mediocre. Your confidence in “garlic improves flavor” is high but not absolute. If you later try garlic in desserts and it fails, your confidence becomes context-dependent: “garlic improves savory dishes.” HypoExplore does this for architecture principles. A hypothesis like “residual connections help deep networks” starts with neutral confidence. As experiments accumulate, the score rises if residual connections consistently improve results, or becomes nuanced if they only help in certain depth ranges. The score is Bayesian—it updates with evidence and can represent uncertainty.

• Dual Exploration Strategy: The system must balance two competing goals: exploit validated principles (build on what works) versus explore uncertain ones (test unproven ideas). Pure exploitation leads to local optima—you keep refining the same idea. Pure exploration wastes time on random guesses. HypoExplore uses confidence scores to balance both. High-confidence hypotheses are selected more often (exploitation), but medium-confidence ones get chances too (exploration). It’s like a scientist who mostly builds on established theories but occasionally tests fringe ideas that might be breakthroughs. The selection probability is proportional to confidence, but not deterministic—even low-confidence hypotheses can be chosen, just rarely.

• Trajectory Tree: This isn’t just a log of experiments; it’s a genealogy of ideas. Each node is an architecture, and edges show parent-child relationships. If architecture B was created by modifying architecture A based on hypothesis H, there’s an edge A→B labeled with H. This structure reveals which principles are fertile (many successful descendants) versus dead ends. You can trace any successful architecture back to its conceptual roots and see the sequence of insights that led there. It also enables transfer learning of principles: if two independent branches both discover that “attention mechanisms improve accuracy,” the Memory Bank aggregates evidence from both lineages, strengthening confidence faster than either branch alone could.

Framework Shift

Before (mainstream NAS):              After (HypoExplore):

Search Space                          Hypothesis Space
     |                                      |
     v                                      v
Random/Gradient Sampling              LLM-Guided Ideation
     |                                      |
     v                                      v
Train & Evaluate                      Train & Multi-Agent Analysis
     |                                      |
     v                                      v
Keep Best Architecture                Update Principle Confidence
     |                                      |
     v                                      v
[Discard reasoning]                   Accumulate in Memory Bank
                                             |
                                             v
                                      Transfer to New Searches

From implicit optimization to explicit reasoning, the core shift is making design principles first-class citizens that persist and transfer across search runs.

Expert Assessment

Problem choice: This is a real gap. NAS has matured in performance but remains a black box. The field is ready for interpretability—practitioners want to understand *why architectures work, not just which ones work. The timing is right given LLMs’ reasoning capabilities. However, the problem is somewhat manufactured in that many researchers already informally track design principles; this paper formalizes it.

Method maturity: The core insight—treating architecture search as hypothesis testing—is elegant. The dual exploration strategy and confidence tracking are well-motivated. However, the reliance on LLMs for hypothesis generation is a double-edged sword: it enables rich reasoning but introduces brittleness (prompt sensitivity, hallucination risk). The multi-agent feedback mechanism feels slightly over-engineered; simpler confidence update rules might suffice. The method is more “clever orchestration” than “novel algorithm.”

Experimental integrity: The 18.91% → 94.11% improvement is dramatic but starts from a deliberately weak baseline (likely a shallow network). More informative would be starting from a reasonable baseline like ResNet-18. The CIFAR-10 results are solid, and generalization to CIFAR-100, Tiny-ImageNet, and MedMNIST strengthens the claims. The claim that “hypothesis confidence scores grow increasingly predictive” is supported but not rigorously quantified—correlation plots would help. Baselines are fair but not exhaustive; comparison to recent NAS methods like AutoFormer or OFA would be valuable.

Writing quality: The paper is well-structured and clearly written. The Trajectory Tree and Memory Bank concepts are explained intuitively. However, Section 3.2 (feedback agents) is vague—what exactly do these agents compute? The paper would benefit from showing example hypotheses and their confidence trajectories over time. The related work section is thorough but could better position this work relative to meta-learning and neural architecture transfer.

Verdict: weak accept — The hypothesis-driven framing is a valuable contribution to NAS interpretability, and the results demonstrate feasibility, but the method’s complexity and LLM dependence raise questions about practical adoption and reproducibility.

Takeaways

Confidence-weighted exploration: The dual strategy of balancing exploitation and exploration via confidence scores is broadly applicable. In hyperparameter tuning, you could track confidence in “learning rate schedules” or “regularization strategies” and prioritize well-validated approaches while occasionally testing uncertain ones. This beats pure grid search or random search.

Genealogy tracking for iterative design: The Trajectory Tree concept transfers to any iterative design process. In prompt engineering, track which prompt modifications descended from which, and which principles (e.g., “adding examples improves accuracy”) have strong evidence across branches. In A/B testing, maintain a tree of feature variants and their rationales.

Multi-perspective evaluation: Using specialized agents to analyze results from different angles (accuracy, efficiency, stability) is a pattern worth stealing. In model evaluation, don’t just look at aggregate metrics—have separate “agents” check for fairness, robustness, calibration, and latency. Consolidate findings into actionable insights.

Explicit principle extraction: The biggest takeaway is making implicit knowledge explicit. Most practitioners informally track “what works,” but formalizing it into a queryable knowledge base with confidence scores enables transfer and collaboration. If you’re doing any form of iterative experimentation, maintain a structured log of hypotheses, evidence, and confidence—not just results.