AiScientist: Autonomous Long-Horizon ML Research Through File-as-Bus Architecture

Paper: 2604.13018 Authors: Guoxin Chen, Jie Chen, Lei Chen, Jiale Zhao, Fanzhe Meng, Wayne Xin Zhao, Ruihua Song, Cheng Chen, Ji-Rong Wen, Kai Jia Categories: cs.CL

The Gap

Autonomous AI agents can handle short tasks well, but ML research engineering spans hours or days across task comprehension, environment setup, implementation, experimentation, and debugging. Existing agent systems rely on conversational context passing between agents, which degrades over long horizons as context windows fill up and handoffs lose fidelity. Systems like AutoGPT and MetaGPT struggle with multi-day research tasks because they treat long-horizon work as extended conversations rather than as persistent project state that needs coordination.

Problem: Long-horizon ML research (hours/days)
   |
   v
Existing approach: Conversational handoffs between agents
   |
   +---> Context window limits
   +---> Information loss at handoffs
   +---> No durable project state
   |
   v
Assumption: Need structured coordination + persistent state
   |
   v
Method: Hierarchical orchestration + File-as-Bus workspace
   |
   +---> Orchestrator: thin control layer
   +---> Specialized agents: re-ground on files
   +---> Files: durable artifacts (code, logs, plans)
   |
   v
Evidence: +10.54 PaperBench, 81.82% MLE-Bench Lite
         Ablation: -6.41/-31.82 without File-as-Bus
   |
   v
Conclusion: Long-horizon = systems problem (coordinate over state)
            not reasoning problem (extend conversations)

The Increment

One sentence: Before this paper, autonomous agents treated long-horizon research as extended conversations that degrade over time; after, they coordinate through durable file-based state that agents repeatedly re-ground on.

Core Mechanism

AiScientist has two layers. The top layer is an Orchestrator that maintains stage-level control (task comprehension, setup, implementation, experimentation, debugging) using concise summaries and a workspace map. It doesn’t do the work—it routes tasks to specialized agents and tracks progress through file artifacts.

The bottom layer consists of specialized agents (Planner, Coder, Experimenter, Debugger) that operate on a File-as-Bus workspace. Instead of receiving instructions through conversation and passing results back conversationally, agents read from and write to permission-scoped files: analysis documents, implementation plans, code files, experiment logs, debug traces. Each agent re-grounds on these durable artifacts when it starts work, rather than relying on what the previous agent said in chat.

Orchestrator (thin control)
   |
   +---> maintains: stage tracker, workspace map, summaries
   |
   v
   Routes tasks to:
   |
   +---> Planner ----+
   +---> Coder   ----+---> File-as-Bus Workspace
   +---> Experimenter+      (permission-scoped)
   +---> Debugger ---+      |
                            +---> analysis.md
                            +---> plan.json
                            +---> code/*.py
                            +---> logs/experiment_*.txt
                            +---> debug_trace.log
   
   Each agent:
   1. Reads relevant files (re-grounds)
   2. Performs specialized work
   3. Writes results to files
   4. Reports completion to Orchestrator

Think of it like a construction site. The Orchestrator is the general contractor who tracks which phase the project is in (foundation, framing, electrical, plumbing) and assigns crews. But the contractor doesn’t tell the electrician every detail of what the plumber did—instead, the electrician walks onto the site, looks at the actual pipes and walls (the durable artifacts), and figures out where to run the wiring. The building itself is the communication medium. If the electrician only had the contractor’s verbal summary of the plumbing work, they’d miss critical details. By inspecting the actual pipes, they get ground truth. Similarly, AiScientist agents inspect actual code, logs, and plans rather than relying on summarized handoffs.

Key Concepts

• File-as-Bus: Instead of agents communicating through conversational messages (which get lost or summarized), they communicate by reading and writing files in a shared workspace. Files become the “bus” that carries information between agents. A Planner writes plan.json, the Coder reads it and writes model.py, the Experimenter reads both and writes results.log, the Debugger reads all three. Each file is a durable artifact that any agent can re-ground on. This is like using a shared Google Drive folder where everyone leaves notes and code, versus trying to remember everything said in a long Slack thread.

• Thin control over thick state: The Orchestrator maintains “thin” control—it only tracks high-level stage transitions and routes tasks. It doesn’t micromanage or hold detailed project state in memory. The “thick” state lives in files: thousands of lines of code, experiment logs, debug traces. Agents operate directly on this thick state. This inverts the typical agent architecture where a central controller holds all context and issues detailed instructions. Here, the controller is lightweight and the state is externalized.

• Re-grounding: Each time an agent starts work, it reads the relevant files from scratch rather than relying on what was said earlier in conversation. If the Debugger needs to fix a bug, it doesn’t just read the Experimenter’s chat message saying “test failed”—it reads the actual test output file, the code file, and the plan file. This prevents information decay over long horizons. It’s like a doctor reviewing your full medical chart before each appointment rather than relying on what the nurse said in the hallway.

Framework Shift

Before (conversational agents):        After (AiScientist):

  Agent A                                Orchestrator (thin)
     |                                      |
     v (chat message)                       v (task routing)
  Agent B                                Planner --> plan.json
     |                                      |
     v (chat message)                       v
  Agent C                                Coder --> code/*.py
     |                                      |
     v (chat message)                       v
  Agent D                                Experimenter --> logs/*.txt
                                            |
  [Context degrades over time]              v
  [Information lost at handoffs]         Debugger --> debug_trace.log
  
                                         [State persists in files]
                                         [Agents re-ground on artifacts]

From conversational relay race to construction site coordination, the core shift is externalizing state into durable files that agents repeatedly inspect rather than passing summaries through a degrading conversation chain.

Expert Assessment

Problem choice: Real gap. Long-horizon autonomy is the frontier for AI agents, and the paper correctly identifies that existing systems fail not because of weak reasoning but because of architectural choices (conversational handoffs). This sits at the intersection of agent systems and software engineering, which is timely given the push toward AI-assisted research.

Method maturity: The File-as-Bus idea is elegant and feels obvious in hindsight—it’s how human teams actually work (shared repos, not just Slack). The hierarchical orchestration is less novel (similar to MetaGPT’s role-based structure), but the combination is well-motivated. However, the paper doesn’t deeply explore failure modes: what happens when files get corrupted, when agents write conflicting updates, or when the workspace grows too large? The permission-scoping mechanism is mentioned but not detailed.

Experimental integrity: Two benchmarks (PaperBench, MLE-Bench Lite) with solid improvements (+10.54, 81.82%). The ablation study is critical—removing File-as-Bus drops performance by 6.41 and 31.82 points, which strongly supports the core claim. Baselines include AutoGPT, MetaGPT, and OpenHands, which are reasonable comparisons. However, PaperBench is relatively new and may not be widely validated. The paper doesn’t report variance or multiple runs, which is a minor red flag. Also, 81.82% on MLE-Bench Lite is good but not dominant—there’s room for improvement.

Writing quality: The abstract and introduction are crisp. The method section could be tighter—there’s some repetition between describing the Orchestrator and the File-as-Bus protocol. The related work section is adequate but doesn’t deeply engage with software engineering literature on build systems and artifact-based coordination (e.g., Make, Bazel), which would strengthen the positioning. The results section is clear but lacks error analysis or qualitative examples of where the system succeeds or fails.

Verdict: Weak accept — solid contribution with a clean architectural insight and decent empirical validation, but lacks depth on failure modes and could benefit from more rigorous experimental reporting.

Takeaways

For agent builders: Stop passing information through conversational context. Use files or databases as the communication medium. Design agents to re-read ground truth artifacts rather than trusting summaries. This applies beyond ML research—any multi-step workflow (data pipelines, testing, deployment) benefits from durable state.

For orchestration systems: Separate control (routing, stage tracking) from state (code, logs, plans). Keep the controller thin and stateless. Let specialized workers operate directly on thick, persistent state. This is the Unix philosophy applied to agents: small tools, shared files.

For benchmarking: The ablation study here is a model—removing File-as-Bus causes massive performance drops, which proves it’s not just incremental. When designing agent systems, build in ablation points from the start so you can isolate what actually matters.

Concrete technique: Permission-scoped file access. Don’t give every agent read/write access to everything. The Planner writes plans but doesn’t touch code. The Coder reads plans but doesn’t modify experiment logs. This prevents agents from stepping on each other and makes debugging easier. Implement this with file system permissions or a simple access control layer.