BrainTaskonomy: Organise Pretraining and Transfer by Measured Learning Relations

Paper: 2609.10518 Authors: Junfeng Xia, Wenhao Ye, Junxiang Zhang, Jiayu Zuo, Mo Wang, Quanying Liu Categories: cs.CV, q-bio.NC

The Gap

fMRI foundation models aggregate data across brain states, cohorts and acquisition settings, and two defaults have gone unexamined. Pretraining domains are treated as a flat mixture — every domain sampled as if interchangeable. And downstream tasks are adapted independently — each task trained from the pretrained model as if no task could help another.

Neither assumption is obviously true, and both are auditable. Domains differ in difficulty and in how much they facilitate learning of other domains. Tasks transfer asymmetrically: knowing task A may help task B more than the reverse. If those relations exist and are measurable, then treating both stages as flat sets discards information that is already available.

   TWO DEFAULTS IN fMRI FOUNDATION MODELS

   data aggregated across BRAIN STATES, COHORTS, ACQUISITION SETTINGS
        |
        +--------------------+---------------------+
        v                                          v
   PRETRAINING DOMAINS treated as a             DOWNSTREAM TASKS adapted
   FLAT MIXTURE                                 INDEPENDENTLY
     every domain sampled as if                   each task trained from the
     INTERCHANGEABLE                              pretrained model as if NO
                                                  TASK COULD HELP ANOTHER
        |                                          |
        v                                          v
   [NEITHER ASSUMPTION IS OBVIOUSLY TRUE, AND BOTH ARE AUDITABLE]
     domains differ in DIFFICULTY and in HOW MUCH they FACILITATE
       learning of other domains
     tasks TRANSFER ASYMMETRICALLY: knowing task A may help B more
       than the reverse
        |
        v
   [GAP] if those relations exist and are MEASURABLE, treating both
         stages as FLAT SETS discards information that is ALREADY
         AVAILABLE

The Increment

One sentence: Before this paper, fMRI pretraining and adaptation treated domains and tasks as flat sets; after it, measured facilitation and transfer organise both stages into a curriculum and a taskonomy, improving three metrics without modifying the backbone.

Core Mechanism

The constraint that shapes the whole contribution is stated up front: without modifying the backbone. So the organisation happens in what to sample, in what order, and what to adapt from — not in the architecture. That is the right constraint for this claim, because it isolates the value of the learning relations from any capacity change.

During pretraining, a lightweight Brain-DiT proxy estimates difficulty and directed facilitation across ten fMRI domains. Two words matter. Directed means the proxy measures which domain helps which, not merely which are similar — a symmetric similarity matrix would not distinguish A-helps-B from B-helps-A, and the paper’s whole premise is that direction matters. And proxy plus lightweight means the measurement is cheap enough to be worth doing, which is what makes the approach practical rather than a one-off analysis.

Those estimates feed a priority-guided cumulative domain curriculum combined with high-to-low-noise timestep scheduling and joint consolidation. Two things are being sequenced simultaneously here — which domains, and which noise levels — and the paper reports them jointly, which is consistent with the finding that they were evaluated together.

During adaptation, controlled first- and higher-order transfer across fifteen tasks constructs a directed taskonomy, from which budgeted integer programming selects directly supervised source tasks and target-specific routes. The formulation is worth noticing because it is an explicit optimisation rather than a heuristic: routes are selected under a budget, so the choice of which tasks to train and which transfer paths to use is made as an allocation problem. “Higher-order” routes mean chains — A helps B helps C — which is where the search space becomes large enough that an allocation formulation earns its place.

The pretraining result is measured against the honest baseline: relative to uniform sampling over both dimensions, the joint curriculum reduces v-NMSE by 6.5%, PSD-NMSE by 16.3%, and FC-MSE by 10.5%, with strong downstream performance across six in- and out-of-domain tasks. The comparison against uniform sampling over both dimensions is the right control, since the claim is about organising both stages rather than one.

The taskonomy’s characterisation is reported as a property in its own right: it reveals asymmetric, target-dependent transfer. So the structure is not a simple ranking of tasks by usefulness; usefulness depends on the target. And the exploratory sealed-test evaluation shows larger descriptive gains for BIP policies when higher-order route spaces are available than for matched random controls — with the matched random control being the detail that prevents the route-space benefit from being attributed to having more routes at all.

   THE CONSTRAINT THAT SHAPES EVERYTHING: WITHOUT MODIFYING THE BACKBONE
     -> organisation happens in WHAT TO SAMPLE, IN WHAT ORDER, and
        WHAT TO ADAPT FROM -- not in the architecture
     <- the right constraint for this claim: it ISOLATES THE VALUE OF
        THE LEARNING RELATIONS from any capacity change

   [PRETRAINING]
     a LIGHTWEIGHT Brain-DiT PROXY estimates DIFFICULTY and DIRECTED
     FACILITATION across TEN fMRI domains
       "DIRECTED" matters: it measures WHICH DOMAIN HELPS WHICH, not
         merely which are SIMILAR
         <- a SYMMETRIC similarity matrix would not distinguish
            A-helps-B from B-helps-A, and the whole premise is that
            DIRECTION MATTERS
       "LIGHTWEIGHT proxy" matters: the measurement is CHEAP ENOUGH TO
         BE WORTH DOING
         -> what makes the approach PRACTICAL rather than a one-off
            analysis

     -> feeds a PRIORITY-GUIDED CUMULATIVE DOMAIN CURRICULUM combined
        with HIGH-TO-LOW-NOISE TIMESTEP SCHEDULING and JOINT
        CONSOLIDATION
        <- TWO things sequenced simultaneously: WHICH DOMAINS and
           WHICH NOISE LEVELS
        <- reported JOINTLY, consistent with their being evaluated
           together

   [ADAPTATION]
     CONTROLLED FIRST- and HIGHER-ORDER transfer across FIFTEEN TASKS
     constructs a DIRECTED TASKONOMY
       -> BUDGETED INTEGER PROGRAMMING selects DIRECTLY SUPERVISED
          SOURCE TASKS and TARGET-SPECIFIC ROUTES
       <- an EXPLICIT OPTIMISATION, not a heuristic: routes are
          SELECTED UNDER A BUDGET, so choosing which tasks to train and
          which transfer paths to use is an ALLOCATION PROBLEM
       <- "HIGHER-ORDER" routes mean CHAINS (A helps B helps C), which
          is where the search space becomes large enough that an
          allocation formulation EARNS ITS PLACE

   [THE PRETRAINING RESULT, against an honest baseline]
     relative to UNIFORM SAMPLING OVER BOTH DIMENSIONS, the joint
     curriculum reduces
       v-NMSE   by  6.5%
       PSD-NMSE by 16.3%
       FC-MSE   by 10.5%
     with strong downstream performance across SIX in- and out-of-domain
     tasks
       <- comparing against uniform over BOTH dimensions is the right
          control: the claim is about organising BOTH stages, not one

   [THE TASKONOMY'S OWN PROPERTY]
     reveals ASYMMETRIC, TARGET-DEPENDENT transfer
       -> not a simple ranking of tasks by usefulness: usefulness
          DEPENDS ON THE TARGET
     exploratory SEALED-TEST evaluation shows LARGER DESCRIPTIVE GAINS
     for BIP policies WHEN HIGHER-ORDER ROUTE SPACES ARE AVAILABLE, than
       for MATCHED RANDOM CONTROLS
       <- the matched control is the detail that stops the route-space
          benefit from being attributed to merely having MORE ROUTES

Think of it as planning a study sequence instead of taking courses in arbitrary order. A flat mixture is a curriculum where you take whatever course comes up next, at whatever depth; independent adaptation is studying for each exam without noticing that one subject makes the next easier. Two facts a student would exploit are that some courses are prerequisites for others, and that the benefit is asymmetric — mastering statistics helps with machine learning far more than the reverse. The paper’s proxy is the equivalent of measuring those prerequisites before planning, and the integer program is the equivalent of choosing a sequence under a fixed number of credit hours. And the sealed-test comparison against a matched random control is what distinguishes “the prerequisites are real” from “having more course options obviously can’t hurt”.

Key Concepts

• Directed facilitation versus similarity: measuring which domain helps which, not which are alike. A symmetric matrix cannot express the asymmetry the paper is built on.
• A lightweight proxy as the enabler: cheap difficulty and facilitation estimates. It is what makes organising by measured relations practical rather than a one-off study.
• Joint curriculum over two dimensions: domains and noise levels sequenced together. Reporting them jointly is consistent with evaluating them together, and the uniform-over-both control matches that scope.
• A budgeted integer program for routes: choosing source tasks and transfer paths as an allocation under a budget, including higher-order chains. It turns “which task should I learn from” into an optimisation with a constraint.
• Matched random control for route spaces: comparing against random routes rather than against no routes. It separates the value of the structure from the value of having a larger space.

Framework Shift

Before (flat mixture, independent adaptation):
  sample all pretraining domains uniformly
  adapt each downstream task from the pretrained model alone
  -> domains treated as interchangeable
  -> tasks treated as unable to help one another
  -> available structure is discarded

After (measured relations organise both stages):
  proxy measures directed facilitation across ten domains
  -> priority-guided cumulative curriculum + high-to-low noise
  controlled transfer across fifteen tasks builds a taskonomy
  -> budgeted integer programming selects source tasks and routes
  -> -6.5% v-NMSE, -16.3% PSD-NMSE, -10.5% FC-MSE vs uniform
  -> backbone unchanged throughout

From treating domains and tasks as flat sets, to organising both stages by measured directed relations, the core shift is that the ordering and the provenance of training data carry information that a uniform mixture does not use.

Expert Assessment

Problem choice: Excellent, and the two-defaults framing is unusually well aimed. Domain ordering and task transfer are both places where practitioners have intuitions — some data is harder, some tasks are prerequisites — and the paper’s contribution is to replace those intuitions with measurements and then act on them.

Method maturity: The constraint of not modifying the backbone is what makes the result interpretable, because every gain is attributable to organisation. The directed-proxy design is the right measurement for the premise, and insisting on direction rather than similarity is what keeps the finding from collapsing into a domain-similarity ranking. Formulating route selection as a budgeted integer program — rather than a greedy heuristic — is the choice that makes higher-order chains tractable, and the matched random control for route spaces is a well-constructed check.

Experimental integrity: The controls are the strength: uniform sampling over both dimensions, and random routes matched to BIP routes. The reporting is also careful about strength — the sealed-test result is described as exploratory with “descriptive gains”, which is the correct register for an evaluation the paper does not present as confirmatory. The limitations are that the whole argument rests on a proxy’s estimates of difficulty and facilitation, so proxy error propagates into the curriculum, and that the setting is fMRI with ten domains and fifteen tasks, so whether these relations are equally exploitable in domains with weaker structure between datasets is untested.

Writing quality: The two defaults are stated plainly, and every component maps onto one of them, which makes the paper readable as a whole. Because the practical question is whether the proxy can be trusted, a short passage on how well proxy-estimated facilitation agrees with the transfer measurements it predicts — the proxy is exercised in pretraining, the taskonomy in adaptation — would strengthen confidence in the mechanism.

Verdict: strong accept — it audits two unexamined defaults in a data-hungry domain, replaces them with measured directed relations, and shows the reorganisation helps on multiple metrics without touching the backbone.

Takeaways

• Measure the relations, then act on them. Domain difficulty and task transfer are usually handled by intuition; both are measurable, and the measurement is cheap enough to be routine.
• Insist on direction. A symmetric similarity measure cannot tell you that A helps B but not the reverse, which is the asymmetry worth exploiting.
• Match the control to the claim’s scope. Uniform sampling over both dimensions matches a claim about organising both stages.
• Control for the size of the search space, not just for its absence. Comparing structured routes against matched random routes separates the value of structure from the value of options.