Satellite Embeddings as Sub-Grid Descriptors: Teaching a Downscaler What the Ground Looks Like

Paper: 2608.12271 Authors: Pedro Sousa, Will Tebbutt, Sadiq Jaffer, Robin Young, Anil Madhavapeddy, Richard E. Turner Categories: cs.LG, physics.ao-ph

The Gap

Global reanalysis and AI forecast products live on grids of roughly 25 km. Inside one such cell you can have a lake, a ridge, a city centre, and a pine plantation. ERA5 gives all of them the same 2 m temperature. Anyone who actually needs a number at a location — a wind farm, a vineyard, a road-icing service — has to bridge that gap.

The established way to bridge it is statistical downscaling with a learned model that takes the coarse atmospheric state plus a description of the site, and predicts the local observation. The strongest probabilistic version of this is the convolutional conditional neural process (ConvCNP) family, which can predict at arbitrary off-grid points and returns a distribution rather than a point. These models already work. Their weak spot is the “description of the site”: it’s hand-crafted topography. Elevation, slope, aspect, terrain position index at a few smoothing scales. Someone decided in advance that terrain shape is what matters sub-grid.

Terrain shape is clearly part of what matters. But a paved car park and a wet meadow at identical elevation and slope get identical descriptors, and they do not have identical 2 m temperature or 10 m wind. What’s missing is the state of the surface, not just its geometry — albedo, roughness, moisture, vegetation structure, urban fabric. That information exists in satellite imagery, and Earth-observation foundation models now compress satellite time series into per-pixel embeddings. The question this paper asks is uncomfortable on its face: those embeddings summarise a year. Weather downscaling is instantaneous. Can a slow descriptor help a fast prediction?

[Problem]
  one 25 km cell = one number for a lake, a ridge, and a city
        |
        v
[Prior art]
  ConvCNP downscaler + hand-crafted topo descriptors
  (elevation, slope, aspect, TPI)
  ... encodes the SHAPE of the land, not the STATE of the surface
        |
        v
[Assumption]
  the sub-grid departure at a site is mostly PERSISTENT
  => a slow (annual) descriptor can serve a fast (hourly) task
        |
        v
[Method]
  compress a 10 m TESSERA embedding patch -> short vector d
  feed d to the ConvCNP decoder alongside coarse state + topo
        |
        v
[Evidence]
  5 climatically diverse regions
  stations held out in SPACE and in TIME
  CRPS skill: +11.5% (2 m temperature), +6.2% (10 m wind speed)
  survives swapping ERA5 -> Aurora forecasts as the coarse input
  survives brand-new stations with no regional history
        |
        v
[Conclusion]
  annual-timescale EO embeddings are usable sub-grid descriptors
  wind gains more from surface state; temperature more from terrain

The Increment

One sentence: Before, the sub-grid half of a probabilistic downscaler was a human-designed topography vector; after, it can be a learned compression of a satellite foundation model’s embeddings, and that swap buys double-digit CRPS improvement on temperature without touching the atmospheric model.

Core Mechanism

Start with the backbone, which is not new and the authors don’t pretend it is. A ConvCNP takes gridded coarse fields, pushes them through a CNN, and produces a continuous latent function over space — you can evaluate it anywhere, not just at grid nodes. A decoder then evaluates that latent at a query location and outputs the parameters of a predictive distribution for the target variable at that point. That’s why it handles station data: stations sit at arbitrary coordinates, and you want a spread, not just a mean.

The new part sits on the side channel. Around each query location, take a patch of TESSERA embeddings at 10 m resolution. That patch is large — thousands of pixels, each with an embedding vector — so it cannot be fed to the decoder directly. A learned compressor squeezes the patch into a short descriptor vector. Critically, this compressor is trained end to end against the downscaling loss, not with a reconstruction or clustering objective. The model is not asked “summarise this land cover”; it is asked “extract from this land cover whatever predicts how this site departs from its grid cell.” Those are different objectives and the second one is the interesting one.

At prediction time the decoder therefore sees three things: the interpolated coarse atmospheric state at the query point, the hand-crafted topographic features (kept, not replaced), and the learned surface descriptor. The paper’s per-variable analysis then splits the credit: for 2 m temperature, topography explains most of the sub-grid structure and TESSERA adds a modest increment; for 10 m wind speed, topography explains less and the surface descriptor contributes proportionally more of the useful signal. That asymmetry is physically sensible — wind at 10 m is dominated by surface roughness, which is exactly what elevation and slope cannot see.

 ERA5 (or Aurora) coarse fields, ~25 km, multi-variable
        |
        v
 +----------------------------+
 | ConvCNP encoder (CNN)      |   grid --> continuous latent
 +----------------------------+
        |
        |  r(.)  : latent function over space
        |
        |            TESSERA embeddings @ 10 m
        |            patch centred on target site
        |                    |
        |                    v
        |          +----------------------+
        |          | learned compressor   |  thousands of
        |          | patch --> vector d   |  pixels --> one
        |          | (trained end to end) |  short descriptor
        |          +----------------------+
        |                    |
        |    topo features   |
        |    (elev, slope)   |
        |         |          |
        v         v          v
 +--------------------------------------+
 | decoder at query location x          |
 |   inputs: r(x) , topo(x) , d(x)      |
 +--------------------------------------+
        |
        v
   p( y | x )  : mean + spread, off-grid, per station

The metaphor: think of a clinic predicting a patient’s body temperature this afternoon.

The regional weather model is the ambient reading — the clinic’s thermostat says 21 C in the building. That’s true of the whole ward, so it can’t distinguish patients. The hand-crafted topography is like knowing the patient’s height and weight: real, measurable, causally relevant, and chosen in advance by whoever designed the intake form. The TESSERA embedding is a year of wearable data compressed into a constitution profile: it says nothing about this afternoon, but it says this person runs warm, sweats early, has poor circulation in the extremities. The learned compressor is the clinician who has read a thousand of these wearable dossiers and knows which two or three summary numbers actually predict deviation from ward average — as opposed to a generic wellness score, which is what you’d get if you compressed the dossier without knowing the downstream question.

The ConvCNP decoder is the consultation: ambient reading, plus the intake form, plus the constitution profile, out comes a range for this afternoon rather than a single number. And the “new station” result maps cleanly: a patient walks in who has never visited this clinic, so there are no past readings to interpolate from — but they arrive wearing the watch. Their constitution profile is available on day one. That is why the embedding helps at newly deployed stations where a purely data-driven local correction has nothing to fit.

Key Concepts

• Conditional neural process, and why “convolutional” matters: Ordinary regression learns one function from inputs to outputs. A conditional neural process learns a mapping from *a set of observations to a predictive distribution, so at test time you hand it whatever context you happen to have and it returns calibrated uncertainty without retraining. The “convolutional” version adds translation equivariance: if you slide the whole scene 50 km east, the predictions slide with it. In practice this is what makes the model generalise to stations it has never seen, because it learned a local rule (“this configuration of coarse state and surface implies this local departure”) rather than a lookup table of station identities.

• CRPS: the continuous ranked probability score, the standard way to grade a probabilistic forecast. Intuition: your forecast is a cumulative distribution curve, the truth is a step function that jumps at the observed value, and CRPS is the squared area between them. It punishes you both for being wrong on average and for lying about your confidence — a razor-sharp forecast that misses scores terribly, and a hedge-everything wide forecast also scores poorly. If your prediction is a single point, CRPS collapses to mean absolute error, which is a nice sanity anchor. One caveat for reading the paper: “CRPS *skill” is a score relative to some reference, so an 11.5% improvement in skill is not the same as an 11.5% CRPS reduction, and the reference choice matters.

• Earth-observation foundation model embeddings: instead of handing you raw satellite bands, these models pre-digest a full year of multi-sensor satellite time series per pixel and output a fixed-length vector. That vector encodes seasonal behaviour — when the field greened up, how the surface responded to rain, whether it’s permanently paved. Concretely: a wheat field and an evergreen forest can look nearly identical on one cloudless July image, but their annual trajectories are unmistakably different, and the embedding captures that difference. The paper’s bet is that this annual signature is a good proxy for the *persistent physical properties (roughness, thermal inertia, moisture availability) that set a site’s systematic offset from its grid cell.

Framework Shift

Before (mainstream approach):          After (this paper):

 coarse cell ------+                    coarse cell ----------+
                   |                                          |
 topo vector ------+--> [ConvCNP]       topo vector ----------+--> [ConvCNP]
 elev / slope /    |      |                                   |       |
 aspect / TPI      |      v             10 m EO patch         |       v
                   |   p(y|x)              |                  |    p(y|x)
                   |                       v                  |
   ^               |               [learned compressor] ------+
   |               |                       ^
 a human chose     |                       |
 which surface     |               the satellite record chose,
 properties        |               and the DOWNSTREAM LOSS chose
 matter                            which parts of it matter

One sentence: from asking a domain expert what about the ground matters to asking the downstream loss to pick it out of a satellite embedding, the core shift is that the sub-grid descriptor becomes a learned, transferable object rather than a fixed feature list.

Expert Assessment

Problem choice: Real gap, and well positioned. Downscaling literature has been iterating hard on the atmospheric side (better architectures, better generative models, diffusion) while the surface side stayed at “elevation plus a few terrain indices.” Meanwhile EO foundation models arrived and everyone in remote sensing has been looking for load-bearing downstream tasks beyond crop classification and land-cover mapping. This paper connects the two, and the timescale mismatch — annual embeddings for instantaneous prediction — is the kind of question that sounds like it shouldn’t work, which makes a positive result worth reporting. It’s an incremental contribution to a well-defined pipeline, not a new paradigm.

Method maturity: Clever plumbing rather than a new idea. The backbone is an existing ConvCNP downscaler; the contribution is one side channel and the decision to train its compressor against the downscaling loss. That decision is the actually good insight, and the framing of the surface descriptor as encoding “a location’s departure from the coarse-grid state” is a clean way to think about the problem. What I’d want checked is the cheaper alternative: a one-hot land-cover class from an off-the-shelf product like WorldCover, plus impervious fraction, plus distance to water, is a nearly free baseline that captures a lot of what “surface state” means for roughness and thermal inertia. If TESSERA only beats *topography, that’s a weak comparison; the interesting comparison is against a competent hand-crafted land-surface descriptor. There’s also a capacity confound — the embedding branch adds parameters, and a fair ablation needs a size-matched control.

Experimental integrity: The protocol is better than average. Holding stations out in *both space and time is the right thing to do and papers in this area often don’t. Five climatically distinct regions is reasonable coverage. The two robustness results carry real weight: swapping the coarse input from ERA5 to Aurora tests that the descriptor learned surface physics rather than ERA5’s specific biases, and the new-station result is the deployment scenario people actually care about. The per-variable decomposition, where topography wins for temperature and TESSERA contributes more for wind, is the kind of finding that suggests the authors were looking for the truth rather than for a headline number.

Two things I’d press on. First, leakage: TESSERA embeddings are derived from satellite observations over specific years. If the embedding years overlap the evaluation period, the embedding could encode that year’s climate anomaly at that location, not just persistent properties — a subtle and easily overlooked channel. The abstract doesn’t tell us how this was handled and it should be addressed head-on. Second, the reported gains are modest in absolute terms and expressed as skill-score percentages against an unstated reference; 6.2% on wind is small enough that it needs error bars across regions and seeds, not just a pooled average, since regional heterogeneity in downscaling results is notoriously large.

Writing quality: I’m judging mostly from the abstract, which is unusually disciplined — it states the timescale tension, gives the numbers, and admits where the contribution is smaller (temperature is mostly topography). If I could ask for one section to be expanded, it’s the compressor: patch size, embedding dimension, descriptor dimension, and architecture are exactly the knobs a reader needs to reproduce or to judge whether the gain is information or capacity. Second priority would be pushing the per-variable attribution from an observation into an analysis — if wind benefits because the embedding proxies surface roughness, show that by relating the learned descriptor to roughness estimates. That would turn an empirical result into a mechanistic one and lift the paper a full grade.

Verdict: weak accept — a well-executed, honestly evaluated demonstration that EO foundation model embeddings transfer to a task nobody built them for, held back by an architecturally thin contribution and a baseline set that likely omits the cheap land-cover competitor.

Takeaways

Things that transfer:

• Train the compressor against the downstream loss, not a reconstruction objective. The reusable move is: when a foundation model gives you far more representation than your decoder can absorb, don’t reduce it with PCA or clustering or a generic autoencoder — let the task loss decide what to keep. Applies to any pipeline where a big pretrained representation meets a small task head.

• Timescale mismatch is not automatically disqualifying. The reflex is to reject a slow descriptor for a fast prediction. The right test is whether the *residual you’re trying to explain is persistent, even if the target isn’t. Here the target (2 m temperature now) is fast but its departure from the grid-cell mean is slow. That reframing generalises: look for tasks where the bias structure is stable even though the signal is volatile — sensor calibration, per-user recommendation offsets, per-facility demand corrections.

• Hold out in space and time, and swap the upstream input. Two protocol choices worth copying. Spatial-plus-temporal holdout kills the station-memorisation shortcut. Swapping the coarse driver (ERA5 to Aurora) is a cheap, powerful test that your auxiliary feature learned physics rather than the quirks of one data product — the general form is “change the upstream model and see if your improvement survives.”

• *Report where your contribution doesn’t help. The temperature-versus-wind split makes the paper more credible, not less, and it hands the next researcher a hypothesis. Cheap to do, disproportionately valuable.

If you build site-specific weather products, the concrete action item is smaller and more immediate: your topographic descriptor stack is probably leaving surface-state information on the table, particularly for wind, and off-the-shelf EO embeddings are now an easy thing to bolt on and measure.