CHARM: Teaching Time-Series Models to Speak Through Channel Descriptions

Paper: 2605.31580 Authors: Utsav Dutta, Gerardo Pastrana, Sina Khoshfetrat Pakazad, Henrik Ohlsson Categories: cs.LG

The Gap

Time-series foundation models have lagged behind language and vision. Existing approaches treat channels as anonymous vectors—a temperature sensor and a pressure sensor are just “channel 1” and “channel 2.” This creates three problems: (1) models can’t transfer knowledge across datasets where the same sensor appears in different positions, (2) learned representations lack interpretability since there’s no semantic grounding, and (3) models are brittle to channel permutations despite channels having no inherent order.

Prior work either ignores channel semantics entirely (treating multivariate series as opaque vectors) or uses channel-specific parameters that don’t generalize. Meanwhile, language models have shown that grounding in text descriptions enables transfer and interpretability—but nobody has systematically incorporated this into time-series architectures.

Problem: Anonymous channels block transfer & interpretability
    |
    v
Assumption: Text descriptions can ground channel semantics
    |
    v
Method: JEPA + channel-aware gating + order equivariance
    |
    v
Evidence: Linear probe performance across 4 task types
    |
    v
Conclusion: Text acts as channel ID, JEPA drives performance

The Increment

One sentence: Before—time-series models treat channels as anonymous slots; after—channels carry semantic identities through text descriptions, enabling cross-dataset transfer and interpretable attention patterns.

Core Mechanism

CHARM has three interlocking pieces. First, each channel gets a text description (e.g., “outdoor temperature sensor” or “heart rate monitor”) that’s encoded by a frozen language model into a semantic vector. Second, a Transformer encoder processes the time-series data, but instead of standard attention, it uses description-aware gating: the text embeddings modulate how channels attend to each other, creating interpretable cross-channel relationships. Third, the architecture is permutation equivariant—shuffling channel order doesn’t change the output because channels are identified by their descriptions, not positions.

Training uses JEPA (Joint Embedding Predictive Architecture): the model sees a corrupted view of the time series (masked timesteps, added noise) and predicts the representation of a clean view in latent space. This forces the encoder to learn robust, noise-invariant features. A novel loss term encourages temporal stability—nearby timesteps should have similar embeddings—which prevents the model from collapsing to trivial solutions.

Text: "temp sensor"  "pressure"  "humidity"
  |         |            |
  v         v            v
[LM] --> [embed_1] [embed_2] [embed_3]
              |         |         |
              v         v         v
Time series: [x1(t)] [x2(t)] [x3(t)]
              |         |         |
              +----+----+----+----+
                   |
                   v
         [Transformer Encoder]
          (gated by text embeds)
                   |
                   v
              [z_corrupted]
                   |
                   v
            [Latent Predictor]
                   |
                   v
         predict [z_clean] <-- target

Think of CHARM as a multilingual conference interpreter. Each sensor speaks its own “language” (raw signal), but the interpreter (Transformer) has a cheat sheet (text description) telling them who’s speaking. When the temperature sensor talks, the interpreter knows to pay attention to related sensors (humidity, pressure) because the cheat sheet says they’re related. If someone shuffles the seating arrangement (channel order), the interpreter still knows who’s who by checking the cheat sheet. During training, the interpreter practices by listening to noisy audio (corrupted series) and trying to reconstruct what the clean version would sound like (latent prediction). The cheat sheet ensures the interpreter learns meaningful relationships, not just memorizing seat positions.

Key Concepts

• Permutation Equivariance: Imagine a recipe where ingredients are listed in random order. A permutation-equivariant model reads “flour, eggs, sugar” and “sugar, flour, eggs” as the same recipe because it identifies ingredients by name, not position. For time series, this means the model recognizes “temperature sensor” whether it’s channel 1 or channel 5. Technically, if you permute the input channels, the output embeddings permute identically—the function commutes with permutation operations. This is crucial because sensor ordering is arbitrary (a hardware accident), not meaningful.

• JEPA (Joint Embedding Predictive Architecture): Instead of reconstructing raw pixels or tokens (which wastes capacity on irrelevant details), JEPA predicts representations. You show the model a blurry photo and ask it to predict what the embedding of the sharp photo would be—not the sharp photo itself. For time series: corrupt the signal with noise and masking, encode it to get z_corrupted, then predict what z_clean would be. This forces the model to learn features robust to noise (critical for real sensors) while ignoring pixel-level details that don’t matter for downstream tasks.

• Description-Aware Gating: Standard attention lets every token attend to every other token equally. Description-aware gating uses text embeddings to modulate attention weights—if two channels have semantically related descriptions, they attend more strongly. Concretely: compute attention scores, then multiply by a gating function of the text embeddings. This creates interpretable attention patterns (you can see the model learned that temperature and humidity sensors interact) and enables transfer (the model knows “temperature” in dataset A relates to “temperature” in dataset B, even if they’re different physical sensors).

Framework Shift

Before (channel-agnostic):          After (CHARM):

Input: [x1, x2, x3, ...]            Input: [x1, x2, x3, ...]
         |                                   |
         v                                   v
    [Positional                      [Text: "temp", "press", ...]
     Encoding]                              |
         |                                  v
         v                            [Semantic Embed]
    [Transformer]                           |
    (position-based                         v
     attention)                      [Transformer + Gating]
         |                           (description-aware
         v                            attention)
    [Task Head]                             |
                                            v
                                       [Task Head]

Channels identified by:             Channels identified by:
- Position in array                 - Semantic description
- Learned positional codes          - Text embedding

Transfer: Breaks on reordering      Transfer: Works across datasets

One sentence: From position-indexed anonymous vectors to semantically-grounded, order-invariant channel representations—the core shift is treating sensors as named entities rather than array slots.

Expert Assessment

Problem choice: Real gap. Time-series foundation models are genuinely behind language/vision, and the lack of semantic grounding is a legitimate bottleneck for transfer learning. The problem sits at the intersection of multimodal learning and time-series analysis—timely given recent foundation model trends.

Method maturity: Solid engineering, not a breakthrough insight. JEPA is borrowed from vision (I-JEPA), permutation equivariance is standard in graph networks, and text conditioning is everywhere in multimodal models. The contribution is the integration—showing these pieces work together for time series. The description-aware gating is the most novel component, but it’s a straightforward extension of cross-attention. No simpler approach is obviously overlooked, though one could argue for just using channel IDs (integers) instead of text—the paper shows text helps but isn’t the main driver.

Experimental integrity: Baselines are fair (standard Transformers, TSMixer, PatchTST). The ablation study is thorough—they show JEPA matters more than text, which is honest. However, the datasets are relatively small-scale (UCR archive, a few real-world datasets), and there’s no comparison to recent time-series foundation models like TimesFM or Chronos. The claim about “strong performance with linear probes” is supported but not earth-shattering—linear probe accuracy is respectable but not SOTA. One red flag: no discussion of computational cost (text encoding + gating adds overhead).

Writing quality: The abstract and intro are crisp. The method section is dense—Figure 2 (architecture diagram) does heavy lifting but could be clearer about data flow. The ablation study (Section 4.3) is the paper’s strongest section, showing intellectual honesty about what drives performance. The related work section is perfunctory—misses recent time-series SSL work. If I were reviewing, I’d ask them to expand the discussion section: why does JEPA work so well for time series specifically? What’s the failure mode when text descriptions are poor?

Verdict: weak accept — Solid execution of a reasonable idea with honest evaluation, but incremental rather than transformative. The JEPA + equivariance combination is the real contribution; text is a useful but secondary feature. Belongs at a good venue (ICML/NeurIPS workshop or domain conference) but not a spotlight paper.

Takeaways

Steal the JEPA training recipe for time series: Predicting latent representations instead of raw reconstructions is a clean win for noisy sensor data. The temporal stability loss (nearby timesteps should have similar embeddings) is a simple regularizer you can drop into any time-series SSL setup.

Use text as channel metadata, not magic: The paper shows text descriptions help with transfer but aren’t the performance driver—they’re essentially learned channel IDs. If you’re building a time-series model, invest in the pretraining objective (JEPA) first, add text conditioning second. For practitioners: if you have heterogeneous sensor data across deployments, maintaining a simple text description schema (even just “sensor_type: temperature, unit: celsius”) enables this kind of transfer.

Permutation equivariance is table stakes for multivariate time series: If your model breaks when you reorder channels, you’re learning spurious positional correlations. Use set-based architectures (attention without positional encoding, or explicit permutation-equivariant layers) to avoid this trap.

Linear probe performance is a useful sanity check: If your pretrained representations require a complex task head to be useful, the pretraining might not be learning the right features. CHARM’s linear probe results suggest the JEPA objective is doing the heavy lifting—a good sign for transfer learning.