Flow-Guided Neural Operators: Dynamic Noise Levels for Self-Supervised Time-Series Learning

Paper: 2602.12267 Authors: Duy Nguyen, Jiachen Yao, Jiayun Wang, Julius Berner, Animashree Anandkumar Categories: cs.LG

Abstract

This paper presents Flow-Guided Neural Operator (FGNO), a novel self-supervised learning framework for time-series data that treats corruption level as a learnable parameter rather than a fixed hyperparameter. Unlike traditional masked autoencoders that use predetermined masking ratios, FGNO combines operator learning with flow matching to learn mappings in functional spaces. The method leverages Short-Time Fourier Transform to handle multiple time resolutions and extracts hierarchical features by sampling from different network layers and flow times. Evaluated on three biomedical datasets, FGNO demonstrates substantial improvements: 35% AUROC gains in neural signal decoding, 16% RMSE reduction in temperature prediction, and over 20% accuracy improvement in sleep stage classification under low-data conditions.

Key Contributions

• Dynamic Corruption Framework: Introduces corruption level as a continuous degree of freedom, moving beyond fixed masking ratios in traditional SSL methods
• Operator Learning Integration: Combines neural operators with flow matching to learn functional space mappings that unify different time resolutions via Short-Time Fourier Transform
• Hierarchical Feature Extraction: Extracts multi-scale representations by tapping into different network layers and flow times, capturing both low-level patterns and high-level semantics
• Clean Inference Protocol: Uses clean inputs during inference while training with noise, eliminating randomness and improving downstream task accuracy
• Strong Empirical Results: Achieves state-of-the-art performance across three biomedical domains, particularly excelling in low-data regimes

Methodology: Flow Matching Meets Operator Learning

The core innovation of FGNO lies in its treatment of the corruption process as a continuous flow. Traditional SSL methods like MAEs apply a fixed masking ratio $r$ to all samples, limiting their ability to capture features at different abstraction levels. FGNO instead models the corruption as a time-dependent process parameterized by flow time $t \in [0, 1]$.

The framework operates in functional spaces using neural operators, which learn mappings between function spaces rather than finite-dimensional vectors. This is particularly powerful for time-series data where signals can have varying lengths and sampling rates. The Short-Time Fourier Transform (STFT) serves as the bridge, converting time-series into a unified frequency-time representation:

$\text{STFT}(x)[n, k] = \sum_{m} x[m] w[m-n] e^{-j2\pi km/N}$

where $w$ is a window function, $n$ is the time index, and $k$ is the frequency bin.

The flow matching objective trains the model to predict the direction of the flow at any corruption level $t$, learning a velocity field $v_\theta(x_t, t)$ that connects clean data $x_0$ to increasingly corrupted versions. During training, the model sees inputs at various corruption levels, learning to denoise from any intermediate state. This multi-level training enables the extraction of features at different semantic levels by querying the model at specific flow times.

Hierarchical Representation Extraction

A key insight of FGNO is that different corruption levels and network depths capture complementary information. Early layers with low corruption ($t \approx 0$) preserve fine-grained temporal patterns, while deeper layers with higher corruption ($t \approx 1$) capture global structure and long-range dependencies.

The authors propose a systematic feature extraction strategy:

1. Layer-wise features: Extract activations from multiple network depths to capture hierarchical abstractions
2. Flow-time features: Sample representations at different $t$ values during the denoising process
3. Aggregation: Combine these multi-scale features through concatenation or learned pooling

This approach yields a rich feature bank without requiring multiple models or training runs. For a specific downstream task, practitioners can select the optimal combination of layers and flow times, or use all features with a lightweight classifier.

Critically, during inference for downstream tasks, FGNO uses clean inputs ($t=0$) rather than noisy ones. This contrasts with some generative SSL methods that inject noise during inference, introducing unwanted randomness. By training with noise but inferring with clean data, FGNO achieves both robust representations and deterministic predictions.

Experimental Results and Analysis

The authors evaluate FGNO on three challenging biomedical time-series benchmarks:

BrainTreeBank (Neural Signal Decoding): This dataset involves decoding cognitive states from neural recordings. FGNO achieves up to 35% improvement in AUROC over baselines including SimCLR, BYOL, and standard MAEs. The gains are most pronounced in low-data scenarios where only 10-20% of labeled data is available, demonstrating FGNO’s superior sample efficiency.

DREAMT (Skin Temperature Prediction): For physiological signal forecasting, FGNO reduces RMSE by 16% compared to supervised baselines and other SSL methods. The continuous corruption framework proves particularly effective for capturing the multi-scale temporal dynamics in circadian rhythms and acute temperature changes.

SleepEDF (Sleep Stage Classification): In this multi-class classification task with severe class imbalance, FGNO improves both accuracy and macro-F1 by over 20% in low-data regimes. The hierarchical features enable better discrimination of subtle differences between sleep stages, which often manifest at different temporal scales.

Ablation studies reveal that both the flow-based corruption and operator learning components are essential. Removing flow matching and reverting to fixed corruption reduces performance by 12-18%, while replacing the neural operator with standard convolutions decreases accuracy by 8-15%. The STFT-based functional representation is particularly important for handling variable-length sequences and multi-resolution analysis.

Implications and Future Directions

FGNO represents a significant advance in self-supervised learning for time-series data, with several broader implications:

Theoretical: The work bridges operator learning, flow matching, and SSL, opening new avenues for theoretical analysis of representation learning in functional spaces. The continuous corruption framework provides a more principled way to think about data augmentation and invariance learning.

Practical: The method’s strong performance in low-data regimes makes it particularly valuable for domains like healthcare and scientific computing where labeled data is expensive or scarce. The single-model, multi-scale approach is also computationally efficient compared to training separate models for different tasks.

Limitations and Extensions: While FGNO excels on biomedical data, its performance on other time-series domains (finance, climate, audio) remains to be thoroughly evaluated. The reliance on STFT may be suboptimal for signals with non-stationary or highly irregular patterns. Future work could explore adaptive time-frequency representations or learnable basis functions.

The clean-input inference strategy, while effective, raises questions about the train-test distribution gap. Investigating whether intermediate corruption levels during inference could provide uncertainty estimates or robustness to noisy inputs is an interesting direction.

Finally, extending FGNO to multivariate time-series with complex dependencies, or to spatiotemporal data, could unlock applications in climate modeling, traffic prediction, and video understanding.

Takeaways

1. Treating corruption level as a continuous parameter rather than a fixed hyperparameter enables more flexible and powerful self-supervised learning for time-series data
2. Combining neural operators with flow matching provides a principled framework for learning in functional spaces, naturally handling variable-length sequences and multiple resolutions
3. Hierarchical feature extraction from different network layers and flow times captures complementary information from fine-grained patterns to global structure
4. Using clean inputs during inference while training with noise eliminates randomness and improves downstream task performance
5. FGNO achieves substantial gains (up to 35%) over established baselines across multiple biomedical domains, particularly excelling in low-data regimes where labeled samples are scarce