Impermanent: Testing Time Series Models on Data That Won't Sit Still

Paper: 2603.08707 Authors: Azul Garza, Renée Rosillo, Rodrigo Mendoza-Smith, David Salinas, Andrew Robert Williams, Arjun Ashok, Mononito Goswami, José Martín Juárez Categories: cs.LG

The Gap

Time series foundation models are making bold claims about generalization, but we’re testing them with broken thermometers. Current benchmarks use static train-test splits from datasets frozen in time. This creates three problems: (1) test data can leak into training as models get retrained, (2) researchers can tune hyperparameters on test scores since the test set never changes, and (3) we measure one-off accuracy on a snapshot rather than sustained performance under temporal drift.

The field has reached a point where we can build models that memorize patterns, but we can’t tell if they adapt to change. Prior work (M4, Monash forecasting archive, etc.) provides valuable datasets but all share the static split limitation. This paper asks: what if the test set kept moving?

Problem: Static benchmarks can't detect temporal brittleness
    |
    v
Assumption: Real forecasting requires adapting to ongoing change
    |
    v
Method: Live benchmark with daily-updated GitHub activity streams
    |
    v
Evidence: Infrastructure + protocols for continuous evaluation
    |
    v
Conclusion: Shift from snapshot accuracy to sustained performance

The Increment

One sentence: Before this paper, we tested forecasting models on frozen datasets; after, we can test them on data streams that update daily, exposing temporal fragility that static benchmarks hide.

Core Mechanism

Impermanent builds on a simple insight: GitHub activity is naturally live and highly non-stationary. The benchmark tracks four time series types from the top 400 repositories by star count: issues opened, pull requests opened, push events, and new stargazers. Each day, new data arrives and models must forecast the next window.

The evaluation protocol uses rolling windows. A model makes forecasts, the actual data arrives, performance is scored, and the window slides forward. This happens continuously with daily updates. The system maintains standardized protocols and live leaderboards so researchers can submit models and track performance over time without touching the test data directly.

The data source matters: GitHub activity exhibits real distributional shifts from software releases, changing contributor behavior, platform updates, and external events (like a popular library getting mentioned in a viral post). These aren’t synthetic perturbations—they’re the actual temporal dynamics that forecasting models claim to handle.

GitHub API --> Data Collection --> Time Series Construction
                                          |
                                          v
                    [Issues, PRs, Pushes, Stars] x 400 repos
                                          |
                                          v
                              Rolling Window Evaluation
                                          |
                    +---------------------+---------------------+
                    |                     |                     |
                    v                     v                     v
              Model submits          Actual data           Score & update
              forecasts              arrives daily         leaderboard
                    |                     |                     |
                    +---------------------+---------------------+
                                          |
                                          v
                                  Window slides forward
                                  (repeat daily)

Think of this like a weather forecasting competition, but instead of waiting for the season to end and grading everyone’s predictions at once, you’re grading forecasts every single day as the weather actually happens. A model that memorized historical patterns might do well initially, but when an unusual cold front arrives (analogous to a major GitHub platform change), you immediately see which models adapt and which ones break. The “live” aspect isn’t just a gimmick—it’s the core mechanism that prevents overfitting to a fixed test set and forces models to demonstrate temporal robustness rather than pattern memorization.

Key Concepts

• Temporal contamination: In static benchmarks, the test set is fixed and known. As foundation models get retrained or researchers iterate, test data can inadvertently influence training (direct leakage) or model selection (indirect leakage through hyperparameter tuning on test scores). It’s like studying for an exam when you already know the questions. Live benchmarks prevent this because the test data doesn’t exist yet—you can’t memorize tomorrow’s GitHub activity today.

• Distributional shift: Time series data doesn’t stay still. A repository’s activity pattern changes when a major release drops, when core contributors leave, or when a competing tool emerges. Static benchmarks capture one slice of time, but they can’t tell you if a model will handle the shift from “steady growth” to “sudden spike” to “gradual decline.” Impermanent’s GitHub data naturally exhibits these shifts, making them part of the evaluation rather than an afterthought.

• Rolling window evaluation: Instead of train-once-test-once, you continuously slide a window forward through time. At each step: forecast the next period, wait for actual data, score the forecast, move forward. This mimics real deployment where you’re constantly making predictions about the immediate future. A model might nail the first month but degrade over time as patterns drift—rolling windows catch this degradation that static splits miss.

Framework Shift

Before (static benchmarks):          After (Impermanent):

[Train Data]-->[Test Data]           [Train]-->[Test Day 1]
     ^              |                    |          |
     |              v                    v          v
  Fixed split   One-time score       [Test Day 2]
                                         |
                                         v
                                     [Test Day 3]
                                         |
                                         v
                                     [Test Day N...]
                                         |
                                         v
                                   Continuous scoring

From measuring accuracy on a frozen moment to measuring sustained performance through ongoing change, the core shift is from snapshot evaluation to temporal stress testing.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The time series community has been building increasingly complex foundation models while using evaluation protocols designed for simpler methods. The contamination risk is genuine—I’ve seen papers where “test” performance suspiciously improves in later versions as models get retrained. The timing is right: foundation models are mature enough that we need better evaluation infrastructure.

Method maturity: This is infrastructure work, not algorithmic innovation. The core idea (live benchmarks) isn’t new—Kaggle has done this for years. What’s new is applying it systematically to time series forecasting with standardized protocols. The choice of GitHub data is clever (naturally live, relevant to ML practitioners) but also limiting (single domain, potentially biased toward tech trends). A simpler approach might be to just use multiple static benchmarks with different time periods, but that doesn’t solve the contamination problem.

Experimental integrity: This paper doesn’t present experimental results—it presents a benchmark. The integrity question is whether the infrastructure is sound. The protocols seem reasonable (rolling windows, daily updates, standardized submission). The main risk is maintenance: will this actually stay live? Many “live” benchmarks die after a year when funding runs out. The GitHub dependency is both a strength (free, reliable API) and weakness (what if GitHub changes their API or rate limits?).

Writing quality: The paper is clear about what it is (infrastructure) and what it isn’t (a new forecasting method). The motivation is well-articulated. However, the paper would benefit from more discussion of limitations—why only 400 repos? How representative is GitHub activity of other forecasting domains? What happens when a repo gets archived or deleted? The related work section could better position this against other live benchmark efforts in ML.

Verdict: weak accept — Addresses a real methodological gap with practical infrastructure, but limited to one domain and success depends on long-term maintenance commitment.

Takeaways

If you’re building forecasting models, steal the evaluation philosophy: test on data that keeps changing, not frozen snapshots. Even if you can’t set up a live benchmark, you can simulate it by evaluating on multiple time periods and measuring performance degradation over time.

For benchmark designers in other domains: the “live” aspect prevents contamination and forces temporal robustness. Consider whether your field has naturally updating data sources (social media, sensor networks, financial markets) that could support similar infrastructure.

The specific technical contribution here is modest (it’s mostly engineering), but the conceptual shift matters: from “does this model work on this dataset?” to “does this model keep working as the world changes?” That framing transfers to any domain where temporal dynamics matter.