Strait: Priority-Aware ML Inference Serving Through Contention Modeling

Paper: 2604.28175 Authors: Haidong Zhao, Nikolaos Georgantas Categories: cs.LG

The Gap

Existing ML inference serving systems like TensorFlow Serving, TorchServe, and Triton handle request scheduling across GPUs but treat all requests equally. When you have dual-priority traffic (urgent user-facing requests mixed with batch analytics), these systems lack two critical capabilities: accurate latency prediction when multiple requests compete for GPU resources, and priority-aware scheduling that protects high-priority deadlines. Prior work either ignores interference effects during concurrent execution or relies on software preemption (context switching), which introduces overhead and fairness issues.

Problem: High-priority requests miss deadlines under GPU contention
   |
   v
Assumption: Interference is predictable + priority can guide scheduling
   |
   v
Method: Model data transfer contention + adaptive kernel interference prediction
   |                                    + priority-aware scheduler
   v
Evidence: 1.02-11.18pp fewer deadline violations for high-priority tasks
   |
   v
Conclusion: Contention modeling enables better priority differentiation than preemption

The Increment

One sentence: Before Strait, inference servers scheduled blindly under contention; after Strait, they predict interference and schedule accordingly to protect priority deadlines.

Core Mechanism

Strait has three components working in sequence. First, a contention-aware data transfer model tracks PCIe bandwidth usage and predicts delays when multiple requests transfer data simultaneously between CPU and GPU. Second, an adaptive kernel interference predictor estimates how much slower a GPU kernel runs when sharing the GPU with other kernels—this uses a lightweight profiling phase to build a model, then updates predictions online as actual interference is observed. Third, a priority-aware scheduler uses these latency predictions to decide which requests to admit and when to launch them, prioritizing high-priority requests when resources are tight.

Request arrives
   |
   v
[Contention Model] --predicts data transfer delay--> Estimated latency
   |                                                        |
   v                                                        v
[Interference Model] --predicts kernel slowdown--> Adjusted latency
   |                                                        |
   v                                                        v
[Scheduler] --admits/rejects based on priority + deadline--> GPU execution

Think of Strait like an airport managing two types of flights: commercial (high-priority) and cargo (low-priority). Most airports just queue planes first-come-first-serve, causing delays when runways get congested. Strait is like an air traffic controller who: (1) predicts taxi time based on current runway traffic (contention model), (2) estimates how much longer takeoff takes when multiple planes use the runway simultaneously (interference model), and (3) holds cargo flights on the ground when commercial flights risk missing their departure windows (priority scheduler). The controller doesn’t stop a plane mid-takeoff (no preemption)—instead, it makes smarter admission decisions upfront.

Key Concepts

• Kernel interference: When multiple GPU kernels execute concurrently, they compete for compute units, memory bandwidth, and caches, causing each kernel to run slower than it would in isolation. The slowdown isn’t uniform—it depends on which kernels are paired and their resource usage patterns. Strait measures this by profiling kernel pairs offline, then adapts the model online as it observes actual execution times. For example, two memory-intensive kernels might slow each other down by 40%, while a compute-bound kernel paired with a memory-bound one might only see 10% slowdown.

• Contention-aware data transfer: PCIe bandwidth is shared between all data transfers happening at the same time. If three requests each need to transfer 100MB from CPU to GPU, and PCIe bandwidth is 16GB/s, naive estimation would predict 6.25ms per transfer. But if all three transfer simultaneously, they share the bandwidth, so each takes ~18.75ms. Strait tracks active transfers and adjusts predictions accordingly, avoiding the optimistic bias that causes deadline misses.

• Priority-aware admission control: Instead of accepting all requests and hoping for the best, Strait predicts whether a new request can meet its deadline given current GPU load. For low-priority requests, it admits them only if there’s slack. For high-priority requests, it may reject low-priority requests already queued to make room. This is fundamentally different from preemption (interrupting running tasks) because it makes decisions before execution starts, avoiding the overhead of context switching.

Framework Shift

Before (mainstream approach):        After (this paper):

Request queue (FIFO)                 Request queue (priority-sorted)
   |                                    |
   v                                    v
Scheduler (blind to contention)      Contention predictor
   |                                    |
   v                                    v
GPU (concurrent execution)           Interference predictor
   |                                    |
   v                                    v
Measure actual latency               Priority-aware scheduler
   |                                    |
   v                                    v
React to deadline misses             GPU (controlled concurrency)

From reactive scheduling to predictive admission control, the core shift is moving the intelligence from post-execution measurement to pre-execution estimation.

Expert Assessment

Problem choice: Real gap. On-premises deployments often mix latency-sensitive and batch workloads on shared GPU clusters, and existing systems genuinely struggle with priority differentiation. The problem sits at the intersection of systems and ML, which is currently active territory.

Method maturity: Solid engineering rather than algorithmic novelty. The contention model is straightforward queuing theory, and the interference predictor is a simple regression model updated online. The insight is recognizing that these two sources of unpredictability (data transfer + kernel execution) need separate modeling. No simpler approach is being overlooked—this is the right level of complexity for the problem.

Experimental integrity: Baselines are fair (compares against Clockwork, a strong prior system, and software preemption). The 1.02-11.18pp improvement range is honest reporting—shows the method works better in some scenarios than others. One weakness: experiments use synthetic workloads with fixed priority ratios. Real deployments have bursty, unpredictable priority distributions, and it’s unclear how Strait adapts when the workload shifts suddenly.

Writing quality: The paper front-loads motivation well but buries implementation details. Section 4 (system design) would benefit from a clearer separation between offline profiling and online adaptation—readers have to piece together the workflow from scattered paragraphs. The evaluation section is thorough but could use a failure case analysis: when does Strait’s prediction model break down?

Verdict: weak accept — Solves a real problem with practical engineering, but the contribution is incremental rather than transformative.

Takeaways

Practitioners building multi-tenant GPU systems can steal two ideas: (1) Model data transfer and compute interference separately—they have different characteristics and need different prediction strategies. (2) Admission control beats preemption for priority differentiation when tasks are short-lived (inference requests). Preemption makes sense for long-running training jobs, but for inference, the overhead of context switching often exceeds the benefit. If you’re building a serving system, invest in better latency prediction rather than fancier preemption mechanisms.