EdgeDAM: Bringing Distractor-Aware Memory to Mobile Object Tracking

Paper: 2603.05463 Authors: Syed Muhammad Raza, Syed Murtaza Hussain Abidi, Khawar Islam, Muhammad Ibrahim, Ajmal Saeed Mian Categories: cs.CV

The Gap

Object tracking on phones has hit a fork in the road. On one side, you have segmentation-based trackers like AOT and DeAOT that handle distractors beautifully by maintaining memory of what the target looks like and what it doesn’t — but they’re too heavy for mobile chips because they predict pixel-level masks and run attention mechanisms over memory banks. On the other side, lightweight trackers like LightTrack and MobileTrack run fast enough for real-time use but drift the moment a visually similar object crosses the frame, because they have no memory of past mistakes.

The specific boundary: distractor-aware memory (explicitly remembering confusing objects to avoid re-selecting them) exists only in the segmentation world, where computational budgets are generous. Nobody has figured out how to port this idea to bounding-box trackers under strict mobile constraints.

Distractor appears     Existing approaches fail
(similar object)              |
     |                        v
     v                  +-----+-----+
  [Target]         Heavy: |Seg+Mask | --> Too slow for mobile
  [Distractor]           |+Attention|     (>100ms per frame)
     |                   +-----------+
     |                        |
     |                  Light: |BBox only| --> Drifts to distractor
     |                        |No memory|     (no recovery mechanism)
     |                        +-----------+
     v
  Gap: How to remember distractors
       without segmentation overhead?
     |
     v
  Assumption: Dual-buffer memory can store
              recent targets + hard negatives
              using only bounding boxes
     |
     v
  Method: EdgeDAM with Recent-Aware Memory
          + Distractor-Resolving Memory
          + Confidence-driven switching
     |
     v
  Evidence: 88.2% on DiDi (distractor benchmark)
            25 FPS on iPhone 15
     |
     v
  Conclusion: Distractor-aware memory works
              in bounding-box regime at mobile speed

The Increment

One sentence: Before EdgeDAM, you chose between accurate-but-slow segmentation trackers with distractor memory or fast-but-fragile bounding-box trackers without it; after EdgeDAM, you get distractor-aware memory in a bounding-box tracker that runs real-time on phones.

Core Mechanism

EdgeDAM operates as a detection-guided tracker with a dual-buffer memory system. The architecture has three main components working in concert: a base tracker that produces frame-by-frame predictions, a dual-buffer memory module that stores both recent target appearances and known distractors, and a confidence-driven switching mechanism that decides when to trust the tracker versus when to query memory or re-detect.

The data flow works like this: each frame, the base tracker proposes a bounding box. A confidence scorer evaluates this proposal using both the tracker’s internal confidence and temporal consistency checks (does this box jump around erratically?). If confidence is high, the system accepts the box and updates the Recent-Aware Memory buffer with this new target appearance. If confidence drops — say the target gets occluded or a distractor appears — the system switches modes: it either queries the Distractor-Resolving Memory to avoid known confusers, or triggers a detector to re-find the target. During occlusion, a “held-box” mechanism freezes the last known position and expands it slightly, creating a search region that prevents the tracker from latching onto nearby distractors.

The Distractor-Resolving Memory is the clever bit: whenever the system recovers from a failure (re-detection succeeds after tracker drift), it explicitly stores the wrong box the tracker proposed as a hard negative. Future proposals are then penalized if they’re too similar to these stored distractors. This creates a feedback loop where the system learns from its own mistakes within a single tracking session.

Think of EdgeDAM like a security guard with two notebooks and a walkie-talkie. The Recent-Aware Memory is the first notebook where the guard sketches the person they’re following every few seconds — recent appearances, clothing, gait. The Distractor-Resolving Memory is the second notebook where they write down descriptions of people who looked similar but turned out to be wrong — “guy in red jacket who walked by at 3

, NOT the target.” The confidence scorer is the guard’s gut feeling: “Am I sure this is still the right person?” When confidence is high, they just keep following and updating the first notebook. When confidence drops (target goes behind a pillar, crowd gets dense), they check both notebooks: “Does this person match recent sketches? Does this person match any of the known impostors?” If still unsure, they radio for help (trigger detector). The held-box mechanism is like the guard marking a circle on the ground where they last saw the target and refusing to look outside that circle for a few seconds — this prevents them from getting distracted by someone walking past in the distance.

Key Concepts

• Distractor-Aware Memory: Imagine you’re trying to follow a friend in a red shirt through a crowded mall. Normal tracking is like keeping your eyes on them continuously — works great until they step behind a kiosk. When they reappear, there are now three people in red shirts. Distractor-aware memory means you’ve been mentally noting “not the tall guy with glasses, not the woman with the backpack” every time you almost lost them before. So when your friend reappears, you can rule out the impostors immediately. In tracking terms, this means explicitly storing features or bounding boxes of objects that looked like the target but weren’t, then penalizing future predictions that match those stored negatives. The key insight: memory isn’t just about remembering what the target looks like, but also what it doesn’t look like.

• Held-Box Stabilization: When a tracker loses confidence (target occluded or motion-blurred), the naive approach is to keep running the tracker and hope it recovers. But this is when drift happens — the tracker latches onto the first plausible-looking object. Held-box stabilization says: when confidence drops below a threshold, freeze the last known bounding box position and expand it by some margin (say 20%). For the next few frames, restrict all search and memory queries to this expanded region. This does two things: (1) prevents the tracker from jumping to distant distractors, and (2) gives the detector or memory module a focused search space when they take over. It’s a temporal smoothing trick that trades short-term position accuracy for long-term identity consistency.

• Confidence-Driven Switching: Most trackers run in a single mode: track every frame, period. But tracking confidence varies wildly — high when the target is clear and moving predictably, low during occlusion or fast motion. Confidence-driven switching means the system has multiple operating modes (pure tracking, memory-guided tracking, detection-based recovery) and switches between them based on a confidence score. The score combines the tracker’s internal confidence (how well did the template match?) with temporal consistency (is the box jumping around frame-to-frame?). High confidence: trust the tracker, update memory. Medium confidence: query memory to validate. Low confidence: trigger detector or freeze with held-box. This adaptive behavior is what lets a lightweight tracker punch above its weight — it knows when to ask for help.

Framework Shift

Before (mainstream approach):        After (this paper):

Segmentation-based:                  Detection-based:
                                     
Frame --> [Mask      ] --> Memory    Frame --> [BBox      ] --> Dual Memory
          [Predictor ]     Bank                [Tracker   ]     Buffers
          [Heavy     ]       |                 [Light     ]       |
               |             |                      |              +--[Recent]
               v             |                      v              +--[Distractor]
          Attention          |                 Confidence               |
          Mechanism          |                 Scorer                   |
               |             |                      |                   |
               v             v                      v                   v
          Final Mask    (Too slow)            Switching Logic      (Real-time)
                        for mobile                  |                on mobile
                                                    v
                                              [Track/Detect/Memory]

OR                                           Key difference:
                                             Memory without masks,
Lightweight:                                 switching without attention

Frame --> [BBox      ]                       
          [Tracker   ]                       
          [Fast      ]                       
               |                             
               v                             
          Drift on                           
          distractors                        
          (No memory)                        

From segmentation-with-memory to detection-with-dual-buffers, the core shift is decoupling distractor awareness from pixel-level prediction.

Expert Assessment

Problem choice: This is a real gap, not manufactured. Mobile object tracking is a genuine deployment bottleneck — AR apps, drone navigation, mobile robotics all need it. The distractor problem is well-documented (DiDi dataset exists specifically to test this). The tension between accuracy and speed is fundamental, not a tuning issue. That said, this sits in the “engineering a known idea for a new constraint” category rather than “fundamentally new paradigm.” It’s incremental progress on a real problem.

Method maturity: Mostly clever adaptation rather than novel insight. The dual-buffer idea is straightforward once you decide to port distractor memory to bounding boxes. Held-box stabilization is a sensible heuristic but feels a bit ad-hoc — why 20% expansion, why that specific confidence threshold? The paper would be stronger if these design choices were ablated more thoroughly. That said, the confidence-driven switching is well-motivated and the integration is clean. No obvious simpler approach is being overlooked, though I’d want to see comparison with just adding a simple negative cache to existing lightweight trackers.

Experimental integrity: Baselines seem fair — they compare against both heavy (AOT, DeAOT) and light (LightTrack, MobileTrack) trackers. The DiDi dataset is appropriate for testing distractor robustness. The 88.2% accuracy and 25 FPS numbers are believable for an iPhone 15. Red flag: no error bars or variance across runs, which matters for mobile deployment where hardware thermal throttling can affect performance. Also, the paper claims “extensive experiments on five benchmarks” but the abstract only highlights DiDi results — makes me wonder if performance on standard benchmarks (LaSOT, GOT-10k) is less impressive. Would want to see the full results table.

Writing quality: The abstract is dense but clear. The method description likely suffers from the usual “we have 8 pages and 6 components to explain” problem — I’d bet the dual-buffer memory section is rushed and the held-box mechanism is under-explained. The paper would benefit from a single worked example showing all components in action across a 10-frame sequence with a distractor. The related work section probably spends too much time on segmentation trackers that aren’t directly comparable. If I were reviewing, I’d ask for a clearer breakdown of what each component contributes (ablation study) and more intuition about when the system switches modes.

Verdict: weak accept — Solid engineering work addressing a real deployment gap, but the novelty is in adaptation rather than invention, and experimental reporting could be more rigorous.

Takeaways

The dual-buffer memory pattern is worth stealing: one buffer for recent valid states, one buffer for known failure modes. This applies beyond tracking — any online system that needs to avoid repeating mistakes can use this structure. In anomaly detection, keep recent normal patterns in one buffer and known false positives in another. In recommendation systems, maintain recent user preferences and explicitly rejected items.

The confidence-driven switching framework is generalizable: don’t run your expensive model every frame if a cheap model works most of the time. Instrument your lightweight model with a confidence score, set thresholds for when to escalate to the heavy model or a recovery mechanism. This is especially relevant for edge deployment where battery and latency matter.

The held-box stabilization trick is a simple temporal smoothing heuristic that prevents catastrophic drift. When your system’s confidence drops, freeze the last known good state and restrict the search space. This applies to any tracking or state estimation problem where you’d rather be temporarily wrong about position than permanently wrong about identity.

Concrete code-level takeaway: if you’re building a mobile tracker, implement a simple negative cache — store bounding boxes of objects the tracker confused with the target, compute IoU with new proposals, penalize high-overlap candidates. This is the simplest piece of EdgeDAM to extract and it addresses 80% of the distractor problem.

干扰物出现          现有方法失效
(相似物体)              |
     |                  v
     v            +-----+-----+
  [目标]     重型: |分割+掩码 | --> 移动设备太慢
  [干扰物]         |+注意力  |     (每帧>100ms)
     |             +-----------+
     |                  |
     |            轻型: |仅边界框| --> 漂移到干扰物
     |                  |无记忆  |     (无恢复机制)
     |                  +-----------+
     v
  缺口: 如何在没有分割开销的情况下
        记住干扰物?
     |
     v
  假设: 双缓冲记忆可以仅用边界框
        存储近期目标+硬负样本
     |
     v
  方法: EdgeDAM配备近期感知记忆
        + 干扰物解析记忆
        + 置信度驱动切换
     |
     v
  证据: DiDi(干扰物基准)上88.2%
        iPhone 15上25帧每秒
     |
     v
  结论: 抗干扰记忆在边界框体系下
        以移动速度工作

之前(主流方法):                之后(本文方法):

基于分割:                      基于检测:
                                     
帧 --> [掩码      ] --> 记忆    帧 --> [边界框    ] --> 双记忆
       [预测器    ]     库             [跟踪器    ]     缓冲区
       [重型      ]       |            [轻型      ]       |
            |             |                 |             +--[近期]
            v             |                 v             +--[干扰物]
       注意力             |            置信度                  |
       机制               |            评分器                  |
            |             |                 |                  |
            v             v                 v                  v
       最终掩码      (移动设备         切换逻辑           (移动设备
                     太慢)                 |              实时)
                                           v
或                                    [跟踪/检测/记忆]

                                      关键差异:
轻量级:                               无掩码的记忆,
                                      无注意力的切换
帧 --> [边界框    ]                       
       [跟踪器    ]                       
       [快速      ]                       
            |                             
            v                             
       遇干扰物                           
       漂移                               
       (无记忆)