PNPTrack tracking a target in complex scenarios (Image by SIA)

Visual object tracking is a fundamental computer vision task that aims to continuously localize a specific target or region across successive video frames.It holds significant practical value in fields such as autonomous driving and security surveillance. However, conventional tracking approaches relying purely on initial templates suffer from limited robustness amid target deformation, occlusion and dramatic appearance variations. To boost tracking stability, such methods commonly incorporate spatiotemporal cues derived from historical video frames.

Existing methods that exploit spatiotemporal information generally face two problems. On one hand, spatiotemporal modules are deeply coupled with the backbone network, often requiring full-parameter retraining from scratch, which incurs high training costs. On the other hand, introducing dedicated spatiotemporal processing modules increases inference complexity, and since they must be executed serially after the backbone network during inference, inference latency is increased, making it difficult to achieve satisfactory results on high-performance backbones.

To address this issue, a research team from the Opto-Electronic Information Technology Department of the Shenyang Institute of Automation(SIA), Chinese Academy of Sciences, has proposed a plug-and-play visual tracking framework named PNPTrack. This method effectively leverages spatiotemporal information to improve tracking stability and accuracy without requiring full-parameter retraining and without introducing additional spatiotemporal processing modules. The related work was published in the journal Expert Systems with Applications under the title Adaptive Positive-Negative Spatiotemporal Attention Feature Modulation for Plug-and-Play Visual Tracking. Doctoral student SHI Jian from the SIA is the first author, and Researcher LUO Haibo is the corresponding author.

The research team divided the backbone encoder into shallow and deep parts: the shallow encoder remains frozen during training and does not participate in spatiotemporal interaction, thereby substantially reducing trainable parameters and training resources; the deep encoder integrates spatiotemporal information through positive and negative spatiotemporal features.

The team adopted the following approach to allow positive and negative spatiotemporal features to each play their respective roles. On one hand, positive spatiotemporal features extract target-related information to enhance the target response, helping the model identify targets more accurately in complex scenes. On the other hand, negative spatiotemporal features extract background information to suppress background interference and refine target boundaries. The two types of spatiotemporal features independently interact with search-region features, avoiding mutual suppression. Meanwhile, the team decoupled the “extraction” and “utilization” processes of spatiotemporal features, allowing the two to be executed in parallel within the network, thereby reducing the impact on inference speed.

Rather than simply superimposing spatiotemporal information onto the original matching information, the method assigns separate roles to different information streams. The spatiotemporal features are mainly responsible for adjusting the search-region features through enhancement and suppression—enhancing target-related responses and suppressing background interference—while minimally altering the original template–search feature matching process, thereby reducing the impact of inaccuracies from any single source of information.

Benefiting from this architecture, the spatiotemporal branch can be flexibly toggled on or off at the inference stage. When disabled, the framework reverts to the vanilla tracker, with no need for model re‑loading or parameter adjustment — realizing genuine plug‑and‑play functionality.

The research team tested the method on multiple public visual tracking datasets. The results showed that the method achieved leading performance on mainstream datasets including GOT-10k, LaSOT, and LaSOText. When trained using only the official GOT-10k training set, it achieved an average overlap (AO) of 75.8%. In backbone generality tests, the method delivered consistent performance gains across multiple backbones such as ViT, HiViT, and Fast-iTPN. Experiments also showed that the method requires only a small number of additional projection parameters to improve tracking accuracy while maintaining relatively high inference speed.

This research provides a novel technical approachto balance tracking accuracy and computational efficiency.. By targeted mining and utilization of historical information about both the target and the background, the method can improve tracking performance in complex scenes with low model upgrade cost. It provides valuable references for advancing visual perception technologies applied in intelligent surveillance, mobile robots, autonomous driving and other domains.