Object tracking in motion imagery

Available with Image Analyst license.

One capability of motion imagery is tracking specific objects in video data while it plays. These objects can be stationary or moving, may become obscured and re-emerge, or change shape (such as a person entering a vehicle). The object tracking capability in full-motion video (FMV) provides automated and computer-assisted tools to address a variety of situations when tracking objects in video imagery. It relies on deep learning and computer vision technology to assist in object tracking, feature extraction, and matching. You can build a deep learning object tracking model, and use the suite of tools to select, and track an object of interest. The centroids corresponding to the object's identification rectangles can be digitized and saved as a point class in the project's geodatabase. The saved points can then be displayed as the archived video plays.

Requirements

Object tracking capability in FMV is available in ArcGIS Pro with the ArcGIS Image Analyst extension.

Note:

Ensure that your video card drivers are current.

Deep learning model

Tracking objects in a video requires a trained deep learning model. The effectiveness of tracking depends on the quality of the deep learning training sample data and how closely your object of interest is associated with the training data. For example, to track a truck moving along a highway, you must have labelled training samples of trucks from many different angles. The source of the training samples (annotated images) must be motion imagery. The labelled training samples are used to train the deep learning model to track objects, for instance trucks in this case. The model will have limited ability to track objects with different appearance, such as cars, but may have success in tracking larger recreational vehicles or buses.

You must install deep learning framework packages to perform deep learning workflows in ArcGIS Pro. Use a variety of tools to prepare video and still imagery training data, label objects, create deep learning models, inference, and review results. For information about how to install these packages, see Install deep learning frameworks for ArcGIS.

Deep learning is computationally intensive, and a powerful GPU is recommended with CUDA Compute Capability support, version 6.0 or later.

For details about deep learning and deep learning workflows, see Introduction to deep learning and Deep learning in ArcGIS Pro. For more information about the suite of deep learning tools in ArcGIS, see An overview of the Deep Learning toolset.

Tracking tab

The Tracking tab is enabled on the Standalone Video contextual tab set when you select a video in the Contents pane.

Note:

The Tracking tab is available once the deep learning packages are installed and enabled in ArcGIS Pro.

The tools in the Object Tracking group allow you to identify and manage object tracking in your video data:

• Tracking Pane—Open the Object Tracking for Video pane, where you can specify the object tracking deep learning model and configure parameter settings.
• Enable—Activate the interactive object tracking tools.
• Add Object—Add an object to perform object tracking by interactively drawing a rectangle around the object.
• Reposition Object—Click an existing object's tracking rectangle to select it, and redraw the rectangle around the object's updated position.
• Remove Object—Click an existing object's tracking rectangle to remove it from being actively tracked.
• Object(s) to Feature—Save the centroids of the object detection rectangles as a new feature class.

Tracking Pane

Click the Tracking Pane button to open the Object Tracking for Video pane. The pane contains the Settings and Tracked Objects tabs.

Settings

On the Settings tab, you can choose the deep learning model and set parameters for tracking objects.

Click the Add object tracking model using local path or URL button to open the Add Deep Learning Model From Path dialog box. Specify the path to your deep learning model package file (.dlpk) by providing a URL or by browsing to the file in a local directory. You can assign an alias for the model's name in the Model text box. Click Add to load the model and close the dialog box. The model name appears in, and is selected from, the Model drop-down list.

The Settings tab contains two options that help control the object tracking: Detect Track Failure and Recover Track.

• Detect Track Failure—Specifies whether the object is successfully tracked based on changes in appearance. The default is checked.
  • Interval (frames)—The interval, in number of frames, at which the application checks for object appearance changes. The default is 5 frames.
  • Minimum Object Size (pixels)—Object tracking will cease if an object is smaller than the specified size. The default is 10 pixels.
• Recover Track—Specifies whether an attempt is made to find an object after the track has been lost. The default is checked.
  • Confidence Threshold (0-1)—The minimum ratio between matched source image features and searched object features for successful recovery. The confidence threshold is a number between 0 and 1. The default value is 0.1.
  • Minimum Overlap Threshold (0-1)—The minimum overlap ratio between the detected object and searched object for successful recovery. The threshold is a number between 0 and 1. The default value is 0.1.

Tracked Objects

The Tracked Objects tab is where you can view and manage tracked objects. The ID, Source, and Status values of every tracked object are listed in the table.

• ID—The unique identifier for each tracked object.
• Source—The source video file in which the object is identified.
• Status—The status of each tracked object: actively tracked, lost, or in a search.

The Tracked Objects tab also contains the following five action buttons for managing tracked objects:

• Add—Add one or more objects to track by drawing a rectangle around each object of interest.
• Reposition—Click an existing object's tracking rectangle to select it, and reposition the rectangle around the object's updated position.
• Remove—Click an existing object's tracking rectangle to remove it from being actively tracked.
• Reassociate—Reassociate the listed object with the Lost status by redrawing the object’s updated position.
• Delete—Delete the selected object from the tracked object list.

Object tracking workflow

A summary of the object tracking workflow is as follows:

• Load a deep learning model and set the tracking parameters on the Settings tab in the Object Tracking for Video pane.
• Click Enable to activate the object tracking tools.
• Click Add Object to draw a rectangle around the object you want to track in the video player. The object will be tracked in every video frame.
• On the Tracked Objects tab, view the status of the tracked objects.
• If the object becomes obscured and tracking is lost, click Reassociate, and redraw a rectangle around the object's updated position to re-engage tracking.
• If a tracking rectangle becomes misaligned with an object being tracked, click Reposition, select its tracking rectangle, and reposition it around the object.
• Optionally, click Add to add an object to be tracked.
• Optionally, click Remove to remove an object from active tracking.
• On the Tracking tab, in the Object Tracking group, click Object(s) to Feature and specify the output location and prefix name to use to store the object centroids to a feature class. Optionally, specify whether the feature class will be added to the map, as well as the frequency interval in seconds by which the centroids will be saved.