Fusion tracker

The Fusion Tracker module combines output from several specialized trackers to produce more complete and accurate tracking data. It also supports features such as object snapshots and geographic coordinates.

The data processing pipeline that includes a Fusion Tracker instance is shown below:

A Fusion Tracker instance depends on instances of the following module types:

• Video Object Detection Tracker
• Video Motion Tracker
• Radar Motion Tracker (if Radar Video Fusion Feature is supported)

Because this module builds on data from other modules, the configuration of its input modules also affects its output.

Instances

This module is generally preconfigured with one instance per physical video channel.

Notes:

• On multisensor cameras, stitched panorama views are supported, but not individual sensor views.
• On multidirectional cameras, individual channels are supported, but not grouped views.
• On select fisheye cameras, some dewarped views are supported, see Dewarped views for details.

Function

The Fusion Tracker module tracks objects in the scene and estimates their state. It continually updates each object's estimated state to reflect its current state in the scene.

• Update frequency: approximately 10 Hz. On multidirectional cameras, the frame rate may be lower when multiple instances produce data.
• Latency: approximately 1 s, although it varies.
• Heartbeat: every 2 s. If no objects are detected, the module sends a heartbeat message containing an empty scene.

The module sends a delete track operation when an object's track ends and will no longer be updated. It may re-identify tracked objects classified as humans (ReID), which it indicates with a rename track operation. For more information, see the ReID concept page.

Object state

The estimated state depends on the module's configuration, the sensors it uses, and other factors. The following list summarizes the state estimated for each object.

General:

• Class - Object classification.
  • Human
  • Human Face
  • Vehicle - Car
  • Vehicle - Bus
  • Vehicle - Truck
  • Bike
  • License Plate
• Vehicle Color - Vehicle color
• Human Upper Clothing Color - Human upper clothing color
• Human Lower Clothing Color - Human lower clothing color
• License Plate Text - License plate text (requires license plate text feature)
• Image - The current snapshot image of the object (requires Object Snapshot feature)

2D Image Perspective:

• Bounding Box - Object position as bounding box in the image
• CenterOfGravity/Centroid - Position of the object as a point in the image

3D Device Perspective:

• Spherical Coordinate - Position of the object in spherical coordinates (requires radar-video fusion feature)
• Speed - Speed of the object (requires radar-video fusion feature)
• Direction - Direction of the object (requires radar-video fusion feature)

3D World Perspective:

• Geolocation - Geographic coordinates of the object (requires geographic coordinates feature)

Track operations

• Delete - marks track end
• Rename - renames one track id to another id
• Merge - marks a merger to two ids into one (only on M1135 Mk II, M1137 Mk II and Q3819-PVE)
• Split - marks a split of one id into two (only on M1135 Mk II, M1137 Mk II and Q3819-PVE)

Dewarped views

On fisheye cameras with an ARTPEC-8 or ARTPEC-9 SoC, the Fusion Tracker module runs only on the 360° overview. You can still retrieve data for objects visible in the dewarped views: Panorama, View Area 1-4, Corner Left, and Corner Right. In these views, the objects' bounding box coordinates are transformed to align with the respective video view. Note that, due to an object's orientation in the 360° overview, its bounding box may become larger after transformation to a dewarped view.

Be aware of limitations when consuming data from multiple channels.

Output protocols

This module's output can be retrieved using a variety of methods. Depending on what method is used the data can be sent on different formats, below is a full list of ways to receive the output.

Protocol	Name/Address	Format	Guide
RTSP	Address: rtsp://ip-address/axis-media/media.amp?analytics=polygon Source: AnalyticsSceneDescription	ONVIF tt:Frame	Configure scene metadata over RTSP
MQTT	com.axis.scene.frame.v1	ADF Frame	scene metadata over MQTT
MQTT	com.axis.scene.object_snapshot.v1	ADF Object Snapshot	scene metadata over MQTT
Device Data Hub	com.axis.analytics_scene_description.v0.beta	ADF Beta Frame	ACAP example (consume-scene-metadata)

Configuration

Several configuration options affect this module's output. The following options affect all instances unless otherwise stated.

Some options require a device restart to take effect. Restart the device after changing an option to ensure that the change takes effect.

Not all options are available on all devices.

Description	Method	Default value	Note	Configuration guide
Object Snapshot feature	Rest API, http://<servername>/config/rest/object-snapshot/v1/enabled	OFF	See details below.	Enable and Retrieve Object Snapshots
Geographic coordinates feature	Configure device geolocation and orientation using designated CGIs	OFF	This feature is implicitly enabled when you configure the device's geolocation and orientation. Once enabled, it can only be turned off by resetting the device. Available only on some devices. See Geographic coordinates for details.	Enable Geographic Coordinates
License plate text feature	Install AXIS License Plate Verifier ACAP	OFF	This feature is implicitly enabled by installing the AXIS License Plate Verifier ACAP. Only available on some devices, see more details below.	Install ACAP
Image rotation	Image rotation CGI	Device dependent	Object state in the 2D image perspective is rotated according to the input video's rotation. This option is configured per instance.	Image source rotation

Object Snapshot feature

To find supported devices, visit the AXIS Scene Metadata product page and filter by the "Object Snapshot" functionality.

The Object Snapshot feature captures images of objects in a scene. When enabled, it adds a base64-encoded, cropped image to classified and unclassified objects. The current implementation prioritizes bounding box size over the lifetime of a track. The module may send more than one snapshot for an object if it finds a better image.

The following diagram shows how this feature connects to the Fusion Tracker module:

To enable this feature, see the guide.

Compression

This feature uses resolution downscaling and JPEG quality compression to fit the snapshot generation requirements. The JPEG quality does not go below 50% to ensure a useful snapshot.

Snapshot generation requirements

All the requirements are checked after the image compression step

• Size less than 64 kB
• Pixel count
  • Less than 500k
  • More than 800

Snapshot limitations

• Tracks consisting of less than four frames are unlikely to receive a snapshot

Radar video fusion feature

info

This feature is only available radar video fusion cameras such as Q1686-DLE and Q1656-DLE.

The radar video fusion feature enables the Fusion Tracker module to also fuse tracking data generated from radar scans.

This feature will add additional information to the object state for objects detected by the radar. The additional info includes

• Spherical Coordinate
• Speed
• Direction

As the radar sensor is designed to detect objects far away, using this feature will also improve detection range.

This feature can also be used in combination with the Geographic Coordinates this feature to receive geolocation data.

License plate verifier integration feature

info

This feature is only available on Q1686-DLE.

The AXIS License Plate Verifier ACAP enables capturing and recognizing of license plate text. By integrating this data into the Fusion Tracker module through this feature, the same information can now be provided for the Fusion Tracker module output.

The ALPV ACAP acap can be visualized as a processing module together with the Fusion Tracker module as such,

Geographic coordinates

info

This feature is only available radar video fusion cameras such as Q1686-DLE and Q1656-DLE.

The geographic coordinates feature adds coordinates in decimal degrees to the object state. To calculate accurate coordinates, configure the device's location and orientation precisely. The bottom center of each object's bounding box is used to calculate its coordinates. If a person's feet aren't visible, the coordinates may be inaccurate.

This feature uses data from the Radar Motion Tracker. Only objects detected by the radar in a given frame receive geolocation data for that frame.

Read the how-to on how to enable this feature.

Limitations

• Mirroring is not supported.
  • If the object tracking should be visualized on a mirrored image, the user must apply mirroring.
• Objects must be moving to be considered by this tracker.
• This tracker only supports video stream default aspect ratios.

Consuming data from multiple channels

On devices with multiple video sensors, or fisheye cameras that support dewarped views, consuming scene metadata from three or more channels simultaneously may result in dropped messages. To reduce this risk, consume data from no more than two channels at a time.