Skip to main content

Transform scene metadata with jq

This guide shows how to use the Data Transformation module to transform scene metadata using jq expressions. It includes ready-to-use jq examples for frame, object track, and object snapshot topics, as well as practical tips you can apply when building your own expressions.

Prerequisites​

Overview​

  1. Retrieve available input topics
  2. Add a new transform
  3. Consume the transformed data over MQTT (Optional)
  4. Verify with statistics (Optional)
  5. Remove the transform (Optional)

Extra:

  • Topics specific jq expression examples
  • Practical tips for writing jq expressions

Lets get started!

Step 1: Retrieve available input topics​

Check which scene metadata topics are available on the device.

Remember to replace <servername>, <username> and <password> where applicable.

curl --request GET \
  --anyauth \
  --user "<username>:<password>" \
  --header "Accept: application/json" \
  "http://<servername>/config/rest/data-transformation/v1/availableTopics/topics"
Output
200 OK
Content-Type: application/json

{
    "status": "success",
    "data": [
        {
            "topicName": "com.axis.scene.frame.v1",
            "description": "Scene metadata for a single frame in a video stream, including all object detections present in the frame.",
            "version": "1.0.1",
            "keys": [
                "channel_id"
            ]
        },
        {
            "topicName": "com.axis.scene.object_snapshot.v1",
            "description": "A snapshot of an object, along with information about the source of the image.",
            "version": "1.0.1",
            "keys": [
                "channel_id"
            ]
        },
        {
            "topicName": "com.axis.scene.object_track.v1",
            "description": "A detected and tracked object observed over time.",
            "version": "1.0.1",
            "keys": [
                "channel_id"
            ]
        }
    ]
}
info

Topics can appear at any time, so you can set up transforms for topics that aren't currently available.

Or use the Swagger UI for /data-transformation/v1/availableTopics/topics:

http://<servername>/config/web-ui/swagger-ui/?url=/config/discover/apis/data-transformation/v1/openapi.json

Step 2: Add a new transform​

Create a transform that strips non-human detections from the frame topic. Every frame is forwarded, but the detections array is filtered to contain only Human entries.

Remember to replace <servername>, <username> and <password> where applicable.

curl --request POST \
  --anyauth \
  --user "<username>:<password>" \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \
  "http://<servername>/config/rest/data-transformation/v1/transforms" \
  --data '{
    "data": {
        "inputTopic": "com.axis.scene.frame.v1",
        "jqExpression": ".detections |= (. // [] | map(select(.class.type == \"Human\")))",
        "outputTopic": "com.axis.dt.humans_only"
    }
}'
Output
201 Created
Content-Type: application/json

{
    "status": "success",
    "data": {
        "inputTopic": "com.axis.scene.frame.v1",
        "jqExpression": ".detections |= (. // [] | map(select(.class.type == \"Human\")))",
        "outputTopic": "com.axis.dt.humans_only",
        "outputTopicDescription": "JQ transformation of the topic com.axis.scene.frame.v1 of version 1.0.1",
        "outputTopicVersion": "1.0.1"
    }
}

The transformed data is now published on com.axis.dt.humans_only. The outputTopicVersion is copied from the input topic.

Keep in mind when writing jq expressions
  • The expression must produce a JSON object or null — not a string, number, or other primitive. If it produces null (for example, when select() doesn't match), no message is published.
  • The output must include all key fields from the input topic, such as channel_id for the metadata topics. If a key field is missing, the message fails to publish.

See Output behavior for details.

Or use the Swagger UI for /data-transformation/v1/transforms:

http://<servername>/config/web-ui/swagger-ui/?url=/config/discover/apis/data-transformation/v1/openapi.json

Step 3: Consume the transformed data over MQTT (Optional)​

Set up an MQTT publisher using the Analytics MQTT API. The data source key is the output topic name with a #channelID suffix.

Remember to replace <servername>, <username> and <password> where applicable.

curl --request POST \
  --anyauth \
  --user "<username>:<password>" \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \
  "http://<servername>/config/rest/analytics-mqtt/v1/publishers" \
  --data '{
    "data": {
        "id": "humans_only_publisher",
        "data_source_key": "com.axis.dt.humans_only#1",
        "mqtt_topic": "my_humans_only"
    }
}'
Output
201 Created
Content-Type: application/json

{
    "status": "success",
    "data": {
        "id": "humans_only_publisher",
        "data_source_key": "com.axis.dt.humans_only#1",
        "mqtt_topic": "my_humans_only",
        "qos": 0,
        "retain": false,
        "use_topic_prefix": false
    }
}
info

The #1 suffix refers to the channel ID. On single-channel devices this is always 1. On multidirectional cameras, 2, 3, and 4 might also be available.

Step 4: Verify with statistics (Optional)​

With a consumer active, check the statistics to confirm the transform is working correctly. Let it run under realistic conditions before drawing conclusions.

Remember to replace <servername>, <username> and <password> where applicable.

curl --request GET \
  --anyauth \
  --user "<username>:<password>" \
  --header "Accept: application/json" \
  "http://<servername>/config/rest/data-transformation/v1/transforms/com.axis.dt.humans_only/statistics"
Output
200 OK
Content-Type: application/json

{
    "status": "success",
    "data": {
        "msgInCount": 1200,
        "msgOutCount": 1200,
        "byteInCount": 845000,
        "byteOutCount": 580000,
        "avgMsgProcessingTimeNs": 4000,
        "avgMsgsPerSec": 10.0,
        "msgDroppedCount": 0,
        "jqTransformationErrCount": 0,
        "publishErrCount": 0,
        "lastErrMsg": ""
    }
}
note

If nothing is consuming the output topic — no MQTT publisher configured — all counters will be zero.

A healthy transform has:

  • msgDroppedCount at 0 — no messages are being dropped
  • jqTransformationErrCount at 0 — the expression handles all messages
  • publishErrCount at 0 — all transformed messages are published
note

msgOutCount should equal msgInCount here — every frame is forwarded, just with non-human detections stripped. byteOutCount will be lower than byteInCount because the payloads are smaller.

If msgOutCount > msgInCount, your expression is emitting multiple outputs per message. See Tips for writing jq expressions.

If any error count is increasing, see chapter Statistics in the module reference.

warning

Test under a representative load, not just a quiet scene. See Performance consideration for what can go wrong.

Or use the Swagger UI for /data-transformation/v1/transforms/{outputTopic}/statistics:

http://<servername>/config/web-ui/swagger-ui/?url=/config/discover/apis/data-transformation/v1/openapi.json

Step 5: Remove a transform (Optional)​

If you no longer need the transform, remove it.

Remember to replace <servername>, <username> and <password> where applicable.

curl --request DELETE \
  --anyauth \
  --user "<username>:<password>" \
  --header "Accept: application/json" \
  "http://<servername>/config/rest/data-transformation/v1/transforms/com.axis.dt.humans_only"
Output
200 OK
Content-Type: application/json

{
    "status": "success"
}

Or use the Swagger UI for /data-transformation/v1/transforms/{outputTopic}:

http://<servername>/config/web-ui/swagger-ui/?url=/config/discover/apis/data-transformation/v1/openapi.json

Example jq expressions​

Ready-to-use expressions for each scene metadata topic. Refer to the schema references for all available fields.

Frame (com.axis.scene.frame.v1)​

See the Frame schema for all available fields.

Drop detections, keep track_events and all other fields:

jq expression
del(.detections)

Object Track (com.axis.scene.object_track.v1)​

Object Tracks are emitted once per tracked object when tracking ends. See the Object Track schema for all available fields.

Drop path (trajectory samples) and image (embedded base64 snapshot) — the two largest fields — keep classification and timing summary:

jq expression
del(.path, .image)

Object Snapshot (com.axis.scene.object_snapshot.v1)​

Object Snapshots are emitted when a representative image of a tracked object is captured. See the Object Snapshot schema for all available fields.

Drop data (base64-encoded image bytes — typically over 99% of message size), keep all metadata:

jq expression
del(.data)

Tips for writing jq expressions​

Filtering the detections array​

Scene metadata messages contain arrays like detections and classes. To filter elements within an array, use map() — it produces exactly one output message per input message:

.detections |= (. // [] | map(select(.class.type == "Human")))

Be aware that .detections[] is an iterator — using it directly inside a condition causes the expression to run once per element and emit one output per match. A frame with three Human detections would emit three copies of the frame. msgOutCount > msgInCount in the statistics indicates this phenomenon.

If you instead want to drop the whole message when no element matches, use any():

select(any(.detections[]?; .class.type == "Human"))

Dropping heavy fields​

Object Tracks include path (trajectory samples) and image (base64 snapshot). Object Snapshots include data (base64 image). These fields dominate message size. Use del() to remove them while keeping everything else:

del(.path, .image)

del() and |= preserve all fields you don't touch, including timestamp and any new fields added to the schema in the future. Building a message with {field1, field2, ...} only keeps what you list explicitly.

Preserving key fields​

Operators like del() and |= preserve all fields you don't explicitly remove, including key fields like channel_id. If you construct the output manually with {field1, field2}, remember to include them:

{channel_id, my_field: .some.nested.value}