Data transformation

The Data Transformation module takes a scene metadata topic as input, transforms each message using a jq expression compatible with jq 1.8.1, and publishes the result as a new topic. The new topic is available both for ACAPs running on the device and externally via MQTT, just like any other scene metadata topic.

This enables extracting, filtering, or reshaping data on the device before it's consumed, without changing the original topic.

Instances

Unlike other modules, Data Transformation instances are not preconfigured. Create, update, and remove transform instances through the Data Transformation configuration API. Each transform instance maps an input topic to an output topic using a jq expression. Every output topic is unique and prefixed with com.axis.dt..

Update an existing transform by changing its inputTopic, jqExpression, outputTopicDescription, or outputTopicVersion. All statistics reset when a transform is modified in any way.

Function

Common use cases include:

• Filtering — remove data or events that don't meet a specific criteria
• Reducing bandwidth — extract only the fields you need
• Reformatting data — convert timestamps, rename fields, or restructure the JSON to match the format expected by an external consumer
• Precomputing derived data — add computed fields to reduce processing requirements for downstream consumers

To get started, see the Transform scene metadata with jq guide.

Output behavior

The jq expression must produce a JSON object or null. Primitive values such as strings, numbers, or booleans are not valid output. If the expression produces null no message is published.

The output must include all key fields defined by the input topic. For scene metadata topics, this is channel_id. If a key field is missing, the message fails to publish — see publishErrCount below.

Statistics

Each transform exposes runtime statistics through the configuration API. Statistics reset when the transform is modified or the device restarts.

Field	Description
msgInCount	Total number of input messages received
msgOutCount	Total number of output messages successfully published
byteInCount	Total bytes received
byteOutCount	Total bytes sent after jq processing
avgMsgProcessingTimeNs	Average processing time per message in nanoseconds, over the last 30 messages
avgMsgsPerSec	Average input message rate in messages per second, over the last 30 messages
msgDroppedCount	Number of messages dropped before jq processing, often due to congestion
jqTransformationErrCount	Number of messages where the jq expression failed to produce a result
publishErrCount	Number of messages that were transformed but failed to publish, for example due to a missing key field
lastErrMsg	The most recent error message, including a UTC timestamp and error type

Performance considerations

The device runs jq expressions on every incoming message in real time, so a poorly written expression can cause messages to be dropped or delayed. Transforms share CPU and memory with the rest of the device.

Always test expressions under realistic conditions — run the transform while the device is under a representative load (many objects in the scene) and monitor the statistics to catch problems early.

Possible pitfalls

• Slow jq expressions — if an expression is too complex to process each message before the next one arrives, the input queue fills up and messages start being dropped. Monitor avgMsgProcessingTimeNs and msgDroppedCount to catch this early
• Message volume multiplication — if an expression produces many output messages per input (for example, using .[] as a top-level iterator instead of map()), the output rate multiplies and can overwhelm downstream consumers
• Transform chains — since transforms can use other transform output topics as input, it's possible to create a chain of transforms. This increases the resources needed per round trip more than combining the expressions into one
• Unbounded recursion in jq — some jq constructs can trigger deep or infinite recursion, consuming large amounts of CPU/memory per message

Interpreting statistics

• msgDroppedCount increasing — the jq expression or data serialization/deserialization is too slow to keep up with the input rate. Try to simplify the expression to generate less output or make the expression less calculation heavy
• jqTransformationErrCount increasing — the expression doesn't handle all input messages correctly. Check lastErrMsg for details
• publishErrCount increasing — transformed messages can't be published, often because a required key field is missing from the output
• msgOutCount lower than msgInCount — expected if the expression uses select() or other methods to filter messages, but unexpected otherwise. This means that the message was processed but the jq expression produced no output
• avgMsgProcessingTimeNs compared to avgMsgsPerSec — if the average processing time per message is close to or exceeds the average time between messages (the inverse of the message rate), the transform is at risk of falling behind and dropping messages

Output protocols

Transformed topics can be retrieved using the same protocols as other scene metadata topics:

Protocol	Format	Description
MQTT	JSON	Subscribe to the output topic over MQTT

Limitations

• Only scene metadata topics (com.axis.scene.*) and other Data Transformation output topics (com.axis.dt.*) are supported as input
• Output topic names must be prefixed with com.axis.dt.
• A maximum of 8 transforms can be configured at the same time
• jq expressions have a maximum length of 20480 characters