Skip to main content

AI-enriched object insights

AXIS Scene Metadata provides representative snapshots of tracked objects. Send them to an image-capable AI model to identify details beyond standard classifications, such as a vehicle's model, body style, and visible text, or a person's clothing, accessories, and carried items.

Prompt sent to AI
Analyze the person in the image. Output hair type, accessories, clothing, and estimated age in JSON format.
AI analysis result
{
    "gender": "Female",
    "hair_type": "Short, light brown/blonde",
    "accessories": ["Eyeglasses", "Shoulder bag (black)"],
    "clothing": ["White short-sleeved t-shirt", "Light blue denim jeans", "White sneakers"],
    "estimated_age": "20-30"
}
View original scene metadata
{
    "channel_id": 1,
    "id": "6a385c75-2e36-47e7-8758-7b0d0c7ae503",
    "start_time": "2026-06-23T14:34:48.565889Z",
    "end_time": "2026-06-23T14:34:56.665929Z",
    "duration": 8.1,
    "classes": [
        {
            "type": "Human",
            "score": 0.8823,
            "carries_bag": true,
            "upper_clothing_colors": [
                { "name": "white", "score": 0.613 },
                { "name": "gray", "score": 0.2945 }
            ],
            "lower_clothing_colors": [
                { "name": "blue", "score": 0.6605 },
                { "name": "gray", "score": 0.2985 }
            ]
        }
    ],
    "image": {
        "id": "ed428166-7bc7-4b19-b6c3-1c3b5166a15b",
        "timestamp": "2026-06-23T14:34:49.765895Z",
        "crop_box": { "top": 0.1433, "right": 0.2654, "bottom": 0.4575, "left": 0.1568 },
        "data": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBD…"
    },
    "path": [
        {
            "timestamp": "2026-06-23T14:34:48.565889Z",
            "bounding_box": { "top": 0.2662, "right": 0.0753, "bottom": 0.5256, "left": 0.0092 }
        }
    ]
}
note

AI-generated details depend on the image, prompt, and model. Validate important results before using them for decisions, alerts, or automated actions.

How it works​

When you enable object snapshots, AXIS Scene Metadata captures representative images of tracked objects. Your application can send these images to an image-capable AI model to extract more details. Because scene metadata already identifies the broad object class, your application can use that class to select a focused prompt for the model.

You can receive the images from two data sources:

Data sourceSnapshot output
Object TrackContains the latest representative snapshot available when the object track ends.
Object SnapshotSends each generated snapshot update during tracking, which can mean more MQTT messages and AI requests.

This example uses Object Track. When tracking ends, the application receives the consolidated object information and latest representative image. It then selects a prompt from the most likely object class, sends the image and prompt to the AI model, and prints the returned details as JSON.

The full flow is:

  1. Enable the Object Snapshot feature on the device.
  2. Configure an Analytics MQTT publisher for the Object Track data source.
  3. Read the most likely object class and representative snapshot from each payload.
  4. Choose a prompt based on the object class.
  5. Send the image and prompt to an image-capable AI model.

Try it yourself​

Prerequisites​

Update the MQTT settings and implement analyze_image() for your chosen model provider. Then run the script and move an object through the camera view. The Object Track message is emitted when tracking ends.

View full script
analyze_object_snapshots.py
import json

import paho.mqtt.client as mqtt

MQTT_BROKER_HOST = "<MQTT broker IP>"
MQTT_BROKER_PORT = 1883
MQTT_TOPIC = "<Analytics MQTT publisher topic>"

DEFAULT_PROMPT = "Describe the object in the image. Return the result as JSON."

CLASS_PROMPTS = {
    "car": "Analyze the car in the image. Output its model, type, color, and estimated year in JSON format.",
    "human": "Analyze the person in the image. Output hair type, accessories, clothing, and estimated age in JSON format.",
    "bike": "Analyze the bike in the image. Output type, color, and accessories in JSON format.",
}


def analyze_image(base64_image: str, prompt: str) -> str:
    """Send a base64-encoded image to an image-capable AI model."""
    raise NotImplementedError("Add your model call here")


def get_most_likely_object_type(object_track: dict) -> str | None:
    classes = object_track.get("classes", [])
    if not classes:
        return None

    return classes[0].get("type", "").lower()


def analyze_object_track(_client, _userdata, message):
    object_track = json.loads(message.payload)

    object_snapshot = object_track.get("image")
    if not object_snapshot:
        return

    object_type = get_most_likely_object_type(object_track)
    prompt = CLASS_PROMPTS.get(object_type, DEFAULT_PROMPT)
    analysis = json.loads(analyze_image(object_snapshot["data"], prompt))

    result = {
        "object_track_id": object_track["id"],
        "object_snapshot_id": object_snapshot["id"],
        "analysis": analysis,
    }
    print(json.dumps(result))


client = mqtt.Client()
client.on_message = analyze_object_track
client.connect(MQTT_BROKER_HOST, MQTT_BROKER_PORT)
client.subscribe(MQTT_TOPIC)
client.loop_forever()

Understand the code​

1. Connect to the consolidated object-track stream​

Configure an Analytics MQTT publisher with com.axis.scene.object_track.v1#1 as its data_source_key. The #1 suffix selects channel 1; use the channel that matches your device configuration. The publisher's mqtt_topic is the broker topic that the script subscribes to through MQTT_TOPIC.

import paho.mqtt.client as mqtt

client = mqtt.Client()
client.on_message = analyze_object_track
client.connect(MQTT_BROKER_HOST, MQTT_BROKER_PORT)
client.subscribe(MQTT_TOPIC)
client.loop_forever()

For publisher setup and connection details, see Configure scene metadata over MQTT.

2. Read the snapshot and object class​

Each Object Track payload represents one completed track. Its image.data field contains the latest representative snapshot, and the first item in classes is the most likely object classification. A track might not contain image, for example, if it was too short to produce a snapshot.

def get_most_likely_object_type(object_track: dict) -> str | None:
    classes = object_track.get("classes", [])
    if not classes:
        return None

    return classes[0].get("type", "").lower()


def analyze_object_track(_client, _userdata, message):
    object_track = json.loads(message.payload)

    object_snapshot = object_track.get("image")
    if not object_snapshot:
        return

    object_type = get_most_likely_object_type(object_track)

3. Select a focused prompt​

Scene metadata already identifies the broad object class. Use it to ask the model a focused question instead of requesting a generic image description. A default prompt handles classes that aren't in the mapping.

prompt = CLASS_PROMPTS.get(object_type, DEFAULT_PROMPT)

Example prompt mapping:

CLASS_PROMPTS = {
    "car": "Analyze the car in the image. Output its model, type, color, and estimated year in JSON format.",
    "human": "Analyze the person in the image. Output hair type, accessories, clothing, and estimated age in JSON format.",
    "bike": "Analyze the bike in the image. Output type, color, and accessories in JSON format.",
}

4. Call an AI model​

The example keeps the provider-specific code behind one function. It expects the model to return a JSON object encoded as text:

def analyze_image(base64_image: str, prompt: str) -> str:
    """Send a base64-encoded image to an image-capable AI model."""
    raise NotImplementedError("Add your model call here")

Implement this function with a local model or a hosted service. It must accept a base64-encoded image and a prompt, then return a JSON object as text. Provider SDKs and request formats change independently of AXIS Scene Metadata, so this page leaves that boundary explicit rather than prescribing one provider.

Keep credentials outside your source code. When the provider supports structured output, use a JSON schema and validate the response before storing or acting on it.