Scene Change Detection with Savant: Catching Camera Tampering with ReID Embeddings

· Savant Team
Scene Change Detection with Savant: Catching Camera Tampering with ReID Embeddings

Most video analytics pipelines assume the camera is pointing where it was yesterday. That assumption fails quietly and often: a camera gets bumped, a PTZ head drifts, someone tapes over the lens, a truck parks in front of the view, or a maintenance crew repositions the mount and never tells anyone. When the scene changes out from under a pipeline, every downstream model keeps running — on the wrong picture. Detections still fire, dashboards still fill, and nothing signals that the footage stopped being meaningful.

We have added a new sample to Savant that watches for exactly this: samples/scene_change. It flags when a monitored region of interest has changed significantly compared to a few seconds earlier — camera tampering, repositioning, occlusion, or any large shift in what the camera sees — and it does so cheaply, with a single embedding per frame rather than a full detection stack.

The idea

The naive way to detect a scene change is to diff pixels. That breaks immediately in the real world: lighting flickers, auto-exposure hunts, sensor noise shimmers, a cloud passes, and the pixel delta spikes even though nothing about the scene actually changed. Pixel differencing confuses the image changing with the scene changing, and those are not the same thing.

The sample takes a more robust route. It computes a ReID (re-identification) embedding vector for a region of interest on each frame and compares it against the embedding of the same region N seconds ago. Embeddings are trained to be invariant to exactly the nuisances that wreck pixel diffing — small lighting and exposure changes barely move the vector, while a genuinely different scene moves it a lot. The comparison is a cosine distance between L2-normalized vectors:

distance = 1 - dot(a, b)

When that distance rises above a configurable threshold, the region is flagged as changed. Because the reference is a frame from a few seconds back rather than a fixed keyframe, the detector tracks the scene as it legitimately evolves and only fires on abrupt, embedding-level shifts.

How the pipeline is built

The whole thing is three pipeline elements on top of Savant’s standard decode/encode, and it reads top to bottom:

  1. RoiInjector (a pyfunc) attaches the region of interest to each frame as an object, so the inference model knows which part of the picture to embed.
  2. scene_reid (an nvinfer element) runs a ReID ONNX model over that ROI and emits an embedding. The model takes a 3×224×224 crop and produces a vector attribute named reid, converted to a plain vector via Savant’s TensorToVectorConverter. TensorRT compiles the ONNX engine automatically on first launch.
  3. SceneChangeDetector (a pyfunc) keeps a short per-source history of (timestamp, embedding) pairs in a deque, finds the closest reference frame at roughly now − delay, computes the cosine distance, and writes the result back onto the frame.

A final overlay draws the outcome: the ROI rectangle plus a status label — green “stable”, red “scene changed”, or orange when there is not yet enough history to judge — with the live distance value printed alongside so you can watch it move.

The two knobs that matter are both small and both configurable:

  • Reference delay — how far back to compare (default 5.0 seconds). Longer delays tolerate slower legitimate change; shorter delays react faster.
  • Distance threshold — how much embedding movement counts as a change (the demo ships with 0.1). Lower is more sensitive; higher suppresses minor drift.

Dynamic regions of interest over Etcd

You rarely want to watch the whole frame — you want to watch the doorway, the conveyor, the gate, the shelf. The sample lets you set the ROI per source at runtime through Etcd, with no pipeline restart. The RoiInjector reads the key roi/{source_id} (cached with a short TTL so updates take effect within about a second) and falls back to a default region when the key is empty. ROIs are validated against the frame bounds and a minimum size before they are applied.

A helper script wraps the Etcd write:

# Watch a specific region on the "video" source: left,top,right,bottom
./samples/scene_change/set-roi.sh video "540,100,1000,400"

# Run with no coordinates to restore the default ROI
./samples/scene_change/set-roi.sh video

This is the same dynamic-reconfiguration pattern Savant uses elsewhere — configuration lives in Etcd, the pipeline watches it, and operators retune a live deployment without touching the process.

Running the demo

Clone the repository and pull the Git LFS assets first, then bring up the bundle for your hardware:

# x86 + discrete NVIDIA GPU
docker compose -f samples/scene_change/docker-compose.x86.yml up

# NVIDIA Jetson (L4T)
docker compose -f samples/scene_change/docker-compose.l4t.yml up

The first launch compiles the TensorRT engine, so give it a moment. If you would rather build the engine ahead of time:

./scripts/run_module.py --build-engines samples/scene_change/module.yml

Once it is running, watch the annotated stream over RTSP at rtsp://127.0.0.1:554/stream/video or LL-HLS at http://127.0.0.1:888/stream/video/. Move the source scene, block the ROI, or swap the input, and you will see the label flip from stable to changed as the distance crosses the threshold.

Why this is worth having

The cost profile is the point. Scene change detection here is one embedding per frame over a single ROI — no object detector, no tracker, no per-object inference. That makes it cheap enough to run alongside a real analytics pipeline as a health check, or on its own at the edge as a tamper and integrity monitor. It is embedding-based, so it shrugs off the lighting and exposure noise that defeats pixel diffing. And it is event-driven: instead of a human noticing three days later that a camera has been staring at a wall, the pipeline emits a signal the moment the view stops matching what it saw a few seconds ago.

Typical uses: camera tamper and integrity monitoring (lens covered, camera turned or unplugged), PTZ drift detection, occlusion alerts on a critical region, and input-validity gating — pausing or flagging downstream analytics when the scene is no longer the one the pipeline was configured for.

What is Savant

Savant is an open-source, high-performance framework for building real-time streaming AI applications on the NVIDIA stack. Built on top of DeepStream and TensorRT, it gives you a high-level, Python-first way to assemble fault-tolerant inference pipelines that run efficiently on both data-center GPUs and Jetson edge devices — while keeping the heavy lifting in optimized native code.

Source code

The demo lives in the Savant repository: https://github.com/insight-platform/Savant/tree/develop/samples/scene_change

You will find many more samples covering different models and pipeline patterns here: https://github.com/insight-platform/Savant/tree/develop/samples

Consider visiting our GitHub, and don’t forget to subscribe to our X for updates. We also have a Discord where we help users get started.