Small Objects in the Wild, Part 4: Tracking What You Can Barely See
Part four of five. Part 1 covers the geometry. Part 2 covers stock defaults. Part 3 covers detection. Part 5 covers blur, sensors and deployment.
The published baseline for the MVA 2025 small-flying-object challenge paired YOLOX with an unmodified OC-SORT and scored 9.90 SO-HOTA. The winning entry used the same tracker family with three geometric changes and a tiled detector, and scored 50.59. A 5.1× improvement, and the tracker architecture never changed.
That result is the shape of this whole subject. Association research has produced twenty years of increasingly sophisticated methods, and on six-pixel targets the winning configuration is still a 2016 Kalman filter with the IoU replaced by something that works.
Five Paradigms, and What Each Costs You
| Paradigm | Mechanism | Latency profile | On tiny objects |
|---|---|---|---|
| Tracking-by-detection (SDE) | Detector → optional ReID → motion predict → bipartite assignment | Detector dominates (10–100 ms); association 0.1–3 ms | Best. Tile the detector independently, swap it freely, tune association without retraining |
| Joint detection and embedding (JDE) | One backbone emits boxes and ReID vectors (JDE, FairMOT, CSTrack) | Single forward pass, no separate ReID model | Fails. The ReID head samples features at the object centre on a stride-4/8 map, so a 6 px bird yields about one pixel of embedding support |
| Tracking-by-attention / end-to-end | Track queries persist across frames (MOTR family, MeMOTR, MOTIP, LA-MOTR) | Slowest class, roughly 5–20 FPS | Poor. Needs high-resolution feature maps that scale quadratically, has no tiling story, and has zero small-object benchmark wins to date |
| Tracking-by-regression | Detector head regresses previous boxes forward (Tracktor, CenterTrack) | Cheap, no separate association | Dead for birds and balls moving 20+ px per frame — the small-displacement assumption is violated by an order of magnitude |
| Offline graph optimisation | Min-cost flow, multicut, hierarchical GNN (SUSHI) | Batch only, seconds to minutes per clip | Best identity recovery through long gaps. Unusable online |
| Track-before-detect | Accumulate energy along candidate trajectories before thresholding | Expensive motion-hypothesis search | The only principled route below about 3 px for SNR-limited targets |
The evidence for the JDE row is direct rather than theoretical: FairMOT on VisDrone-MOT reaches MOTA 34.3 and IDF1 46.1 at 17.2 FPS, below plain ByteTrack on the same benchmark. One-shot architectures underperform tracking-by-detection on aerial video for a stride reason, not an engineering-quality reason.
The end-to-end row deserves its own measurement, because the family is genuinely strong elsewhere. It owns association quality on appearance-degenerate but well-resolved targets — DanceTrack and SportsMOT, with AssA in the 59–63 range against 38–48 for the SORT family. On MMOT, a drone multispectral benchmark, MOTRv2 scored 49.2 HOTA and MeMOTR 42.3 while plain BoT-SORT scored 53.6.
Why the Kalman Assumptions Break
SORT’s state is seven-dimensional: centre u, v, scale s, aspect r, and velocities for the first three, under constant velocity and constant aspect ratio. On pedestrians both assumptions hold. Constant velocity plus constant aspect is also the most-violated assumption in this domain, because birds, shuttlecocks and agile drones accelerate hard and change apparent aspect every frame. And the IoU cost matrix is degenerate below about 20 px.
Each fix removes one specific failure:
EMA velocity direction (α = 0.8) instead of raw Kalman velocity. The Kalman velocity estimate on a tiny box is dominated by localisation noise, so smoothing the direction is more robust than filtering the position.
Observation-centric re-update. During occlusion the filter’s variance grows and its velocity estimate drifts; when the object reappears the filter snaps and the velocity is garbage. OC-SORT re-runs the filter over a virtual trajectory between the last and current observations, and uses observation direction rather than filter velocity for the motion cue.
Buffered or expanded IoU (C-BIoU, Deep-EIoU) enlarges boxes before computing overlap, which makes the metric non-degenerate again.
Constant-acceleration local models. WASB predicts p̂ₜ₊₁ = pₜ + vₜ + aₜ/2 from finite differences and drops the Kalman filter entirely, beating TrackNetV2 on all five sports it covers.
NSA-Kalman scales measurement noise by detection confidence.
The SMOT4SB ablation attributes +3.716, +3.471 and +3.818 SO-HOTA to three of these in sequence — measured, additive and free. One caution on expanded IoU: it raises false associations in crowded scenes, so it is trading precision for the ability to associate at all. That is right when targets are sparse and wrong when they are dense. Tune the expansion factor to your target density, not your target size. Two birds 15 px apart with 2× expansion will swap identities.
Worth noting what got displaced and why. JPDA, MHT, particle filters, min-cost flow and graph multicut all handle association ambiguity more principledly than a greedy per-frame assignment. They were displaced not because they were wrong but because a cheap two-stage cascade captures most of the benefit at a thousandth of the cost. Particle filters remain genuinely relevant for tiny erratic targets, because they represent multi-modal posteriors, which is the correct model when a 6-pixel blob could be either of two things. They become impractical above roughly 50 simultaneous tracks.
The SORT Family: The Part You Will Actually Ship
All MOT17 and MOT20 figures below are test-set with private (YOLOX-X) detections unless stated. Two warnings before the table.
Cells marked R are tuned third-party reimplementations rather than the figure the original paper published, and the difference is large: SORT’s official DanceTrack HOTA is 47.9, the tuned reimplementation 54.3. (That 47.9 itself comes from OC-SORT’s comparison table, since the DanceTrack paper has no SORT row at all.) They are included because for several older trackers they are the only DanceTrack numbers measured under a uniform detector, but they are not interchangeable with paper values.
And read the FPS column with suspicion. ByteTrack’s 29.6 includes the detector on a V100, while OC-SORT’s ~700 and UCMCTrack’s 1000+ are association-only. A 30× discrepancy hides in that column.
| Tracker | Key idea | MOT17 HOTA | MOT20 HOTA | DanceTrack HOTA | FPS (hardware, detector included?) | Licence |
|---|---|---|---|---|---|---|
| SORT (2016) | Kalman + Hungarian on IoU | 60.4 R | — | 47.9 · 54.3 R | ~260 Hz assoc-only | GPL-3.0 |
| FairMOT (2021) | Anchor-free JDE, balanced heads | — | — | 39.7 | 17.2 (VisDrone, incl.) | MIT |
| ByteTrack (2022) | Associate low-score boxes in a second stage | 63.1 | 61.3 | 47.7 | 29.6 MOT17, V100, incl. | MIT |
| BoT-SORT (2022) | GMC + better KF state + IoU⊕ReID fusion | 64.6 | 62.6 | 57.8 R | 4.5, RTX 3060 + i9 | MIT |
| BoT-SORT-ReID | + FastReID | 65.0 | 63.3 | — | — | MIT |
| OC-SORT (2023) | Observation-centric re-update, momentum | 63.2 | 62.4 | 55.1 | ~28 RTX 2080 Ti incl.; ~700 assoc-only | MIT |
| StrongSORT / ++ | BoT-ReID + ECC + NSA-Kalman + AFLink + GSI | 64.4 | 62.6 | — | 7.5 on a V100; 0.96 on a Jetson TX2 | GPL-3.0 |
| Deep OC-SORT (2023) | Adaptive feature weighting, dynamic appearance | 64.9 | 63.9 | 61.3 | not reported | MIT |
| Hybrid-SORT (2024) | Weak cues: confidence, height, velocity direction | 63.6 | 62.5 | 62.2 (65.7 w/ ReID) | 27.8 V100 incl.; 15.5 with ReID | see repo |
| C-BIoU (2023) | Buffered IoU cascade, no Kalman filter | 64.1 | — | 60.6 | not reported | Apache-2.0 reimpl. |
| SparseTrack (2023) | Pseudo-depth cascade | 65.1 | 63.4 | — | — | MIT |
| BoostTrack | Boost detection confidence from tracklet evidence | — | — | — | 65.5 without appearance / 15.4 with, RTX 3090 | MIT |
| BoostTrack++ | Soft-BIoU + Mahalanobis + shape | 66.6 | 66.4 | — | not reported | MIT |
| UCMCTrack (2024) | Mahalanobis distance on the ground plane | 64.3 · 65.8 ++ | 62.8 · 63.4 | — | >1000 on one CPU, given detections | MIT |
| DiffMOT (2024) | Diffusion motion predictor replacing the KF | 64.5 | 61.7 | 62.3 | 22.7–30.3, RTX 3090, incl. | unstated |
| Deep-EIoU (2024) | Iterative scale-up expansion IoU, no Kalman | — | — | SportsMOT 77.2 | 14.6, RTX 4080, all-inclusive | unstated |
Four of these matter disproportionately for small objects.
ByteTrack is the correct default, for a reason that generalises. Low-confidence detections are not noise — they are mostly occluded or small true objects. Tiny objects live permanently in the 0.1–0.4 confidence band, which is exactly the band a single-threshold tracker discards. ByteTrack keeps them and associates them in a second pass against tracks that failed the first pass, and that second stage is IoU-only at threshold 0.5 with no score fusion, by explicit design. It is MIT, trivially simple, and deployable as-is at a detector-inclusive 29.6 FPS on a V100. Two limits: it is still IoU-gated, so it inherits the degeneracy below 20 px, and it has no camera-motion compensation, which is disqualifying for aerial video. Density is also a latency variable — MOT20 drops it to 17.5 FPS on the same GPU (paper Table 5).
OC-SORT is the base every small-object challenge winner starts from. Association alone runs at roughly 700 FPS on an i9. Every top-five entry in the MVA 2025 challenge that used a SORT-family tracker started here.
The appearance-heavy branch is where the cost lives, and the numbers are worth internalising before you enable ReID. BoostTrack goes from 65.5 to 15.4 FPS on identical hardware when embeddings are switched on — a 4.3× tax. StrongSORT with full ReID runs at 7.5 FPS on a V100 by its own paper, and at 0.96 FPS on a Jetson TX2 in the study that measures it on embedded hardware. That 7.8× spread is itself the finding: the appearance tax scales with how little compute you have. Selective extraction recovers most of it on the TX2 — at a 0.5 filter setting, 3.21 FPS (+234%) at roughly HOTA parity. And on tiny objects you are paying that tax for an embedding computed from about one pixel of support, which is the worst trade in this whole series. StrongSORT is also GPL-3.0.
UCMCTrack is conceptually the cleanest answer to camera motion, and its limitation is exactly the case this series cares about. Image-plane motion of a target on a moving platform is dominated by camera motion, so project detections onto the ground plane and the target’s motion becomes nearly linear again — a simple Mahalanobis distance suffices and per-frame camera-motion estimation becomes unnecessary. It reaches MOT17 HOTA 65.8 with association at over 1000 FPS on a single CPU core. But it requires a ground plane and a homography, which exist for surveillance, sports and nadir aerial, and do not exist for targets in the air against sky. That is precisely the counter-UAS and bird case.
For offline work, SUSHI is the right tool and it composes with an online tracker: MOT17 HOTA 66.5 / IDF1 83.1, MOT20 64.3 / 79.8. Identity is a global property of a trajectory rather than a local property of a frame pair, and solving association hierarchically over progressively longer subclips lets evidence from far in the future resolve an ambiguity in the present. Run online for alerts; re-run offline for the archive.
Re-Identification Does Not Merely Degrade. It Stops Being Defined
Three independent reasons, each sufficient on its own.
Pixel budget. ReID networks expect 256 × 128 crops. An SMOT4SB bird is typically under 32 × 32 px; TDTIV drones are 20 × 20 or smaller; BEE24 objects average about 3,099 px². Upsampling a 12 × 12 crop to 256 × 128 manufactures no information — embeddings collapse toward a class prototype and cosine distances stop being informative.
Intra-class identity is genuinely absent. Birds in a flock, bees in a hive, identical drones, players in one kit. Appearance is non-discriminative by construction. DanceTrack was built to expose this; SportsMOT and BFT extend it.
Feature-map stride. At stride 8–16 a 6-pixel object occupies under one cell.
What replaces it: motion models, camera-motion compensation, geometric projection, learned motion (DiffMOT, TrackSSM, MambaTrack, DMTrack), distance-based similarity instead of IoU, and — the interesting one — frame dynamics used as the appearance channel. The 2025 Anti-UAV challenge winners fed frame-difference and flow maps to the detector as extra input channels rather than using motion as a hard gate.
ReID still pays in three places, and they are all above this scale: crowded pedestrians (Deep OC-SORT is +1.7 HOTA over OC-SORT on MOT17), sports players at moderate scale (Hybrid-SORT-ReID is +3.5 HOTA on DanceTrack), and medium-density insects (TOPICTrack’s ReID adds 0.7–4.0 HOTA at 1–18 ms).
Camera Motion: The Aerial Tax
On a drone, ego-motion routinely exceeds target motion, so an image-plane Kalman filter is modelling the aircraft rather than the bird. The DynUAV ablation confirms that camera-motion compensation substantially improves association accuracy and cuts identity switches.
| Method | Mechanism | Cost | When to use it |
|---|---|---|---|
ECC (findTransformECC) |
Iterative photometric alignment | Tens of ms at full resolution — downscale aggressively | Most accurate; StrongSORT’s choice; offline or high-compute |
| Sparse optical flow GMC | Track sparse features, fit affine | ~1–5 ms at 640-px scale | The production default |
| ORB feature GMC | Descriptor matching, robust fit | Moderate | When flow fails on low-texture scenes |
| Ground-plane projection | One homography per sequence, skip per-frame CMC | Near zero | Surveillance, sports, nadir aerial. Not targets against sky |
| Learned keypoints (DISK + LightGlue) | Deep feature matching for affine estimation | GPU-costly | Texture-poor sky, where classical features fail |
That last row was used by an MVA 2025 top-five team, and it exists because sparse feature tracking needs texture that a clear sky does not have.
The UAV-specific leaderboard shows what the accumulated fixes are worth. Note that the leader is appearance-free:
| Method | VisDrone MOTA | VisDrone IDF1 | UAVDT MOTA | UAVDT IDF1 | FPS |
|---|---|---|---|---|---|
| ByteTrack | 35.7 | 48.3 | 41.6 | 59.1 | 28.9 |
| FairMOT | 34.3 | 46.1 | 41.5 | 59.2 | 17.2 |
| OC-SORT | 39.6 | 50.4 | 47.5 | 64.9 | 26.4 |
| UAVMOT (CVPR 2022) | 36.1 | 51.0 | 46.4 | 67.3 | 12.0 |
| FOLT (optical flow) | 42.1 | 56.9 | 48.5 | 68.3 | 29.4 |
| TrackSSM | 41.9 | 55.3 | 48.1 | 65.4 | 24.1 |
| MambaTrack | 43.7 | 57.3 | 48.3 | 67.3 | 25.9 |
| MM-Tracker | 44.7 | 58.3 | 51.4 | 68.9 | 31.1 |
| DMTrack (Oct 2025, appearance-free) | 47.8 | 61.7 | 54.8 | 72.2 | 28.4 |
One Warning About Oracle Detections
HDST-GNN (June 2026) reports VisDrone MOTA 94.51 with oracle detections — and 11.58 with a real YOLOv8n detector. The same gap appears wherever detections are handed to the tracker: SoccerNet-Tracking’s own ground-truth-box baselines reach HOTA 69.55 (DeepSORT) and 71.50 (ByteTrack), while the 82.9–85.7 span sometimes quoted for that benchmark is a tuned reimplementation with YOLOX detections — a different setup rather than a stronger association result.
Any tracking number quoted without naming the detector is describing a different problem from yours.
Single-Object Tracking, Where the Story Gets Strange
Two facts frame this section, and neither is what you would predict.
The best result on the hard benchmark is a zero-shot segmentation model with a Kalman filter bolted on, beating a 1.2-billion-parameter supervised tracker. And the largest single measured gain on tiny thermal targets came from changing one preprocessing hyperparameter — the search region size — not from any architectural or training change.
Search-Region Geometry Is the Finding
A standard single-object tracker crops a square of side roughly 4–5 × √(wh) around the last box and resizes it to a fixed 224, 256 or 384 input. For a 6-pixel target this is pathological in both directions.
Take OSTrack-256 exactly. Search factor 4 on a 6-pixel target means a 24-pixel crop upsampled 10.7× to 256 px, so the target now spans about 64 px — roughly 4 patches per side, about 16 of the 256 search tokens. The problem is not that the target is squeezed into one token. It is that those ~16 tokens are interpolated from about one patch’s worth of genuine sensor signal, 36 px². Upsampling multiplies tokens, not information. Then widen the window to survive fast motion and even that token share collapses.
Two papers measure this precisely, and both win by decoupling field of view from target resolution rather than by scaling the backbone.
ZoomTrack names the dilemma outright: enlarging the context factor at fixed input size “leads to degraded performance due to the decreased target area resolution”. So instead of resizing uniformly, warp non-uniformly — magnify where the target probably is, compress the periphery. A 17 × 17 controllable grid, solved as a quadratic program balancing zoom energy against aspect distortion under a full-coverage constraint, with the importance map being simply a Gaussian at the previous result. Overhead is 1.58 ms on CPU and 1.28 M MACs.
The gain pattern is the proof. OSTrack-256 gains +3.1 on LaSOT-ext, +2.5 on GOT-10k, +2.2 on TNL2K, +1.1 on LaSOT — and +0.1 on TrackingNet, where objects are large and well-resolved. Identical 21.5 G MACs, 119 → 100 FPS. OSTrack-256 with ZoomTrack beats OSTrack-384 on TNL2K (56.5 against 55.9 AUC) at 100 against 61 FPS, using 44.5% of its MACs. (The paper’s prose says “50% faster”, which does not follow from its own 100/61 table — quote the FPS pair instead.)
FocusTrack quantifies the trade on genuinely tiny thermal targets and then makes the search factor adaptive. At the default factor of 4, abrupt camera motion takes the UAV out of the search region entirely. Widening to 6 gives +6.1% AUC. Widening to 7 degrades, and the paper says why numerically: “with a search factor of 7, the target occupies only 1/49 of the total feature space, compared to 1/16 with a factor of 4”, plus more clutter.
Its ablation is the most useful table in this area. From retrained OSTrack’s 53.7 AUC on Anti-UAV410:
| Change | AUC |
|---|---|
| Baseline (retrained OSTrack) | 53.7 |
| + search factor 4 → 6 | 59.8 (+6.1) |
| + DropMAE pretraining | 60.2 (+0.4) |
| + learned search-region adjustment | 62.3 (+2.1) |
| + attention-to-mask module | 62.6 (+0.3) |
| + two-phase training | 62.8 (+0.2) |
On the size subsets it reaches 57.8 AUC on “Small” and 47.3 on “Tiny”, so +12.3 and +11.5 over OSTrack, with attribute gains of +14.1 on out-of-view, +11.8 on fast motion, +14.7 on occlusion and +16.6 on scale variation.
One practical note: full FocusTrack drops to 44 FPS from OSTrack’s 137, mostly from the mask module, while the search-region-adjustment-only variant keeps 143 FPS at 62.3 AUC. Take that one.
Before you buy a bigger model, sweep your search factor. It costs an afternoon.
A measurement-hygiene aside from the same tables: SiamDT is quoted in three places in the literature as 66.8 (AUC), 67.69 (mean state accuracy, as re-measured by the CST Anti-UAV paper) and 68.19 (state accuracy, own paper). Those are three different measurements of the same tracker rather than a discrepancy — and they are not interchangeable.
The LaSOT / LaSOT-ext Inversion
SAMURAI adds two things to SAM 2 and nothing else: a Kalman-filter-based motion score to disambiguate competing mask candidates when appearance cannot, and motion-aware memory selection that refuses to write low-confidence frames into the memory bank. No retraining or fine-tuning at all.
SAMURAI-L reaches LaSOT 74.2 (+5.69 over SAM 2.1-L), GOT-10k 81.7, TrackingNet 85.3 — and LaSOT-ext 61.0. That beats ODTrack-L (53.9), HIPTrack (53.0), LoRAT-L (56.6) and even the 1.2-billion-parameter LoRAT-g378 (56.5). SAMURAI-B at 57.5 also beats every supervised model listed.
Be careful how that margin is constructed. SAMURAI’s own Table 1 does not contain the LoRAT-378 variants — it lists LoRAT-B224 at 50.3 and LoRAT-L224 at 52.8, with ODTrack-L’s 53.9 as its best supervised entry — so the narrower 4.4-point endpoint comes from importing LoRAT-L378’s 56.6 from the LoRAT paper. Inside SAMURAI’s own table the margin reads +7.1 to +10.7. It also loses on OTB100 (71.5 against LoRAT-L’s 72.3), the oldest and most saturated benchmark.
Why does the ordering flip? LaSOT’s training split is part of the standard tracking recipe, so supervised trackers are partly reading a memorised class-and-appearance prior on the LaSOT test set. LaSOT-ext is a disjoint 150-sequence extension built around smaller, faster, more heavily occluded objects from classes largely absent from the training distribution. The memorised prior stops paying, and what remains is the class-agnostic representation plus the motion model. SAM 2 supplies a mask prior trained on vastly more segmentation data than any tracker sees — SA-V holds 50.9 K videos, 642.6 K masklets, 35.5 M masks — and SAMURAI supplies the motion model.
The practical implication is sharp: if your targets are 6 px, LaSOT-ext and Anti-UAV results should outrank LaSOT AUC entirely in model selection. Benchmark rank order on the in-distribution set does not predict rank order on the hard set.
Two footnotes on the SAM family. SAM 2 as a bare tracker is non-monotonic in size on LaSOT — SAM 2.1 T 66.70, S 66.47, B 65.97, L 68.54 — because the failure mode is memory contamination rather than representation quality. And SAMURAI publishes no FPS figures at all, only that overhead is “minimal”, so any real-time claim for it is a claim rather than a measurement; budget around 30 FPS for the L variant from SAM 2’s own 30.2 on an A100. Also note that mask output is the wrong representation at the smallest scales, because mask quality degrades before box quality does. Use box-mode readout for 6-pixel targets.
DAM4SAM is the strongest accuracy configuration available for cluttered tiny-target tracking when paired with SAMURAI, and neither requires training. It splits SAM 2’s single memory buffer into two — recent appearances and distractor-resolving — so a re-entering target is not matched to its look-alike, which is exactly the failure that kills drone and bird tracking in a cluttered sky. First place on the DiDi distractor benchmark (Q 0.694, robustness 0.944), state of the art on 10 of 13 benchmarks in the journal version, and +11% when dropped into EfficientTAM, +4% into EdgeTAM.
There is a hard blocker for product use, though, and it is worth stating plainly: the repository ships no licence file at all. No LICENSE on either branch, no licence statement in the README, no licence label on GitHub. Being a SAM 2 derivative does not grant you a licence to this repository’s code, so the default is all rights reserved. Ask the authors before shipping it.
Efficient Trackers, and What Exports
If you need this on a CPU or a Jetson, the useful frontier is small:
| Tracker | Params | MACs | GPU FPS | CPU FPS | Jetson FPS | LaSOT AUC |
|---|---|---|---|---|---|---|
| HiT-Tiny | 9.59 M | 0.99 G | 204 | 76 | 77 | 54.8 |
| HiT-Small | 11.03 M | 1.13 G | 192 | 72 | 68 | 60.5 |
| HiT-Base | 42.14 M | 4.34 G | 175 | 33 | 61 | 64.6 |
| MixFormerV2-S | 16.2 M | — | 325 | 30 | — | 60.6 |
| Aba-ViTrack | — | — | 181 | 50.3 | 35.6 (TX2) | UAV123 66.4 |
| UETrack-T (2026) | 5 M | 1.1 G | 208 | 79 | 82 | ~63.4 |
| UETrack-B (2026) | 13 M | 3.2 G | 163 | 56 | 60 | 69.2 |
UETrack-B is the strongest single argument here that a small model trained well beats a large model trained badly: LaSOT 69.2 at 13 M parameters and 3.2 GFLOPs statistically ties OSTrack-256’s 69.1 at roughly one-seventh the FLOPs, and runs 56 FPS on CPU. It is very recent, so independent reproduction is thin.
On export, the received wisdom is worth correcting. “Transformer trackers cannot export because of grid_sample” appears misattributed for OSTrack specifically — its search-region crop happens in NumPy and OpenCV preprocessing, not in the graph. What actually breaks is dynamic shapes from candidate elimination: the repo’s own issue thread traces the tracing warnings to two torch.linspace calls, a math.ceil and an equality comparison, all data-dependent control flow around the dynamic top-k. Community forks export with candidate elimination disabled or shapes fixed, which forfeits the 13–40% speedup it was buying.
Where grid_sample genuinely bites is precision. TensorRT does have a native grid-sample layer, but it supports FP16, FP32 and BF16 only — not INT8 — so any tracker that resamples inside the graph gets a forced floating-point island with reformat nodes on both sides. On a bandwidth-bound Orin that can cost more than the layer itself. The practical fix is to replace in-graph resampling with an integer ROI crop plus a bilinear resize on the ISP, VIC or NPP, keeping the network fully INT8 and accepting sub-pixel misalignment.
Working exports do exist: OSTrack at 256² FP16 runs 27 ms end-to-end on a Jetson Orin Nano 8 GB (LightTrack manages 11 ms), and a TensorRT FP16 C++ path with cv::cuda reaches roughly 284 FPS on an RTX 2060 and about 49 FPS on a Jetson AGX NX.
One more piece of trivia that saves a search: there is no “OSTrack V2”. The paper has four arXiv versions, no journal extension, and the authors’ repository contains zero mentions of V2. People saying it usually mean ARTrackV2 or MixFormerV2, or a community ONNX fork.
When a Six-Pixel Target Is Lost
Synthesised from the verified results above rather than quoted from one source:
- Never write low-confidence frames into the template or memory. SAMURAI’s motion-aware selection is the validated version of this.
- On loss, escalate the search region monotonically — local, then 8×, then full frame — rather than trusting a fixed 4× window.
- Run a global class-agnostic small-object detector on the full frame, not on the tracker’s search crop.
- Gate re-acquisition on a motion or Kalman consistency test. A 6-pixel target has almost no appearance evidence to discriminate on, so the trajectory is the identity.
- Require k consecutive consistent detections before re-arming the appearance model.
And pick the hand-off pattern deliberately, because at this scale the hand-off logic is worth more than a model upgrade and costs far less. Detect-to-init is cheapest and fails silently on drift, since nothing ever notices. Tracker-to-fill runs the detector at 1–5 Hz on the full frame and interpolates every frame, which is the right pattern on Jetson-class hardware and exactly what DeepStream’s interval=N plus NvDCF implements in production. Absence-aware requires the tracker to emit a presence score and evaluates with VOT-LT F-score or Anti-UAV state accuracy, which matches how counter-UAS and detect-and-avoid systems are actually judged.
On long-term protocols, the metric is the point. VOT-LT scores precision, recall and F-score on confidence-scored predictions, so the tracker must emit a target-presence score — exactly what you need for a 6-pixel drone that leaves frame. KeepTrack still leads both long-term protocols (VOT-LT F 0.709, OxUvA MaxGM 0.809) at 18 FPS. Plain AUC hides the failure that kills a drone-tracking system.
Training Beats Architecture Here Too
Three results, all with the architecture held fixed.
Pretraining corpus. Tracking needs temporal correspondence, and image-based masked autoencoding never has to learn it. DropMAE does masked autoencoding on video, with dropout applied to spatial attention so the model cannot solve the reconstruction from within-frame context alone. Swapping ImageNet-MAE initialisation for it in an otherwise unchanged architecture gives GOT-10k 73.7 → 75.9 (+2.2) and LaSOT-ext 50.5 → 52.7 (+2.2), for zero inference cost.
Read the paper’s “2× faster pre-training” claim carefully, because it is easy to invert: it means half the epochs to a better result, not a cheaper epoch. On the same 64 × V100 rig, the Kinetics-400 variant reaches 74.8 AO in 800 epochs and 42.2 h against MAE’s 73.7 in 1600 epochs and 84 h — identical per-epoch cost, half the epochs. The best row, Kinetics-700 at 800 epochs, costs 92.4 h, which is 2.2× more per epoch.
The whole 2024–2026 top end sits on this one decision. HIPTrack builds on frozen DropTrack weights; FocusTrack uses DropMAE initialisation. If you are training a tracker from ImageNet weights, you are starting two points behind for no reason.
Parameter-efficient fine-tuning. LoRAT reports LaSOT 0.742 with LoRA against 0.730 with full fine-tuning on the L-224 variant, everything else held constant. Read the whole table, though, not the best row: the effect is variant-dependent, at B-224 −0.8, B-378 +0.3 in full fine-tuning’s favour, L-224 −1.2 and L-378 −0.2, a mean of about −0.5 AUC. So the honest claim is that LoRA matches or modestly beats full fine-tuning while costing a fraction as much, rather than that it reliably wins by a point.
The cost side is where it earns its place: LoRAT-B224 trains in 5.9 h using 2.4 GB, and g378 in 60 h using 25.8 GB — for a 1.2-billion-parameter model. Two design details were prerequisites, and both are about not breaking pretrained structure: positional embeddings decoupled into token-type embeddings plus a sliced sub-matrix of the ViT’s native positional embedding, and an MLP-only head, because convolutional heads failed to converge under LoRA at all.
There is a saturation result buried in the LoRAT ladder that matters more than the method. LaSOT goes B224 71.7 → L378 75.1 → g378 76.2, but LaSOT-ext peaks at 56.6 (L378) and g378’s 56.5 is below it despite four times the parameters. On hard, small, unseen targets, scale stops paying. That is also why a zero-shot mask model beats all of them on LaSOT-ext.
Multi-task data. SUTrack shows that joint multi-task training is better rather than merely convenient, beating per-task models by +19.5% on RGB-D, +9.8% on RGB-T and +6.0% on RGB-E. The sampling ratio alone is worth about a point: RGB at 2× the multimodal rate is optimal, while uniform sampling costs −0.9% average, which the paper attributes to “the limited diversity of multi-modal datasets”. One counterexample in the same ablation is worth knowing, because it stops the lesson from being over-generalised: giving each modality a separate representation instead of channel-concatenating gains 0.9%, so “unified is always better” is not the takeaway.
The actionable version for thermal or depth small-target work: co-train with RGB, do not fine-tune away from it.
One protocol trap to guard against while reading any of these tables. OSTrack states plainly that its GOT-10k figures marked with an asterisk use the one-shot protocol, while its LaSOT and TrackingNet figures come from a model trained on COCO plus LaSOT plus GOT-10k plus TrackingNet. Adjacent columns in a tracking results table describe different models. GOT-10k’s one-shot protocol is also the only column in the standard suite that measures generalisation to unseen object classes, which is exactly what a tiny-target deployment needs, because your target class is almost certainly not in LaSOT.
Benchmarks Where the Objects Are Actually Small
Evaluating on the wrong benchmark measures nothing. Moving from Anti-UAV410 to CST Anti-UAV drops the best state accuracy from 67.69% to 35.92%. DynUAV drops all eleven tested trackers by 12–25% MOTA relative to MOT17. MCATrack gains +5.1 success on TDTIV but only +1.0 on Anti-UAV410. A method that looks marginal on an easy benchmark may be the only thing that works on yours, and the reverse holds too.
| Benchmark | Scale | Object size | Leader |
|---|---|---|---|
| SMOT4SB (MVA 2025, birds from UAV) | 211 seqs / 108,192 frames / 371,690 boxes / 2,240 IDs | most under 32 × 32 px | YOLOv8-SMOT, SO-HOTA 50.59 (baseline 9.90) |
| BEE24 (IEEE TIP 2025) | 36 videos / 446,908 annotations | mean area ~3,099 px²; up to 1,961 tracks in one video | TOPICTrack 46.6 HOTA (ByteTrack 43.2) |
| BFT (CVPR 2024, bird flocks) | 106 seqs / 22 species / 6 continents | small, highly deformable | MOTIP 70.5 HOTA; NetTrack 68.7 |
| DynUAV (CVPR 2026) | 42 seqs / 37,893 frames / >1.7 M boxes | mean object/frame area ratio 1.17 × 10⁻³ | TrackTrack 62.74 HOTA |
| MMOT (Oct 2025, drone multispectral) | 125 seqs / 8 bands / 488.8 K annotations | small, oriented boxes | BoT-SORT 53.6 HOTA — multispectral adds +3.5 to +4.1 |
| CST Anti-UAV (ICCVW 2025) | 220 seqs / >240 K frames | ~96% of instances under 30 px diagonal | best 35.92% state accuracy |
| SportsMOT | 240 clips / 3 sports | players, moderate | Deep-EIoU 77.2 HOTA (CC BY-NC 4.0) |
MOTIP is the most credible transformer candidate for airborne work, precisely because it posts a strong result on a genuinely small-object dataset: BFT HOTA 70.5 / AssA 71.8 / IDF1 82.1, Apache-2.0. Two caveats — it runs about 12 FPS on an RTX 4090 and was trained on eight of them, and it has no tiling story, so it cannot exploit the single largest small-object lever. Also, BFT birds are larger than SMOT4SB birds; a strong BFT result does not imply a strong result at 6 px.
Metrics That Do Not Lie
Every default metric here is IoU-based, and IoU is broken below twenty pixels.
| Metric | Use it when | Behaviour on small objects |
|---|---|---|
| AP / mAP (IoU 0.50:0.95) | Comparing detectors on well-resolved objects | For a 6 px box a 2 px shift gives IoU 0.50, so the 0.75 and 0.90 thresholds are unreachable regardless of model quality |
| Pd / Fa | Any deployed system — the infrared community’s standard | Fine. This is the metric that decides deployability, and it is invisible in AP |
| FROC | Medical, and anything with fixed review capacity | Fine. Recall at a fixed false-positive budget |
| τ-threshold F1 (centre error ≤ 4 px) | Sports balls and point-like targets | Fine. Deliberately ignores unmeasurable shape |
| MOTA / MOTP | Legacy comparison only | Actively misranks. Detection-dominated, unbounded below. Two SMOT4SB top-five entries scored negative MOTA alongside SO-HOTA 43.87 |
| HOTA | The default for MOT | Still IoU-based underneath, so it inherits the cliff |
| SO-HOTA | Tiny-object tracking — the MVA 2025 standard | Fine. Substitutes DotD = exp(−D/S) and averages over 19 thresholds from 0.05 to 0.95 |
| State accuracy | Anti-UAV, and anything where the target leaves frame | Fine. Credits a correct “target absent” decision |
| Streaming AP | Any tiled real-time pipeline | Fine — and the only honest way to score a 32-tile 3.8 FPS configuration against a 6-tile 20 FPS one |
Six Reasons Published FPS Is Incomparable
- Detector included or not. ByteTrack’s 29.6 includes YOLOX-X on a V100; OC-SORT’s ~700 and UCMCTrack’s >1000 are association-only. A 30× discrepancy hides here.
- Hardware generation. V100 → 2080 Ti → 3090 → 4080 → 4090 spans roughly 4×.
- Input resolution. YOLOv8-SMOT’s 5.7 FPS is at 2160 × 3840; WASB’s 30 FPS is at 288 × 512. Same GPU class, 50× the pixels.
- Appearance on or off. BoostTrack goes 65.5 → 15.4 FPS on identical hardware.
- Scene density. ByteTrack drops from 29.6 FPS on MOT17 to 17.5 on MOT20, same GPU.
- Batch versus streaming. Offline methods quote throughput, not latency, and cannot be used online at all.
And Five Pitfalls Specific to Small Objects
Resolution is an uncontrolled variable. Inference resolution moves AP_small more than most architectural claims do. Any comparison that does not fix it is uninterpretable — which is why Cross-DINO’s AI-TOD result is reported “@1200 px”.
AP hides false alarms. Report Pd/Fa or FPPI alongside AP for anything you intend to deploy.
Definition shopping. SODA’s bands are eS (0, 144], rS (144, 400], gS (400, 1024] and Normal (1024, 2000] px², with anything above 2000 px² discarded as ignore. COCO’s “small” is ≤1024 px², so SODA’s first three bands sit inside it while its entire Normal band is COCO-medium. A “small object AP” figure means nothing without the dataset’s definition.
Tiling changes the evaluation, not just the model. SAHI gains are reported as AP50 rather than AP50:95, because the high-IoU thresholds are where tiling helps least.
Leaderboards do not track this problem. The aggregator leaderboards expose no AP_small column, so progress on small objects is invisible on the field’s most-watched scoreboard.
What to Actually Build
For a tiny-object tracker, in this order:
- Fix the detector first. Tiling and resolution, per Part 3. Detection improvement dominated the MVA result: SO-DetA went 8.67 → 47.27.
- Start from OC-SORT or ByteTrack. MIT-licensed, and the base every winner used.
- Replace IoU with expanded IoU plus a normalised centre-distance penalty. That plus EMA velocity is +11 SO-HOTA measured, with no new model.
- Add camera-motion compensation if the platform moves. Sparse optical flow for production; ground-plane projection if a ground plane exists.
- Set
track_bufferto about 1.5 × fps and pair the long coast with distance-based re-association. - Do not buy appearance re-identification. At this scale you are paying a 4.3× throughput tax for an embedding computed from one pixel.
- Evaluate with SO-HOTA or state accuracy, never plain MOTA.
On libraries: roboflow/trackers has the best licence-to-quality ratio (Apache-2.0, detector-agnostic, publishes tuned tables). Roboflow’s supervision is MIT and its InferenceSlicer supports SAHI-style tiling — use NON_MAX_MERGE rather than NMS at tile seams. BoxMOT is the de-facto benchmark harness but AGPL-3.0, which blocks closed-source products. NVIDIA DeepStream is the only production stack with true batched multi-stream tracking, with sub-batching to avoid GPU idling behind CPU-bound association. And skip MMTracking for new work: its last release was 1.0.0rc1 in October 2022.
Part 5 covers the cases where the target is not just small but degraded — blurred, faint, camouflaged — and then the deployment arithmetic that decides your hardware, where a 4K frame is 32 forward passes and that multiplies everything downstream.