Small Objects in the Wild, Part 5: Blur, Sensors, and the Tiling Tax
Part five of five. Part 1 covers the geometry. Part 2 covers stock defaults. Part 3 covers detection. Part 4 covers tracking.
Blur is usually treated as damage to be repaired. The strongest results treat it as information, because a streak encodes velocity, and velocity is exactly what a tiny fast object’s tracker needs.
That reframing is the first half of this article. The second half is the arithmetic that decides your hardware, which is not the model’s FLOPs but the tile count.
How Each Degradation Breaks Things
Small and degraded arrive together, and the degradations fail differently. Knowing which one you have determines whether you reach for restoration, a different loss, or a different sensor.
| Degradation | Physical mechanism | What it destroys | Consequence for assignment |
|---|---|---|---|
| Object motion blur | Displacement > 1 px during exposure; blur length L ≈ v·f·T/(z·p) | Spreads target energy over L px, so peak SNR falls ~1/L; texture gone, only streak orientation survives | There is no single box. Labelling the blur centre versus the leading edge changes both the training target and the recovered velocity |
| Camera shake / vibration | Gimbal jitter, airframe vibration | Global PSF — the background blurs too, so background subtraction breaks | Registration residual explodes across the whole frame |
| Rolling shutter | Row-wise readout skew | A fast small object becomes sheared, elliptical or split | Box regression biased; multi-view triangulation biased |
| Defocus | Long focal length, shallow depth of field at range | Low-pass — sub-pixel targets vanish into the noise floor | The target ceases to exist as a detectable structure |
| Atmospheric turbulence | Refractive-index fluctuation: tilt, blur and scintillation, time-varying | Geometric warp and blur; the target jitters by an amplitude comparable to its own size | Catastrophic: YOLO11m 0.547 → 0.093 mAP on synthetic-turbulence video |
| Low light / high ISO | Photon-limited; short exposure to kill blur starves SNR | Target contrast falls below sensor noise | Training data matters more than restoration — adding night data moved event-camera vehicle mAP 0.566 → 0.761 |
| Compression | H.264/265 rate control discards high-frequency residual in small regions | Exactly the signal a tiny object consists of | A cloud-first pipeline streaming at 2–4 Mbps has already destroyed part of what it is paying to analyse |
| Sub-pixel targets | Target smaller than one pixel footprint | Only a single-pixel intensity bump remains | IoU is undefined. This is anomaly detection, not detection |
| Thermal low contrast | Small ΔT, NETD-limited | No texture at all — the target is a Gaussian blob | Drives the infrared literature to IoU/nIoU plus Pd/Fa instead of box AP |
| Camouflage | Deliberately low boundary contrast | Boundary and region cues degrade together | Camouflaged-object metrics replace AP |
| Occlusion | Player occludes ball; foliage occludes drone | Whole-object evidence absent for 1–20 frames | Must be solved at track level |
Motion Blur: Repair It, Train Through It, or Read It
Three strategies, and the evidence favours the third and second over the first. Pixel-space restoration optimises PSNR, which is not the detector’s objective, and it adds 30–200 ms.
Why Deblur-Then-Detect Usually Loses
The restoration literature is genuinely strong. BSSTNet (CVPR 2024) reaches 35.98 dB on GoPro at 133 GFLOPs and 28 ms per frame, against Shift-Net+’s 35.88 dB at 146 GFLOPs and 45 ms. BSSTNet is notable because it uses a blur map as side information to sparsify attention and gate propagation — treating blur as a signal rather than only as noise. The AIM 2025 efficient-deblurring challenge (under 5 M parameters, under 200 GMACs at 1920 × 1200) was won by a reparameterised NAFNet variant at 31.13 dB, and its conclusion was that efficient deblurring on mobile “remains challenging”.
Sequential deblur-then-detect loses anyway, and there are three independent measurements of it losing.
Tracking. On a 500-video blur benchmark across five blur levels and 23 trackers, light blur improved accuracy for 17 of the 23. Averaging neighbouring frames injects motion evidence. Blanket deblurring with DeblurGAN or SRN reduced accuracy on lightly-blurred video. Selective deblurring, applied only to frames flagged by localisation-error analysis, gave up to +9.3% relative improvement.
Turbulence. LWDETR-Med goes 0.628 clean → 0.192 turbulent. The best restore-then-detect pipeline recovered 0.369. Joint end-to-end training reached 0.419.
Infrared UAV blur. Deblur-then-detect with a strong restorer gave AP50 0.673. Feature-level joint restoration gave 0.767, and on real blurred infrared 0.419 → 0.623 — at 3.5 M parameters, 4.7 GFLOPs and 25.7 FPS.
So, in priority order: blur-robust training with blur augmentation; feature-level restoration with a clean-branch teacher; joint optimisation with an alternating schedule (a weighted joint loss actually degraded mitigation quality in the turbulence study, so alternating was required); and blur as side information. Reserve pixel deblurring for human review or selective per-frame triggering.
“Light blur helps trackers” is a real, measured, counter-intuitive result. Do not deblur reflexively.
Reading the Streak: Deblatting and BlurBall
One literature treats the streak as signal outright. A fast-moving object’s blur is a rendering of its sub-frame trajectory, so solving the image formation model recovers not just a sharp appearance but the path the object took within the exposure — temporal super-resolution from a single frame.
Deblatting (deblurring plus matting) solves I = H*F + (1 − H*M)·B for trajectory H, appearance F and mask M given background B. The line runs Tracking by Deblatting (BMVC 2019) → non-causal TbD (ICCV 2019) → the IJCV 2021 journal version → DeFMO (CVPR 2021), which replaces the optimisation with an encoder-decoder that renders sharp appearances at arbitrary sub-frame offsets, self-supervised by sharpness and time-consistency losses. DeFMO is the best method on the MIT-licensed FMO Deblurring Benchmark: TIoU 0.684, PSNR 26.83 dB, SSIM 0.753 on Falling Objects. It needs a background estimate, so it suits fixed or registerable cameras, and it is heavy relative to a detector.
BlurBall is the productised form, and it is the most transferable idea in this article for any tiny fast object.
Two insights. First, the ground-truth heatmap should cover the whole motion-blurred streak rather than the ball centre, and the label should sit at the blur centre rather than the leading edge — otherwise the network is trained to predict a point that corresponds to no physical instant. Second, if you regress the blur’s length and orientation, you have measured the object’s image-plane velocity from a single frame, which turns trajectory fitting from a multi-frame problem into a nearly single-frame one.
Mechanically it is WASB’s HRNet-style architecture with SE blocks over a 3-frame input at 288 × 512, outputting [heatmap, blur angle θ, half-length ℓ], with θ and ℓ recovered by PCA on the thresholded heatmap. It trained for 30 epochs with Adam and quality focal loss on a new 64,119-frame, 26-recording table-tennis dataset in which 62% of frames are blurred and blur reaches 73 px, with camera calibration included.
Read its two configurations separately, because the difference matters. The default model (+SE) scores F1 96.52, accuracy 93.47, AP 98.23, with blur length MAE 1.5 ± 1.2 px and angle 6.5 ± 18.9°, at 1.49 M parameters and 79 FPS. A threshold-tuned variant (δ = 0.7) reaches F1 97.17 and accuracy 94.75 with tighter blur MAEs, but drops AP from 98.23 to 97.34 — it trades ranking quality for a better operating point. Against baselines under the same mid-blur labelling: WASB 96.00 F1, TrackNetV3 95.93, MonoTrack 94.97. So the fair headline against WASB is 96.52 against 96.00 — half a point, not the 1.2 the tuned row suggests.
The headline that survives the threshold choice is the useful one: using the blur derivative for trajectory prediction cuts error from 84.4 ± 136.6 px to 53.0 ± 87.1 px. Blur is velocity information for free.
Independent corroboration comes from the event-camera side, where predicting image-plane velocity alongside position improved 3D bounce prediction by 36% (RMSE 7.7 ± 16.5 cm). One licence note: BlurBall’s code is MIT, but the dataset ships as a bare download link with no stated licence, so clear it before training a product on it.
The TrackNet Lineage, and What Generalises Out of It
This is the most complete worked example anywhere of a recipe designed for tiny, blurred, fast objects. Its three load-bearing ingredients are all domain-agnostic: temporal stacking to make a sub-10-pixel object detectable, a heatmap instead of a box, and distance-threshold evaluation instead of IoU.
| Version | Year | Input | Key change | Headline accuracy | Params | FPS |
|---|---|---|---|---|---|---|
| TrackNet | 2019 | 1 or 3 frames, 640 × 360 | VGG16-style encoder + DeconvNet decoder, per-pixel softmax heatmap | 1 frame 92.5 F1; 3 frames 98.2 F1 | — | — |
| TrackNetV2 | 2020 | 3 frames, multiple-out | U-Net, multiple-in multiple-out, focal loss | 94.98 acc / 97.03 F1 | 11.3 M | 27.7 |
| TrackNetV3 | 2023 | 8 frames + background image | Video-level mixup; trajectory rectification by inpainting occluded frames | 97.51 acc / 98.56 F1 (from 87.72 acc) | — | 25.11 |
| TrackNetV4 | 2024 | plug-in for V2/V3 | Motion prompt layer: 2 learnable parameters over frame-difference maps | tennis 97.1 → 97.5 F1 | +2 params | 155–163 |
| TrackNetV5 | Dec 2025 | 3 frames, 512 × 288 | Motion Direction Decoupling (signed polarity fields) + residual refinement | 0.9733 acc / 0.9859 F1 | 14.77 M | 114 (T4) |
| TrackNetV6 | ICMR 2026 | — | Linear-multistep decoding, wavelet high-frequency fusion | claims SOTA | — | 183 |
| WASB | BMVC 2023 | 3 frames, 288 × 512 | HRNet-style high-resolution modules, strides removed from stem | tennis 95.6, badminton 93.1, soccer 88.2, volleyball 88.0, basketball 82.6 F1 | 1.5 M | 15–77 |
| MonoTrack | 2022 | — | TrackNetV2 variant, layers removed | 92.1 / 90.9 / 85.2 F1 | 2.9 M | — |
| BallSeg | 2020 | 2 frames (frame ⊕ diff) | Modified ICNet, segmentation-based | volleyball 19.5, basketball 16.8, soccer 36.1, badminton 79.9 | 12.7 M | — |
| DeepBall | 2019 | 1 frame | Tiny CNN heatmap + peak detection | 0.0–64.4 F1 — fails outright on basketball | 0.1 M | — |
The founding result is still the cheapest large gain in this whole series: TrackNet’s Model I (one frame) scores 95.7 precision / 89.6 recall / 92.5 F1, and Model II (three frames) scores 99.8 / 96.6 / 98.2. Note also that it is evaluated at a 5-pixel positioning error, within one ball diameter, rather than by IoU. And note the honest caveat in the same paper: ten-fold cross-validation across nine additional videos drops it to 95.3 P / 75.7 R / 84.3 F1, so the single-video number is optimistic.
TrackNetV3 is worth understanding as a pattern rather than a model. Once per-frame detection is good, the residual error is almost entirely occlusion — the ball behind a player produces no peak. That is a trajectory problem rather than a detection problem, so V3 solves it as one: analyse the predicted trajectory, identify the gaps, and inpaint them with a separate network. Accuracy 97.51, precision 97.79, recall 99.33. The recall figure is the point.
It buys that recall with a precision loss, which is the classic amodal trade-off, and it means the pipeline now emits hallucinated positions. The paper’s own table gives false negatives 4.72% → 0.57% and first-type false positives 0.26% → 1.49% — an 8.3× cut in misses bought with a 5.7× rise in phantom detections. Be careful with cost comparisons here, too: in V3’s own paper on its own hardware it runs 25.11 FPS against V2’s 27.70, about 9% slower rather than an order of magnitude. The 155–163 FPS figures for V2 are V4’s measurements on different hardware, and V4 separately clocks V3 at 15.1 FPS with an explicit warning that it timed “the entire script, including data loading, file writing”. One more naming hazard: an unofficial and more popular repository also called “TrackNetV3” is a different attention-based 3-frame model.
WASB is the strongest cross-sport baseline and the one to start from. Three things done properly rather than one new idea: keep the feature map at high resolution, train with a position-aware objective on real-valued targets, and enforce temporal consistency at inference with an explicit constant-acceleration motion model (p̂ₜ₊₁ = pₜ + vₜ + aₜ/2 from finite differences) and candidate filtering — no Kalman filter at all. A single 1.5 M-parameter recipe transfers across five sports, beating TrackNetV2 at one-seventh the parameters, with AP gains of +7.8 to +16.8. It is MIT-licensed.
The most instructive rows in that table are the failures. Segmentation-based BallSeg collapses to 19.5 F1 on volleyball and 16.8 on basketball while scoring 79.9 on badminton. Single-frame DeepBall scores 0.0 F1 on basketball. Those are exactly the sports where the ball is smallest, fastest and most often occluded. It is the clearest evidence anywhere that stacked-frame heatmap regression is the right inductive bias for blurry tiny objects, and that a box or a mask is not.
The Transfer Gap
Here is the clearest hole in this field. The heatmap-stack recipe has never left sports.
Repository and citation searches for TrackNet or WASB applied to birds, drones or insects returned nothing during compilation, and TrackNetV6’s own abstract frames cross-domain generalisation as still open while staying inside sports.
It should transfer, because all three load-bearing ingredients are domain-agnostic and BlurBall’s blur-as-velocity result applies directly to birds and drones. What does not transfer is the sports prior: fixed broadcast camera geometry, one target instance, a background that can be median-estimated, and a bounded trajectory class. Those four assumptions are precisely what the counter-UAS literature has to abandon.
A registered or stabilised background substitute for the median image is the missing piece, and the real obstacle is multi-instance — heatmap peak-picking with a reject option handles one ball elegantly and N birds awkwardly. The currently-validated alternative is the MVA winners’ answer from Part 3: tiled box detection plus distance-based association.
Infrared: The Most Honest Metrics in the Field
This subfield has the healthiest edge story here and the most honest reporting, and the reason is that it reports probability of detection and false-alarm rate rather than only IoU — because Fa is what decides deployability and it is invisible in AP.
The IRSTD-1k table has to be split by protocol, because the two halves are not comparable:
| Method | Year | IoU | Pd | Fa (×10⁻⁶) |
|---|---|---|---|---|
| Standard protocol — trained and tested on IRSTD-1k | ||||
| DNA-Net | 2023 | 65.71 | 91.84 | 17.61 |
| UIU-Net | 2023 | 66.64 | 89.23 | 16.02 |
| MSHNet | CVPR 2024 | 67.16 | 93.88 | 15.03 |
| Cross-dataset generalisation — trained on three merged sets, evaluated independently | ||||
| UIU-Net | 2023 | 39.10 | 92.62 | 7008.86 |
| DNA-Net | 2023 | 64.14 | 95.64 | 40.90 |
| RealScene-ISTD method | 2025 | 72.44 | 95.30 | 14.08 |
The two halves of that table are the whole lesson. UIU-Net scores IoU 66.64 with Fa 16.02 when trained and tested on IRSTD-1k, and IoU 39.10 with Fa 7008.86 — a 440× worse false-alarm rate — when the same architecture is asked to generalise. DNA-Net degrades the same way.
So the honest statement is not “UIU-Net has a bad false-alarm rate”. It is that false alarms are what explodes first when an infrared detector is asked to generalise, long before IoU tells you anything is wrong. Always report Pd/Fa or FPPI alongside IoU, and always report which protocol produced them.
Two methods worth knowing. MSHNet’s contribution is a scale- and location-sensitive loss, addressing the fact that IoU-style objectives are insensitive to both for a tiny blob. That is the same diagnosis as NWD and SAFit from Part 3, arrived at independently in the infrared community — strong convergent evidence that the diagnosis is right. It also improves IoU and false alarms together, which is rare.
WSNet (January 2026) inverts the usual design reasoning deliberately: infrared imagery carries very little semantic content, because a target is a blob, so depth is wasted while width buys multi-scale spatial sensitivity. So the network is wide and shallow — a width-extension module with multi-branch 3/5/7 and dilated convolutions plus channel-spatial hybrid attention. The result is 0.054 M parameters and 1.050 GFLOPs, at up to 146 FPS on GPU and 30 FPS on CPU. That is two to three orders of magnitude below DNA-Net-class models, which makes CPU-only counter-UAS payloads genuinely plausible. Its efficiency claim is fully verified; its head-to-head accuracy claims were not extractable during compilation, so weight them less.
The design principle generalises past infrared: if your modality carries no texture, spend parameters on scale coverage rather than depth.
The classical family (IPI, NRAM, RIPT, PSTNN) models a frame as low-rank background plus sparse target, needs no training, and is still the baseline every 2024–2026 paper must beat. It stays relevant because it is deterministic, explainable and certifiable, which matters more than AP in some procurement contexts.
One perspective-setting measurement from the sensor-fusion side: F1 by modality runs thermal IR 0.61–0.88, visible 0.69–0.86, and audio around 0.93, with radar-vision fusion at 93.0% precision and 83.7% recall in occluded scenes. Single-modality vision is the weakest link at range.
Camouflage, and Why It Is the Same Problem
When appearance is deliberately non-discriminative, motion is the only reliable cue — so video camouflaged-object detection is the closest well-studied analogue to faint small-target detection, and it reached the same conclusion independently.
Vcamba (2025) uses a state-space backbone with frequency-domain sequential scanning and beats SLT-Net by +4.3% Sα, +7.0% weighted F and −52.3% MAE at 16.49 G MACs and about 14 FPS. CAMotion (2026) is the new benchmark: 474 sequences, roughly 150 K frames, 30,028 annotated frames, 151 species, with HGINet (Sα 0.774) and ZoomNeXt (0.779) leading.
The transferable warning is in the corruption study, and it is unambiguous. Camouflage architectures buy low-contrast sensitivity but not blur robustness. Geometric corruptions — motion blur, Gaussian blur, noise — are the most harmful, while photometric ones (brightness, contrast, fog, JPEG, rain) are much less so. SINet-V2 loses 18.5 Dice points to motion blur (0.734 → 0.549); ZoomNet goes 0.699 → 0.523; PFNet 0.681 → 0.530.
So take the zoom, boundary-uncertainty and frequency-domain machinery from the camouflage literature, and add the blur machinery above on top. Neither alone is sufficient.
A related limit worth naming: semantics-based motion methods break at this scale for a structural reason. ZBS (CVPR 2023) raises background modelling from pixels to instances using a zero-shot detector, and gains +4.70% F-measure over the previous best unsupervised method on CDnet 2014, with better shadow and night handling. But it inherits the detector’s blind spot, and an open-vocabulary detector will not fire on a 6-pixel blurred streak — see Part 3. Semantics require pixels, and the whole premise here is that there are none.
Below the appearance floor, around 3 px, there is no per-frame decision worth making, and track-before-detect is the only principled route. Accumulate evidence along a hypothesised trajectory before thresholding. The modern learned forms are frame-differencing and flow maps as extra detector input channels, sparse spatio-temporal point clouds, and shift-and-stack synthetic tracking. The detection delay this creates is a design parameter rather than a defect: the AOT dataset’s 10 Hz capture rate exists precisely so a detect-and-avoid system can accumulate about three seconds of track before committing.
Event Cameras: The Strongest Single Result Here Is a Sensor Change
On the EV-UAV benchmark, a sparse event-based network reaches IoU 55.18% against YOLOv10-S’s 32.55% and Faster R-CNN’s 26.93% — roughly a 70% relative gain over the best frame-based detector on the same scenes. The targets average 6.8 × 5.4 px, with about 45% under 64 px².
The mechanism is geometric. Small moving targets form continuous curves in the spatiotemporal event point cloud, and those curves are separable from noise (which is isolated) and from background surfaces (which are extended). The target’s trajectory becomes the detectable structure, at microsecond resolution and with no motion blur at all.
The sports evidence is the most striking corroboration. A 2026 table-tennis system with four Prophesee EVK4 cameras (1280 × 720) achieves 400 Hz detection with three cameras, at 12.5 ms or less end to end, with 2D localisation error of 0.75 px — about 8 mm at 3 m — and 3D bounce prediction at 7.7 ± 16.5 cm RMSE. That beats the best frame-based table-tennis system, which runs at 149 Hz. An egocentric ping-pong system reports 92.59% detection success within 5 px at 1.5 ms ball-detection latency.
On the general event-detection side, SMamba (AAAI 2025) reaches Gen1 50.4 and 1Mpx 49.3 mAP at 24–26 ms by pruning tokens on spatio-temporal continuity, cutting FLOPs 22–23% while raising accuracy — because activity events are spatio-temporally continuous while noise is not, so pruning by continuity both accelerates and denoises.
Two real limits. Training data availability is the binding one: eTraM shows how sharply performance depends on it, with day-only training giving 0.566 vehicle mAP against 0.761 for day-plus-night. There is no dominant open corpus for faint targets against sky. And event streams give motion, not classification, so the realistic architecture is hybrid — event or infrared for detection, frame for classification — rather than event-only.
On accessibility in 2026, the low end is genuinely cheap. The Prophesee GenX320 is 320 × 320 at 6.3 µm pitch, with over 140 dB dynamic range, under 150 µs latency, over 10,000 fps equivalent, 36 µW to 3 mW power, a 0.05 lux cutoff, in a 3 × 4 mm package — available as bare die, EVKs, an STM32 Discovery kit and an OpenMV module. HD event sensing still means EVK-priced hardware.
The Optics Trade That Sets the Algorithm’s Job
Four decisions upstream of the model, each of which changes what the model has to do. Our camera selection guide covers the general case; these are the small-object specifics.
Exposure. Halving exposure halves blur length but also halves photons, so SNR falls by roughly √2 in the shot-noise regime. There is a genuine optimum, and at long range in low light the blurred frame often carries more detectable energy than the noisy short-exposure one. This is why “light blur helps trackers” is not a fluke, and why BlurBall’s strategy of accepting blur and regressing it beats fighting blur with exposure.
Resolution versus field of view. Pixels on target at range z is D·f/(z·p). Counter-UAS operates at 2–10 px on target, which is why the infrared small-target literature exists at all. Fixing it optically means a narrow field of view, which then requires a cue to point the sensor — hence radar, acoustic, or a wide-angle motion detector, and hence the fusion numbers above where audio scores 0.93 against visible’s 0.69–0.86.
Frame rate. A ball at 100+ km/h (about 28 m/s) needs 200–400 Hz to keep inter-frame displacement inside a plausible search radius. At 30 fps a shuttlecock moves multiple body-lengths per frame and IoU-based association is hopeless — which is exactly why sports trackers are heatmap-plus-trajectory rather than detect-and-associate.
Shutter and dynamic range. Rolling shutter shears fast small objects and biases triangulation, so use global shutter for metric ball tracking. Sky-background targets sit against a five-decade luminance ramp, where the event sensor’s >140 dB is the cleanest answer and HDR bracketing costs frame rate. One gap: no published measurement of rolling-shutter-induced ball-localisation bias surfaced, despite it being a first-order error for triangulated ball tracking.
And gimbal stabilisation is a perception component rather than a mechanical nicety. It is what keeps the warp residual near the noise floor, which is what makes registration-based change detection viable at all.
The Tiling Tax, Quantified
Now the arithmetic that decides your hardware. At 640 × 640 tiles with 20% overlap (stride 512):
tiles = (⌈(W−640)/512⌉+1) × (⌈(H−640)/512⌉+1)
That gives 32 tiles for 4K, 77 for 20 MP, and 135 for 8K — plus one more for the downscaled full-frame pass that standard practice includes to catch objects larger than a tile.
Those figures cover only the forward passes. They exclude the 32 to 135 crops and resizes per frame, the 32 to 135 NMS invocations plus a cross-tile merge with duplicate removal in the overlap bands (routinely more CPU time than the GPU inference on a dense 4K scene), and the roughly 4× activation memory that batching 32 tiles at 640² requires versus batch 8 — a real constraint on an 8 GB Orin NX.
Effective full-frame throughput, derived from measured per-tile latencies, inference only:
| Device / model | ms per tile | 4K (32 tiles) | 20 MP (77 tiles) | 8K (135 tiles) |
|---|---|---|---|---|
| Orin NX 16 GB, YOLO26n INT8 | 3.49 | 111.7 ms → 8.9 FPS | 269 ms → 3.7 | 471 ms → 2.1 |
| Orin NX 16 GB, YOLO26s INT8 | 4.78 | 153 ms → 6.5 FPS | 368 ms → 2.7 | 645 ms → 1.6 |
| Orin NX 16 GB, YOLO26m INT8 | 8.22 | 263 ms → 3.8 FPS | 633 ms → 1.6 | 1110 ms → 0.9 |
| Orin Nano Super, YOLO26s INT8 | 5.25 | 168 ms → 6.0 FPS | 404 ms → 2.5 | 709 ms → 1.4 |
| AGX Thor, YOLO26s INT8 | 1.75 | 56 ms → 17.9 FPS | 135 ms → 7.4 | 236 ms → 4.2 |
| Hailo-8, yolov8s (vendor zoo) | 2.04 | 65 ms → 15.3 FPS | 157 ms → 6.4 | 275 ms → 3.6 |
| Hailo-8, yolov8s derated 0.55× | 3.71 | 119 ms → 8.4 FPS | 286 ms → 3.5 | 501 ms → 2.0 |
| L40S, RT-DETR 640 FP16 | 1.56 | 49.8 ms → 20.1 FPS | 120 ms → 8.3 | 210 ms → 4.8 |
A tiled 4K pipeline on Orin-class hardware lands at 4–9 FPS. That is the number the rest of your design has to work around.
Mitigations, Ranked by Payoff
NMS-free heads. YOLOv10 and YOLO26-style one-to-one heads remove per-tile NMS entirely. On a 32-tile 4K pipeline this is the biggest CPU win available.
Cascade: low-resolution full frame, then crop where something fired. One 640² pass costs 3.5–4.8 ms, then tile only the k active regions. Typical aerial scenes need 3–8 tiles instead of 32, a 4–10× cost reduction. It costs recall on objects invisible at the downscaled resolution — which is exactly the smallest ones — so gate the proposal on a P2-level feature response rather than on final detections.
Motion-gated tiling. Use the ISP or a cheap frame difference on the decoded NV12 to select tiles. Excellent for static cameras, useless on a moving platform where global motion dominates.
Frame skipping plus tracking to fill. Detect every Nth frame and propagate with a cheap tracker. DeepStream exposes this as interval=N on nvinfer, and this is where NvDCF’s near-zero cost pays: at 4K with 32 tiles and interval = 4, effective throughput quadruples with identity maintained. This is the standard production answer.
Dynamic scheduling. “Towards Streaming Perception” shows that latency is sometimes minimised by sitting idle rather than starting a frame you will finish too late, and that asynchronous tracking plus forecasting emerge as the right internal representation.
Hardware ISP and scaler. On Jetson, do crop-and-resize with VIC, nvvidconv or NPP and keep buffers in NVMM. Never extract tiles in NumPy.
Batch the tiles as if they were streams. Push them through nvstreammux to get the batched-engine gains (2–3× from measured batch-1 to batch-8 comparisons) plus zero-copy and a tracker downstream.
One reporting obligation comes with all of this: if you build a tiled pipeline, report streaming AP. Offline mAP will make a 32-tile 3.8 FPS configuration look better than a 6-tile 20 FPS one, which is the wrong answer for anything that moves.
Quantisation Takes Small Objects First
What good post-training quantisation costs, measured on Hailo’s Dataflow Compiler at COCO 640²: yolov8n 37.0 → 36.4 (−0.6), yolov8s 44.6 → 44.0, yolov8m 49.9 → 49.2, yolov11n 39.0 → 37.8 (−1.2), yolov11m 51.1 → 49.8 (−1.3). That is a well-engineered flow with per-channel weights, careful calibration and equalisation.
What naive quantisation costs, on YOLO12 under static TensorRT INT8: nano 0.4047 → 0.3325, a −7.2 point drop, against small −6.5, medium −3.7, large −2.7 and x-large −3.1. Smaller backbones lose two to three times more accuracy to INT8 than large ones, which is the exact opposite of what an edge engineer wants.
Small objects go first for a specific reason. Their signal lives on the highest-resolution, lowest-magnitude feature maps. Per-tensor activation scales are set by the max over the whole tensor, which is dominated by high-activation large-object channels, so small-object channels get quantised into a handful of levels. And the regression head’s sub-cell distinctions fall below one INT8 step. Per-channel weight quantisation plus per-channel or histogram activation quantisation is not optional here.
How much is at risk? DEIM and D-FINE report AP_small around 36.9–38.8 against overall AP of 54.7–56.5, so AP_small is already 17–18 points below AP. A hit that reads as “only −1.3 mAP overall” can be several points of AP_small, because small objects are a minority of instances and dominate the marginal-confidence band where clipping flips detections to misses.
One honest gap: no public source reporting AP_small separately under PTQ INT8 surfaced during compilation. The mechanism is well-founded; the constant is unmeasured. Which means every edge deployment is currently making this trade blind, and the practical response is to evaluate AP_small when accepting a quantised model, and calibrate on 500–1000 images that actually contain your small objects. A COCO-generic calibration set is malpractice for aerial data. Prefer FP8 on Hopper, Ada, Blackwell and Thor as the middle path for transformer detectors; INT4 is for LLM weights, not detection activations.
Four Ops That Break Small-Object Pipelines
Worth knowing before you pick an architecture, because these decide whether it exports at all:
grid_sample. TensorRT has a native grid-sample layer but supports FP16, FP32 and BF16 only — not INT8. Any detector or tracker that resamples inside the graph therefore has a forced floating-point island in an otherwise-INT8 engine, with reformat and quantise nodes on both sides. On a bandwidth-bound Orin that can cost more than the layer itself. ONNX opset 16’s GridSample also does not support 5D input, so temporal deformable sampling needs a custom plugin. Fix: integer ROI crop plus a bilinear resize on the ISP, VIC or NPP.
Deformable attention. MSDeformAttn is not a standard ONNX operator. Every Deformable-DETR-derived model needs a TensorRT plugin or a decomposition into gather + grid_sample + einsum, which reintroduces the floating-point island above. This is a large part of why RT-DETR, D-FINE, DEIM and RF-DETR deliberately use simpler attention: they export cleanly.
NMS variants. TensorRT’s EfficientNMS and INMSLayer plugins are fast but constrain max detections and box format. Soft-NMS, cluster-NMS and class-agnostic-with-per-class-thresholds usually land on the CPU. This is the single largest hidden cost in tiled high-resolution pipelines, where NMS runs over N tiles’ worth of boxes. NMS-free heads are a genuine deployment win rather than a paper trick.
Dynamic shapes. Adaptive tiling, variable tile counts and variable-aspect letterboxing force min/opt/max profiles in TensorRT, a full recompile in the Hailo and RKNN toolchains, or outright rejection by the Intel NPU plugin and Apple’s ANE. Fix: pad to a fixed tile batch and mask.
Sizing the Hardware
Comparing the whole Jetson and accelerator range is a longer subject, and we covered when Jetson Orin is the right fit separately. Four findings here are specific to small-object work.
Do not size Thor from its TOPS number. The headline 2070 TFLOPS is FP4 sparse. Detection and tracking run INT8 and FP16, and the measured scaling for CNN detectors is roughly 2.5–4.0× Orin Nano Super and 1.5–2.2× AGX Orin 64 GB depending on precision (INT8 2.50× / 1.51×; FP16 3.29× / 1.88×; FP32 3.96× / 2.20×, on YOLO26n at 640) — not the roughly 8× the TOPS ratio implies. Memory bandwidth only rose from 205 to 273 GB/s, a factor of 1.33, and high-resolution small-object pipelines are bandwidth-bound. Also halve every “sparse” figure: Orin Nano Super is 67 sparse and 33 dense.
Pick the board for the power budget, not the TOPS. An independent externally power-metered study (INA260 inline sensor, batch 1, 640²) found YOLO11s INT8 running at 145.82 FPS and 0.0767 J per inference on an Orin Nano Super at 15 W, against 80.90 FPS and 0.1814 J on an AGX Orin 64 GB at the same 15 W. At matched 15 W the Orin Nano Super beats the AGX Orin 64 GB by 1.8× and uses 2.4× less energy per inference, because the larger die is inefficient when clocked down into a tight envelope. The same study found that Jetson’s internal tegrastats power telemetry underestimates true board-input power by about 1.99 W (pooled bias over 76,691 aligned samples), so every watts-per-FPS figure derived from it on a 7–15 W device is optimistic by 13–28%.
Derate vendor accelerator numbers by roughly half. The cleanest measurement of this is a Raspberry Pi 5 with a Hailo-8, running YOLOv8m at 640² from 1280 × 760 video at batch 8:
| Concurrent streams | Gen2 per stream | Gen2 aggregate | Gen3 per stream | Gen3 aggregate |
|---|---|---|---|---|
| 1 | 39.82 | 39.8 | 76.99 | 77.0 |
| 2 | 19.86 | 39.7 | 38.21 | 76.4 |
| 4 | 8.45 | 33.8 | 16.94 | 67.8 |
| 8 | 3.85 | 30.8 | 8.15 | 65.2 |
| 12 | 2.94 | 35.3 | 5.43 | 65.2 |
(Per-stream figures are measured; the aggregate columns are the product, computed here.)
The Hailo model zoo reports yolov8m at batch 8 as 141 FPS. Real single-stream throughput on a Pi 5 is 77 FPS — 55% of the vendor number — and the PCIe generation matters by 1.9×, meaning the host link, not the 26 TOPS NPU, is the constraint. Use a 0.5–0.6× derating factor on zoo figures for planning.
Three accelerator traps. The Hailo-10H’s INT8 vision throughput is lower than the Hailo-8’s, despite the higher number — it is GenAI-oriented, so do not “upgrade” a detector to it expecting speed. The Hailo-8L is 1.2–5.1× slower than the Hailo-8, and the spread is the point: yolov8n/s at batch 1 are 4.5–5.1× slower but yolov11n/s only 1.2×, so benchmark your own model rather than applying a factor. And the Google Coral Edge TPU is effectively abandoned — its news page has been unchanged since May 2022, the Gasket driver is broken on modern kernels, Frigate now steers users elsewhere, and the product page redirects to a developer landing page with no price at all. Do not design it in.
For phone-class deployment, three readings from Qualcomm AI Hub NPU-only measurements at 640²: INT8 is about 4.6× faster than FP16 on the same NPU (YOLOv8n on Snapdragon X2 Elite, 0.67 ms w8a8 against 3.124 ms float); w8a16 sits in the middle and is the pragmatic fallback when w8a8 loses too much small-object recall; and the flagship-to-industrial gap is about 15× (0.67 ms against 10.423 ms on a Dragonwing Q-6690), so never size an industrial box from phone-SoC numbers. Sub-millisecond inference also means the NPU is not your bottleneck — camera capture, YUV→RGB conversion, letterboxing and NMS dominate, so budget 8–20 ms per frame of non-NPU work.
The Table That Includes the Tracker
Most benchmarks omit the tracker, which makes them useless for sizing a real pipeline. DeepStream 9.1’s official end-to-end figures include it, with rendering off:
| Model | Res | Tracker | AGX Orin | AGX Thor | L40S | RTX PRO 6000 | B200 |
|---|---|---|---|---|---|---|---|
| RT-DETR | 640² | none | 96 | 208 | 643 | 1063 | 1448 |
| RT-DETR | 640² | NvDCF | 96 | 189 | 642 | 1024 | 1417 |
| PeopleNet 2.6.3 | 640² | MV3DT | 147 | 349 | 1114 | 1899 | 3572 |
| PeopleNet Transformer | 640² | MV3DT | 16 | 25 | 142 | 224 | 243 |
| Grounding-DINO | 544 × 960 | none | 13 | 24 | 107 | 178 | 216 |
Two readings. NvDCF is nearly free on GPU — 643 → 642 FPS on an L40S. A correlation-filter tracker maintaining identity across frames where the detector did not run is the cheapest throughput multiplier available, and it is the mechanism behind interval=N.
And note the transformer cost: PeopleNet Transformer at 142 FPS against PeopleNet CNN at 1114 on the same GPU is a 7.8× multiplier at matched resolution, precision and tracker. Be careful how you read that, though — it is not a clean architecture ablation, because PeopleNet Transformer is a ResNet-50 Deformable-DETR while PeopleNet 2.6.3 is a small DetectNet_v2-class CNN, so capacity and accuracy differ too. It is a real cost you will pay rather than a measurement of “transformers cost 7.8×”.
Two pipeline facts that catch people out. Decode is almost never the bottleneck on Ada or newer — an L4 has 4 NVDEC engines and decodes roughly 120 × 1080p30 H.264 — but it is a bottleneck on an Orin Nano. And note that A100 and H100 have zero NVENC engines, so they cannot re-encode at all. On zero-copy: one host round-trip per frame per stream at 4K NV12 is about 12 MB, so at 30 streams × 30 fps that is 10.8 GB/s, roughly 11% of an Orin NX’s entire 102 GB/s bandwidth budget spent on copies before a single convolution runs.
Realistic 1080p30 stream counts for a detect-and-track pipeline, derived from that table:
| Device | RT-DETR 640 + NvDCF | PeopleNet 640 + MV3DT | PeopleNet Transformer |
|---|---|---|---|
| AGX Orin 64 GB | 3.2 | 4.9 | 0.5 |
| AGX Thor | 6.3 | 11.6 | 0.8 |
| L40S | 21.4 | 37.1 | 4.7 |
| RTX PRO 6000 Blackwell | 34.1 | 63.3 | 7.5 |
What Is Realistic Per Tier
| Tier | Stack | Resolution / FPS | Small-object reality |
|---|---|---|---|
| MCU-class (~1 W, $10–20) | STM32N6-class, INT8 single-class detector, no tracker or a 2-state Kalman | 320², 5–15 FPS | Only objects above ~32 px. No tiling. AP_small effectively zero |
| Mobile phone | YOLO11n/YOLOv8n w8a8 (0.67–0.73 ms NPU on a flagship) + ByteTrack in native code | 640², 30 FPS with headroom | 2-tile or cascade tiling is affordable. Use w8a16 if AP_small drops too far. Phones throttle in 3–5 min |
| Small edge box (5–15 W, $150–500) | Pi 5 + Hailo-8, or Orin Nano Super at 15 W; YOLOv8s/11s INT8 + NvDCF | 1–2 × 1080p30, or 4K/32-tile at ~8 FPS | Insist on PCIe Gen3 — a 1.9× difference. Use cascade plus interval=3 to reach ~25 FPS effective |
| High-end edge (30–60 W, $1–2 k) | Orin NX 16 GB or AGX Orin + DeepStream; RT-DETR or YOLO26s INT8 + NvDCF | ~3 × 1080p30, or one 4K/32-tile stream at 6.5 FPS | The sweet spot for tiled aerial and maritime work. Thor at $3,499 and 130 W is rarely justified for detection alone |
| On-prem server | 1–2 × L4 or L40S; DeepStream or Triton; CNN detector FP16 + NvDCF | L40S: 37 × 1080p30 with tracker, or 21 with RT-DETR | The highest-ROI tier for 10–40 cameras. Watch NVENC counts if you re-encode |
| Cloud | L4 for burst and backfill; L40S for dense multi-stream; H100/B200 only for training | On demand | The right place to run the expensive tiled or transformer configuration for forensic re-processing |
Cost, and the Thing That Usually Decides It
Verified August 2026 cloud pricing: g4dn.xlarge (1 × T4) at $0.526/hr on demand, $0.331 reserved; g6.xlarge (1 × L4) at $0.80; g6e.xlarge (1 × L40S) at $1.861; market medians of $2.00 for A100, $4.24 for H100 and $7.88 for B200.
Cloud per stream. An L40S at 37.1 concurrent streams costs $0.050 per stream-hour, about $36.6 per stream-month at full utilisation. The same GPU running a transformer detector (4.7 streams) costs $0.394 per stream-hour, about $288 per stream-month — the same 7.8× multiplier, now in dollars.
Edge per stream. A Pi 5 8 GB (about $80) plus a Hailo-8 M.2 ($179) plus HAT, PSU and enclosure is roughly $320–400 for 1–2 streams, so $160–400 per stream one-time at about 8 W. Amortised over three years that is $4.4–11 per stream-month plus about 5.8 kWh/month. An Orin NX box at $900–1,400 for 2–3 streams works out to $300–470 per stream, or $8–13 per stream-month over three years at 25 W.
So edge is 3–8× cheaper per stream than cloud on a three-year TCO, before bandwidth. Cloud wins on elasticity, iteration speed, and access to configurations edge cannot run at all.
Bandwidth is usually what decides it. A 1080p30 stream at 4 Mbps is about 1,300 GB per month per camera. At 8 Mbps — which you need if small objects are to survive compression at all — it is about 2,600 GB. AWS charges nothing for ingest, so the cost is the site uplink plus any egress; at a commonly-published $0.09/GB, pulling back even 10% of one camera’s stream is roughly $11.70 per camera-month.
Cellular or satellite backhaul kills cloud-first designs outright. At even $1/GB, 1,300 GB is $1,300 per camera per month — two orders of magnitude above the amortised cost of an edge box.
And compression is an accuracy problem, not only a cost problem. Small objects are the first casualty of H.264 and HEVC rate control, because they occupy few macroblocks and low-contrast high-frequency detail is exactly what gets quantised away. Detect at the edge, on the pre-encode frame.
The Licence Problem Is Bigger Than It Looks
At least fifteen of the datasets in this field prohibit commercial use, and they include the two most heavily used aerial small-object benchmarks.
Blocked for commercial use: AI-TOD (dataset CC BY-NC-SA 4.0, with AI-TOD-v2 reusing the same images and AI-TOD-R publishing no dataset licence at all, so treat both as inherited non-commercial), DOTA (“academic purposes only… any commercial use is prohibited”), SODA-D and SODA-A (CC BY-NC 4.0), xView (CC BY-NC-SA 4.0), SportsMOT (CC BY-NC 4.0), MVTec AD (CC BY-NC-SA 4.0), TT100K (CC BY-NC), SKU-110K, IP102, DeepPCB, Argoverse-HD (inheriting Argoverse 1’s CC BY-NC-SA 4.0), Drone-vs-Bird (signed agreement, research only) and SoccerNet (video behind an NDA).
Clean: AOT under CDLA-Permissive-1.0 with commercial use permitted, SeaDronesSee (dataset CC0 1.0, code MIT), DUT Anti-UAV (Apache-2.0), WASB (MIT), VISEM-Tracking and LUNA16 (CC BY 4.0), COCO (CC BY 4.0).
Unresolved — a code licence exists but dataset terms are simply not published, so clear them before a commercial deployment: VisDrone, SOD4SB, the SMOT4SB dataset, TinyPerson, RGBT-Tiny, EV-UAV, TTPLA, the SIRST family, BEE24, BFT, DynUAV, MMOT, the Anti-UAV sets, TDTIV, the TrackNet datasets and the BlurBall dataset.
On code: Ultralytics, YOLOv10, YOLOE and BoxMOT are AGPL-3.0. PyTracking and StrongSORT are GPL-3.0. DEIM, DEIMv2, D-FINE, RT-DETR, RF-DETR (N–L), SAM 2, SAMURAI, SPMTrack, YOLOX, MMDetection, PaddleDetection, roboflow/trackers and FiftyOne are Apache-2.0. SAHI, supervision, ByteTrack, OSTrack, Aba-ViTrack, UCMCTrack, WASB and TrackEval are MIT.
Two traps in particular. RF-DETR’s XL and 2XL ship from a separate package under PML 1.0, which conditions your rights on holding a current platform plan and is not OSI-permissive. And DAM4SAM — the best distractor-handling tracker available — ships no licence file at all, so the default is all rights reserved.
Four Domain Playbooks
Aerial and drone video. SliceTrain at 1024–1280 px tiles with 20–25% overlap; SAHI plus a full-image pass at inference with Cluster-DIoU-NMS; imgsz=1280–1536, rect=True; add a P2 head; swap the assignment metric to NWD or DotD and drop the TAL β from 6.0 to 2–4; oriented boxes if objects are dense and aligned. Augmentation scale=0.2, translate=0.05, close_mosaic=25–30, multi_scale=0.3, and honour ignore regions. Postprocessing conf=0.05–0.10, iou=0.55, max_det=1000–2000, and raise evaluation maxDets above 100. Track with the OC-SORT family plus sparse-flow GMC (or ground-plane projection for nadir), expanded IoU plus a distance penalty, track_buffer ≈ 1.5 × fps. Expect +12 to +15 AP50 from tiling alone, +2 to +3 from P2, and +4 to +10 from assignment.
Ground camera and surveillance. Tiling is often unnecessary — crop fixed regions of interest at native resolution instead. imgsz=960–1280, add P2 only if targets are under 20 px, conf=0.15, iou=0.6. Mine hard negatives from the site’s own clutter, which is stationary and therefore learnable. Track with BoT-SORT, gmc_method: none for a static camera, track_buffer = fps, match_thresh=0.9, fuse_score=False, and with_reid=True — appearance is genuinely available above about 30 px and worth +1.7 to +3.5 HOTA. In production, set tracker-width/tracker-height to the full stream resolution rather than the 640 × 384 sample values. An L4 or L40S on-prem is the highest-ROI choice for 10–40 cameras.
Airborne small flying objects. Follow the MVA winners literally. SliceTrain at 1280², detection at full 4K, P2 head, and hard-negative mining of sky, cloud and foliage — the SOD4SB baseline gained +3.7 mAP from 20 extra epochs of it. Track with OC-SORT plus EMA velocity (α = 0.8) plus 2× box expansion plus a normalised distance penalty: +11.0 SO-HOTA measured. Never use appearance ReID, because there is no appearance. Never leave min_box_area set. On DeepStream, drop minDetectorConfidence to about 0.1 and raise maxShadowTrackingAge. Consider a different sensor — event cameras beat the best frame-based detector by about 70% relative IoU on tiny fast movers, and the GenX320 is a tens-of-dollars, 3 × 4 mm, 36 µW part. Evaluate with SO-HOTA and state accuracy, never plain MOTA.
Sports. Track the ball with a separate model, because a bounding box is the wrong output representation. Use a heatmap over three stacked frames at 288 × 512 (WASB: 1.5 M parameters, MIT), evaluate at τ = 4 px, and add trajectory rectification if occlusion is your residual error. If blur is significant, regress it. If you must stay in a box detector: single class, P2 head, imgsz ≥ 1280, mosaic=0 (a ball has no context worth shuffling), heavy motion-blur augmentation, max_det=10, agnostic_nms=True, and a constant-acceleration trajectory smoother rather than IoU gating. For players, Deep-EIoU or Hybrid-SORT-ReID online, with pitch calibration and BEV projection turning association into a nearly-linear problem. Frame rate is a design variable: a 28 m/s ball needs 200–400 Hz.
What the Field Has Not Measured
Worth knowing, because these are the places where you cannot look up an answer and will have to measure it yourself.
AP_small under INT8 quantisation. No public source reports per-size accuracy under post-training quantisation. Every edge deployment is making this trade blind.
Tiled-teacher to full-frame-student distillation. The idea is obvious and the hook now ships, but no measured deltas surfaced. This looks like the most promising unexplored lever available, because the teacher’s tiling cost is paid once at training time.
Domain pretraining versus COCO initialisation. No controlled A/B — a cheap experiment nobody has published.
A controlled super-resolution ablation. SR to 2× against plain bicubic to 2× at the same detector input size. Almost the entire SR-for-detection literature omits this control.
Rolling-shutter-induced localisation bias, and “remove P5” on aerial data. Both are first-order in practice and unablated in any primary source.
And two structural holes. The heatmap-stack recipe has never left sports, despite all three of its load-bearing ingredients being domain-agnostic. And no leaderboard tracks small-object progress — the tiny benchmarks are fragmented across MVA, Anti-UAV, SODA, AI-TOD, DynUAV and half a dozen others, each with its own metric, while the field’s most-watched scoreboard shows no size breakdown at all.
If You Build One Thing From This Series
Sweep four preprocessing parameters before you commission a training run: tile size and overlap, input resolution, and — for tracking — the search factor and the association similarity function.
Across every measured ablation collected across these five articles, those four account for ten to fifteen points where architecture accounts for two to five. They cost hours. A training run costs days.
The field’s hardest problems here are not architectural. They are geometric, statistical and procedural, and they are solved in the preprocessing function, the assignment metric, the association geometry and the training protocol — which is a less exciting answer than a new backbone, and a considerably cheaper one.