Small Objects in the Wild, Part 3: What Actually Moves the Number
Part three of five. Part 1 covers the five geometric failures. Part 2 covers stock-checkpoint defaults. Part 4 covers tracking. Part 5 covers blur, sensors and deployment.
The detector lineage matters less than the layer you change. Backbone and neck work has been worth two to five points for a decade. Assignment, resolution and protocol work is worth ten to fifteen.
That claim needs evidence rather than assertion, so here is the arithmetic across four verified ablation tables. No paper states the split explicitly, and the derivation is mine, but the inputs are all published.
AI-TOD, R-50, 12 epochs. Upgrading the architecture from Faster R-CNN to DetectoRS — recursive FPN, switchable atrous convolution, a far heavier model — moves mAP 11.1 → 14.8, so +3.7. Changing the label-assignment metric on the same unchanged Faster R-CNN moves it 11.1 → 21.1, so +10.0. Assignment is 2.7× the architecture gain, at zero inference cost. And a light one-stage model with the right pyramid and assignment (FCOS with P2 plus RFLA, at 16.3) beats the heavy architecture with stock assignment (DetectoRS, at 14.8).
VisDrone AP50. Architecture, FCOS → TOOD: 25.8 → 29.4, so +3.6. Data and inference protocol on the same FCOS: 25.8 → 38.5, so +12.7. Protocol is 3.5× architecture. Applied to TOOD it adds +14.1 — the two compose rather than substituting.
VisDrone mAP, YOLOv5 at 1536. Architecture terms sum to +4.75: P2 head +2.15, transformer blocks +1.81, CBAM +0.79. Inference and training terms sum to +2.11: multi-scale testing +1.27, self-trained classifier +0.84. This is the one case where architecture wins, and the reason is instructive — resolution had already been fixed at 1536.
SMOT4SB SO-HOTA. Architecture change: none. Association-geometry tuning: +11.0.
So the rule has a boundary condition. Until you have native-scale pixels and a scale-appropriate assignment metric, architecture work is worth two to five points and everything else is worth ten to fifteen. Once those are fixed, architecture becomes the remaining lever. The order of operations dominates the choice of model.
Assignment and Loss: Where the Points Are
If you change one thing, change this.
The reason goes back to the IoU cliff from Part 1. Under a fixed IoU threshold, a tiny ground-truth box receives zero or one positive anchor while a medium box receives dozens. No amount of loss reweighting fixes a sample that was never labelled positive.
NWD — Normalised Wasserstein Distance
IoU is exactly zero for non-overlapping boxes, so it provides no gradient and no ranking signal. For tiny objects, predicted and ground-truth boxes frequently do not overlap at all. Modelling each box as a 2D Gaussian and measuring Wasserstein distance gives a metric that is smooth, non-zero everywhere, and scale-normalisable.
Convert box (cx, cy, w, h) to a Gaussian with mean at the centre and covariance from (w/2)² and (h/2)². Compute the 2-Wasserstein distance between the two Gaussians in closed form. Map it to a similarity with exp(−√W₂²/C), where C is a dataset-dependent scale constant. Use it in place of IoU in the assigner, the loss, or both.
It is worth +6.7 AP over the fine-tuning baseline on AI-TOD, and +6.0 over the then state of the art, at zero inference cost. The reference implementation is Apache-2.0 in an MMDetection fork.
The catch is that C encodes the typical object scale and must be tuned per dataset — get it wrong and you lose most of the benefit. Applied globally it actively hurts medium and large objects, because for well-resolved boxes IoU is the better metric. The RFLA authors say so plainly: their repo “is unsuited for generic object detection … since this problem is not obvious in medium and large objects”. Use a size-gated blend if your data has both regimes.
NWD-RKA
Even with a better distance metric, a fixed threshold still assigns an unstable number of positives per object. Ranking-based assignment fixes the count instead of the threshold: each ground truth gets its top-k candidates by NWD, so every tiny object is supervised.
Worth +4.3 AP over the prior state of the art on AI-TOD-v2, and it removes the threshold hyperparameter entirely. It shipped alongside a re-annotated AI-TOD-v2 (752,754 instances) that fixes label noise in v1 — a correction that matters independently, because v1’s label noise was itself suppressing measured progress. Compare v1 and v2 numbers carefully; they are different label sets and papers do not always say which they used.
RFLA — Gaussian Receptive-Field Label Assignment
This one asks the correct question. The relevant comparison is not “does this anchor box overlap the object?” but “does this feature point’s effective receptive field match the object?” — because that is what determines whether the feature can represent the object at all. For a 6-pixel object almost no receptive field matches well, and RFLA turns that mismatch into a smooth, rankable quantity instead of a hard fail.
Model each feature point’s effective receptive field as a Gaussian, define a Receptive Field Distance between it and the ground-truth Gaussian, replace IoU with RFD in assignment, and add a cascaded Hierarchical Label Assignment scheme to remove the bias toward large objects.
It is the single largest verified assignment gain in the literature: Faster R-CNN 11.1 → 21.1 and DetectoRS 14.8 → 24.8 on AI-TOD at 12 epochs, with AP on very tiny objects going 0.0 → 9.5. From literally never detecting them to detecting them, at zero inference cost.
Read that 0.0 baseline carefully, because it is the most useful number in this section. Faster R-CNN with a full P2–P6 RPN still scored exactly zero on very tiny objects before its assignment was fixed. Resolution without assignment buys nothing at the smallest scale.
DotD — Dot Distance
For an object a few pixels across, the only meaningful question is whether the centre is in roughly the right place. Shape agreement is unmeasurable and mostly annotation noise, so normalised centre distance discards the unmeasurable part: Euclidean centre distance normalised by a dataset-level scale, typically mapped to a similarity by exp(−D/S).
It is the simplest of the family and the most widely adopted downstream, used as an assignment metric, as an association metric in tracking, and — through SO-HOTA — as an evaluation metric. The MVA 2025 organisers substituted it into HOTA precisely because “IoU-based metrics struggle with sensitivity to small spatial displacements”, illustrating it with a 16 × 16 px box for which “a minor displacement of just 8 pixels in the x-direction from perfect alignment causes IoU to drop dramatically from 1.00 to 0.5” — and to zero once the centre displacement exceeds half the box width.
It ignores scale entirely, so it cannot distinguish a correct box from a correct centre with a wrong size. That matters if downstream logic uses box area for range estimation or size gating. And if you adopt distance-based assignment, adopt distance-based evaluation too, or your metric will disagree with your training objective and you will tune the wrong knob.
SAFit, DCFL and STAL
SAFit resolves the tension that makes NWD dangerous on mixed-scale data: a sigmoid blend between the IoU term and the NWD term, with the transition centred at an object size of 32 px — the COCO small/medium boundary. It is defined in the RGBT-Tiny paper (arXiv 2406.14482) first as a metric, “Scale Adaptive Fitness”, then as a loss, L_SAFit = 1 − SAFit, which the paper reports “can provide stable and accurate optimization guidance on targets with varied sizes”. RGBT-Tiny’s own statistics justify the design: about 48% of its annotations are 8²–16² px and over 97% are small or smaller. Read the exact weighting function off the paper rather than reimplementing it from a description.
DCFL handles oriented tiny objects, where the cliff is worse because a small angular error on a 10 px elongated object destroys overlap. Dynamic priors plus coarse-to-fine positive-sample selection let the assigner commit gradually rather than demanding a good box before granting supervision. On AI-TOD-R — mean object size 10.6 ± 4.9 px, the smallest of any oriented-detection dataset — it reaches 17.1 AP against an 11.2 baseline. It is reported with an extended training schedule, so part of the gain is schedule length, and the ablation separating the two was not confirmed during compilation.
STAL is the same diagnosis, productised. YOLO26 ships small-target-aware label assignment as a default, described as enhancing positive-label coverage for small objects, paired with Progressive Loss Balancing in a head that also drops DFL and runs NMS-free. Its presence in a mainstream release is the clearest signal that the assignment diagnosis is now consensus rather than a research position. The implementation details are not published as a paper, so it cannot be ablated against NWD or RFLA on a common benchmark.
Resolution and Tiling: The Largest Single Win
Nothing else here moves AP as much. Slicing-aided fine-tuning plus slicing-aided inference is worth +12.7 to +14.5 AP50 on VisDrone, and it takes xView from 2.1 to 20.4 AP50 — roughly a tenfold improvement. It is also the most expensive thing you can do, and the arithmetic is unforgiving; Part 5 does that arithmetic.
SAHI
SAHI attacks the root cause directly. A tiny object is undetectable because the pipeline destroys its pixels before inference. Slicing keeps every object at native scale and presents each tile to the detector at the resolution it was trained for. Nothing else in the stack has to change, and it is MIT-licensed and detector-agnostic.
There are two independent components, and this is the part teams skip. SF (slicing-aided fine-tuning) extracts overlapping patches from the training set and resizes them to the detector’s native input, so training and test scale statistics match. SAHI (inference) tiles the frame with overlap, detects per tile, optionally adds one downscaled full-image pass to catch large objects, then merges with NMS. The reference implementation defaults to 512 px slices at 0.2 overlap.
Roughly half the gain comes from the training side. Inference-only slicing is worth about +5 to +7 AP50 on VisDrone; adding sliced fine-tuning takes it to +13 to +15. A model that has never seen a tile at training time is being asked to generalise across a scale shift at test time. Do the fine-tuning.
Three caveats worth internalising. Time grows roughly linearly with tile count — a 4K frame at 640 px tiles with 20% overlap is 32 tiles, and 20 MP is 77. Objects larger than a tile get fragmented. And the reported gains are AP50, not AP50:95, because tiling helps least at the high IoU thresholds where annotation noise dominates. Memory stays fixed regardless of frame size, though; only time grows. It also works with models you cannot retrain.
ASAHI
Fixing the slice size means the tile count explodes with resolution, and much of that compute lands on empty sky. ASAHI fixes the slice count instead — 6 or 12 slices chosen from the input resolution — and merges with Cluster-DIoU-NMS rather than plain NMS.
It reaches mAP50 56.4 on VisDrone2019-DET-val while cutting computation 20–25% against state-of-the-art slicing, with a measured throughput gain of +0.48 img/s on the test split averaged over five runs. Better on both axes at once, which is rare. The DIoU-based merge is doing real work: keep plain NMS at the merge step and you will not see the full gain. The cost is that variable slice dimensions reintroduce dynamic shapes for export-constrained runtimes.
SliceTrain
The MVA 2025 winner names the problem it solves the resolution–diversity dilemma: at full 4K you can fit one image in a batch, which destroys batch statistics, and at reduced resolution the targets vanish.
Partition each 2160 × 3840 frame into overlapping 1280 × 1280 tiles at 20% overlap covering the whole frame, apply independent random augmentation per tile, and train on tiles. Then — importantly — infer on the full frame without slicing, relying on the detector’s learned scale robustness. That raised the effective batch from 1 to 6 on a single RTX 3090 while preserving information density, and it means inference stays single-pass, which is a very different deployment story from SAHI.
Full 4K inference speeds are modest: 17.6 FPS for YOLOv8-S, 8.96 for -M, 5.70 for -L on one RTX 3090. And label handling at tile boundaries is where teams go wrong — clip boxes to the tile and drop boxes whose retained area falls below a threshold, or you train on box fragments and teach the model to emit them.
Raise imgsz, and Stop Letterboxing Pixels Away
Pure arithmetic from Part 1: the only way a 20 px object in a 4K frame reaches 8 px at the input is a scale factor of at least 0.4, so imgsz ≥ 1536. TPH-YOLOv5 trains VisDrone at 1536 with batch 2 on an RTX 3090; Ultralytics’ own VisDrone page recommends 1280 plus SAHI.
Necessary, not sufficient. Raising resolution without adding a P2 head still leaves six-pixel-in-frame objects sub-cell, and it does not touch assignment.
Super-Resolution: Unproven
The hope is that a learned upsampler restores high-frequency detail a bicubic resize cannot. Mechanistically this is suspect: the information is not in the sensor data, so anything restored is a prior, and a prior that invents plausible texture on a four-pixel blob is manufacturing evidence.
One controlled study does report real gains — 4× SR of Sentinel-2 toward PlanetScope resolution with YOLOv8 plus SAHI moved precision from 0.51 to 0.72 and F1 from 0.53 to 0.73. But that gain is confounded with a density-filter fusion step and includes no ablation isolating SR, and the same authors state objects “get hallucinated” at SR factors of 8 or higher, with extra-small objects undetected regardless.
Almost none of this literature runs the controlling comparison: SR to 2× versus plain bicubic to 2× at the same detector input size. Until someone does, prefer high-resolution training plus tiling, which has hard numbers. If you must use SR, train it jointly with the detector so the objective is detection rather than PSNR — task-driven SR consistently beats sequential SR, the same pattern that shows up with deblurring in Part 5.
Small-Object Copy-Paste
Small objects are rare in pixels even when common in count. Copy-pasting small instances raises their pixel share without new data collection, and it is worth +9.7% relative instance-segmentation AP and +7.1% relative detection AP on small COCO objects. It was part of the MVA 2023 winning recipe, and it is cheap and orthogonal to everything else. Pasted objects lack correct shadows, blur and atmospheric attenuation, so they create a domain gap that partly cancels the gain, and the method explicitly trades large-object AP.
Temporal Detection: The One Signal You Have Plenty Of
A 6-pixel object has almost no spatial information and, at 30 fps, a great deal of temporal information.
The founding measurement is TrackNet’s: going from a single frame to three stacked frames took F1 from 92.5 to 98.2 on tennis balls. That is the cheapest large gain in the small-object literature, and it has been rediscovered repeatedly since.
Concatenating N consecutive frames on the channel axis lets the very first convolution compute temporal differences implicitly. A stationary 6-pixel grey blob and a moving one are indistinguishable in one frame and trivially distinguishable in three — and in this domain, “moving” is close to equivalent to “is the target”. Three frames is the repeatedly-rediscovered optimum. Only the first layer changes, so there is no optical flow, no recurrence, no extra pass.
More frames is not better. TrackNetV3’s 8-frame input plus 16-frame rectification costs an order of magnitude in throughput (15–25 FPS against 155–163) for a few points of accuracy. And the method requires registered frames: on a moving platform the background moves too, and the difference is dominated by ego-motion.
What the MVA 2025 Challenge Decomposes Into
This is the best-documented competition in exactly this problem — birds from a moving UAV, most instances below 32 × 32 px, both camera and target moving, 211 sequences and 108,192 frames — and the results decompose cleanly.
The winner (YOLOv8-SMOT) used SliceTrain tiled training at 1280², full-frame inference at 2160 × 3840, then OC-SORT with three appearance-free association changes. Runners-up: Cascade R-CNN + Swin into Hybrid-SORT (46.22); Co-DETR + SAHI + synthetic copy-paste into BoostTrack++ (43.87); a YOLOv7 and YOLOv12 ensemble with adaptive weighted box fusion into OC-SORT with DIoU (43.71); a CenterNet ensemble with motion-compensated OC-SORT (40.49).
Detection improvement dominated: SO-DetA went 8.67 → 47.27 while SO-AssA went 11.32 → 54.30. Every top entry used tiling. Three of the top five used ensembling with weighted box fusion. None used appearance re-identification.
One exposed weakness is worth knowing before you copy the leaderboard. Two of the top five scored negative MOTA (−8.12 and −8.11) — the high-recall, high-false-positive regime that maximises SO-HOTA is punished savagely by MOTA. If your product is false-positive-sensitive, you are optimising a different objective from this leaderboard.
HiEUM
A different exploitation of the same signal, and the cleanest existence proof that “detect the motion, classify later” beats appearance-first below about 20 px. A tiny moving target in a registered stack is a sparse structure in 3D space-time — a thin continuous curve. Convert the stack to a sparse point cloud, run sparse convolution only on the occupied voxels, and multi-frame integration becomes cheap instead of expensive.
It reaches F1 89.7% against DSFNet’s 74.4% on satellite video, running at 98.8 FPS at 1024 × 1024. That is +15.3 F1 points over the supervised state of the art while running 28.7× faster, and with no labels at all. It needs a registerable, largely static background — satellite video and fixed cameras qualify, an agile drone does not — and sparse convolution is a deployment liability outside CUDA.
One family to be sceptical of: the feature-aggregation video detectors (FGFA, SELSA, MEGA, TransVOD). In principle, aggregating features across nearby frames is exactly the small-object remedy. In practice they were designed and validated on ImageNet-VID, where objects are large, no verified evaluation of the family on tiny-object benchmarks surfaced, and the flow-warping step is itself unreliable on 6-pixel targets. The cheap version of the same idea — channel stacking — has actual evidence.
Where the Detector Families Actually Sit
Two structural changes matter in the CNN lineage, and neither is an architecture with a marketing name. Multi-scale prediction with a top-down semantic path (FPN, 2017) is why small-object AP roughly doubled that year: before it, small objects were predicted either from shallow features with resolution and no semantics, or deep features with semantics and no resolution. And anchor-free dense prediction (FCOS, CenterNet, 2019) removed the ill-conditioned k-means-on-tiny-boxes problem described in Part 2. Everything since is refinement.
Note that FPN still starts at P3 by default, so it solved the semantics problem and not the stride problem. Extending the pyramid down to P2 is a separate decision from choosing a neck.
A few members worth knowing:
QueryDet is the most defensible way to afford a P2 level under a latency budget. High-resolution feature maps are what small objects need and what makes inference expensive, but tiny objects are sparse — so predict coarsely where they might be, and only spend high-resolution computation there. It is the Viola-Jones cascade, learned, on feature pyramids: 3.0× measured speedup on COCO, 2.3× on VisDrone, with +1.0 mAP and +2.0 AP_small. Two cautions: sparse convolution support and performance vary sharply across TensorRT, ONNX Runtime and NPU toolchains, and recall is bounded by the coarse stage — anything invisible at low resolution is never queried. Gate the query stage on a high-resolution feature response rather than on final low-resolution detections, or you re-import the blindness you were trying to avoid.
CFINet holds the best-verified SODA-D result in the two-stage family: 30.7 AP / 60.8 AP50 / 14.7 on extremely small objects, against RFLA’s 29.7/60.2/13.2 and Faster R-CNN’s 28.9/59.4/13.8. Its feature-imitation idea — regressing tiny-object RoI features toward high-quality features of the same category — is transferable independently, but it needs larger instances of the same class to exist in the data, which is false in some domains. A distant drone has no near-field counterpart in the same dataset.
Crop-then-detect exploits the fact that aerial objects are clustered rather than uniformly distributed. The measured VisDrone AP progression is ClusDet 26.7 → DMNet 28.5 → YOLC 30.3 → YOLC+MoonNet 32.9, with adaptive cost, so sparse scenes are cheap. The deployment problem is real: a variable tile count per frame breaks fixed-shape export and makes latency non-deterministic. Also, the density stage is itself a small-object detector, so it inherits every failure in this series, and isolated single targets against sky — the counter-UAS case — are its worst case.
The YOLO line as of August 2026 (v12, v13, YOLO26) leads on accuracy-per-millisecond at 640 × 640 with clean export, which is what most products need. YOLOv12 covers COCO 40.4 → 55.4 across n→x at 1.60–10.38 ms on a T4 with TensorRT 10; YOLOv13 spans 41.6 to 54.8 at 1.97–14.67 ms; YOLO26 spans 40.9 → 57.5 at 1.7–11.8 ms and is the first release in the family with an explicitly small-object-motivated change. All three are AGPL-3.0, none has a P2 level by default, and all are pretrained on COCO. One measurement hygiene note: the published mAP columns in the vendor’s Jetson tables are internally inconsistent — one row reports FP16 mAP 0.045 alongside INT8 0.464 — so use their latency columns only.
The DETR Lineage and Its Structural Disadvantage
The set-prediction family arrived with a real handicap on small objects and has spent four years engineering it away. The handicap is one-to-one matching: each ground-truth object receives exactly one positive query, so a tiny object contributes one gradient path per image where a dense head’s top-k assignment gives it ten.
The evidence is blunt. On SODA-D — a benchmark built entirely from objects of 2000 px² or less — Deformable-DETR scores 19.2 AP, the lowest of twelve published baselines, against Faster R-CNN’s 28.9, while training four times as long. On extremely small objects it manages AP_eS 6.3 against Faster R-CNN’s 13.9 and Cascade R-CNN’s 14.1.
Be precise about how far that pattern goes, though. Sparse R-CNN, also one-to-one, scores AP_eS 8.8 — fifth-lowest of twelve, above CenterNet (5.1), Deformable-DETR (6.3), CornerNet (6.5) and FCOS (6.9). So one-to-one matching is not the only thing that hurts, and anchor-free heatmap detectors trained with stock assignment do worse still. What the SODA-D column does establish is that the two set-prediction detectors sit at or near the bottom, and that every subsequent advance in the family has been an attempt to restore dense supervision at training time without paying for it at inference.
| Model | Mechanism | Small-object result | Cost |
|---|---|---|---|
| Deformable DETR | Sampled offsets per query per level make multi-scale attention affordable | SODA-D 19.2 AP — fixes resolution, leaves supervision untouched | MSDeformAttn is not a standard ONNX op |
| RT-DETR / v2 | Attention on P5 only, CNN cross-scale fusion, IoU-aware query selection | R18 46.5 → X 54.8 AP, 217 → 74 FPS T4 FP16 | No P2; 300 queries is a hard recall ceiling |
| RT-DETRv3 | Auxiliary dense supervision at training, discarded at inference | R18 46.5 → 48.1 at identical latency | Training memory and complexity |
| D-FINE | Box regression as iterative refinement of an edge-position distribution | AP_S 36.5 (L); N 42.8 → X 55.8 AP at 2.12–12.89 ms | Extra decoder passes; still no P2 |
| DEIM | Dense O2O — mosaic and mixup used to multiply targets per image — plus a matchability-aware loss | +0.4 to +1.4 AP_small over D-FINE at identical inference cost | Changes object scale statistics |
| DEIMv2 | DINOv3 ViT with a spatial tuning adapter (S–X); pruned HGNetv2 (Atto–Nano) | AP_S 39.2 at 57.8 AP — best verified in real-time detection | Large SSL checkpoint; ViT quantises poorly |
| RF-DETR | NAS over a DINOv2-backboned real-time DETR, with export as a search constraint | N 48.4 @ 2.3 ms → 2XL 60.1 @ 17.2 ms | Headline latency is at 384²; AP_small unpublished |
| Cross-DINO | Category-size soft labels so the loss knows an instance is tiny | COCO AP_S 36.4 vs DINO’s 32.0; SODA-D 32.0, beating CFINet’s 30.7 | 45 M params, 277 GFLOPs, no real-time claim |
Three readings of that table matter more than the ordering.
RT-DETRv3 is the shape of the answer. +1.6 AP for zero inference cost on R18, purely from training-time dense supervision. The R101 variant gains only +0.3, and the shrinking gain as the backbone grows is itself the point: dense supervision matters most where capacity is scarce. Training-only changes are the highest-value category in this whole area because they cost nothing at deployment.
DEIM is the clearest proof that a recipe can beat an architecture. It adds +0.7 to +0.9 AP and +0.4 to +1.4 AP_small over D-FINE at identical inference cost, plus roughly 50% less training time, by changing what the training data looks like and how matches are weighted. Apache-2.0, and free at inference. Adopt it before considering a bigger model. One caution: if your target distribution is already at the resolution floor, aggressive mosaic and mixup can push instances below detectability. Tune the scale range rather than inheriting it.
Read the x-axis of any latency plot with suspicion. RF-DETR-N’s 2.3 ms is measured at 384 × 384, a resolution at which small objects have largely ceased to exist, while the D-FINE and DEIM points are at 640 × 640. For small-object work, compare at 704 × 704 (RF-DETR-L, 6.8 ms) instead. Only the D-FINE / DEIM / DEIMv2 line publishes AP_small alongside AP, and that is the column that matters here. RF-DETR’s XL and 2XL also ship under PML 1.0 via a separate package, which is not an OSI-permissive licence — read it before planning around 60.1 AP.
For context on the ceiling: the COCO detection leaderboard has plateaued at roughly 66 AP. The official leaderboard does publish an AP^S column and cocoeval.py computes it as a standard output, but the aggregator leaderboards most people actually read rank on overall AP alone and display no size breakdown. Progress on small objects is therefore invisible where the field’s attention is, even though the number exists.
Foundation Models: One Negative Result and One Positive One
The negative result is worth stating bluntly, because the expectation runs the other way. Open-vocabulary detectors do not work on tiny objects.
A January 2026 zero-shot evaluation on LAE-80C (3,592 aerial images, 86,558 instances, 80 classes aggregated from DOTA-v2, DIOR, FAIR1M and xView) put the best model — OWLv2, with a 428 M-parameter ViT-L/14 — at 27.6% F1 with a 69% false-positive rate. Grounding DINO scored 0.5–7.4%. YOLO-World and YOLOE scored 2.8–3.9%. Performance collapsed specifically on xView (tiny objects under 20 px) and FAIR1M (dense and fine-grained).
Two compounding reasons. Vision-language alignment is learned on web images where objects are large and centred, so the text-image correspondence for a 10-pixel object is essentially untrained. And vocabulary size dominates: cutting the class list from 80 classes to 3.2 classes yielded a 15× F1 improvement, meaning the models are being defeated by class confusion as well as by size.
Keep the evidence in proportion. That study is a single-author preprint reporting F1 rather than AP and including no supervised baseline, so it establishes that open-vocabulary detection is weak in absolute terms on aerial data rather than a precise gap against supervised models. A 69% false-positive rate is not an operating point, though. It is a warning. These models remain genuinely useful for auto-labelling and for bootstrapping a dataset when your classes are large enough to be found — restrict the vocabulary aggressively if you use them at all.
The positive result is one level down: foundation models pay as backbones. DINOv3 is what gets DEIMv2 to 39.2 AP_small; DINOv2 is what gets RF-DETR to 60.1 AP. Self-supervised training at scale produces features whose local structure survives aggressive downsampling better than supervised-classification features, which are optimised to discard spatial detail. On small objects the binding constraint is feature quality at low resolution, and this is a direct improvement to it.
The compute comparison is stark. DEIMv2-Pico delivers 38.5 AP from 1.5 M parameters and 5.2 GFLOPs at 2.13 ms, while the open-vocabulary models above spend 26–435 M parameters to reach single-digit F1 on aerial data. Two limits on the backbone story: ViT backbones quantise less gracefully to INT8 than CNNs, and below roughly 4 M parameters the answer is still a pruned CNN — credit DEIMv2’s genuinely edge-deployable tiers to the pruned HGNetv2 line rather than to DINOv3.
On the SAM family: mask quality for a 6-pixel target degrades before box quality does, so a mask-mode readout is the wrong output representation at the smallest scales. Using SAM to refine a detector’s boxes is common practice, but no rigorous tiny-object mask-quality measurement surfaced during compilation. Verify on your own data before trusting refined masks at small scale.
The Build Layer
Two things that decide whether any of the above is measurable.
Annotation is the AP ceiling. At 10 px mean object size, a 2-pixel disagreement is a 33-point IoU swing. The AI-TOD-R authors name “label acquisition difficulty due to limited appearance information” as one of three primary obstacles in the domain, which is dataset authors treating label noise as a first-class problem rather than an afterthought.
So prefer point annotation plus a size prior for sub-20-pixel targets. SMOT4SB’s SO-HOTA and WASB’s τ = 4 px F1 both effectively evaluate against a point, so point labels lose very little information. Use FiftyOne (Apache-2.0) for the review loop — it gives per-size AP breakdown and a mistakenness feature that surfaces likely annotation errors directly. The practical bottleneck is a zoomable magnifier rather than any tool’s feature list, because annotating 10-pixel targets at 1:1 on a 4K image is the actual work. And expect poor results from foundation models for auto-labelling here, for the reasons above; the realistic path is tiled auto-labelling with a domain-fine-tuned detector plus human verification.
Synthetic data gets you most of the way, not all of it. MOTSynth is the reference existence proof: 768 sequences, over 1.3 M densely annotated frames, roughly 40 M pedestrian instances, 9,519 identities. With a detector trained on MOTSynth alone and no fine-tuning, Tracktor reaches 45.0% MOTA on MOT17-train (45.5 with ReID), against 43.5 for a COCO-trained detector — and 49.8–50.3 once fine-tuned on real data. That gap between 45.0 and 49.8 is the honest measure.
Sim-to-real is harder for tiny objects than for normal ones, and the reason is specific: the domain gap is concentrated in exactly the signal you depend on. Sensor noise, motion blur, compression artefacts, atmospheric scatter and rolling shutter are the entire content of a 10-pixel target. Randomising scene layout is beside the point. Randomise sensor parameters — exposure, noise, blur kernel, compression, point-spread function.
What to Do First
The ordering, restated as a sequence rather than a list:
- Slice for training and inference, at 20–25% overlap. Both halves.
- Raise input resolution to 1280–1536 with rectangular inference.
- Replace the assignment metric — NWD, RFLA or DotD — and lower the alignment exponent.
- Add a P2 head.
- Only then consider a different architecture.
Steps 1 and 2 cost compute. Step 3 costs a code fork and nothing at inference. Step 4 costs 18% GFLOPs. Step 5 is where most projects start, and it is worth two to five points.
Part 4 takes the same treatment to tracking, where the equivalent finding is stranger: on the hard benchmark, a zero-shot segmentation model with a Kalman filter bolted on beats a 1.2-billion-parameter supervised tracker, and the largest measured gain on tiny thermal targets came from changing one preprocessing hyperparameter.