Small Objects in the Wild, Part 1: Why Six Pixels Breaks Everything
This is part one of a five-part series on detecting and tracking small objects. Part 2 covers what a stock YOLO checkpoint does wrong. Part 3 covers what actually moves the number. Part 4 covers tracking. Part 5 covers blur, sensors and deployment.
A team trains a detector on 4K aerial footage, gets 8% mAP, and concludes the task is too hard. The model was never the problem. Somewhere between the camera and the loss function, the twenty-pixel object they cared about became a two-pixel object, received no positive label, contributed no gradient, and was trained as background. The model did not fail to detect it. The model was never told it existed.
That failure has five independent causes, and they multiply rather than add. Understanding which one you are hitting is the difference between a two-point improvement and a ten-point one. None of the five is architectural.
The IoU Cliff
Take two same-sized square boxes, side s, offset by d pixels along one axis. Their intersection over union is:
IoU = (s − d) / (s + d)
That expression is nearly flat for large s and nearly vertical for small s. It is the single most consequential fact in this field, and it is not a modelling problem. It is the metric’s geometry.
Shift a 36-pixel box by two pixels and IoU is 0.89. Shift a 6-pixel box by the same two pixels and IoU is 0.50. One pixel of error costs a tiny object half its overlap.
A convention worth fixing before going further, because the literature mixes the two and the mixture is common: every displacement figure in this series is a single-axis translation. A diagonal shift of d in both axes is harsher, at (s − d)² / (2s² − (s − d)²). RGBT-Tiny’s much-quoted result — a 2 px error on an 8 × 8 px object drops IoU to 0.39 — is the diagonal figure. The single-axis value for the same box is 0.60. Both are correct. Reading one as the other is not.
The cliff matters three separate times over, because IoU appears in three places:
Label assignment. A tiny ground-truth box wins zero or one positive anchor where a medium box wins dozens. No amount of loss reweighting rescues a sample that was never labelled positive.
Non-maximum suppression. At the usual 0.7 threshold, two duplicate 8-pixel boxes two pixels apart have IoU 0.60 and both survive. Two genuinely distinct objects three pixels apart have IoU 0.45 and also both survive. The threshold produces duplicates instead of merging them, and fails to separate real neighbours, at the same time.
Data association in tracking. An 8-pixel target that moves 8 pixels between frames has IoU 0.00. It cannot be matched at all. Not matched badly — matched with zero available signal.
Stride Versus Object Size
A standard detection head predicts at P3, P4 and P5 — strides 8, 16 and 32. An 8 × 8 px object occupies exactly one cell at P3 and is sub-pixel at P5. Two of the three heads are structurally incapable of localising it, and their positives are noise.
At 640 × 640 the entire anchor budget is 80² + 40² + 20² = 8,400 positions. Adding a stride-4 P2 level adds 160² = 25,600 more, a 4.05× increase. That arithmetic explains both why “add a P2 head” is the most reliable architectural fix and why it is expensive.
The measured effect is consistent across benchmarks. On AI-TOD, extending FCOS’s FPN down to P2–P6 moves mAP from 12.0 to 15.4, with AP on very tiny objects going 2.5 → 6.0 (RFLA Table 5, R-50, 12 epochs). On VisDrone, adding a P2 head to YOLOv5 at 1536 px buys +2.15 mAP for +18% GFLOPs.
Note the size of those gains. Three points, two points. Hold that thought — it becomes the point of Part 3.
Resolution Destroyed Before the Network Runs
The default inference path resizes the long side to 640. Here is what that does:
| Input frame | Scale factor | A 20 px object becomes | Original px needed to fill one P3 cell |
|---|---|---|---|
| 1920 × 1080 | 0.333 | 6.7 px | 24 px |
| 3840 × 2160 | 0.167 | 3.3 px | 48 px |
| 5472 × 3648 (20 MP) | 0.117 | 2.3 px | 68 px |
| 3840 × 2160 at imgsz 1280 | 0.333 | 6.7 px | 24 px |
| 3840 × 2160 at imgsz 1536 | 0.400 | 8.0 px | 20 px |
An object must be 48 px in the original 4K frame merely to fill one P3 cell at the default input size. That threshold sits above the entire AI-TOD distribution, whose mean object size is 12.8 px.
Nothing downstream recovers information the resize deleted. This is the cheapest failure to fix and the most common one to leave in place.
Signal, Not Just Size
Small and faint are different problems that usually arrive together.
A target spread over L blur pixels loses roughly 1/L of its peak signal-to-noise ratio. A shorter exposure removes the blur and costs photons instead, so there is a genuine optimum rather than a free choice. At long range through atmosphere the target position itself jitters by an amplitude comparable to its own diameter. The clearest single measurement of what that costs: YOLO11m drops from 0.547 to 0.093 mAP on synthetic-turbulence video.
Where appearance carries no information at all — a 6-pixel grey speck against bright sky — re-identification by appearance is not merely weak. It is undefined. Every top entry in the 2025 bird-tracking challenge was explicitly appearance-free, and that was a design decision rather than a shortcut.
Annotation Noise Becomes the Ceiling
At a 10-pixel mean object size, a 2-pixel labelling disagreement is a 33-point IoU swing (8/12 = 0.67), and 53 points under the diagonal convention.
This makes AP@0.75 on tiny objects a measurement of your annotators, not your detector. It is visible directly in the SODA-D baseline table: AP on extremely small objects spans 5.1 to 14.7 across twelve detectors, while AP on generally small objects in the very same images spans 29.6 to 36.4. The detectors are not four times worse on the smaller band. The labels are.
The field’s response has been to abandon IoU rather than patch it. Dot Distance replaces it for assignment. A τ = 4 px centre error replaces it for sports balls. SO-HOTA replaces it for tiny-object tracking, and normalised Wasserstein distance replaces it in the loss. Those are covered in Part 3 and Part 4.
The Compounding
Follow one object through a stock pipeline. A 20 px target in a 4K aerial frame is letterboxed to 3.3 px. The default scale-jitter augmentation, drawing from U[0.5, 1.5], halves it again in half of all training samples, to 1.7 px. At that size it has no in-box anchor centre, so it receives no positive assignment, so it contributes no gradient, so it is trained as background.
Every step in that chain is a reasonable default for COCO-like data. Together they delete the task.
Sixteen Fields Where This Is the Whole Job
It is worth naming the domains explicitly, because the engineering answer changes with the requirement rather than with the object. A 6-pixel person and a 6-pixel drone are the same detection problem and completely different systems problems. One tolerates a hundred false alarms an hour because a human reviews every one. The other closes a runway.
Counter-UAS and airspace security works at 1–8 px at 1–3 km. The pixel arithmetic is instructive: a 0.35 m quadcopter at 1 km subtends 350 µrad, which on a 300 mm lens is a 105 µm image. So pixel count is set entirely by pixel pitch — 6.2 px at a 17 µm LWIR pitch, 8.8 px at 12 µm, but 30 px at a 3.45 µm visible pitch. The few-pixel regime is a thermal regime. A good visible sensor on the same lens is not resolution-limited at 1 km; it is contrast- and atmosphere-limited. The cost anchor for getting it wrong is Gatwick, December 2018: roughly 1,000 flights cancelled or diverted, about 140,000 passengers affected, £5 m subsequently spent on a counter-drone system, and no drone ever confirmed. Both a false positive and a miss are expensive.
Detect-and-avoid for uncrewed aircraft is the purest few-pixel, high-consequence problem here. A Cessna-class aircraft at 2 NM in a ~50° field of view is 5–15 px, and first detection is typically a 1–3 px smudge. The public proxy dataset, Amazon Prime Air’s Airborne Object Tracking set, encodes the requirement in every design choice: 4,943 flight sequences of about 120 s at 10 Hz, 2448 × 2048 8-bit grayscale for photon efficiency, over 5.9 M images, 3.3 M annotations, plus 3.3 M frames containing no airborne object at all. The empty frames exist to set the false-alarm operating point. The 10 Hz capture rate exists so the system can accumulate about three seconds of track before committing.
Maritime dark-vessel detection has the hardest physical floor. At Sentinel-1’s roughly 20 m resolution, a 20 m vessel is one pixel. The definitive public quantification covers more than 67 M Sentinel-1 tiles and 53 bn AIS positions over 2017–2021: detection runs above 90% for vessels 50 m and longer, about 80% for 25–50 m, and declines steeply below 25 m. That decline is the small-object frontier for the field, and the headline finding — that 72–76% of industrial fishing vessels are not publicly tracked — is entirely a function of it.
Wind-farm curtailment is the one field where the precision/recall trade has a published price per event, which makes it the best available template for setting a threshold from a business requirement rather than from a PR curve. Every false alarm is metered lost generation; every miss is a permit problem. IdentiFlight publishes the clearest requirement set anywhere in this space: over 100 species, 98% protection accuracy, curtailment of targeted turbines to 2 rpm or below over the operator’s SCADA network, more than 520 stations on six continents. Its efficacy figures are genuinely contested and should not be presented as settled — McClure et al. 2021 reported an 82% (75–89%) reduction in eagle fatalities, Huso and Dalthorp 2023 reanalysed it, identified four errors, and obtained “an unremarkable estimate of 50% (−159%, 89%)”, a confidence interval that includes increased mortality, and McClure et al. published a rebuttal. The engineering lesson survives the dispute. The magnitude of the benefit does not.
Sports officiating is the only field where sub-second latency, sub-centimetre accuracy and essentially zero false positives coexist. It is met by engineering the sensing, not by a cleverer network. Hawk-Eye uses up to ten high-performance cameras, typically under the stadium roof, and states 2.6 mm accuracy with 3.6 mm average error. FoxTenn states 2,500 images per second per camera across more than 40 synchronised cameras, so over 100,000 images per second in total. When someone asks whether a model can hit that spec, the answer is that no model can and the rig can.
Autonomous driving inverts the usual posture. Everywhere else in this list, a false positive costs an operator’s attention. Here, phantom braking at highway speed is itself a hazard, so a false positive can kill. Combine that with hard latency, a tight power budget and a safety argument, and it is the hardest combination in the survey. The class list is also unbounded — road debris, lost cargo, bricks, tyre fragments — which is why the field uses open-set and anomaly segmentation rather than closed-set detection. LiDAR does not save you: a 20 cm object at 80 m may return one to three points, which is why vision carries small-obstacle detection at range.
Wildfire detection shows how revisit interval defines a detection problem. ALERTCalifornia operates more than 1,200 HD PTZ cameras as of February 2026, sweeping 360° roughly every two minutes, with near-infrared at night, seeing up to 60 miles by day and 120 by night. Catch a faint smudge within a couple of sweeps or lose the early-detection window. Operators accept meaningful false alarms for minutes of earlier detection, but not so many that dispatchers stop believing the alerts, which is why every such system routes to a human console.
The rest of the list runs through aerial infrastructure inspection (a 0.2 mm defect at survey range), search and rescue (a person in water is the hardest case here — head and shoulders only, low contrast, moving with the waves), defence wide-area motion imagery, agriculture, traffic and ITS, retail shelf audit, industrial inspection, medical imaging, space domain awareness, anti-poaching, construction safety, crowd analytics and drone light shows.
The Rule That Cuts Across All of Them
One question predicts most of the architecture:
False alarms are tolerable exactly where a cheap verification loop exists and where acting on a false alarm is harmless.
Search and rescue tolerates very high false-positive rates because a human reviews every hit and the cost of a miss is irreversible. Aerial inspection tolerates them because defects are triaged. Industrial QA tolerates them because there is a re-verify station. Counter-UAS does not, because the alarm is a runway closure. Wide-area ISR does not, because at 100 km² and 1 Hz even 10⁻⁶ false positives per pixel floods the operator. Traffic enforcement does not, because of due process. Everything else about the system follows from that one question, and it is worth answering before choosing a model.
Three Recipes, and What Does Not Transfer
Across all sixteen fields, three recipes cover almost everything. They transfer freely within their group and essentially never across it.
Tiled high-resolution appearance detection. Slice a 4K–16K frame, run a modern detector per tile, merge. This moves cleanly between aerial inspection, drone-based traffic, retail shelves, PCB inspection and maritime SAR imagery. It requires the object to be at least about 10 px and texturally distinct.
Motion-based track-before-detect on a registered background. Register frames, difference them or decompose into low-rank background plus sparse target, then associate over many frames before declaring anything. This moves between wide-area motion imagery, satellite video, infrared search-and-track, long-range counter-UAS, first-detection detect-and-avoid, and astronomical streak detection. It works at 1–3 px, below the appearance floor, which is why it dominates wherever the target is dimmer or smaller than a classifier can handle.
Anomaly and open-set detection against a learned normal. One-class or student-teacher models with pixel-precise output. This moves between industrial inspection, road-obstacle segmentation and incidental medical findings — anywhere the defect taxonomy is unbounded.
What does not transfer is worth stating as plainly. Anything demanding sub-second and sub-centimetre accuracy is solved by ten to fourteen synchronised calibrated high-speed cameras, not by a better network. Spectral-signature problems, like methane plume detection, are not image detection at all. And at extreme density the box abstraction itself fails, which is why crowd analytics abandoned detection for density regression — a useful warning about the limits of the representation rather than of the model.
Where This Leaves You
The five failures are geometric, statistical and procedural. The IoU cliff is the metric’s shape. Stride mismatch is arithmetic. Resolution loss happens in the preprocessing function. Missing signal is physics. Annotation noise is the label set.
Not one of them is fixed by a newer backbone, which is why the rest of this series spends most of its time on resolution, label assignment, association geometry and training protocol. Those four knobs account for ten to fifteen points across every ablation I could find, where architecture accounts for two to five.
Part 2 starts with the pipeline most teams actually run — a stock checkpoint with default settings — and works through the eleven defaults that make it close to a worst case for this problem, most of them silently.
Figures in this series are drawn from primary sources: papers, repositories, vendor benchmark pages and challenge leaderboards. Where a number is arithmetic performed on published constants rather than a published result, it is described that way. Where a widely repeated figure could not be confirmed against a primary source, it is flagged as unconfirmed rather than quoted as fact.