Skip to main content

Geolocation from video evidence

Design proposal, October 7, 2026. The localization engine, telemetry importer and location-claim schema below are not implemented. This refines the video evidence roadmap. Accuracy requires qualification on the intended hardware and footage.

A GoPro GPS sample describes the recorder's position. The physical traffic camera visible in a frame is a separate object. Waldo should retain the image evidence even when it cannot locate that object. Detector confidence, track continuity and location confidence answer different questions.

What each case can establish​

Available evidenceDefensible outputMissing constraint or failure case
Recorder GPS and detectionRecorder trajectory and a source-linked observationNo asset position; copying GPS onto the asset is invalid.
Qualified pixel-to-ray calibrationCamera-relative bearing or angular extentNo world heading or metric distance.
Calibration plus synchronized Earth-referenced camera poseWorld ray or bearing cone from an uncertain originA single ray leaves distance unresolved.
World ray plus measured metric depthEstimated 3D point and uncertaintyDepth must refer to the selected feature, with range versus optical-axis depth specified.
World ray plus surveyed plane/terrain/3D surfaceConditional intersection estimateThe observed feature must lie on that surface; terrain below an elevated housing does not locate the housing.
Matched static feature in several calibrated views with metric poses and parallaxTriangulated estimatePure rotation, weak baseline, wrong association or moving features invalidate this method.
Surveyed asset or independently checked reference matchVerified claim with measurement/review provenanceA manual map click or high detector score alone is not verification.

An approximate height, object size or learned monocular depth can be an explicit prior in an experimental estimate. It must carry its uncertainty and validation results; it is not interchangeable with measured metric depth. Relative visual reconstruction without an established metric scale cannot supply world distances.

Calibrate the image that was actually observed​

Define a versioned CalibrationProfile for the camera/serial, firmware, lens, capture resolution, field of view, stabilization/horizon mode, zoom/crop and valid recording interval. Store the projection model, intrinsics, distortion, render transform and held-out residuals. Do not reuse a profile after a setting or mount change without qualification.

GoPro documents several stabilization, zoom and projection modes. Its telemetry distinguishes capture-relative camera orientation (CORI) from image orientation relative to the body (IORI); gravity and magnetometer availability vary by camera/firmware. Inspect the actual file instead of assuming these streams exist or provide absolute heading. GoPro metadata definitions.

Invert the detector's resize, letterbox, tile and crop transforms back to the decoded source frame. Then apply a qualified source-pixel-to-optical-ray mapping. For a pinhole image this includes distortion correction; fisheye images require the matching model. OpenCV provides distinct fisheye calibration and unprojection operations. OpenCV fisheye model.

Do not assume a quaternion reverses all HyperSmooth, horizon-lock, dynamic crop or nonlinear view warping. Qualify the complete rendered-image mapping per mode, using available metadata and control images. If it cannot be recovered or validated, retain observations without a metric world ray. Rolling-shutter readout, motion blur and ambiguous feature localization must enter the error budget or cause abstention. The selected feature needs a semantic identity (housing corner, mounting point, pole base); a bounding-box center is not automatically any of these.

Align telemetry, exposure and coordinate frames​

TelemetrySample preserves raw stream/key, units/scales, sample timing, decoded values, fix/quality indicators and source-file identity. GPMF sensor clocks and sample rates need reconciliation with MP4 timing; payload order or nominal FPS alone is insufficient. Its scaling and axis conventions are explicit metadata. GoPro timing documentation, GPMF stream definitions.

TimeAlignment records the mapping from source video PTS/timebase to telemetry and UTC, its offset/drift, uncertainty, valid intervals and discontinuities. Keep upload time separate. Chapter joins, trims, proxies, duplicated/skipped frames and telemetry gaps need explicit mappings. Resolve pose at exposure time; interpolate only over qualified gaps, using rotation-aware interpolation. Account for exposure/readout timing during fast motion; do not silently extrapolate.

PoseEstimate records position, orientation, frame conventions, quality and the alignment/calibration revisions used. Validate quaternion order, handedness, axis direction and transform direction against control observations. Gravity alone leaves yaw unresolved. Capture-relative orientation needs a surveyed or otherwise qualified Earth-frame alignment. Course of travel must not become optical heading for a side-facing, reversed or movable camera.

Represent measured camera-to-body/IMU rotation and receiver-to-optical-center lever arm as extrinsics with validity intervals. A rigid vehicle mount and a moving helmet/gimbal need different contracts. An externally measured vehicle heading helps only when its relationship to the optical camera is known.

Use a declared metric local frame for geometry, then convert to the declared geographic CRS. Preserve altitude datum and uncertainty; do not combine unknown altitude with a terrain model as if their heights share a reference. GeographicLib documents geodetic/local Cartesian conversion and ellipsoidal height. GeographicLib coordinate conversion.

Solve the appropriate geometry​

After validated image unprojection, let d_c be a unit optical ray, C the optical center in the metric world frame, and R_wc the rotation from camera to world. The candidate feature lies on:

d = R_wc d_c
X(lambda) = C + lambda d, lambda > 0

This is a ray, not a point. For a plane n · X + b = 0, substitution gives lambda = -(n · C + b) / (n · d). Reject nearly parallel rays, intersections behind the camera and a plane assumption incompatible with the feature. A known pole base may use a ground constraint; its elevated camera requires separate geometry or an explicit measured offset. A terrain model supplies terrain elevation, not the height of every object above it. Projection and pose must use consistent conventions. OpenCV projection and pose reference.

A measured range rho gives X = C + rho d. Optical-axis depth z instead requires rho = z / d_c,z; reject an incompatible sign or degenerate ray. Never mix these two depth conventions. A calibrated stereo or depth sensor also needs its transform and time alignment to the image camera.

For fixed cameras, triangulate corresponding physical features across translated views using qualified projection matrices. OpenCV's multi-view triangulation API takes corresponding image points and projection matrices. OpenCV triangulation.

The proposed solver must check correspondence quality, parallax, metric pose scale, positive depth, reprojection residuals and stability under input errors. More frames cannot cure a shared pose bias or a wrong asset association. Keep competing associations rather than forcing a merge. A moving object requires synchronized stereo or an explicit time-dependent motion model; applying static triangulation across its successive positions is not valid.

Preserve uncertainty and query it honestly​

Propagate GPS, pose, timing, pixel/feature, calibration and depth/surface errors through each method, using a documented Jacobian or Monte Carlo model. Include correlated errors and bias assumptions. Detector scores do not define positional covariance. DOP describes satellite geometry; it is not a radius in meters. NovAtel explanation of DOP.

As first-order sensitivity checks, position timing error grows roughly as speed times time offset, and transverse angular error as range times angular error in radians. These checks do not replace the complete uncertainty model. Do not average correlated GPS or heading errors away by treating every frame as an independent measurement. If a probability model is unqualified, show a stated bound/scenario range rather than labeling it a calibrated confidence interval.

Proposed LocationClaim fields:

  • Workspace, asset hypothesis and supporting observation IDs; immutable revision and superseded-claim link.
  • unknown, estimated or verified state; point/region/ray geometry and CRS, or an explicit unavailable reason.
  • Method, calibration/alignment/pose/depth/surface references, assumptions and validation diagnostics; uncertainty representation and its interpretation.
  • Reference measurement and reviewer provenance for verification. Verification does not mean zero error or establish every track as the same physical asset.

Location revisions must not overwrite source evidence or existing training artifacts. Resolve all inputs in the authenticated workspace; an LLM proposes a typed query, while server code applies permission and location-policy checks.

Area search should return located matches, possible matches whose qualified uncertainty region overlaps the area, and a separate unlocated group by default. Expose an explicit strict-location option with excluded/unlocated counts. An unbounded ray is not a finite location region unless a justified distance bound exists. A disjoint recorder path or a point estimate outside the area cannot alone exclude an asset: the recorder can see across the boundary. Label whether time filtering concerns recording/capture time, observation PTS or an asset's known existence interval; a timestamped sighting does not prove continued presence.

Staged contracts and acceptance gates​

These are proposed tests, not passing implementation results. Choose tolerances from the intended hardware, use case and independently surveyed held-out data.

StageImplementable contractRequired acceptance evidence
1. Recorder evidenceOriginal MP4/GPMF inventory, trajectory and source-time mapping; no asset coordinatesGolden files with missing GPS, poor fix, chapters and gaps decode reproducibly. Known time events expose offset/drift. Missing metadata remains unknown.
2. Calibrated bearingsMode-qualified source-pixel mapping and camera-relative rays; world rays only with qualified Earth poseHeld-out targets across image edges, camera rotations and crop/stabilization conditions. Wrong profile, unresolved yaw or stale mount calibration prevents world-ray output.
3. Conditional positionsIndependent depth, surface and static triangulation methods with declared assumptionsSurveyed targets at several heights/ranges. Elevated housing rejects a ground shortcut; pure rotation/weak parallax, moving features and incorrect matches abstain. Injected clock/pose/GPS errors change uncertainty or invalidate output.
4. Reviewed assets and searchVersioned location claims, reversible associations and typed area/time policyIndependent passes recover measured positions within empirically qualified uncertainty; shared bias is retained. Boundary, unbounded-ray and unlocated fixtures never silently become outside-area results. Foreign-workspace evidence is inaccessible.

Measure horizontal/vertical error, uncertainty coverage, abstention rate and false spatial exclusions by capture mode and geometry. Also report observation recall separately: good localization of a few detections does not establish that all visible cameras were found. Fix acceptance thresholds before a held-out test; do not invent an accuracy promise from metadata labels.

Practical GoPro test material​

Start with original, unedited MP4s and all related chapters, the camera model and firmware, capture settings, mount description and GPS availability. Preserve the originals even when analysis uses proxies. Inspect which telemetry actually exists before selecting a localization method.

Record a calibration target throughout the field of view in the intended mode; include controlled rotations and translated views of surveyed stationary targets at different heights and distances. Use paired stabilization/horizon/crop modes where comparison is needed. Measure the mounting transform and optical-center offset to any external pose sensor. Include a visible event with an independently recorded time reference to validate alignment.

A useful first road fixture includes repeat passes with meaningful lateral parallax, elevated camera housings, poles/lights as hard negatives, occlusion, moving objects and poor-GPS segments. Ground-truth asset coordinates and heights must have their own measurement uncertainty. Footage alone can test detection and tracking; trustworthy world-location evaluation needs these independent controls. No new collection or engine implementation is implied by this proposal.