Ego motion prediction for AR-HUD latency compensation

A contact-analog overlay is lit 50-500 ms after the newest sensor sample, so it must be drawn for where the car will be at display time, not where it was. Replay of recorded drives with the real fitted predictors of the v0.4.0 benchmark.

Report GitHub Demo video (78 s) All drive-replay demos
-

Driver view, simulated AR-HUD each method's arrow, anchored 20 m ahead; white brackets = true target at display time

Simulated view from recorded nuScenes data, not a Mercedes-Benz HUD. Lane band = the recorded future path; HUD frame and eye point are illustrative.

Bird's-eye view heading up, follows the car

Controls

Display latency (prediction horizon)200 ms
50100150200300500
Injected input noise (report section 3)

Level 1: pose position 0.05 m, pose heading 0.17 deg, wheel speed 0.1 m/s, yaw rate 0.29 deg/s, acceleration 0.2 m/s^2, steering wheel 0.57 deg (white, per sample); level 2 doubles it.

Methods

Errors at display time

Now = the current start point. Scene mean = mean |error| over all start points of this scene (every 0.1 s), the same numbers the benchmark computes. Test split = all 192 test scenes from the report (noise levels: 200 ms only). 20 m offset = lateral error + 20 m x sin(heading error); + = arrow left of the target.

Ego motion prediction for
AR-HUD latency compensation

Recorded nuScenes CAN bus drives: physics models, Kalman filters and learned predictors on a held-out test split

Pavan Yadav Annappa · open-source portfolio project · github.com/pavanyadava007/drive-replay

Held-out test split: 192 scenes, 1.04 h

huggingface.co/spaces/pavanyadava07/drive-replay

github.com/pavanyadava007/drive-replay

Open-loop replay, one car model. Simulated view, not a production HUD. Data CC BY-NC-SA 4.0.