Kosmic Worlds
Action labels for driving video
Physics-Aware Control Extraction
Estimate steering wheel angle, throttle, and brake from how a vehicle moves. Synchronize those controls with video for world-model and driving-policy training.
Video-input evaluation, speed > 5 m/s. Errors measured at the steering wheel; equal-dataset weighting. See evaluation scope ↓
Method
PACE uses a proprietary physics-based model to estimate control inputs from egocentric driving video. The resulting action labels are synchronized with the video for model training.
Driving footage with frame timestamps.
Proprietary physics-based control extraction.
The public overview presents capabilities, validation results, and limitations. Algorithmic details are available to partners under NDA.
Evaluation datasets
One representative front-camera frame from each of the eight evaluation datasets, spanning passenger cars, a van, a coach, and a semi-truck.
Recorded-control validation
Predicted steering is compared with synchronized CAN steering targets at speeds above 5 m/s. Results include both equal-dataset and pooled summaries.
Steering wheel angle · errors in degrees
Download CSVSteering wheel angle · speed > 5 m/s · 24,983,552 scored frames
| Dataset | Scored frames | MAE | RMSE |
|---|---|---|---|
| A2D2 | 32,325 | 1.99° | 3.03° |
| comma2k19 | 2,043,677 | 2.18° | 3.11° |
| Turing Japan | 2,354,277 | 1.97° | 2.83° |
| Yaak L2D | 18,873,404 | 5.70° | 12.32° |
| ZOD | 284,532 | 4.42° | 5.93° |
| Udacity Self-Driving Car Dataset | 693,153 | 1.31° | 2.24° |
| RoboBus | 690,225 | 6.67° | 10.46° |
| VTTI CONOPS | 11,959 | 9.03° | 11.02° |
| Equal-dataset weighting | 8 datasets | 4.16° | 7.50° |
| All scored frames pooled | 24,983,552 | 4.95° | 10.95° |
MAE is the mean absolute error. RMSE gives greater weight to large errors. Equal-dataset MAE averages the eight dataset MAEs; equal-dataset RMSE is the square root of the mean dataset MSE. Pooled metrics weight every scored frame equally.
One front-camera timeline is counted per dataset: Udacity’s center camera, RoboBus camera 200, and synchronized Yaak state rows counted once. Original finite-value, duplicate, direction, and camera-alignment masks are preserved.
The public steering benchmark scores frames above 5 m/s. Selected demonstration excerpts below have a different scope. Detailed speed breakdowns are available in the full partner report.
Longitudinal controls
Pedal channels are scored only where the recorded signal is comparable to pedal position. The public overview reports all-speed throttle results on seven datasets, and brake results on Yaak L2D.
| Dataset | Scored frames | ThrottleMAE | ThrottleRMSE | BrakeMAE | BrakeRMSE |
|---|---|---|---|---|---|
| A2D2 | 65,371 | 2.51 | 4.51 | — | — |
| comma2k19 | 2,360,839 | 7.21 | 10.17 | — | — |
| Yaak L2D | 24,885,560 | 5.05 | 8.37 | 1.95 | 5.15 |
| ZOD | 351,055 | 7.71 | 11.54 | — | — |
| Udacity Self-Driving Car Dataset | 976,609 | 4.36 | 6.78 | — | — |
| RoboBus | 847,852 | 9.96 | 17.01 | — | — |
| VTTI CONOPS | 39,765 | 9.10 | 16.15 | — | — |
| Equal-dataset weighting | 7 datasets | 6.56 | 11.48 | 1.95* | 5.15* |
Source: public overview, Table 2. These are separate all-speed pedal results, not the steering threshold populations. A dash denotes unavailable comparable ground truth. *Brake averages are Yaak-only. Turing is omitted because it has no usable pedal signal in this evaluation.
Download CSVPedal accuracy depends on the vehicle, signal definition, and evaluation setup. These scores do not establish equivalent accuracy on an unseen vehicle.
Frame progression
Four sequences compare PACE with recorded controls across Yaak L2D, RoboBus, VTTI CONOPS, and comma2k19. Each row follows one maneuver through four camera frames.
Selected maneuver
A 15-second RoboBus excerpt shows the entire turn, including the wheel unwinding. Magenta is the recorded steering; green is PACE.
Scope and limitations
Small motion errors are amplified at low speed. A stationary trajectory cannot reveal steering while stopped. Parking and reversing remain important open cases. Detailed lower-speed analysis is available in the full partner report.
Accuracy varies across vehicles and operating conditions. Results for known evaluation vehicles do not guarantee equivalent performance on an unknown vehicle.
The current dynamics model treats the vehicle as one rigid body. The coach and truck results do not validate trailer articulation or a general multi-body model.
Steering paths are approximate projections for visualization, not measured future trajectories. Unsupported channels are omitted or marked N/A. The demonstrations use real dataset footage with control overlays.
Documentation
October 2026 · 7 pages · PDF
The public overview covers the method, eight-dataset steering evaluation, pedal results, qualitative examples, and limitations.
Dataset attributions and notices are included in the overview. A fuller technical report is available to partners under NDA.
Research and partnerships
PACE supports our internal preparation of driving data for Kosmic Worlds. Contact us about validation, dataset integration, or world-model design partnerships.