UZH FPV Temporary Leader Board

This is the temporary leader board based on the 2019 and 2020 UZH FPV competitions. It evaluates position estimation accuracy on six dataset sequences for which the ground truth is not public.

We have recently discovered that there is an issue with the rotations provided by the ground truth in the UZH FPV dataset, see here for more details. The evaluation method has changed to address such issue.
We are planning to implement a new method for ground truth pose estimation. Once we can thoroughly test the new ground truth we will release it and re-evaluate the competition submissions to get final results.

Sequences

Sequence	Video	Length	Start Time Snapdragon	Start Time Davis	Download Snapdragon	Download Davis
indoor forward 11		85.68	1540824001 s	1540824001 s	bag, zip	bag, zip
indoor forward 12		124.07	1540824296 s	1540824296 s	bag, zip	bag, zip
indoor 45deg 3		119.82	1623 s	1545305934 s	bag, zip	bag, zip
indoor 45deg 16		58.72	133 s	1545315222 s	bag, zip	bag, zip
outdoor forward 9		314.41	372 s	1540102003 s	bag, zip	bag, zip
outdoor forward 10		455.63	674 s	1540102304 s	bag, zip	bag, zip

Evaluation Method

The submission will be ranked based on the position accuracy. We use an adapted metric with respect to the one adopted by KITTI to account for the issue with the rotations provided by the current ground truth (see above).
We will switch back to the same metric as in KITTI, as soon as the new ground truth is available.

The average translation error (in percentage) over all possible subsequences of a set of certain lengths is used for ranking the submitted results. We use our publicly available trajectory evaluation toolbox to evaluate the estimated trajectories.

For each x in {40m, 60m, 80m, 100m, 120m}, subtrajectories of length x are uniformly sampled (in time) from the entire trajectory. The smaller the subtrajectory, the more samples there are in a given sequence. Each of those subtrajectories is then aligned to the ground truth.
To avoid relying on rotation estimates, we align each subtrajectory based on the first 10% (distance-wise, not time-wise) worth of positions. Since gravity is observable, we only optimize position and yaw. For the ranking, the average of relative position error is taken across all such samples. The length of the sequence affects the amount of samples in that sequence. For example, outdoor_forward_9 has {964, 916, 862, 807, 762} samples for trajectory lengths {40, 60, 80, 100, 120}. If a trajectory is shorter than x, no samples are generated for x.

Submission

To make a submission, you will need to create an account, follow the instructions below. Each account can make only one submission, but it can be changed afterwards. Submissions are not automatically published – you can review the results and decide whether to publish by yourself (evaluation usually takes around 1 minute). Submissions can also be withdrawn completely.

Click here for the submission format specification.