Rank

Public Ranked Leaderboard

Disclaimer: "The test metric below is computed on the validation set. The final ranking will be computed on a holdout set, so the final standings may be different."
Classification Leaderboard
Rank Username Latency Acc on Classified # Classified Acc/Time Metric ref_acc
1st MIT_HAN_Lab 33.0 0.80585 20000.0 1.07263e-06 0.11115 0.69470
2nd MIT_HAN_Lab 35.0 0.80775 20000.0 1.01701e-06 0.11039 0.69736
3rd MIT_HAN_Lab 34.0 0.80605 20000.0 1.03729e-06 0.11000 0.69605
4th MIT_HAN_Lab 36.0 0.80825 20000.0 9.89368e-07 0.10961 0.69864
5th MIT_HAN_Lab 35.0 0.80695 20000.0 1.00797e-06 0.10959 0.69736
6th MIT_HAN_Lab 35.0 0.7996 20000.0 9.99501e-07 0.10224 0.69736
7th MIT_HAN_Lab 35.0 0.7876 20000.0 9.79268e-07 0.09024 0.69736
8th MIT_HAN_Lab 35.0 0.7742 20000.0 9.71376e-07 0.07684 0.69736
9th EdgeAI 27.0 0.74725 20000.0 1.24542e-06 0.06165 0.68560
10th EdgeAI 27.0 0.74725 20000.0 1.24542e-06 0.06165 0.68560
11th MIT_HAN_Lab 35.0 0.7329 20000.0 9.21792e-07 0.03554 0.69736
12th MIT_HAN_Lab 36.0 0.73405 20000.0 8.90741e-07 0.03541 0.69864
13th MIT_HAN_Lab 33.0 0.7191 20000.0 9.52343e-07 0.02440 0.69470
14th EdgeAI 26.0 0.7078 20000.0 1.17310e-06 0.02391 0.68389
15th EdgeAI 26.0 0.7078 20000.0 1.17310e-06 0.02391 0.68389
16th EdgeAI 32.0 0.717 20000.0 9.70721e-07 0.02370 0.69330
17th MIT_HAN_Lab 33.0 0.718 20000.0 9.45673e-07 0.02330 0.69470
18th EdgeAI 32.0 0.71565 20000.0 9.67765e-07 0.02235 0.69330
19th EdgeAI 34.0 0.69875 20000.0 9.07091e-07 0.00270 0.69605
20th MIT_HAN_Lab 30.0 0.6924 20000.0 9.97975e-07 0.00202 0.69038
21st MIT_HAN_Lab 30.0 0.6924 20000.0 9.97975e-07 0.00202 0.69038
22nd MIT_HAN_Lab 23.0 0.67585 20000.0 1.12642e-06 -0.00442 0.68027
23rd EdgeAI 30.0 0.68355 20000.0 9.97791e-07 -0.00683 0.69038
24th EdgeAI 28.0 0.67525 20000.0 1.06503e-06 -0.01200 0.68725
25th expasoft-denisov 17.0 0.66625 20000.0 1.11042e-06 -0.01402 0.68027
26th expasoft-denisov 17.0 0.66625 20000.0 1.11042e-06 -0.01402 0.68027
27th EdgeAI 24.0 0.6621 20000.0 1.10350e-06 -0.01817 0.68027
28th expasoft-denisov 17.0 0.7147432964 10927.0 2.18000e-06 -0.03337 0.74812
29th expasoft-denisov 17.0 0.7147432964 10927.0 2.18000e-06 -0.03337 0.74812
30th expasoft-denisov 17.0 0.7147432964 10927.0 2.18000e-06 -0.03337 0.74812
31st EdgeAI 26.0 0.2281 20000.0 3.76541e-07 -0.45579 0.68389
32nd MIT_HAN_Lab 22.0 0.00105 20000.0 1.75000e-09 -0.67922 0.68027
33rd MIT_HAN_Lab 22.0 0.00105 20000.0 1.75000e-09 -0.67922 0.68027
34th MIT_HAN_Lab 24.0 0.00105 20000.0 1.75000e-09 -0.67922 0.68027
35th MIT_HAN_Lab 25.0 0.00105 20000.0 1.75000e-09 -0.68107 0.68212
36th EdgeAI 26.0 0.001 20000.0 1.63930e-09 -0.68289 0.68389
37th wildkid1024 33.0 0.00115 20000.0 1.65308e-09 -0.69355 0.69470
38th EdgeAI 33.0 0.00115 20000.0 1.65308e-09 -0.69355 0.69470
39th EdgeAI 33.0 0.00115 20000.0 1.65308e-09 -0.69355 0.69470
40th MIT_HAN_Lab 35.0 0.00095 20000.0 1.20058e-09 -0.69641 0.69736
  • timerLatency
    Latency (ms) is single-threaded, non-batched runtime measured on a single Pixel 2 big core of classifying one image.
  • whatshotMetric
    Accuracy improvement over the reference accuracy from the Pareto optimal curve. See track 1 description for details.
  • adjustAcc on Classified
    Acc on Classified is the accuracy in [0, 1] computed based only on the images classified within the wall-time.
  • class# Classified
    # Classified is the number of images classified within the wall-time.
  • infoAcc/Time
    Acc/Time is the accuracy divided by either the total inference time or the wall-time, whichever is longer.
  • inforef_acc
    The reference accuracy of models from the Pareto optimal curve that have the same latency as the submission.
  • infoBucket
    The latency bucket against which the submission is scored. Should be either [24, 36], [80, 120], or None if the model is too slow.


Detection Leaderboard
Rank Username Metric Runtime mAP over time mAP of processed
1st MIT_HAN_Lab 0.05132 114.0 2.37349e-03 2.70577e-01
2nd MIT_HAN_Lab 0.05076 112.0 2.40995e-03 2.69914e-01
3rd MIT_HAN_Lab 0.04921 116.0 2.31526e-03 2.68571e-01
4th MIT_HAN_Lab 0.04835 113.0 2.36777e-03 2.67558e-01
5th MIT_HAN_Lab 0.04646 114.0 2.33084e-03 2.65715e-01
6th MIT_HAN_Lab 0.04264 114.0 2.29735e-03 2.61898e-01
7th MIT_HAN_Lab 0.03800 115.0 2.23748e-03 2.57310e-01
8th MIT_HAN_Lab 0.03227 114.0 2.20636e-03 2.51525e-01
9th MIT_HAN_Lab 0.02520 114.0 2.14440e-03 2.44461e-01
10th MIT_HAN_Lab 0.02343 116.0 2.09300e-03 2.42788e-01
11th MIT_HAN_Lab 0.01445 116.0 2.01563e-03 2.33813e-01
12th MIT_HAN_Lab 0.01396 116.0 2.01135e-03 2.33317e-01
13th MIT_HAN_Lab 0.01325 116.0 2.00527e-03 2.32611e-01
14th MIT_HAN_Lab 0.00371 116.0 1.92299e-03 2.23067e-01
15th MIT_HAN_Lab -0.00270 113.0 1.91594e-03 2.16502e-01
16th Orange-Control -0.00491 119.0 -8.40336e-03 2.14599e-01
17th Orange-Control -0.00491 119.0 -8.40336e-03 2.14599e-01
18th Orange-Control -0.00491 119.0 -8.40336e-03 2.14599e-01
19th Orange-Control -0.00491 119.0 -8.40336e-03 2.14599e-01
20th Orange-Control -0.00491 119.0 -8.40336e-03 2.14599e-01
21st Orange-Control -0.00491 119.0 -8.40336e-03 2.14599e-01
22nd Orange-Control -0.00491 119.0 -8.40336e-03 2.14599e-01
23rd Orange-Control -0.00491 119.0 -8.40336e-03 2.14599e-01
24th Orange-Control -0.00927 118.0 1.78131e-03 2.10195e-01
25th Orange-Control -0.00927 118.0 1.78131e-03 2.10195e-01
26th Orange-Control -0.00927 118.0 1.78131e-03 2.10195e-01
27th Orange-Control -0.00927 118.0 1.78131e-03 2.10195e-01
28th Orange-Control -0.01069 114.0 -8.78884e-03 2.08560e-01
29th MIT_HAN_Lab -0.01275 111.0 1.85903e-03 2.06353e-01
30th Orange-Control -0.01433 118.0 1.73844e-03 2.05137e-01
31st Orange-Control -0.01433 118.0 1.73844e-03 2.05137e-01
32nd Orange-Control -0.01433 118.0 1.73844e-03 2.05137e-01
33rd Orange-Control -0.01647 109.0 -9.17431e-03 2.02521e-01
34th Orange-Control -0.01647 109.0 -9.17431e-03 2.02521e-01
35th Orange-Control -0.01647 109.0 -9.17431e-03 2.02521e-01
36th Orange-Control -0.01647 109.0 -9.17431e-03 2.02521e-01
37th Orange-Control -0.01647 109.0 -9.17431e-03 2.02521e-01
38th Orange-Control -0.01647 109.0 -9.17431e-03 2.02521e-01
39th Orange-Control -0.01647 109.0 -9.17431e-03 2.02521e-01
40th Orange-Control -0.01647 109.0 -9.17431e-03 2.02521e-01
41st MIT_HAN_Lab -0.01697 111.0 1.82101e-03 2.02132e-01
42nd Orange-Control -0.01976 108.0 1.84424e-03 1.99178e-01
43rd Orange-Control -0.02291 107.0 1.83148e-03 1.95968e-01
44th MIT_HAN_Lab -0.02297 108.0 1.81452e-03 1.95968e-01
45th MIT_HAN_Lab -0.02297 108.0 1.81452e-03 1.95968e-01
46th MIT_HAN_Lab -0.02297 108.0 1.81452e-03 1.95968e-01
47th Orange-Control -0.02310 108.0 1.81335e-03 1.95841e-01
48th Orange-Control -0.02443 118.0 1.65284e-03 1.95035e-01
49th Orange-Control -0.02974 108.0 1.75182e-03 1.89197e-01
50th MIT_HAN_Lab -0.03102 93.0 2.01112e-03 1.87034e-01
51st MIT_HAN_Lab -0.03102 93.0 2.01112e-03 1.87034e-01
52nd MIT_HAN_Lab -0.03102 93.0 2.01112e-03 1.87034e-01
53rd Orange-Control -0.03102 93.0 2.01112e-03 1.87034e-01
54th Orange-Control -0.03102 93.0 2.01112e-03 1.87034e-01
55th Orange-Control -0.03102 93.0 2.01112e-03 1.87034e-01
56th Orange-Control -0.03102 93.0 2.01112e-03 1.87034e-01
57th Orange-Control -0.03102 93.0 2.01112e-03 1.87034e-01
58th Orange-Control -0.03102 93.0 2.01112e-03 1.87034e-01
59th Orange-Control -0.03518 115.0 1.60109e-03 1.84125e-01
60th Orange-Control -0.03518 115.0 1.60109e-03 1.84125e-01
61st Orange-Control -0.04820 108.0 1.58094e-03 1.70742e-01
62nd Orange-Control -0.04820 108.0 1.58094e-03 1.70742e-01
63rd MIT_HAN_Lab -0.05020 108.0 1.56236e-03 1.68735e-01
64th MIT_HAN_Lab -0.05020 108.0 1.56236e-03 1.68735e-01
65th Orange-Control -0.07395 59.0 2.42716e-03 1.43202e-01
66th Orange-Control -0.07395 59.0 2.42716e-03 1.43202e-01
67th Orange-Control -0.07397 58.0 2.46872e-03 1.43186e-01
68th MIT_HAN_Lab -0.18589 104.0 3.15649e-04 3.28275e-02
69th MIT_HAN_Lab -0.19696 117.0 1.91909e-04 2.24534e-02
70th MIT_HAN_Lab -0.19724 117.0 1.89487e-04 2.21700e-02
71st MIT_HAN_Lab -0.19797 117.0 1.83257e-04 2.14410e-02
72nd MIT_HAN_Lab -0.20016 117.0 1.64531e-04 1.92501e-02
73rd Orange-Control -0.21715 49.0 2.18948e-07 1.07284e-05
74th Orange-Control -0.21715 51.0 2.10362e-07 1.07284e-05
75th Orange-Control -0.21882 106.0 1.01213e-08 1.07285e-06
76th MIT_HAN_Lab -0.21888 107.0 7.06129e-10 7.55558e-08
77th MIT_HAN_Lab -0.21888 107.0 7.06129e-10 7.55558e-08
  • timerRuntime
    Latency per image (ms).
  • whatshotMetric
    COCO mAP computed on the entire minival dataset.
  • adjustmAP Over Time
    COCO mAP on the minival dataset divided by latency per image.
  • classmAP of Processed
    COCO mAP computed only on the processed images.