criterion performance measurements
overview
want to understand this report?
| serialization (time) | |
| deserialization (time) | |
| serialization (time) | |
| deserialization (time) | |
| serialization (time) | |
| deserialization (time) | |
| serialization (time) | |
| deserialization (time) | |
| serialization (time) | |
| deserialization (time) |
serialization (time)/[Direction]-flat
3.2 3.30 3.40 3.5 3.6 3.7 3.80 3.90
mean |
20 30 40 50 60 70 80 10 iters 100 150 200 250 300 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.27 ms | 3.29 ms | 3.32 ms |
| R² goodness-of-fit | 0.998 | 0.999 | 1.000 |
| Mean execution time | 3.26 ms | 3.29 ms | 3.36 ms |
| Standard deviation | 44.4 μs | 110 μs | 208 μs |
Outlying measurements have moderate (17.7%) effect on estimated standard deviation.
serialization (time)/[Direction]-store
3 2.80 2.90 3.1 3.2 3.30
mean |
20 30 40 50 60 70 80 90 10 iters 100 150 200 250 300 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.82 ms | 2.84 ms | 2.86 ms |
| R² goodness-of-fit | 0.995 | 0.998 | 1.000 |
| Mean execution time | 2.84 ms | 2.85 ms | 2.90 ms |
| Standard deviation | 33.1 μs | 77.6 μs | 148 μs |
Outlying measurements have moderate (12.9%) effect on estimated standard deviation.
serialization (time)/[Direction]-binary
15 16 14.5 14.8 15.3 15.5 15.8
mean |
10 15 20 25 5 iters 200 300 400 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 14.4 ms | 14.5 ms | 14.6 ms |
| R² goodness-of-fit | 0.998 | 0.999 | 1.000 |
| Mean execution time | 14.6 ms | 14.7 ms | 14.9 ms |
| Standard deviation | 138 μs | 309 μs | 572 μs |
Outlying measurements have slight (3.8%) effect on estimated standard deviation.
serialization (time)/[Direction]-cereal
10 11 12 10.5 11.5 12.5
mean |
10 15 20 25 30 5 iters 100 150 200 250 300 350 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 9.63 ms | 9.79 ms | 9.88 ms |
| R² goodness-of-fit | 0.990 | 0.997 | 1.000 |
| Mean execution time | 9.94 ms | 10.1 ms | 10.4 ms |
| Standard deviation | 214 μs | 523 μs | 975 μs |
Outlying measurements have moderate (25.3%) effect on estimated standard deviation.
serialization (time)/[Direction]-persist
2.55 2.6 2.65 2.7 2.75 2.8 2.85
mean |
40 60 80 20 iters 100 150 200 250 300 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.55 ms | 2.58 ms | 2.63 ms |
| R² goodness-of-fit | 0.996 | 0.998 | 1.000 |
| Mean execution time | 2.56 ms | 2.57 ms | 2.59 ms |
| Standard deviation | 23.1 μs | 51.2 μs | 102 μs |
Outlying measurements have slight (7.2%) effect on estimated standard deviation.
serialization (time)/[Direction]-packman
70 80 90 100 110
mean |
4 6 8 10 2 iters 400 600 800 0 s 200 ms 1 s 1.2
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 76.5 ms | 98.0 ms | 116 ms |
| R² goodness-of-fit | 0.757 | 0.908 | 0.995 |
| Mean execution time | 73.1 ms | 80.7 ms | 95.7 ms |
| Standard deviation | 11.7 ms | 17.1 ms | 22.6 ms |
Outlying measurements have severe (68.3%) effect on estimated standard deviation.
serialization (time)/[Direction]-serialise
25 26 27 28 25.5 26.5 27.5 28.5
mean |
5 8 10 13 15 18 2.5 iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 25.0 ms | 25.5 ms | 26.1 ms |
| R² goodness-of-fit | 0.997 | 0.999 | 1.000 |
| Mean execution time | 25.3 ms | 25.5 ms | 26.2 ms |
| Standard deviation | 269 μs | 798 μs | 1.58 ms |
Outlying measurements have slight (9.6%) effect on estimated standard deviation.
deserialization (time)/[Direction]-flat
10 15 20 25 30
mean |
10 15 20 25 5 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 12.2 ms | 16.4 ms | 20.6 ms |
| R² goodness-of-fit | 0.480 | 0.688 | 0.842 |
| Mean execution time | 12.9 ms | 14.6 ms | 18.0 ms |
| Standard deviation | 3.74 ms | 5.77 ms | 9.37 ms |
Outlying measurements have severe (95.6%) effect on estimated standard deviation.
deserialization (time)/[Direction]-store
15 20 25 30 35 40
mean |
10 15 20 5 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 16.0 ms | 21.3 ms | 26.3 ms |
| R² goodness-of-fit | 0.481 | 0.711 | 0.878 |
| Mean execution time | 16.4 ms | 18.7 ms | 23.2 ms |
| Standard deviation | 4.41 ms | 7.09 ms | 11.6 ms |
Outlying measurements have severe (95.0%) effect on estimated standard deviation.
deserialization (time)/[Direction]-binary
45 50 55 60 65 70
mean |
4 6 8 10 12 2 iters 200 300 400 500 600 700 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 47.0 ms | 49.5 ms | 59.6 ms |
| R² goodness-of-fit | 0.757 | 0.912 | 1.000 |
| Mean execution time | 46.7 ms | 48.7 ms | 59.8 ms |
| Standard deviation | 481 μs | 6.78 ms | 12.9 ms |
Outlying measurements have severe (52.0%) effect on estimated standard deviation.
deserialization (time)/[Direction]-cereal
15 20 25 30 35
mean |
10 15 20 5 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 16.7 ms | 21.6 ms | 26.7 ms |
| R² goodness-of-fit | 0.593 | 0.763 | 0.894 |
| Mean execution time | 17.0 ms | 18.7 ms | 22.2 ms |
| Standard deviation | 3.60 ms | 5.79 ms | 8.76 ms |
Outlying measurements have severe (90.3%) effect on estimated standard deviation.
deserialization (time)/[Direction]-persist
15 20 25 30 35
mean |
10 15 20 25 5 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 13.5 ms | 17.2 ms | 21.8 ms |
| R² goodness-of-fit | 0.512 | 0.706 | 0.867 |
| Mean execution time | 14.2 ms | 16.0 ms | 19.1 ms |
| Standard deviation | 2.99 ms | 5.63 ms | 8.52 ms |
Outlying measurements have severe (95.3%) effect on estimated standard deviation.
deserialization (time)/[Direction]-packman
80 100 120 140 160 180
mean |
2 3 4 5 6 7 8 1 iters 400 600 800 0 s 200 ms 1 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 85.0 ms | 118 ms | 143 ms |
| R² goodness-of-fit | 0.668 | 0.907 | 0.995 |
| Mean execution time | 102 ms | 118 ms | 144 ms |
| Standard deviation | 16.2 ms | 29.3 ms | 43.8 ms |
Outlying measurements have severe (73.2%) effect on estimated standard deviation.
deserialization (time)/[Direction]-serialise
60 65 70 75 80 85 90
mean |
4 6 8 10 2 iters 400 600 800 0 s 200 ms 1 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 63.4 ms | 70.7 ms | 86.3 ms |
| R² goodness-of-fit | 0.844 | 0.933 | 1.000 |
| Mean execution time | 62.9 ms | 65.4 ms | 72.5 ms |
| Standard deviation | 638 μs | 7.63 ms | 12.9 ms |
Outlying measurements have moderate (43.3%) effect on estimated standard deviation.
serialization (time)/BinTree Int-flat
89 90 91 92 93
mean |
4 6 8 10 2 iters 400 600 800 0 s 200 ms 1 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 89.4 ms | 90.4 ms | 91.3 ms |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 89.6 ms | 90.2 ms | 91.3 ms |
| Standard deviation | 804 μs | 1.32 ms | 2.00 ms |
Outlying measurements have slight (9.0%) effect on estimated standard deviation.
serialization (time)/BinTree Int-store
100 110 120 130 140 150
mean |
2 3 4 5 6 7 8 1 iters 400 600 800 0 s 200 ms 1 s 1.2
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 95.6 ms | 124 ms | 146 ms |
| R² goodness-of-fit | 0.655 | 0.907 | 0.994 |
| Mean execution time | 104 ms | 115 ms | 133 ms |
| Standard deviation | 14.6 ms | 21.4 ms | 29.0 ms |
Outlying measurements have severe (60.3%) effect on estimated standard deviation.
serialization (time)/BinTree Int-binary
2 2.05 2.1 2.15 2.2 2.25
mean |
1 2 2 3 3 4 4 0.5 iters 2 4 6 8 10 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.68 s | 2.11 s | 2.48 s |
| R² goodness-of-fit | 0.982 | 0.995 | 1.000 |
| Mean execution time | 2.03 s | 2.11 s | 2.21 s |
| Standard deviation | 26.9 ms | 101 ms | 135 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
serialization (time)/BinTree Int-cereal
1.7 1.73 1.75 1.78 1.8 1.83
mean |
1 2 2 3 3 4 4 0.5 iters 1 2 3 4 5 6 7 8 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.68 s | 1.77 s | 1.94 s |
| R² goodness-of-fit | 0.998 | 0.999 | 1.000 |
| Mean execution time | 1.71 s | 1.76 s | 1.79 s |
| Standard deviation | 18.9 ms | 52.5 ms | 72.2 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
serialization (time)/BinTree Int-persist
90 100 110 120 130 140 150
mean |
2 3 4 5 6 7 8 1 iters 400 600 800 0 s 200 ms 1 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 81.3 ms | 113 ms | 148 ms |
| R² goodness-of-fit | 0.663 | 0.872 | 0.999 |
| Mean execution time | 100 ms | 111 ms | 129 ms |
| Standard deviation | 16.1 ms | 21.3 ms | 27.7 ms |
Outlying measurements have severe (60.4%) effect on estimated standard deviation.
serialization (time)/BinTree Int-packman
3.6 3.62 3.64 3.66 3.68 3.7
mean |
1 2 2 3 3 4 4 0.5 iters 5 10 15 20 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.42 s | 3.68 s | 4.01 s |
| R² goodness-of-fit | 0.997 | 0.999 | 1.000 |
| Mean execution time | 3.60 s | 3.67 s | 3.69 s |
| Standard deviation | 7.40 ms | 46.5 ms | 60.1 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
serialization (time)/BinTree Int-serialise
2.48 2.5 2.52 2.54
mean |
1 2 2 3 3 4 4 0.5 iters 2 4 6 8 10 12 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.46 s | 2.56 s | 2.67 s |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 2.47 s | 2.51 s | 2.54 s |
| Standard deviation | 18.9 ms | 36.3 ms | 46.5 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
deserialization (time)/BinTree Int-flat
450 500 550 600
mean |
1 2 2 3 3 4 4 0.5 iters 2 3 0 s 500 ms 1 s 1.5 2.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 454 ms | 615 ms | 776 ms |
| R² goodness-of-fit | 0.967 | 0.991 | 1.000 |
| Mean execution time | 475 ms | 551 ms | 592 ms |
| Standard deviation | 13.5 ms | 72.4 ms | 94.9 ms |
Outlying measurements have moderate (23.6%) effect on estimated standard deviation.
deserialization (time)/BinTree Int-store
425 450 475 500 525 550 575
mean |
1 2 2 3 3 4 4 0.5 iters 2 3 0 s 500 ms 1 s 1.5 2.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 455 ms | 616 ms | 795 ms |
| R² goodness-of-fit | 0.964 | 0.990 | 1.000 |
| Mean execution time | 450 ms | 516 ms | 562 ms |
| Standard deviation | 24.5 ms | 64.6 ms | 88.4 ms |
Outlying measurements have moderate (23.4%) effect on estimated standard deviation.
deserialization (time)/BinTree Int-binary
2 1.96 1.97 1.98 1.99 2.01 2.02 2.03 2.04
mean |
1 2 2 3 3 4 4 0.5 iters 2 4 6 8 10 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.85 s | 1.93 s | 2.05 s |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 1.96 s | 2.00 s | 2.03 s |
| Standard deviation | 29.1 ms | 36.2 ms | 42.7 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
deserialization (time)/BinTree Int-cereal
475 500 525 550 575 600 625
mean |
1 2 2 3 3 4 4 0.5 iters 2 3 0 s 500 ms 1 s 1.5 2.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 510 ms | 660 ms | 808 ms |
| R² goodness-of-fit | 0.975 | 0.993 | 1.000 |
| Mean execution time | 506 ms | 571 ms | 607 ms |
| Standard deviation | 17.2 ms | 62.3 ms | 83.4 ms |
Outlying measurements have moderate (23.0%) effect on estimated standard deviation.
deserialization (time)/BinTree Int-persist
450 500 550 600
mean |
1 2 2 3 3 4 4 0.5 iters 2 3 0 s 500 ms 1 s 1.5 2.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 471 ms | 640 ms | 824 ms |
| R² goodness-of-fit | 0.964 | 0.990 | 1.000 |
| Mean execution time | 468 ms | 537 ms | 577 ms |
| Standard deviation | 20.7 ms | 67.8 ms | 92.3 ms |
Outlying measurements have moderate (23.5%) effect on estimated standard deviation.
deserialization (time)/BinTree Int-packman
4.1 4.2 4.3 4.4 4.5
mean |
1 2 2 3 3 4 4 0.5 iters 5 10 15 20 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.71 s | 4.26 s | 4.87 s |
| R² goodness-of-fit | 0.991 | 0.997 | 1.000 |
| Mean execution time | 4.16 s | 4.32 s | 4.45 s |
| Standard deviation | 77.6 ms | 170 ms | 235 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
deserialization (time)/BinTree Int-serialise
2.05 2.1 2.15 2.2
mean |
1 2 2 3 3 4 4 0.5 iters 2 4 6 8 10 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.99 s | 2.14 s | 2.37 s |
| R² goodness-of-fit | 0.998 | 0.998 | 1.000 |
| Mean execution time | 2.07 s | 2.14 s | 2.18 s |
| Standard deviation | 25.4 ms | 68.8 ms | 93.1 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
serialization (time)/BinTree Direction-flat
128 129 129 130 130
mean |
2 3 4 5 6 7 8 1 iters 400 600 800 0 s 200 ms 1 s 1.2
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 128 ms | 128 ms | 128 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 128 ms | 128 ms | 129 ms |
| Standard deviation | 150 μs | 684 μs | 1.07 ms |
Outlying measurements have moderate (10.9%) effect on estimated standard deviation.
serialization (time)/BinTree Direction-store
115 120 125 130 135 140 145
mean |
2 3 4 5 6 7 8 1 iters 400 600 800 0 s 200 ms 1 s 1.2
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 114 ms | 127 ms | 158 ms |
| R² goodness-of-fit | 0.872 | 0.953 | 1.000 |
| Mean execution time | 115 ms | 119 ms | 131 ms |
| Standard deviation | 100 μs | 10.8 ms | 16.5 ms |
Outlying measurements have moderate (23.5%) effect on estimated standard deviation.
serialization (time)/BinTree Direction-binary
2.20 2.25 2.3 2.35
mean |
1 2 2 3 3 4 4 0.5 iters 2 4 6 8 10 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 2.40 s | 2.43 s | 2.50 s |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 2.21 s | 2.29 s | 2.34 s |
| Standard deviation | 25.1 ms | 77.7 ms | 105 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
serialization (time)/BinTree Direction-cereal
1.15 1.20 1.25 1.3 1.35
mean |
1 2 2 3 3 4 4 0.5 iters 1 2 3 4 5 6 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.12 s | 1.26 s | 1.56 s |
| R² goodness-of-fit | 0.988 | 0.993 | 1.000 |
| Mean execution time | 1.16 s | 1.25 s | 1.31 s |
| Standard deviation | 45.5 ms | 87.6 ms | 121 ms |
Outlying measurements have moderate (20.0%) effect on estimated standard deviation.
serialization (time)/BinTree Direction-persist
90 95 100 105 110 115 120 125
mean |
2 3 4 5 6 7 8 9 1 iters 400 600 800 0 s 200 ms 1 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 89.7 ms | 97.9 ms | 126 ms |
| R² goodness-of-fit | 0.823 | 0.934 | 1.000 |
| Mean execution time | 89.8 ms | 93.5 ms | 111 ms |
| Standard deviation | 625 μs | 10.6 ms | 16.9 ms |
Outlying measurements have moderate (31.8%) effect on estimated standard deviation.
serialization (time)/BinTree Direction-packman
3.28 3.30 3.32 3.34 3.36
mean |
1 2 2 3 3 4 4 0.5 iters 5 10 15 0 s 2.5 7.5 12.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.15 s | 3.32 s | 3.45 s |
| R² goodness-of-fit | 0.999 | 1.000 | 1.000 |
| Mean execution time | 3.28 s | 3.31 s | 3.35 s |
| Standard deviation | 16.5 ms | 38.6 ms | 51.0 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
serialization (time)/BinTree Direction-serialise
3.47 3.48 3.49 3.5 3.51 3.52 3.53 3.54
mean |
1 2 2 3 3 4 4 0.5 iters 5 10 15 0 s 2.5 7.5 12.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.47 s | 3.56 s | 3.71 s |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 3.47 s | 3.49 s | 3.53 s |
| Standard deviation | 445 μs | 32.7 ms | 39.6 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
deserialization (time)/BinTree Direction-flat
425 450 475 500 525
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5 2.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 429 ms | 511 ms | 660 ms |
| R² goodness-of-fit | 0.984 | 0.989 | 1.000 |
| Mean execution time | 434 ms | 489 ms | 520 ms |
| Standard deviation | 12.7 ms | 52.6 ms | 70.5 ms |
Outlying measurements have moderate (22.9%) effect on estimated standard deviation.
deserialization (time)/BinTree Direction-store
475 500 525 550 575 600 625
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5 2.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 431 ms | 566 ms | 750 ms |
| R² goodness-of-fit | 0.979 | 0.984 | 1.000 |
| Mean execution time | 501 ms | 560 ms | 601 ms |
| Standard deviation | 22.8 ms | 57.2 ms | 76.4 ms |
Outlying measurements have moderate (22.7%) effect on estimated standard deviation.
deserialization (time)/BinTree Direction-binary
3.1 3.15 3.2 3.25 3.30 3.35 3.40
mean |
1 2 2 3 3 4 4 0.5 iters 5 10 15 0 s 2.5 7.5 12.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.03 s | 3.27 s | 3.73 s |
| R² goodness-of-fit | 0.995 | 0.998 | 1.000 |
| Mean execution time | 3.15 s | 3.26 s | 3.37 s |
| Standard deviation | 70.0 ms | 128 ms | 179 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
deserialization (time)/BinTree Direction-cereal
1.50 1.55 1.6 1.65 1.70 1.75
mean |
1 2 2 3 3 4 4 0.5 iters 1 2 3 4 5 6 7 8 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.02 s | 1.72 s | 2.52 s |
| R² goodness-of-fit | 0.917 | 0.975 | 1.000 |
| Mean execution time | 1.54 s | 1.66 s | 1.73 s |
| Standard deviation | 58.3 ms | 119 ms | 146 ms |
Outlying measurements have moderate (20.2%) effect on estimated standard deviation.
deserialization (time)/BinTree Direction-persist
450 475 500 525 550 575
mean |
1 2 2 3 3 4 4 0.5 iters 2 0 s 500 ms 1 s 1.5 2.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 409 ms | 516 ms | 692 ms |
| R² goodness-of-fit | 0.968 | 0.987 | 1.000 |
| Mean execution time | 455 ms | 505 ms | 539 ms |
| Standard deviation | 26.0 ms | 52.3 ms | 73.5 ms |
Outlying measurements have moderate (22.7%) effect on estimated standard deviation.
deserialization (time)/BinTree Direction-packman
4 3.85 3.90 3.95 4.05 4.10 4.15 4.2 4.25
mean |
1 2 2 3 3 4 4 0.5 iters 5 10 15 20 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.49 s | 3.82 s | 4.08 s |
| R² goodness-of-fit | 0.997 | 0.999 | 1.000 |
| Mean execution time | 3.89 s | 3.99 s | 4.18 s |
| Standard deviation | 16.6 ms | 176 ms | 222 ms |
Outlying measurements have moderate (18.7%) effect on estimated standard deviation.
deserialization (time)/BinTree Direction-serialise
3.7 3.75 3.80 3.85 3.90
mean |
1 2 2 3 3 4 4 0.5 iters 5 10 15 20 0 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.58 s | 3.65 s | 3.72 s |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 3.73 s | 3.79 s | 3.88 s |
| Standard deviation | 25.6 ms | 89.0 ms | 119 ms |
Outlying measurements have moderate (18.8%) effect on estimated standard deviation.
serialization (time)/Cars-flat
6 5.5 5.60 5.7 5.80 5.9 6.10 6.2 6.30
mean |
20 30 40 50 10 iters 100 150 200 250 300 350 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 5.56 ms | 5.68 ms | 5.93 ms |
| R² goodness-of-fit | 0.992 | 0.996 | 1.000 |
| Mean execution time | 5.57 ms | 5.59 ms | 5.69 ms |
| Standard deviation | 14.1 μs | 120 μs | 269 μs |
Outlying measurements have slight (7.2%) effect on estimated standard deviation.
serialization (time)/Cars-store
4 6 8 10 12
mean |
20 30 40 50 60 70 10 iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.18 ms | 4.28 ms | 5.40 ms |
| R² goodness-of-fit | 0.517 | 0.665 | 0.834 |
| Mean execution time | 3.26 ms | 3.68 ms | 4.50 ms |
| Standard deviation | 853 μs | 1.77 ms | 3.08 ms |
Outlying measurements have severe (97.7%) effect on estimated standard deviation.
serialization (time)/Cars-binary
38 39 40 37.5 38.5 39.5
mean |
5 8 10 13 15 2.5 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 37.4 ms | 37.6 ms | 38.1 ms |
| R² goodness-of-fit | 0.997 | 0.999 | 1.000 |
| Mean execution time | 37.8 ms | 37.9 ms | 38.5 ms |
| Standard deviation | 246 μs | 583 μs | 994 μs |
Outlying measurements have slight (6.2%) effect on estimated standard deviation.
serialization (time)/Cars-cereal
20 25 30 17.5 22.5 27.5
mean |
10 15 20 5 iters 200 300 400 500 600 700 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 17.0 ms | 20.7 ms | 25.7 ms |
| R² goodness-of-fit | 0.729 | 0.838 | 1.000 |
| Mean execution time | 17.1 ms | 18.2 ms | 20.3 ms |
| Standard deviation | 124 μs | 3.55 ms | 6.09 ms |
Outlying measurements have severe (76.2%) effect on estimated standard deviation.
serialization (time)/Cars-persist
6 8 10 12 14 16
mean |
10 15 20 25 30 35 40 5 iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 6.77 ms | 7.82 ms | 9.91 ms |
| R² goodness-of-fit | 0.633 | 0.762 | 1.000 |
| Mean execution time | 6.76 ms | 7.27 ms | 8.29 ms |
| Standard deviation | 104 μs | 2.01 ms | 3.67 ms |
Outlying measurements have severe (90.9%) effect on estimated standard deviation.
serialization (time)/Cars-packman
30 35 40 45 50 55 60
mean |
5 8 10 13 15 2.5 iters 200 300 400 500 600 700 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 29.4 ms | 38.0 ms | 49.7 ms |
| R² goodness-of-fit | 0.585 | 0.799 | 0.994 |
| Mean execution time | 32.6 ms | 35.6 ms | 41.8 ms |
| Standard deviation | 4.46 ms | 8.73 ms | 13.7 ms |
Outlying measurements have severe (78.9%) effect on estimated standard deviation.
serialization (time)/Cars-serialise
25 30 35 22.5 27.5 32.5
mean |
10 15 20 5 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 21.5 ms | 23.7 ms | 29.1 ms |
| R² goodness-of-fit | 0.809 | 0.898 | 1.000 |
| Mean execution time | 21.5 ms | 22.2 ms | 24.9 ms |
| Standard deviation | 55.3 μs | 2.95 ms | 5.82 ms |
Outlying measurements have severe (58.4%) effect on estimated standard deviation.
deserialization (time)/Cars-flat
20 25 30 35 40 45
mean |
5 8 10 13 15 18 2.5 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 19.5 ms | 26.0 ms | 33.5 ms |
| R² goodness-of-fit | 0.522 | 0.717 | 0.884 |
| Mean execution time | 21.3 ms | 24.4 ms | 29.8 ms |
| Standard deviation | 5.92 ms | 8.76 ms | 12.4 ms |
Outlying measurements have severe (94.2%) effect on estimated standard deviation.
deserialization (time)/Cars-store
20 25 30 35 40 45
mean |
5 8 10 13 15 18 2.5 iters 200 300 400 500 600 700 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 22.2 ms | 30.0 ms | 36.7 ms |
| R² goodness-of-fit | 0.580 | 0.786 | 0.911 |
| Mean execution time | 23.1 ms | 25.7 ms | 30.8 ms |
| Standard deviation | 4.82 ms | 7.72 ms | 11.4 ms |
Outlying measurements have severe (88.1%) effect on estimated standard deviation.
deserialization (time)/Cars-binary
110 120 130 140 150 160
mean |
2 3 4 5 6 7 8 1 iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 125 ms | 154 ms | 177 ms |
| R² goodness-of-fit | 0.836 | 0.955 | 0.998 |
| Mean execution time | 119 ms | 129 ms | 146 ms |
| Standard deviation | 15.5 ms | 20.3 ms | 27.5 ms |
Outlying measurements have moderate (48.0%) effect on estimated standard deviation.
deserialization (time)/Cars-cereal
45 50 55 60 65 70 75 80 85
mean |
4 6 8 10 12 2 iters 400 600 800 0 s 200 ms 1 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 47.3 ms | 62.9 ms | 76.3 ms |
| R² goodness-of-fit | 0.639 | 0.855 | 0.962 |
| Mean execution time | 49.7 ms | 55.6 ms | 64.6 ms |
| Standard deviation | 9.82 ms | 13.7 ms | 18.1 ms |
Outlying measurements have severe (73.8%) effect on estimated standard deviation.
deserialization (time)/Cars-persist
20 25 30 35 40 45
mean |
5 8 10 13 15 18 2.5 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 20.4 ms | 24.8 ms | 31.8 ms |
| R² goodness-of-fit | 0.574 | 0.756 | 1.000 |
| Mean execution time | 21.4 ms | 23.5 ms | 27.8 ms |
| Standard deviation | 3.70 ms | 6.68 ms | 10.2 ms |
Outlying measurements have severe (88.7%) effect on estimated standard deviation.
deserialization (time)/Cars-packman
40 50 60 70 80
mean |
4 6 8 10 12 2 iters 400 600 800 0 s 200 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 48.7 ms | 57.5 ms | 64.9 ms |
| R² goodness-of-fit | 0.650 | 0.879 | 0.974 |
| Mean execution time | 42.6 ms | 48.9 ms | 58.5 ms |
| Standard deviation | 10.2 ms | 14.6 ms | 21.7 ms |
Outlying measurements have severe (83.7%) effect on estimated standard deviation.
deserialization (time)/Cars-serialise
137 138 138 139
mean |
2 3 4 5 6 7 8 1 iters 500 750 0 s 250 ms 1 s 1.25
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 138 ms | 138 ms | 139 ms |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 137 ms | 138 ms | 138 ms |
| Standard deviation | 348 μs | 500 μs | 765 μs |
Outlying measurements have moderate (10.9%) effect on estimated standard deviation.
serialization (time)/Iris-flat
10 15 20 25
mean |
10 15 20 25 30 5 iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 8.54 ms | 11.6 ms | 15.5 ms |
| R² goodness-of-fit | 0.377 | 0.580 | 0.770 |
| Mean execution time | 9.17 ms | 10.9 ms | 13.5 ms |
| Standard deviation | 3.59 ms | 5.47 ms | 8.07 ms |
Outlying measurements have severe (96.2%) effect on estimated standard deviation.
serialization (time)/Iris-store
2 1.95 2.05 2.1
mean |
40 60 80 100 120 20 iters 100 150 200 250 300 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 1.93 ms | 1.97 ms | 2.01 ms |
| R² goodness-of-fit | 0.996 | 0.998 | 1.000 |
| Mean execution time | 1.93 ms | 1.95 ms | 1.97 ms |
| Standard deviation | 29.1 μs | 49.5 μs | 72.6 μs |
Outlying measurements have moderate (13.0%) effect on estimated standard deviation.
serialization (time)/Iris-binary
160 170 180 190 200 210
mean |
2 3 4 5 6 1 iters 400 600 800 0 s 200 ms 1 s 1.2
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 161 ms | 183 ms | 240 ms |
| R² goodness-of-fit | 0.842 | 0.948 | 1.000 |
| Mean execution time | 163 ms | 171 ms | 192 ms |
| Standard deviation | 1.51 ms | 17.7 ms | 24.7 ms |
Outlying measurements have moderate (30.6%) effect on estimated standard deviation.
serialization (time)/Iris-cereal
30 35 40 45 50 55
mean |
5 8 10 13 15 2.5 iters 200 300 400 500 600 700 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 29.6 ms | 38.8 ms | 47.3 ms |
| R² goodness-of-fit | 0.648 | 0.824 | 0.942 |
| Mean execution time | 31.2 ms | 34.5 ms | 40.3 ms |
| Standard deviation | 5.84 ms | 9.03 ms | 13.1 ms |
Outlying measurements have severe (86.4%) effect on estimated standard deviation.
serialization (time)/Iris-persist
60 65 70 75 80
mean |
4 6 8 10 2 iters 400 600 800 0 s 200 ms 1 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 61.9 ms | 71.8 ms | 85.7 ms |
| R² goodness-of-fit | 0.909 | 0.947 | 1.000 |
| Mean execution time | 62.1 ms | 63.9 ms | 71.0 ms |
| Standard deviation | 191 μs | 5.88 ms | 10.2 ms |
Outlying measurements have moderate (26.0%) effect on estimated standard deviation.
serialization (time)/Iris-packman
50 55 60 65 70 75
mean |
4 6 8 10 12 2 iters 400 600 800 0 s 200 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 53.4 ms | 64.1 ms | 76.5 ms |
| R² goodness-of-fit | 0.836 | 0.917 | 0.994 |
| Mean execution time | 51.7 ms | 54.9 ms | 62.0 ms |
| Standard deviation | 2.38 ms | 8.52 ms | 11.7 ms |
Outlying measurements have severe (56.5%) effect on estimated standard deviation.
serialization (time)/Iris-serialise
15 20 25 30 35 40 45 50
mean |
5 8 10 13 15 18 2.5 iters 200 300 400 500 600 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 19.6 ms | 26.6 ms | 33.5 ms |
| R² goodness-of-fit | 0.499 | 0.722 | 0.886 |
| Mean execution time | 20.3 ms | 23.6 ms | 28.9 ms |
| Standard deviation | 6.38 ms | 9.12 ms | 13.1 ms |
Outlying measurements have severe (94.2%) effect on estimated standard deviation.
deserialization (time)/Iris-flat
40 50 60 70 80
mean |
4 6 8 10 12 2 iters 200 300 400 500 600 700 800 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 45.8 ms | 56.2 ms | 65.0 ms |
| R² goodness-of-fit | 0.635 | 0.864 | 0.961 |
| Mean execution time | 41.6 ms | 48.1 ms | 58.4 ms |
| Standard deviation | 11.0 ms | 14.7 ms | 22.8 ms |
Outlying measurements have severe (83.7%) effect on estimated standard deviation.
deserialization (time)/Iris-store
30 35 40 45 50 55 60 65 70
mean |
5 8 10 13 15 2.5 iters 200 300 400 500 600 700 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 30.9 ms | 41.9 ms | 52.9 ms |
| R² goodness-of-fit | 0.529 | 0.775 | 0.935 |
| Mean execution time | 33.9 ms | 39.6 ms | 47.7 ms |
| Standard deviation | 10.3 ms | 13.1 ms | 18.1 ms |
Outlying measurements have severe (92.5%) effect on estimated standard deviation.
deserialization (time)/Iris-binary
280 290 300 310 320 330
mean |
2 3 4 5 1 iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 285 ms | 332 ms | 361 ms |
| R² goodness-of-fit | 0.958 | 0.984 | 1.000 |
| Mean execution time | 286 ms | 300 ms | 318 ms |
| Standard deviation | 13.5 ms | 20.7 ms | 25.7 ms |
Outlying measurements have moderate (17.4%) effect on estimated standard deviation.
deserialization (time)/Iris-cereal
100 120 140 160 180
mean |
2 3 4 5 6 7 8 1 iters 500 750 0 s 250 ms 1 s 1.25
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 109 ms | 141 ms | 182 ms |
| R² goodness-of-fit | 0.770 | 0.907 | 0.983 |
| Mean execution time | 108 ms | 126 ms | 147 ms |
| Standard deviation | 22.3 ms | 30.5 ms | 40.1 ms |
Outlying measurements have severe (73.1%) effect on estimated standard deviation.
deserialization (time)/Iris-persist
35 40 45 50 55 60 65 70 75
mean |
4 6 8 10 12 2 iters 200 300 400 500 600 700 800 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 39.1 ms | 52.0 ms | 64.2 ms |
| R² goodness-of-fit | 0.567 | 0.814 | 0.960 |
| Mean execution time | 39.8 ms | 45.7 ms | 54.5 ms |
| Standard deviation | 10.6 ms | 13.7 ms | 18.5 ms |
Outlying measurements have severe (83.7%) effect on estimated standard deviation.
deserialization (time)/Iris-packman
70 80 90 100 110 120 130
mean |
2 3 4 5 6 7 8 9 1 iters 400 600 800 0 s 200 ms 1 s
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 83.4 ms | 101 ms | 119 ms |
| R² goodness-of-fit | 0.747 | 0.925 | 0.990 |
| Mean execution time | 77.6 ms | 88.4 ms | 105 ms |
| Standard deviation | 16.4 ms | 21.5 ms | 30.8 ms |
Outlying measurements have severe (76.0%) effect on estimated standard deviation.
deserialization (time)/Iris-serialise
100 110 120 130 140
mean |
2 3 4 5 6 7 8 9 1 iters 400 600 800 0 s 200 ms 1 s 1.2
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 96.0 ms | 122 ms | 137 ms |
| R² goodness-of-fit | 0.789 | 0.938 | 1.000 |
| Mean execution time | 98.3 ms | 106 ms | 119 ms |
| Standard deviation | 11.2 ms | 17.2 ms | 22.7 ms |
Outlying measurements have severe (53.5%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.