criterion performance measurements
overview
want to understand this report?
| teaspoon | |
| const (control) | |
| Just (control) | |
| justToJust (control) | |
| catchMethod1 | |
| catchMethod2 | |
| catchMethod3 | |
| catchMethod4 | |
| catchMethod5 | |
| catchMethod6 | |
| primspoon | |
| throwsMethod0 | |
| throwsMethod1 | |
| throwsMethod2 | |
| throwsMethod3 | |
| throwsMethod4 | |
| throwsMethod5 | |
| throws |
teaspoon /Check undefined
130 140 150 160 170 180 190 200
mean |
2 3 4 5 6 7 8 9 1×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 146 ns | 148 ns | 151 ns |
| R² goodness-of-fit | 0.997 | 0.998 | 0.999 |
| Mean execution time | 149 ns | 151 ns | 154 ns |
| Standard deviation | 9.52 ns | 11.2 ns | 15.0 ns |
Outlying measurements have severe (85.7%) effect on estimated standard deviation.
teaspoon /Check defined
22 23 24 25 26 27 28
mean |
20 30 40 50 10×10⁶ iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 23.9 ns | 24.4 ns | 24.9 ns |
| R² goodness-of-fit | 0.997 | 0.998 | 0.999 |
| Mean execution time | 24.1 ns | 24.4 ns | 24.8 ns |
| Standard deviation | 1.24 ns | 1.45 ns | 1.69 ns |
Outlying measurements have severe (81.2%) effect on estimated standard deviation.
const (control) /Check undefined
7 8 6.5 7.50 8.5
mean |
100 150 200 50×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 6.86 ns | 6.92 ns | 6.99 ns |
| R² goodness-of-fit | 0.999 | 0.999 | 0.999 |
| Mean execution time | 7.03 ns | 7.11 ns | 7.19 ns |
| Standard deviation | 321 ps | 384 ps | 531 ps |
Outlying measurements have severe (79.3%) effect on estimated standard deviation.
const (control) /Check defined
7 8 6.5 7.50 8.5
mean |
100 150 50×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 7.10 ns | 7.19 ns | 7.30 ns |
| R² goodness-of-fit | 0.998 | 0.999 | 0.999 |
| Mean execution time | 7.13 ns | 7.21 ns | 7.30 ns |
| Standard deviation | 337 ps | 397 ps | 504 ps |
Outlying measurements have severe (79.8%) effect on estimated standard deviation.
Just (control) /Check defined
7 8 9 10 6.5 7.50 8.5 9.5
mean |
50 75 100 125 150 175 25×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 7.45 ns | 7.66 ns | 7.86 ns |
| R² goodness-of-fit | 0.993 | 0.996 | 0.998 |
| Mean execution time | 7.65 ns | 7.78 ns | 7.93 ns |
| Standard deviation | 491 ps | 602 ps | 785 ps |
Outlying measurements have severe (88.8%) effect on estimated standard deviation.
justToJust (control)/check Nothing
8 9 10 11 7.50 8.5 9.5 10.5
mean |
50 75 100 125 150 25×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 8.09 ns | 8.22 ns | 8.38 ns |
| R² goodness-of-fit | 0.998 | 0.998 | 0.999 |
| Mean execution time | 8.31 ns | 8.42 ns | 8.55 ns |
| Standard deviation | 431 ps | 531 ps | 742 ps |
Outlying measurements have severe (83.9%) effect on estimated standard deviation.
justToJust (control)/Check Just
7 8 9 10 7.50 8.5 9.5
mean |
50 75 100 125 150 25×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 7.74 ns | 7.82 ns | 7.95 ns |
| R² goodness-of-fit | 0.996 | 0.998 | 0.999 |
| Mean execution time | 7.74 ns | 7.87 ns | 8.00 ns |
| Standard deviation | 522 ps | 604 ps | 762 ps |
Outlying measurements have severe (88.4%) effect on estimated standard deviation.
catchMethod1 /Check undefined
31 32 33 34 35 36 37 38 30.0
mean |
10 15 20 25 30 35 40 5×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 33.6 ns | 34.2 ns | 35.0 ns |
| R² goodness-of-fit | 0.997 | 0.998 | 0.999 |
| Mean execution time | 33.6 ns | 34.0 ns | 34.4 ns |
| Standard deviation | 1.61 ns | 1.84 ns | 2.10 ns |
Outlying measurements have severe (77.8%) effect on estimated standard deviation.
catchMethod1 /Check defined
14 16 18 20 22 24
mean |
20 30 40 50 60 70 80 10×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 15.6 ns | 15.8 ns | 16.0 ns |
| R² goodness-of-fit | 0.996 | 0.998 | 0.999 |
| Mean execution time | 16.1 ns | 16.4 ns | 16.8 ns |
| Standard deviation | 1.23 ns | 1.55 ns | 2.33 ns |
Outlying measurements have severe (91.7%) effect on estimated standard deviation.
catchMethod2 /Check undefined
32 34 36 38 30.0
mean |
10 15 20 25 30 35 40 5×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 33.1 ns | 33.6 ns | 34.0 ns |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 33.0 ns | 33.3 ns | 33.6 ns |
| Standard deviation | 1.32 ns | 1.54 ns | 2.01 ns |
Outlying measurements have severe (71.7%) effect on estimated standard deviation.
catchMethod2 /Check defined
14 16 17 18 15.0
mean |
20 30 40 50 60 70 80 90 10×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 15.2 ns | 15.4 ns | 15.5 ns |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 15.2 ns | 15.4 ns | 15.6 ns |
| Standard deviation | 694 ps | 822 ps | 1.01 ns |
Outlying measurements have severe (78.0%) effect on estimated standard deviation.
catchMethod3 /Check undefined
35 40 30.0 32.5 37.5 42.5
mean |
10 15 20 25 30 35 40 5×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 32.9 ns | 33.3 ns | 33.8 ns |
| R² goodness-of-fit | 0.998 | 0.999 | 0.999 |
| Mean execution time | 33.2 ns | 33.6 ns | 34.1 ns |
| Standard deviation | 1.86 ns | 2.20 ns | 3.15 ns |
Outlying measurements have severe (83.8%) effect on estimated standard deviation.
catchMethod3 /Check defined
14 16 17 18 14.5 15.0 15.5 16.5 17.5
mean |
20 30 40 50 60 70 80 10×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 15.6 ns | 16.1 ns | 16.5 ns |
| R² goodness-of-fit | 0.994 | 0.995 | 0.997 |
| Mean execution time | 15.1 ns | 15.3 ns | 15.5 ns |
| Standard deviation | 838 ps | 964 ps | 1.14 ns |
Outlying measurements have severe (83.2%) effect on estimated standard deviation.
catchMethod4 /Check undefined
32 34 36 38 40 30.0
mean |
10 15 20 25 30 35 40 5×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 34.4 ns | 35.0 ns | 35.9 ns |
| R² goodness-of-fit | 0.994 | 0.996 | 0.997 |
| Mean execution time | 33.7 ns | 34.2 ns | 34.9 ns |
| Standard deviation | 2.35 ns | 2.69 ns | 3.04 ns |
Outlying measurements have severe (88.0%) effect on estimated standard deviation.
catchMethod4 /Check defined
14 16 17 14.5 15.0 15.5 16.5 17.5
mean |
20 30 40 50 60 70 80 90 10×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 15.1 ns | 15.3 ns | 15.5 ns |
| R² goodness-of-fit | 0.998 | 0.999 | 0.999 |
| Mean execution time | 15.1 ns | 15.2 ns | 15.4 ns |
| Standard deviation | 639 ps | 748 ps | 907 ps |
Outlying measurements have severe (74.9%) effect on estimated standard deviation.
catchMethod5 /Check undefined
35 40 45 30.0 32.5 37.5 42.5
mean |
10 15 20 25 30 35 40 5×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 32.8 ns | 33.3 ns | 33.8 ns |
| R² goodness-of-fit | 0.998 | 0.999 | 0.999 |
| Mean execution time | 33.2 ns | 33.6 ns | 34.3 ns |
| Standard deviation | 1.82 ns | 2.35 ns | 3.48 ns |
Outlying measurements have severe (85.4%) effect on estimated standard deviation.
catchMethod5 /Check defined
14 16 17 18 19 20 15.0
mean |
20 30 40 50 60 70 80 10×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 15.6 ns | 16.0 ns | 16.6 ns |
| R² goodness-of-fit | 0.990 | 0.993 | 0.997 |
| Mean execution time | 15.2 ns | 15.5 ns | 15.8 ns |
| Standard deviation | 1.03 ns | 1.24 ns | 1.52 ns |
Outlying measurements have severe (88.9%) effect on estimated standard deviation.
catchMethod6 /Check undefined
28 32 34 36 38 40 30.0
mean |
10 15 20 25 30 35 40 5×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 31.0 ns | 31.4 ns | 31.8 ns |
| R² goodness-of-fit | 0.999 | 0.999 | 0.999 |
| Mean execution time | 31.4 ns | 31.8 ns | 32.2 ns |
| Standard deviation | 1.66 ns | 1.97 ns | 2.51 ns |
Outlying measurements have severe (82.2%) effect on estimated standard deviation.
catchMethod6 /Check defined
28 32 34 36 38 30.0
mean |
10 15 20 25 30 35 40 5×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 31.5 ns | 32.2 ns | 33.2 ns |
| R² goodness-of-fit | 0.993 | 0.995 | 0.997 |
| Mean execution time | 31.8 ns | 32.3 ns | 32.7 ns |
| Standard deviation | 2.02 ns | 2.30 ns | 2.65 ns |
Outlying measurements have severe (86.1%) effect on estimated standard deviation.
primspoon /Check undefined
35 40 45 30.0
mean |
10 15 20 25 30 35 40 5×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 32.4 ns | 32.8 ns | 33.1 ns |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 32.4 ns | 32.9 ns | 33.5 ns |
| Standard deviation | 1.92 ns | 2.57 ns | 3.82 ns |
Outlying measurements have severe (88.0%) effect on estimated standard deviation.
primspoon /Check defined
14 16 17 18 15.0
mean |
20 30 40 50 60 70 80 90 10×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 14.9 ns | 15.1 ns | 15.3 ns |
| R² goodness-of-fit | 0.998 | 0.999 | 0.999 |
| Mean execution time | 15.1 ns | 15.3 ns | 15.5 ns |
| Standard deviation | 859 ps | 1.01 ns | 1.27 ns |
Outlying measurements have severe (84.5%) effect on estimated standard deviation.
throwsMethod0 /Check undefined
35 40 30.0 32.5 37.5 42.5
mean |
10 15 20 25 30 35 40 5×10⁶ iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 32.5 ns | 33.2 ns | 33.7 ns |
| R² goodness-of-fit | 0.998 | 0.998 | 0.999 |
| Mean execution time | 32.4 ns | 32.8 ns | 33.4 ns |
| Standard deviation | 1.74 ns | 2.15 ns | 3.06 ns |
Outlying measurements have severe (83.8%) effect on estimated standard deviation.
throwsMethod0 /Check defined
14 16 17 18 19 20 21 22 15.0
mean |
20 30 40 50 60 70 80 10×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 15.6 ns | 15.8 ns | 16.0 ns |
| R² goodness-of-fit | 0.999 | 0.999 | 0.999 |
| Mean execution time | 15.5 ns | 15.7 ns | 16.0 ns |
| Standard deviation | 873 ps | 1.15 ns | 1.72 ns |
Outlying measurements have severe (87.3%) effect on estimated standard deviation.
throwsMethod1 /Check undefined
32 34 36 38 40 30.0
mean |
10 15 20 25 30 35 40 5×10⁶ iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 31.9 ns | 32.7 ns | 33.7 ns |
| R² goodness-of-fit | 0.994 | 0.997 | 0.999 |
| Mean execution time | 32.2 ns | 32.6 ns | 33.2 ns |
| Standard deviation | 1.86 ns | 2.24 ns | 2.92 ns |
Outlying measurements have severe (85.1%) effect on estimated standard deviation.
throwsMethod1 /Check defined
13 14 16 17 18 19 15.0
mean |
20 30 40 50 60 70 80 90 10×10⁶ iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 14.5 ns | 14.9 ns | 15.4 ns |
| R² goodness-of-fit | 0.995 | 0.996 | 0.999 |
| Mean execution time | 14.4 ns | 14.5 ns | 14.8 ns |
| Standard deviation | 721 ps | 939 ps | 1.42 ns |
Outlying measurements have severe (83.9%) effect on estimated standard deviation.
throwsMethod2 /Check undefined
35 40 30.0 32.5 37.5 42.5
mean |
10 15 20 25 30 35 40 5×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 32.1 ns | 32.6 ns | 33.2 ns |
| R² goodness-of-fit | 0.997 | 0.998 | 0.999 |
| Mean execution time | 32.2 ns | 32.7 ns | 33.3 ns |
| Standard deviation | 2.04 ns | 2.42 ns | 3.19 ns |
Outlying measurements have severe (86.7%) effect on estimated standard deviation.
throwsMethod2 /Check defined
13 14 16 17 18 15.0
mean |
20 30 40 50 60 70 80 90 10×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 14.6 ns | 14.7 ns | 14.9 ns |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 14.7 ns | 14.9 ns | 15.2 ns |
| Standard deviation | 873 ps | 1.06 ns | 1.33 ns |
Outlying measurements have severe (86.4%) effect on estimated standard deviation.
throwsMethod3 /Check undefined
32 34 36 38 30.0
mean |
10 15 20 25 30 35 40 5×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 31.6 ns | 31.9 ns | 32.3 ns |
| R² goodness-of-fit | 0.999 | 0.999 | 0.999 |
| Mean execution time | 31.9 ns | 32.3 ns | 32.7 ns |
| Standard deviation | 1.60 ns | 1.89 ns | 2.21 ns |
Outlying measurements have severe (80.5%) effect on estimated standard deviation.
throwsMethod3 /Check defined
14 16 17 18 15.0
mean |
20 30 40 50 60 70 80 90 10×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 14.9 ns | 15.2 ns | 15.4 ns |
| R² goodness-of-fit | 0.997 | 0.998 | 0.999 |
| Mean execution time | 14.6 ns | 14.8 ns | 15.0 ns |
| Standard deviation | 695 ps | 834 ps | 1.10 ns |
Outlying measurements have severe (79.9%) effect on estimated standard deviation.
throwsMethod4 /Check undefined
35 40 45 30.0
mean |
10 15 20 25 30 35 40 5×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 32.0 ns | 32.3 ns | 32.7 ns |
| R² goodness-of-fit | 0.997 | 0.998 | 0.999 |
| Mean execution time | 34.1 ns | 34.9 ns | 35.7 ns |
| Standard deviation | 3.39 ns | 3.86 ns | 4.66 ns |
Outlying measurements have severe (93.7%) effect on estimated standard deviation.
throwsMethod4 /Check defined
13 14 16 17 15.0
mean |
20 30 40 50 60 70 80 90 10×10⁶ iters 2 0 s 500 ms 1 s 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 14.4 ns | 15.1 ns | 15.8 ns |
| R² goodness-of-fit | 0.990 | 0.992 | 0.999 |
| Mean execution time | 14.5 ns | 14.6 ns | 14.9 ns |
| Standard deviation | 799 ps | 941 ps | 1.13 ns |
Outlying measurements have severe (83.9%) effect on estimated standard deviation.
throwsMethod5 /Check undefined
35 40 45 30.0 32.5 37.5 42.5
mean |
10 15 20 25 30 35 40 5×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 32.5 ns | 32.9 ns | 33.2 ns |
| R² goodness-of-fit | 0.999 | 0.999 | 0.999 |
| Mean execution time | 32.4 ns | 32.8 ns | 33.4 ns |
| Standard deviation | 1.77 ns | 2.39 ns | 3.95 ns |
Outlying measurements have severe (86.4%) effect on estimated standard deviation.
throwsMethod5 /Check defined
14 16 17 18 15.0
mean |
20 30 40 50 60 70 80 90 10×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 14.7 ns | 14.9 ns | 15.1 ns |
| R² goodness-of-fit | 0.996 | 0.998 | 0.999 |
| Mean execution time | 14.5 ns | 14.8 ns | 15.0 ns |
| Standard deviation | 822 ps | 1.05 ns | 1.37 ns |
Outlying measurements have severe (86.4%) effect on estimated standard deviation.
throws /Check undefined
32 34 36 38 40 30.0
mean |
10 15 20 25 30 35 40 5×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 31.9 ns | 32.2 ns | 32.5 ns |
| R² goodness-of-fit | 0.999 | 0.999 | 1.000 |
| Mean execution time | 32.3 ns | 32.8 ns | 33.3 ns |
| Standard deviation | 1.78 ns | 2.17 ns | 2.70 ns |
Outlying measurements have severe (84.1%) effect on estimated standard deviation.
throws /Check defined
13 14 16 17 18 15.0
mean |
20 30 40 50 60 70 80 90 10×10⁶ iters 500 750 0 s 250 ms 1 s 1.25 1.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 15.0 ns | 15.4 ns | 15.8 ns |
| R² goodness-of-fit | 0.995 | 0.996 | 0.998 |
| Mean execution time | 14.6 ns | 14.9 ns | 15.2 ns |
| Standard deviation | 1.05 ns | 1.18 ns | 1.36 ns |
Outlying measurements have severe (88.9%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.