criterion performance measurements
overview
want to understand this report?
| Right-assoc | |
| Right-assoc/Free | |
| Right-assoc | |
| Left-assoc | |
| Left-assoc/Free | |
| Left-assoc |
Right-assoc/Free
4.45 4.5 4.55 4.60 4.65 4.7 4.75 4.80
mean |
20 30 40 50 10×10³ iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 4.49 μs | 4.50 μs | 4.51 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 4.50 μs | 4.51 μs | 4.53 μs |
| Standard deviation | 26.8 ns | 53.4 ns | 100 ns |
Outlying measurements have slight (8.4%) effect on estimated standard deviation.
Right-assoc/Free/lazy
4.45 4.47 4.5 4.52 4.55 4.58 4.6 4.63
mean |
20 30 40 50 10×10³ iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 4.48 μs | 4.49 μs | 4.50 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 4.48 μs | 4.49 μs | 4.50 μs |
| Standard deviation | 17.9 ns | 28.8 ns | 52.4 ns |
Outlying measurements have no (0.6%) effect on estimated standard deviation.
Right-assoc/Chruch
8 10 12 14 16
mean |
10 15 20 25 5×10³ iters 200 300 400 500 0 s 100 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 8.13 μs | 9.94 μs | 11.8 μs |
| R² goodness-of-fit | 0.865 | 0.876 | 0.957 |
| Mean execution time | 8.02 μs | 8.65 μs | 9.74 μs |
| Standard deviation | 1.62 μs | 2.50 μs | 3.34 μs |
Outlying measurements have severe (98.7%) effect on estimated standard deviation.
Right-assoc/Codensity
7 8 9 10 11 12 13 14
mean |
10 15 20 25 5×10³ iters 100 150 200 250 300 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 7.25 μs | 8.22 μs | 9.50 μs |
| R² goodness-of-fit | 0.877 | 0.897 | 0.966 |
| Mean execution time | 7.89 μs | 8.55 μs | 9.54 μs |
| Standard deviation | 2.03 μs | 2.62 μs | 3.04 μs |
Outlying measurements have severe (98.7%) effect on estimated standard deviation.
Right-assoc/NoRemorse
8 10 12 14 16
mean |
10 15 20 5×10³ iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 8.47 μs | 8.98 μs | 9.81 μs |
| R² goodness-of-fit | 0.882 | 0.925 | 0.975 |
| Mean execution time | 8.73 μs | 9.37 μs | 10.4 μs |
| Standard deviation | 1.71 μs | 2.65 μs | 3.55 μs |
Outlying measurements have severe (98.1%) effect on estimated standard deviation.
Right-assoc/Freer
3.33 3.35 3.38 3.40 3.43 3.45
mean |
20 30 40 50 60 10×10³ iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 3.35 μs | 3.36 μs | 3.38 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 3.35 μs | 3.35 μs | 3.36 μs |
| Standard deviation | 21.9 ns | 28.4 ns | 38.1 ns |
Outlying measurements have no (0.5%) effect on estimated standard deviation.
Right-assoc/MTL
47 48 46.5 46.8 47.3 47.5 47.8
mean |
2 3 4 1×10⁶ iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 46.7 ns | 46.8 ns | 46.8 ns |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 46.7 ns | 46.8 ns | 46.9 ns |
| Standard deviation | 163 ps | 246 ps | 390 ps |
Outlying measurements have no (0.4%) effect on estimated standard deviation.
Left-assoc/Free
60 61 59.5 60.5 61.5
mean |
1000 1500 2000 2500 3000 3500 500 iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 59.8 μs | 60.0 μs | 60.3 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 59.8 μs | 60.0 μs | 60.1 μs |
| Standard deviation | 372 ns | 477 ns | 705 ns |
Outlying measurements have no (0.8%) effect on estimated standard deviation.
Left-assoc/Free/lazy
60 65 70 75 80 85 90 95 100
mean |
1000 1500 2000 2500 3000 3500 500 iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 62.1 μs | 62.7 μs | 63.7 μs |
| R² goodness-of-fit | 0.987 | 0.994 | 0.999 |
| Mean execution time | 63.0 μs | 64.5 μs | 67.6 μs |
| Standard deviation | 3.26 μs | 6.60 μs | 11.9 μs |
Outlying measurements have severe (83.3%) effect on estimated standard deviation.
Left-assoc/Chruch
8 10 12 14 16 18
mean |
10 15 20 5×10³ iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 8.88 μs | 8.99 μs | 9.11 μs |
| R² goodness-of-fit | 0.971 | 0.983 | 0.991 |
| Mean execution time | 10.3 μs | 11.2 μs | 12.3 μs |
| Standard deviation | 3.01 μs | 3.76 μs | 4.31 μs |
Outlying measurements have severe (98.7%) effect on estimated standard deviation.
Left-assoc/Codensity
8 9 8.5 9.5
mean |
10 15 20 25 5×10³ iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 8.19 μs | 8.42 μs | 8.71 μs |
| R² goodness-of-fit | 0.993 | 0.996 | 0.999 |
| Mean execution time | 8.23 μs | 8.31 μs | 8.45 μs |
| Standard deviation | 273 ns | 357 ns | 569 ns |
Outlying measurements have severe (53.2%) effect on estimated standard deviation.
Left-assoc/NoRemorse
15 20 25 17.5 22.5 27.5
mean |
5 8 10 13 3×10³ iters 100 150 200 250 300 350 400 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 14.6 μs | 16.3 μs | 18.3 μs |
| R² goodness-of-fit | 0.906 | 0.948 | 0.999 |
| Mean execution time | 14.8 μs | 15.3 μs | 16.5 μs |
| Standard deviation | 1.13 μs | 2.26 μs | 4.43 μs |
Outlying measurements have severe (93.1%) effect on estimated standard deviation.
Left-assoc/Freer
20 19.5 19.8 20.3 20.5 20.8
mean |
4 6 8 10 2×10³ iters 100 150 200 250 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 19.8 μs | 19.8 μs | 19.9 μs |
| R² goodness-of-fit | 1.000 | 1.000 | 1.000 |
| Mean execution time | 19.8 μs | 19.9 μs | 20.0 μs |
| Standard deviation | 150 ns | 242 ns | 359 ns |
Outlying measurements have slight (7.6%) effect on estimated standard deviation.
Left-assoc/MTL
140 150 160 170 180 190 200
mean |
500 750 1000 1250 1500 250×10³ iters 100 150 200 250 300 0 s 50 ms
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | 146 ns | 153 ns | 162 ns |
| R² goodness-of-fit | 0.978 | 0.987 | 0.999 |
| Mean execution time | 146 ns | 150 ns | 155 ns |
| Standard deviation | 7.71 ns | 13.5 ns | 19.7 ns |
Outlying measurements have severe (88.6%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.