criterion performance measurements
overview
want to understand this report?
Right-assoc/Free
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.496309671764716e-6 | 4.507848454887979e-6 | 4.532996548202542e-6 |
| Standard deviation | 2.6759443526443448e-8 | 5.337630939081432e-8 | 1.0038574670532733e-7 |
Outlying measurements have slight (8.399691638860637e-2%) effect on estimated standard deviation.
Right-assoc/Free/lazy
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.4828828275670695e-6 | 4.489391899068001e-6 | 4.5019045096494305e-6 |
| Standard deviation | 1.7853120124437838e-8 | 2.8832230927589546e-8 | 5.2372744137404136e-8 |
Outlying measurements have no (5.5246913580247635e-3%) effect on estimated standard deviation.
Right-assoc/Chruch
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.024927614937138e-6 | 8.646621273868009e-6 | 9.740545664222338e-6 |
| Standard deviation | 1.6230587205941088e-6 | 2.5034314028830486e-6 | 3.3442610076778144e-6 |
Outlying measurements have severe (0.9868558213949039%) effect on estimated standard deviation.
Right-assoc/Codensity
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.88665651016658e-6 | 8.552365035027449e-6 | 9.535830133156762e-6 |
| Standard deviation | 2.02996938880265e-6 | 2.6167378092226254e-6 | 3.0449616592244426e-6 |
Outlying measurements have severe (0.9871252634181289%) effect on estimated standard deviation.
Right-assoc/NoRemorse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.726906684022631e-6 | 9.370788315510773e-6 | 1.0396227801151339e-5 |
| Standard deviation | 1.7118527545468777e-6 | 2.6511749472714766e-6 | 3.5465157306229867e-6 |
Outlying measurements have severe (0.9806777385057214%) effect on estimated standard deviation.
Right-assoc/Freer
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.347096688145364e-6 | 3.354442456053234e-6 | 3.3642445640258957e-6 |
| Standard deviation | 2.18945710459088e-8 | 2.8440163423315714e-8 | 3.8092752138884746e-8 |
Outlying measurements have no (5.347439010289873e-3%) effect on estimated standard deviation.
Right-assoc/MTL
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.674918005964803e-8 | 4.680548243816792e-8 | 4.689783307630922e-8 |
| Standard deviation | 1.6342461876864283e-10 | 2.464523602690172e-10 | 3.901526202788249e-10 |
Outlying measurements have no (3.6363152005965548e-3%) effect on estimated standard deviation.
Left-assoc/Free
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.984055044472469e-5 | 5.99611989733279e-5 | 6.012953562669336e-5 |
| Standard deviation | 3.72085977809039e-7 | 4.771339552874426e-7 | 7.054233996212338e-7 |
Outlying measurements have no (7.812015624031212e-3%) effect on estimated standard deviation.
Left-assoc/Free/lazy
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.304665307112526e-5 | 6.447256632868442e-5 | 6.758368158728052e-5 |
| Standard deviation | 3.2609786597301568e-6 | 6.6016800276629984e-6 | 1.185743620557848e-5 |
Outlying measurements have severe (0.8328991544321732%) effect on estimated standard deviation.
Left-assoc/Chruch
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0307126731360323e-5 | 1.1208259574436796e-5 | 1.2326305312962383e-5 |
| Standard deviation | 3.0077456441055128e-6 | 3.7615267561962194e-6 | 4.305065419998736e-6 |
Outlying measurements have severe (0.9869794740645862%) effect on estimated standard deviation.
Left-assoc/Codensity
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.229454464063152e-6 | 8.313597730640533e-6 | 8.45052390857398e-6 |
| Standard deviation | 2.727872345330792e-7 | 3.5656568522970373e-7 | 5.689081113607044e-7 |
Outlying measurements have severe (0.5324566892509678%) effect on estimated standard deviation.
Left-assoc/NoRemorse
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4811040419301926e-5 | 1.5274647439591986e-5 | 1.6518303909670677e-5 |
| Standard deviation | 1.129448486058532e-6 | 2.2582392129478466e-6 | 4.426225113789385e-6 |
Outlying measurements have severe (0.9314477568505284%) effect on estimated standard deviation.
Left-assoc/Freer
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.980512338910006e-5 | 1.9861163771887065e-5 | 1.995436280240223e-5 |
| Standard deviation | 1.4961764459586009e-7 | 2.421762812219432e-7 | 3.587245186836761e-7 |
Outlying measurements have slight (7.55161103701509e-2%) effect on estimated standard deviation.
Left-assoc/MTL
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4647576147537096e-7 | 1.495587764691872e-7 | 1.5482174355112384e-7 |
| Standard deviation | 7.706863224499795e-9 | 1.3465261036996903e-8 | 1.974145879670133e-8 |
Outlying measurements have severe (0.885865149567117%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.