criterion performance measurements
overview
want to understand this report?
Environment: AMD Opteron 6168 on Ubuntu 12.04 with GHC 8.0.2
create/relation 10x100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.891481267739451e-4 | 5.906283388143756e-4 | 5.919056866982677e-4 |
| Standard deviation | 3.7867077713250133e-6 | 4.502571301832575e-6 | 5.8062391798041006e-6 |
Outlying measurements have slight (1.2343749999999981e-2%) effect on estimated standard deviation.
create/relation 10x1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.760191573102873e-3 | 5.770838309782314e-3 | 5.779948952798578e-3 |
| Standard deviation | 2.2935863601065537e-5 | 2.8815215559732134e-5 | 3.914144050420156e-5 |
Outlying measurements have slight (2.5623268698060614e-2%) effect on estimated standard deviation.
create/relation 10x10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.772489494946795e-2 | 5.784195729887314e-2 | 5.792298187695035e-2 |
| Standard deviation | 1.0422507194091924e-4 | 1.7121491767487851e-4 | 2.73724491538287e-4 |
Outlying measurements have slight (7.638888888888888e-2%) effect on estimated standard deviation.
restrict/restrict relation 10x10000 to 10x1
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4337830338577313e-2 | 1.448797309139284e-2 | 1.4763601823824296e-2 |
| Standard deviation | 2.772536754203617e-4 | 4.774249175945004e-4 | 8.291091461013828e-4 |
Outlying measurements have moderate (0.11563227985262721%) effect on estimated standard deviation.
restrict/restrict relation 10x10000 to 10x5000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.322941573263729e-2 | 8.447258684075218e-2 | 8.511386931168204e-2 |
| Standard deviation | 6.118316959029327e-4 | 1.4013810341920257e-3 | 2.3016514403696887e-3 |
Outlying measurements have slight (9.0e-2%) effect on estimated standard deviation.
project/project 10x1000 to 1x1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.2211661066456561e-2 | 1.243577405705988e-2 | 1.2583044006972743e-2 |
| Standard deviation | 3.0211460384957275e-4 | 4.600204612133745e-4 | 5.956804020691042e-4 |
Outlying measurements have moderate (0.14345146934193906%) effect on estimated standard deviation.
project/project 10x1000 to 5x1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5271674270594341e-2 | 1.547382196820991e-2 | 1.5622307025724294e-2 |
| Standard deviation | 2.873217405621163e-4 | 3.978143402171812e-4 | 5.247713511789067e-4 |
Outlying measurements have slight (4.1587901701323225e-2%) effect on estimated standard deviation.
union/union identical 10x1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.908323998442862e-2 | 4.923173514046889e-2 | 4.94547578201357e-2 |
| Standard deviation | 2.09615306506087e-4 | 3.275115670368287e-4 | 4.977889379852797e-4 |
Outlying measurements have slight (7.100591715976319e-2%) effect on estimated standard deviation.
union/union identical 10x10000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.5144192261126985 | 0.5157808375258943 | 0.5164000385072457 |
| Standard deviation | 0 | 1.1935851498586624e-3 | 1.285892588023751e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
join/join identical 10x100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3355841931137941 | 0.3374978199984254 | 0.3385756312293291 |
| Standard deviation | 0 | 1.6965376723717264e-3 | 1.8668238128934738e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
group/group 10x100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.666001885635249e-2 | 4.70336206675725e-2 | 4.7476184359486216e-2 |
| Standard deviation | 5.178696454087703e-4 | 7.389605929837668e-4 | 1.1404867684675546e-3 |
Outlying measurements have slight (7.100591715976322e-2%) effect on estimated standard deviation.
write/write 10x1000
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5472953190097016e-3 | 1.5673276303702817e-3 | 1.5890440248897237e-3 |
| Standard deviation | 6.110376965683773e-5 | 6.970009193368111e-5 | 7.954403273020534e-5 |
Outlying measurements have moderate (0.30346203307926556%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.