# criterion performance measurements

## overview

want to understand this report?

Environment: AMD Opteron 6168 on Ubuntu 12.04 with GHC 8.0.2

## create/relation 10x100

lower bound | estimate | upper bound | |
---|---|---|---|

OLS regression | xxx | xxx | xxx |

R² goodness-of-fit | xxx | xxx | xxx |

Mean execution time | 5.891481267739451e-4 | 5.906283388143756e-4 | 5.919056866982677e-4 |

Standard deviation | 3.7867077713250133e-6 | 4.502571301832575e-6 | 5.8062391798041006e-6 |

Outlying measurements have slight (1.2343749999999981e-2%) effect on estimated standard deviation.

## create/relation 10x1000

lower bound | estimate | upper bound | |
---|---|---|---|

OLS regression | xxx | xxx | xxx |

R² goodness-of-fit | xxx | xxx | xxx |

Mean execution time | 5.760191573102873e-3 | 5.770838309782314e-3 | 5.779948952798578e-3 |

Standard deviation | 2.2935863601065537e-5 | 2.8815215559732134e-5 | 3.914144050420156e-5 |

Outlying measurements have slight (2.5623268698060614e-2%) effect on estimated standard deviation.

## create/relation 10x10000

lower bound | estimate | upper bound | |
---|---|---|---|

OLS regression | xxx | xxx | xxx |

R² goodness-of-fit | xxx | xxx | xxx |

Mean execution time | 5.772489494946795e-2 | 5.784195729887314e-2 | 5.792298187695035e-2 |

Standard deviation | 1.0422507194091924e-4 | 1.7121491767487851e-4 | 2.73724491538287e-4 |

Outlying measurements have slight (7.638888888888888e-2%) effect on estimated standard deviation.

## restrict/restrict relation 10x10000 to 10x1

lower bound | estimate | upper bound | |
---|---|---|---|

OLS regression | xxx | xxx | xxx |

R² goodness-of-fit | xxx | xxx | xxx |

Mean execution time | 1.4337830338577313e-2 | 1.448797309139284e-2 | 1.4763601823824296e-2 |

Standard deviation | 2.772536754203617e-4 | 4.774249175945004e-4 | 8.291091461013828e-4 |

Outlying measurements have moderate (0.11563227985262721%) effect on estimated standard deviation.

## restrict/restrict relation 10x10000 to 10x5000

lower bound | estimate | upper bound | |
---|---|---|---|

OLS regression | xxx | xxx | xxx |

R² goodness-of-fit | xxx | xxx | xxx |

Mean execution time | 8.322941573263729e-2 | 8.447258684075218e-2 | 8.511386931168204e-2 |

Standard deviation | 6.118316959029327e-4 | 1.4013810341920257e-3 | 2.3016514403696887e-3 |

Outlying measurements have slight (9.0e-2%) effect on estimated standard deviation.

## project/project 10x1000 to 1x1000

lower bound | estimate | upper bound | |
---|---|---|---|

OLS regression | xxx | xxx | xxx |

R² goodness-of-fit | xxx | xxx | xxx |

Mean execution time | 1.2211661066456561e-2 | 1.243577405705988e-2 | 1.2583044006972743e-2 |

Standard deviation | 3.0211460384957275e-4 | 4.600204612133745e-4 | 5.956804020691042e-4 |

Outlying measurements have moderate (0.14345146934193906%) effect on estimated standard deviation.

## project/project 10x1000 to 5x1000

lower bound | estimate | upper bound | |
---|---|---|---|

OLS regression | xxx | xxx | xxx |

R² goodness-of-fit | xxx | xxx | xxx |

Mean execution time | 1.5271674270594341e-2 | 1.547382196820991e-2 | 1.5622307025724294e-2 |

Standard deviation | 2.873217405621163e-4 | 3.978143402171812e-4 | 5.247713511789067e-4 |

Outlying measurements have slight (4.1587901701323225e-2%) effect on estimated standard deviation.

## union/union identical 10x1000

lower bound | estimate | upper bound | |
---|---|---|---|

OLS regression | xxx | xxx | xxx |

R² goodness-of-fit | xxx | xxx | xxx |

Mean execution time | 4.908323998442862e-2 | 4.923173514046889e-2 | 4.94547578201357e-2 |

Standard deviation | 2.09615306506087e-4 | 3.275115670368287e-4 | 4.977889379852797e-4 |

Outlying measurements have slight (7.100591715976319e-2%) effect on estimated standard deviation.

## union/union identical 10x10000

lower bound | estimate | upper bound | |
---|---|---|---|

OLS regression | xxx | xxx | xxx |

R² goodness-of-fit | xxx | xxx | xxx |

Mean execution time | 0.5144192261126985 | 0.5157808375258943 | 0.5164000385072457 |

Standard deviation | 0 | 1.1935851498586624e-3 | 1.285892588023751e-3 |

Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.

## join/join identical 10x100

lower bound | estimate | upper bound | |
---|---|---|---|

OLS regression | xxx | xxx | xxx |

R² goodness-of-fit | xxx | xxx | xxx |

Mean execution time | 0.3355841931137941 | 0.3374978199984254 | 0.3385756312293291 |

Standard deviation | 0 | 1.6965376723717264e-3 | 1.8668238128934738e-3 |

Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.

## group/group 10x100

lower bound | estimate | upper bound | |
---|---|---|---|

OLS regression | xxx | xxx | xxx |

R² goodness-of-fit | xxx | xxx | xxx |

Mean execution time | 4.666001885635249e-2 | 4.70336206675725e-2 | 4.7476184359486216e-2 |

Standard deviation | 5.178696454087703e-4 | 7.389605929837668e-4 | 1.1404867684675546e-3 |

Outlying measurements have slight (7.100591715976322e-2%) effect on estimated standard deviation.

## write/write 10x1000

lower bound | estimate | upper bound | |
---|---|---|---|

OLS regression | xxx | xxx | xxx |

R² goodness-of-fit | xxx | xxx | xxx |

Mean execution time | 1.5472953190097016e-3 | 1.5673276303702817e-3 | 1.5890440248897237e-3 |

Standard deviation | 6.110376965683773e-5 | 6.970009193368111e-5 | 7.954403273020534e-5 |

Outlying measurements have moderate (0.30346203307926556%) effect on estimated standard deviation.

## understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel
density estimate is built. The
*x*axis indicates the number of loop iterations, while the*y*axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

*OLS regression*indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the*mean*estimate below it, as it more effectively eliminates measurement overhead and other constant factors.*R² goodness-of-fit*is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.*Mean execution time*and*standard deviation*are statistics calculated from execution time divided by number of iterations.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.