criterion performance measurements
overview
want to understand this report?
teaspoon /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4924031895263246e-7 | 1.5129562369525601e-7 | 1.5401903833435578e-7 |
| Standard deviation | 9.522374587535147e-9 | 1.1240989439825158e-8 | 1.4990883034268208e-8 |
Outlying measurements have severe (0.8567473698978191%) effect on estimated standard deviation.
teaspoon /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.4115004478057808e-8 | 2.4444835652883048e-8 | 2.4761793382239424e-8 |
| Standard deviation | 1.2388009230453414e-9 | 1.453864286217957e-9 | 1.6929701720966379e-9 |
Outlying measurements have severe (0.8117985162830379%) effect on estimated standard deviation.
const (control) /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.02593645866747e-9 | 7.109290674892493e-9 | 7.192453080398449e-9 |
| Standard deviation | 3.21466635923414e-10 | 3.8376810205198573e-10 | 5.305500519265233e-10 |
Outlying measurements have severe (0.7929820013640598%) effect on estimated standard deviation.
const (control) /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.125488126378474e-9 | 7.205165735308796e-9 | 7.296783288630013e-9 |
| Standard deviation | 3.3658919964630515e-10 | 3.9667277577447943e-10 | 5.040443761483633e-10 |
Outlying measurements have severe (0.7983787467695437%) effect on estimated standard deviation.
Just (control) /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.651774885828448e-9 | 7.775875397092576e-9 | 7.933527550172554e-9 |
| Standard deviation | 4.914890276009866e-10 | 6.021715441537476e-10 | 7.85366229452107e-10 |
Outlying measurements have severe (0.8876987114543708%) effect on estimated standard deviation.
justToJust (control)/check Nothing
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.312979029067131e-9 | 8.421315121053112e-9 | 8.550929514418925e-9 |
| Standard deviation | 4.30927227700022e-10 | 5.311558114994625e-10 | 7.420937951862758e-10 |
Outlying measurements have severe (0.8386804619740228%) effect on estimated standard deviation.
justToJust (control)/Check Just
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.740755679681477e-9 | 7.865429917651177e-9 | 8.000004194868036e-9 |
| Standard deviation | 5.218699915219296e-10 | 6.039248018607391e-10 | 7.62297984189059e-10 |
Outlying measurements have severe (0.8844333293196542%) effect on estimated standard deviation.
catchMethod1 /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.363285048718316e-8 | 3.401088739976908e-8 | 3.442789040391171e-8 |
| Standard deviation | 1.6104261247108203e-9 | 1.840348782530137e-9 | 2.096335540540165e-9 |
Outlying measurements have severe (0.778173262490266%) effect on estimated standard deviation.
catchMethod1 /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6119460652829615e-8 | 1.6409716101500247e-8 | 1.6802037537745936e-8 |
| Standard deviation | 1.228490286931892e-9 | 1.5467428021156642e-9 | 2.331935325788394e-9 |
Outlying measurements have severe (0.9174331245213555%) effect on estimated standard deviation.
catchMethod2 /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.2989303864499474e-8 | 3.3292041223272526e-8 | 3.3626969682864695e-8 |
| Standard deviation | 1.3228633608426778e-9 | 1.5392011076030736e-9 | 2.012195441877587e-9 |
Outlying measurements have severe (0.7173247265893776%) effect on estimated standard deviation.
catchMethod2 /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5242746100964277e-8 | 1.5406729763916314e-8 | 1.5586662383594134e-8 |
| Standard deviation | 6.935950515136367e-10 | 8.217144723206271e-10 | 1.0122530730453175e-9 |
Outlying measurements have severe (0.7801444998323301%) effect on estimated standard deviation.
catchMethod3 /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.319602445041095e-8 | 3.362503090937419e-8 | 3.4106975276016166e-8 |
| Standard deviation | 1.8568408417608713e-9 | 2.196095586953116e-9 | 3.151612103585927e-9 |
Outlying measurements have severe (0.8376693134859329%) effect on estimated standard deviation.
catchMethod3 /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5088985772822725e-8 | 1.5287016663295174e-8 | 1.5517580542356826e-8 |
| Standard deviation | 8.383623303960394e-10 | 9.63648829214993e-10 | 1.1397170111651993e-9 |
Outlying measurements have severe (0.8324922866615498%) effect on estimated standard deviation.
catchMethod4 /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.3719875971655485e-8 | 3.42445248097354e-8 | 3.4896623178217235e-8 |
| Standard deviation | 2.3541706505038565e-9 | 2.685005709919358e-9 | 3.0390250901295942e-9 |
Outlying measurements have severe (0.8801352414240017%) effect on estimated standard deviation.
catchMethod4 /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5064852480536768e-8 | 1.522630128638654e-8 | 1.5400623505327448e-8 |
| Standard deviation | 6.387953593939218e-10 | 7.476315669392351e-10 | 9.074944386882587e-10 |
Outlying measurements have severe (0.7492572648516422%) effect on estimated standard deviation.
catchMethod5 /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.315016445563423e-8 | 3.3602988265779877e-8 | 3.4255289401955615e-8 |
| Standard deviation | 1.823446245692651e-9 | 2.354109034736141e-9 | 3.4791962141857287e-9 |
Outlying measurements have severe (0.8542025913210011%) effect on estimated standard deviation.
catchMethod5 /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5206362982068948e-8 | 1.5459463957027658e-8 | 1.5767391641157087e-8 |
| Standard deviation | 1.025676528397493e-9 | 1.23893941245456e-9 | 1.5213227603610325e-9 |
Outlying measurements have severe (0.8890441848104582%) effect on estimated standard deviation.
catchMethod6 /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.140218864636789e-8 | 3.181083714562984e-8 | 3.2243243375823093e-8 |
| Standard deviation | 1.6610698073310777e-9 | 1.974863430579961e-9 | 2.505030600161803e-9 |
Outlying measurements have severe (0.8219506761477989%) effect on estimated standard deviation.
catchMethod6 /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.175225003210051e-8 | 3.226464874558321e-8 | 3.273734536392492e-8 |
| Standard deviation | 2.0176029411136767e-9 | 2.3011633974733336e-9 | 2.645394736774145e-9 |
Outlying measurements have severe (0.8606481411534501%) effect on estimated standard deviation.
primspoon /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.239773019988799e-8 | 3.285550622995881e-8 | 3.348575171417948e-8 |
| Standard deviation | 1.9168768139088827e-9 | 2.5668761534825786e-9 | 3.818353968192504e-9 |
Outlying measurements have severe (0.8804618746262052%) effect on estimated standard deviation.
primspoon /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5090452044620134e-8 | 1.5293001980598076e-8 | 1.5512158868192142e-8 |
| Standard deviation | 8.586082839339355e-10 | 1.0062695868480365e-9 | 1.272877195910416e-9 |
Outlying measurements have severe (0.8454260872675284%) effect on estimated standard deviation.
throwsMethod0 /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.2412834206199564e-8 | 3.283537468336913e-8 | 3.340816100815261e-8 |
| Standard deviation | 1.7374763853671148e-9 | 2.14966443996524e-9 | 3.061620815337233e-9 |
Outlying measurements have severe (0.837708701821855%) effect on estimated standard deviation.
throwsMethod0 /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5471820099939138e-8 | 1.568664378049676e-8 | 1.600368999713757e-8 |
| Standard deviation | 8.734980579518805e-10 | 1.1534119408340632e-9 | 1.7220461431608265e-9 |
Outlying measurements have severe (0.8731607389716959%) effect on estimated standard deviation.
throwsMethod1 /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.2156656403772365e-8 | 3.25704606935117e-8 | 3.317839521417611e-8 |
| Standard deviation | 1.8573791859692262e-9 | 2.237458869009485e-9 | 2.91993427895722e-9 |
Outlying measurements have severe (0.8508169221278034%) effect on estimated standard deviation.
throwsMethod1 /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4361310795425033e-8 | 1.4523992226134635e-8 | 1.4780834797708924e-8 |
| Standard deviation | 7.207903661985004e-10 | 9.39190569260209e-10 | 1.416783649409945e-9 |
Outlying measurements have severe (0.839273493411498%) effect on estimated standard deviation.
throwsMethod2 /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.224970606581321e-8 | 3.272147956860883e-8 | 3.3303728574026226e-8 |
| Standard deviation | 2.0420168161819467e-9 | 2.4237745747088254e-9 | 3.1888790990224077e-9 |
Outlying measurements have severe (0.8673698416456419%) effect on estimated standard deviation.
throwsMethod2 /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4709343141172197e-8 | 1.4923315765524755e-8 | 1.5178952997140425e-8 |
| Standard deviation | 8.728232782395199e-10 | 1.0625749578206934e-9 | 1.3307933340500174e-9 |
Outlying measurements have severe (0.8642529694210173%) effect on estimated standard deviation.
throwsMethod3 /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.1879101431959e-8 | 3.225480392394766e-8 | 3.2741461940877276e-8 |
| Standard deviation | 1.5988584087913286e-9 | 1.8927431976670206e-9 | 2.2142067861217905e-9 |
Outlying measurements have severe (0.8049999075265992%) effect on estimated standard deviation.
throwsMethod3 /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4612284345025871e-8 | 1.4775882350659054e-8 | 1.4971945787301427e-8 |
| Standard deviation | 6.948124898517939e-10 | 8.338570742495826e-10 | 1.1024432625677649e-9 |
Outlying measurements have severe (0.7987841058665628%) effect on estimated standard deviation.
throwsMethod4 /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.4082669537280217e-8 | 3.487225588340948e-8 | 3.569074356124327e-8 |
| Standard deviation | 3.391799877728726e-9 | 3.859300359052451e-9 | 4.660631155878453e-9 |
Outlying measurements have severe (0.9367048762995922%) effect on estimated standard deviation.
throwsMethod4 /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.445191641653295e-8 | 1.4636505124167117e-8 | 1.4871293776428604e-8 |
| Standard deviation | 7.98832108776118e-10 | 9.413309215742662e-10 | 1.132345633562719e-9 |
Outlying measurements have severe (0.8391856211413933%) effect on estimated standard deviation.
throwsMethod5 /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.2369968853293514e-8 | 3.2815287586104066e-8 | 3.344106616436695e-8 |
| Standard deviation | 1.7658752018636861e-9 | 2.3858918414443085e-9 | 3.947403110044294e-9 |
Outlying measurements have severe (0.8640165843951324%) effect on estimated standard deviation.
throwsMethod5 /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.454984466982508e-8 | 1.4761647878021656e-8 | 1.502473728823934e-8 |
| Standard deviation | 8.22445591840974e-10 | 1.047319762797153e-9 | 1.3694700182139215e-9 |
Outlying measurements have severe (0.864202799015545%) effect on estimated standard deviation.
throws /Check undefined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.233682761284755e-8 | 3.2750182342321156e-8 | 3.3263680128946826e-8 |
| Standard deviation | 1.7832357388245536e-9 | 2.173776808533904e-9 | 2.6951890806665613e-9 |
Outlying measurements have severe (0.8410164614425646%) effect on estimated standard deviation.
throws /Check defined
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4621843664045197e-8 | 1.4888949745138716e-8 | 1.517667542083771e-8 |
| Standard deviation | 1.0483930526219434e-9 | 1.184054051720054e-9 | 1.3561517787030583e-9 |
Outlying measurements have severe (0.8892815824922827%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.