criterion performance measurements
overview
want to understand this report?
bench/./Curry/ReplicateDie 2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.508165775029291e-2 | 2.547894736397135e-2 | 2.606210945532307e-2 |
| Standard deviation | 6.965984825148872e-4 | 1.0450193007930137e-3 | 1.5064995518359694e-3 |
Outlying measurements have moderate (0.14570719632426354%) effect on estimated standard deviation.
bench/./Curry/ReplicateDie 3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.5080658200985564e-2 | 2.5832952193163422e-2 | 2.7868237276572998e-2 |
| Standard deviation | 9.615370492021815e-4 | 2.5780474740517857e-3 | 4.832636903644789e-3 |
Outlying measurements have moderate (0.42722779074022105%) effect on estimated standard deviation.
bench/./Curry/ReplicateDie 4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.554045648809823e-2 | 2.6419943799268143e-2 | 2.7718163767097903e-2 |
| Standard deviation | 1.7379350771683597e-3 | 2.3316529860060715e-3 | 3.0087128898990075e-3 |
Outlying measurements have moderate (0.3540895657600912%) effect on estimated standard deviation.
bench/./Curry/ReplicateDie 5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.4771141532390538e-2 | 2.511100633304272e-2 | 2.5391188395788626e-2 |
| Standard deviation | 4.867143085117779e-4 | 6.960455149974617e-4 | 1.031179502233606e-3 |
Outlying measurements have slight (4.986149584487535e-2%) effect on estimated standard deviation.
bench/./Curry/ReplicateDie 10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.4876202351592196e-2 | 2.5159573310981e-2 | 2.545348529365808e-2 |
| Standard deviation | 4.6168992591860415e-4 | 6.333501662920411e-4 | 9.5761156583288e-4 |
Outlying measurements have slight (4.986149584487534e-2%) effect on estimated standard deviation.
bench/./Curry/ReplicateDie 15
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.500839410412954e-2 | 2.531172529413352e-2 | 2.566770712450938e-2 |
| Standard deviation | 6.086066070845804e-4 | 7.357792835931941e-4 | 8.949431283543341e-4 |
Outlying measurements have slight (7.388911829262826e-2%) effect on estimated standard deviation.
bench/./Curry/ReplicateDie 25
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.4774115263364198e-2 | 2.5005770096581517e-2 | 2.5227096485020598e-2 |
| Standard deviation | 4.1443209776581503e-4 | 4.997376781032426e-4 | 6.112244131387863e-4 |
Outlying measurements have slight (4.986149584487534e-2%) effect on estimated standard deviation.
bench/./Curry/ReplicateDie 50
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.642744659841225e-2 | 3.696895474885466e-2 | 3.782627720734787e-2 |
| Standard deviation | 9.057335390768865e-4 | 1.4610799733472083e-3 | 2.365249775320408e-3 |
Outlying measurements have moderate (0.11523108262142467%) effect on estimated standard deviation.
bench/./Curry/ReplicateDie 100
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.10568303056545819 | 0.10896648666280408 | 0.11476658104904115 |
| Standard deviation | 3.3755195228827063e-3 | 6.717739794810498e-3 | 1.187863141043722e-2 |
Outlying measurements have moderate (0.1996646669488161%) effect on estimated standard deviation.
bench/./Curry/ReplicateDie 200
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.33417877661607537 | 0.37749019149244606 | 0.46214062470729306 |
| Standard deviation | 1.314931025262922e-3 | 8.390083019548283e-2 | 9.850507658862204e-2 |
Outlying measurements have moderate (0.48023126543061473%) effect on estimated standard deviation.
bench/python ProbLog/replicateDie.py 2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.17209205141730813 | 0.177056817933529 | 0.18280528789877684 |
| Standard deviation | 5.376172743407427e-3 | 7.714913761498982e-3 | 9.822594699721039e-3 |
Outlying measurements have moderate (0.13888888888888887%) effect on estimated standard deviation.
bench/python ProbLog/replicateDie.py 3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4968080395483412 | 0.5630756642640335 | 0.6919188005122123 |
| Standard deviation | 1.6660935361869633e-3 | 0.1276671469428748 | 0.14973715834385593 |
Outlying measurements have moderate (0.4810034273462169%) effect on estimated standard deviation.
bench/python ProbLog/replicateDie.py 4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.74575905338861 | 10.270813969798231 | 10.766786075352382 |
| Standard deviation | 0.48664841879377607 | 0.5757989592817259 | 0.631820469902542 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
bench/python ProbLog/replicateDie.py 5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 330.03310889808927 | 335.667910458229 | 341.2018055341905 |
| Standard deviation | 4.544901117611669 | 6.7640183178945605 | 7.924973705005383 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/replicateDie.wppl 2
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.722593838974717 | 1.841412473959887 | 1.934525809182863 |
| Standard deviation | 7.867169153844462e-2 | 0.13755580277696441 | 0.1931999124441112 |
Outlying measurements have moderate (0.20624936285731%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/replicateDie.wppl 3
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6614248967671301 | 1.9726969030161854 | 2.4612955674820114 |
| Standard deviation | 6.721651338650614e-2 | 0.4702339778757757 | 0.6157407294054976 |
Outlying measurements have moderate (0.48281328975753873%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/replicateDie.wppl 4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.587018496444216 | 1.6422711295211532 | 1.6806608671710515 |
| Standard deviation | 3.527048198588694e-2 | 5.950228161201445e-2 | 8.401417924239712e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/replicateDie.wppl 5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8574789731064811 | 1.9180661091086222 | 1.9732728056551423 |
| Standard deviation | 3.1268049409845844e-2 | 6.922931034370765e-2 | 8.801261530433822e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/replicateDie.wppl 6
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.681885625749904 | 3.779433753753741 | 3.8301079681744645 |
| Standard deviation | 6.994500095534303e-3 | 9.492237247007859e-2 | 0.11702708534711065 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/replicateDie.wppl 7
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 19.976683511777082 | 20.91385816804056 | 21.63352700312195 |
| Standard deviation | 0.41175792954163626 | 0.9090918928586171 | 1.09558161750057 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.