criterion performance measurements
overview
want to understand this report?
bench/./Curry/Strings bs 5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.478168742697722e-2 | 2.498564793389119e-2 | 2.525110704502709e-2 |
| Standard deviation | 3.437114650510261e-4 | 5.231935371373576e-4 | 8.042243378571322e-4 |
Outlying measurements have slight (4.986149584487534e-2%) effect on estimated standard deviation.
bench/./Curry/Strings bs 6
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.46179000169562e-2 | 2.4906054725250846e-2 | 2.5107910504103922e-2 |
| Standard deviation | 3.735470280857124e-4 | 5.167437012375148e-4 | 6.842811263710826e-4 |
Outlying measurements have slight (4.986149584487533e-2%) effect on estimated standard deviation.
bench/./Curry/Strings bs 7
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.4385549383082177e-2 | 2.4907801566817718e-2 | 2.509558853870334e-2 |
| Standard deviation | 2.0799819175146717e-4 | 6.896385929442692e-4 | 1.2684945520493865e-3 |
Outlying measurements have slight (4.986149584487534e-2%) effect on estimated standard deviation.
bench/./Curry/Strings bs 8
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.416588737809028e-2 | 2.4549361938866542e-2 | 2.4873523474890684e-2 |
| Standard deviation | 5.802787016659808e-4 | 7.675962117418001e-4 | 1.092581197430016e-3 |
Outlying measurements have slight (9.594277000756908e-2%) effect on estimated standard deviation.
bench/./Curry/Strings bs 9
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.5429941008010964e-2 | 2.6702239617851422e-2 | 2.8298047188279503e-2 |
| Standard deviation | 2.2915994670928484e-3 | 2.980692233106394e-3 | 3.991285041539471e-3 |
Outlying measurements have moderate (0.48258399120444073%) effect on estimated standard deviation.
bench/./Curry/Strings bs 10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.780683582673161e-2 | 3.925113024275346e-2 | 4.141192984237538e-2 |
| Standard deviation | 2.4289428037739293e-3 | 3.6950711484845895e-3 | 5.185043828690764e-3 |
Outlying measurements have moderate (0.381193521881071%) effect on estimated standard deviation.
bench/python ProbLog/strings.py bs 5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.25470619306666775 | 0.2668753542789879 | 0.27590041007303323 |
| Standard deviation | 7.63803755763246e-3 | 1.314332871553352e-2 | 1.7513848990439146e-2 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
bench/python ProbLog/strings.py bs 6
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.4038947492663283 | 0.4186751557572279 | 0.42726985225453973 |
| Standard deviation | 5.563316025596928e-3 | 1.4549879381519956e-2 | 1.984860134530444e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
bench/python ProbLog/strings.py bs 7
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.906505729139705 | 0.9781123764745038 | 1.0728463448031107 |
| Standard deviation | 2.4295500275911763e-2 | 0.10267039480252145 | 0.13731444120926228 |
Outlying measurements have moderate (0.2278421766105705%) effect on estimated standard deviation.
bench/python ProbLog/strings.py bs 8
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.017229679346201 | 3.0715956086302563 | 3.124930851355505 |
| Standard deviation | 5.2481563285100605e-2 | 6.164213390036006e-2 | 6.420038975093104e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
bench/python ProbLog/strings.py bs 9
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 10.11210215309984 | 10.396871128318404 | 10.824132638789402 |
| Standard deviation | 0.10494075866575475 | 0.41147624045827363 | 0.552705019331412 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
bench/python ProbLog/strings.py bs 10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 38.506998091877904 | 40.5827059968918 | 43.70695603711647 |
| Standard deviation | 0.7702007877524011 | 2.975344580506619 | 3.9042908065973947 |
Outlying measurements have moderate (0.2045799595475725%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/strings.wppl bs 5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.552370374755507 | 1.5798058721071964 | 1.5948884818741742 |
| Standard deviation | 2.101341115659211e-3 | 2.5914535372615136e-2 | 3.2730454120527726e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/strings.wppl bs 6
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5529755750321783 | 1.5846002447409167 | 1.5955875434641105 |
| Standard deviation | 4.007946699857712e-5 | 2.122672974596775e-2 | 2.5374079502011128e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/strings.wppl bs 7
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.563585592142772 | 1.5884871783200651 | 1.6066005505854264 |
| Standard deviation | 1.613508680175408e-2 | 2.8127233376501085e-2 | 3.969690278880247e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/strings.wppl bs 8
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5749438547936734 | 1.6067366190836765 | 1.6239232640655246 |
| Standard deviation | 5.161051347386092e-3 | 3.10626339035213e-2 | 3.969085642694607e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/strings.wppl bs 9
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6039975782769034 | 1.624694835030823 | 1.6372662896174006 |
| Standard deviation | 9.613674046704546e-3 | 2.0680034261924655e-2 | 2.7731544355351575e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/strings.wppl bs 10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6230444001800304 | 1.6495569616624077 | 1.6629502285892765 |
| Standard deviation | 2.1090402980717543e-4 | 2.506023157828042e-2 | 3.071952103241772e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.