criterion performance measurements
overview
want to understand this report?
bench/./Curry/Strings fast 5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.4608384107080782e-2 | 2.4824514360770277e-2 | 2.499785125361543e-2 |
| Standard deviation | 2.976569251886693e-4 | 4.181931892974109e-4 | 5.971002138196778e-4 |
Outlying measurements have slight (4.986149584487534e-2%) effect on estimated standard deviation.
bench/./Curry/Strings fast 10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.5061152019446056e-2 | 2.5346126255483818e-2 | 2.5795988500058042e-2 |
| Standard deviation | 5.489094536708504e-4 | 7.83701071394409e-4 | 1.0848190839174109e-3 |
Outlying measurements have slight (9.57320909221408e-2%) effect on estimated standard deviation.
bench/./Curry/Strings fast 15
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.894824520795961e-2 | 4.1874811428272006e-2 | 4.715789193653653e-2 |
| Standard deviation | 3.867989218692066e-3 | 7.940382972078445e-3 | 1.2544850504888528e-2 |
Outlying measurements have severe (0.717753799844224%) effect on estimated standard deviation.
bench/./Curry/Strings fast 20
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.15242723871987046 | 0.16171339317929748 | 0.16863901786685156 |
| Standard deviation | 5.301992273842078e-3 | 1.1297236693381514e-2 | 1.3995864481940195e-2 |
Outlying measurements have moderate (0.13352302178424005%) effect on estimated standard deviation.
bench/./Curry/Strings fast 25
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.7318278246675618 | 0.789869758800099 | 0.8479116929326362 |
| Standard deviation | 5.174191951345151e-2 | 6.836843831699052e-2 | 7.945732397813539e-2 |
Outlying measurements have moderate (0.2174133507788208%) effect on estimated standard deviation.
bench/python ProbLog/stringsFast.py 5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.16841012778847167 | 0.1852364112021557 | 0.2100767446992298 |
| Standard deviation | 1.4506282978539248e-2 | 2.7823703242263156e-2 | 3.985460096822947e-2 |
Outlying measurements have moderate (0.47114429857357487%) effect on estimated standard deviation.
bench/python ProbLog/stringsFast.py 10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.248319277507253 | 0.2558378758421168 | 0.2630109216785058 |
| Standard deviation | 5.378813707466398e-3 | 8.852637123624201e-3 | 1.3411574865724123e-2 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
bench/python ProbLog/stringsFast.py 15
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.35284717815617717 | 0.3562154554577622 | 0.36259521688043606 |
| Standard deviation | 4.7510714891055295e-4 | 6.281849897918761e-3 | 7.641017189810116e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
bench/python ProbLog/stringsFast.py 20
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1923811023589224 | 1.231073061513598 | 1.2888123728480423 |
| Standard deviation | 1.7652852659986548e-2 | 5.4901538033453025e-2 | 7.170239947371757e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
bench/python ProbLog/stringsFast.py 25
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.178917607263429 | 2.300884317238039 | 2.405974314955529 |
| Standard deviation | 8.687052148732562e-2 | 0.12505174236664324 | 0.14958713963452164 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/stringsFast.wppl 5
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5913679461664287 | 1.7306238726547842 | 1.826863287036152 |
| Standard deviation | 8.718059640765807e-2 | 0.14950181137438806 | 0.21113263409267902 |
Outlying measurements have moderate (0.2172845782674133%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/stringsFast.wppl 10
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.661249644588679 | 1.7335733827785589 | 1.7976072953315452 |
| Standard deviation | 3.6777969260955896e-2 | 7.954048452315438e-2 | 0.10190124405593219 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/stringsFast.wppl 15
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5245499142523233 | 1.736168790628047 | 1.906530347361695 |
| Standard deviation | 0.1722135861442064 | 0.22495989555331247 | 0.2674932845497856 |
Outlying measurements have moderate (0.23545834828244236%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/stringsFast.wppl 20
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.561704711522907 | 1.5752526139064382 | 1.5970622622796025 |
| Standard deviation | 5.129958502948284e-3 | 2.1078165312604463e-2 | 2.8145395640963713e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
bench/./WebPPL/node_modules/.bin/webppl WebPPL/stringsFast.wppl 25
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0712027875706553 | 2.104103980731452 | 2.160064159310423 |
| Standard deviation | 1.8000674391091022e-2 | 5.577886749935113e-2 | 7.524865556247826e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.