criterion performance measurements

overview

want to understand this report?

Environment: AMD Opteron 6168 on Ubuntu 12.04 with GHC 8.0.2

create/relation 10x100

580
585
590
595
600
create/relation 10x100 time densities
mean
100
150
200
250
300
350
50 iters
100
150
200
250
0 s
50 ms
regression
create/relation 10x100 times
lower bound estimate upper bound
OLS regression 599 μs 599 μs 600 μs
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 589 μs 591 μs 592 μs
Standard deviation 3.79 μs 4.50 μs 5.81 μs

Outlying measurements have slight (1.2%) effect on estimated standard deviation.

create/relation 10x1000

5.68
5.7
5.73
5.75
5.78
5.80
5.83
create/relation 10x1000 time densities
mean
20
30
40
10 iters
100
150
200
250
300
0 s
50 ms
regression
create/relation 10x1000 times
lower bound estimate upper bound
OLS regression 5.82 ms 5.82 ms 5.83 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 5.76 ms 5.77 ms 5.78 ms
Standard deviation 22.9 μs 28.8 μs 39.1 μs

Outlying measurements have slight (2.6%) effect on estimated standard deviation.

create/relation 10x10000

57.5
57.6
57.7
57.8
57.9
58.0
58.1
create/relation 10x10000 time densities
mean
4
6
8
10
12
2 iters
200
300
400
500
600
700
800
0 s
100 ms
regression
create/relation 10x10000 times
lower bound estimate upper bound
OLS regression 58.0 ms 58.1 ms 58.2 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 57.7 ms 57.8 ms 57.9 ms
Standard deviation 104 μs 171 μs 274 μs

Outlying measurements have slight (7.6%) effect on estimated standard deviation.

restrict/restrict relation 10x10000 to 10x1

14
15
16
14.5
15.5
restrict/restrict relation 10x10000 to 10x1 time densities
mean
10
15
20
25
5 iters
100
150
200
250
300
350
400
0 s
50 ms
regression
restrict/restrict relation 10x10000 to 10x1 times
lower bound estimate upper bound
OLS regression 14.3 ms 14.5 ms 14.7 ms
R² goodness-of-fit 0.998 0.999 0.999
Mean execution time 14.3 ms 14.5 ms 14.8 ms
Standard deviation 277 μs 477 μs 829 μs

Outlying measurements have moderate (11.6%) effect on estimated standard deviation.

restrict/restrict relation 10x10000 to 10x5000

81
82
83
84
85
86
restrict/restrict relation 10x10000 to 10x5000 time densities
mean
4
6
8
10
2 iters
400
600
800
0 s
200 ms
1 s
regression
restrict/restrict relation 10x10000 to 10x5000 times
lower bound estimate upper bound
OLS regression 84.1 ms 85.0 ms 86.2 ms
R² goodness-of-fit 0.999 1.000 1.000
Mean execution time 83.2 ms 84.5 ms 85.1 ms
Standard deviation 612 μs 1.40 ms 2.30 ms

Outlying measurements have slight (9.0%) effect on estimated standard deviation.

project/project 10x1000 to 1x1000

12
13
11.5
12.5
project/project 10x1000 to 1x1000 time densities
mean
10
15
20
25
5 iters
100
150
200
250
300
350
400
0 s
50 ms
regression
project/project 10x1000 to 1x1000 times
lower bound estimate upper bound
OLS regression 12.8 ms 13.0 ms 13.1 ms
R² goodness-of-fit 0.999 0.999 1.000
Mean execution time 12.2 ms 12.4 ms 12.6 ms
Standard deviation 302 μs 460 μs 596 μs

Outlying measurements have moderate (14.3%) effect on estimated standard deviation.

project/project 10x1000 to 5x1000

15
16
14.5
14.8
15.3
15.5
15.8
project/project 10x1000 to 5x1000 time densities
mean
10
15
20
5 iters
200
300
400
0 s
100 ms
regression
project/project 10x1000 to 5x1000 times
lower bound estimate upper bound
OLS regression 15.8 ms 16.0 ms 16.2 ms
R² goodness-of-fit 0.999 0.999 1.000
Mean execution time 15.3 ms 15.5 ms 15.6 ms
Standard deviation 287 μs 398 μs 525 μs

Outlying measurements have slight (4.2%) effect on estimated standard deviation.

union/union identical 10x1000

49
50
48.8
49.2
49.4
49.6
49.8
union/union identical 10x1000 time densities
mean
4
6
8
10
12
2 iters
200
300
400
500
600
700
0 s
100 ms
regression
union/union identical 10x1000 times
lower bound estimate upper bound
OLS regression 48.9 ms 49.2 ms 49.6 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 49.1 ms 49.2 ms 49.5 ms
Standard deviation 210 μs 328 μs 498 μs

Outlying measurements have slight (7.1%) effect on estimated standard deviation.

union/union identical 10x10000

515
515
516
516
517
union/union identical 10x10000 time densities
mean
1
2
2
3
3
4
4
0.5 iters
2
0 s
500 ms
1 s
1.5
2.5
regression
union/union identical 10x10000 times
lower bound estimate upper bound
OLS regression 509 ms 516 ms 521 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 514 ms 516 ms 516 ms
Standard deviation 0 s 1.19 ms 1.29 ms

Outlying measurements have moderate (18.7%) effect on estimated standard deviation.

join/join identical 10x100

336
336
337
337
338
338
339
339
join/join identical 10x100 time densities
mean
1
2
2
3
3
4
4
0.5 iters
500
750
0 s
250 ms
1 s
1.25
1.5
regression
join/join identical 10x100 times
lower bound estimate upper bound
OLS regression 341 ms 343 ms 346 ms
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 336 ms 337 ms 339 ms
Standard deviation 0 s 1.70 ms 1.87 ms

Outlying measurements have moderate (18.8%) effect on estimated standard deviation.

group/group 10x100

46
47
48
45.5
46.5
47.5
48.5
group/group 10x100 time densities
mean
4
6
8
10
12
2 iters
200
300
400
500
600
700
0 s
100 ms
regression
group/group 10x100 times
lower bound estimate upper bound
OLS regression 46.4 ms 47.0 ms 47.4 ms
R² goodness-of-fit 0.999 1.000 1.000
Mean execution time 46.7 ms 47.0 ms 47.5 ms
Standard deviation 518 μs 739 μs 1.14 ms

Outlying measurements have slight (7.1%) effect on estimated standard deviation.

write/write 10x1000

1.45
1.50
1.55
1.6
1.65
write/write 10x1000 time densities
mean
50
75
100
125
25 iters
100
150
200
250
0 s
50 ms
regression
write/write 10x1000 times
lower bound estimate upper bound
OLS regression 1.51 ms 1.56 ms 1.62 ms
R² goodness-of-fit 0.992 0.995 0.998
Mean execution time 1.55 ms 1.57 ms 1.59 ms
Standard deviation 61.1 μs 69.7 μs 79.5 μs

Outlying measurements have moderate (30.3%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.