criterion performance measurements
overview
want to understand this report?
fromList/HashMap.Strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.430087062139572e-2 | 5.5306861611212614e-2 | 5.6675910888740404e-2 |
| Standard deviation | 1.3997181820719936e-3 | 2.0765077546525262e-3 | 3.1342865183418074e-3 |
Outlying measurements have slight (7.691876843351375e-2%) effect on estimated standard deviation.
fromList/LinkedHashMap.Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.1118768702656383 | 0.11365900137786449 | 0.11530133712650943 |
| Standard deviation | 1.8375882037615183e-3 | 2.518686102060762e-3 | 3.348851243754148e-3 |
Outlying measurements have moderate (0.109375%) effect on estimated standard deviation.
fromList/LinkedHashMap.IntMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.13616156056465584 | 0.13991178703941676 | 0.1465616479489535 |
| Standard deviation | 2.495435373816608e-3 | 7.137081410157575e-3 | 1.0954786477752082e-2 |
Outlying measurements have moderate (0.11327217785466746%) effect on estimated standard deviation.
fromList/LinkedHashSet
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.10832844180195576 | 0.11112111195283281 | 0.11456346264201635 |
| Standard deviation | 3.495516077831306e-3 | 4.783726891218055e-3 | 6.395679181516847e-3 |
Outlying measurements have slight (9.95459176947333e-2%) effect on estimated standard deviation.
insert/HashMap.Strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.36859746050086e-2 | 9.564038285186564e-2 | 9.765368661116038e-2 |
| Standard deviation | 2.1439719721177337e-3 | 3.059855153748262e-3 | 4.096123128351332e-3 |
Outlying measurements have slight (9.876543209876541e-2%) effect on estimated standard deviation.
insert/LinkedHashMap.Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.12925660774424838 | 0.13214900029892288 | 0.13636362410684086 |
| Standard deviation | 3.2869328172669327e-3 | 5.008508351789427e-3 | 7.444706798703425e-3 |
Outlying measurements have moderate (0.109375%) effect on estimated standard deviation.
insert/LinkedHashMap.IntMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.1381321991664535 | 0.1408451551438188 | 0.14382993468392277 |
| Standard deviation | 2.593116369287535e-3 | 3.994899078013638e-3 | 5.690067129933355e-3 |
Outlying measurements have moderate (0.12244897959183673%) effect on estimated standard deviation.
insert/LinkedHashSet
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.12084342120217309 | 0.12276720309503271 | 0.12496266379612098 |
| Standard deviation | 2.1659102822166884e-3 | 2.8456342197920293e-3 | 3.6654346083441975e-3 |
Outlying measurements have moderate (0.109375%) effect on estimated standard deviation.
toList/HashMap.Strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1250000938920172e-2 | 1.1564078052990583e-2 | 1.1916531254640409e-2 |
| Standard deviation | 6.836980631636226e-4 | 8.690256302020189e-4 | 1.1079661243709234e-3 |
Outlying measurements have moderate (0.37760649966406246%) effect on estimated standard deviation.
toList/LinkedHashMap.Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.668126983910865e-3 | 1.0264724493343624e-2 | 1.134531974756007e-2 |
| Standard deviation | 1.1502333521670755e-3 | 2.1723309778124496e-3 | 3.7370394414292024e-3 |
Outlying measurements have severe (0.8579320155725961%) effect on estimated standard deviation.
toList/LinkedHashMap.IntMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.915954672256332e-3 | 6.455850815980168e-3 | 7.105598447030257e-3 |
| Standard deviation | 1.3042923623282844e-3 | 1.6841884988224372e-3 | 2.573598913431011e-3 |
Outlying measurements have severe (0.9136793173275716%) effect on estimated standard deviation.
toList/LinkedHashSet
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.059316847661113e-3 | 6.573212020409705e-3 | 7.337737697536092e-3 |
| Standard deviation | 1.376369784330068e-3 | 1.8097239184442714e-3 | 2.5927745775057246e-3 |
Outlying measurements have severe (0.9142145289337344%) effect on estimated standard deviation.
lookup/HashMap.Strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.473729355735698e-2 | 2.504179551543866e-2 | 2.5553656780240602e-2 |
| Standard deviation | 5.91114966714086e-4 | 8.6654628690708e-4 | 1.2780424853415851e-3 |
Outlying measurements have slight (9.765347406472392e-2%) effect on estimated standard deviation.
lookup/LinkedHashMap.Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.08267747073094e-2 | 3.093286253272737e-2 | 3.12540073378293e-2 |
| Standard deviation | 1.2408752013628844e-4 | 3.421632926602405e-4 | 6.268687888922882e-4 |
Outlying measurements have slight (5.536332179930796e-2%) effect on estimated standard deviation.
lookup/LinkedHashMap.IntMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.426628612118356e-2 | 3.488420362072054e-2 | 3.570621402615471e-2 |
| Standard deviation | 1.0725611598340382e-3 | 1.464618428605021e-3 | 1.843688983459404e-3 |
Outlying measurements have moderate (0.11634377161476366%) effect on estimated standard deviation.
lookup/LinkedHashSet
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8353393095660316e-2 | 1.8592965281378555e-2 | 1.8854385608461005e-2 |
| Standard deviation | 4.905290524957243e-4 | 6.002411348437584e-4 | 7.889004841042409e-4 |
Outlying measurements have slight (8.527715398147008e-2%) effect on estimated standard deviation.
delete/HashMap.Strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.657259152672139e-2 | 6.880135554728915e-2 | 7.184987411789664e-2 |
| Standard deviation | 2.910503358264081e-3 | 4.35907675998751e-3 | 6.886786435481943e-3 |
Outlying measurements have moderate (0.17012030837842088%) effect on estimated standard deviation.
delete/LinkedHashMap.Seq
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.3088634549358604 | 0.312881759024926 | 0.31803884141788674 |
| Standard deviation | 8.651382806075028e-4 | 5.108554903587085e-3 | 7.063215807089006e-3 |
Outlying measurements have moderate (0.15999999999999998%) effect on estimated standard deviation.
delete/LinkedHashMap.IntMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.29083887617878446 | 0.3197353696833014 | 0.3428401871998568 |
| Standard deviation | 1.731677156351609e-2 | 3.0840905315095434e-2 | 4.202518238869635e-2 |
Outlying measurements have moderate (0.1865650164292625%) effect on estimated standard deviation.
delete/LinkedHashSet
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.31055797004782754 | 0.3163024372206242 | 0.32054798234246284 |
| Standard deviation | 3.120241838309029e-3 | 6.005503526427491e-3 | 7.690315050020456e-3 |
Outlying measurements have moderate (0.16%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.