Parameter Tuning for Search-Based Test-Data Generation Revisited

Support for Previous Results

Anton Kotelyanskii

Gregory M. Kapfhammer

creative commons licensed ( BY-NC-ND ) flickr photo shared by sunface13

Software Testing

Test Suites

Automatic Generation

Confronting Challenges

Evaluation Strategies

Empirical Studies






Amazing test suite generator

Uses a genetic algorithm

Input: A Java class

Output: A JUnit test suite

creative commons licensed ( BY-SA ) flickr photo shared by mcclanahoochie

Parameter Tuning

RSM: Response surface methodology

SPOT: Sequential parameter optimization toolbox

Successfully applied to many diverse problems!

Defaults or Tuned Values?

Experiment Design

Eight EvoSuite parameters

Ten projects from SF100

475 Java classes for subjects

100 trials after parameter tuning

Aiming to improve statement coverage

creative commons licensed ( BY-NC ) flickr photo shared by Michael Kappel


Parameter Name Minimum Maximum
Population Size 5 99
Chromosome Length 5 99
Rank Bias 1.01 1.99
Number of Mutations 1 10
Max Initial Test Count 1 10
Crossover Rate 0.01 0.99
Constant Pool Use Probability 0.01 0.99
Test Insertion Probability 0.01 0.99


184 days of computation time estimated

Cluster of 70 computers running for weeks

Identified 139 "easy" and 21 "hard" classes

Mann-Whitney U-test and

Vargha-Delaney effect size


Category Effect Size p-value
Results Across Trials and Classes 0.5029 0.1045
No "Easy" and "Hard" Classes 0.5048 0.0314

Using lower-is-better inverse statement coverage

Effect size greater than 0.5 means that tuning is worse

Testing shows we do not always reject the null hypothesis

Additional empirical results in the QSIC 2014 paper!


Tuning improved scores for 11 classes

Otherwise, same as or worse than defaults

A "soft floor" may exist for parameter tuning

Additional details in the QSIC 2014 paper!

creative commons licensed ( BY ) photo shared by Startup Stock Photos

Practical Implications

Fundamental Challenges

Tremendous Confidence

Great Opportunities

Important Contributions

Comprehensive Experiments

Conclusive Confirmation

For EvoSuite, Defaults = Tuned

creative commons licensed ( BY-NC-ND ) flickr photo shared by sunface13