Performance Awareness in Software Development

Software performance tends to be easier to observe after the fact than to design up front. Focusing on the implementation phases of software development, we work on methods that help improve developer awareness of software performance.

Performance Unit Testing strives to construct specialized unit tests that validate performance. The obstacles we address include portable test construction and robust evaluation of noisy test results.

• We have introduced a formalism for describing performance test conditions in an ICPE 2012 paper [1].
• We have evaluated the use of performance unit testing in a software project in an EPEW 2013 paper [2].
• We are developing tool support for performance unit testing in Java.

Performance Documentation incorporates information on code performance into standard code documentation. Performance is collected from corresponding performance unit tests and displayed interactively in standard documentation.

• We have introduced the concept of performance documentation with a case study in an ICPE 2015 paper [3].
• We are developing tool support for performance documentation in Java.

Performance Unit Testing

Compared to functional unit testing, performance unit testing is more difficult, partially because correctness criteria are more difficult to express for performance than for functionality. Using the Stochastic Performance Logic (SPL) [1], we aim to express assertions on code performance in relative, hardware-independent terms. Using the performance unit testing tools, these assertions can be automatically validated [2], [5]. Besides performance unit testing, we experiment with using the performance unit tests to generate data for software documentation extended with performance information [3]. Other research includes incorporating performance awareness into adaptive applications [4].

Example

Stochastic Performance Logic (SPL) allows the developer to capture assumptions about performance of a code fragment. For example, to express that a proprietary sorting method is faster than the one provided by the Java Class Library, we can execute a test with the following annotation (somewhat simplified syntax):

@SPL("for n in 100, 200, 500, 1000 SELF(n) < java.util.Arrays#sort(n)")
public static void proprietarySort(int[] data) {
    ...
}

As another example, it is possible to also express than newer version of the method is faster, helping to prevent performance regressions:

@SPL("for n in 100, 200, 500, 1000 SELF@rev951(n) < SELF@rev840(n)")
public static void proprietarySort(int[] data) {
    ...
}

Complete examples are included with the tools.

Tools

The core tool is a command line utility to execute the performance unit tests. The utility scans a given project and locates and executes the tests specified using annotations. The results are provided in the form of an HTML report with a quick overview and a detailed information page for each test, including visualization of the measurements. The utility can cooperate with both Git and Subversion to fetch particular project versions used as baselines, it can also use SSH to execute the measurements on a remote host, which can help improve measurement stability.

In addition to the core command line utility, we also provide plugins for the Eclipse integrated development environment and the Hudson continuous integration server. The Eclipse plugin provides project configuration, annotation editor with content assist, and interactive results browser also for results from Hudson.

We also make available research prototypes under development, listed below.

Experiments

We have performed a retroactive study where we have picked an existing software project, augmented it with tests and evaluated the results. Our project of choice was JDOM, a software package “for accessing, manipulating, and outputting XML data from Java code”. With about 15000 lines of code, it is of reasonable size for an experiment that requires frequent compilation and manual code inspection. With over 1500 commits spread across 13 years of development, it provided ample opportunity to observe performance regressions. JDOM is an open-source project with public source code repository. Details of the case study can be found in [2].

The table below contains the results from the measurements on three different machines.

¹ Open index.html in the ZIP archive to see the overview page of the HTML report.

² The data files are plain text, one execution time per line. The first line of each file contains the measurement identification, including machine, method, repository, revision, workload generator and date. To process the data files in R, use read.table("filename.dat")[,1], which gives access to the measured values as a vector of execution times in nanoseconds.

Performance Documentation

To increase performance awareness in software development, we use the performance unit tests to enrich reference documentation with performance related information. To this end, our tools extend the standard Javadoc generator to include an interactive performance browser with each method that has a performance unit test. The tools execute measurements on the fly, gradually increasing the accuracy of presented results [3], [6].

Tools

Acknowledgments

This work has been supported by the EU project 257414 ASCENS, the GACR project P202/10/J042 FERDINAND, and by the Charles University institutional funding SVV-2012-265312 and SVV-2013-267312.

Contact

• Petr Tůma
• Vojtěch Horký

References

1. L. Bulej, T. Bureš, J. Keznikl, A. Koubková, A. Podzimek, P. Tůma: Capturing Performance Assumptions Using Stochastic Performance Logic, in Proc. 3rd ACM/SPEC International Conference on Performance Engineering (ICPE), pp. 311–322, 2012, ISBN: 978-1-4503-1202-8, DOI: 10.1145/2188286.2188345
2. V. Horký, F. Haas, J. Kotrč, M. Lacina, P. Tůma: Performance Regression Unit Testing: A Case Study, in Computer Performance Engineering: Proc. 10th European Performance Engineering Workshop (EPEW), pp. 149-163, 2013, ISBN: 978-3-642-40724-6 978-3-642-40725-3
3. V. Horký, P. Libič, L. Marek, A. Steinhauser, P. Tůma: Utilizing Performance Unit Tests To Increase Performance Awareness, in Proc. 6th ACM/SPEC International Conference on Performance Engineering (ICPE), pp. 289–300, 2015, ISBN: 978-1-4503-3248-4, DOI: 10.1145/2668930.2688051
4. L. Bulej, T. Bureš, V. Horký, J. Keznikl, P. Tůma: Performance Awareness in Component Systems (Vision Paper), in Proc. 36th IEEE Annual Computer Software and Applications Conference (COMPSAC) CORCS Workshop, pp. 514-519, 2012, DOI: 10.1109/COMPSACW.2012.96
5. Tomáš Trojánek: Capturing Performance Assumptions using Stochastic Performance Logic, Master Thesis, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic, Charles University Thesis Repository Entry, 2012
6. Jakub Náplava: PerfJavaDoc: Extending API Documentation With Performance Information, Bachelor Thesis, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic, Charles University Thesis Repository Entry, 2015