In the cutthroat world of iOS app development, speed is a critical factor in an app's success. With users having no patience for slow and unresponsive apps, it's essential to evaluate an app's performance before its release. This ensures a seamless user experience and sets an app apart from its competitors.
At Apple's 2014 Worldwide Developers Conference (WWDC), the company introduced performance testing support for the XCTest framework. This framework allows developers to measure code performance within unit or UI tests. In this article, we'll explore the process of conducting automated performance testing using the XCTest framework, a crucial step in ensuring an app's success.
The Evolution of Performance Testing in iOS
Apple provides a range of developer tools to assess the performance of iOS apps, including Instruments, Leaks, Profiler, and Allocations. These tools help detect performance issues, memory leaks, and other bottlenecks. However, traditional manual monitoring and execution of these checks are no longer viable in modern iOS development. A single code change can render an app unresponsive, unstable, and unusable. Fortunately, there are ways to achieve automated performance testing of iOS apps:
- Scheduling periodic Instruments checks.
- Implementing automated performance tests using the XCTest framework, which we'll focus on in this article.
Manual performance checks are not feasible, and an automated approach is necessary to regularly evaluate an app's performance.
Automated Performance Testing with XCTest
The XCTest framework features measure blocks that can be applied to any code within a test method to evaluate its performance. Here's how it works:
- The measure block executes the code multiple times, depending on the set number of invocations – the default being ten.
- Upon completion, it calculates the average value, which can be used as a baseline for future test executions. This baseline value can also be set manually.
- The performance test will fail if the average time exceeds the baseline.
Applying Performance Tests at the UI Level
Identifying suitable candidates for performance testing is a team-based decision. According to XCTest documentation, performance tests can be applied to any XCTest, regardless of the test level. This means that measure blocks can be utilized in both unit tests and UI tests. By applying performance tests to XCUItests, we can analyze which service or API responds faster or slower. While this approach has its advantages and disadvantages, it provides valuable insights into an app's performance.
For more information on ongoing quality assurance of iOS applications, visit https://carsnewstoday.com/programming/testing/ongoing-quality-assurance-of-ios-applications-via-xctest-driven-performance-evaluation/.
Assessing App Efficiency with Customizable Measurement Tools
To demonstrate this concept in a real-world scenario, let's develop a demo application, XCUItest-Performance, featuring a UI test target. Begin by adding a button to the main storyboard, assigning it the accessibility identifier "press" without linking it to any IBAction. At this point, we will have a functional demo app with a button. In the XCUItest template, create a test specifically designed for performance assessment, incorporating a measure block. Within this block, we can initiate the app launch sequence and simulate a button press. A typical test would resemble the following:
func evaluateAppPerformance() {
self.measure {
XCUIApplication().launch()
XCUIApplication().buttons["press"].tap()
}We can execute the test from Xcode and observe that it will iterate the code within the block ten times. During the initial test run, it will fail, prompting the developer to establish an appropriate benchmark value for future test executions. The test will subsequently pass or fail based on the comparison of the average value and the benchmark value.
Observe this in the GIF below:
Now that we have established the benchmark value for our future performance tests, these tests will run in tandem with our normal unit tests, failing if the average value exceeds the benchmark. This enables us to automatically test the performance of our code alongside unit test execution.
Refined Performance Tests with Measure Metrics
In the preceding test, we calculated the average time from the app launch until the user presses the button. However, we can delve deeper and specify the exact points from which to start and stop measuring performance. This allows us to apply performance tests to specific areas of the test code within the unit test by utilizing the start/stop method, achievable through the measure metric method.
Let’s apply performance tests solely when the button is pressed. We can launch the app normally and then employ the measure metric method to initiate and terminate performance measurement. We can add the test as follows:
func testperformancemeasuremetrics() {
xcuiapplication().launch()
self.measuremetrics([xctperformancemetric.wallclocktime], automaticallystartmeasuring: false) {
startmeasuring()
xcuiapplication().buttons["press"].tap()
stopmeasuring()
}
}Upon executing this test, we can observe that the application launches only once, yet we can calculate its performance by invoking it multiple times.
The measure metrics method offers a convenient means of optionally setting up start and end points for performance measurement during test execution, thereby providing valuable insights.
Performance Test Result Analysis
Xcode visually represents the results of performance tests, with a typical result resembling this:
The resulting dialogue reports average, baseline, and max stddev values, indicating that the performance test is 0.779% worse than the previous execution. These results can also be accessed in the console output when running performance tests from a continuous integration server.
Pros and Cons of XCTest as a Performance Test Tool
XCTest, a unit and UI level functional testing tool, is also capable of performing performance testing of iOS modules. However, there are certain advantages and disadvantages to using XCTest as a performance testing tool.
Advantages
Developers can benefit from using XCTest for performance testing in several ways:
- As a core Apple framework, XCTest eliminates the need to add third-party dependencies for performance testing, thereby streamlining the process.
- Unit tests written using XCTest can be easily extended to performance tests by adding measure blocks around code snippets, eliminating the need to learn additional performance test tools.
- Performance tests can be written using Swift, eliminating the need to learn additional languages.
- XCTest is easily pluggable to any continuous integration server, allowing for seamless execution using Xcodebuild or Fastlane Scan.
- It offers the ability to match the baseline using device types, eliminating the need to worry about comparing results with valid data sets.
- XCTest provides the ability to utilize the Xcode IDE for performance testing, making it a convenient option.
Disadvantages
Despite its benefits, using XCTest for performance testing also has some drawbacks:
- The performance test results generated by XCTest are not easily interpretable by non-technical individuals, and cannot be exported to a readable format.
- The performance test results cannot be shared or hosted easily, unlike other performance test tools.
- As the number of performance tests increases, the test suite becomes slower, necessitating separate execution from the development workflow.
- XCTest is incapable of simulating a large number of users to check the performance of an iOS app.
- XCTest cannot alter hardware device settings like CPU or memory to cover complex behaviors for performance tests.
Mastering the Art of Precise Performance Testing Strategies
In Xcode, performance tests are seamlessly integrated with unit and UI tests, courtesy of the XCTest framework. Consequently, it falls on the developer to ensure the test itself is correctly set up and functioning as intended. When establishing performance tests, several crucial considerations come into play:
Establish Clear Testing Parameters
Prior to embarking on performance testing, it's vital to define what and how specific code will be tested for performance. Identifying the level at which performance tests can be executed – whether at the unit, integration, or UI level – is essential. The XCTest framework allows us to execute tests at any level.
Eliminate Interference Between Invocations
Performance tests involve multiple iterations of the same code block. It's essential to ensure each invocation is self-contained and independent, without exhibiting significant deviations. Any deviations will result in an error report from Xcode.
Regulate Your Test Data
Predefined test data is more likely to produce reliable test results compared to random data when running a performance test. Random data can lead to inconsistent results, which are of little value.
Run Performance Tests Isolated in a Dedicated Scheme
As the number of performance tests increases, the time taken by test suites increases significantly. Configuring a separate scheme to run only performance tests, without affecting the normal flow of unit and UI tests, is highly recommended.
Run Performance Tests Continuously on a CI Server
Now that we have created a separate scheme for performance tests, it's relatively easy to configure with any continuous integration server. If you're using the Xcode server, it's even simpler. We can create a separate CI job to run the performance tests on a regular basis, such as daily or overnight.
Source Code for This Demo
The source code for the performance test discussed in this article is available on GitHub. You can clone the source code and try running performance tests yourself.
GitHub repo: xctest-performance
Conclusion
Apple has expanded the core XCTest framework to include performance testing, enabling us to detect performance bottlenecks in our iOS apps early in the development cycle. It's relatively easy for any iOS developer to get started with performance testing without needing to learn a new language or framework. With the latest Xcode, it's even smoother. We hope you'll try this yourself and integrate performance tests into your iOS apps soon!
Top comments (0)