In the cutthroat world of iOS app development, speed is the name of the game. With users having no patience for slow and unresponsive apps, it's crucial to evaluate an app's performance before its release. A single misstep can lead to a poor user experience, ultimately affecting an app's success.
At Apple's 2014 Worldwide Developers Conference (WWDC), the tech giant introduced performance testing support for the XCTest framework. This framework allows developers to measure code performance within unit or UI tests. In this article, we'll explore the process of conducting automated performance testing using the XCTest framework, a game-changer in the world of iOS development.
The Evolution of Performance Testing in iOS
Apple provides a range of developer tools to assess the performance of iOS apps, including Instruments, Leaks, Profiler, and Allocations. These tools help detect performance issues, memory leaks, and other bottlenecks. However, traditional manual monitoring and execution of these checks are no longer viable in modern iOS development. With a single code change capable of rendering an app unresponsive, unstable, and unusable, an automated approach is necessary to regularly evaluate an app's performance. Fortunately, there are ways to achieve automated performance testing of iOS apps, such as:
- Scheduling periodic Instruments checks.
- Implementing automated performance tests using the XCTest framework, a more efficient approach.
In this article, we'll delve into automated performance tests using the XCTest framework, as regular Instruments checks are not a suitable solution due to the repetitive nature of the task. For more information on ongoing quality assurance of iOS applications, visit https://carsnewstoday.com.
Streamlining Performance Testing with XCTest
The XCTest framework features measure blocks that can be applied to any code within a test method to evaluate its performance. Here's how it works:
- The measure block executes the code multiple times, depending on the set number of invocations – the default being ten.
- Upon completion, it calculates the average value, which can be used as a baseline for future test executions. This baseline value can also be set manually.
- The performance test will fail if the average time exceeds the baseline.
Applying Performance Tests at the UI Level
Identifying suitable candidates for performance testing is a team-based decision. While XCTest documentation suggests that performance tests can be applied to any XCTest, regardless of the test level, we'll focus on XCUItests for this demo. This approach enables communication with all application endpoints, providing an opportunity to analyze which service or API responds faster or slower. Applying performance tests to UI tests has its advantages and disadvantages. While XCUItests cover diverse areas, the performance test may become exceedingly slow. In contrast, applying performance tests at the unit level yields quick results.
Assessing App Efficiency with Customizable Measurement Tools
To demonstrate this concept in a real-world scenario, let's develop a demo application, XCUItest-Performance, featuring a UI test target. Begin by adding a button to the main storyboard, assigning it the accessibility identifier "press" without linking it to any IBAction. At this point, we will have a functional demo app with a button. In the XCUItest template, create a test specifically designed for performance assessment, incorporating a measure block. Within this block, we can initiate the app launch sequence and simulate a button press. A typical test would resemble the following:
func evaluateAppPerformance() {
self.measure {
XCUIApplication().launch()
XCUIApplication().buttons["press"].tap()
}We can execute the test from Xcode and observe that it will iterate the code within the block ten times. During the initial test run, it will fail, prompting the developer to establish an appropriate benchmark value for future test executions. The test will subsequently pass or fail based on the comparison of the average value and the benchmark value.
Observe this in the GIF below:
Now that we have established the benchmark value for our future performance tests, these tests will run in tandem with our normal unit tests, failing if the average value exceeds the benchmark. This enables us to automatically test the performance of our code alongside unit test execution.
Refined Performance Tests with Measure Metrics
In the preceding test, we calculated the average time from the app launch until the user presses the button. However, we can delve deeper and specify the exact points from which to start and stop measuring performance. This allows us to apply performance tests to specific areas of the test code within the unit test by utilizing the start/stop method, achievable through the measure metric method.
Let’s apply performance tests solely when the button is pressed. We can launch the app normally and then employ the measure metric method to initiate and terminate performance measurement. We can add the test as follows:
func testperformancemeasuremetrics() {
xcuiapplication().launch()
self.measuremetrics([xctperformancemetric.wallclocktime], automaticallystartmeasuring: false) {
startmeasuring()
xcuiapplication().buttons["press"].tap()
stopmeasuring()
}
}Upon executing this test, we can observe that the application launches only once, yet we can calculate its performance by invoking it multiple times.
The measure metrics method offers a convenient means of optionally setting up start and end points for performance measurement during test execution, thereby providing valuable insights.
Performance Test Result Analysis
Xcode visually represents the results of performance tests, with a typical result resembling this:
The resulting dialogue reports average, baseline, and max stddev values, indicating that the performance test is 0.779% worse than the previous execution. These results can also be accessed in the console output when running performance tests from a continuous integration server.
Pros and Cons of XCTest as a Performance Test Tool
XCTest, a unit and UI level functional testing tool, is also capable of performing performance testing of iOS modules. However, there are certain advantages and disadvantages to using XCTest as a performance testing tool.
Advantages
Developers can benefit from using XCTest for performance testing in several ways:
- As a core Apple framework, XCTest eliminates the need to add third-party dependencies for performance testing, thereby streamlining the process.
- Unit tests written using XCTest can be easily extended to performance tests by adding measure blocks around code snippets, eliminating the need to learn additional performance test tools.
- Performance tests can be written using Swift, eliminating the need to learn additional languages.
- XCTest is easily pluggable to any continuous integration server, allowing for seamless execution using Xcodebuild or Fastlane Scan.
- It offers the ability to match the baseline using device types, eliminating the need to worry about comparing results with valid data sets.
- XCTest provides the ability to utilize the Xcode IDE for performance testing, making it a convenient option.
Disadvantages
Despite its benefits, using XCTest for performance testing also has some drawbacks:
- The performance test results generated by XCTest are not easily interpretable by non-technical individuals, and cannot be exported to a readable format.
- The performance test results cannot be shared or hosted easily, unlike other performance test tools.
- As the number of performance tests increases, the test suite becomes slower, necessitating separate execution from the development workflow.
- XCTest is incapable of simulating a large number of users to check the performance of an iOS app.
- XCTest cannot alter hardware device settings like CPU or memory to cover complex behaviors for performance tests.
Mastering Performance Testing: Expert Techniques for Optimal Results
In Xcode, performance tests are seamlessly integrated with unit and UI tests via the XCTest framework. Consequently, developers must ensure that their tests are properly configured and functioning as intended. When setting up performance tests, several crucial factors come into play:
Establish Clear Testing Objectives
Before initiating performance testing, it's vital to define what and how specific code will be evaluated for performance. Identifying the testing level – whether unit, integration, or UI – is also essential. Fortunately, the XCTest framework allows for testing at any level.
Prevent Interference Between Test Iterations
Performance tests involve executing multiple iterations of the same code block. Ensuring that each iteration is self-contained and independent, with minimal deviation, is critical. Xcode will flag any significant deviations as errors.
Optimize Test Data for Reliable Results
Predefined test data is more likely to yield consistent results compared to random data when running performance tests. Random data can produce erratic results, which are of limited value.
Execute Performance Tests in a Dedicated Scheme
As the number of performance tests grows, test suite execution time increases substantially. Configuring a separate scheme to run only performance tests, without impacting unit and UI tests, is a practical solution.
Integrate Performance Tests into Your CI Workflow
With a dedicated scheme in place, integrating performance tests into your continuous integration server is relatively straightforward. If you're using the Xcode server, the process is even simpler. Create a separate CI job to run performance tests regularly, such as daily or overnight.
Access the Demo Source Code
The source code for the performance test discussed in this article is available on GitHub. Clone the repository and try running performance tests yourself.
GitHub repo: xctest-performance
Conclusion
Apple's extension of the XCTest framework for performance testing enables developers to identify performance bottlenecks early in the development cycle. With the latest Xcode, getting started with performance testing is easier than ever. We encourage you to try this approach and integrate performance tests into your iOS apps soon!
Top comments (0)