AWS Device Farm is a mobile application performance testing application that provides real-time automated testing and reproduction of issues, simulating and testing issues that may occur on a variety of platforms (e.g. iPhone or Samsung mobile device, or multiple operations systems, etc).
$0.01
per instance minute
IBM DevOps Test Performance
Score 8.0 out of 10
N/A
IBM DevOps Test Performance helps software testing teams test earlier and more frequently by shifting testing left. IBM DevOps Test Performance validates the scalability of web and server applications, identifies the presence and cause of system performance bottlenecks and reduces load testing. Software testing teams can execute performance tests that analyze the impact of load on applications.
As long as you have the right scenarios in mind where you can use AWS Device Farm, you will be very happy. It's a cloud platform for testing automation support, where you can use real devices in multiple configs to validate your Android or iOS use cases. It does easily integrate with your CI pipeline, but reporting and UI are not perfect. A pain point is also the JUnit4 implementation, which could be more mature.
Go for IBM RPT if: 1. You're testing a Java-based Web application with HTTP protocol 2. You wanted to distribute the load across machines easily 3. Your team is in learning phase/not really introduced to a wide range of performance testing tools Do not go for IBM RPT if: 1. You wanted to test REST or any other advanced protocols 2. Your system under test demands a very high user load 3. Your application is written in .NET or any other platform except Java.
Memory utilization could have been improved.(Eats up system's RAM)! It may crash if a test is conducted with the heavy load if adequate RAM is not available in the VM/host machine.
Licensing could have been made simpler. IBM's licensing method is difficult to follow.
Support for protocols other than HTTP. Not really up to the current trend.
We had the enterprise support with AWS, so overall support experience was good with great engineers on the back providing answers. As you may know, overall AWS support is different and this is not different. Responses through the regular web support channel came easily, fast and accurate. We had questions/issues which were solved fast. Documentation is good as well, especially around the test automation pieces.
We haven't used anything like AWS Device Farm before. I am familiar with Amazon Web Services and when we had our MVP ready to test, we turned to AWS for a solution. AWS Device Farm was exactly what we were looking for as we have a really small team and limit resources.
Cost/Licensing: While JMeter is an opensource testing tool from Apache, compared to IBM RPT and HP LoadRunner, RPT is much cheaper than Loadrunner. Functionality:JMeter provides basic functionalities which are adequate for performance testing, however advanced features are not available (such as load testing with GUI, reporting is very basic etc.). But when it comes to Loadrunner, it offers very broad features and supports a variety of protocols. So in this category, Loadrunner is a winner, but RPT is better than JMeter. Ease of operating:JMeter is easy compared to LoadRunner, but it has old GUI and look and feel is not that great to understand. Also, most of the things are to be done in a command line, non-GUI mode. While LoadRunner is very advanced with many options, which also confusing sometimes. But RPT, on the other hand, maintains a balance between simplicity and offering of different features. So winner: RPT.
