Every industry has some degree of automation. So why is automated testing of digital health environment so rare? Why do we constantly see the testing phase of every EMR. EHR, or HIS implementation planned with thousands of hours of manual testing by clinicians, business analysts and IT professionals?
Among the concerns we hear are:
These are all myths.
Aversan’s test automation and test execution services can help free up your clinicians and business alanysts so they can do the work they are best suited for. Aversan creates a tailored environment using our Automated Test Framework (ATF) to automate testing of the manual test scripts that you now use, saving hours of cumbersome manual test labor. We also create variants of those scripts to broaden and deepen the test coverage which gives you even more confidence your digital health systems and connected medical devices are functioning as expected.
How do you make the decision to invest in test automation? It is the correct choice for some projects, not every project. Also, it is the preferred approach for certain aspects of a project, not every aspect. We believe there needs to be a business case for test automation, in other words a return on investment (ROI). One needs to make the case for the intial investment in the test automation methodology, tools and setup, which then generates a payback based on use.
In this article we will look at two types of ROI. One is the traditional financial ROI. The other ROI is more subjective, but likely much more important because of the human element. This ROI is based on an assurance that the system meets a level of acceptable quality – in other words patient safety.
The return on investment of test automation is very high. Several case studies from past and current clients support this claim. Very often the initial cost to create the automation environment is recovered after 6 to 7 full test runs. A typical organization will have dozens of test runs over a 2 to 3 year period. Numerous test runs are necessary once an organization has procured a new software system as they incrementally configure it and migrate in legacy data. Each incremental releae must not only test the new functionality but also re-test what was previously developed and validated. Then one needs to factor in releases that integrate iterations of these release and test cycles as well. Finally, before and after promoting the solution into production, the various vendors invovled may have software upgrades and patches that must be applied. Best practice requires performing a full regression test once again. Doing all those regression tests manually is just not feasible from cost, timeline or quality perspectives.
Let’s look at patient safety ROI case study. British Columbia deployed a lab reporting solution across the province connecting dozens of systems. Diagnostic test reporting is not something you want to get wrong. Whether is is an error in results transmission, an error in transformation of the raw data into standardized format, a report associated with the wrong patient or a report sent to the wrong clinician – any of these wil affect patient safety.
Aversan performed component, integration, performance, user interface and workflow verification services, 80% of which used automated testing. New releases were being introduced as often as every week during development the phase. There were nearly 1700 test scenarios automated during this phase (currently this number has grown to 2100 due to changes and enhancements to the application). Each major release of the system during development requires a pre- and post-candidate test confirmation run. This equates to roughly 3000 hours of manual testing for each release (as each run takes approximately 1500 man hours). Assuming the average fully loaded cost of testers (a blend of clinicians, business analysts, IT professionals) is $100/hour, manual validation of the system would cost about $300,000 per release.
The upfront investment required to implement the Automated Test Framework and regression test suite for this program was $1,500,000. The cost to execute the automated test suite is $2500 per run ($5000 per release). Using very modest number of 20 test runs, the cost to manually test would have been $6.0M but was only $1.6M with automation, roughly one quarter of the cost. the upfront investment in the test automation suite was break even by the sixth test run.
Another case study involves Manitoba Blue Cross and is a fantastic illustration of a financial ROI. The claims processing system processes approximately 250,000 claims per day, at an estimated value of $10-$50 million per day. These metrics show the magnitude and complexity of the system and its impact on the business and its clients. Prior to Aversan being engaged, the project team had only 120 test cases. Aversan broadened and deepened the test coverage to 5,500 test cases, and given the use of automation, for virtually no additional cost to each regression test. Defects and shortcomings in the implementation that were otherwise not detectable could now be identified. Thorough testing of all the test cases would be about 5,500 hours of people time. With the implementation of the ATF, the execution time for the complete suite of 5,500 test cases is 24 hours. Therefore, there was a significant savings in labor costs associated with manual testing. There was an increase in solution quality, reduced and predictable project timelines, and reduced costs and timelines for testing.
Besides the immediate results, the ROI continues even after the system is live. The Aversan ATF and automated test cases are left with the client along with training and support on its use.
In summary, test automation does have some upfront cost. But there is little or no impact on project timelines given the setup runs in parallel with other project streams. There is clearly an ROI after 6 to 7 complete regression tests. Futhermore, there is an undeniable increase in the quality of the system due to broader and deeper test coverage. The business case is easy to calculate financially. It is a bit more subjective when one factors in the patient safety ROI. But we can all agree that avoiding human hard or death due to software errors in the digital health system is priceless.
Even though MBSE can help companies reduce cost of testing and develop better quality products that address customer needs with reduced time to market, different engineering companies can look at MBSE in different ways based on their areas of expertise.
Aversan specializes in the design and development of mission and safety-critical avionics & test equipment, offering turn-key product development engineering services to our customers. whenever possible, Aversan applies Model Based Systems Engineering approach for developing our products within aerospace, defense and rail verticals.
Aversan has built multiple Controls Integration and Verification test facilities for its clients over the past decade in order to integrate and verify systems engineering specification controls requirements in an automated fashion where the controls components were simulated using real-time dynamic models. This capability saves us significant hours in the development process by allowing engineering teams simulate their use cases instead of physically integrating and testing every single component of the product under development.
If you’d like to explore how Aversan can help accelerate your product roadmap, reduce time-to-market and optimize the development and testing process for increased quality and efficiency, get in touch with us!
