Some common challenges in Product Testing:
- Product Road map is not clearly defined and we aren’t sure if automation is appropriate for us?
- In most ISVs & Software Enabled Businesses, developers double up as testers and there is no formal testing process in place
- QA is typically given a step-motherly treatment and is looked at only when schedule and time permits
- Though Test automation may be a good choice, but still is expensive and takes ages to justify the investments.
- Automation resources are top-heavy and would mean a lot of upfront costs.
These are some of the initial reactions that we get to hear whenever we think of Product Testing. These reactions aren’t unfounded and it merits careful assessment. This, if unaddressed most likely to result in increased rework, decreased customer satisfaction, decrease in profits, loss of business, increased bug fixing costs and product recalls. Testing can help ISVs & Software Enabled Businesses in removing all the pains & inhibitions listed above.
Some of the reasons that substantiate usage of testing are:
- Good software development practice calls for separating the development team from the testing team. This allows provision of unbiased view of the product quality and greater visibility of product issues.
- ISV’s can save up to 40% in overall cost by doing a proper testing
- Time to market can be greatly reduced with increased product quality, enhancing their competitive advantage
- Software product testing is a specialist skill, and cannot be accomplished by the developer community and can result in compromised product quality
Thus product testing is critical to ensure quality of product and test automation is a viable choice to ensure time to market & quality. Test automation is not about removing manual testers but to make use of their time better. It is possible to automate testing completely, and it has to be a mix of expertise from manual testers and automation testers
Some questions to consider before deciding to automate are:
- Is your product complex, fairly stable and do you have paying customers for its initial version?
- Are you following Agile Development? Do you overlook to thoroughly regression test before release?
- Does the product have a clearly defined Road map?
- Does every version upgrade, both minor and major warrant running all the functional tests over and over again? Essentially, the test scripts would be re-used multiple times justifying the investments?
- Does your product have a number of test cases that need to be validated?
- Do you maintain different versions for each of your customer?
If you answer yes to any of the above questions, then most likely, your product is a candidate for automation testing. Still, there are number of issues that need to be considered at the product level and also the larger objective of the organization before deciding to invest on automation.
If you have decided to automate testing, below are some practical tips for implementing any test automation initiative:
- Focus on the Framework/methodology and not the tool
A clearly defined automation framework/methodology that covers how the automation process will be conducted can eliminate most of the frustrations associated with automation by providing stakeholders with an upfront understanding. This would provide a fair idea on what is needed to automate tests, which includes tool selection as well as the rest of the automation process.
- Choose tools that are scalable to meet future needs and doesn’t gets you locked with them
Ensure that the tools can be customized to your needs and address the following: reusability, scalability, maintainability, visibility, measurability and manageability. Also, look at the support each tool has for scripting in different languages, where there are strengths available internally. Otherwise, you will need to organize an appropriate program that trains the team on the selected tools.
Keep in mind that no single tool will satisfy all the requirements; so choose a tool that meets the majority of the evaluation criteria.
- Perform a POC using the selected tool
Even if the tool appears to satisfy the evaluation criteria, it is advised to conduct a few test scenarios using the tool. Proof of concept (POC) should be done in such a way that the test scenarios cover an end to end business scenario, most of the controls and a few common features. POCs can be done with the evaluation versions of most tools and hence it is a fairly easier option to do.
- Test automation should not lead test design
Test design must be kept independent of test automation and they form the input for the automation discipline and not the other way around. Essentially, quality is the prerogative of the QA function and it has to be in control without worrying about the automation function and the problems faced by the automation team.
- Expect reasonable ROI
Test automation requires substantial investments upfront in terms of tool licenses(commercial) and creating automation scripts. It should be clear to all stakeholders that it will take time to see a defined return on these investments. However, there are intangibles such as improved quality and increased test coverage that can be observed in the short-term.