Chapter 6: Tools Support for Testing
In this chapter we will discuss on the following topics.
- 6.1 Types of Test Tools (U)
- 6.2 Effective Use of Tools: Potential Benefits and Risks (U)
- 6.3 Introducing a Tool into an Organization (U)
6.1 Types of Test Tools
There are many types of test tools such as incident management tools, debugging tools, coverage tools etc.
6.1.1 Tool Support for Testing
Test tools can be used for one or more activities that support testing. These include:
- Tools that are used in exploration For example: tools that monitor file activity for an application.
- Any tool that aids in testing ( Ex: spreadsheet )
- Tools that are directly used in testing such as test execution tools and result comparison tools
- Tools that help in managing the testing process such as those used to manage tests, test results, data, requirements, incidents, defects, etc., and for reporting and monitoring test execution
Tool support for testing can have one or more of the following purposes depending on the context:
- Automate activities that cannot be executed manually (Example: large scale performance testing of client-server applications)
- Increase reliability of testing (Example:, by automating large data comparisons or simulating behavior)
- Improve the efficiency of test activities by automating repetitive tasks or supporting manual test activities like test planning, test design, test reporting, and monitoring
- Automate activities that require significant resources when done manually
The term “test frameworks “ is also frequently used in the industry, in at least three meanings:
- Overall process of execution of testing
- A type of design of test automation (Example: data-driven, keyword-driven)
- Reusable and extensible testing libraries that can be used to build testing tools (called test harnesses as well)
6.1.2 Test Tool Classification
There are a number of tools that support different aspects of testing. Tools can be classified based on several criteria such as purpose, commercial/free/open-source/shareware, technology used and so forth.Here we classify tools according to the testing activities that they support.
Some tools clearly support one activity; others may support more than one activity, but are classified under the activities with which they are most closely associated. Tools from a single provider, especially those that have been designed to work together, may be bundled into one package.
Some type of test tools can be intrusive, which means that they can affect the actual outcome of the test. For example, The actual timing may be different due to the extra instructions that are executed by the tool, or you may get a different measure of code coverage. The consequence of intrusive tools is called the probe effect.
Some tools offer support more appropriate for developers For example: tools that are used during component and component integration testing.
6.1.3 Tool Support for Management of Testing and Tests
Management tools apply to all test activities over the entire software life cycle.
Test Management Tools
These tools provide interfaces for executing tests, tracking defects and managing requirements, along with support for quantitative analysis and reporting of the test objects. They also support tracing the test objects to requirement specifications and might have an independent version control capability or an interface to an external one.
Requirements Management tools
These tools store requirement statements, store the attributes for the requirements such as priority, provide unique identifiers and support tracking the requirements to individual tests. These tools may also help with identifying inconsistent or missing requirements.
Incident Management tools (Defect Tracking Tools)
These tools store and manage incident reports, i.e., defects, failures, change requests or perceived problems and anomalies, and help in managing the life cycle of incidents, optionally with support for statistical analysis.
Configuration Management tools
Although not strictly test tools, these are necessary for storage and version management of testware and related software especially when configuring more than one hardware/software environment in terms of operating system versions, compilers, browsers, etc.
6.1.4 Tool Support for Static Testing
Static testing tools provide a cost effective way of finding more defects at an earlier stage in the development process.
Review Tools
These tools assist with review processes. Checklists, review guidelines and are used to store and communicate review comments and report on defects and effort. They can be of further help by providing aid for online reviews for large or geographically dispersed teams.
Static Analysis Tools
These tools are mostly used by developers. These tools help developers and testers find defects prior to dynamic testing by providing support for enforcing coding standards, analysis of structures and dependencies. They can also help in planning or risk analysis by providing metrics for the code.
Modeling Tools
This tool also mostly used by developers. These tools are used to validate software models For example: physical data model (PDM) for a relational database, by enumerating inconsistencies and finding defects. These tools can often aid in generating some test cases based on the model.
6.1.5 Tool Support for Test Specification
Test Design Tools
These tools are used to generate test inputs or executable tests and/or test oracles from requirements, graphical user interfaces, design models or code.
Test Data preparation Tools
Test data preparation tools manipulate databases, files or data transmissions to set up test data to be used during the execution of tests to ensure security through data anonymity.
6.1.6 Tool Support for Test Execution and Logging
Test Execution Tools
These tools enable tests to be executed automatically, or semi-automatically, using stored inputs and excepted outcomes, through the use of scripting language and usually provide a test log for each test run. They can also be used to record tests, and usually support scripting languages or GUI-based configuration for parameterization of data and other customization in the tests.
Test harness or unit test framework tools
This tools helps in testing component or a part of system by simulating the environment in which test objects will run, through the provision of mock objects as stubs or drivers.
Test comparators
Test comparators determine the difference between files, databases or test results. A test comparator may use test oracle.
Coverage Measurement Tools
Test tools measure the percentage of specific type of code structure that have been exercised by a set of tests. This tool is used by developers.
Security Testing Tools
These tools are used to evaluate the security characteristics of a software under test. This includes evaluating the ability of the software to protect data confidentiality, integrity, authentication, authorization, availability and non-repudiation.
6.1.7 Tool Support for Performance and Monitoring
Dynamic Analysis Tools:
Dynamic analysis tools find defects that are evident only when software is executing, such as time dependencies or memory leaks.
Performance Testing/Stress Testing /Load Testing Tools
Performance testing tools monitor and report on how a system behaves under a variety of simulated usage conditions in terms of number of concurrent users, their ramp-up pattern, frequency and relative percentage of transactions. The simulation of load is achieved by means of creating virtual users carrying out out a selected set of transactions, spread across various test machines commonly known as load generators.
Monitoring Tools
Monitoring tools continuously analyze, verify and report on usage of specific system resources, and give warnings of possible service problems.
6.1.8 Tool Support for Specific Testing Needs
Data Quality Assessment
Data is at the center of some projects such as migration and data conversion projects and applications like data warehouses and its attributes can very in terms of criticality and volume. In such cases, tools need to be employed for data quality assessment to review and verify the data conversion and migration rules to ensure that the processed data is correct, complete and complies with a pre-defined context-specific standard.
6.2 Effective Use of Tools: potential Benefits and Risks
6.2.1 Potential Benefits and Risks of Tool Support for Testing
Simply purchasing or leasing a tool does not guarantee success with that tool. Each type of tool may require additional effort to achieve real and lasting benefits. There are potential benefits and opportunities with the use of tools in testing , but there are also risks.
Potential benefits of using tools include:
- Ease of access to information about tests or testing. For example: statistics and graphs about test progress, incident rates and performances.
- Repetitive work is reduced. For example: running regression tests, re-entering the same test data, and checking against coding standards.
- Greater consistency and repeat-ability. For example: tests executed by a tool in the same order with the same frequency, and tests derived from requirements.
Risks of using tools include:
- Underestimating the time, cost and effort for the initial introduction of a tool.
- Unrealistic expectations of the tool, such as functionality and ease of use.
- Underestimating the effort required to maintain the test assets generated by the tool.
- Underestimating the time and effort needed to achieve significant and continuing benefits from the tool, including the need for changes in the testing process and continuous improvement of the way the tool is used.
- Neglecting version control of test assets within the tool.
- Over-reliance on the tool.
- Risk of tool vendor going out of business, retiring the tool, or selling the tool to a different vendor.
- Poor response from vendor for support, upgrades, and defect fixes.
- Risk of suspension of open-source/ free tool project.
- Unforeseen such as the inability to support a new platform.
- Neglecting relationships and interoperability issues between critical tools, such as requirements management tools, version control tools, incident management tools, defect tracking tools and tools from multiple vendors.
6.2.2 Special Considerations for Some Types of Tools
Test Execution Tools
Test execution tools execute test objectives using automated test scripts. This type of tool often requires significant effort in order to achieve significant benefits.
Capturing tests by recording the actions of a manual tester seems attractive, but this approach does not scale to large numbers of automated test scripts. A captured script is a linear representation with specific data and actions as part of each script. This type of script may be unstable when unexpected events occur.
A data-driven testing approach separates out the test inputs (the data), usually into a spreadsheet, and uses a more generic test script that can read the input data and execute the same test script with different data. Testers who are not familiar with the scripting language can then create the test data for these predefined scripts.
There are other techniques employed in data-driven techniques. Where instead of hard-coded data combinations placed in a spreadsheet, data is generated using algorithms based on configurable parameters at run time and supplied to the application. For example, a tool may use an algorithm, which generates a random user ID, and for repeatability in pattern, a seed is employed for controlling randomness.
In a keyword-driven testing approach, the spreadsheet contains keywords describing the actions to be taken (also called as action words), and test data. Testers even if they are not familiar with the scripting language can then define tests using the keywords, which can be tailored to the application being tested.
Technical expertise in the scripting language is needed for all approaches.
Regardless of the scripting technique used, the expected results for each test need to be stored for later comparison.
Static Analysis Tools
Static analysis tools applied to source code can enforce coding standards, but if applied to existing code may generate a large quantity of messages. Warning messages do not stop the code from being translated into an executable program, but ideally should be addressed so that maintenance of the code is easier in the feature. A gradual implementation of the analysis tool with initial filters to exclude some messages is an effective approach.
Test Management Tools
Test management tools need to interface with other tools or spreadsheets in order to produce useful information in a format that fits the needs of the organization.
6.3 Introducing a Tool into an Organization
The main considerations in selecting a tool for an organization include:
- identify changes needed to that infrastructure to effectively use the tool
- Evaluation of the vendor or service support suppliers in case of non-commercial tools
- Identification of internal requirements for coaching and mentoring in the use of the tool
- Evaluation of training needs considering the current test team’s test automation skills
- Estimation of a cost-benefit ratio based on a concrete business case
- Assessment of organizational maturity, strengths and weaknesses and identification of opportunities for an improved test process supported by tools
- Evaluation against clear requirements and objective criteria
- A proof-of-concept, by using a test tool during the evaluation phase to establish whether it performs effectively with the software under test and within the current infrastructure
Introducing the selected tool into an organization starts with a pilot project, which has the following objectives:
- Evaluate how the tool fits with existing process and practices, and determine what would need to change
- Decide on standard ways of using, managing, storing and maintaining the tool and the test assets
- Assess whether the benefits will be achieved at reasonable cost
- Learn more detail about the tool
Success factors for the deployment of the tool within an organization include:
- Implementing a way to gather usage information from the actual use
- Monitoring tool use and benefits
- Providing support for the test team for a given tool
- Gathering lessons learned from all teams
- Rolling out the tool to the rest of the organization incrementally
- Adapting and improving processes to fit with the use of the tool
- Providing training and coaching/mentoring for new users
- Defining usage guidelines
Posts
