In this post, I'm going to describe techniques and strategies for
software testing. Techniques cover different ways testing can be
accomplished. Testing techniques can be defined in three ways:
Preparation, Execution and Approach.
Preparation: From preparation point of view there are two testing techniques: Formal Testing and Informal Testing.
Formal Testing: Testing performed with a plan, documented set of
test cases, etc that outline the methodology and test objectives. Test
documentation can be developed from requirements, design, equivalence
partitioning, domain coverage, error guessing, etc. The level of
formality and thoroughness of test cases will depend upon the needs of
the project. Some projects can have rather informal ‘formal test cases’,
while others will require a highly refined test process. Some projects
will require light testing of nominal paths while others will need
rigorous testing of exceptional cases.
Informal Testing: Ad hoc testing performed without a documented
set of objectives or plans. Informal testing relies on the intuition and
skills of the individual performing the testing. Experienced engineers
can be productive in this mode by mentally performing test cases for the
scenarios being exercised.
From the execution point of view, the two testings types are: Manual Testing and Automated Testing.
Manual Testing: Manual testing involves direct human interaction
to exercise software functionality and note behavior and deviations from
expected behavior.
Automated Testing: Testing that relies on a tool, built-in test
harness, test framework, or other automatic mechanism to exercise
software functionality, record output, and possibly detect deviations.
The test cases performed by automated testing are usually defined as
software code or script that drives the automatic execution.
From the testing approach point of view, the two testings types are: Structural Testing and Functional Testing.
Structural Testing: Structural testing depends upon knowledge of
the internal structure of the software. Structural testing is also
referred to as white-box testing.
Data-flow Coverage: Data-flow coverage tests paths from the definition of a variable to its use.
Control-flow Coverage
Statement Coverage: Statement coverage requires that every statement in the code under test has been executed.
Branch Coverage: Branch coverage requires that every point of
entry and exit in the program has been executed at least once, and every
decision in the program has taken all possible outcomes at least one.
Condition Coverage: Condition coverage is branch coverage with
the additional requirement that “every condition in a decision in the
program has taken all possible outcomes at least once.” Multiple
condition coverage requires that all possible combinations of the
possible outcomes of each condition have been tested. Modified condition
coverage requires that each condition has been tested independently.
Functional Testing: Functional testing compares the behavior of
the test item to its specification without knowledge of the item’s
internal structure. Functional testing is also referred to as black box
testing.
Requirements Coverage: Requirements coverage requires at least
one test case for each specified requirement. A traceability matrix can
be used to insure that requirements coverage has been satisfied.
Input Domain Coverage: Input domain coverage executes a function
with a sufficient set of input values from the function’s input domain.
The notion of a sufficient set is not completely definable, and complete
coverage of the input domain is typically impossible. Therefore the
input domain is broken into subsets, or equivalence classes, such that
all values within a subset are likely to reveal the same defects. Any
one value within an equivalence class can be used to represent the whole
equivalence class. In addition to a generic representative, each
extreme value within an equivalence class should be covered by a test
case. Testing the extreme values of the equivalence classes is referred
to as boundary value testing.
Output Domain Coverage: Output domain coverage executes a
function in such a way that a sufficient set of output values from the
function’s output domain is produced. Equivalence classes and boundary
values are used to provide coverage of the output domain. A set of test
cases that “reach” the boundary values and a typical value for each
equivalence class is considered to have achieved output domain coverage.
Various Software Testing Levels
Although many testing levels tend to be combined with certain
techniques, there are no hard and fast rules. Some types of testing
imply certain lifecycle stages, software deliverables, or other project
context. Other types of testing are general enough to be done almost any
time on any part of the system. Some require a particular methodology.
When appropriate common utilizations of a particular testing type will
be described. The project’s test plan will normally define the types of
testing that will be used on the project, when they will be used, and
the strategies they will be used with. Test cases are then created for
each testing type.
Unit Testing: A unit is an abstract term for the smallest thing
that can be conveniently tested. This will vary based on the nature of a
project and its technology but usually focuses at the subroutine level.
Unit testing is the testing of these units. Unit testing is often
automated and may require creation of a harness, stubs, or drivers.
Component Testing: A component is an aggregate of one or more
components. Component testing expands unit testing to include called
components and data types. Component testing is often automated and may
require creation of harness, stubs, or drivers.
Single Step Testing: Single step testing is performed by stepping
through new or modified statements of code with a debugger. Single step
testing is normally manual and informal.
Bench Testing: Bench testing is functional testing of a component
after the system has been built in a local environment. Bench testing
is often manual and informal.
Developer Integration Testing: Developer integration testing is
functional testing of a component after the component has been released
and the system has been deployed in a standard testing environment.
Special attention is given to the flow of data between the new component
and the rest of the system.
Smoke Testing: Smoke testing determines whether the system is
sufficiently stable and functional to warrant the cost of further, more
rigorous testing. Smoke testing may also communicate the general
disposition of the current code base to the project team. Specific
standards for the scope or format of smoke test cases and for their
success criteria may vary widely among projects.
Feature Testing: Feature testing is functional testing directed
at a specific feature of the system. The feature is tested for
correctness and proper integration into the system. Feature testing
occurs after all components of a feature have been completed and
released by development.
Integration Testing: Integration testing focuses on verifying the
functionality and stability of the overall system when it is integrated
with external systems, subsystems, third party components, or other
external interfaces.
System Testing: System testing occurs when all necessary
components have been released internally and the system has been
deployed onto a standard environment. System testing is concerned with
the behavior of the whole system. When appropriate, system testing
encompasses all external software, hardware, operating environments,
etc. that will make up the final system.
Release Testing: Release tests ensure that interim builds can
successfully deployed by the customer. This includes product deployment,
installation, and a pass through the primary functionality. This test
is done immediately before releasing to the customer.
Beta Testing: Beta testing consists of deploying the system to
many external users who have agreed to provide feedback about the
system. Beta testing may also provide the opportunity to explore release
and deployment issues.
Acceptance Testing: Acceptance testing compares the system to a
predefined set of acceptance criteria. If the acceptance criteria are
satisfied by the system, the customer will accept delivery of the
system.
Regression Testing: Exercises functionality that has stabilized.
Once high confidence has been established for certain parts of the
system, it is generally wasted effort to continue rigorous, detailed
testing of those parts. However, it is possible that continued evolution
of the system will have negative effects on previously stable and
reliable parts of the system. Regression testing offers a low-cost
method of detecting such side effects. Regression testing is often
automated and focused on critical functionality.
Performance Testing: Performance testing measures the efficiency
with respect to time and hardware resources of the test item under
typical usage. This assumes that a set of non-functional requirements
regarding performance exist in the item’s specification.
Stress Testing: Stress testing evaluates the performance of the
test item during extreme usage patterns. Typical examples of “extreme
usage patterns” are large data sets, complex calculations, extended
operation, limited system resources, etc.
Configuration Testing: Configuration testing evaluates the
performance of the test item under a range of system configurations.
Relevant configuration issues depend upon the particular product and may
include peripherals, network patterns, operating systems, hardware
devices and drivers, user settings.
