Challenges in Software Testing

All software engineering areas face lot of challenges during execution. So, I would say, as a tester, never ever get surprised when you face challenges in software testing. But, it is also a hard fact for tester community that most of the companies are not testing oriented. Management always do a good appraisal of development teams. Sometimes, appreciation goes to development teams only.
Few points that I want highlight to management / project managers & development teams:

1. If development teams are so good / intelligent, then why you need testers to test the application.
2. Remember - by testing a build / release, testers evaluate the Quality of work done by developers.
3. During every regression testing, lot of bugs got re-opened (what quality of work was done by developers)
4. Sometimes, a customer release requires 5-7 regression testing cycles. Think Why?
5. During these 5 to 7 regression cycles, developers made so many mistakes. Testers help to prevent the defect leakage to customer. But, at the end, if even if 2-3 minor bugs goes to customer, everybody from top to bottom will catch testers. Just imagine, with one mistake of a tester, everybody will catch testers from the neck. Are testers bound to make not even a single mistake? But, developers can repeat the same mistakes multiple times.
6. Most of the times, developers eat tester's time.
7. I never understand why management always keep a difference between developers and testers. You might be having answer to this question, but, I'm not.

How to do System Testing

Testing the software system or software application as a whole is referred to as System Testing of the software. System testing of the application is done on complete application software to evaluate software's overall compliance with the business / functional / end-user requirements. The system testing comes under black box software testing. So, the knowledge of internal design or structure or code is not required for this type of software testing.

In system testing a software test professional aims to detect defects or bugs both within the interfaces and also within the software as a whole. However, the during integration testing of the application or software, the software test professional aims to detect the bugs / defects between the individual units that are integrated together.

During system testing, the focus is on the software design, behavior and even the believed expectations of the customer. So, we can also refer the system testing phase of software testing as investigatory testing phase of the software development life cycle.

At what stage of SDLC the System Testing comes into picture:

After the integration of all components of the software being developed, the whole software system is rigorously tested to ensure that it meets the specified business, functional & non-functional requirements. System Testing is build on the unit testing and integration testing levels. Generally, a separate and dedicated team is responsible for system testing. And, system testing is performed on stagging server.

Why system testing is required:

• It is the first level of software testing where the software / application is tested as a whole.
• It is done to verify and validate the technical, business, functional and non-functional requirements of the software. It also includes the verification & validation of software application architecture.
• System testing is done on stagging environment that closely resembles the production environment where the final software will be deployed.

Entry Criteria for System Testing:

• Unit Testing must be completed
• Integration Testing must be completed
• Complete software system should be developed
• A software testing environment that closely resembling the production environment must be available (stagging environment).

System Testing in seven steps:

1. Creation of System Test Plan
2. Creation of system test cases
3. Selection / creation of test data for system testing
4. Software Test Automation of execution of automated test cases (if required)
5. Execution of test cases
6. Bug fixing and regression testing
7. Repeat the software test cycle (if required on multiple environments)  

Contents of a system test plan: The contents of a software system test plan may vary from organization to organization or project to project. It depends how we have created the software test strategy, project plan and master test plan of the project. However, the basic contents of a software system test plan should be:

- Scope
- Goals & Objective
- Area of focus (Critical areas)
- Deliverables
- System testing strategy
- Schedule
- Entry and exit criteria
- Suspension & resumption criteria for software testing
- Test Environment
- Assumptions
- Staffing and Training Plan
- Roles and Responsibilities
- Glossary

How to write system test cases: The system test cases are written in a similar way as we write functional test cases. However, while creating system test cases following two points needs to be kept in mind:

- System test cases must cover the use cases and scenarios
- They must validate the all types of requirements - technical, UI, functional, non-functional, performance etc.

As per Wikipedia, there are total of 24 types of testings that needs to be considered during system testing. These are:

GUI software testing, Usability testing, Performance testing, Compatibility testing, Error handling testing, Load testing, Volume testing, Stress testing, User help testing, Security testing, Scalability testing, Capacity testing, Sanity testing, Smoke testing, Exploratory testing, Ad hoc testing, Regression testing, Reliability testing, Recovery testing, Installation testing, Idem potency testing, Maintenance testing, Recovery testing and failover testing, Accessibility testing

The format of system test cases contains:

• Test Case ID - a unique number
• Test Suite Name
• Tester - name of tester who execute of write test cases
• Requirement - Requirement Id or brief description of the functionality / requirement
• How to Test - Steps to follow for execution of the test case
• Test Data - Input Data
• Expected Result
• Actual Result
• Pass / Fail
• Test Iteration

Difference between Smoke & Sanity Software Testing

Smoke Testing: Software Testing done to ensure that whether the build can be accepted for through software testing or not. Basically, it is done to check the stability of the build received for software testing.

Sanity testing: After receiving a build with minor changes in the code or functionality, a subset of regression test cases are executed that to check whether it rectified the software bugs or issues and no other software bug is introduced by the changes. Sometimes, when multiple cycles of regression testing are executed, sanity testing of the software can be done at later cycles after through regression test cycles. If we are moving a build from staging / testing server to production server, sanity testing of the software application can be done to check that whether the build is sane enough to move to further at production server or not.

Difference between Smoke & Sanity Software Testing:

• Smoke testing is a wide approach where all areas of the software application are tested without getting into too deep. However, a sanity software testing is a narrow regression testing with a focus on one or a small set of areas of functionality of the software application.
• The test cases for smoke testing of the software can be either manual or automated. However, a sanity test is generally without test scripts or test cases.
• Smoke testing is done to ensure whether the main functions of the software application are working or not. During smoke testing of the software, we do not go into finer details. However, sanity testing is a cursory software testing type. It is done whenever a quick round of software testing can prove that the software application is functioning according to business / functional requirements.
• Smoke testing of the software application is done to check whether the build can be accepted for through software testing. Sanity testing of the software is to ensure whether the requirements are met or not.

Test Entry Criteria and Test Exit Criteria

Test Entry Criteria

Test Planning Deliverables

1. Test Plan/Specification approved

2. Test Resources available

Code Readiness

1. Code complete for all requirements

2. Code changes migrated to environment and verified

3. Unit/Development testing activities completed and results available

Testing Environment Readiness

1. Environment and infrastructure properly installed and/or set up

2. Environment validation activity should be completed

3. No SEV1 or SEV2 defects from environment validation activity

Test Preparation Deliverables

1. Test Cases completed and available for review

2. Test Schedule completed

3. Test Management Tool set up with project’s information

4. Metrics Portal set up with project's information

Test Exit Criteria

Based on Defect Trend

1. No SEV1 or SEV2 defects - All the SEV1 or SEV2 defects MUST be in “Closed” , “Canceled”, “Duplicate” or “Rejected” status in the Test Management Tool. There should not be any Ready to Verify/Ready to Promote defects pending.        

2. All SEV3 and SEV4 defects are deferred - All the SEV3 and SEV4 defects which are not closed, should be moved to Deferred status.

Based on Testing Strategy

1. 100% test cases attempted - Test Scope is completed according to the Test Specification.

2. Above 90% planned test cases should be passed.

3. Failed test cases must have defect associated with it and defect severity should not be above SEV3 or SEV4.

Based on Evidences

1. Test Sign-off assessment result should be submitted for audit.

2. Test result should be available in Test management tool. 

What is the Difference between Verification & Validation?

Verification: Are we building the produce right?

It’s a static process which involves evaluation of project documents, requirements, code. Checklists are used to do the verification.

 Validation: Are we building a right Product?

Validation is actual testing where the product itself is tested with set of inputs and verify that it is working fine as per the requirement.

What is system testing?

After integrating all the modules the software is tested whether the entire system is working properly. All functional and non functional testing should be done under system testing. The software should be work as defined in the requirement document. 

Below mentioned testing types are considered under system testing:

Installation testing

Load testing

Performance testing

Smoke testing

Ad hoc testing

Regression testing

Exploratory testing

Security testing

What is the difference between static testing and dynamic testing?

Static Testing: In static testing is review the process using checklist, it involves code review, walkthrough, inspection. Static testing will be done before dynamic testing. The actual application or software will not be tested in this process.  

Eg. Code review

Dynamic Testing: In Dynamic testing the actual application is tested. Test cases are executed in this stage to find the defect.

Eg. Functional testing, system testing, integration testing etc