Fundamentals of Testing

Testing is a process throughout the software development lifecycle that helps evaluate whether a component or system satisfies specified requirements, demonstrates that it is fit for purpose, and detects defects.

Common misconception: Testing is only executing tests. In reality, testing also includes planning, analysis, design, implementation, reporting on progress and results, and evaluating the quality of a test object.

Typical Testing Objectives

• Evaluating work products such as requirements, user stories, designs and code
• Triggering failures and finding defects
• Ensuring required coverage of a test object
• Reducing the level of risk of inadequate software quality
• Verifying whether specified requirements have been fulfilled
• Verifying that a test object complies with contractual, legal and regulatory requirements
• Providing information to stakeholders to allow them to make informed decisions
• Building confidence in the quality of the test object
• Validating whether the test object is complete and works as expected by the stakeholders

Testing vs Debugging

Testing can show that failures are caused by defects. Debugging is the development activity that finds, analyzes and fixes the defect. Subsequent confirmation testing checks whether the fix resolved the problem.

Testing is necessary because software is complex and humans are fallible. Rigorous testing reduces the probability of failures occurring during operation, and contributes to the quality of the test object.

Testing's Contributions to Success

• Involving testers in requirement reviews can find defects earlier (cheaper to fix)
• Involving testers with stakeholders helps build the right product
• Involving testers with developers raises understanding of the code and tests
• Verifying and validating delivered software reduces the level of risk

Testing and Quality Assurance (QA)

Quality management includes all activities that direct and control an organization with regard to quality. Quality assurance (QA) is process-oriented (right process → right product). Quality control (QC) is product-oriented and includes testing.

Errors, Defects, Failures and Root Causes

• Error (mistake): a human action that produces an incorrect result.
• Defect (bug, fault): an imperfection or deficiency in a work product, e.g. wrong code.
• Failure: an observable event where a component or system does not perform as required.
• Root cause: the source of a defect that, if removed, prevents the defect from recurring.

1. Testing shows the presence, not the absence, of defects. Testing reduces the probability of undiscovered defects but cannot prove correctness.
2. Exhaustive testing is impossible. Testing everything is not feasible except for trivial cases; risk analysis and prioritization should be used.
3. Early testing saves time and money. Defects found early are cheaper to fix (shift-left).
4. Defects cluster together. A small number of modules usually contain most of the defects (Pareto-style).
5. Tests wear out (pesticide paradox). Same tests repeated find fewer new defects; tests must evolve.
6. Testing is context dependent. Safety-critical software is tested differently from a mobile game.
7. Absence-of-errors is a fallacy. A system that meets specs but doesn't meet user needs is still a failure.

Testing is context dependent but always includes a similar set of activities. These are not strictly sequential — they may overlap, be combined or omitted depending on context.

The Test Activities

• Test planning — defining objectives and approach to meet objectives within constraints.
• Test monitoring & control — comparing actual progress with the plan and taking action.
• Test analysis — analyzing the test basis to identify testable features and test conditions.
• Test design — elaborating test conditions into test cases and other testware.
• Test implementation — preparing testware needed for execution (test data, environment, scripts).
• Test execution — running tests and comparing actual with expected results.
• Test completion — collecting data from completed activities, finalizing testware, archiving and lessons learned.

Test Work Products (Testware)

Test plan, test schedule, risk register, test conditions, test cases, test procedures, test data, test environment requirements, test logs, test progress reports, test summary reports, defect reports, change requests, test completion reports.

Traceability

Traceability between the test basis (e.g. requirements) and test work products supports impact analysis, coverage analysis and reporting. It provides confidence that requirements have been adequately covered.

Generic Skills Required for Testing

• Testing knowledge (to increase effectiveness)
• Thoroughness, carefulness, curiosity, attention to detail
• Good communication, active listening, being a team player
• Analytical and critical thinking, creativity
• Technical knowledge appropriate to the test object
• Domain knowledge to understand the users and their environment

The Whole Team Approach

All members of the team — developers, testers, BAs, POs, designers — are jointly responsible for quality. Testers are integrated into the team to share knowledge, design tests collaboratively, and increase ownership of quality.

Independence of Testing

A degree of independence makes a tester more effective at finding defects due to differences between the author's and the tester's cognitive biases. Levels range from "developer tests own code" to "an external organization tests".