Testing terminology

definitions, terms, vocabularies and acronyms used in testing

This article is not intended to tell a story, but rather serves as a reference for ones who seek knowledge or reminder related to testing. I build this knowledge base to be a strong foundation of different software testing approaches, methods and activities. You may use it to help you understand the testing area or to take your next steps toward a career in software testing. For myself, its serves as a reference when I speak with my peers and mostly for my articles.

This reference is used by the following articles :

• Testing Strategy — toward Agility, a guideline to define testing strategy that fits in an Agile environment
• Automated Quality Gate — a rail guard to support the continuous quality, a first step to improve the code quality

Test Levels

Diamond of Test | Pyramid of Test

There are 5 levels of test show on the pyramid of test :

• End to End Testing is the level in the software testing process where a product is given the green light or not.
• System Integration Testing is about the whole system under test. Here we may have interaction with external system or using some stubbing or fake system.
• Components Integration Testing is meant to test multiple components integration together, usually in the same functional scope (composition of component, e.g. few micro-services together as one), but this is still under test in isolation within that functional scope.
• Component testing is also called as module testing. The whole component (composition of classes, functions and procedures) is tested. If we are talking about the frontend, it is commonly known as E2E testing, but in fact it is a component from a system point of view.
• Unit testing aims to verify each part of the software in isolation and then perform tests to demonstrate that each individual part of code (like functions, methods, classes, interfaces and procedures) is correct.

Frontend vs Backend

Frontend vs Backend

Pyramid of Test

Original Pyramid of Test invented by Mike Cohn

The “Pyramid of Test” is a concept developed by Mike Cohn. The pyramid is an essential part to structure a software test strategy. The idea is to break automated tests into three levels : unit, integration, and E2E (with UI). That would help developers and QA specialists to optimise their efforts in building software with higher quality. Visually, it suggests to have wider coverage at the bottom and more exhaustive tests as it is cheaper to write and run unit tests. At the opposite, E2E tests are ways more heavier and costly to design, implement and run.

Diamond of Test

Original Diamond of Test

The “Diamond of Test” is a variation of the “Pyramid of Test”. It serves the same purpose, but suggests to focus more on the integrations. The emergence of microservices architecture requires our test strategy to be adapted to components’ characteristics. Most of components are quite small so that trying to cover exhaustively them with unit tests become more painful than they bring added value. The pragmatic way to do testing is to focus more on the integration tests, because most business requirements are responded with business processes and rules that are powered by multiples components.

As testing is not only about technical matter, and we aim more on functional testing, it make senses to try to have more testing on the mid-level. However, it does not mean we should have less or a lot less on unit level. As a matter of fact, unit and integration are complementary.

Test Types

There more than a hundred of test types, and each has its own naming and meaning :

• Unit Testing : Unit testing is the lowest level, glued to the source code of your application. They consist in testing individual methods and functions of the classes, components or modules used by your software. Unit tests are in general quite cheap to automate and can be run very quickly by a continuous integration server.
• Service Test : There is no much difference between Component and Service test as a server may be a component. On a multi-layer architecture, we could refer this to the service layer, so any service should be tested. On a Service Oriented Architecture, any service should be tested. On a Web Service Architecture, any web service must tested, and here a web service expose an API, so finally it’s an API test.
• API Test : On a Web Service Architecture, any web service must tested, and here a web service expose an API. If it’s not a web based API, API would likely be exposed on another protocol (websocket, TCP, UDP, etc.), so it doesn’t matter what the protocol is, the API should be tested.
• Component Tests : It is also called as module testing. The basic difference between the unit testing and component testing is in unit testing the developers test their piece of code but in component testing the whole component (composition of classes, functions and procedures) is tested.
• Components Integration Tests : is done when modules or components are integrated together. This type of testing is basically done to ensure that the code should not break after integrating the two or many modules together. Can be using either Black Box Testing or White Box Testing approach.
• System Integration Tests : is a type of testing where we basically test all the related systems together in the same environment because they should maintain data integrity and must operate in coordination with other systems as a whole.
• End-to-end testing : aim to replicate a user behaviour with the software in a complete application environment. It verifies that various user flows work as expected and can be as simple as loading a web page or logging in or much more complex scenarios verifying email notifications, online payments, etc…
  End-to-end tests are very useful, but they’re expensive to perform and can be hard to maintain when they’re automated. It is recommended to have a few key end-to-end tests and rely more on lower level types of testing (unit and integration tests) to be able to quickly identify breaking changes.
• User Acceptance Tests : are formal tests executed to verify if a system satisfies its business requirements. They require the entire application to be up and running and focus on replicating user behaviours. But they can also go further and measure the performance of the system and reject changes if certain goals are not met.
• Technical Acceptance Tests : Acceptance tests are formal tests executed to verify if a system satisfies its business requirements. They require the entire application to be up and running and focus on replicating user behaviours. But they can also go further and measure the performance of the system and reject changes if certain goals are not met.
• Exploratory Testing : Exploratory Testing is an approach that is concisely described as simultaneous learning, test design and test execution. While the software is being tested, the tester learns things that together with experience and creativity generates new good tests to run. Exploratory testing is often thought of as a black box testing technique, and all of this is done manually at first.
• UI tests : The objective of this UI testing is to validate the UI as per the business requirement. The expected UI of the application is mentioned in the Detailed Design Document and UI mockup screens.
  The UI testing includes the size of the buttons and input field present on the screen, alignment of all text, tables and content in the tables.
  It also validates the menu of the application, after selecting different menu and menu items, it validates that the page does not fluctuate and the alignment remains same after hovering the mouse on the menu or sub-menu.
• Functional Tests : focus on the business requirements of an application. They only verify the output of an action and do not check the intermediate states of the system when performing that action.
  There is sometimes a confusion between integration tests and functional tests as they both require multiple components to interact with each other. The difference is that an integration test may simply verify that you can query the database while a functional test would expect to get a specific value from the database as defined by the product requirements.
• Sanity Checks : is done to determine if a new software version is performing well enough to accept it for a major testing effort or not. If an application is crashing for the initial use then the system is not stable enough for further testing. Hence a build or an application is assigned to fix it.
• Regression Tests : Testing an application as a whole for the modification in any module or functionality is termed as Regression Testing. It is difficult to cover all the system in Regression Testing, so typically automation testing tools are used for these types of testing.
• Black Box Testing : Internal system design is not considered in this type of testing. Tests are based on the requirements and functionality.
  Detailed information about the advantages, disadvantages, and types of Black box testing can be seen here.
• White Box Testing : White Box testing is based on the knowledge about the internal logic of an application’s code.
  It is also known as Glass box Testing. Internal software and code working should be known for performing this type of testing. Under this tests are based on the coverage of code statements, branches, paths, conditions etc.
• Smoke Tests : a Smoke testing are a subset of test cases that cover the most important functionality of a component or system. It is used to take preliminary measure to reveal simple failures, but severe enough. We would run smoke tests on each new build of a product to verify that the build is testable before the build is released. Is the build stable or not ?
  Smoke tests, also known as “Build Verification Test”, are basic tests, thus non-exhaustive set of tests that check basic the most important functionalities of the application. They are meant to be quick to execute, and their goal is to give you the assurance that the major features of your system are working as expected. The result of this testing is used to decide if a build is stable enough to proceed with further testing.
  Smoke tests can be useful right after a new build is made to decide whether or not you can run more expensive tests, or right after a deployment to make sure that the application is running properly in the newly deployed environment. A Smoke test can be performed manually or can be automated.
• Monkey Tests : Monkey testing is carried out by a tester assuming that if the monkey uses the application then how random input, values will be entered by the Monkey without any knowledge or understanding of the application. The objective of Monkey Testing is to check if an application or system gets crashed by providing random input values/data. Monkey Testing is performed randomly and no test cases are scripted and it is not necessary to.
  Monkey Testing is performed randomly and no test cases are scripted and it is not necessary to be aware of the full functionality of the system.
• Performance Tests : Performance tests check the behaviours of the system when it is under significant load. These tests are non-functional and can have the various form to understand the reliability, stability, and availability of the platform. For instance, it can be observing response times when executing a high number of requests, or seeing how the system behaves with a significant of data. Performance tests are by their nature quite costly to implement and run, but they can help you understand if new changes are going to degrade your system. I also designate performance tests when talking about load, stress, spike and endurance tests.
• Endurance Test : The system is tested with a load extended over a significant amount of time, to analyse the behaviour of the system under sustained mode. This type of testing is performed at the last stage of performance run cycle. It ensures that the application is capable enough to handle the extended load without any deterioration of response time.
• Load Test : It is a type of non-functional testing and the objective of Load testing is to check how much of load or maximum workload a system can handle without any performance degradation.
  Load testing helps to find the maximum capacity of the system under specific load and any issues that cause the software performance degradation. Load testing is performed using tools like JMeter, LoadRunner, WebLoad, Silk performer etc.
• Stress Test : This testing is done when a system is stressed beyond its specifications in order to check how and when it fails. This is performed under heavy load like putting large number beyond storage capacity, complex database queries, continuous input to the system or database load.
• Spike Test : a sub type of a performance test. Here we test an extreme increase or decrease in load. The goal is to determine the robustness of an application when it receives extreme variations in traffic.
• Penetration Tests : a Security testing aiming to penetrate (attack) into the system using multiple techniques. It’s a meticulous process where a skilled security tester attempts to find weaknesses and security breaches. Is there any security issue ?
• Static Code Analysis : examining the code without executing the program to provides an understanding of the code structure and verify if it adheres to industry standards. This aims to identify whether or not the code complies with the set rules (code coverage, bugs, vulnerabilities, magic numbers…). This would require an automated tool, and is used to scan code typically on the continuous integration pipeline.
• Static Application Security Testing :
• Dynamic Application Security Testing :
• A/B Tests : a way to compare two versions of a single variable, typically testing a subject’s response of variant A against variant B, and determining which of the two variants is more effective.
• Property-Based Testing : It’s a method to explore possible edge cases using a very useful pattern to discover properties of a system under test. We can use parameterized tests to test the functions, with several inputs and asserting their outputs. Finding edge cases is hard work. It would be great to be able to automatically explore and find possible edge cases so we don’t need to rely on QA specialists. In some cases, we can leverage Property-based testing to do that exploration for us.
• Mutation Testing :
• Usability Testing :
• Compatibility Testing :
• UI Testing or Visual Testing :
• Parity Testing : This is similar to A/B Testing. It is used when replacing part or all of an old system, and ensures a new system behaves the same way as an old one. In short, you mirror the traffic to both systems and compare the results, logging any difference. It requires a feature-flagging system to setup multiple processing implementations. Typically, there are four options: NEW, OLD, MIRROR, and PARITY. NEW routes all of the traffic to the new system. OLD sends all of the traffic only to the old system. MIRROR sends the traffic to both old system and new system at the same time. PARITY does the same thing as MIRROR but also sends the response from the old system to compare and log when the responses don’t match.
• Alpha Testing
• Beta Testing

Group of Test Types

Among all the types, we can group them in Testing Approach, Testing Categories, and Testing Method.

• High Order Testing Approach
• High Order Testing Categories
• High Order Functional Testing Methods
• High Order Non-Functional Testing Methods

High Order Testing Approaches

• Black box vs White box Method : Each test type would fall onto either white box or blackbox testing. In a black box testing, the internal structure of the software is hidden from the tester. White box testing is the opposite, the tester does know the internal structure of the software.
• Manual vs Automated : Also each test type must be or may be manual or automated testing or both.

High Order Testing Categories

• Functional testing Method
• Non-functional testing Method

High Order Functional Testing Methods

• Unit testing
• Integration testing
• System testing
• Acceptance testing
• Regression Testing
• Smoke Testing

Non-Functional Testing Methods

• Performance testing
• Security testing
• Usability testing
• Compatibility testing
• Stress testing

Testing Activities

There are several activities that could done in software testing. Here a list :

• Any Testing execution of test type or level : any test execution is also defined as an activity or task. Examples are : unit testing, integration testing, E2E testing, regression testing, white-box testing, performance testing, A/B Testing… This is the most common term I would use in all my articles.
• Test Planning : when starting a project, we need to anticipate things to be covered to ensure that requirements are met before shipping the software. So we need to identify all activities, that has to be done thorough the project phases, even in an agile approach. On top of that, scope, objectives and risks are identified. It’s a matter of defining a plan. That means to plan who, what and where on the timeline (either on particular dates or fit into each iteration). For example, test analysis, design, implementation, execution, evaluation, or any other activities are part of the plan.
• Test Monitoring and Control : reporting and show of progress, analysis of result against expected results, and determine whether the results have met the exit conditions. Also it is important to check if there is any risk and measure the impacts of any gap if the objectives can not be reached.
• Test Analysis : here it is about “what to test”. We need to determine and prioritise any test that is appropriate to the test levels, according to the strategy (functional, non-functional, and structural characteristics, business and technical factors and levels of risks). For example, user stories, features, sets of features are to be analysed.
• Test Design : here it is about “how to test”. We design test cases and and define the data, testing environmental infrastructure and tools to support test conditions and test cases. We also define how to capture metrics and measure the progress, as we should able to generate reports once a testing activity is completed.
• Test Implementation : here it is about “what is the sequence to run the tests?”. So we would implement new tests and create test suites accordingly and update or create scripts to trigger tests execution either manually or automatically, either used on the CI or further environments. This should also cover the environment on which the tests are ran, such as now platform or virtual machine or container. Preparing test data and ensuring it is properly loaded in the test environment so the context is set for tests to be ran.
• Test Execution : execute tests either manually or by using any execution tools. Nowadays more and more tests are automated at unit level and running them is also a test execution. When execution tests, we would compare actual results with expected one, analyse anomalies to find their root causes. Some failures may occur due to defects in the code, but could be false positive. Also we would report defects and log the outcome.
• Test Completion : create a test summary report, including feedbacks for improvement, to be communicated and keep trace. Update and archive the test data environment infrastructure configuration for reuse. May include checking whether defect are closed or creating a new one.

Testing Strategy

A testing strategy usually has a mission statement which could be focused on the software being built or may be related to the wider business goals and objectives.

Manual Testing Strategy

Testing Automation Strategy

We would refer to the diamond or pyramid of test.

Test Doubles

To help in understanding what are the differences between the test doubles are, we need to define what a system under test is.

A System under Test (SUT) is a subject which we want to test against, verifying its behaviour and state so we can confirm the subject is conformed to what we expected. But most of the time, it requires to interact with objects and its dependencies. We thus need test doubles to replace them.

There five type of test doubles :

• Fake Object aims to replace something real with something else. That fake object would have the same attributes or less but should serve the purpose to substitute the real one so that we can run a test against the subject. It’s a shortcut version of the real one.
• Dummy Object is similar to a fake object, but has much less attributes with absence (or near) of behaviour. It’s a very simple and lightweight version of the real one.
• Mock Object is the must commonly known, and is used to simulate behaviours of an object which interacts with the subject. It’s pre-programming to have specific pre-determined behaviour and state, and most of the time re-pre-programming within the same test.
• Stub is a bit similar to the above one in the sense that it has pre-determined with a state and behaviour, but the difference lies on the fact that it is not locked to any specific test. It may respond like a real one (object or function) and it is a shortcut version of the real one. Stubs are likely to be wrapped and deployed as a dedicated running service. What really matter is the input and output.
• Spy is a mock or a stub, and on top of it records information during the interaction with the subject so that we can verify how the double has interacted with the subject. Typical information are arguments of a function call and the number of times a function has been called.

In some case, people refer “stubs” as a way to verify the state of the object under test, and while “mocks” as a way to verify its behaviour. In fact, a “stub” is a test double to replace real object by providing “indirect inputs” to the subject under test, while a “mock” is a test double to replace a real object to verify “indirect output” of the subject under test. A “spy” on the other hand is a test double to replace the real one by recording “indirect output” of the object under test, so that we can do ulterior verification.

Test Environment

At last, we need to define the environment in which tests will be executed. There are typically commonly used environments :

• Local : local testing (on developers’s workstation). Any automated test that can run on local should be runnable to the CI as well.
• CI: Continuous Integration Environment (each component of the software being built is containerised — the component is integrated inside a box, aiming to test only the component in isolation, but does not yet interact with external APIs of other component outside of the scope). Whatever the test is run on CI should be runnable on local as well.
• SIT : System Integration Testing Env (fully integration, used by the dev team including QA)
• QA : Quality Assurance Testing Env (fully integrated, used only by QA)
• UAT : User Acceptance Testing Env (fully integrated, used by QA, alpha, beta users)
• Perf : Performance Testing Env (fully integrated, used by QA and architects or teams, normally with a short timeframe)
• Staging : Iso Production Testing Env (fully integrated, used by QA, alpha, beta users)
• Prod : this environment aims to serve end-users (fully maintained by ops team, used by QA and product teams, used by end users), but some testing activities may be performed here, such as A/B Testing, performance testing, sanity check or smoke testing.

Acronyms

• API : Application Programming Interface
• BDD : Behaviour Driven Development
• CI & CD : Continuous Integration & Continuous Delivery
• CIT : Components Integration Testing
• CT : Component Testing
• DDD : Domain-Driven-Design
• E2E : End-to-end
• INT : Integration
• KPI : Key Performance Indicator
• OKR : Objective Key Results
• OST : Opportunity Solution Tree
• PR : Pull Request
• QA: Quality Assurance
• SIT : System Integration Testing
• SOA : Service Oriented Architecture
• TDD : Test Driven Development
• TLDR : Too long drop it
• UAT : User Acceptance Testing
• UFT : Unit Function Testing
• SAST : Static Application Security Testing
• SDLC : Software Development Life Cycle
• SQLC : Software Quality Life Cycle
• DAST : Dynamic Application Security Testing