How to find the problems, bugs and leaks before the clients, users or hackers do
With the growing technology, there has been a massive change in the software testing industry with the help of newly upgraded tools and trends. These changes are aiming for shorter cycle times, a better product quality providing products to the market, and reduction of costs in development and maintenance. Looking at the evolution in the field of test technology and testing processes a lot of skills and expertise is required from the software testers as well in adapting due to the changes and challenges every day.
There are a lot of different types and approaches of testing the different systems to be released, considering:
- Dependencies on the used development process and development phases of the product
- Different conditions and states of the product in these phases
- Interactions with surrounding systems and interfaces
- Conditions of the environment in which the product will operate
- Criteria of related standards and regulations
- Usability and user experience
- Safety and security aspects
- Compatibility with previous versions of the product
- Proof of delivery, meeting requirements and quality assurance
Main steps of defining and following the test process
- Set the test requirements according to the contractual requirements (hardest requirements first to be considered)
- Break it down and consider to create a special test framework
- Test Strategy & project governance definition
- Define special tests and as well partners for the execution (e.g. from Client, special skills, trial & friendly users, regulatory support, …)
- Team setup
- Reporting tools setup
- Documentation setup
- Test Environment setup
- Test Case Generation
- Test Case Execution
- Result and Analysis
- Bug fixing
- Re-test
- Final documentation
BASE TEST TYPES
4 main types of testing approaches
The decisions of selecting the several types of how to test a given HW and/or SW product must be made during the general test planning as they are important related to requirements, effectiveness needed and available test environment, budgets, test time and available tools.
Manual testing
Manual testing is the process of manually testing the software for defects. It requires a tester to play the role of an end user whereby they use most of the application’s features to ensure correct behaviour. It refers to a test process in which a QA manually tests the software application in order to identify bugs. To do so, QAs follow a written test plan that describes a set of unique test scenarios.
Automated testing
Automated tests provide much faster feedback when things go wrong. Faster feedback from automated tests (whether run locally or on a build server) makes it easier for developers to ensure that their changes don’t break existing work, and reduces the time wasted during integration.
Black box testing
Black box testing involves testing a system with no prior knowledge of its internal workings. A tester provides an input, and observes the output generated by the system under test.
White box testing
White box testing is a software evaluating method used to examine the internal structure, design, coding and inner-working of software. Developers use this testing method to verify the flow of inputs and outputs through the application, improving usability and design and strengthening security.
PROGRESS AND STAGE RELATED TYPES
during product development test phases.
PoC Proof of Concept test
Is proofing and justifying the ideas and design approaches for the new product. These tests are executed on a (simple) pre-product version which is tested to check if the assumptions are fulfilled and the main base functions of the planned solution are properly working and harmonised between each other.
See as well out article: IOT Connected Prototypes | Overview and Experience
SW Module test
Module testing is a process where you need to test each unit of these modules to ensure they adhered to the best coding standards. Unless a module passes the testing phase, it cannot go for the application testing process. Module testing, component testing, helps to early detection of errors in application testing.
Bring up test
Board bring-up is a phased process whereby an electronics system, inclusive of assembly, hardware, firmware, and software elements, is successively tested, validated and debugged, iteratively, in order to achieve readiness for manufacture.
End to End testing (E2E)
End-to-end testing is a methodology used in the software development lifecycle (SDLC) to test the functionality and performance of an application under product-like circumstances and data to replicate live settings. The goal is to simulate what a real user scenario looks like from start to finish.
Field test
Field Validation Table (FVT) is a test design technique, which mainly helps for validating fields present in the application. This technique adds value to an application or project and gives very good test coverage for field validation. And this technique easily helps to find defects lying in the system or application.
Field trials
Field trials are real-life experiments which test directly whether proposed interventions actually work. This makes them powerful tools for gathering evidence for making policy. But, as with all research methods, they come with costs, such as time and resources.
Clinical trials
Clinical trials are research studies performed in people that are aimed at evaluating a medical, surgical, or behavioural intervention. They are the primary way that researchers find out if a new treatment, like a new drug or diet or medical device (for example, a pacemaker) is safe and effective in people.
Usability tests
Usability testing refers to evaluating a product or service by testing it with representative users. Typically, during a test, participants will try to complete typical tasks while observers watch, listen and take notes.
Acceptance test
This is a type of testing done by users, customers, or other authorised entities to determine application/software needs and business processes.
Acceptance testing is the most important phase of testing as this decides whether the client approves the application/software or not which has mostly direct impact to payments, reputation and further engagement.
Test plan example | overview
If a new product is to be developed which consists of HW (e.g. one or more devices) and one or more SW parts, this HW is needed in early stages of the product and of course its development cycle has to start earlier as well to get the first samples for the protype. The SW devliveries have to be scheduled in several phases which make the code available in-sync with the HW stages of PoCs, first samples and prototype phase, and HW ready for production. The Integration and first testing scenarios have to focus on these areas which prevent expensive HW changes in the later stages. The special test cases to ensure use-ability, performance, KPIs and reliability could be in parallel with or part of the E2E real system configuration tests. The acceptance tests to proof the requirements have to be aligned with the customer before hand and are starting at Ready for Acceptance and ending with the acceptance approval which is as well allowing payments according the terms.
STRATEGY & INFRASTRUCTURE RELATED TYPES
Different scenarios on test infrastructure and supporting nodes at the different development stages and strategies
SW offline test / SW test with interface simulations
An offline exam means an exam that can either be administered paper-based or offline using examination software systems, like Qorrect, in which the test is run within an examination facility with computers only locally connected, for example.
Program branch coverage tests
Branch coverage is a metric that indicates whether all branches in a codebase are exercised by tests. A “branch” is one of the possible execution paths the code can take after a decision statement (e.g., an if statement) gets evaluated. Special debugging tools or markers in the code can determine whether the branch was visited or not.
To calculate Branch Coverage, one has to find out the minimum number of paths which will ensure that all the edges are covered. In this case there is no single path which will ensure coverage of all the edges at once. The aim is to cover all possible true/false decisions.
Environment or testbed
A testing environment is a setup of software and hardware for the testing teams to execute test cases. In other words, it supports test execution with hardware, software and network configured. Test bed or test environment is configured as per the need of the Application Under Test.
SW in the Loop test SIL
Software-in-the-Loop testing, also called SiL testing, means testing embedded software, algorithms or entire control loops with or without an environment model on a PC, thus without ECU hardware. In fact, SiL Testing is an integral part of automotive software testing.
HW in the Loop test HIL
Hardware-in-the-loop testing provides a way of simulating sensors, actuators and mechanical components in a way that connects all the I/O of the ECU being tested, long before the final system is integrated. It does this by using representative real-time responses, electrical stimuli and functional use cases.
Product test with Simulators
A simulator creates an environment that simulates interfaces, content and protocols of a real device.
It is a software or Software/Hardware that helps to test your product to test the interfaces, functions, and features not be connected to the real nodes and interfaces during operation.
The system reactions are pre-programmed per test case and need the full call flow scenario- and data model know-how from the test designers and programmers. Protocol test equipment is used to monitor and store the test results in compliance with interface standards and parameters.
Product test with Emulators
An emulator creates (emulates) an environment that mimics the behaviour and configurations of a real device (it acts like a real connected device, but sometimes in a more simple way).
It is software and Hardware that helps to test your product via the interfaces, functions, and features not connected to the real nodes and interfaces during operation. Examples are test equipment from Anritsu or Tektronix.
The emulator is e.g. supporting the interface standards and logic flow reactions during an early stage to help product vendors to test their products as long as no partner and corresponding nodes are still on the market or company internally available.
Real equipment E2E test
End-to-end testing is a methodology used in the software development lifecycle (SDLC) to test the functionality and performance of an application under product-like circumstances and data to replicate live settings. The goal is to simulate what a real user scenario looks like from start to finish.
TARGET & COVERAGE RELATED TYPES
Important are the definitions which targets are needed to achieve for the best product quality. I must be aligned with the whole QA system and as well be focused on the key areas of clients requirements and expectations.
Basic operation tests
Operational testing is a type of non-functional acceptance testing that confirms that a product, service, process or system meets operational requirements.
Examples are Load & Performance Test Operation, Security Testing, Backup and Restore Testing, and Failover Testing.
Functional tests
Functional testing is a type of software testing that validates the software system against the functional requirements/specifications. The purpose of Functional tests is to test each function of the software application, by providing appropriate input, verifying the output against the Functional requirements.
Non functional tests
Non-functional testing is the testing of a software application or system for its non-functional requirements which means the way a system operates, rather than specific behaviours of that system.
Functional requirements explain how the system must work, while non functional requirements explain how the system should perform.
Error / Scenarios tests
Scenario testing is a software testing activity that uses scenarios:
- hypothetical stories to help the tester work through
- a complex problem or test system
- The ideal scenario test is a credible, complex, compelling or motivating story.
- The outcome of which is easy to evaluate.
- Error scenarios like power outage, alarms, data corruptions or wrong interface approaches can be be created by using simulators
Regression tests
Regression testing is testing existing software applications to make sure that a change or addition hasn’t broken any existing functionality.
Load testing
Some basic examples of load testing are: Testing a printer by transferring a large number of documents for printing. Testing a mail server with thousands of concurrent users. Testing a word processor by making a change in the large volume of data.
Stress tests
Stress testing (sometimes called torture testing) is a form of deliberately intense or thorough testing used to determine the stability of a given system, critical infrastructure or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results.
Performance tests
Performance testing is the practice of evaluating how a system performs in terms of responsiveness and stability under a particular workload. Performance tests are typically executed to examine speed, robustness, reliability, and application size.
Interaction tests
Interaction-based testing is a design and testing technique that emerged in the Extreme Programming (XP) community in the early 2000’s.
Focusing on the behaviour of objects rather than their state, it explores how the object(s) under specification interact, by way of method calls, with their collaborators.
Interface tests
Interface Testing is defined as a software testing type which verifies whether the communication between two different software systems is done correctly. A connection that integrates two components is called interface. This interface in a computer world could be anything like API’s, web services, etc.
In the telecommunication are often interoperability tests between the different telecom vendors executed which are proof that the signalling standards are filufilled.
Certification tests
Justifies the usage of a product in a specified environment and under defined conditions. The process for certification of a product is generally summed up in the steps:
- Application (including testing of the product)
- Evaluation (does the test data indicate that the product meets qualification criteria)
- Decision (does a second review of the product application concur with the Evaluation)
This is a very important step and test type for Medical Devices (according MDR) and Common criteria for High Security requirements.
HARDENING TYPES
Hardening tests – security
In computer security, hardening is usually the process of securing a system by reducing its surface of vulnerability, which is larger when a system performs more functions; in principle a single-function system is more secure than a multipurpose one.
Examples: Side channel attacks and channel attacks; Security measures: Secure SW writing which allows to mask and hide exchange of security keys and fault injection systems which simulate a breach.
Hardening tests – environmental
Hardening of an electronic product concerns the resistance of this product against surrounding environments and effects like earthquake danger, theft danger, radiation impacts, mechanical load, extreme temperatures, humidity, usage in water and vacuum.
To test the stable performance of such products, the operation in a climatic chamber, systems to check robustness and simulating earthquakes, vacuum, water basins and generators for radiation and electromagnetic pulses.
Radiation-hardened electronics, also called rad-hard electronics, are electronic components (circuits, transistors, resistors, diodes, capacitors, etc.), single-board computer CPUs, and sensors that are designed and produced to be less susceptible to damage from exposure to radiation and extreme temperatures (-55°C to 125°C).
