Consider a test assignment where hundreds of IP cameras are required to test a video surveillance solution; or, where we require multiple routers to test the functionality of a protocol implementation; or, even a scenario where we have to test an application on multiple operating systems over a WAN connection.
Often, testing such types of complex deployment scenarios are compromised due to infrastructure and logistics costs. This article gives an insight on how we can leverage virtualisation and simulation techniques to set up test environments that can reduce operational costs.
You will also get an overview of how to come up with test bed setup solutions for testing applications across areas such as communications, video conferencing and surveillance and element management systems.
Test environments
When we talk of test environment, usually the following comes to our mind:
1. Physical infrastructure - servers, network equipment, storage
2. Network and storage configurations - IPv4/v6, VPN, RAID, iSCSI
3. Applications and services - domain related, network related, access control and file system services
4. Test repository - OS and application builds, tools, test cases, test data
Typically, significant effort goes into the following associated activities:
1. Administration - user management and access control to test beds
2. Test bed management
a. Test bed topology definition
b. Setup, configuration, and teardown
c. Backup and recovery
d. Health monitoring
3. Automation
a. Programmatic control of test equipment (startup/shutdown, reboot etc.)
b. Integration with test automation framework and defect management systems
Thus, developing a base framework that reduces the cost of ownership for these tasks can result in a good return on investment. This framework can be extended and customised for testing applications in specific domains.
Building blocks
There are several open source and freeware utilities and indeed commercial tools that we can explore and integrate to build the framework. However, wherever required we have to develop our own custom utilities to address any specific testing needs.
The following are some areas where we need to identify solutions for integration:
1. Virtual Machine Monitor (VMM) - A virtual machine monitor is a programme which allows a single computer to support multiple isolated execution environments for running applications. There are several open source and freeware virtual machine monitors that can be used to run multiple operating systems on a single physical machine. Examples include VMware server, Virtual Box, KVM, and XEN.
2. Virtual Infrastructure Managers (VIMs) - These are applications which provide an abstraction layer for the underlying physical machines which actually host the virtual machines (VMs). They take care of deployment of VMs across the physical machine in the cluster, providing features like VM life cycle management, virtual network management, user management and fault tolerance. Some examples are open Nebula, ConVirture and Enomaly.
3. Network and storage configurations - There are utilities for setting up and configuring IPv4/IPv6, firewall, DHCP, routing protocols, and network services. There are also applications for configuring storage services like RAID, file systems, LUNs etc. Vyatta, NetSNMP and FreeNAS are some examples.
4. Automation - Test automation frameworks and re-usable libraries are available to manage test equipment and run automated test cases. IBM STAF, robotframework, CPAN and Libvirt APIs are good examples.
Features
The identified applications and utilities are customised and integrated to come up with a set of features to ease administration, test bed management and automation.
Some examples are:
• Physical cluster management - ease of addition or removal of physical machines in the cluster for scalability
• Test bed topology definition - usage of a drawing tool to define the test bed topology, specify server and OS details, network and storage configurations, service and application requirements; and export to different topology files
• Test bed provisioning and management - automatically deploy and tear down test beds, configure IPv4/IPv6 networks, and configure network and storage services from the stored topology files
• Test automation - integration with test automation frameworks and other domain specific utilities (e.g. protocol packet generators). Integration with defect management tools like Bugzilla for automatic logging of defects
• Backup and recovery - backup and recover entire set of test beds for troubleshooting at a later point in time
• Monitoring and access- view real time status of test equipments and access equipment consoles from a single location
• Central image repository - upload or download the VM images, application images, tools, etc. from central repository
Approach
The virtualisation and simulation concepts or the base framework can be customised or extended for specific testing requirements. Let us see some example approaches:
- Testing implementation of routing protocols
- Testing of deployment scenarios usually involve costly test infrastructure like routers, switches and traffic generators and considerable provisioning effort.
Consider a test scenario involving a routing protocol like OSPF (open shortest path first) where the objective is to verify the convergence in case of node failure. This would typically involve multiple routers and couple of host machines apart from the device under test.
As part of the test execution steps, we have to bring down a router after connectivity between both the test hosts is established. Thereafter, we have to verify that it is able to recalculate and take the next shortest route to the second test host.
We can see that apart from the cost involved, there is lot of effort in provisioning and maintenance of the test bed. It is a significant requirement to tear down and re-deploy a test bed corresponding to a different topology to be used by another test case. Also, in case of any failure, troubleshooting can become challenging because of the extensive cabling and the need to access multiple equipment consoles.
The entire test bed can be deployed using two physical machines (maybe one - depends on CPU and RAM capacity) and a switch to connect the device under test, as depicted in the following diagram.
The test hosts can be deployed with the help of virtual machines using any VMM of our choice. The network and protocol configurations can be done using virtual routers like ZebraOS, Vyatta etc.
These virtual routers behave like physical routers and traffic will be exchanged between the device under test and virtual routers. Using VM management APIs and PERL modules, we can automate the entire test bed deployment and teardown, network and protocol configurations and outage simulation.
This approach provides the following benefits, apart from the obvious cost advantages:
• Scalability - ease of addition and removal of physical machines depending on the requirement
• Automated test bed deployment and teardown - execution of test cases having diverse topology requirements one after another without manual intervention
• Ease of troubleshooting failure scenarios - we can take a backup of the test bed on discovery of defects and continue with other test cases. At an appropriate time, we can recover the test bed to the failure state for debugging, thus reducing the defect resolution time.
• Automation - we can develop re-usable device independent APIs for network and protocol configuration, outage simulation and integrate with the test automation framework
Let us take another scenario.
Testing of a Video Surveillance Application
A video surveillance application configures, manages, and controls a number of IP cameras located at different locations and displays multiple video streams at the same time.
In order to test such applications, we have to simulate
1. Video streaming - over protocols like RTSP
2. Control information- brightness and contrast control, panning, and zoom, etc. over protocols like SNMP
The video streaming can be simulated using open source and freeware solutions like VLC player, Darwin and live555 streaming servers. We have to create a number of virtual or logical network interfaces on the physical network interface in the test host. Once we assign IP addresses to each of the virtual interfaces, we can stream video files on these IP addresses and also run SNMP agents on each of the interfaces. Thus, for the video surveillance application, it would seem that the video streams are originating from live IP cameras to which it can send control messages.
Camera control simulation can be done by modifying values in the specific MIBs of the IP camera for a particular type of control. The corresponding SNMP agents would be contacted for this purpose by the SNMP manager in the video surveillance application. Apart from SNMP get and set operations, we can also use the TRAP utility for simulating faults. When we receive a control message for increasing the brightness of a particular camera, we can start streaming a video file to the corresponding IP address which is similar to the one which is currently being streamed, except that its brightness settings are modified.
Thus we see that for this requirement, we have to integrate video streaming solutions, SNMP application packages, MIB databases, and other custom utilities. Part of the solution involving the SNMP simulation can be extended to test EMS/NMS applications as well.
Addressing challenges
In this article, we have seen:
• Ideas for integrating and packaging open source and freeware solutions and developing your own utilities for setting up test environments at reduced cost and effort
• Suggestions for reducing defect resolution time
• Overview of end-to-end automation approaches including test bed deployment and teardown
However, there are some limitations in terms of
• Availability of emulators to address domain specific requirements - custom development often requires significant effort and expertise
• Performance testing - requires actual hardware for benchmarking
• Customisation effort - sometimes a lot of effort is required for framework or solution customisation which can impact re-usability
Nevertheless, these solutions help us in addressing challenges in areas of scalability testing and in many cases functionality testing as well. They also increase the extent of automation. However, planning and exploration have to be done carefully to avoid technical and reliability issues at a later point of time.
Harish Nagaraj, Test Manager, MindTree Test Labs, has over 10 years of experience in IT, mostly in development, product testing and project life cycle management. He is currently driving a project on virtualisation for the Testing BU division of MindTree. His responsibilities include providing thought leadership and team competency development. He has worked in various areas like implementation of test automation frameworks, virtual machine monitors, virtual infrastructure managers and test life cycle management.
Sudip Naha, Co-Head, MindTree Test Labs, Sudip Naha has over 10 years of experience in product testing and project life cycle management. He holds a Bachelor of Engineering degree in Computer Science. He is currently co-heading the Test Labs in MindTree. In this role, his responsibilities include ownership of testing frameworks evangelism and development catering to specific industry segments.