Customized Software Testing Framework for Web Applications

elis.pelivani@hotmail.com Abstract— Software testing is gaining more attention and investments in the IT companies. We have both manual and automated approaches, considering the dynamic of the project the appropriate one is chosen. There have been presented and used several frameworks and tools for this purpose. However, the outcome is that a single tool does not cover all the testing needs of IT systems. In this paper, we will present a customized framework based in Selenium that tackle this gap within the IT systems where several modules with many cross function are involved. We will present the architecture overview and a case


Applying the styles to an existing paper
Software testing is gaining more importance nowadays and the most of the IT companies are investing a lot of time, effort and resources in this field. Moreover, the testing reporting and percentage of success is mandatory in the final steps of many applications acceptance. Especially for some industries where the accepted margin of the failure is close to zero, there is the need to use a set of software testing tools and frameworks for different modules or applications. As we have shown in our previous researches, none of the existing software tools and frameworks offer all the features and testing needs for all the companies. For this reason, most of them have a mixed of testing, manual and automated [1,2].
The goal is to have as much as it is possible the automation of the software testing. Consequently, various testing tools and frameworks are reviewed to have the best choice [3]. For many of them, the best solution would be to have a customized framework that gets the most needed features from several frameworks and optimizes the outcome of the testing. This comes with high costs of investment in time, money and resources.
In this research, we will try to introduce a new customized framework that is based on the most powerful and used software-testing framework, Selenium. We will show in detail the architecture and the information flow of this customization [4]. At the end of it, we will also show a real case study with all the results of its implementation. This will show how this customized framework improves our testing capabilities for this kind of applications that are composed by many modules. Writing a new document with this template.

Customized framework
The aim of our customized automation framework is: • creating and running tests and test suites with full control over test concurrency and producing extremely thorough yet understandable test reports and • providing technological extensions for testing systems of any type.

Main architecture components
In the below figure it is shown the architecture overview of the framework, Figure 1. The following are the primary component types found in the framework: A platform core that includes everything needed for creating, running and reporting tests. [5] One of the core classes of the framework contains main methods that are needed to perform user actions on an WebElement(DOM element) such as click(), get-Text(), selectCheckbox(), deselectCheckbox(), moveMouseOverElement(), rightClick-OnElement(). Every general driver method is located in this class.
The Core components of the architecture of the customised framework are: • config: Parses the configuration files and provides the framework with their parameters.
• Scheduler: Parses test suites and concurrency data and organizes it into a tree structure. Executes the tests in the tree while adhering to the specified concurrency requirements. • Content: Provides the test classes with broad data/document-based validation functionality. • Service: Manages the life cycle of services and declares service provider interfaces for platform extensions. • logfortesting: Keeps track of test execution and logs it.
• Invoker: Provides an implementation for default test class mechanism and defines a service provider interface for executing tests. Execution extensions that allow running tests from the command line, as well as the JUnit unit testing framework.
Services which contain service provider interfaces for interacting with a system under test and give mechanisms to execute tests from the command line and the JUnit unit testing framework [6] (e.g. a web GUI, database, JMS).

The schedules class
The basic entry point for parsing a given Suite or Test class and constructing the described tree structure is SuiteParser. The class org.test.scheduler.RunnerTree is used to group all pending tests into a tree structure. The tree is made up of multiple types of nodes (represented by child classes of the org.test.scheduler.node.RunnerNode parent class): • Parent nodes: Child nodes are tree nodes that combine other nodes. A test suite is represented by a parent node, which combines test cases and/or test suites. The class org.test.scheduler.node.RunnerGroup represents them. • Leaf nodes are tree nodes that do not have any children. Each leaf node represents an atomic test case. The class org.test.scheduler.node.RunnerLeaf represents them.
The root node of the tree is mapped to the base test suite class. With RunnerNode as the component, RunnerNode as the leaf, and RunnerGroup as the composite, the tree node classes form a Composite Pattern. Because a test class method might be called multiple times, Java class elements are assigned to nodes in the following way: Child node test class -> child node test method -> child node test invocation (or parameter set) -> leaf node.

Other settings
Java's Executor mechanism: RunnerTree schedules and executes Runnables that invoke the test cases using Java's ExecutorService mechanism. The essential concept is that an abstract service definition (ExecutorService) is used to asynchronously execute tasks (Runnable or Callable) and that an object handle (Future) is used to query execution status, obtain execution results, wait for termination, or cancel execution.
Some of the main plugins used from this customised framework are: Test rail Connector: Allows framework test to send testrail results via the testrail connection module (using Rest API), Figure 2. The utility class that creates the TestPlanEntry is called first, followed by the Listener that updates the results case by case, and finally a utility class that cleans up the un-tested cases from the testPlanEntry [7].

Fig. 2. Jenkins results displayed in test rail
Test Cloud Manager: The Cloud Manager acts as a "Load Balancer" for the Selenium hosts that are needed to run all the test cases. It is a comprehensive application made up of various parts (and supports extensions).
DatabaseTest, Database service is one of the most significant topics in database testing. This Database Service allows creating, updating, and querying databases.
Jenkins-plugin: Jenkins Plugin provides useful data for tasks that run framework-based tests. Each task must perform a unique Post-Build phase provided by this Plugin to obtain the framework data from the executed run in order to use the charts. Following that, both the run and the job have an additional button on the left sidebar. The statistics for the run concentrate on the current and maybe past runs, while the statistics for the job concentrate on the change in results over time. Jenkins Plugin offers sophisticated statistics and graphics for test executions [8]. This plugin helps analysing how test cases perform over multiple releases, allowing to rapidly spot potentially unstable test cases (or instabilities in Application under test). It also helps identifying trends by comparing results with previous builds, Figure 3. Visual Data Editor (VDE): For test data, framework presently supports two formats: Excel files in a customized format and XML files in framework's own testdata xml format. To edit the latter, the framework ecosystem provides the Visual Data Editor as an Eclipse plugin. Because XML files are difficult to edit by hand, this plugin provides an editor for.testdata.xml files in Eclipse, Figure 4.

Fig. 4. The screen of the VDE in eclipse
Logfortesting is a component that offers the technical basis for test execution functional logging. It is used to create HTML or XML reports that are well structured and easy to read [9, 10], Figure 5. There are several statuses for the executed test cases. Full list is below [11,12], Figure 6: • Pending, a test is scheduled to run, but it has not yet begun.
• Running, a test is presently in progress.
• Passed, a test was run, and no issues were found.
• Ignored, a test was disregarded. This can be due to a variety of factors for different sorts of test objects: This indicates to TestSuites and TestCases that a test data set has been designated to be ignored by the user, that the test has been run, but that the result code has not been propagated to the suite. It indicates that both a prior step was incorrect, and hence this TestStep was skipped. If a test step was incorrect but was set up to be retried despite an initial failure. • Failed, a test failed due to a functional error e.g. incorrect value.
• Failed Performance, this implies that the SUT is stuck or takes an inordinate amount of time to reply to a user action. • Failed Access, A test failed owing to a failed system access. The system is most likely unavailable, or the access setting is incorrect. • Failed Automation, A test failed owing to an error that a Test Developer might resolve.
Technical errors, such as a changing technical identifier, are the most common. • Inconclusive, a test failed due to an error that has to be examined by a Test Automation Framework Developer.

Our experiments
A test case is used to get a better understanding of how the framework operates. The test case consists of eleven different configurations translated in eleven different test data to go through test steps. Aim of this scenario is to assure that the user can book a seat on a certain date and check that the booking is successfully saved and displayed in My Bookings screen. Scenario steps are displayed in the following print screen (part of the generated report after test case finishes running).
At first, starting and finishing time of test case execution is displayed. Then, steps in test case implementation are organized to form testGroups. Each testGroup gives complete information of the TAS and SUT at that step, Figure 7. Often, certain functionality must be invoked before (set-up) or after (teardown) test execution. This can be achieved by putting relevant code into methods. For this reason, each test case class that is added in the framework has to extend a base class: BaseTestCaseClass.
This class contains all the methods that are needed to be executed before a test starts running and after it finishes execution (e.g., installation and configuration issues with test environment: database setup and initial load, services start/stop). Beside these methods, other Jenkins jobs are configured to start automatically before test Suite begins running [13] (e.g., database clear up).
Login method is same for all test cases, Figure 8. It can take different test data (username and password value) based on the functionality it will test. Following print screen is part of the generated report, login method is displayed with full information how it operates. Firstly, it goes to the provided URL (application domain), then maximizes the chrome driver window. Afterwards, it interacts with the SUT (System under Test), inserts values in username and password input fields. Technical Locator column indicates how xpath, classname or CSS locator locates the WebElements. Assertions are made for each operation conducted on the application to ensure that it performs properly. The test logs give detailed information about the execution steps, actions and responses of a test case and/or test suite. However, the logs alone cannot provide a good overview of the overall execution result. For this, it is necessary to have in place reporting functionality. After each execution of the test suite or test case, a concise report must be created and published [14].
Whenever a test case encounters a failure, the framework makes sure that all information needed to analyse the problem is available/stored, as well as any information regarding the continuation of testing, if applicable. Screenshots and other visual captures are saved during test execution for further use during failure analysis.
Below it is a part of the report that is generated automatically when a test case finishes execution. This report is customized for the framework because Selenium itself does not have reporting tools, extra configuration is needed to be done to produce such detailed reports and log files. For each step, there is full information regarding WebElements, user actions, test data, assertion of that step.
In case of second run, test case has failed, and its execution time is significantly higher than the other runs. A maximum execution time has been specified in the framework core, a corresponding timer is started that will terminate (and fail) the test case if it has not finished in the configured time. Besides that, for each action performed on the SUT there is a taskCompletionTimeout which checks whether an action in performed which a predefined time in milliseconds, Figure 9.

Conclusion
There are many different types of automated testing frameworks, so choosing the proper one for your needs is important. Using one that is well structured, can improve the team's productivity by boosting test correctness, maximizing test coverage, and minimizing costs and maintenance-eventually providing a higher return on investment. We believe that our customized framework complies with the key concepts that support easy development, evolution, and maintenance of a Test Automation Solution.
Each component of the framework has a single responsibility [15], it is in charge of a one task, e.g.: setting up the driver, logging results, generating detailed execution reports, creating and reading data files, executing teardown methods. All these functionalities are secluded in platform core, which leads to another good principle being applied: tests (scenarios) are separated from automation framework (as already mentioned in case study, test data and test cases are in different packages from framework core). [16] In conclusion, although some of the principles must be followed, it is important that the automation framework serves the business needs and is customized according to the application. That is why this kind of customized framework is wanted; otherwise, one test automation solution would work for all kinds of applications. Further analysis and discussion will be done for the presented customized framework with more real case studies.