Comparison of Effectiveness of Different Learning Technologies

E-learning has become one of the powerful supporting tools that expand traditional teaching in higher education. Designers of learning objects (LOs) for blended learning higher education face number of challenges; one of them is choosing the right technology to develop learning objects. This study adopts the Bloom-Redeker-Guerra (B-R-G) mapping model which guides designers to transform the contents and objectives of a traditional course into a number of suggested LOs for a blended course. The study attempts to empirically validate the first dimension of its evaluation scale which measures the effectiveness of learning objects that targets achieving lower order thinking skills (i.e. Knowledge and Comprehension) according to Bloom's Taxonomy. This paper presents the results of the empirical study that validates the students' learning achievement and students' perceived satisfaction differ for receptive learning objects that have been developed with different learning technologies. The empirical study has been implemented using pretest-posttest experiments, in addition to a questionnaire that measures students' satisfaction. Participants were about 100 Information Technology (IT) students enrolled in different courses. Results show that students' learning achievement and students' perceived satisfaction improve with learning objects designed with advanced learning technologies (according to Guerra scale), hence better achieve the targeted learning objectives.


INTRODUCTION
The development of Information and Communication Technology (ICT) has affected the education process, which is known as e-learning. According to the European Commission, e-learning is defined as "the use of new multimedia technologies and the Internet to improve the quality of learning by facilitating access to resources and services, as well as remote exchange and collaboration" [8]. Therefore, research and development of e-learning materials focus now on the inclusion of new technical features. However, less effort is going into assessing the achievement of the learning objectives in this new educational material.
The thinking process and learning objectives are classified into six thinking skills levels according to Bloom's Taxonomy (BT), which is widely adopted nowadays [4]. The levels of thinking from simple to complex are: Knowledge, Comprehension, Application, Analysis, Synthesis and Evaluation. According to Bloom, Knowledge is the least complex level of the taxonomy that emphasizes remembering. Meanwhile, Comprehension emphasizes basic understanding of a communication. Thus, in this research, we consider: Knowledge and Comprehension as the lower order thinking skills, whereas Application, Analysis, Synthesis and Evaluation as the higher order thinking skills. Although, in higher education the final goal is usually for students to reach higher order thinking skills, Bloom indicates that each subsequent level is dependent upon the learner's ability to perform at the level preceding it. Hence, higher education students will always need to master knowledge and comprehension skills in any course before achieving higher goals.
The learning object (LO) is defined as self-contained instructional units that include heterogeneous learning sources (text, presentation, audio, or video) or a combination of any of these media [3]. Redeker's taxonomy classifies LOs into three types: (1) Receptive: where the learner is consuming information. (2) Internally interactive: where the learner interacts with the LO. (3) Cooperative: where the learner is required to perform communicative activities among other learners [17].
The list of available technologies that can be utilized in developing LOs is rapidly expanding. Furthermore, the features provided by any of these technologies are changing; making it difficult to classify a particular system or tool into a specific category. This research utilizes Guerra Scale [12], which is developed by the American Society for Training and Development. The scale outlines the range of online content on a scale from one to ten based on several factors such as increased user interactivity, complexity of development and functionality. This study limits the investigation of learning technologies to the lower end of this scale in particular GS1, GS2, GS4 and GS5. Figure 1 shows the 10 levels of the Guerra Scale.
In [7], instructional design (ID) learning theories and taxonomies have been utilized to represent the components of the technology-based learning design. A Bloom-Redekr-Guerra (B-R-G) mapping model has been devised to assist developers to transform the contents and objectives of a traditional course into a number of suggested LOs for a blended course. This approach takes into consideration the learning objectives of the course while integrating technology with course content; thus assisting in improving the learning process. In addition, the study suggests weight scales that are used during an evaluation process to pre-assess the effectiveness of the blended course while developing it [7]. This paper validates the design and evaluation model proposed in [7]. Thus, an empirical study has been conducted to evaluate the effectiveness of learning technologies in achieving lower order thinking skills when used to design learning objects. The remainder of this paper is structured as follows: section 2 provides information related to the adopted instructional design process for developing blended courses; section 3 reviews literature related to empirical evaluation of e-learning. Section 4 states the purpose of the study. Section 5 describes the methodology of the empirical study. Section 6 demonstrates the results of the study. Results are discussed in section 7, while, conclusions and future work are presented in section 8.

II. OVERVIEW OF THE DESIGN AND EVALUATION PROCESS OF DEVELOPING BLENDED COURSES
According to Ally in [2], "the development of effective online learning material should be based on proven learning theories". Bower proposed the following three components of "technology-based" learning design: content design, activity design and technology design [5]. Thus, the design process proposed in [7] adopted Bloom's Taxonomy, Redeker Taxonomy, and Guerra Scale in developing blended courses: (1) Bloom's Taxonomy is utilized for representing learning objectives of the course content; (2) Redeker Taxonomy is employed to classify the activity component; (3) Guerra Scale is used to classify available learning tools and technologies. The Bloom-Redeker-Guerra (B-R-G) mapping model as shown in Table 1 has distributed the Bloom's levels and the Guerra levels on the three Redeker interaction levels. The mapping model recognizes that in order to achieve Bloom's lower order levels of thinking (Knowledge and Comprehension), only receptive learning is required, which can be supported by technologies in the Guerra Scale levels 1, 2, 4 and 5.
Generally, during the development of a blended course, the course objectives and syllabus are determined during the analysis process. Then, the course content is divided into modules and the objectives of each module are identified. The realization of the module objectives produces a set of learning objects. The main activities of the proposed design and evaluation process consist of the following steps: 1. The process starts with identifying the objectives and modules of (face-to-face) course. 2. For each module, BT is used to classify each module objective into the six thinking levels. 3. For each module, the B-R-G mapping model is used to propose a number of alternative technologies that can be used to implement the LOs. The designer may select one of the technologies that support this learning objective based on other constraints such as expertise in the technology, budget, and availability of the technology. 4. When all modules of the course are designed, the proposed course design is evaluated by giving three weights for each learning object related to Redeker interaction levels: receptive, interactive and collaborative. Table II shows the weights used in evaluating a learning object which are chosen based on human computer interaction principles and theories.
In this study, the goal is to test the validity of the first dimension (receptive) of the evaluation scale. Hence, the designed learning objects for the study are restricted to be receptive learning objects that aim to achieve Bloom's lower order thinking skills, and only technologies in the Guerra Scale levels 1, 2, 4 and 5 are utilized in the selected learning objects. Evaluation of learning objects has two distinct paths: Summative or Formative. The summative approach evaluates the quality or success after the LO is implemented. Formative evaluation, on the other hand, assesses the LO during its design and development phases in order to adjust it before implementation [6]. The evaluation scale suggested in [7] is a type of formative evaluation, since it is an estimation of effectiveness during the design of LOs. Nevertheless, in order to empirically validate this evaluation scale, summative evaluation will be employed in this study.
Many studies assess the quality of online or blended learning objects using the summative approach via experts or end users reviews. For example, [15] investigated the effect of gender and computer experience on self-efficacy and motivation toward e-learning; and consequently, on the intention of using e-learning. They collected data from students' surveys after using an e-learning system. Furthermore, Lau and Woods examined external variables and users' beliefs and perceptions that influence the usage of learning objects. The study utilized web-based surveys to collect students' opinions. The results of this study indicated that learning object characteristics (such as technical quality and content quality) were important external stimuli for learners as they formed the perception and intention to use learning objects [13]. Leacock and Nesbit devised a tool called "The Learning Object Review Instrument" (LORI), which allows the users of learning objects to rate them according to nine criteria: content quality, learning goal alignment, feedback and adaptation, motivation, presentation design, interaction usability, accessibility, reusability, and standards compliance [14]. While, the later tool is used to evaluate multimedia learning objects, Wang and others devised a more general instrument to be used to evaluate an e-learning system. The results of their factor analysis identified six factors that measure the success of e-learning systems: System Quality, Information Quality, Service Quality, System Use, User Satisfaction, and Net Benefit. Their instrument was based on data collected via a survey of e-learning developers [19].
All the previous examples are qualitative evaluations based on surveys. Chawla and others took a different approach for evaluating LOs. They proposed a quantitative system that automatically evaluates the learning object in terms of defined parameters. First, the tool extracts the metadata fields of a learning object supplied by the contributor. The tool estimates the correctness and accuracy of metadata records, and then estimates other parameters such as reusability, granularity, linkage and complexity using defined metrics [6].
All these studies are useful in revealing factors that affect LOs quality and success based on the views of stakeholders. However, they do not show the contribution of these factors on the learning process and objectives. A study in [1] investigated the effect of students' learning styles on their perception of the learning process and their learning achievement. The results of this study showed that students' views of the learning environment (measured using a questionnaire) differ according to their learning styles. Meanwhile, students' achievement level (measured using exams and assignments results) was not significantly different. Another study in [10] identified four factors that affected the students' perception of elearning and showed that these factors were correlated with the students' approaches to study and achievements. Rodgers studied the effect of level of engagement in the eleaning system (measured as number of hours spent online) on student achievement (measured as the final module mark). The study results showed that the greater the e-learning engagement leads to better academic performance [18].
All these studies were evaluating e-learning systems which contain a variety of learning objects (e.g. course materials, discussion forms, practical exercises, etc.), which makes it difficult to identify the individual effect of each learning object or each learning technology. This study will evaluate each LO individually and will compare the effect of the learning technology on achieving learning objectives and on learners' perception.

IV. RESEARCH PURPOSE
As mentioned earlier, the evaluation scale adopted in this study has been based on human computer interaction principles and theories. The objective of this study is to validate the first dimension of this scale empirically. Hence, the study will answer the following question: Is there empirical evidence that LOs designed with multimedia technologies will better achieve the learning objectives of knowledge and comprehension and will better satisfy students than LOs designed with text and graphics technologies?

A. Research Hypotheses:
To meet the research objective, the hypotheses of the study are: Hypothesis 1: The lower order thinking skills (Knowledge and Comprehension) will improve more using learning objects designed with a more advanced technology according to Guerra scale.
Hypothesis 2: Students' satisfaction with learning objects designed using advanced technologies will be greater.

B. Participants of the Study
In order to test the hypotheses, this study was conducted as multiple experiments in the Faculty of Information Technology (IT) for two academic semesters. Three different courses were selected: Introduction to E-Commerce (EC), Software Engineering (SE) and Advanced Software Engineering (ASE). The selection of different courses in this study is intentional, in order to provide supporting evidence of the hypotheses that is not limited to a specific course. Participants were second and third year students enrolled in these courses. A total of one hundred students participated in this study.

C. Instruments of the Study
Two data collection instruments were used in this study: tests and questionnaires. Hence, quantitative analysis was used to analyze the data collected from the questionnaires and test scores. In order to measure the degree of learning achieved from the learning objects, the results of two small tests (Pre and Posttests) were traced to make comparisons. In each experiment the sample was divided PAPER COMPARISON OF EFFECTIVENESS OF DIFFERENT LEARNING TECHNOLOGIES into two groups, each of which was exposed to a different learning object. The learning achieved by students in each group was then evaluated. Also, a questionnaire was utilized to discover what students' opinions and attitudes were towards the learning objects content and presentation. The questionnaire was made up of 17 structured questions. The questions encompassed two evaluation criteria: utilized multimedia in the learning objects presentation and content. The questionnaire items were chosen from an online courses evaluation survey [9].

D. Learning Objects Preparation
Learning objects utilized in the experiments were developed based on the approach suggested in [7] for developing blended courses. However, this approach was applied on modules of the selected courses. The preparation process consisted of the following steps: 1. The process starts with defining the objectives of the modules for the selected courses. 2. Then the module objectives are categorized according to Bloom's Taxonomy. . Therefore, to test the hypotheses of the study, the first learning objects for all experiments were power point slides that were developed by the instructors of the courses. Meanwhile, the second learning objects were multimedia video files that included the same content provided in the first learning objects. Table IV shows the selected technologies utilized in each experiment. 4. The course instructors review the content in the two learning objects. They also approve that the two LOs are of satisfactory quality to achieve learning. 5. Finally, the assessment approach described in [7] is utilized to pre-evaluate the effectiveness of the selected learning objects. Since all learning objectives are classified as receptive learning objects, Interactive and collaborative weight values are zeros. Table  V shows the pre-assessment weight scales for each LO used in the experiments.

E. Procedure
All experiments followed the generic procedure shown in Figure 2, which went as follows: 1. Define the learning objectives of the selected modules. 2. Implement the learning objects using different technologies that support knowledge and comprehension thinking skills according to the B-R-G mapping model. 3. Prepare the pretest and posttest. Questions of the pre and posttests match the learning objectives and the content covered by the learning objects. 4. Introduce the students to the module content in a face-to-face learning mode. 5. Conduct the pretest among all participants to measure the students' pre-knowledge acquired from the faceto-face introduction of the subject. 6. Divide the participants into two groups and introduce them to two different learning objects. 7. Conduct the posttest among all participants. 8. Conduct the perception questionnaire among all participants. 9. Analyze the collected data using proper data analysis techniques, as will be explained in the results section. All statistical tests reported in this study were conducted with a significance level of 0.05.

A. Students' Achievement
The first research hypothesis was, "the lower order thinking skills will be improved using learning objects designed with advanced technologies according to Guerrra scale". In order to test this hypothesis, students' prior knowledge and post knowledge achievement test results were compared for all learning objects.
An independent-samples t-test was conducted to compare the students' average score for LO1 and LO2. Table  VI shows the results of the t-test. There was a significant difference in scores for LO1 (Mean= 8.28) and LO2 (mean = 9.17) with a p-value=0.03. The magnitude of difference in the means was small (eta squared = 0.04). This means that learning objects implemented with advanced technologies had more effect on students in achieving lower order thinking skills, although this effect was very small.

B. Students Perceived Satisfaction
There were 17 questions included in the student perception questionnaire. Students rated each item on a 5-point scale (5=strongly agree; 4= agree; 3=neutral; 2=disagree; and 1=strongly disagree). The aim of the survey was to compare students' opinions about the technology used to implement the selected learning object, thus testing the second hypothesis of the study. Four students in the experiments took the pre and post tests, but did not fill out the students' questionnaire. Table VII shows the mean scores of the 17 items for LO1 and LO2 for all experiments. In general most items were rated 4 (which is "agree") and above which indicated the satisfaction of the respondents with the learning objects. Items related to audio (such as item 4, 5, 6, and 11) were rated "neutral" (mean score around 3) because not all the learning objects contained audio. Further tests were done to explore whether students' satisfaction was different between LO1 and LO2.
An exploratory factor analysis was conducted to verify how items in the questionnaire relate to each other. We used Principal Component Analysis with Varimax rotation. The result of the Kaiser-Meyer-Olkin Measure is calculated (0.749) to assess the suitability of the data for factor analysis. Table VIII shows the loading of four independent factors. Factor 1 shows 6 coherent items related to the LO content (items 1, 13-17). Factor 2 shows 4 items related to the LO presentation (items 2, 4, 7, and 9). Factor 3 consists of items related to audio (which was not used in all learning objects). Finally, factor 4 consists of items related to suggestions for improvement.
Step 1 Define Learning Objectives(LOJ) that achieve lower order skills (Knowledge and Comprehension) of a Topic.
Step 9 Data collection and comparative analysis Step 6 Utilize LO2 developed using LT2 to Group 2 Step 6 Utilize LO1 developed using LT1 to Group 1 Step 5 Conduct pre-test Step 7 Conduct post-test Step 2 A. Prepare two Learning Objects (LO1, LO2) that satisfy LOJ using different Learning Technology (LT).
B. Design of a student perspective Questionnaire Step 8 Conduct the student Perception Questionnaire Step 3 Prepare pretest and posttest Step 4 Introduce students to the module face to face

PAPER COMPARISON OF EFFECTIVENESS OF DIFFERENT LEARNING TECHNOLOGIES
The first and second factors concur with factors identified in literature that affect LO acceptance [1,13,19]. Hence, we will consider them for further detailed investigation.
In order to investigate whether these factors are statistically significant between LO1 and LO2, independentsamples t-tests were conducted.
First, an independent-samples t-test for the content factor was conducted to explore whether students perceived differences between learning objects in terms of content.
The results of the t-test are shown in table IX. A p-value of 0.161 shows no statistical significance between learning objects in terms of content.
Furthermore, an independent-samples t-test for the presentation factor was conducted to explore whether students perceived differences between learning objects in terms of presentation. The means of the presentation factor were 3.4364 for LO1 and 4.2636 for LO2. The p-value was 0.0, which shows that there is a statistical significance between learning objects in terms of presentation. iJET -Volume 9, Issue 9: "Blended Learning", 2014 PAPER COMPARISON OF EFFECTIVENESS OF DIFFERENT LEARNING TECHNOLOGIES Next, an independent-samples t-test for the improvement factor was conducted to explore whether students' suggestions for improvements on learning objects were different when advanced technologies are used to implement the LO. The p-value was 0.24, which shows no statistical significance between learning objects in terms of suggested improvements. The means show that students were neutral in suggesting improvements on LO (LO1 mean= 3.85, while LO2 mean= 3.64).

VII. DISCUSSION
In this study, two assessment criteria were used: learning achievement and perceived satisfaction.
The results of the independent-samples t-test (table 6) show that when using advanced technology to design LOs, the learning achievement of the students is improved. This supports the first hypothesis in this research. It also concurs with studies reported in literature (e.g. [1,10,18]) that report evidence of improvement in students' grades when using more e-learning material. Furthermore, the results validate the weight scales assigned to LOs in the pre-assessment done in table 5.
Secondly, when evaluating participants' satisfaction, presentation design and content quality were the focus of the students' perception questionnaire.
When designing the experiments for this study, learning objects were chosen carefully to ensure they will have the same scope of content. In addition pre and post test questions were measuring knowledge covered within the learning objects content. No inference or any kind of high level processing of the content was needed to answer the exams questions since the goal of this study is to measure lower order thinking skills obtained from exposure to LOs. Hence, the result of the independent-samples t-test (Table  9) for the content factor of both types of learning objects was statistically insignificant, because the content of the material was controlled.
Furthermore, the result of the independent-samples ttest for the presentation factor of the two types of learning objects was statistically significant in favor of more advanced technology learning objects. This shows that students' perception of the learning object presentation design was toward more advanced technologies, which supports the second hypothesis.
This study has some limitations: (1) the sample size was small; (2) participants were IT students only. Despite this limitation, results provide empirical evidence that support the first dimension of the evaluation scale (Table  2), which is the purpose of this research.

VIII. CONCLUSIONS
This study has adopted the B-R-G mapping model to design and evaluate LOs for blended learning. The advantages of this model are: (1) Assist developers to systematically choose the learning technology based on the targeted learning objectives. (2) Provide early assessment of the effectiveness of LOs. (3) Reduce the cost of trial and error during the development of LOs.
The paper reports a number of experiments that were conducted to measure the effect of using different technologies to implement learning objects in achieving lower order thinking skills. Two aspects were considered and measured: students' learning achievement and students' perceived satisfaction. Learning achievement was evaluated using a pretest-posttest experimental model; while a questionnaire was used to measure the students' perceived satisfaction. Two types of statistical analysis techniques were used. Independent-samples t-tests were used to check whether statistical significant differences exist between learning objects implemented in different technologies. Factor analysis was used to investigate the relationship among items of the perception questionnaire.
The results of the empirical study presented in this paper show that when content is controlled, learning objects designed with advanced technologies better satisfy the lower order thinking skills objectives of a course and user satisfaction, which validates the first dimension of the evaluation scale.
It has been proven by studies reported in literature that collaborative technologies (such as forums, blogs, and video conferencing) assist in achieving higher order thinking skills (e.g. [11,16]). Fox and Mackeogh show that proper pedagogical design of e-learning facilities (discussion and peer-tutoring) has an effect on student engagement and subsequently on their learning [11]. While [16] suggests that asynchronous interactions and synchronous group activities can facilitate the development of higher level cognitive skills. These studies support the third collaborative dimension (synchronous and asynchronous) of the evaluation scale in Table 2.
However, it remains a future work for this group to validate the suggested weight scales for the second (interactive) dimension of the evaluation scale.