Providing a Multi-fold Assessment Framework to Virtualized Collaborative Learning in Support for Engineering Education

In previous research we proposed a virtualization process of live collaborative sessions from Web discussion forums and chats with the aim to produce interactive and attractive online learning resources to be played by learners, thus having a positive effect in learner engagement. In order to enhance further learning engagement, in this paper we endow our virtualization process with a multifold assessment framework that provides effective awareness and constructive feedback to learners from the original collaborative interactions amongst group members. The research here presented focuses on e-assessment of collaborative and social learning and extends it with Learning Analytics and Social Network Analysis techniques that are able to analyze and represent cognitive and social interactions underlying live collaborative sessions. The interaction data extracted from collaborative knowledge and social networking is integrated into the virtualized collaborative learning to produce an efficient and personalized awareness and feedback system about the collaborative activity and the social behaviour of the original participants of live collaboration. This paper describes both the conceptual and methodological research to build our multi-fold e-assessment framework. The research is evaluated in real context of e-learning and validated by empirical data and interpretation.


INTRODUCTION
Collaborative learning has become a controversial research field that focuses on the change from traditional education, oriented to individual learning, to a new environment where the learner and a learning community are the principal actors of their own learning [1]. Collaborative learning incorporates the learning community level and the learning consists in the evolution process of the learning community and the classroom must be reconceptualised as a knowledge-building community [4]. A broad literature has been provided over the last two decades on collaborative learning building new learning theories and dimensions [1][2][3][4].
Main collaborative learning theories [3] concern certain aspects, such as the definition of the collaborative situation, the interactions, the processes and effects as well as common dimensions, namely the collaboration scale (group size and time span), level of learning, and the depth of the collaboration. Computer-Supported Collaborative Learning (CSCL) [2] is one of the most important educational paradigms in the collaborative learning domain focused on how collaborative learning is supported by technology so as to enhance peer interaction and work in groups, and how collaboration and technology facilitate sharing and distributing of knowledge and expertise among community members. To achieve these aims, CSCL has become a broad multidisciplinary area, where technology meets psychology, philosophy, and pedagogy. Instructional designers and software developers, educational psychologists, learning theorists, computer scientists, and even sociologists are interested in this area of research [4].
In addition, virtualization can have significant impact on the efficacy of collaborative learning [5]. In particular, it has the potential to allow for aspects such as cultural cohesion to be fostered through the methods in which virtual worlds can facilitate embodiment whilst preserving anonymity [6]. As such virtualization impacts learner identity, it has pedagogic as well as technical implications; a principal consideration in the context of this research is how to effectively incorporate pedagogic, as well as technical elements into learning objects. This leads to new needs and expectations for collaborative learning.
Current collaborative technical support involves online collaborative tools that allow for interacting and learning through socialization. For the sake of socialization, Social Learning (SL) theories [7] play a fundamental role in this context by emphasizing observation and modelling of the behaviours, attitudes, and emotional reactions, in terms of continuous reciprocal interaction between cognitive, behavioural, and environmental influences [8]. The SL models are essential but there are more factors to consider, such as the different learner profiles: diverging, assimilating, converging, and accommodating. Translating these SL concepts to CSCL implies introducing factors, such as self-efficacy, incentive and benefit in the system. For instance, the serious games are an interesting environment where to develop the collaborative and social learning and offer particularly compelling outcomes [9].
Technological tools also provide learners with a wide range of social and interactive experiences though they need a holistic approach, which integrates all available social media in learning scenarios [10]. Scenario creation must reflect the learning requirements and also show its limitations in pedagogy. Therefore, the collaborative learning process must be arranged during the CSCL and SL scenarios, becoming a difficult task [11]. Certain requirements are to be considered when constructing a SL scenario using a CSCL model, such as the ability to observe the modelled behaviour, recall this behaviour and SPECIAL FOCUS PAPER PROVIDING A MULTI-FOLD ASSESSMENT FRAMEWORK TO VIRTUALIZED COLLABORATIVE LEARNING IN SUPPORT FOR… reproduce it. Tools such as IMS Learning Design and scripts also help educators construct effective learning experiences though they lack to specify several characteristics of the use of tools that mediate collaboration [12]. A further problem is the use of CSCL and SL scenarios in the context of formal, informal and intentional learning experiences [13]. It is necessary to define and differentiate the mentioned approaches to each of these areas.
Assessment of collaborative learning is a mature research field at present with a great impact on collaborative learning. Building a consistent assessment framework involves to know how knowledge can be extracted from CSCL and SL activities. This knowledge can then be used for assessing and also for monitoring and scaffolding the collaborative learning process. Driven by the users' interaction, a high amount of quantitative and qualitative information can be managed from on-line collaboration. For example, students' posts in on-line forums are labelled with certain indicators that provide qualitative information about performance and behavioural aspects of the collaboration [15]. Visualization techniques of this information have also a great impact on on-line social tools [10].
All above approaches and ideas lead to new needs and expectations for supporting assessment in CSCL and SL. In this paper, we identify the unique challenges associated with using learning objects across advanced technological platforms and within pedagogic frameworks that reflect on the strengths of these technologies to virtualize live sessions of collaborative learning while enhancing complex aspects, such as cognitive and social assessment [5]. To this end, collaboration and complexity are key aspects firmly considered as extension from the traditional learning objects to exploit the notion of Collaborative Complex Learning Objects integrated into the new technological concept of Collaborative Complex Learning Resource (CC-LR) [16].
The paper is structured as follows: Section 2 presents related work with concepts and technologies used in later sections. Section 3 presents our multi-fold assessment framework based on cognitive and social information to be integrated in the CC-LR approach. Section 4 shows a research methodology to empirically demonstrate the educational value of the CC-LR to support engineering education in real context of learning. Section 5 evaluates and validates the approach by an analytical data discussion and interpretation of the effects of using the CC-LR enriched with our assessment framework in our real contexts of learning. Section 6 concludes the paper by highlighting the key results achieved and outlining future work.

II. RELATED WORK
In this section, we provide an extensive overview of each of the technologies and paradigms related to this work, namely (i) Interaction data analysis in CSCL; (ii) Social Network Analysis; (iii) Virtualization of Collaborative and Social Learning. This overview will serve as background for setting the main goals of this research at the end of this section becoming the very rationale of our cognitive and social e-assessment framework for Collaborative Complex Learning Resources presented in this paper. This overview is based on previous research work [17].

A. CSCL Interaction Data Analysis
Computer-Supported Collaborative Learning (CSCL) is one of the most influencing research paradigms dedicated to improve teaching and learning with the help of modern information and communication technology [4] [18] [19]. A vast literature production and outstanding contributions have been published reporting on studies where the CSCL interaction process has been exhaustively analyzed [18]. In [20] it was investigated whether CSCL in asynchronous discussion groups results in enhancing academic discourse and knowledge construction. Their research work showed that students in the discussion groups were fundamentally task-oriented and that higher proportions of high phases of knowledge construction were observed. It was also evidenced by [19] that students were able to construct their own understanding based on their interactions.
In addition, intensive and successful research from the interaction analysis field has been achieved over the last years to facilitate the management by computers of the large amounts of interaction data from online discussions. Current efforts [20] aim to alleviate manual procedures while considering relevant aspects of the collaboration, such as how all participants are actually performing during the discussion and the dynamics of each participant with respect to the group. To this end, two levels of interaction analysis are considered, quantitative and qualitative level [22]. Quantitative indicators measure the participants' performance and dynamics (e.g., number of contributions written and read by each participant) as relevant information to model the group functioning and task performance [23]. Qualitative information has been also considered valuable to complete the labored task of interaction analysis and evaluation of contributions. [21], used a merging view of quantitative analysis within a qualitative methodology to build a model for the analysis of collaborative knowledge building in asynchronous discussion. However, the mere consideration of the depth of discussion threads, the number of messages and he relation among messages do not guarantee by itself the quality of the discussion; students' postings can be simply driven by socialization reasons and not directly linked to the development of the learning tasks [22] Quantitative indicators measure the participants' performance and dynamics (e.g., number of contributions written and read by each participant) as relevant information to model the group functioning and task performance [23]. An initial model based on the above considerations was already built and evaluated in previous research [24]. In this paper we enhance the social perspective of our model by incorporating Social Network Analysis (SNA) techniques in order to provide a clear way of analyzing the network structure and discovering the collaborative knowledge hidden in large volumes of well structured data. The use of SNA in the context of CSCL is reviewed next

B. Social Network Analysis in CSCL
Social Network Analysis is a method of Educational Data Mining whose goal is to understand and measure the relationships between entities in networked information [25]. SNA views social relationships in terms of network theory consisting of nodes (representing individual actors within the network) and connections or links, which represent relationships between the individuals [26]. In EDM, SNA can be used for mining in order to interpret and ana-SPECIAL FOCUS PAPER PROVIDING A MULTI-FOLD ASSESSMENT FRAMEWORK TO VIRTUALIZED COLLABORATIVE LEARNING IN SUPPORT FOR… lyze the structure and relations in collaborative tasks and interactions with communication tools. [26] [25]. To this end, the specialists performing the analysis must collect and process data through mining techniques [25] and do the required inference for filling up those indicators defined for measuring and evaluating the collaboration acts. In order to measure collaboration, SNA techniques include centrality indices, which are related with the position and relevance within a network [27], and measure how influential a person is within a social network.
When measuring collaboration by SNA, most works [27], [26] show a three-step process of collecting the information, analyzing and presenting the results [22]. In [26], for instance, SNA techniques are used in the context of discussion forums to measure effective participation. First, log files of the e-learning course are used to extract the social network underneath the discussion threads. Second, appropriate social network measures for effective participation are selected, and, finally, the measures are reported (i.e. visualized) appropriately. The attributes of messages in a forum, such as time, task phase, sender and how the message is received (degree centrality, etc), as well as the content, reveal the nature of the social relationships [28].
On the other hand, there are diverse ways to determine the boundaries of a social network [27] [29]. A straightforward way uses event-based approach to defining the boundaries of the network instead of identifying the participants of the collaborative tasks [30]. A different perspective starts from a relation-based approach and a small set of nodes deemed centred in the network and then expands the selection including others nodes sharing particular types of relations with [1]. These approaches are not mutually exclusive and can be used together [29]. Following this approach in the context of e-learning, once the network boundaries are known, outside students can be identified even if they still formally are involved in the network. This is found by realizing the lack of informal ties between these students and the rest of the group, showing the existence of some formal relationships but no actual involvement in the group [28].
The next question in SNA is about the type of interactions produced within the learning network. SNA study patterns of relations, not just relations between pairs [27]. Relations are measured in pairs of nodes, but the study of the effect and meaning of this relation requires broader patterns of ties within the network [31]. In [26], the interaction patterns in the network were also analyzed in order to understand many aspects of the collaboration, such as how the discussion was developed over time, collaborative work sequences, planning issues, delays, etc. Therefore, SNA may help identify patterns of relationship between people who are part of a social network [28], and the social environment can be mapped as patterns of relationships among interacting members [32].
According to [33], patterns in social networks can be identified and constructed using mainly two methodologies: inductive pattern mining, seeking what is the pattern from a set of models, or deductive pattern mining, identifying the essence of generic models for solutions to recurrent problems that learning designers identified previously [34]. Depending on the granularity or the detail level, prestructuring collaboration can be accomplished in a coarsegrained process level and/or fine-grained level of detailed learning actions.

C. Virtualization of Collaborative and Social Learning
Collaborative Complex Learning Object (CC-LO) and their particularization, such as Storyboard Learning Object (SLO) were first presented and discussed in [35]. This new concept was justified by setting up two research questions about what makes a Learning Object (LO) [36] collaborative and what makes a LO complex, which cannot be easily answered by current standard learning objects.
The key differentiators from the standard LO include multiple levels of abstraction from pedagogic context, learners, and representational medium (complexity), as well as intrinsic support for interaction across the object (collaboration). To accommodate the above concepts with the model proposed by the authors [35], a CC-LO is embedded into a Virtualized Collaborative Session (VCS). A VCS is a registered collaboration session augmented by alternative flows, additional content, assessment, etc., during an authoring phase (subsequent to the registration phase). The VCS is interactive and animated (by movies or comic strips) and learners can observe how knowledge is constructed, refined and consolidated (see Fig. 1).
In this paper, we extend our research towards the invention of an innovative learning resource based on the CC-LO approach. We propose the concept of Collaborative Complex Learning Resource (CC-LR) by integrating CC-LO into a complex learning material that is used, adapted and reused extensively in academic courses beyond the original collaboration (Fig. 1). To this end, the CC-LOs can be edited by the VCS to include complex aspects of the learning process, such as cognitive and social assessment.

III. AN ASSESSMENT FRAMEWORK FOR COLLABORATIVE AND SOCIAL LEARNING SCENARIOS
The assessment framework proposed in this research is based on the integration of several models and methods (see Table 1, [15] and [37], for the details of this model).
Two main dimensions are considered in our framework: cognitive and social assessment. They are described next.

A. Cognitive assessment approach for evaluating collaboration in CC-LR
The structure of a long interaction in a collaborative discussion is constructed cooperatively by using the exchange as the basic unit for communicating knowledge. Three general exchange structure categories are considered: give-information, elicit-information and raise-anissue, which consist of different types of moves [38] and describe a generic discourse goal. More specifically, the goal of the actor who initiates the give-information exchange is to inform his/her partners about a certain situation with the aim to change the partners' mental states. Informing includes moves that explain, give an opinion, describe or remind a situation in different ways. The actor goal of the elicit-information exchange is to elicit the partners' state of mind (knowledge, beliefs, attitude, desire or abilities) of a situation, in which the actor is not aware or certain about. The actor goal of the raise-an-issue exchange is to raise a problem or question to be resolved by the participants, which causes to explore their state of mind (knowledge, beliefs, etc). Finally, peer assessment can be effected to complete the evaluation of each contribution made. The roles that these moves play in the exchange as well as the degree of success of that role determine the successful completion of the exchange goal (see Table 1 for the whole picture of indicators that assess collaboration and the exchanges involved).
Based on the above considerations, deferred or retrospective time assessment of live discussions allows for understanding how the original collaborative interactions developed over time. This information is displayed in the CC-LR by a variety of elements that contribute to the understanding of the nature of the collaborative interactions, such as the learners' passivity, proactivity, reactivity as well as the effectiveness and impact of their contributions to the overall goal of the collaborative learning activity (see Table 1). Note that passivity is related to participation frequency, while proactivity is related to contents' quality [15]. Therefore, the learner consuming a CC-LR achieves a better understanding of the collaborative learning process while improving the overall collaborative and social experience. For instance, by constantly showing in the CC-LR cognitive assessment information about the live collaboration, the learner can develop reflective and experiential learning skills by analysis and application [40]. Large amounts of information data are considered in the CC-LR from asynchronous interaction, which includes complex issues of the collaborative work and learning process (e.g., group well-being as well as self-, peer-and group activity evaluation) [41].

B. An SNA Approach for Social e-Assessment in Collaborative Complex Learning Resources
As mentioned previously, students use CC-LRs to develop their collaborative competences through a sequential process, in which assessment and emotional indicators, rules, and the use of the CC-LR are continuously evaluated. New forms of social network assessment are incorporated to empower the learning experience and improve the student's engagement. In order to measure the networking interactions from the live discussion, we consider the following centrality indices from SNA [2]:  [15].
• Degree centrality: It is measured counting the number of connections to other persons. • Closeness centrality: it shows the distance between one student and the other in the network. • Betweenness centrality: it measures how important a student is in bridging two different parts of a network. • Eigenvector centrality is a measure of the importance of a node in a network. Next, in order to understand and measure the social perspective of the networking interactions, we follow [4] and consider the following SNA indicators, which help identify the network members (nodes) and the relations between them (see Table 2): member similarities, social relations, social and social flows. This social network approach annotates and examines a variety of elements that contribute to the understanding of the social nature of the collaborative interactions in the network. The aim is to provide the overall assessment model.

IV. RESEARCH GOALS AND METHODOLOGY
The aim of this research is to develop an innovative assessment framework for CSCL combining personalization, collaboration and networking aspects within a cognitive and social based approach. The ultimate goal is to overcome the quoted limitations of interactivity, challenging and context aware of current e-learning systems and content, while enabling learners' demand of empowerment, social identity, and authentic learning experience. To this end, we evaluate our multi-fold assessment framework by considering new ways to assess the meaningful interactive behavior collected during live

Metrics from the original CSCL sessions
Degree centrality Number of ties directed to each node Indegree). Number of ties that each node directs to others (Outdegree).

Closeness centrality
Measured for each pair of nodes: number of nodes in the path from one node to other.

Betweenness centrality
Number of times a node acts as a bridge along the shortest path between two other nodes.

Eigenvector centrality
Assigns scores to all nodes in the network. Connections to high-scoring nodes contribute more to the score of the node in question than equal connections to lowscoring nodes.

Member similarities
Based on previous collected data, from a previous test or institutional data base.

Social relation
Social relation is qualified from the emotional information added and messages collected.

Social interaction
Measured by comparing a pre and post test of the collaborative activity.
Social flow Measured from the material exchanged during the collaborative activity.
retrospective sessions of collaborative learning and facilitating an easy implementation of it in a variety of environments, and in particular the CC-LR approach, which is the application context of our research (see Fig. 1). Following this research goal, the aim of this section is to present the results of the execution of the different rounds of experimentation and validation of the research and technology developed in this research. To this end, a practical method oriented to the experimentation of the tools developed and organized as prototype scenarios and its validation in real situations in different educational sites is followed.

A. General methodology
A comprehensive experimentation study is developed in this subsection describing all activities that have been undertaken during the experimentation, evaluation and validation stages of this research. The purpose of the experiments is to collect information about the experience of incorporating the multi-fold assessment framework (Section 3) based on cognitive and social aspects into real collaborative learning and its impact in the learning process.
The study includes, for both the Open University of Catalonia (UOC) 1 and Cadiz University (UCA) 2 pilot sites, details on the goals and hypotheses, the method (including number and type of participants, apparatus and stimuli, and procedure), and the evaluation and validation results. This is the standard structure to report empirical 1 The Open University of Catalonia (UOC) is located in Barcelona, Spain. The UOC offers distance education through its Virtual Campus since 1994. Currently, about 50,000 students and 3,700 lecturers conduct 6,400 online classrooms from a great variety of graduate, postgraduate and doctorate programs. UOC web site: http://www.uoc.edu 2 The Cadiz University (UCA) is located in Cádiz, Spain. UCA offers blended learning through its Campus Virtual supported by Moodle. The School of Engineering of UCA, which participates in this research, has more than 3300 students and 200 teachers. The school offers a total of 6 engineering degree and 3 master's programs, and a doctoral program. results following APA guidelines (American Psychological Association, 2010).
Communities of user groups (in general, students and teachers/lecturers) were organised in each pilot site (i.e., UOC and UCA), which are educational environments with full or relatively limited e-learning quotes (i.e. full virtual education and blended learning), and in which the extended computational capabilities of this thesis enabled the exploitation by teachers and students of existing advanced educational technologies. For each scenario of use a devoted user group was developed drawing from two different academic contexts, namely Software Engineering and Computing Fundamentals.
All of the experiments are based on the VCS system and the embedded CC-LR learning resource developed, experimented and validated in previous research participated by the authors. In the experiments reported below, the UOC experiments results [16] were used as a reference (i.e. control group), which facilitated the data comparison and interpretation of the results extracted from the UCA experiments (experimental group).

B. Experiments
This subsection presents the objective of experimenting the developed tools and methodologies in order to provide feedback to theoretical and technological activities presented in Section 2. It includes, as well, the evaluation and validation of the impacts of the innovative features offered by CC-LR inside the selected learning environments. The following three scenarios were experimented at UOC and UCA sites (see also Table 3): • UOC: This scenario provides live sessions of collaborative learning based in CC-LR tools to be experienced and played by learners. During the resource execution, learners observe how avatars discuss and collaborate, how discussion threads grow, and how knowledge is constructed, refined and consolidated. The novelty of this experiment is the evaluation of a multi-fold cognitive assessment approach included in the CC-LR that provided effective awareness and constructive feedback to learners from both the live collaborative interaction amongst group members. • UCA-1: The aim of this scenario is to enhance learning engagement using CC-LR with the multi-fold cognitive assessment approach by adding a social dimension with new SNA-based indicators on the original participants. SNA drives the implementation scenario with special emphasis on social learning dynamics. • UCA-2: This scenario provides a new step on designing a coherent and efficient assessment system for collaborative and virtual social learning. The experience is specifically focused on social assessment based on SNA techniques that are able to analyze and represent social network interaction during live sessions of collaboration virtualized by using CC-LR.

C. Research goals and hypotheses
The main goal of the experiments is to evaluate the enhancement and improvement of the learning performance and outcomes by means of the CC-LR approach that provides full support to the collaborative and networking interactions occurring during live collaborative learning activities. In particular, the aim is to evaluate the influence SPECIAL FOCUS PAPER PROVIDING A MULTI-FOLD ASSESSMENT FRAMEWORK TO VIRTUALIZED COLLABORATIVE LEARNING IN SUPPORT FOR…  [16] for more details.

UCA-1
Experimenting with the UCA classroom with 117 undergraduate students enrolled in the first-grade course "Computer Structure Fundamentals" of the Electrical and Computer Engineering program. Eventually, 17 students participated in the experiment" during the first 2 weeks of April 2014.

UCA-2
Experimenting with the UCA classroom with 24 participants enrolled in the third-grade course "Advanced Computer Structure" (ASC) and first year master subject: "Ubiquitous Systems" (US). The experiment was performed in the last week of April and the first week of May 2015.
in the learning process of the cognitive and social information included in the CC-LR. We focused on the following scenario's goals (G) and hypotheses (H): G1. To enhance and improve learning performance and knowledge acquisition by means of the CC-LR approach G2. To improve the assessment features of the CC-LR approach by self-evaluation and formative feedback in terms of cognitive and social information. G3. To improve usability, emotional aspects and functionality of the CC-LR approach by CC-LR resources generated from Collaborative Learning and Social Networking. G4. To improve the efficiency of the CC-LR. H1. The use of cognitive and social indicators applied to CC-LRs allow users to acquire more knowledge from collaborative learning activities. H2. The use of the CC-LR created from real collaborative activities enriched with LA and SNA techniques contributes to improve the virtual learning activities, increasing students' motivation and engagement. H3. The levels of usability, emotional aspects and functionality of the CC-LR are satisfactory. H4. CC-LRs are considered as a valuable educational resource by students.

D. Method
Following the APA guidelines to report empirical results proposed by [14], information about the participants, the apparatus used for experimentation and the procedure of the experiment are provided in this subsection.

1) Participants
The participants in the mentioned three scenarios of use are those shown in Table 3.

2) Apparatus and stimuli
The CC-LR contained cognitive and social assessment information, both deferred and immediate (see Section 2 for further information). Deferred cognitive assessment of past sessions can be added in the CC-LR by the Editor tool as performance indicators of the original participants in terms of activity and quality as well as social indicators, such as (betweeness) centrality and (degree) communication during the whole discussion. In the CC-LR, every avatar assumes representative icons (e.g., coloured hats, medals, etc.) that show selected performance indicators of the collaboration and make the deferred social interactions easier to understand. In addition, extra information about the contribution is also provided in order to show the primitive type of interaction occurred (e.g., request or give information, clarify, explain, solve a problem, etc.). The ultimate purpose is to help learners during knowledge acquisition. See Fig. 2 for the mentioned CC-LR features.
Immediate cognitive assessment was provided in the CC-LR by incorporating assessment scenes with formative assessment tests in certain points of the storyboard in order to evaluate the acquisition of knowledge of the current learner after a certain amount of scenes consumed (see [16]).

3) Procedure
Quantitative and qualitative data were collected from questionnaires containing quantitative and qualitative questions. The answer categories will vary between rating scales, multiple choice or open answers. Rating scales were selected to make the evaluation by the students easier, and made it possible a fair evaluation in terms of agreement or disagreement levels.
After the assignment, the students were required to fill out a questionnaire, which included the following 5 sections: (i) identification data (names and username); (ii) open questions about the knowledge acquired during the discussion; (iii) test-based evaluation of the tool supporting CC-LR; (iv) test-based evaluation of the knowledge acquisition from using the CC-LR; (v) test-based evaluation on the usability and emotional aspects of the VCS system.
For qualitative statistical analysis, open answers were summarized in the questionnaire. For the quantitative statistical analysis basic statistics were employed, such as Mean (M) and Standard Deviation (SD). This quantitative analysis was complemented by employing accepted statis-PROVIDING A MULTI-FOLD ASSESSMENT FRAMEWORK TO VIRTUALIZED COLLABORATIVE LEARNING IN SUPPORT FOR… tical procedures, such as Chi-square (!2) so as to compare the observed scores to the expected scores [34].
The rating scales for the quantitative questions were based on the usual 0-10 point scale. The rating scale was from the lowest (0) to the highest (10) considerning a "excellent" assessment from 8 to 10, "good" from 6 to 8, "pass" (5-6) and a "fail" assessment from 0 to 4, being the "pass" assessment the expected score for each question (df = 1 and p < 0.05 for the calculated !2).
For the section v (usability of the VCS player showing the CC-LR) we used the System Usability Scale (SUS) developed by Brooke (1996) hich contains 10 items and a 5 point Likert scale to state the level of agreement or disagreement. SUS is generally used after the respondent had an opportunity to use the system being evaluated.
Finally, to investigate in which emotional state the students using the new system, which include 12 items of the Computer Emotion Scale (CES) (Kay and Loverock, 2008). The CES scale is used to measure emotions related to learning new computer software. Research showed that the 12 items are describing four emotions: • Happiness ("When I used the tool, I felt satisfied/excited/curious.") • Sadness ("When I used the tool, I felt disheartened/dispirited.") • Anxiety ("When I used the tool, I felt anxious/insecure/helpless/nervous.") • Anger ("When I used the tool, I felt irritable/frustrated/angry") The answer categories in this section are "Never", "Sometimes", "Usually" or "Always".
Finally, for one of the experiments (UCA-1), an additional procedure was added to evaluate the knowledge acquisition of the group when studying with the CCLR. The procedure was developed in two sequential steps: (1) a pre-evaluation test (before starting the experience with the CC-LR) and (2) a post-evaluation test (right after finishing the study with the CC-LR). The pre-evaluation test measured the previous knowledge of the participants in the CC-LR experiment. Even if the CC-LR is designed as a self-sufficient learning tool, the pre-evaluation test is useful to know what kind of previous concepts students know. The post-evaluation test made a global learning evaluation of the topic. The results of the pre-and postevaluation tests are calculated in terms of correct and wrong answers with a Mean Median and Standard Deviation of the scores obtained of all the answers. This way we can compare the evolution of the knowledge acquired in the group of participants between both tests through the study with the CC-LR.

V. EXPERIMENTAL RESULTS AND DISCUSSION
Following the methodology described above, in this section we focus on usability and emotional aspects (evaluation results) as well as knowledge acquisition and worthiness of the CC-LR (validation results). UCA-1 formed the experimental group reported below and the results were compared to UOC results (control group). The UOC experiment was already reported in [16] from where we extracted the data.

1) Usability of the CC-LR
To evaluate student's satisfaction of the experimental group with the tool, enriched with cognitive and social information, as for an efficient and user-friendly management (H3), we collected data from students' ratings and open comments on the usability/functionality/integration of the tool.
To investigate the overall usability of the CC-LR, we used the SUS included in section (v) of the questionnaire. As mentioned, the answers were given on the 5-point Likert scale, so that students could state their level of agreement or disagreement. The rating scale ranged from "Strongly disagree" (1), "Disagree" (2), "Neither agree nor disagree" (3) to "Agree" (4), "Strongly agree" (5).
SUS scores have a range of 0 to 100 with an average score of 68, obtained from 500 studies. A Score above a 68 would be considered above average and anything below 68 is below average. A score above an 80.3 is considered an A (the top 10% of scores). Scoring at the mean score of 68 gets you a C and anything below a 51 is an F (putting you in the bottom 15%).
After calculating the SUS score for each student, we got an average for 17 SUS scores of 72.50 for the experimental group, thus above the SUS mean and also above the control group (25 SUS scores of 69.27). Next, we present the most relevant results of the SUS score by providing several statistics: Mean (M), Standard Deviation (SD) and Median (Md).

2) Emotional aspects
Regarding the students' emotions of the both experimental group (UCA-1) and control group (UOC) during the work with the CC-LR (H3), the results from a 4-point rating scale are presented next, and compared to the results of the control group: • Happiness (M=1.52, SD=1, Md=1). This result is slightly better than the control group (M=1.13, SD=0.67, Md=1) showing the students found the CC-LR interesting and were curious with the assessment scenes incorporated in the CC-LR (cognitive and social). • Sadness (M=0.88, SD=0.99, Md=1). This result is slightly worse than the control group (M=0.50, SD=0.78, Md=0) though with high SD showing that a few students felt quite sad when using the new resource while others felt not sad at all. All in all, most of students felt no sadness at all or had that negative feeling just sometimes. • Anxiety (M=0.52, SD=0.79, Md=0). This result is slightly worse than the control group (M=0.45, SD=0.72, Md=0). However, this result is still very good with Md=0, which means that students of experimental group did not experienced this bad feeling. • Anger (M=0.52, SD=0.79, Md=0). This result is slightly better than the control group (M=0.54, SD=0.77, Md=0) and in line with Anxiety, which means that students of experimental group did not experienced these bad feelings.
In summary, students felt more often happiness than sadness, anxiety or anger when using the CC-LR to study with the CC-LR equipped with cognitive and social based assessment indicators The results in general are similar in SPECIAL FOCUS PAPER PROVIDING A MULTI-FOLD ASSESSMENT FRAMEWORK TO VIRTUALIZED COLLABORATIVE LEARNING IN SUPPORT FOR… the experimental group than the control group though, being the most noticeable result the highest value in happiness while the students felt the same very low levels of anxiety and anger.
A particular increase of Sadness from the control group can be explained by the incorporation of SNA information which was not easy to understand. This might have caused higher steps of sadness, especially when a few of students failed to interpret the meaning of SNA indicators. However, as mentioned, negative feelings and in particular sadness emotion was assessed quite low.
In overall, this is a good result that confirms that the incorporation and increase of new assessment indicators does not influence the emotions negatively when studying by the CC-LR approach. Finally, this result is in line with the results presented above concerning usability.
At this point, following the methodology set out above proceed to validate the improvement of knowledge acquisition (H2), worthiness from the control group of the CC-LR (CC-LR) enriched with assessment as an educational tool (H4) as well as the acquisition of collaborative knowledge with this new type of learning material (H1).

3) Knowledge acquisition
As mentioned in the methodology, the knowledge acquisition was evaluated in UCA-1 by pre-and post-tests. Both included different cognitive questions though very related to the topic shown in the CC-LR. Question of both tests had the same level of knowledge and difficulty so that the results of both groups of tests could be compared.
The pre-test was developed to assess the previous concepts necessary to understand the CC-LR. Six key questions were considered for assessing the process of making chronograms of numeric digital operations. Some questions had a list of predefined answers to choose the correct one while other questions were open with free text to provide the answer. Open questions were assessed manually. Each question was assessed with a score in the typical academic scale of 0-10.
The group scores obtained by the pre-test (M=7.08, SD=1.47; Md=8) showed that most students were able to recognize the main cognitive concepts of the CC-LR, evaluated with a success rate ranging from 54.54% to 90.9%. Only two questions showed low percentages near the 50%. These two questions were considered advanced as they required deeper understanding of arithmetic digital circuits.
The post-test was launched right after the study with the CC-LR. The procedure, type and content of questions and assessment was similar to the pre-test. The group scores obtained by the post-test were better than the pre-test as all the participants answered most of questions correctly (94%), thus showing that most of students understood better the main concepts of the CC-LR after the experience with the CC-LR. All wrong answer were concentrated in a specific student. On the other hand, scores obtained (M=7.26, SD=2.12; Md=8) were quite similar to the pretest-scores (M=7.08, SD=1.47; Md=8) though slightly better, with SD higher in the post-test as the outlier coming from the specific student with wrong answers, who got very low scores. Overall, these results confirmed the improvements in knowledge acquisition through studying with the CC-LR.

4) The VCS as a valuable resource
In this section we evaluated the level of worthiness of the improved version of the CC-LR enriched with SNA indicators for the assessment information supported by the VCS as an educational tool (H4). To this end, we collected quantitative and qualitative data in order to know the user's satisfaction in the experimental group with the tool. Both quantitative and qualitative data were collected in section (iv) from 5 open questions of the questionnaire addressed to students. All this data was also collected with a similar questionnaire and questions from students of the control group (see UOC experiment in [16]). This will make it possible a fair comparison between both groups.
The following 6 questions were asked to the experimental group at UCA-1 related to evaluate the CC-LR: Q1. What you liked and what you did not like from the CC-LR (assess the CC-LR from this view in the scale 0-10). Q2. Let us know your opinion about the potential of the CC-LR to observe how people discuss and collaborate, and how knowledge is constructed (assess the CC-LR from this view in the scale 0-10). Q3. Do you think the CC-LR have helped you acquire more knowledge about the discussion topics in comparison to the text-based forums? (assess the CC-LR from this view in the scale 0-10). Q4. Let us know how the CC-LR material generated from a real debate by students has helped you to better understand the contents, compared to traditional materials (books, web, etc.). Q5. Do you think that both the cognitive indicators (activity and quality) depicted below the avatar and the test questions integrated in the CC-LR allowed you to understand the contents of the CC-LR and acquire more knowledge? (assess the CC-LR from this view in the scale 0-10). Q6. Do you think that the social indicators (centrality and communication) depicted below the avatars have helped you to follow the discussion and discern between participants and their contributions? (assess the CC-LR from this view in the scale 0-10). Q1 through Q5 are the same as those included in the control group's questionnaire (see the UOC experiment in [16] On the other hand, Q6 was new for UCA-1 experiment (not performed in UOC) and particularized to the incorporation of SNA indicators into the assessment information provided. Therefore, Q1 through Q5 were compared between the experimental and control groups whilst Q6 was analyzed by collecting data from both UCA-1 and UCA-2 experience so as to validate the inclusion of SNAbased indicators into CC-LR as assessment information.
All the students of the experimental group (UCA-1) provided assessment marks. After calculating the 0-10 scale for all the questions of the experimental group we got a general Mean score of 6.73 (SD=2.06 and Md=6.8). This result is slightly better than the control group (M=6.52, SD=1.42 and Md=6.7) (see above the UOC experiment) and in line with the previous results on usability and emotions, hence these results confirm the CC-LR enriched with assessment information as valuable educational resource. Next we present the results of each question for the UCA-1 experiment (see the results of the corresponding questions in the UOC experiment in [16]. SPECIAL FOCUS PAPER PROVIDING A MULTI-FOLD ASSESSMENT FRAMEWORK TO VIRTUALIZED COLLABORATIVE LEARNING IN SUPPORT FOR… Q1: Students of the experimental group in general liked the CC-LR enriched with SNA information (M=6.40, SD=1.44, Md=6.75) more than the control group without this information (M=6.76, SD=1.92, Md=7). In general, students liked the functionality and video format of the learning resource (talking avatars, test scenes, graphical information to assess the contents, etc) while they did not like the length of the resource as it took too long to study all of it. They also did not like the "robotic" voice of the avatars. As for the assessment aspects, students did not like the too many test questions included in the scenes and that some of the tests were not very related to the CC-LR contents (they needed to check other materials).
Q2: The experimental group (M=6.94, SD=1.60, Md=7) also achieved better scores than the control group (M=6.54, SD=1.93, Md=7). The students in general mentioned that the CC-LR is a powerful learning tool to understand complex topics in an easier way from the contributions of other peers. Other students also provided positive opinions by mentioning that the CC-LR fosters to seek and learn extra-curricular concepts from the material attached in the CC-LR.
Q3: The experimental group achieved significant better scores (M=6.82, SD=2.32, Md=7) than the control group (M=5.92, SD=2.34, Md=6.50). This result confirms the didactical purpose of the CC-LR as learning materials and the enrichment with SNA information reinforces their didactical purpose. The students in general found the CC-LR easier to read and understand from listening and observe the scenes than traditional reading long lists of posts in traditional web forums, though sometimes requiring a speed rate of learning too fast. Certain students, however, considered the CC-LR was not very helpful though they found that many times it was easier to understand a complex explanation given by peers (within the CC-LR scenes) than from reading text-based posts.
Q4: The experimental group (M=6.82, SD=2.32, Md=7) also achieved significant better scores than the control group (M=5.88, SD=2.09, Md=6). Some students indicated that the CC-LR looks like a video, thus being more attractive than traditional materials. The students also indicated that the CC-LR attracted more their attention. However, some students found the CC-LR useful to complement traditional material as it served to understand certain concepts. Even if the CC-LR contents come from actual collaboration, this material lacks interaction, thus not providing a real collaborative experience. Finally, certain students indicated they preferred to take notes rather than watching the material.
Q5: This question about the incorporation of cognitive assessment indicators was quite positive. The experimental group got a mean score as high as 6.53 (SD=2.12, Md=6.0) (see the below figure), and slightly worse scores than the control group (M=6.94 (SD=1.12, Md=7.0) (see the UOC experiment and Fig. 3). However, the SD is quite high pointing out the dispersion of opinions. The scores of both groups are well above the average (5.0), which confirms these indicators were useful and are in line with the positive effect in knowledge acquisition with the CC-LR addressed above.
Q6: The sixth and last question evaluated the social assessment indicators based on SNA and how useful they were to show the social dimension. As mentioned in the evaluation methodology (see above the methodology sub- section), for this specific question a different methodological approach was followed. The participants from the UCA-2 experiment (May 2015) were used as experimental group while the UCA-1 participants (April 2014) formed the control group (see Fig. 4 and the details of the UCA-1 and UCA-2 experiment above).
The following table and figure show that the experimental group (UCA-2) got a mean of 6.38 (SD=1.8, Md=6), which is slightly lower than the control group UCA-1 (M=6.63, SD=2.12, Md=7). However, the scores of both groups are well above the average (5.0), which confirmed that the SNA indicators were also useful and are in line with the positive effect in knowledge acquisition with the CC-LR addressed above.
The students of UCA-1 and UCA-2 provided balanced opinions on the SNA indicators. In particular, most of UCA-1 students found interesting to have this information to understand better the content of the scenes. However, they complained that the indicators were hard to see and interpret, and suggested providing better integration of these indicators into the graphical interface of the CC-LR (eg. show avatar's faces with social features, etc) so as to best understand the social dimension of the avatar.
The UCA-2 students commented more specifically on the SNA indicators. For instance, some of the students mentioned that the meaning of the "centrality" indicator was not well understood, whether it was 'green' or 'red', while other students pointed out the this indicator was taken into account to understand the degree of acceptance of the avatar's involvement in the discussion. The state of each indicator (green, yellow and red) was also commented by students as during the first part of the study with the CC-LR students paid attention to the state of each indica- tor (especially if 'green' or 'red') but for long CC-LRs with many scenes, they reported to lose attention as the CC-LR went by.

5) Conclusion of the experiments
To conclude the UCA experiments, the CC-LR approach was found to potentially and easily create a collaborative learning environment anywhere and anytime this learning model is hard to incorporate. Overall, the UCA students understood the CC-LR was a good opportunity to explore further for learning through virtualized collaboration.
The design and implementation of the CC-LR leveraged a previous real collaborative experience from Web discussion forums, which was selected and improved to meet the requirements of the CC-LR, with the main objective of re-creating (ie. virtualizing) a live collaborative learning situation while correcting an improving the learning process. SNA analysis can helped to complete the previous collaboration, using the collected relevant information acquired from previous collaborative forums. The data provided information to assess and measure the social interactions and features underlying the collaboration, thus augmenting and visualizing in the CC-LR the information of the collaborative processes. The data analysis from SNA techniques provided a good reference for different levels of assessment: collaborative group, collaborative tools, and collaborative dynamics and the assessment methodology.
Therefore, it is possible to state that the SNA indicators integrated into the CC-LR became a reliable source of information at different level of learning assessment, such as individual and group performance and dynamics. Although the SNA indicators were found useful, experiences conducted in this research showed that this information should be integrated in a more natural way in the CC-LR. In addition, levels of frustration appeared, produced by high expectations to learn faster and easier.
Finally, based on the results obtained in the UCA experiments, the formulated goals (see above in the Methodology section) for these experiments were achieved, as follows: In general the students liked the extended features of the CC-LR and found them interesting to understand better the content of the CC-LR (G1), even more than the control group. During the study with the new version of the CC-LR, the students found them very easy to use as no relevant technical problems were reported and also from the usability perspective (G3). The majority of students could generate the CC-LR efficiently (G4).
The experience with the CC-LR in terms of SNA indicators to assess the original collaborative was found positive (G2) though the improvements of the assessment approach was found not significant. Further work is required to enhance the social information provided in terms of both number of indicators and better integration in the CC-LR for making it easier their interpretation. All in all, the social information provided was appreciated in line with the cognitive indicators of previous experiences (UOC experiment).
Finally, students provided some hints to improve the CC-LR in general (G1-G4) as well as they suggested to use this type of learning resources to learn extra-curricular topics and for complex concepts.

VI. CONCLUSIONS AND FUTURE WORK
This paper presents an assessment framework from the collaborative knowledge and social networking interactions found in CSCL to be integrated into the CC-LR approach produced from live discussion activities. The assessment framework includes a model for cognitive and social assessment in CSCL and provides a complete assessment of the actual performance of the original CSCL participants. This in turn improves and enhances the CC-LR experience with cognitive and social outcomes when this learning resource is consumed by individual students as regular teaching material.
To sum up, we believe that social and cognitive indicators become a reliable source of information at different level of learning assessment, such as individual and group performance and dynamics. Although these indicators were found useful, experiences conducted in this research showed that this information should be integrated in a more natural way in the CC-LR. In addition, levels of frustration appeared, produced by high expectations to learn faster and easier.
Future work is to address the above issues by shortening and summarizing CC-LR sequences for achieving significant learning more effectively. We also plan to improve the test scenes embedded in the CC-LR for selfevaluation purposes by providing more dynamic interaction without interrupting the learning process. Finally, next steps will include research on further LA and SNA indicators to assess other dimensions of the cognitive and social networking found in live CSCL activities.
Finally, the statistical tools used have the inherent limits of descriptive statistics utilized. As the experiment was conducted with subjects randomly assigned, this research could improve the statistical results and conclusions using inferential statistics, thus asserting whether there are significant differences among the studied variables, correlations among variables, etc. We plan to consider this approach in next research steps.