Quality Assurance for Reusable Learning Objects on a Peer-To-Peer Network

— In this research, improving on the quality of Reusable Learning Objects (RLOs) on a Peer-To-Peer (P2P) network is considered. The RLO was first redesigned to have a fundamentally inherent pedagogical structure, which gave it an immediate foundational level of quality in terms of opportunities related to reusability. Applying the Learning Object Review Instrument 1.5 (LORI 1.5) demonstrated that some of the elements are inherent in this new RLO design, so there was no need to constantly have such features evaluated with LORI. A modified LORI was therefore developed in order to evaluate the remaining features of the RLO. The research identified these remaining elements to produce a Review Rubric for scoring the RLO’s quality. In addition, an algorithm is given which considers one or more subject-matter experts as part of a review process. Utilizing the subject-matter experts in a P2P network involved the creation of special nodes to ensure data integrity and post-availability of the review scores for RLOs. The research concludes that the redesigned RLOs along with the corresponding Review Rubric and scoring algorithm produces a system suitable for a P2P network, where for the first time, RLOs can be shared of assured quality to promote eLearning within P2P networks.


INTRODUCTION
If Learning Content is to be shared on a Peer-To-Peer (P2P) network to promote eLearning goals, then quality assurance of the content will be an important consideration. Achieving quality assurance on such a P2P network is however not the same as it would be for the traditional sharing of content in a client/server eLearning system. In [1], it was noted that many researchers have adopted P2P file-sharing technologies to share Learning Content as files. In these cases they rely strictly on the traditional eLearning standards-based metadata schemas [2,3,4] to facilitate the discovery of the content for reuse. The reuse would generally translate to the content being imported into a server-based eLearning application, where the quality may then be reviewed before being selected for structuring into courses for student consumption.
The research found no attempts to preempt the quality assurance of the Learning Content while it is authored and actively shared over the P2P network. In previous research by [5], the Learning Content to be shared on a P2P network was redesigned for a better fit into a P2P network so as to promote greater opportunities for reuse. The redesigned content is referred to by [5] as a photonOne Reusable Learning Object (RLO). The research reviewed this design structure of the photonOne RLO, and found that if the Learning Object Review Instrument 1.5 (LORI 1.5) is applied, then some aspects related to evaluating the quality were redundant, and as such, a modified LORI was considered and presented as a new evaluation Review Rubric.
In the many traditional client/server eLearning systems, the quality of the learning content is determined through a peer-review process, sometimes by subject-matter experts. A similar process is developed in the research for sharing the photonOne RLOs, however, it is combined with the new Review Rubric and elements of P2P networking in an algorithm to arrive at a rating score for the RLOs' quality.
The following Section II "Background" gives a more detailed background for the research, followed by Section III "Peer-Review of photonOne RLOs in a P2P Network for Quality Assurance", which identifies the goal of the research. Section IV "Adapting LORI 1.5 into the Quality Assurance Process for photonOne RLOs" develops the new Review Rubric for evaluating photonOne RLOs along with an Algorithm for Calculating a Rating Score for a photonOne RLO, involving one or more subjectmatter expert. A discussion follows in Section V, then Section VI gives a summary and conclusion.

II. BACKGROUND
In first considering P2P architectures for managing eLearning content, [6] looked at the challenge faced by content authors and instructors when using client-server solutions. These ranged from managing different versions across different systems, offerings from one time period and learning situation to another, to a lack of comfortlevel when placing content on Servers outside the control of the content authors and instructors. It was felt that if content authors and instructors had a facility on their own machine for storing their content, it will be easier to maintain one authoritative version of the content, and by introducing application features similar to traditional P2P file sharing network, it will allow for the content to be Internet accessible as in client-server models but without added overheads of hardware and a complex network infrastructure. Overall, there should be an improvement in opportunities for content reuse. This proposal produced Lion-Share [7,8], a P2P application for sharing eLearning content. LionShare's requirement for the use of the authentication mechanism of Shibboleth [9] restricted adoption to only a few selected Tertiary level institutions that had or can access the required infrastructure but [10] later developed an open communication layer for improving on the P2P content repository security and scalability to have a wider adoption. This however is a complex design by their own admission. [11] also worked on a P2P learning object system POOL, POND and SPLASH, adopting use of the communication layer mentioned above. These successes and other attempts [12,13] at creating a P2P system for sharing eLearning content was the motivation for [5] to look at designing a similar P2P system, but this time PAPER QUALITY ASSURANCE FOR REUSABLE LEARNING OBJECTS ON A PEER-TO-PEER NETWORK to consider any Internet-based users, not just technologically mature Tertiary level institutions.
The P2P repository design of [5], built on a definition of Reusable Learning Objects (RLOs) that seems to be a better fit for P2P sharing than just the traditional eLearning content, standard-based or otherwise. Figure 1 shows a graphical representation of the RLO, which [5] called a photonOne RLO. The RLO to be used on P2P networks according to this definition must have a required single stated Learning Objective, reference to the content that relate to the Learning Objective, and, reference to the formative and summative assessment content that relate to the learning content. In a later prototype enhancement of the repository, the assessment content must also be formatted using the Question and Test Interoperability (QTI) standard [14]. This simple RLO, would make it easier to store, manage and share content and handle one of the major challenges of acquiring assessment content when the RLO is used in a course. This is something which none of the popular client-server systems can do without maintaining databases with questions and answers and providing search and discovery features to match content with possible assessment items. Because of these overheads, attempts to automate the selection and generation of assessment items for aggregated RLOs has received some research attention [15,16,17]. These approaches have been met with varied degrees of success.
[5] also introduced the use of email certificates for signing the RLO so that changes can be properly managed. The email certificates are also used to identify users, which acts as a method to discourage copyright violations, common to the traditional P2P file sharing networks and applications. This email certificate identity also allows for users to optionally form groups and restrict sharing to amongst their group members. The photonOne P2P Reusable Learning Object Repository (RLOR) prototype is based on the open source file sharing Shareaza project [18] and restricted to only using the more mature P2P communication protocol Gnutella2 [19].
The following work on Peer-Review of RLOs for Quality Assurance extends the photonOne P2P RLOR prototype to produce a more powerful and useful application for elevating a P2P eLearning system to a closer or better level than client-server eLearning systems. It allows for quality assurance to take place within the P2P network environment, without having to export the RLOs to clientserver eLearning systems.

III. PEER-REVIEW OF PHOTONONE RLOS IN A P2P NETWORK FOR QUALITY ASSURANCE
On P2P file-sharing networks, the files shared over the network are sometimes tagged with metadata containing a rating score for the file. Rating these files using the traditional P2P file-sharing applications are done by users. Generally the users can assign a value between 1 (Poor) and 5 (Excellent), and they may also have the facility to enter a comment. This rating score is however a popularity rating, for example, a user indicating if the file is a good song or video. It is not the type of rating score that would be suitable to indicate pedagogical quality of RLOs when the RLOs are to be shared on a P2P network. Further, the "rating" metadata element in file-sharing P2P networks is accessible by users for edits, since the data is stored on the client where it is easy to manipulate and compromise the integrity of previously stored review data This seems to suggest that any peer-review data for RLOs need to be either digitally signed or kept on specially managed Database (D/B) server nodes. The latter appears to be especially necessary, if the review data is to be kept fully updated at all times to all copies of a given RLO residing on different peers on the P2P network. Further, it will be more useful to have subject matter experts perform reviews on the RLOs and not just any random user on the P2P network, again suggesting some kind of account management for reviewer registration using specially managed server nodes.
In further reviewing the approach used by the wellknown server-based repository MERLOT [20], and considering the above challenges along with the reviews done in [1] related to work by [21,22,23,24] with respect to rating Learning Objects for quality and effectiveness, it was decided that an adapted LORI 1.5 approach would be a good fit for reviewing RLOs on the photonOne P2P Network. The design would incorporate the use of a scalable D/B server architecture for the peer-review data, with account management features for managing reviewers, along with a registration system for users to identify RLOs that are to receive a review.
The use of the special server nodes will not include storing eLearning content on the D/B or other server nodes. Instead, the eLearning content will remain entirely on the P2P network where the P2P application will be extended, so that a registered RLO assigned to a reviewer will be automatically downloaded to the reviewer's pho-tonOne P2P RLOR. The extension will facilitate the reviewer to enter all review related data using the photo-nOne P2P application, where it is then automatically and transparently uploaded via the reviewer's account, to the D/B on which the RLO is registered. The process allows for any network peer with a copy of a given registered RLO, to access and retrieve the peer-review data transparently from the server node on which it is registered, thereby addressing the previously mentioned issue of not having the most current review data across the entire P2P network.

IV. ADAPTING LORI 1.5 INTO THE QUALITY ASSURANCE PROCESS FOR PHOTONONE RLOS
In adapting LORI 1.5, it was determined that only four of the nine dimensions needed to be captured explicitly, as iJET -Volume 11, Issue 10, 2016 5 PAPER QUALITY ASSURANCE FOR REUSABLE LEARNING OBJECTS ON A PEER-TO-PEER NETWORK Figure 2. photonOne RLO Review Rubric some of the items were either, not applicable because they were already inherent in the photonOne RLO design, or, the dimension is covered sufficiently, or, it is not applicable at the RLO level, again because of the design. This is to be given consideration when aggregating and sequencing for delivery to the learner. The following gives the adaption to LORI 1.5 which is used in the design, where the nine dimensions summary descriptions were taken from [25]: 1. Content Quality (veracity, accuracy, balanced presentation of ideas, and appropriate level of detail) -the reviewer is presented with a form to fill-in, built from the rubric given in Figure 2 (photonOne RLO Review Rubric). 2. Learning Goal Alignment (alignment among learning goals, activities, assessments, and learner characteristics) -the RLO has a required learning objective by design, so the reviewer only needs to indicate if the stated objective is aligned with the reviewed content, using a given scale. 3. Feedback and Adaptation (adaptive content or feedback driven by differential learner input or learner modeling) -the goal is to have RLOs with only a single learning objective (as fine a granularity as possible). The Research proposes transferring this dimension to a higher level, where RLOs are aggregated and sequenced into a course. It is at the course level that a more meaningful feedback and adaptation for learning styles can be given, possibly leveraging information from a learner's profile which can be maintained for structured course offerings over a given period of time. 4. Motivation (ability to motivate, and stimulate the interest or curiosity of, an identified population of learners) -as part of the RLO registration process, the user is required to fill out a form which captures the attributes Category, sub-Category, Educational Level and pre-Requisites. The reviewer is asked if the RLO content matches against the user selections. If a "No" is given, the reviewer should enter a comment. These same attributes are also used to match against the reviewer's profile to select possible subject matter experts to be assigned for the review.

Presentation Design (design of visual and auditory
information for enhanced learning and efficient mental processing) -this is part of the reviewer's form to fill-in, as given in the rubric of Figure 2 (photonOne RLO Review Rubric). 6. Interaction Usability (ease of navigation, predictability of the user interface (UI), and the quality of UI help features) -this is part of the reviewer's form to fill-in, as given in the rubric of Figure 2 (photonOne RLO Review Rubric). 7. Accessibility (support for learners with disabilities)the belief is that this may be better determined at the course level and not at the RLO level. It is therefore omitted from the RLO review. 8. Reusability (ability to port between different courses or learning contexts without modification) -the RLO is designed to have the single learning objective, content, and assessment contents, so no additional in-Scale: 0 (None), 1 (Poor, needs work), 2 (Acceptable, but can be improved), 3 (Average), 4 (Above average), 5 (Excellent) PAPER QUALITY ASSURANCE FOR REUSABLE LEARNING OBJECTS ON A PEER-TO-PEER NETWORK formation has to be explicitly gathered from the reviewer. 9. Standards Compliance (adherence to international standards and specification) -the main content may or may not be standards-based. The reviewer indicates if it is in any standard-based format, such as IMS or SCORM. This is only informational and does not impact on the review score. In the updated design of the RLO it is now necessary for all assessments to be in QTI format. The reviewer checks this and indicates with a Yes/No. A "No" immediately causes the assessment score to be set to zero.
A. An Algorithm for Calculating a Rating Score In [26], it was noted that the Peer-Review process can be a very time consuming activity. As a result, the Research develops Algorithm "P2P peer-review algorithm for calculating a Rating score" to improve on the time commitment required in obtaining the rating score for a photonOne RLO. The photonOne RLO has a structure which already assures a significant level of quality. The Algorithm takes this into account, along with the adapted dimensions of the LORI 1.5 given above, to focus the Peer-Review process and reduce the time spent on a review.
Algorithm 1 -P2P peer-review algorithm for calculating a Rating score. All rating and review data is stored in the database where the RLO is registered and is retrieved transparently using the GUID by the photonOne P2P RLOR, where any copy of the registered RLO may subsequently exist. The user who initially registered and digitally signed the RLO can also resubmit an improved version for a new review. Any copy of the previous version will receive a notification that it has been updated, when the rating and review data is accessed from the P2P RLOR. The P2P RLOR will be notified that a new version exists on the P2P network, giving the new registered GUID of the RLO, and allowing for an automatic process for downloading the new version of the RLO if desired.

V. PEER-REVIEW DISCUSSION
It is clear that the peer-review approach to determine quality of RLOs in P2P networks cannot be the simple process used to rate files as in a traditional P2P file sharing application. It needs to be similar to strategies adopted in client-server eLearning systems, and in this case, the adaptation of LORI 1.5 was a reasonable fit, given the successful implementations found in client-server eLearning systems.
It was interesting to find that the photonOne RLOs did not need evaluation in LORI with respect to several of the dimensions because of the inherent pedagogical design. This assisted considerably to produce a more focused and reduced Review Rubric to consider the remaining applicable dimensions.
In the updated design of the photonOne RLO, adding QTI content for the Formative and Summative assessments within the RLOs further assisted with quality, since the QTI items must relate to the learning content and iJET -Volume 11, Issue 10, 2016 PAPER QUALITY ASSURANCE FOR REUSABLE LEARNING OBJECTS ON A PEER-TO-PEER NETWORK Learning Objective, it is also at the same time in a standards-based format.
While the design of the photonOne RLO gives good support for quality assurance, there is no guarantee that the RLOs will be authored to follow the guided format, hence the Review Rubric combined with subject-matter expert reviews were considered. The strategy of the user initiating a registration of photonOne RLOs for review, allows for the user to consider only when the RLO is matured. It may be possible to employ a peer-review process amongst users on the P2P network, similar to [27] for maturing the RLOs, prior to submission for peer-review by the subject-matter experts. The reviewer's task is kept to a minimum as much as possible by having users supply some information similar to what is required for filling-in metadata tags for standards-based content such as IMS [28], and then having the reviewers validate with a simple option selection to facilitate some aspects of the rating. This reduces the work considerably, so more focus can be placed on reviewing the learning content and assessment content itself against the learning objective for the quality determination.
The algorithm giving the overall peer-review process results in a final rating score for a photonOne RLO. It would be possible later on, to consider further validation on the accuracy of the quality in terms of the rating score that was calculated using quantitative studies when there is a greater amount of photonOne RLOs being shared on the photonOne P2P network.
The special server nodes used for managing the registration of RLOs, managing subject-matter experts and maintaining the data integrity of the rating score assigned per RLO, is built using technologies that allows for a scalable implementation, where any number of server nodes with different owners can participate in the process. This modularity and scalability is important, since it lends support for different groups of P2P users to have their own server nodes and management process, as well as for more public groups and individuals to use publicly available server nodes for their management of the reviews.
In [29], the photonOne P2P RLOR was extended to be able to aggregate and sequence RLOs into eBooks. Figure  3 (Mobile Device eBook: TOC, Page, URL, RLO and Learning Objective) shows six different screenshot elements of an eBook APP for a mobile device. It was automatically generated in one mouse-click within the photo-nOne RLOR after the RLOs were retrieved from peers, selected, then aggregated and sequenced into a course. It is a significant achievement to have quality assured RLOs authored, evaluated and used to build courses, all within the photonOne P2P network environment. There was no reliance on any client-server eLearning system, from the time of authoring the RLO, to the time of producing the eBook for student consumption. The eBook contains the assessment items and each RLO within the eBook retains the GUID link to the rating score, so both the course designer and user can view the respective ratings.
In the Figure 3, Screenshot 1 identifies the drop-down menu while on the start page. The menu items are TOC (Table of Contents), Cover and About. Screenshot 2 shows the TOC screen. This APP template generates a simple hyperlink file as the TOC. Screenshot 3 shows the Page for the "Course Details" document, and Screenshot 4 shows the Page with a URL linking to the Lecture's web page.
The Page objects are not RLOs. Screenshot 5 is a RLO with a five TAB layout for Content, Practice, Exam, Rating and Signature. There is a Menu dropdown for the RLO which gives Screenshot 6, the Learning Objective for the RLO. This eBook demonstrates and hints at several other aspects of the overall research undertaken, with this research on quality assurance being an important component.

VI. SUMMARY AND CONCLUSION
The inherent design of the photonOne RLOs to have the Learning Objective, learning content and assessment content, all related and contained as the structure of the RLO, along with the digital signature of the author, inherently introduced a level of reusability which reduced the effort required to implement LORI 1.5 for a peer-review process within a P2P network. The Review Rubric and associated algorithm given to guide the process, assigns the rating score as an indicator of the level of assured quality of the RLO.
Peer-review of RLOs in a P2P network is more challenging than a client-server environment because the integrity of the feedback from the reviews can be easily compromised if it resides on a peer. The decision was to maintain the data integrity using a specially managed node in the photonOne P2P network. This special node can be referred to as the photonOne LORI node. In the algorithm, the registration process for a RLO is designed to use the domain name of the node as part of its GUID, thereby achieving some scalability for photonOne LORI nodes. In the design, a user can, from the peer, initiate a registration of a RLO on the special node, and reviewers can, from their own peer complete the review without accessing the photonOne LORI node directly.
Apart from the registration process, all other interactions to have the feedback stored and accessed on the photonOne LORI node, is transparent to all peers, and subsequently transparent to the peers that will later access the rating information when users are viewing the feedback.
Using the photonOne P2P RLOR, it is possible to aggregate and sequence RLOs into eBooks within the P2P network environment. The result is, for the first time, a self-contained end-to-end, P2P eLearning system with no reliance of any sort on client-server eLearning systems. This research on quality assurance has successfully contributed in achieving the overall goal of P2P eLearning system independence.