A Preliminary Analysis of Software Engineering Metrics-based Criteria for the Evaluation of Learning Objects Reusability

—Reusability of learning objects is evaluated on the basis of a priori software reusability analysis, which are related to cohesion and coupling aspects. A number of reusability metrics extracted from metadata records are defined and analyzed to provide an aggregate reusability evaluation for learning objects in a repository. The evaluation is validated and compared with an expert-based a posteriori evaluation method


I. INTRODUCTION
Reusability is a key issue on e-learning contents and systems.Providing reusable learning objects can facilitate its further development and adaptation, augment learning object development productivity, reduce development costs and improve quality.Although reusability is an intrinsic characteristic of the learning object that can provide a priori a measure of its quality, reusing learning objects is an empiric and observable fact that can be compared with such measures by means of a posteriori data compiled from their actual use.Nevertheless, studies on reusability indicators and design criteria that guarantee reusability are scarce [1].
The objective of this work is to identify concrete metrics that can be used to qualify learning objects with aspects related to the capability of being reusable.Such measures can be useful to learning object producers, who can have quantitative data on the reusability of the designed objects, as well as to learning object consumers, who can search in repositories for objects that can be more easily adapted to their specific needs.

II. LEARNING OBJECT EVALUATION
Several initiatives have approached the evaluation of learning objects to provide an estimation of the guaranteed quality.MERLOT (http://www.merlot.org)classify objects in seven discipline categories (i.e.Arts, Economy, Education, Humanities, Mathematics, Science and Technology) and compile experts and users' evaluations on three dimensions (i.e.content quality, usability and effectiveness as a learning tool) on a 1-5 numeric scale [2].
ELera (http://www.elera.net)extends this evaluation scheme by the LORI (Learning Object Review Instrument) tool [3], which evaluates aspects such as content quality, objective fulfillment, feedback and adaptation capability, motivation, presentation, usability, accessibility, reusability and standards compliance.Each aspect is assessed on a 1-5 scale, based on a Delphi-style collaborative evaluation scheme with the participation of groups of experts, in which objects are first evaluated on an asynchronous, individual basis; afterwards individual evaluations are discussed to agree on eventual assessments.
Usual learning object evaluation methods are based upon compiling opinions from users and experts about different aspects of a learning object.In contrast to these, the learning object reusability evaluation model proposed here is aprioristic and is based upon the learning object structure and common metadata that describe it.However, harnessing learning object metadata for that aim depicts some issues, particularly related to information fragmentation and the potential lack of integrity on the harvested metadata.Therefore, we need to augment metadata with extended information that enacts reusability.

III. EVALUATION METHODOLOGY
We have based on an evaluation methodology used to measure reusability of object-oriented software [4], based on the following steps: 1) Study and identify the learning object aspects and factors having influence on the capability of reusing 2) Define metrics to measure reusability factors that have been identified, based upon analysis of IEEE Learning Object Metadata (LOM) standards [5] and the learning object structure 3) Formulate an aprioristic evaluation model formed by the aggregation of the metrics according to their significance for evaluating reusability 4) Evaluate the model though application and comparison with the reusability data obtained by LORI for a significant set of learning objects of the eLera repository.
IV. REUSABILITY FACTORS The factors that determine the ability of a learning object to be reused [6] From a contextual viewpoint, the more contextdependent and context-specific a learning object is, the more limited its reusability will be.We can deal with contextual factors in the following dimensions: technological, educational and social.
• The technological dimension of context includes platform dependencies and software needed to run the learning object, as well as representation issues (reusable learning objects should separate contents and format issues).• The social and educational contexts require the following features: learning objects must be generic, i.e. independent from a given subject or discipline; they have to be prepared for using on different education and assessment levels; they must be pedagogically neutral, i.e. do not involve a specific pedagogical method; they must lack institutional, legal, social and cultural dependencies; they are independent of time and location in which they are run.
We have to mention that some factors described above cannot be actually considered up to its extreme in order to achieve the greatest reusability; for instance, a generic, discipline-independent learning object is more reusable than a discipline-specific one, but clearly it is not useable, since it has to commit the learning objectives for which it is intended, and such objectives are always subjectspecific.A different thing is that, for instance, a learning object dealing with Statistics is more reusable if it does not involves examples that deal with a given discipline (e.g.mechanical engineering) that hinders to include it in another object (e.g. a biology course).Similar issues can be discussed about the pedagogical neutrality or timeindependence features, to say only some of them.
Designers tend to produce objects with multiple dependences to enrich the learning process, in contrast to independent and self-contained objects that contribute with not much significant knowledge.This situation is a challenge to design cohesive, uncoupled objects containing both structural and contextual aspects that do not jeopardize reusability [9].

V. LEARNING OBJECT REUSABILITY METRICS
We have analyzed common software metrics in order to provide reusability metrics for learning objects, based upon the reusability factors discussed above.Traditionally, software engineering based upon an old design principle to strive for strong cohesion and loose coupling [9].These two principles head for building maintainable software that easily adapt to new requirements.Since learning objects are designed for reuse, we analyzed how these principles apply to determine learning object reusability.Although reusability metrics are mainly related to cohesion and coupling, we have also analyzed metadata elements to evaluate other reusability factors, such as portability, size and complexity and difficulty of comprehension.Clearly, these are not completely independent factors, but they depict clear intersections up to some extent.We will describe further on how this issue can be managed.
In order to evaluate and compare our aprioristic model with a posteriori values, we have normalized metrics values in the [1,5] interval, which is the same scale of readily available evaluation models such as MERLOT and LORI.

A. Cohesion
Cohesion analyzes the kind of relationships among different modules.A module must realize a single task to be maximally cohesive [11].Greater cohesion implies greater reusability [12].Cohesion is a software quality indicator that, applied to learning objects, is fulfilled by the following elements: • A learning object involves a number of concepts (LOM 9 Classification category).The lesser number of concepts, the greater cohesion it will depict [13].• A learning object must have an only and clear learning objective [1].The more learning objectives it has, the lesser cohesive it will be.• The semantic density of a learning object (LOM 5.4 Educational category) indicates how concise it is.The more conciseness, the more cohesion for the learning object.• A learning object must be self-contained to be highly cohesive [13].LOM 7 Relation category is used to define as many instances as relationships the learning object has (notably is-version-of, hasversion, is-format-of, has-format, references, isreferenced-by, is-based-on, is-basis-for, requires, isrequired-by, is-part-of and has-part).The more relationship instances a learning object has, the less self-contained and, therefore, less cohesive.Moreover, LOM 1.8 Aggregation level element summarizes the level of aggregation of a learning object as ranging from 1 for single resources to 4 for a set of related courses.The lower level of aggregation, the more cohesion.
We can conclude that learning object cohesion is directly proportional to semantic density and inversely proportional to the number of relationships, aggregation level, number of concepts dealt with, and number of

B. Coupling
Coupling measures interdependencies among software modules and must be minimized [12].A module must communicate with the minimum number of modules and must exchange as minimal information as possible, in order to minimize the impact provoked from changes on other modules.Learning object coupling describes interrelationships among distinguishable objects, so the lesser coupling, the greater reusability [13].
LOM 9 Relation category indicates the number of objects related to a given learning object, so we conclude that coupling is directly proportional to the number of relationships present in that category.

C. Size and complexity
Software size and complexity can be measured through several methods, e.g.lines of code, McCabe's software complexity, Halstead's difficulty, etc.In general terms, granularity provides clear information on learning object reusability, since fine-grained objects are more easily reusable.Learning object granularity is directly proportional to the following LOM elements: • Size: the number of bytes of a learning object.These data should be weighted depending on the learning object format, since there are different interpretations of size for texts, images and videos, for instance.• Duration: the estimated time to run the learning object.This is specifically useful for videos or animations.• Typical Learning Time: the estimated time required to complete the learning object.This is a reliable source of information to estimate the size and complexity of a learning object Learning object size and complexity can be classified according to values of Table II.

D. Portability
Portability metrics measures the ability to transfer software from one system to another and is based on analyzing modularity and hardware/software context independence [14].Learning objects portability can be measured as the context dependence at technological and socio-educational levels.The few dependendencies found, the more portable the learning object.

1) Technical portability
The following LOM values can be analyzed when considering portability at a technical level: • Format: determines the learning object components' delivery format, such as video/mpeg, application/xtoolbook, text/html, etc.Some formats are more readily portable (e.g.text/html is more widespread than application/x-toolbook.• Requirements: involves the hardware and software required to run the object.As the number of requirements increase and these are more complex, less portable is the object.
Learning objects' technical portability can be qualified by means of the values shown in Table III [15].

Very high
The object is based on an technology available on practically all systems and platforms (e.g.html).

High
The object is based on an technology available on many systems and platforms (e.g.pdf).4

Medium
The object is based on an technology that is not available on many systems (i.e. common platform-specific file format).

Low
The object is based on an technology that is hardly available on different systems (i.e.uncommon proprietary file formats).

Very low
The object is based on a proprietary technology that is not available on many systems (i.e. a specific server technology) 1

2) Educational portability
When moving at the educational level, we can deal with vertical or horizontal portability [15].Vertical portability means the possibility for a learning object to be used and reused on different educational levels; in contrast, horizontal portability determines the inter-disciplinarity of the object.We have considered the following LOM values: • Context: potential educational contexts in which an object can be used (i.e.school, high school, higher education, professional training, etc.) Educational portability is greater for learning objects that can be used and reused on more different educational contexts.
A PRELIMINARY ANALYSIS OF SOFTWARE ENGINEERING METRICS-BASED CRITERIA FOR THE EVALUATION OF LEARNING OBJECTS REUSABILITY • Typical age range: potential age ranges in which an object can be used.Educational portability increases as the number of ranges grows.• Language: the human languages supported by the object.An object is more reusable if it is available on more languages.• Classification: information used to classify a learning object within the discipline it belongs or is related to.The more specific the classification scheme, the lesser reusable the learning object can be.

Very high
The object is generic, pedagogically neutral and can be used on different educational levels 5

High
The object can be used for several disciplines and educational levels 4

Medium
The object can be used without modifications on a specific area and educational level 3

Low
The object depicts educational dependencies and can be reused with several modifications on a different educational context and level 2

Very low
The object depicts many educational dependencies and can be hardly reused on different educational contexts and levels 1

E. Difficulty of comprehension
Software difficulty measures the cognitive effort to understand a software component.It is based on analyzing the component complexity, how self-descriptive and well documented it is [14].
We can state that the difficulty to comprehend a learning object directly influences the capability of a designer to reformulate and reuse it on another aggregated object.We can consider here the LOM 5.8 Difficulty category, although other LOM elements can be clearly correlated (e.g.LOM 5.4 Semantic density or LOM 5.9 Typical learning time).Even LOM 7 Relation category or LOM 1.8 Aggregation level elements can be heavily correlated to the difficulty.For this reason, and since these correlated elements have been considered for inclusion in other reusability factors above, we do not consider this factor separately.

VI. REUSABILITY EVALUATION MODEL
Learning object reusability depends on cohesion, coupling, portability and difficulty category elements.Several LOM values can be aggregated to build an a priori evaluation model.We discard the difficulty of comprehension factor due to the great number of dependencies it shows with elements from all other categories.Moreover, we assume that coherent metadata values are available for all considered LOM elements on analyzed objects.Let be the set of evaluation criteria as extracted from LOM records.To estimate the reusability of a learning object we require an aggregation process.
For that aim we used first an ordered weighted averaging operator: Weight values wi are provided by the evaluation as parameters that can be estimated and agreed to enhance or soften the contribution of a given factor to the aggregated reusability evaluation.For instance, Table V provides an estimated, primitive set of values that depend on the number of available evidences extracted from LOM.This must be calibrated and validated if needed.VII.REUSABILITY MODEL VALIDATION Validation of our model has been carried out by a detailed analysis of eight learning objects from the eLera repository.After that, we compared the aggregated reusability metric with LORI reusability evaluations as done by experts.The learning objects have been selected because they received the highest number of expert evaluations in the repository, so it guarantees the reliability of such evaluations.However, we found that a lot of metadata information was missing to compute our aprioristic reusability value, so we had to complete that information.
Table VI shows reusability values obtained and compared with LORI evaluations.They are graphically depicted on Figure 1.If we consider a 0.5 permissible difference, we have that the aprioristic reusability evaluation model fits 62.5% of cases with experts' opinions.If we consider a permissible difference of 1.0, the model fits 87.5% of cases with experts' evaluations.
There exists a significant 95%-confidence correlation between size and educational portability, and a 90%confidence correlation between cohesion and size and between cohesion and educational portability.Therefore, we can assume that there is a degree of interdependence between the selected metrics of cohesion, size and educational portability.VIII.CONCLUSIONS We can conclude that the aprioristic reusability estimations provided by the model approximate to those provided a posteriori by expert evaluation.Although some aspects of the model must be improved, it provides an approach to develop a formal, aprioristic reusability model.Therefore we can conclude that reusability metrics adapted from traditional software engineering reusability factors can provide a clear measurement of learning objects reusability.Including such computed reusability values as metadata records allows to enhance indexing and searching capabilities [16] as well as developing new reusable learning objects, so improving productivity and quality in learning object-based systems.
Aspects to be improved include the ordered weights estimation, and the treatment of interdependencies among analyzed reusability factors.The latter can be managed by utilizing more powerful aggregation operators, such as the Choquet integral, which takes into account existing interdependencies and reduces their influence on the aggregate evaluation value [17].The former can be managed through studying the correlation on a learning object-basis among LORI reusability value and concrete metadata on a significant amount of repository objects.However, metadata records must be filled-in and readily available for that aim.

Figure 1 .
Figure 1.Aprioristic reusability values compared with a posteriori LORI reusability values A PRELIMINARY ANALYSIS OF SOFTWARE ENGINEERING METRICS-BASED CRITERIA FOR THE EVALUATION OF LEARNING OBJECTS REUSABILITY learning objectives covered.These metadata elements can be source for a valid estimation of the reusability of a learning object.This way, we can classify learning objects cohesion values as depicted on TableI.

TABLE I .
COHESION VALUES TO MEASURE LEARNING OBJECT REUSABILITY

TABLE II .
VALUES TO MEASURE LEARNING OBJECT SIZE