ALIGNING CURRICULUM AND EVIDENCING LEARNING EFFECTIVENESS USING SEMANTIC MAPPING OF LEARNING ASSETS The Common

The ability to evidence learning effectiveness in virtual learning environments for accreditation purposes has posed as a challenge to higher education institutions who offer online courses. Many instructional designers have tried several methods for retrieving learning object, resulting in varying levels of object quality. For institutions who offer online courses and who are initially applying for or seeking renewal accreditation, rigorous alignment of curriculum to goals to learning objects is paramount. To date, there has been no evidence in the literature of this type of semantic mapping to harvest high quality learning objects that further align to course content to learning objects to course goals and objectives.A research and development team at a fully online higher education institution addressed this challenge by employing an open source repository and semantic engine for analysis and alignment of content, materials, and learning activities to goals and objectives across all courses within the School of Business. The result was a highly detailed, accurate mapping of the programs' knowledge base to established goals and objectives. Discussion, conclusions, and future directions are provided.


I.
INTRODUCTION As educational policy-makers continue to struggle with definitions of quality in terms of learning effectiveness in traditional face-to-face classrooms, a new set of challenges have emerged in online learning environments.Defining quality-in either environment, is heavily reflective of current economic and societal trends; therefore establishing definitions and associated indicators continues to be a highly fluctuant undertaking.
As many institutions decide to transform traditional instruction delivery to online learning delivery platforms, they have been faced with the task of re-defining, reassessing, re-evaluating, and re-evidencing quality in terms of: (1) online learning objects; and (2) accreditation.Ensuring the alignment of course content against desired goals and objectives has always been at the core of effective instructional course design-whether the instruction is delivered face-to-face, or via the support of distance technologies.Nonetheless, with the latter method of instructional delivery, the architecture of a Learning Management System (LMS) challenges online instructors and Information Technology (IT) departments to locate and/or create high quality digital learning objects that align to goals and objectives.From the organizational perspective, the overwhelming amount of information stored within internal and/or external systems presents a significant challenge when codification of tacit knowledge is required for accreditation purposes.While keyword and metadata strategies have provided some benefit, these methods suffer from a lack of robustness.As such, they provide only incremental improvements to what remains a highly manual process.

A. Accreditation and Quality
Developed over 100 years ago, accreditation has been a long-standing pillar in education.Narrowly defined, accreditation is a set of standards and competencies used to evaluate an institution's overall organizational effectiveness.Definitions of educational quality and measurement have historically evolved around changing socioeconomic forces, thus prompting continual changes and adaptations in institutional practices and policies.It is likely these continuous transformations have contributed to negative perspectives of accreditation as being a cumbersome and obligatory process [1].Viewed in a more positive light, accreditation has the transformative ability to raise institutions to exceeding levels of excellence in teaching and learning.
Accreditation of institutions of higher education are governed by 6  Recent innovative developments against a grim economic backdrop have led to extensive policy changes regarding accreditation criteria.These policy changes have had far-PAPER ALIGNING CURRICULUM AND EVIDENCING LEARNING EFFECTIVENESS USING SEMANTIC MAPPING OF LEARNING ASSETS reaching effects for higher education institutions and have also extended to organizations accrediting programs within an institution.For online programs, these policy changes have additional implications in terms of aligning content to goals and objectives and for evidencing learning effectiveness.

B. Course Content Alignment to Goals and Objectives
The rapid growth of a technologically-driven society has triggered limitless opportunities in the areas of teaching and learning-particularly in distance education.Online learning has increasingly become the preferred method of instructional delivery among learners worldwide [2].Due to a myriad of components and tools that allow for effective teaching and learning, learning management systems (LMS) have primarily been the platform of choice for online learning.Nevertheless, this disruptive technology [3] has also prompted institutions who employ learning management systems to determine best practices and policies aligning with this delivery platform [4].Moreover, the unique architecture of a learning management system, has presented new challenges in terms of defining and evidencing instructional quality that starkly contrast traditional definitions and measures.
At the program level, curriculum alignment has always been a critical component for effective face-to-face learning for centuries [5] [6].Alignment of digital learning assets to goals and objectives, however, has necessitated the exploration and evaluation of online learning content [7][8] [9] and in what ways this digital content can be aligned to goals and objectives.This task has been largely the responsibility of the instructor; however, considered a disruptive technology [10], online learning has rapidly proliferated into all areas and levels of education, therefore allowing new possibilities for collaboration.
Learning objects and learning management systems: Online course creators have commonly labored over locating content pertinent to the course(s) they teach.Some may also opt to use packaged educational materials tailored to their specific discipline.Content, or digital objects used to deliver instruction in an LMS are commonly referred to as a learning object.The Institute of Electrical and Electronic Engineers broadly defines a learning object as "any entity, digital, or non-digital, which can be used, reused, or referenced during technology supported learning" [11].Albeit there are many high quality learning objects available, learning objects designed within an institution's LMS are not reusable (as originally meant to be) as they are either stored privately and/or they are not visible to external systems [12].The reusability issues, which are common to many IT departments have primarily been due to: (a) misaligned metadata standards [13]; (b) learning objects that are only reusable within the systems for which they were built and reside; and (c) the lack of semantic metadata.These issues require semantic analysis to efficiently and effectively manage and align digital content, however, this approach has been largely unexplored.
Semantic Mapping Analysis:In the Learning or Content Management System environments, content management frequently translates into a single-purpose allocation of content resources, with cataloging and meta-tagging being a haphazard affair.The amount of stored, yet accessible information is so substantial, that IT departments consistently seek efficient and effective ways to manage and make use of this abundance of information [14].This focus has led to the need to efficiently and effectively design and update online courses.As with any pedagogically-sound instruction, a core requirement in course design must be the alignment of goals and objectives to formative and summative learning activities [15].By automating the meta-tagging and gap analysis process, semantic analysis allows one to not only smartly survey existing learning objects in a specific curriculum area, but also to examine more learning objects across unrealized curriculums.The ability to determine content interrelationships through the mapping of assets across the content universe enables one to effectively and efficiently facilitate object reusability towards curricular goal and objective fulfillment.This process allows for the actualizing of opportunities to locate learning objects to fulfill course level objectives for alignment across course level objectives, programmatic outcomes and industry standards.Improvement of instructional outcomes, through the ingestion of work products from implementation of content distillation and semantic analysis, ultimately increases return on investment and time on task.
Similar to contemporary object oriented programming language, semantic analysis is reliant on defining data in terms of classes with attributes and instances.The vision of the semantic aware applications builds upon this concept by refining these ontologies through comparisons of associated metadata.Currently, there are two approaches for developing semantic applications; the bottom-up approach and the top-down approach.The bottom-up approach is problematic in that it assumes metadata will be added to each piece of content to include information about its context; tagging at the concept level, if you will.The top-down approach appears to have a far greater likelihood of success, as it focuses on developing natural language search capability that can make those same kinds of determinations without any special metadata [16].

C. The Current Study
American Public University System (APUS): American Public University System (APUS) is a regionally and nationally accredited, private, fully online university offering an extensive variety of fully online academic programs that do not require students to physically attend classes.Demonstrating a commitment to a high quality education for its students, APUS strives "to provide quality higher education with emphasis on educating the nation's military and public service communities by offering respected, relevant, accessible and affordable, student-focused online programs, which prepare them for service and leadership in a diverse, global society" [17].reputation of failing to proactively devise innovative strategies that quickly respond to future economic and societal trends [18].The APUS School of Business continues to demonstrate a commitment to innovative academic advancements as reflected in its mission "to prepare students to be principled leaders in the global business community through a flexible learning environment that leverages technology and best practices focused on the practical application of knowledge" [19].
The APUS School of Business holds accreditation by both the Accreditation Council for Business Schools & Programs (ACBSP) and the North Central Association Higher Learning Commission (NCAHLC) in their Business Administration and Marketing academic programs.For students enrolled in these programs, accreditation provides two overarching advantages, it: (a) creates the impetus for relevancy and currency of faculty, programs and courses to best serve students; and (b) enhances the ability to serve students by assuring a focus on quality performance.
To ensure high levels of academic quality, the APUS Instructional Design department commissioned an outside entity to provide an unbiased and rigorous assessment of accreditation criteria coverage for both Business Administration and Marketing programs.In past years, providing evidence of student learning derived from online learning systems has typically been a difficult and time-consuming process-largely due to the inconsistencies in online course design and misalignment of course goals and objectives.Therefore, the results of this study could have substantial implications for managing, streamlining, and refining the process of providing accrediting entities with the required information to grant initial or renewal accreditation.
APUS ID Process Model:At American Public University System (APUS) the Instructional Design and Development (IDD) Team created an Instructional Design Process Model to design and develop curricula.The APUS ID Process Model is a continuous course development process allowing for the collaboration with Subject Matter Experts (SMEs), Instructional Designers (IDs), Graphic and Media specialists, and Content Area experts both internal and external to the institution.A derivative of the ADDIE model [20], the APUS Instructional Design Process Model also integrates the foundational principles of the Community of Inquiry (CoI) [21] and was applied to every step of the process; planning, pre-development, development, design, and evaluation, and maintenance as shown below in Figure 1.This balanced, six-phase process model is an architecture which enables the team to focus on three important elements of the courseware development.First, the team implemented accepted best practices in online Instructional Design (ID) systems pedagogy, andragogy, and heutagogy modeling to structure overall courseware development.Next, the CoI was applied to the online learning framework in the courseware to ensure successful student outcomes.Finally, the team utilized agile project management principles to allow for collaboration and communication, both internally and with subject matter experts, while still maintaining discipline, quality, and rigor throughout the project's lifecycle [22].
Barriers and Solutions:Though highly efficient and effective in rapidly creating quality content across a network of over 130 geographically dispersed contributors, the APUS IDD team is still confronted with three problems that consistently confound the field.First, content and learning activities created with the premise that SMEs have a mastery of their area and will ensure that goals and objectives are met through tacit embedding of area knowledge within course structures.However, subject matter experts are human and prone to the same tendency to have expert blind spots; a phenomenon long noted in the traditional classroom [23].Second, metatagging data is a time intensive process that even when executed with a high degree of accuracy provides little more than key word associations.Finally, because all contributors have knowledge of the underlying taxonomies or common vernacular that the information is based upon, it is difficult for organizations to survey their content universe for existing objects that can be incorporated into emerging workflows.
As previously mentioned in terms of accreditation, APUS is also confronted with the issue of providing adequate data for accreditation by external entities.Within the accreditation process is the task of demonstrating that curricula satisfies both course level and program level goals and objectives.While the IDD process, described above, is designed to help expedite goal and objective alignment, the limitations previously described still limit a robust execution.
To solve this problem, APUS investigated the feasibility of using semantic analysis to: (a) match program and course level goals and objectives to course content and activities; (b) create a gap analysis to note where additional resources should be applied to meet goals and objectives; and (c) create robust content repositories that have granular associations between course components and over-arching ontologies that can be applied in a cross-curricular fashion.
Interestingly, the 2009 Horizon Report [24] indicates that semantically aware applications are not likely to become mainstream for four to five years, however, a few cutting edge prototypes are currently being utilized.Even as these applications are still undergoing refinement, the prototypes demonstrate the potential power of semantic applications for both formal and informal learning.The IDD team at APUS vetted several of these cutting edge solutions, both open source and proprietary for the purposes previously described.The Common Library solution, an open source repository available under an Apache 2 license on Source Forge, was ultimately selected.

PAPER ALIGNING CURRICULUM AND EVIDENCING LEARNING EFFECTIVENESS USING SEMANTIC MAPPING OF LEARNING ASSETS
The Common Library: Developed from the ground-up to address specific needs in education, Common Library (http://commonlibrary.org) is the first standards-based content management system to enable true collaborative potential through the integration of content development and social networking.The Common Library latent semantic search engine defines a unique and powerful aspect of the application.In the current 2.0 implementation of the system, the metadata and content of each learning object are compared against defined standards systems.As shown below in the Learning Object Lifecycle, Figure 2, this higher-order logic enables Common Library to dynamically suggest interconnections between content items and applicable state standards, providing immediate value for users in the K-12 educational market.This functionality also defines the potential for constructing dynamic relationships between state standard systems that evolve over time.
Implementation of search and aggregate technology generates references that feed new granularly addressable connections between content and curriculum structures as more is learned about individual user requirements.III.PROCEDURE An instance of this solution was stood up for the APUS IDD team and all course components for the APUS Business Program were federated.As semantic analysis is an emerging technology it is still considered somewhat suspect in the realm of higher education.To assess efficacy, random sampling of Common Library output-both matching and gap analysis, was compared against human to Common Library agreement, as well as human to human agreement.An interrater reliability analysis using the Kappa statistic [25] was performed to determine consistency among raters.The first test (human to Common Library interrater reliability was established at Kappa = .90(p< 0.001).For the second instance (human to human agreement), the interrater reliability was established at Kappa = .93(p< 0.001).Thus, following value of Kappa ranges: (a) 0.40 to 0.59 (moderate); (b) 0.60 to 0.79 (substantial); and (c) 0.80 (outstanding), the reliability of the Common Library analysis would be considered outstanding when compared to human analysis.
Granularity Model: After disaggregation of the materials, a granular analysis was conducted using Common Library's underlying latent Dirichlet analysis engine [26][27][28].The Granularity Model, Figure 3, demonstrates the full mapping of program goals and objectives created across the content universe.Where incidents of deficiency were noted, remedial action was taken to provide additional resources.

IV. RESULTS
Content and activities from 29 courses in the APUS Business Program were federated in Common Library.Disaggregation of content yielded 5227 granular level assets and ontological ordering, using Dirichlet analysis, was conducted and categorical structuring was implemented using an iterative, multi-pass approach.A total of 538 goals and objectives, from both the program and course level, were input into the system.Using a natural language approach, these goals and objectives were specified as being representative of over-arching ontological Of the 538 goals and objectives, matches were identified among 465 respective assets.The asset match ranged in scope from between two and 38 assets per goal.With respect to identification of gaps, 73 goals and objectives were identified for which there were no corresponding assets.These areas were noted and recommendations were made to both programs.Subsequent content development, which was added to the existing repository, resulted in a second run in which all but three goals and objectives could clearly be mapped to corresponding course content and activities.
V. DISCUSSION Application of this technique allows for large scale analysis of correlations between goals / objectives and associated course assets.In turn, the ability to identify areas of deficiency and construct learning pathways ensures that all desired goals / objectives are addressed in a timely and thorough fashion.
While not yet implemented, the ability to aggregate student work products, as well as discussion threads, also exists.Using this approach the power of semantic analysis can be extended to include relating learning outcomes back to goals / objectives, providing a complete map of the learning cycle.

PAPER ALIGNING CURRICULUM AND EVIDENCING LEARNING EFFECTIVENESS USING SEMANTIC MAPPING OF LEARNING ASSETS
The ability to rapidly map assets to goals and objectives is significantly more effective when using a semantic engine than when similar processes are implemented by human coders.For this implementation, 137 hours were required from the IDD Team, project management, and technical personnel.A corresponding test case revealed that mapping one course against goals and objectives required 64 hours.When expanded to the 29 course sequence that was reviewed the mapping process would have translated into 1856 hours.Thus, the application of semantic analysis resulted in a 92.7% reduction in human labor.Translated into monetary terms the savings, including fringe benefits would be approximately $80,000 (U.S.).
Notably, standing up the instance of Common Library required 32 hours which will not be required in future mapping initiatives.Given the demands of mapping for both internal and external purposes, and including multiple programs, it is easy to visualize how this technique could result in savings of over $200,000 per year while drastically streamlining institutional course review and development processes.The ROI on this later aspect should be the subject of more comprehensive program evaluations for instance of implementing this technique.
Evidence illustrates that the ability to round-trip content and goals / objectives is a key benefit of this process.In other words, a content universe can be checked for assets that are linked to goals and objectives or individual assets may be viewed and their correlation to goals and objectives are then identified.

VI. CONCLUSIONS AND FUTURE DIRECTIONS
From an ID perspective, semantic analysis of content allows for confirmation that all goals and objectives have been fulfilled, identification of existing gaps, and the need for generation of other materials.In addition, instructional designers can rapidly identify existing assets that can be repurposed for use in new courses or programs; thus delivering on the concept of highly reusable learning objects.This latter capability is especially useful in instances where unrealized cross-curricular content may have already been developed but not realized due to programmatic silo effects and the related lack of awareness on the part of subject matter experts.
At the institutional level, this technique provides administrators the ability to rapidly assess existing materials and effectively plan for future staffing and development needs.When approaching accreditation, either regional or program specific, institutions become empowered to designate only those resources necessary to insure success, as opposed to current models in which countless hours are spent by faculty and staff checking and rechecking documentation to avoid potential lack of compliance.Presently APUS has engaged full time staff to extend upon this study by systematically applying semantic analysis to over 1500 courses, with the intent of facilitating continuous quality improvement across the institution.
With respect to the accreditation process itself, governing bodies can leverage semantic analysis to make processes far more transparent and efficient.On demand, any goal or objective can be produced and corresponding content reviewed, alleviating the need for countless hours of spot checking manually compiled materials.However, it should be noted that this process also brings with it virtually unlimited transparency into any organization, thus increasing the level of scrutiny evaluators may apply.
Moving forward, semantic analysis has to potential to allow for round-tripping of student data.In other words, student work products could be ingested into semantic engines in the same fashion as content.Goal matching could then be applied to each students work and correlations that demonstrate fulfillment of objectives identified.When this process is achieved, the ability to assess learning outcomes will be automated in the same fashion as content alignment, thus shedding complete transparency upon the academy.
regional accrediting bodies (Middle States Commission on Higher Education (MSCHE); New England Association of Schools and Colleges Commission on Institutions of Higher Education (NEASC-CIHE) North Central Association of Colleges and Schools Higher Learning Commission (NCA-HLC); Southern Association of Colleges and Schools (SACS); Western Association of Schools and Colleges Accrediting Commission for Community and Junior Colleges (WASC-ACCJC); and Western Association of Schools and Colleges Accrediting Commission for Senior Colleges and Universities (WASC-ACSCU) who grant accreditation against overarching criteria including: (1) Mission and Integrity; (2) Preparing for the Future; (3) Student Learning and Effective Teaching; (4) Acquisition, Discovery, and Application of Knowledge; and (5) Engagement and Service.
The institution's core values support this mission in the following areas: (a) learningquality; (b) integrity; (c) diversity; (d) freedom of inquiry and expression; (e) accountability;(f) access to underserved; (g) adaptive and responsive; (h) innovation; and (i) collaboration.The institutional mission, vision, and core values support and guide all APUS academic programs, thus demonstrating a continual commitment to quality and excellence in student learning.Quality Assurance for Schools and Programs in Higher Education: The field of education has maintained a lasting iJET -Volume 7, Issue 2, June 2012 PAPER ALIGNING CURRICULUM AND EVIDENCING LEARNING EFFECTIVENESS USING SEMANTIC MAPPING OF LEARNING ASSETS

Figure 2 .
Figure 2. The Learning Object Lifecycle