An adaptation model for personalized e-assessments

Over the last decade, a large number of eassessment system and tools were developed. However, they all provide assessments that are characterized by a static structure (ordered sequence of questions, predefined start and ending, etc.). This approach does not consider students individual knowledge or characteristics. Instead, each student has to answer the same questions. The work presented in this paper focuses on a personalized e-assessment system (askMe!) that not only identifies, but also supports and even compensates deficits in students’ individual learning by considering students’ strengths and preferences. In this paper, the core, the adaptation model of the system is presented that defines what can be adapted, as well as when and how it is to be adapted.


INTRODUCTION
Assessment tests in traditional e-assessment systems and tools are characterized by a static structure, which present an ordered sequence of questions and have a predefined start and ending.They are self-paced by the students and provide immediate feedback, both positive and negative.This linear procedure corresponds to Skinner's approach to education called programmed instruction [1].However, this approach does not consider students individual knowledge or characteristics.Instead, each student has to answer the same questions.But, e-assessment system and tools in the 21 st century are not only responsible for identifying, but also to support and even compensate deficits in students' individual learning by considering students' strengths and preferences.This requires that students' responses should determine what question is presented next.This corresponds to Crowder's approach of learning [2].He states that instructions should be branched so that some students can be presented with additional information if they do not respond well enough and that more advanced students can be exposed to more challenging content.
The institute of the main author has recognized this need for new e-assessment technologies and has developed a personalized e-assessment system called askMe!.The system corresponds to Crowder's approach of learning and dynamically selects appropriate questions and feedback at run-time [3,4,5].It follows the AHAM reference model [6] meaning that it consists of a user model, a domain model and an adaptation model (AM), which closely work together.The focus of this paper is to present the design of the AM.It can be regarded as the core of each adaptive system, because it defines what can be adapted, as well as when and how it is to be adapted.The design of the model and their methods is presented using Specht's [7] four dimensions of adaptive methods namely adaptation means, adaptation information, adaptation goals and adaptation process.
The remainder of the paper is organized as follows: The second chapter gives an insight into related work in the field of adaptive assessment.The third chapter proposes the AM and chapter four presents their implementation in the askMe!system.Finally, concluding remarks and references complete the paper.

II. RELATED WORK
In general, adaptive assessment is defined as a form of assessment, where the sequence of questions depends on the correctness of the students' responses to the previously answered questions.A system or tool that provides adaptive assessments is referred to as Adaptive Assessment System (AAS).There are two types of adaptive techniques that are applied in AASs namely adaptive testing and adaptive questions.

A. Adaptive Testing
The technique of adaptive testing is a process of test administration in which the selection of test items is dynamically adapted to the student's responses to previously administered items [8].It is an iterative algorithm and has the following steps: 1.An item pool is searched for an optimal item that matches the current estimation of the student's knowledge.2. The test item is asked and the student responds.

3.
According to the answer, the estimation of the student's knowledge is updated.4. Steps 1-3 are repeated until a termination criterion is met.
The technique of adaptive testing has made an enormous step forward through the development of the Item Response Theory (IRT) in the middle of the last century.The IRT [9,10] is based on the idea that the probability of a correct response to a test item is a mathematical function of student and item parameters.Several approaches exploit the technique of adaptive testing and IRT such as SIETTE [11] and PASS [12].

B. Adaptive Questions
In contrast, the technique of adaptive questions defines a dynamic sequence of questions depending on students' SPECIAL FOCUS PAPER AN ADAPTATION MODEL FOR PERSONALIZED E-ASSESSMENTS responses [13].That means, the answers provided by students to certain questions are used to determine the next series of questions and to skip unrelated questions.The basis on which these decisions are made are predefined rules.These rules are linked, for example, to the response of the students or to a user model, which represents student knowledge of different concepts or topics.Based on these rules and the last response of the student, appropriate questions are dynamically selected at run-time.The basic principles of adaptive questions go back to computerassisted surveys.Pitkow and Recker [14] showed that web-based adaptive questionnaires can reduce the number and complexity of questions presented to each student.Several approaches exploit the technique of adaptive questions such as CosyQTI [15] and iAdaptTest [16].

C. Comparison
Adaptive testing is an iterative procedure that dynamically selects test items to ask next based on the current estimation of student's knowledge level.The overall goal of this technique is to obtain accurate student knowledge estimations and to minimize the number of test items required for that purpose.That means, testing is continued only as long as necessary for each student.An advantage of adaptive testing is that test items, which are too difficult or too easy, are removed.Thus, the technique ensures that students only see test items, which are very close to his or her level of knowledge.However, this technique is appropriate for measuring students' knowledge levels, but not for assessing student's needs.That means, a teacher is able to determine students' level of knowledge related to a subject, but is not able to draw conclusions about students' strengths, weaknesses or misconceptions.
In contrast, the adaptive questions technique defines a dynamic sequence of questions depending on students' responses.That means that specific responses or students' performance to certain questions trigger the next series of questions according to several predefined rules.The advantage of adaptive questions is that it offers more flexibility to both teachers and students.Teachers are able to express their didactic philosophy and methods through the creation of appropriate rules.This enables creating tests, which are tailored to the characteristics of the individual students or a group of students with specific characteristics.Students can take advantage of this technique, because their prior knowledge, interests, context, etc. is taken into account resulting in shorter tests with more focused items.According to Lazarinis et al. [17], this is especially useful in formative assessments.

III. PROPOSED SOLUTION
In meeting the requirements stated in the introduction, the AM should be able to adapt the sequence, selection and presentation of questions to the individual student.But, feedback also plays an important role in the assessment process.It provides information about student's current areas of strength and weakness and motivates them to increase their effort and attention.Feedback can be regarded as the so called speaking tube of the question and test evaluation and thus able to communicate the result of the assessment to the students as well as other information, which may contain reasons for incorrect answers, hints or advices for continuing the assessment.Due to the fact that there is no one-size-fits-all approach of providing appropriate feedback, feedback needs also to be adapted to the individual student and/or their responses.As shown in earlier research [4], feedback personalization is still neglected or even not addressed in state-of-the-art AASs.For that reason, the AM should also be able to adapt the selection and presentation of feedback to the individual student.This allows providing guidance and support when required and in turn maximizes the motivation of the students [18].
In order to analyze and compare the variety of adaptive methods provided by the different systems and approaches, Specht has compiled a simple taxonomy scheme to meet this challenge.He structures adaptive methods of educational hypermedia applications in four dimensions: • What does the system adapt?(adaptation means) • To what does the system adapt?(adaptation information) • Why does the system adapt?(adaptation goals) • How does the system adapt?(adaptation process) In the following, this scheme is used to present the model and their methods.

A. Adaptation Means
This dimension focuses on the elements of an adaptive system that are adapted by an adaptive method.The elements, which are adapted by the adaptive methods in this AM are: 1. Question Sequence 2. Question Selection 3. Feedback Selection 4. Question Presentation 5. Feedback Presentation 6. Question Difficulty The first adaptation mean encompasses the order of the questions presented to the students.The idea of branching the sequence of questions in the design of the AM was motivated by Crowder's approach of learning.For adapting the sequence of questions, the AM allows to: • Retry questions Using these control structures, a variety of question sequences can be defined (cf. Figure 1).Ranging from simple sequences without any control structures over to complex sequences using branches, returns and loops.
The second adaptation mean describes the composition of the tests.The AM allows dynamically generating a test consisting of questions, which meet specific requirements.Such requirements could be (among other): • The question is referred to a specific domain or concepts of the DM (e.g., biology) • The question is referred to a specific difficulty taxonomy or level (e.g., synthesis level in Bloom's taxonomy) • The question has a specific interaction type (e.g., multiple-choice) • The question has not yet presented to the respective student The third adaptation mean is similar to the previous one in selecting specific feedback as response to a given answer.In order to allow selecting different feedback, the feedback created needs to be classified according to a uniform scheme.In this system, the classification developed in [4] is used.This classification allows selecting feedback that is most suitable for the moment and the respective student.Example conditions for selecting feedback could be (among other): • The feedback is enriched with elaborative information (e.g., response contingent feedback) • The feedback is enriched with graphics or animations • The feedback is limited to textual information • The feedback is given immediately after the student has given the answer The fourth adaptation mean is related to the presentation of the questions.This is realized by choosing a specific interaction type if one and the same question is created using different interaction types and by hiding or showing information of the question text and their associated media.
The fifth adaptation mean is similar to the previous one in adapting the presentation of feedback.Having classified the feedback using the scheme mentioned above, the presentation dimension is used to identify feedback that is best suited for the respective student.
The sixth adaptation mean is related to the difficulty of the questions.The difficulty of the questions is varied by the amount of additional information (hints), which is given during the assessment and the time available to answer the question.

B. Adaptation Information
This dimension focuses on the individual characteristics of a student as well as all situation-related characteristics that are used to provide adequate adaptations.The infor-mation that can be used to provide adaptations in this AM can roughly be divided into: 1. Student performance 2. Student features Student performance encompasses the information, which are obtained during a student takes a test.It can be accessed at any point of the test.This includes the amount of question correctly and incorrectly answered, the scores achieved as well as the time needed to answer the question(s).On the contrary, student features encompass all information, which are stored in the UM.This includes students' knowledge, which is one of the most important user features a personalized e-assessment system must consider as well as interests, goals, background, etc. Depending on the student feature, qualitative (e.g., good, medium, bad) or numeric values (e.g., from 0 to 10) are presented for selection.As the UM can be extended as required, the adaptation information are also not limited to this set of information.

C. Adaptation Goals
This dimension focuses on the pedagogical reasons behind the adaptations.In general, adaptations in educational hypermedia settings mainly aim at compensating knowledge deficits, ergonomic reasons or adapt to learning styles for an easier introduction into a topic [20].
However, the main reasons behind the adaptations in this AM are: 1. Identification of students' strength and weaknesses more valid and reliable 2. Consideration of students' strengths and preferences to compensate weaknesses and deficits 3. Encouragement of students' motivation by considering individual aspects (e.g., context or preferences) These goals correspond to the three heuristic models (assistance, compensation and preference) provided by Salomon [21].

D. Adaptation Process
This dimension focuses on the actual adaptation process and the preceding acquisition of adaptation information.In this AM, the adaptation information are students' performance in the test as well as the current values of students' features.Consequently, the former information are directly obtained from the adaptation engine during the assessment process.In contrast, the latter information are obtained from the user modeling component [22].When creating the adaptive test, these information are referenced inside the UM.At run-time, the adaptation engine uses the user profiles to fill the references with concrete values.In order to update user profiles, the adaptation engine provides the results of the tests to the user modeling component.Then, the user modeling component decides whether and how to update the respective values.The initiative to perform an adaptation can only be taken at specific points during the testing procedure.These so-called trigger points are: • Before presenting the question • During the interaction with the question • After answering the question iJET -Volume 8, Special Issue 2: "EDUCON2013", August 2013

SPECIAL FOCUS PAPER AN ADAPTATION MODEL FOR PERSONALIZED E-ASSESSMENTS
For proposing and aiding decisions about alternatives, the AM uses the rule-based reasoning approach.The main reason for selecting this approach was the ability of the author to directly and easily influence the adaptation process by creating rules.Rules are a very natural knowledge representation method, with a high level of comprehensibility, since they look like natural language expressions (e.g., in such a situation, do this and this).The decision of choosing this approach in favor for representing knowledge instead of case-based reasoning was made, because the author is able to directly influence and optimize students' testing procedure.In contrast, the casebased reasoning approach independently selects a next question based on a reference case.Although the author has defined or at least accepted these cases, it is not guaranteed that the system adapts in the interests of the author.In general, a rule consists of two parts: • IF part.This part is called antecedent, premise or condition • THEN part: This part is called consequent, conclusion or action.

IF <antecedent> THEN <consequent>
The IF part relates given information or facts to some action in the THEN part.A rule can have multiple antecedents joined by conjunctions (AND), disjunctions (OR) or a combination of both.The antecedent of a rule in turn consists of two parts, an object and its value.They are linked by an operator.This AM uses not only symbolic (e.g., is or is not), but also mathematical operators (e.g., !, = or ") to compare an object with a value.In contrast, a consequent uses an operator to assign a value to an object.For example: IF "answer_1" is wrong THEN "action" is "retry" IF "score_1"!5 THEN "action" is "move_to_question(2)" According to Durkin [23], rules can represent different types of semantics namely relations, recommendations, directives, strategies and heuristics.In this AM, rules are mainly used to specify strategies.This means that when the antecedent of a rule is satisfied (matched), a set of actions are triggered (fired) in order to achieve a specific goal (cf. Figure 2).Rule-based reasoning systems also distinguish between the ways in which rules are executed (i.e., forward and backward chaining).In this AM, forward chaining is used as inference method, because the adaptive system firstly gathers (adaptation) information and after that it tries to infer new facts from it.In contrast, backward chaining begins with a hypothetical solution and then attempts to find facts to prove it.Due to the fact that the data determines which rules are selected, this method is also called as data-driven.
In order to avoid conflicts between rules when more than one rule can fired at the same time, but have inconsistent consequents, a conflict resolution is needed.In this AM, the rule with the highest priority is fired.The priority is established by placing the rules in an appropriate order in the knowledge base.

Figure 3. Structural overview of the rule-based reasoning system
Consequently, the adaptation process basically consists of the following steps: 1. Matching: Find all rules whose antecedents are true and mark them as being applicable.2. Conflict Resolution: If more than one rule can fire, select the rule with the highest priority.3. Action: Execute the consequent of the lowest numbered applicable rule.If none applies then stop.4. Reset: Reset the applicability of all rules and return to step (1).
Finally, the complete structure of the rule-based reasoning system is shown in Figure 3.The structure clearly shows the basic structure of a rule-based expert system consisting of a knowledge base, database (facts), inference engine and explanation facilities [24].The knowledge base represents the knowledge about adaptations to be made.These information are specified as a set of rules.In this AM, the rules are associated with questions and compiled to adaptive tests.The compilation of the tests as well as management of the rules (i.e., creating, modifying, etc.) is realized by the adaptation modeling component.Then, the tests are handed over to the adaptation engine, which represents the inference engine and carries out the reasoning.The facts needed for matching the rules are provided both by the engine itself (student performance) and the user modeling component (student features).A crucial element of each expert system is the explanation facility, which provides information how a particular conclusion is reached and why a specific fast is needed.As depicted in the figure, this element is located at the user modeling component.In addition to get general statistics about the tests taken (e.g., reached scores, correct and incorrect answers, domains or concepts covered, etc.), it gives students an opportunity to get informed about which and more importantly, why adaptation decisions were carried out.

SPECIAL FOCUS PAPER AN ADAPTATION MODEL FOR PERSONALIZED E-ASSESSMENTS
In relation to the two types of adaptive techniques that are applied in AASs namely adaptive testing and adaptive questions, this AM corresponds to the latter one.The adaptive question technique is based on rules, which allow dynamically selecting appropriate questions at runtime.For that reason, this AM lays the foundations for adaptive e-assessment system following the adaptive question technique.

E. Modeling of Adaptivity
Even though the AM provides a powerful instrument for tailoring questions and tests to students' knowledge, preferences, etc., the use of it to create complex adaptive tests could be challenging for some authors.In order to prevent this, a novel adaptation modeling approach was developed.It supports the authors in modeling the dynamic behavior of adaptive tests and helps them to cope with the inherent complexity.The approach is based on Finite State Machines (FSMs), a mathematical model of behavior from the field of automata theory [25].FSMs are one of the most widely used models in computer programming and also used in a variety of other areas ranging from electrical engineering, mathematics to linguistics.In general, a FSM is an abstract machine that has a finite number of states and state transitions.A state represents any possible status of a system or object that is waiting to execute a transition.A transition is a state change triggered by a condition.It is also possible to associate actions with a state, for example, when entering or exiting.FSMs can be represented by a directed graph, which is called a state diagram (cf. Figure 4).Each state is represented by a node (circle) and the edges show the transitions from one state to another.Each arrow is labeled with the condition that triggers that transition.The start state is shown with an arrow pointing at it from anywhere and the final states are represented by a double circle.
Analyzing the characteristics of adaptive tests, it shows that the containing question can also be regarded as objects with dynamic behavior.That means, depending on the response of the student, different actions are triggered.This can be the provision of feedback or the selection of other questions.Consequently, the answer options of a question are modeled as states and the opportunities to switch between these options are expressed as transitions.In other words, changing an answer option within the question results in a transition of a state.Having this graphical representation, the author of the adaptive test has a quick overview about any possible status of a question and is able to respond directly by settings trigger points for adaptations.Trigger points cannot only be defined at states, but also at transitions.This not only allows initiating adaptations at any point during the question/test, but also to track students' answer path and to respond accordingly.As an example, Figure 5 shows a FSM model of a multiple-choice question with three answer options (A, B and C).Assumed that A is correct, the following scenarios could be realized: • A student switches between A and B several times.By counting the number of transitions between both states, this uncertainty can be recognized and supported by giving hints (feedback) to the student.• A student firstly selects A, retains the selection for a long time, but in the end submits C.Then, the student could get feedback to rethink the answer.In contrast, if the student switches from C to B, no feedback will be presented.
• A student switches between B and C, and submits B.
It could be derived that A was not an option and that the question was too difficult for the student.The following questions could have a decreased difficulty.
Summarized it can be stated that using FSMs opens up new possibilities for adaptation modeling.They graphically support the authors in defining trigger points for adaptations at any point during the question/test.In addition, students' behavior while answering the question could be tracked and used to determine students' strengths and weaknesses more precisely.However, with a growing number of answer options, the number of states and state transitions is increasing as well and the state diagram could quickly become confusing.Due to the fact that this strongly depends on the question type (e.g., given n answer options, a multiple-choice question results in n + 1 states and n! transitions, whereas a multiple-answer question result in 2 n states and 2n • 2 n-1 transitions), the decision to use this graphical support must be decided on a case-by-case basis.

F. Standard-compliant Implementation
It is important to note that askMe!uses the IMS Question & Test Interoperability (QTI) 1 specification (v2.1) for representing tests, questions, responses, etc.It is an international specification that describes a data model for representing assessment content and their corresponding results, and an Extensible Markup Language (XML) data binding that defines a language for interchanging these materials.It allows assessment content to be authored, delivered and exchanged between authoring and delivery systems, repositories and other Learning Management Systems (LMSs).The IMS QTI specification has been developed by the IMS Global Learning Consortium (GLC), an industry and academic consortium that devel-SPECIAL FOCUS PAPER AN ADAPTATION MODEL FOR PERSONALIZED E-ASSESSMENTS ops specifications for interoperable learning technology, inspired by the need for question interoperability.
The use of the IMS QTI specification enables assessment content reuse across different system and tools.However, having standard-compliant questions on the one hand and adaptation rules specified in an own format on the other hand, would break this principle.As a consequence, this would make it necessary to update any other question engine that would like to render and process these adaptive tests.The most question engine provider would not consider the added effort as justified.For that reason, all the possibilities offered by IMS QTI were fully exploited in order to implement the adaptive tests according to the specification.This is made possible by the use of QTI's response and outcome processing as well as preconditions and branching rule elements.
As mentioned above, the adaptive tests are based on adaptation rules.Depending on the adaptation mean to be addressed, the rules are either implemented on question or on test level.For example, the selection and presentation of feedback is done within an individual question and are realized using the response processing.The response processing of IMS QTI evaluates student's responses and sets outcome variables (prior defined in response declaration sections) accordingly.Then, the values of these variables trigger the selection and presentation of adequate feedback.
In contrast, the sequence, selection and presentation of questions is done within a course of a test.Consequently, adaptation rules addressing this adaptation means are realized using pre-conditions, branching rules and the outcome processing.Pre-conditions and branching rules are new features, which were firstly specified in the IMS QTI specification v2.1.While preconditions are evaluated just before the student enters a test part (a test part consists of at least one question), branching rules determine where next after a test part has been completed by a student.These elements allow skipping and retrying of questions, move to certain questions, etc. based on variables set by the outcome processing.

IV. IMPLEMENTATION
The AM proposed in the previous chapter was realized by the adaptation modeling component within the eassessment system askMe!developed at the institute of the main author.askMe! is a web-based e-assessment system that covers the whole life-cycle of e-assessments starting from creating questions over to presenting it to the students up to preparing the results and presenting them to the authors, tutors, etc.The questions and tests can consider individual aspects so that e-assessments can perfectly be tailored to students or groups of students (personalization).Moreover, the author of the adaptive tests is not limited to traditional question types such as multiplechoice or hotspot, but can use Interactive Content Objects (ICOs) to create sophisticated (interactive) e-assessments.
The adaptation modeling component is responsible for maintaining the AM and structured into several views and controllers for administration and input purposes.Figure 6 exemplarily shows view for the definition of trigger points for adaptation rules using the above mentioned FSM modeling approach.Subsequently, the adaptation rule definition view opens (cf. Figure 7).It requires authors to define the condition(s) under this rule will be triggered and the action that are will be performed.

V. CONCLUSION
This paper has presented the design of the AM and the implementation within the AAS askMe!.The AM can be regarded as the core of each adaptive system, because it defines what can be adapted, as well as when and how it is to be adapted.The model and their methods were presented using Specht's taxonomy scheme consisting of four dimensions.The first dimension (adaptation means) focuses on the elements, which are adapted by the adaptive methods.In this AM, the adaptation means are the question sequence, question selection, feedback selection, question presentation, feedback presentation and the ques-SPECIAL FOCUS PAPER AN ADAPTATION MODEL FOR PERSONALIZED E-ASSESSMENTS tion difficulty.The second dimension (adaptation information) focuses on the information that can be used to provide adaptations.The information used by this AM are student's performance and student's features.The third dimension (adaptation goals) focuses on the pedagogical reasons behind the adaptations.The main reasons behind the adaptations in this AM are the identification of students' strengths and weaknesses more valid and reliable, the consideration of students' strengths and preferences to compensate weaknesses and deficits and the encouragement of students' motivation by considering individual aspects (e.g., context or preferences).The fourth dimension (adaptation process) focuses on the actual adaptation process and the preceding acquisition of adaptation information.In this AM, the adaptation information are directly obtained from the adaptation engine during the assessment process and the user modeling component, respectively.For proposing and aiding decisions about alternatives, the AM uses the rule-based reasoning approach.In relation to the two types of adaptive techniques that are applied in AASs namely adaptive testing and adaptive questions, this AM corresponds to the latter one.Summarized it can be stated that the AM presented in this paper lays the foundation for adaptive e-assessments based on a variety of adaptive methods, allows considering students' strengths and preferences to compensate weaknesses and deficits as well as supports flexible adaptation strategies using rule-based reasoning and forward chaining.
Future work of the authors will address a comprehensive evaluation of the model and their implementation.For that, the askMe!system will be used in a real-life setting at the Ilmenau University of Technology in the course digital systems design for about 80 computer science and computer engineering students.The subjects that will be evaluated are usability, user experience and learning support.While the first two subjects will be evaluated using authors requested to create adaptive tests, the last two subjects will be evaluated by analyzing students using the system for final exam preparation.In order to obtain authentic and useful evaluation results, both quantitative as well as qualitative evaluation methods and in combination (triangulation) are chosen.

Figure 5 .
Figure 5. FSM state diagram of a multiple-choice question