MAPSS an Intelligent Architecture for the Pedagogical Support: Experiment, Results & Discussion

— In this paper, the authors have experimented the MaPSS (Multi-agent pedagogical support system) which is an adaptive architecture based on ontologies and multi-agent systems for the presentation of the pedagogical support in its principal tasks: assessment of the knowledge, analyze of results and adapt remediation. In previous works, the authors have designed an ontology to present the relationship between the test questions and the concepts to evaluate and between those concepts and their remedial activities contents. Also, they have presented the operation of each agent (IT or human) of the system and the principle of collaboration between them. The experience consists on the use of an implemented prototype of MaPSS to support learners of Moroccan qualified secondary school in the domain of algorithmic and programming, and the impact of its use on the improving cognitive decision making to adapt learning.


Introduction
To improve the quality of the learning process and help learners to overcome their difficulties, remedial activities must keep learners in their proximal development zone (ZPD) 1 , and that can't be achieved except by offering an efficient diagnosis who should estimate as accurately as possible their cognitive state.
Despite the noteworthy development in the field of adaptive learning based on the approaches that make learner in the center of his learning process by taking into account his preferences, goals, emotions, style of learning..., the assessment of learners' learning is still in the development phase. One of the consequences of this delay is 1 Vygotsky introduce the PDZ as "the distance between the actual developmental level as determined by independent problem solving and the level of potential development as determined through problem solving under adult guidance, or in collaboration with more capable peers". that: in most e-learning platforms, the assessment has always a classical structure, and the results of the tests presented do not allow an adapted remediation because these results are translated by the assignment of a profile according to the score obtained, while the same score for different learners (even if they have other parameters in common) does not imply that they have same profile concerning their cognitive state; so that the proposed remediation can maintain a learner's ZPD and not absolutely others. [1] To overcome this problem, the authors propose in this paper: The presentation and the experimentation of MaPSS as a pedagogical supporting tool based on a criterion formative assessment of learning that places the learner as an actor in his or her own learning and related regulations. The traditional responsibilities of the teacher (evaluation, remediation, regulation) will be shared with the learner (self-evaluation, self-management of errors, self-regulation), which considerably increases the effectiveness of MaPSS.
The study of the effect of the obtained results on the improvement and progression of the learner in his learning process.

Literature Review
Several kinds of research show that ICTs can contribute to the renewal of teachers' pedagogical practices, they do not necessarily do it automatically and spontaneously. The process of developing and implementing ICT in education should include a fundamental reflection on pedagogical approaches in order to understand their place in teaching and learning practice. In this perspective, this section presents briefly the main pedagogical approaches that influenced pedagogical practices at the end of the twentieth and beginning of the twenty-first centuries.

Pedagogy by objectives (PBO)
Principle: The pedagogy by objectives has been developed to respond to the need for more rigor educational systems. It provides explicit training goals, formal didactic contracts between teachers and learners and clear expectations on what learner should learn. [2][3][4] The approach underlying this pedagogy is as follows: the content of the teaching is divided into simple elements, graduated from the simplest to the most complex. Learners advance objective by objective. After having validated a sub-objective, that is to say after having crossed one of the "small steps", they can go further (to know on the next step), and so on until they reach the top of all the steps.
The teacher's role here is to divide the final objective to be achieved into subobjectives and then to put the learners in a situation to validate each of these subobjectives. When this being done, he proposes training exercises so that the learners apply the discovered notion and appropriate it.
Pedagogy by objectives consisted of dividing the learning into operational objectives to be achieved by the learners. It asked the question "What does a learner need to know or know how to do at the end of a given activity?", It also allowed the teacher to check if the purpose of his intervention is achieved or not, through small assessments (Micro-assessment), enduring or at the end of this activity.
However, learners sometimes have difficulty to make sense of the knowledge they have acquired. Although they have successfully completed the tasks that have been successively proposed, they do not necessarily understand what they have done: there may be a loss of the overall meaning of the discovered notion. The guiding imposed by the teacher prevents the learners from taking the necessary distance.
Designing of assessment activity: The notion of an educational objective is the keystone of the pedagogy by objective. These objectives give the possibility to accurately determine the expected results, from a learner, in terms of knowledge at the end of learning activities.
For the PBO, the objectives are categorized into several levels: Purposes, Goals, General Objectives, Specific Objectives, and Operational Objectives.
Operational objectives are the objectives defined by the teacher in terms of observable behavior which clearly indicate the concrete actions that the learner must perform to prove the mastery of learning.
Assessment activities are created by rewording operational objectives into questions as shown in figure 1. In this approach each objective is examined independently of others and of their interaction: it is a micro-assessment where the notion of integration and mobilization of learning in real situations are not taken into account.

2.2
Competency-based approach (CBA) Principle: The competency-based approach no longer focuses on learning content, but on their integrated mobilization in problem situations. It emphasizes the learner's ability to concretely use what he has learned in school in new and complex tasks and situations.
Depending on the perspectives and implications of this approach, knowledge should be linked to situations that allow the learner to act beyond school; it is a more dynamic vision of educational action and teaching-learning processes.
Presentation of a competency: Most authors today tend to agree on the definition of competence as the spontaneous mobilization of a set of resources in order to apprehend a situation and respond to it in a more or less relevant way. [5][6][7][8][9] Based on this definition, the competency can be represented as in figure 2

Fig. 2. Competency components
The conception of learning activities according to the pedagogy of integration In the competency-based approach, there are essentially two moments in learning: • Specific learning of resources: resources are the subject of specific learning, and it is a priority to develop competency resources. • Integration activities and formative evaluation: The second part of learning time is reserved for what we call "integration activities", it is dedicated to teaching the learner to mobilize his resources in complex situations. This integration can be done gradually, or at once, in an important module, called "integration module".
As example, we represent a competency which requires the development of 7 specific objectives, or resources. Integration can be achieved in two ways.
• Progressively: during the learning (table 1)  The first way proceeds through progressive integration. It is richer, but it is not always possible.
The integration module is followed by a formative assessment module. To conduct the formative assessment, a family of the complex situation related to competency is presented to the learners.

• Designing an assessment activity
As shown in Figure 3, an assessment situation consists of: • Problem situation: identifies the initial data that provides the context of the situation and which are useful to resolve the problem. • A set of internal resources: mainly knowledge and know-how.
• A set of external resources: can be material or informational.
• One or more tasks: the set of instructions that specify the nature and characteristics of the expected output from the learner. • An assessment grid: it includes weighted indicators associated with criteria and a competency level scale. These indicators allow judging the quality of accomplishment of tasks. For this approach, the main objective of assessment activity is to measure the degree of the competence's mastery (Macro-Assessment): the ability of learners to mobilize the set of internal and external resources to solve a problem situation.
While these activities meet largely the needs of educational systems for summative evaluation, they remain very poor in terms of the collected information for diagnostic and formative assessment. They do not examine the degree of resources mastery.

Differential Pedagogy
It is a teaching methodology that, in order to adapt to heterogeneous groups of learners and to take into account differences in learning between them, diversifies and multiplies the management of learning at three levels: by different contents; by different structures, groups of pupils; by different learning processes. [10][11][12] Pedagogical differentiation is essential. It is important for the teacher and the learners find meaning in what they learn and make connections to who they are, what they are capable to do, and what they know. Learning activities can be experienced and assimilated in different ways from one person to another, depending on forms of intelligence, types of personality, learning styles, individual rhythms, Age and life experiences. For teachers, taking differences into account helps to regulate teaching by targeting the needs of each of their learners.

Pedagogical support process
From a pedagogical perspective, pedagogical support is a process who include principal tasks: • The diagnostic of knowledge, the analysis and the treatment of its outcomes.
• The proposition of remedial situations according to learner styles and preferences.
These tasks of supporting process are shown in figure 4.

Fig. 4. Pedagogical support process
Practice assessments: To succeed in a supporting session, some points are unavoidable. The teacher must perform a diagnostic assessment to assess the level of his / her learners and to get information about their knowledge. He must then perform a formative assessment that will show their progress as well as their difficulties.
A criterion formative evaluation allows teachers to select useful information about each learner's situation in order to set up a differentiation. It also gives teachers detailed, clear and precise information on the progress of their learners. So, it represents an indispensable tool which allows the personalization of learning and its adaptation to learners 'needs.
Measures given by a criterion assessment are probably the most appropriate type of measure, either to decide whether the learner should undertake a new teaching unit or to prescribe corrective instruction if necessary.
Analyze of results: Supporting should support the learner in his difficulties, fill his gaps in one or more subjects, learn and understand his lessons while allowing him to acquire methods and therefore autonomy. To do that, the major problem is to measure learners' progress through the evolution of their grades and to focus on describing the skills and the abilities mastered by each learner. An attempt will be made to describe as best as possible the level of each learner's competence in relation to each measured criterion. Reporting results do not necessarily involve quantitative information and may have qualitative aspects.
In this context, the main concern is to measure, for each learner, the degree of achievement of each objective and also to compare this degree of achievement with a specific performance threshold.

Propose remedial activities: adaptation
Taking into account the Proximal development zone (PDZ): On the one hand, the remediation task must take into account the learner's PZD. The teacher is responsible for keeping learners in their PZD. When a learner has to perform a task that is below his zone, this task is too easy and therefore he does not learn. Similarly, when he has to perform a task that is beyond his PZD, this task is too difficult and leads the learner to failure. Taking into account the PZD means to refrain from proposing tasks that are too easy or too difficult.
The task which fits in learner 'PZD allows the learner to mobilize because he feels in a real challenge. In order to allow learners to situate themselves in their PZD, it may be necessary for the teacher to differentiate contents, structures, processes, and productions to avoid learners finding themselves either in a breakaway zone (too difficult = non-mobilization) or in an autonomous zone (too easy = no learning). The teacher must, therefore, offer the learner diversified learning situations that target his PDZ. This will allow them to develop their skills by leveraging their previous knowledge, teacher support, and interaction with their peers.
According to learning styles: Remedial activities must take into account also learners 'learning styles. The term "learning styles" speaks to the understanding that every learner learns differently. Technically, an individual's learning style refers to the preferential way in which the learner absorbs, processes, comprehends and retains information. This notion of individualized learning styles has gained widespread recognition in education theory and classroom management strategy. Individual learning styles depend on cognitive, emotional and environmental factors, as well as one's prior experience. In other words: everyone's different. It is important for educators to understand the differences in their learners' learning styles so that they can implement best practice strategies into their daily activities, curriculum and assessments.

Integration of pedagogical Approaches for the pedagogical support
Whatever is their reference frame, anyone who inscribes their reflection, or their pedagogical practice, in the competency-based approach agree today that they are exercised when it is the question to solving problem situations requiring the mobilization of several resources. Whether it is for the learning competencies or for their evaluation, it is thus advisable to place the learner in a complex environment and to require from him the mobilization of its various resources to solve a complex situation.
The purpose of a competency assessment is to assess the learner's ability to mobilize a set of internal and external resources through complex tasks called an assessment situation. This approach can be applied in the context of a summative evaluation which takes place at the end of learning for certification purposes. If a learner does not succeed in this assessment, the adoption of this approach does not allow to specify its difficulties, its strengths and even the component responsible for this failure.
To overcome this limitation, authors opt in their diagnostic tool for an assessment on two phases as it's represented in figure 5. [13]  Inspiring from the Pedagogy by objectives, each concept Is concretized by a set of operational objectives and Is evaluated and remedied, if necessary, as a learning object in its own right.
After having swept all the resources of the competence, an assessment, based on the competency-based approach, concerning the mobilization of these resources is envisaged. It is the ability to mobilize each resource related to the targeted competency that is measured to affirm that the learner is able or not to use wisely a resource that falls within the definition of competence. Table 3 describes the operationalization of the assessment based on the integration of different pedagogical approaches cited above.

Presentation
MaPSS (multiagent pedagogical support system) is an adaptive architecture based on ontologies and multi-agent systems for the presentation of the pedagogical support in its principal tasks: assessment of the knowledge, analyze of results and adapt remediation. In previous works, the authors have designed an ontology to present the relationship between the test questions and the concepts to evaluate and between those concepts and their remedial activities contents. Also, they have presented the operation of each agent (IT or human) of the system and the principle of collaboration between them.
The system can be used with three scenarios depending on the excepted purposes by the users: in the first scenario MaPSS is used by learners without their teacher for diagnostics purposes, in the second one MaPSS is used by a teacher with a class group for diagnostic purposes and in the third one MaPSS is used for pedagogical support. [14] In this paper, the authors experiment MaPSS with the first Scenario where it is used as a diagnosis tool whose results can be interpreted and exploited by a human agent or IT agent, which offers the possibility of being integrated into an e-learning platform.

Specification Framework for Users: Teachers / Learners
Competency assessment is a complex process that presents many pitfalls for the teacher to get started! For this, authors propose to teachers/assessors who would use MaPSS to: • Identify the resources to mobilize in an assessment situation to be sure that the desired competence will be assessed; • Identify situations that are sufficiently complex and significant for learners; • Ensure that these situations belong to the family of situations related to the competency one wishes to assess; • Propose in the assessment situation three independent opportunities of the same level of complexity that will make it possible to assess each criterion selected; • Identify the best indicators for each occasion and each criterion; • Manage these different indicators to establish the level of control of the criterion;

Application domain
To experiment MaPSS tool, a prototype is designed to be used by ICT 2 teachers in Morocco, for diagnosis purposes, to specify the cognitive state for their learners of common core level in the field of algorithmics and programming.
In Morocco, the teaching of ICT at common core level has at least three purposes: • Pedagogical, insofar as this teaching, facilitates the acquisition of knowledge and promotes autonomy; • Scientific, in the sense that this discipline is considered as a subject in its own right, transversal by analogy with languages, obligatory and not an optional choice; • Professional, because ICT has emerged as an essential tool, regardless of future trades. The accelerated development of computer systems and telecommunications and the new horizons they open up, especially through their interactions, are all determining factors in the objectives assigned to this subject.
The teaching of ICT, at the common core level, must allow: • The introduction of a basic computer culture; • Providing a basic knowledge of the structure and functioning of computer systems; • Introduction to the implementation of computer networks and their reasoned exploitation; • Introduction to the technical construction of computer programs.
The common-core computer program is divided into teaching modules allowing learners to: • Develop an understanding of the concepts of computing; • To acquire essential know-how and knowledge; • To develop knowledge to be specific; • To learn to undertake relevant and responsible actions.
Authors devote this part of the use case and the experimentation of their diagnostic tool in the framework of the fourth module Algorithms and programming: Specific pedagogical consideration of the Algorithmic module.
In this section, authors present the competencies and abilities targeted, the associated know-how and the pedagogical orientations to be taken into consideration. [15]

Competencies and abilities targeted Associated knowledge
The learner must be able to adopt the algorithmic approach to deal with problem situations.
-Algorithm Concept; -Constants, variables, and types; -Basis instructions (write, read, affectation); -Basis Structure of control; -Sequential; -Algebraic operators; -Representation of Sequential algorithms. • Algorithmic activities should be drawn from learners' experiences; • Emphasis must be placed more on the algorithmic approach than on the solution of the problem; • The teacher must ensure that learners choose meaningful identifiers for objects used in simple algorithms; • The teacher must ensure the proper writing of the algorithms by the learners to facilitate their reading, their execution, their maintenance; • It is better to choose a structured programming language that is close to algorithmic thinking; • To optimize machine time, learners should be asked to prepare the programs out of class; • The notions of loops will be treated in the higher levels.

Experimentation
Object of Study: It is an experience focused on pedagogical support. The main objective is to determine the impact of the exploitation of MaPSS tool on the improvement of the results obtained by a group of common core learners, following a session dedicated to the pedagogical support concerning the algorithmic and programming module.
The purpose of this study is to examine the effect of using this diagnostic tool to specify the cognitive state of learners and generate detailed reports on acquired and unvested concepts; on the improvement and progression of the learner in his learning process.
Sample of population: The sample comprises 4 classes of the scientific common core: Participants were N = 118 common core science option (CCS) learners in the framework of formal ICT courses. 52 girls and 66 boys as shown in table 5. The authors choose randomly four heterogeneous classes. Participants are arbitrarily assigned to their classes at the beginning of the school year by the school administration. The four classes chosen are randomly distributed. Experimentation protocol (figure 6) • The experiment is conducted by two ICT teachers from qualified secondary school. For each teacher, a class from the control group and another from the experimental group are already affected by the administration of the institution.
• The two teachers have the same academic and professional profile; the profile of the two teachers was determined by the pedagogical inspector of the subject based on the inspection reports of recent years. • The population is organized into two groups of learners: a control group (CG) and an experimental group (EG). • Both groups of learners belong to the same socio-cultural background and have the same level of achievement in ICT in the first semester. • The Experimental group consists of classes CCS1 and CCS3 (n=61), the learners of the experimental group undergo a pre-assessment test using MaPSS diagnostic tool and have as feedback a detailed report on the origin of the errors they have committed and their difficulties; a report summarizing difficulties and errors is generated for the teacher. • As part of the experiment and in order to be able to compare the progress of the two groups, the scores obtained by the EG are sent to the MaPSS administrator and are not seen by either the learners or their teachers. • The control group consists of classes CCS2 and CCS4 (n =57) which have undergone an assessment in an ordinary manner and have recovered only their score.

Methodology
Step 1: Preparation of Teaching-Learning Activities • Development of an assessment test in coordination with the other teacher, to build a common test for the 4 groups of learners. • Implementation of a prototype based on the elaborated test.
• Before being experimented with the experimental groups, the prototype was introduced to the teachers to get acquainted with.
Step 2: Doing the test (pre-test) Each teacher has two groups • A control group passes a written test in an ordinary way.
• An experimental group passes the same assessment test under MaPSS.

Step3: Correction and analysis of results
• MaPSS provides automatically a cognitive report for each learner and a cognitive report concerning the class group for the teacher. • Both teachers worked in an ordinary way: they correct the copies and assign a final score.
Step4: The remediation • The four groups underwent a remediation session according to the results obtained.
Step5: Doing the test (post-test) • Elaboration of the tests in coordination with the 2 teachers, to build a common test for the 4 groups of learners. • Doing the test by the four groups of learners to identify the progression of their learning.
Step6: Correction and comparison of the results • After correcting the tests of the four groups of learners, the authors made a comparison of the results based on the scores obtained.

Presentation of results
After collecting the scores obtained by the two groups, there are the following results: • Pre-test results Table 6 presents the scores obtained by the control group and the expert group following a pre-test evaluation.

• Post-test results
The scores obtained by the control group and the experimental group following a post-test are presented in table 7. And a comparison of these results is presented in figure 8      Based on the results presented above, table 10 discusses and analyzes the progression of learners' population. Table 10.

Control Group Experimental Group
The remediation did not affect the class [0.5 [: only one out of 8 learners left this class after undergoing a remedial sequence which represents a percentage of 12.5%.
Remediation has significantly influenced the class [0.5[: 7 out of 11 learners left this class after undergoing a remedial sequence which represents a percentage of 64%. Following a remedial sequence, the number of learners with a grade greater than or equal to 10 increased from 30 to 31 learners which is an advance of 1.75%.
Following a remedial sequence, the number of learners with a grade greater than or equal 10 increased from 33 to 47 learners. Which is an improvement of 22.95%. Remediation did not affect grade class [15.20 [: the number of learners increased from 10 to 14; which represents a progression of 7.01% Remediation has significantly influenced the class [15.20 [: the number of learners increased from 11 to 31; which represents a progression of 32.79% The approximate average of the scores increased from 10.03 to 10.59, an increase of 0.56 The approximate average scores increased from 9.84 to 13.43, an improvement of 3.59 after Remediation. So, a difference of 3.03 in favor of the experimental group.
The study has shown that the use of MaPSS for diagnosis purposes has significantly improved the algorithmic and programming outcomes of common core learners.  5,10[ [10,15[ [15,20] Frequency (pre-test) frequency (post-test)

Conclusion & perspectives
This research and experimentation work was conducted on two groups of learners from the same socio-cultural background and have the same level of achievement in ICT in the first semester: • An experimental group made of 61 learners who have undergone remediation based on cognitive relationships generated by MaPSS. • A control group made of 57 learners who have undergone remediation without the benefit of MaPSS.