An Empirical Evaluation of E-Portfolio Critical Success Factors

This paper further explores and empirically evaluates Critical Success Factors (CSFs) for successful implementation of the next generation of ePortfolio systems in academic institutions. Interpretive Structural Modelling (ISM) is used to empirically evaluate the nature of CSFs and to determine their relationships. The identified factors were found to be critical for the successful implementation and deployment of ePortfolios. Moreover, three-level hierarchy between CSFs has been revealed and presented in the final ISM model. In the end, the main drivers for successful implementation of ePortfolio in educational settings were detected and their influence on the remaining CSFs was explained. Keywords—EPortfolio, critical success factors, interpretive structural modelling


Introduction
By no doubts, ePortfolios are very widely used in educational settings. They offer a new philosophy of teaching and learning, giving the learner an opportunity to express oneself, to show ones past work and experience to all interested parties ranging from teachers to potential employers. Electronic Portfolio Consortium [7] indicates numerous benefits of ePortfolios in education such as helping students to develop and recognize skills, assess their progress, demonstrate how skills are developed over time, make career decisions, and promote themselves professionally. As a result, ePortfolio implementation in an academic institution is by no means simple and requires an extended study to enable all the parties involved, i.e. the learner, educator, organization and potential employer, to benefit most from its implementation [11,12]. Supporting claim comes also from [14] who indicate that "ePortfolio system implementation is in general a comprehensive educational innovation and therefore support has to be provided in both a pedagogical and technical sense".
The first authors who approached the implementation problem from a more general perspective and tried to identify a specific set of criteria within an educational institution required to implement ePortfolios are Gathercoal et al. [9]. They identified the existence of factors that they found to be essential for ePortfolio implementation and refer to those as Critical Success Factors (CSF). In addition, they state that CSFs "must be present and active" in order to implement an ePortfolio system [9]. Those factors are particularly reflected in institution's strategy and approach towards ePortfolio implementation and usage, and involve grading system for educators and students, training opportunities, financial and other material resources, etc. Therefore, a set of identified CSFs refers to the institutional level of ePortfolio usage, rather than individual.
Since the initial CSFs were proposed in 2002 and with respect to the rapid changes and improvements in technology and practice in the past several years they became outdated. With this in mind, our first research further explored the identified factors and proposed a new, updated list of CSFs based on the up-to-date research findings and experts' consultations [4].
However, so far there was no empirical attempt to show that those factors are indeed "critical" for the implementation of ePortfolios. An attempt to identify a more specific set of CSFs in tertiary educational institutions based on literature review was done by Kung-Keat [13]. The paper proposed six CSFs primarily to be considered by teachers who implement ePortfolios but without empirical evaluation. Five different studies have been considered and CSFs were re-classified. The six proposed CSFs were: User, Infrastructure, Community, Fun, Social Presence and Learning Approach and Teaching.
Since the focus of previous paper was solely on educators, this research will focus on findings from Gathercoal et al. [9] and will investigate whether their proposed factors can be considered "critical" or "key factors" for successful implementation of ePortfolios in educational settings. Moreover, this paper seeks to further explore the importance of CSFs and to describe the structure of interrelationships among these factors.
Interpretive structural model (ISM) is one of techniques to identify the internal and external barriers and other factors which affect the success of an organization. The ISM process transforms unclear, poorly articulated mental models of systems into a visible and well-defined model. ISM is found to be a well-proven and widely accepted system modelling approach for analyzing the interrelationships between the variables influencing the system [15,21].

Research Background
Several researchers identified factors critical for successful implementation of ePortfolio [9]. However, with respect to the rapid changes and improvements in technology in the past several years, some CSFs became outdated and today some of them, such as the requirement that all classes have an Internet access with computer display projection units, are fulfilled by default so there is no need for them to be categorized as critical. By reviewing these criteria and factors as well as several dozen other sources and ePortfolio project reports, a new set of critical success factors for ePortfolio implementation will be proposed. Critical factors for successful implementation of enterprise systems are discussed and studied in the literature (see for exam-ple [2] and [8]). The results of the ePortfolio CSF literature review can be summarized as follows:  Critical success factors identified in the extant literature are rather outdated since they were identified by [9]. in 2002. They need to be re-examined, since although they observe the ePortfolio in its entirety, some of them are not critical any more, and some of those that should be proclaimed critical due to the technological and pedagogical development are missing.  Several attempts have been made to identify factors that are important for using ePortfolio [5,6,10]. Some of them were rendered only in a narrative manner without any support of quantitative research methods. All studies mostly observed ePortfolio solely from the learner's perspective, thus missing the other perspectives such as the teacher's or institution's one.  Based on the above, all the identified CSFs that have an effect on ePortfolio implementation and usage will be taken into consideration and will be included in the process of identification of critical success factors accordingly. Moreover, except for a revision of the existing CSFs in the current literature; expert opinions and authors' experience in using ePortfolio will be used to update the CSF list.

Research Objectives and Methodology
The objective of this research is to:  Empirically determine whether the identified factors can be considered critical for successful implementation of ePortfolio in educational settings.  Seek for and describe relationships between those factors.
For that reason, Interpretive Structural Modelling (ISM) was chosen, as a wellestablished technique for identification of relationships between different variables or items. Warfield [20] describes ISM as "a computer-assisted learning process that enables an individual or a group user to develop a structure or map showing interrelations among previously determined elements according to a selected contextual relationship". It is a mathematically derived methodology that enables us to describe a complex problem using structural modelling of interconnected matrices. ISM enables the creation of models that represent interdependencies between a set of variables by employing directed graphs or digraphs to represent complex structures diagrammatically [1].
The key steps in ISM modelling followed in this paper are:

Selection of CSF Variables
CSFs identified by [9] are fairly outdated and some of them, such as the requirement that all classes have an Internet access with computer display projection units, are nowadays fulfilled by default so there is no need for them to be categorized as critical. Therefore, a prior research [4] was carried out in order to revise the existing list of CSFs where up-to-date research literature and experts' opinion were taken into account. As a result, a revised list of 16 CSFs was identified as follows:  Rewards for students and educators: Students and educators are encouraged to use ePortfolio (rewards for educators, extra scores for students within the course).  Accessibility of services: All participants have equitable access to the ePortfolio services.  Mandatory usage: Students complete ePortfolios as requirements in courses.  Culture of usage: The student's work in the ePortfolio strongly contributes to define the student to faculty and recruiters.  Grading and providing feedback: Faculty grade and provide feedback on students' work.  Push for adoption: The push for adoption and implementation of ePortfolios comes from faculty management, students and educators.  EPortfolio steering group: A group of faculty members has the commitment and stamina to make the ePortfolio system work.  Developed implementation plan: An implementation plan exists, with reasonable milestones that are measurable and that collectively lead to full implementation (adoption).  Training opportunities: Opportunities exist for student/faculty/mentor training (multiple times and places).  Using ePortfolios for assessment: Faculty commit to casting course assignments into a uniform format to adopt ePortfolio as an assessment tool.  Available financial and technical resources: Financial and other material and technical resources are committed to the implementation and evaluation of ePortfolio.  Frequent review of the curriculum: Faculty teams periodically review and revise the content of the curriculum and are aware of the content of courses making up the entire program.  EPortfolio integrated in institution's vision: The ePortfolio initiative is part of the strategic IT vision of the institution.  Process approach: The ePortfolio is approached as a process, not a product.  Adoption as a change management: The long-term adoption (assimilation) of the ePortfolio system is approached as an organizational change management initiative.
 Existence of post-implementation group: There is a permanent ePortfolio adoption (post-implementation) group monitoring and searching for mutual technologyorganization adaptation.

ISM Process
Interpretive Structural Modelling (ISM was primary introduced by Warfield [19] in order to evaluate complex socioeconomic systems. The process is interpretive because it is based on a decision and judgment of a group of participants about whether and how the system's elements are linked [16]. According to the same source, structural part of the process refers to mathematical procedure taking into account established relationships between variables that ends with a series of matrices. Next, the modelling part reflects the ability to show the results in a form of directed graphical model.
As indicated in previous section, variables selected for the ISM process are as follows:  Rewards for students and educators  Accessibility of services  Mandatory usage  Culture of usage  Grading and providing feedback  Push for adoption  EPortfolio steering group  Developed implementation plan  Training opportunities  Using ePortfolios for assessment  Available financial and technical resources  Frequent review of the curriculum  EPortfolio integrated in institution's vision  Process Approach  Adoption as a change management  Existence of post-implementation group In the next step, seven experts were selected and involved in the review process. They were all ePortfolio experts involved in a large number of ePortfolio projects for more than 15 years focusing mostly on three different target groups: educators and students, policy makers, and industry. During this step, group members were introduced with the process and CSF's. They were asked to identify any contextual relationships between variables using the standard ISM notation described in the next section.

Structural self-interaction matrix
The results of group work provide input to Structural Self-Interaction matrix (SSIM) that shows whether and how variables are related. The relationships between variables established by the group are presented using a standard ISM notation: V: variable i will assist to reach variable j A: variable j will assist to reach variable i X: variables i and j will assist to reach each other O: variables j and i are unrelated Based on the ISM notation above, a SSIM can be created (Table 1) in which relationship between variables is set out in terms of rows(i) and columns(j) and their respective relations.

Reachability matrix
This section presents creation of Initial Reachability Matrix (IRM) and Final Reachability Matrix (FRM). IRM is created by converting symbolic SSIM matrix into binary format (see Table 2) using the rules below [17]: Table 2. Initial Reachability Matrix (IRM) Variables (j) 16  The final IRM is presented in Table 3 using conversion rules as indicated above. After the IRM had been created it was further checked for transitivity. Transitivity here can be described as follows: If variable A is connected to variable B (A → B), and variable B is connected to variable C (B → C), then a transitive relationship between variables A and C (A → C) also exists.
Final Reachability Matrix is presented in Table 3 and the transitivity is highlighted using the notation 1* "to fill the gap, if any, in the opinion collected during development of structural self-instructional matrix" [3].

Level partitions
After exploiting FRM, [19] suggests to derive Reachability and Antecedent sets for each variable in the FRM. The Reachability set consists of the variable itself and any other variable it may help to achieve. On the other hand, Antecedent set consists of the variable itself and any other variable that may help in achieving it. If there is an exact match between the reachability and antecedent sets, then those are level I variables [18] or top level variables. The identified variables are then removed from further consideration and the process is then repeated. The first iteration is shown in Table 4. Where a positive match exists between Reachability and Antecedent sets; a I is inserted against the appropriate variable. The matching variables identified in the first iteration are: 1) Rewards for students & educators, 2) Accessibility of services, 3) Mandatory usage, 4) Culture of usage, 5) Grading and providing feedback, 6) Push for adoption from all faculty members and students, 7) EPortfolio Steering Group; and 10) Using ePortfolios for assessment. Those variables are considered as top-level variables and those will not lead the other variables above their own level at the hierarchy.
After the previous level matchings were removed, a second round of partitioning was carried out. Procedure and the result are visible in Table 5. Four matching variables were identified in this round: 9) Training Opportunities, 11) Available financial and technical resources, 12) Frequent review of the curriculum; and 14) Process Approach. Those variables will lead other variables above their own level.

Classification of variables
In this step, FRM is restructured so the variables are grouped to align with the level partition stage. Such interpretation is called Conical matrix and is presented in Table  7. Besides clustered variables conical form also contains driving and dependence power of each variable that will be used to visually represent variables using Matrics d'Impacts Croises-Multiplication Applique an Classment (MICMAC) analysis. In summary, driving power indicates how strong a variable influences other variables, while dependence power shows how strong is one variable influenced by other variables.  The MICMAC diagram presented in Figure 1 classifies variables according to their driving and dependence power. Based on their driving and dependence power, variables can be classified into four different categories [3]:  Autonomous: Variables that have weak both driving and dependence power and are therefore considered to have low impact and disconnected from the system.  Linkage: Variables with strong driving power as well as strong dependence power.
Those are found to be rather unstable in terms that any action on those variables will have an effect on other variables and a feedback effect on themselves.  Dependent: Variables that have weak driving power but strong dependence power  Independent: Variables that have strong driving power but weak dependence power.
Here it is also important to note that according to [3] variables with very strong driving power are often called "key factors" and fall into category of independent or linkage factors.
A close inspection of the MICMAC diagram shows that all variables are positioned within the Linkage quadrant which means those exhibit strong driving and dependency power. This shows their unstable nature which has the effect that any action on these variables will have an effect on other variables and also a feedback effect on themselves.

Fig. 1. MICMAC diagram
More detailed review of the Linkage quadrant reveals three clusters in the area. First cluster contains variables 6) Push for adoption, 8) Developed Implementation Plan, and 15) Adoption as a change management. Those variables have strong driving power but lower dependence power. The second cluster is made of variables with lower driving power but strong dependence power: 2) Accessibility of services, 9) Training Opportunities, and 12) Frequent review of the curriculum. The last identified cluster with variables that have very strong driving and dependence power is consisted of 1) Rewards for students & educators, 3) Mandatory usage, 4) Culture of usage, 5) Grading and providing feedback, 6) Push for adoption from all faculty members and students, 7) EPortfolio Steering Group, 10) Using ePortfolios for assessment, 11) Available financial and technical resources, 13) EPortfolio integrated in institution's vision; and 14) Process Approach.

ISM digraph
The final step in the ISM process is to create digraph and to convert it into an ISM model. Digraph is a visual representation of the variables and their interdependence [3]. It is created from the conical form of the FRM.
Digraph is converted into an ISM model by replacing nodes of the variables with the statements. The final model is therefore shown in Figure 2.

Fig. 2. ISM digraph
The top level of the ISM graph shows eight variables that were identified as the Level I variables as follows: 1) Rewards for students & educators, 2) Accessibility of services, 3) Mandatory usage, 4) Culture of usage, 5) Grading and providing feedback, 6) Push for adoption, 7) EPortfolio Steering Group, and 10) Using ePortfolios for assessment. These variables have strong links on each other and very high levels of dependence power, but varying levels of driving power. Therefore, they have a high level of dependence on the lower level variables.
Second level of digraph reveals variables 9) Training Opportunities, 11) Available financial and technical resources, 12) Frequent review of the curriculum, and 14) Process Approach as variables that influence all variables at the higher level, but also have a high level of dependence on the lower level variables.
The third level of digraph shows variables that have very strong driving power and influence all variables on the higher level: 8) Developed Implementation Plan, 13) EPortfolio integrated in institution's vision, 15) Adoption as a change management, and 16) Existence of post-implementation group.

Discussion
From the previous analysis, it could be noted that all variables have felt under the Linkage quadrant. By following instructions from ISM related literature, it can be noted that Linkage quadrant is described as a place within MICMAC diagram where variables with strong driving power or "key factors", as named by some authors, are positioned [3]. Since all variables in this research are positioned within the Linkage quadrant, it is empirically confirmed that those variables are indeed "critical factors" for successful implementation of ePortfolio system. Moreover, those factors are also unstable in terms that any action on those variables will have an effect on other variables and a feedback effect on themselves.
In addition, ISM digraph revealed three different level partitions. The four variables at the lowest level of digraph: 8) Developed Implementation Plan, 13) EPortfolio integrated in institution's vision, 15) Adoption as a change management, and 16) Existence of post-implementation group are identified in the model as having the highest influence on other factors and in the MICMAC diagram having the maximum driving power. All of those variables are inter-connected and directly reflect the main tasks of the institution's Management, or in another word, a set of activities that need to be carried out at strategic level of management. Those in fact present the main drivers for successful implementation of ePortfolio in educational settings. This means that the long-term adoption or assimilation of the ePortfolio system needs to be approached as an organizational change management initiative that involves integration of ePortfolio initiative into the strategic IT vision of the institution along with the development of an implementation plan with clear and measurable milestones. All this need to be supported by a permanent ePortfolio post-implementation group who will monitor and search for mutual technology-organization adoption.
The second level of identified variables encompass variables who are directly affected by or are depend on the lowest level variables mentioned in the previous section: 9) Training Opportunities, 11) Available financial and technical resources, 12) Frequent review of the curriculum, and 14) Process Approach. This means that after the main strategy of ePortfolio implementation has been set, a variety of tactical activities need to take place. First it needs to be ensured that there are adequate financial and other material and technical resources that will support the implementation and evaluation of ePortfolio. Educational institution needs to make sure that mentors, educators and students have various opportunities for training. Also, faculty teams need to periodically review and revise the content of the curriculum and need to be aware of the content of courses making up the entire program in order to plan the integration of ePortfolio into teaching and learning. In the end, the ePortfolio implementation needs to be approached as a process, which means that activities are itera-tive, and constant monitoring is required in order to plan and implement improvements.
The top level of the ISM digraph shows factors which exhibit the highest levels of dependency of all the factors in the digraph: 1) Rewards for students & educators, 2) Accessibility of services, 3) Mandatory usage, 4) Culture of usage, 5) Grading and providing feedback, 6) Push for adoption, 7) EPortfolio Steering Group, and 10) Using ePortfolios for assessment. It is suggested that the success of those factors is highly dependent on the success of other connected factors. Those factors mostly reflect tangible activities of the implementation procedure and are mostly operational in terms of management levels. It means that if the main strategy has been set as indicated in previous sections and financial and other resources have been allocated, all prerequisites have been fulfilled for other factors to be met. Those "operationalized" factors mean that faculty ensured that all participants have equitable access to the ePortfolio services and commit to casting course assignments into a uniform format to adopt ePortfolio as an assessment tool. This factor is very tightly connected with other factors that involve rewards for educators and extra scores for students within the course, ePortfolio completion as a requirement in course, and providing feedback on students' work. Students need to feel as if their work in the ePortfolio strongly contributes to define them to faculty and recruiters. Those mentioned factors could be labelled as motivational ones and all these processes need to be monitored by a special group of faculty members who have the commitment and stamina to make the ePortfolio system work. If those factors are in place, then the push for adoption and implementation will also come from students and educators who will recognize the importance and benefits of using ePortfolios, and vice-verse.

Conclusion
In this paper, ISM method was used to empirically confirm that a set of factors identified by the literature review can be interpreted as a critical one for the successful implementation of ePortfolios. Moreover, within CSFs a subset of four factors was identified as the ones who influence, directly or indirectly, all of the remaining factors.
Relationships revealed in this paper can significantly contribute to existing ePortfolio practice by allowing ePortfolio practitioners to prioritize their efforts with some of the CSFs in order to affect the others.