Evaluation of E-Portfolio Software

E-Portfolios are a new type of software and it is still relatively vague to determine, which functions are obligatory – that is which functions constitute characteristic features – and which functions are just optional (“nice to have“). This article describes the concept and the preliminary results of a research project which was conducted to evaluate E-Portfolio software, and aims at providing decision guidance for implementing E-Portfolios in higher education first and foremost from the pedagogical perspective. Which recommendations can be made to an institution which now wants to implement electronic portfolios with a certain objective?


I.
THE CRUCIAL QUESTION: WHAT IS E-PORTFOLIO SOFTWARE?
Presently the market of E-Portfolio software has simply become unmanageable [1].This is due to the fact that the rapid development from the paper portfolio to the electronic portfolio was particularly pushed by various Web2.0 applications, which can be used for certain elements or aspects of an E-Portfolio.The crucial question is: Which software can be called E-Portfolio software?
In his position paper, Serge Ravet -director of the European Institute for E-Learning (EifEL) and initiator of the campaign "E-Portfolio for all" -puts the various E-Portfolio terms straight and tries to outline the foundation for E-Portfolio software-architecture of the future, from a rather organisational point of view [2].According to him, the future of e-learning lies primarily in the creation of a "personal learning environment (PLE)" [3].
Both experts formulate their ideas for the future of individual learning processes, in which the electronic portfolio plays a major role -even if they do so from two totally different perspectives.At the same time they both agree that at the moment the educational and technological implementation with current software products is still far from these visions.
But which recommendations can experts give to an institution in higher education now, if they want to implement a system for the use of E-Portfolios within their studies?
In order to profit from the advantages of online communication and cooperation, it is recommendable to choose a web-based solution.However, the data, which come into consideration for the use in a personal portfolio, can be stored in completely different systems, as figure 2 shows: Figure 2. Overview of software-systems with E-Portfolio share (own model, based on Erpenbeck&Sauter [4]) Which kind of software best suits the intended portfolios?Shall existing learning platforms or contentmanagement-systems be used for portfolio work?Or is it better to fall back on Web2.0 applications?Which aspects are more important: individual or institutional ones?And, which criteria can an institute of higher education utilize to determine which kind of portfolio software is sciencebased and forward-looking?

II. STARTING POINT OF THE EVALUATION: A TAXONOMY FOR E-PORTFOLIOS
A taxonomy is a systematic classification procedure to order things, occurrences, processes, etc. according to consistent and logical principles, methods and rules.[5] The main reason for the presentation, and specifically development of a taxonomy for E-Portfolios is that the various forms of appearance manifestations of E-Portfolios can be classified according to a consistent method.Again and again it is emphasized that E-Portfolios can be used in various ways; that is why there are more than twenty different types of E-Portfolio classifications found in literature on the topic [6].Due to the broad meaning of the term "E-Portfolio" the development of this taxonomy is a tedious and iterative process, which was started by Silke Kleindienst, but has not been completed yet [7].The main goals of the taxonomy are: • Development of a system of descriptors and parameters for the description of different E-Portfolio types • Formation of E-Portfolio types by testing and applying the system of descriptors and parameters on the basis of existing examples The following table shows the extract of the taxonomy which was taken for the evaluation of E-Portfolio software: According to Table I the various types of portfolios mentioned in related literature can be categorised into five major types: from "working portfolio" to "presentation portfolio", in which learners show activities from "collecting" to "presenting" when working on a portfolio.The parameters "joined" and "unjoined" are worth mentioning as well: they describe whether the different elements of the portfolios are stored in a connected or unconnected way.The terms and especially the allocation of these terms to portfolio processes play a major role for the criteria checklist.The development of this checklist as well as the research design will be described in the following section.

III. RESEARCH DESIGN A. The method of "Qualitative Weight and Sum (QWS)"
In literature, various evaluation methods for software products are offered [8].The most important ones are: • Criteria checklists • Recensions

• Comparison groups • Opinion of experts
Each of these methods holds a number of advantages and disadvantages, so that in practice a useful combination of various methods proves to be most suitable.For our evaluation, we applied the method of "Qualitative Weight and Sum (QWS)", originally developed by Michael Scriven [9], which eliminates the disadvantages of mere numerical operations as much as possible.The QWSmethod represents an iterative process of evaluation, which focuses on a weighted list of criteria.The following classification proved to be most suitable: In the first step, a panel of experts assesses the importance (weighting) of the criteria of a prepared catalogue.After that all 0-dimensions can be deleted, as these criteria were not regarded as important.
In the second step, the software product evaluands are assessed by means of the essential criteria (E) in regard to the question if the minimum requirements are met.If not, the software products are deleted from the list, which considerably reduces the amount of work for the process that follows.It is important to make sure that the criteria which were regarded as essential are "pass-fail" attributes (e.g.multilingualism of the surface -Yes/No).
Then, in the third step, the actual evaluation is carried out.It assesses the functions of the software products which are still part of the list.It must be taken into consideration that the weighting of the criterion at the same time represents the maximum possible value which an evaluand can reach in a certain category.
As a result of the previous process a hierarchy is developed (ranking), which can be provided with an integrating final assessment (grading), e.g."will be part of the top 10-list".
A disadvantage of the QWS-method is that it does not contain a definite algorithm for decision making according to a ranking, but has to be reapplied in the form of an iterative procedure several times, in order to receive significant results (e.g.paired comparison of two evaluands).Due to this fact, the method constantly changes between a holistic and an analytical point of view, but always provides meaningful and, above all, comprehensible and revisable results.
The advantages of the method are in particular: • The evaluands, which remain on the final list, basically meet the minimum requirements of an E-Portfolio software product.• The overview in table form (see above) assures a good initial assessment of the list of evaluands.• The method allows a further comparison of evaluands, which after a first assessment by the institution, are taken into account for implementation.http://www.i-jet.org • A weighted criteria checklist with pedagogically motivated meta categories permits a further evaluation with an individual setting of priorities.

B. 3.2 The criteria checklist for the evaluation of E-Portfolio software
The basis for the criteria checklist are the 69 E-Portfolio software criteria, which were developed in course of the WCET-study [10].Additionally, we introduced a metalevel layer, which refers to the taxonomy of E-Portfolios mentioned before.Thus there are five major categories: 1) Collecting, organising, selecting 2) Reflecting, testing, verifying and planning 3) Representing and publishing 4) Administrating, implementing, adapting 5) Usability The first three categories refer to prototypical portfolio processes as were described in the taxonomy; the other two categories comprise of general criteria, which distinguish between the point of view of the administrator (server-based) and the point of view of the user (clientbased).The 69 criteria which were found in literature were commented on and justifiably classified into these metalevel categories.
In course of this process, some criteria (less important, redundant, etc.) were eliminated.As a result a new criteria checklist was created in an iterative process, which was assessed by 25 E-Portfolio experts and weighted according to the QWS-method.
In addition to that, especially in the definition of the minimum requirements, the following pedagogically motivated assumptions were taken into consideration: • Electronic portfolios "belong" to the learners -that means that learners must have the right to use their data; they must be able to individually administer the access to their data themselves.After the portfolio work at a certain institution is finished, their data must still be available to them.• From the perspective of the initiating institution, E-Portfolio software should be suitable to support portfolio processes in higher education.• E-Portfolio software does not serve classroom management, that means that in particular tools for communication and collaboration in the group of learners are not part of the evaluation; however, a feedback option must be guaranteed.• The individual benefit for the learners represents the most important thing; the software is therefore not really regarded as a competence management system of the institution.
Arising from these presumptions it becomes clear that a mere offline-solution of an E-Portfolio software is not included into the market overview.The final criteria checklist for the evaluation of software is illustrated in the following table: Despite the great number of E-Portfolio experts, the dispersion about the weighting of the criteria was low (average always complied with median).

IV. THE RESULTS OF THE EVALUATION PROCESS A. The shortlist of recommendable products
Considering our propositions and minimum requirements, a list of roughly around 60 E-Portfolio providers was created at the end of January 2008.The evaluation of the software products was carried out in the period of April/May 2008 by the same panel of 25 E-Portfolio experts, using the weighted criteria list.After the last evaluation period in June/July 2008 a shortlist of 12 products was created that can be recommended for E-Portfolio implementations in higher education: 1 A term coined by yahoo, see http://developer.yahoo.com/yui/articles/gbs; from our point of view at the moment the most "meaningful" way to assess the form of browser support.• OS: open-source • P: commercial with all-inclusive offer • U: commercial with licenses per user • PU: commercial with a combination of P and U

B. Assessment of evaluands
The evaluands were assessed in the 27 weighted criteria.Due to the weighting of our criteria list the assessment * could be reached six times, the assessment # 19 times and the assessment + two times as a maximum score.
The following table lists the product names in alphabetical order and shows the sums of the individual evaluations for each software: This overview already outlines the following points: 1) All software products are far away from the maximum score possible (at most 3 out of 6 *, at most 11 out of 19 #). 2) No software is convincing in all 27 categories, as can be seen in the evaluations of the columns | and 0.
For a more detailed analysis and as decision guidance for the choice of the most suitable software product for a given institution, the list will be examined from different perspectives.Within the framework of the QWS-method, the following points have to be taken into consideration: 1) The following tables are not supposed to be interpreted as a definite ranking from place 1 to place 12.
2) It would be a mistake to assign an overall achieved score to each software on the basis of a numerical scale.
3) The lists serve as first evidence for a more detailed analysis, in the course of which the products should be compared by pairs in an iterative process.
A first ranking results from arranging the list according to the three "positive" evaluations, which means first of all according to *, and, in the event of a points draw, subsequently according to # and finally according to +.According to this automatic grading, Drupal ED and Elgg would emerge as the two top-quality products, after that a group from PebblePad via Sakai and Mahara to Movable Type with a similarly distributed positive assessment and finally all other products.
However, the examination of e.g. the assessment of PebblePad shows that the automatically generated lists can only serve as a starting point for a more detailed qualitative analysis.In contrast to Drupal ED and Elgg, PebblePad reached the assessment # only six times.On the other hand, it reached the assessment + 15 times, so that it has an overall score of 24 "positive evaluations", in comparison to 20 "positive evaluations" of Drupal ED and 19 of Elgg.
In addition to carrying out an analysis of "strengths", one can also examine the weaknesses of a software product, that means the results in the columns | and 0. The assessment 0 means that a certain feature does not exist at all or is not sufficiently pronounced, | means weakly pronounced: http://www.i-jet.orgAccording to this ranking, PebblePad would be the sole market leader with only three "negative"evaluations, after that Drupal ED and Fronter, which takes a huge step forward in comparison to the first ranking.Movable Type to Taskstream could be regarded as the center span, followed by the rest.
The example of Fronter shows very well that a differentiated examination is inevitable.Fronter could reach the best assessment * only twice; at the same time it also reached the lowest assessment 0 only once and can therefore be regarded as a very balanced product.The following third table tries to bring these considerations in line: products can be carried out with an individual setting of priorities.What such an analysis might look like will exemplarily be described in the following section.

C. Assessment of evaluands in regard to portfolio processes
For the criteria checklist we introduced meta-level categories, the first three of which are pedagogically motivated and correspond to portfolio processes.The five meta-level categories are: 1) Collecting, organizing, selecting 2) Reflecting, testing, verifying, planning 3) Representing and publishing 4) Administrating, implementing, adapting What would the assessment look like if the categories 4 and 5 were initially disregarded?That means which products support typical portfolio processes particularly well?To answer this question we simply sum up the evaluations of the first three meta-level categories and receive the following distribution: In comparison to Table 4, there are several differences.As already described above, we also assort this list first according to the strengths, and after that according to the weaknesses of the products.Eventually we receive the following overview: Important information: As mentioned before, Table 7 must not be interpreted as a final ranking.The ranking resulted from a pairwise comparison of the assessment of individual evaluands, where many products can be regarded "neck and neck", which would make the ranking look a bit different.In our opinion, however, there is clear evidence for three groups: 1) a top trio with PebblePad, Drupal ED and Elgg 2) an upper center span with Mahara, Movable Type and Sakai 3) a second half with Fronter, Taskstream, Factline, Exabis, Wordpress and Epsilen, whereupon Epsilen falls a bit off An important advantage of the QWS-method is that on the basis of the assessment results, an analysis of the In the opinion of the authors, there turn out to be three groups in regard to the criteria of supporting portfolio processes.PebblePad, Mahara and Taskstream represent a top-quality trio with a high assessment; they show weaknesses in only very few categories.These three products have especially been developed as E-Portfolio software and, according to our evaluation, they really meet the requirements.
After that, there is a wide center span with Factline, Fronter, Drupal ED, Elgg, Sakai and Movable Type.These products show restrictions concerning certain portfolio features; the main reason for that lies in the fact that none of these products was developed as E-Portfolio software; they represent different software types but were examined by us with reference to their capability as E-Portfolio software.
Wordpress, Epsilen and Exabis show weaknesses in certain features, which can be traced back to various reasons: Being a Weblog software, Wordpress just partly covers portfolio processes.Epsilen, which was already developed as an E-Portfolio product by Ali Jafari several years ago, offers a newcomer numerous supporting boilerplates.But, on the other hand, it is very inflexible, offers hardly any scope for design and, from the technological point of view, it is not at all comparable to more recent Web 2.0 developments.Finally, Exabis, as a plugin for an LMS, primarily holds weaknesses in regard to representing and publishing.There is no individual scope for design and the processes for publishing and giving feedback are rather laborious.
V. CONCLUSION AND RECOMMENDATIONS As a conclusion, the products were examined separately and compared in all five meta-level categories.On the one hand, all individual evaluations were accounted for in regard to the corresponding criteria.On the other hand, the comments and qualitative analysis, which can be seen in more detail in the individual reviews, were taken into consideration as well.For a clear illustration, a simple scale from one to three ticks was chosen, whereas three ticks represent an explicit recommendation in the respective meta-level category.
The result in alphabetical order: The additional category "effort for first time installation" describes the time expenditure the institution has to count on if it prepares its platform and the users for portfolio work.Epsilen, Exabis, Mahara and PebblePad are "out of the box" systems, which can be used right after the first time installation.Factline, Fronter, Sakai and Taskstream are systems with a modular design principle and hold the advantage that they are very flexible.But, on the other hand, they also require basic adaptations in cooperation with the provider.A similar situation applies to the Blogging software Wordpress.Drupal ED and Movable Type as CMS, as well as the social networking software Elgg are, after the first time installation, just conditionally suitable for portfolio work and require certain adaptations and additional installations, in order to provide users with the full comfort of an E-Portfolio software.
The section "collecting, organizing, selecting" shows a very pleasant result: most of the products are recommendable in this category.A result which is by far worse was achieved concerning the boilerplates for "reflecting, testing, verifying and planning"; just Taskstream is thoroughly convincing in this category.For the composition of a presentation portfolio Factline, Mahara, PebblePad, Sakai and Taskstream can be highly recommended.But it has to be mentioned that Drupal ED, Elgg, Movable Type and Wordpress did not achieve the best assessment in this category because an individual access policy and keeping more portfolios at once are not possible or just possible in a very laborious way.In regard to the category "administrating", five products are highly recommendable; in the "usability" section this holds true for only three products, namely the three "big" opensource projects Drupal, Elgg and Wordpress.
Mahara and PebblePad represent the most balanced products, which can be used for portfolio work without huge time expenditure for installation.Both systems require some acclimatization effort but -once their logic is clear -they are easy to handle.However, it has to be said that by abstaining from traditional structures of homepages (e.g.menu navigation, data management), PebblePad cuts its own idiosyncratic path.As "learning suites" Sakai, Taskstream and Fronter offer, in addition to an E-Portfolio tool, various other tools to support teaching and learning processes; that is why they might under certain circumstances be interesting for institutions which want to install a learning platform as well.Wordpress, Drupal ED, Elgg and Movable Type are completely different software types, but they can definitely be used for the purpose of portfolio work.Although they all require a relatively high adaptation effort at first time installation, they have the advantage, that they represent successful open-source projects with a huge and active community.They are the best available technology and offer individually configurable solutions through various plugins.The Factline Community Server also offers the user individual solutions.But by employing a very particular concept, the software is not easy to handle for newcomers and certainly requires a much more intensive study.Exabis provides Moodle users with an easily operated and structured data pool with an export function, which is hardly offered by any other tool.On the other hand, Exabis shows serious weaknesses concerning the support of portfolio processes, especially in regard to the design of a presentation portfolio.Epsilen is a simple and clearly arranged system, which offers support for designing an E-Portfolio as a personal homepage.But at the same time it is also highly inflexible; the scope for http://www.i-jet.orgdesign is very limited and the technology is partly antiquated, which calls for the further development of the software.
In addition to the qualitative descriptions of the 12 software products, the detailed criteria checklists are available on request for a more detailed analysis.

Figure 3 .
Figure 3. Overview: Results of the evaluation of E-Portfolio software

TABLE I .
EXTRACT OF THE TAXONOMY FOR E-PORTFOLIOS

TABLE II .
THE CRITERIA CHECKLIST FOR THE EVALUATION OF E-PORTFOLIO

TABLE III .
THE SHORTLIST OF RECOMMENDABLE PRODUCTS

TABLE IV .
THE SUMS OF EVALUATION SCORES FOR EACH SOFTWARE PRODUCT

TABLE V .
FIRST RANKING ACCORDING TO THE THREE "POSITIVE" EVALUATIONS (FIRST *, THEN #, FINALLY +)

TABLE VII .
TABLE COMBINING THE RESULTS OF THE TWO AUTOMATICALLY GENERATED LISTS

TABLE VIII .
LIST OF PRODUCTS WITH EVALUATION SCORES ACCORDING TO META-LEVEL CATEGORIES 1, 2 AND 3 (ALPHABETICAL ORDER)

TABLE IX .
TABLE COMBINING DIFFERENT RANKINGS ACCORDING TO EVALUATION SCORES IN META-LEVEL CATEGORIES 1,2 AND 3