Electronic Test or Printed Exercise Book ? Which Individual Mode of Preparation is More Effective ?

At Szent Istvan University we teach database management in the framework of several subjects. For preparation for the data modeling exam students of one of these courses prepared with help the electronic tests of Moodle e-learning education system, while the attendees of the other course used printed data modeling exercise book, we wrote him based on the inspiration of studentsâ?? demand. With examinations based upon statistical methods we looked for an answer to see whether one of the preparation methods yield significantly better results. We analyzed whether there is any remarkable difference between the advantages and disadvantages of the two ways of preparation. We made a survey among students to see which of the preparation methods is favored by them. Based on the outcome of the analysis we are going to decide which one of preparation methods will be recommended by us as teachers.


I. INTRODUCTION
At the Faculty of Mechanical Engineering of Szent István University we teach Database management (ABK) subject for mechanical engineer students specialized for engineer informatics.The course is one semester long with 2 lectures per week.Its main topics are the data modeling and the SQL language.
The environment engineer students of the Faculty of Agricultural and Environment Engineering can attend the Computer studies III.(KM3) course, which is also one semester long, with 2+2 lessons peer week.The main topics of the subject are the database management and the CAD.The students study database management in the first half of the semester, at lectures they study data modeling and at the seminars the SQL language.
The teaching of the database management follows the same syllabus at both courses.Our research covers the individual preparation methods of the data modeling topic.The data modeling topic can be cascaded down to the following parts: fundamental modeling concepts, entity relationship model, relational model, functional and multivalued dependences, normalization and dependence diagrams [2].
In the framework of the university education we give lectures in the classroom with teacher aid.At the same time the Moodle e-learning education system is available also, which allows the use electronic curriculum and tests beside the printed curriculum in the education.So teach-ing of both subjects and data modeling happens in blended-learning.
The blended-learning is an educational form which allows students to see the advantages of both face-to-face and online learning.In this educational form the classroom classes acquit students of the independent studying the curriculum, but at the same time they can use the technology of e-learning in the individual preparation phase.This latter plays an important role in the course of correspondence course, where the number of the classroom classes is low.This learning method allows the teacher to provide more help for students with lower performance [7], [10], [12].
There are areas of database management which can be taught more effectively by the conventional method.The data modeling demands high level of abstraction, the presentation of its process is more effective with teacher explanation in the framework of a conventional lecture.In the course of teaching SQL language the contribution of the teacher is also important in the presentation of the logic of the language, as students meet with declarative programming language at first this time.
In the course of teaching the different subjects the lectures, the consultations and the practices of the database design happen in the framework of conventional classroom classes.The database management practices are held in computer labs.
The students acquire the curriculum through individual learning.The teacher provides them with assistance at the consultations taken in the data modeling and SQL topics.
The data modeling exam paper is a computational test.For this exam paper the individual preparation happens with solving practice tests (ABK courses) or solving of the tasks of the exercise book (KM3 course).At the ABK courses nobody learned with help of the exercise book, while 75%-78% of students of KM3 course prepared with the help of it for the exam paper.70%-80% of the students of ABK courses solved practicing tests.
Reference [9] shares experiences of blended-learning used in distance learning and results of survey made among the students.Accordingly the bigger part of students used the printed curriculum (textbook, exercise book) for the learning.Out of electronic tools most of the students used electronic tests its frequency is higher than the frequency of the use of electronic communication forms (chat, forum, message).The time of the learning with electronic tools was substantially shorter than the PAPER ELECTRONIC TEST OR PRINTED EXERCISE BOOK?WHICH INDIVIDUAL MODE OF PREPARATION IS MORE EFFECTIVE?length of time of learning with printed curriculum.The students did not learn continuously, but campaign-like.
The Moodle e-learning educational system is used on the Department of Informatics of the Szent István University for 7 years and on the university level since 2008.To date more than 800 courses can be found in the university's Moodle system.Every subject of the Department of Informatics has a course in Moodle.The Faculty of Mechanical Engineering is the faculty on the university with the highest number of online courses.
Reference [17] describes ten years of e-learning within the technical education in the Czech Republic.It compares the definition of several authors of e-learning and emphasizes two aspects of e-learning: "First, it is the educational process supported by information and communication technologies; second, it is a set of technological tools supporting education."It presents the process of gradual introduction of e-learning on 19 Czech universities between 1999 and 2007.Finally it evaluates the results of a survey on the usage of e-learning and finds that e-learning has become an integral part of the information society.
Reference [1] also states that nowadays online education basically replaced letter courses and video lectures, education of engineering with online studies became widely used and recognized.The authors in addition summarized the requirements for online courses:  they must be similar or better than the classroom education,  they should be available anywhere and anytime for any student,  they should be available in several engineering topics.Reference [19] reviews the advantages of the usage of Moodle for teachers and students:  for the teachers the system allows course planning support, information sharing with other teachers, monitoring students' activities, follow-up of students' learning process,  for the students the system helps planning their individual works, allows information sharing with each other, and ensures receipt of teacher feedbacks.

II. ELECTRONIC TESTS IN THE TEACHING OF THE ABK SUBJECT
We use the Moodle e-learning education system from the first semester of school year 2007/2008 at the teaching of the ABK subject.We elaborated the questions of the electronic tests based on the works of [2], [15] and [18].In the data modeling tests a small part of the questions are theoretical, the majority of them are exercises for practice.The writing of the questions lasted for the next school years, so the number of the practice and exam tests changed year by year.We appended teacher explanation to the majority of the questions, which were available for students after closing of the tests.
We used a different test series year by year in order to meet new aims yearly.In the first two years there were only practice and exam tests.In the first year there was only one practice test with 21 random questions and one exam test.We gave relatively longer time for solving the practice test, we did not show the correct answers after the closing the test, and we restricted the possible attempt up to 3 times.This way we were going to drive students for thinking and finding answers for the problematic questions in the curriculum.
In the second year the practice tests consist of 4-8 random questions out of the 15-35 available questions in connection with the topic.We did not restrict the number of the attempts.We showed the correct answers after the closing the tests so that students can learn from comparison of their answer and the correct one. 1 on a five degree scale 2 text assessment, the 58% of the students answered, who considered the tests good In Table I. we summarized the characteristics and results of the tests.In 2009 and 2010 students used the same test series; this is the reason of the contraction of the results.
Among the four courses examined by reference [19] three have tests.It is salient, that in two courses the numbers of test attempts by student is 5.03 and 4.4 or rather 4.32, while in the third course 2.44 and 2.04.For comparison in the courses examined by us these values are 2.8, 6.29 and 2.4, but the middle value is not typical, it is due to the extraordinary activity of one student in a small sized group.
In 2008 at the end of the school year we evaluated the results, as well as we examined the learning practice of students [3].Although students practiced substantially more than in the previous year, the results of the exam test were practically unchanged in the first two years.In 2008 one student caused the high level of student practices who solved the tests much more times than the other students of the course.
Examining the learning practice we experienced that in the beginning of the semester students practiced more than in the second half of the course, utmost in the week of the exam test.During the day the most active periods were the early evening hours and the one or two hours after midnight.
In 2009 we prepared three additional tests with new questions in all topics.We wished to measure the level of understanding of the curriculum of the course with the help of the tests "before the practice".Students had to solve the tests at the end of the classes.We established "control before practice" tests to measure the efficiency of practice, which had the same content as the "before the practice" tests, we asked the students to solve them after iJEP -Volume 3, Issue 1, January 2013 PAPER ELECTRONIC TEST OR PRINTED EXERCISE BOOK?WHICH INDIVIDUAL MODE OF PREPARATION IS MORE EFFECTIVE? the practice.The measurement of the acquisition of the knowledge happened with "after the practice" tests, these were the exam tests.
In 2009 we increased the number of the practice tests in the following topics: normalization, dependences and dependence diagrams.We limited the time available for solving the tests, and showed the correct answers after closing it, while we did not limit the number of attempts.We used the same test series in 2010 as in 2009.
In 2010 we evaluated again the learning practice of the students [5], [6].We experienced that the students learned henceforward campaign-wise instead of continuous learning.Only half of the students solved the tests which measured the efficiency of practice.They used most of the time to solve the dependency and normalization tests, these are the most difficult topics of the data modeling.During the day the active periods were the same as at the previous examination.
According to data of Table I. the result of all three years can be evaluated as satisfactory, but in the last period the efficiency has decreased.The reason for the not improving results can be the followings:  The students practiced less. The time devoted to solve the tests decreased. The students do not have the same abilities. For part of students the traditional educational form would be more suitable, that is learning from a book, and practice with the help of an exercise book.
According to the data of Table I. in 2009 and 2010 students solved more tests than in previous years, but since in this period were the number of practice tests the highest, the number of attempts per test solved by student is slightly lower in this period.We raise an issue as teacher: did the students practice less or is the number of tests too high in the third period?
Table II.shows the distribution of the tests by topics.There is more than one test in the relational model topic, due to the importance and the extent of the topic.More tests belong also to the normalization topic which is caused by the adequate learning of the algorithm of the normalization.At this topic each test comes along the normalization of a table.Five sample tasks can't be considered too much, due to the importance of the topic.
The time devoted to solve the tests decreased in the consecutive years, see Table III.This may take us to the conclusion that the aim of achieving better results changed and rather became receiving the correct answers as soon as possible [6].
Although we did not make a survey to compare the ability of the students of the different courses, but it has to be mentioned that the marks of 2009' course were worse than the marks of the other courses.In 2009 there was no better mark then satisfactory.The students of this course lagged with one year from university sample syllabus.Their results influenced the results of third period too.
The majority of the students of ABK courses found the tests useful during the preparation and enjoyed solving the tests, but there were some students in each course who were interested in the exercise book.Partly because of this, on the other hand to improve the efficiency of the teaching we began to compile a data modeling exercise book.

III. EXERCISE BOOK IN THE TEACHING OF KM3 SUBJECT
We published the data modeling exercise book in print in September 2010 containing 300 exercises.Part of these exercises could be found in the electronic tests also, while the other part consisted of new exercises.In the exercise book 50 questions are theoretical, 250 are practicing, shorter modeling tasks [4].
In the autumn semester of 2010 we also recommended the use of the exercise book for the students of the ABK course, but they chose the electronic tests.On the other hand the students of KM3 course prepared for the exam paper with the help of the exercise book.The exam paper was an electronic test in this case too, before exam the KM3 students could only solve a sample test consisting of 22 questions in the Moodle system to try this exam method.
We summarized the advantages and disadvantages of the two individual preparation methods based on our experience and [14] in Table IV.At the electronic tests the necessity of the computational infrastructure can't be considered as a disadvantage, because every student has a computer.The disadvantage is that the exercise book is not free of charge, but the exercise book is cheap compared to Hungarian information technological books and the students have 20% discount at the purchase.There is consultation opportunity for students learning from exercise book too, but it did not happen in an electronic way, but in classroom at a predetermined time.
Two facts supported the use of the exercise book: we experienced that students were learning in teams, which was not typical for the solution of the electronic tests.During the consultation the students may write sort notes into the exercise book, the task and the notes remained available later too.We held an assessment after the solution of the electronic tests too, but in this case the questions and the notes are not available together at the same time, because students can't search in the question bank, and there is not enough time to put down the questions.The circumstances of writing the exam tests of the ABK 2007-2008 and the KM3 courses were similar as we generated the ABK 2007-2008 exam test from practiced questions, and the KM3 exam test from the questions of the exercise book.This did not mean in either of two cases that the students knew all the questions, because they solved only a part of the questions of the question bank during the practicing tests, while the students prepared from exercise book also dealt with only part of the questions of it (see Table XI

.).
In the course of examining the results of the exam tests of the ABK 2007-2008 and KM3 courses the main questions are the followings:  May the results of two groups be characterized similarly? Have the two data rows same distribution? Is the result of either group significantly better?First we calculated the average time devoted to solve the exam test and average scores as well as the correlation coefficients to compare the results of the courses.After this we represented the scores in the function of times devoted to test solving on diagram and determined that 3-3 clusters into which the values can be assigned.Finally we executed the significance test of the correlation coefficients too.
We analyzed too if the data rows are normally distributed, because the results of the exam papers are generally normally distributed.To decide whether the two data rows have the same distribution we made a homogeneity test.
The average scores already show which group's result is better, but to verify this we represented the distribution of the results on diagram too.Finally we made an inde-pendence test to decide whether the results are significantly dissimilar.
The exam tests of the ABK 2009-2010 and KM3 courses were principally similar.We examined 20-20 questions of these exam tests since these questions can be stood into pairs based on their type.With examination of the results of these questions we are going to answer two questions:  In this case is the result of KM3 group significantly better too? Are there any topics which can be considered easy or difficult equally for both groups?
In this case we calculated the average scores too then we made an independence test to decide whether the results of two groups are significantly differing.By our opinion there is not a disadvantage that the exam test of ABK 2009-2010 course did not generate from practicing question, because in practicing tests and in "control before practice" test the students of this course solved same type questions than questions of the exam tests.

V. THE RESULTS OF STATISTICAL AND DATA MINING
EXAMINATIONS We executed the examinations by [11], [13], [16].In the course of the examination of the results of the ABK 2007-2008 and KM3 courses first we calculated the average time devoted to solve the exam test and average scores, then we determined the correlation coefficients, see Table V.
To decide whether there is a relationship between the time used and scores of the ABK 2007-2008 group we analyzed the significance of the correlation coefficient.We made the examination with data of KM3 group too.Our null hypothesis is the following in both cases: shows the results of the examinations.We represented the achieved scores in the function of times devoted to test solving on diagram in case of both groups, see Figure 1.We put the regression lines, their equations and determination coefficients on the diagram too.
Beside the averages at both groups we defined those three-three clusters, into which the values can be assigned, see Table VII., and we depicted their centre on the figure 1.These clusters show tendencies better than the average values.
The result of the exam paper of a group is generally normally distributed.We examined with Chi-squared test whether the results of exam tests of two groups are normally distributed?We found with 95% probability that neither of them is normally distributed.
We made a homogeneity test too to determine whether two data rows have the same distribution.Our null hypothesis is the following: H 0 : two data rows have the same distribution Table VIII.shows the results of the examinations.We prepared a diagram showing the distribution of the results instead of the usual frequency diagram, see Figure 2.
We made an independence test in order to decide whether KM3 group's result is significantly better.Our null hypothesis is the following: H 0 : there is no relationship between the result and the way of the preparation, meaning the data rows are independent.Table IX.shows the results of the independence test.We calculated three contingency coefficients.The value of the Pearson contingency coefficient is 0.284028 (1), the value of the Tschuprov contingency coefficient is 0.148114 (0.707107), while the value of the Cramer contingency coefficient is 0.296227 (1), in the parenthesis are the possible highest values.
At the 20-20 questions examination of the ABK 2009-2010 and KM3 groups first we calculated the average scores, then we examined whether the data rows are normally distributed.The average score of the ABK 2009-2010 group is 16.18, while the average score of the KM3 group is 17.62.So the result of the KM3 group is better with 6.25%.At the normally test we found so that result of ABK 2009-2010 group is normally distributed with 0.05 significance level, but the result of the KM3 group is not.
We made the independence test too Table X.shows the results of it.
We calculated the average scores by topic too, then we divided it with the maximum achievable score in the topic and depicted the values on diagram, see Figure 3.     V. show that the students of the ABK 2007-2008 group devoted by 1.65% more time for the solving of the exam test than the students of the KM3 group, but their average result was worse by 5.2%.The correlation coefficient of the KM3 group is negative its absolute value shows that there is a very weak relationship between the time used and the achieved score.Its negative value shows so that those students who devoted more time for the solving exam test achieved worse result than those students who solved exam test in less time.In the ABK group the value of the correlation coefficients is almost zero, it refers on very weak relationship too.
Based on the result of the significance analysis of the correlation coefficients (Table VI.) we can claim with 95% probability that at ABK 2007-2008 group the correlation coefficient can be considered zero, thus there is no significant relationship between time devoted to solve the test and the achieved score.But at KM3 group the correlation coefficient is not zero.
The clustering shows so that about half of the students of the KM3 group achieved better result with medium time spent on it.In contrast to this 70% of the students of the ABK 2007-2008 group achieved medium result with the longest time spent (Figure 1 and Table VII.).
The analysis indicates that the results of two groups do not work similarly.Neither of two data row is normally distributed.The homogeneity test indicates at 0.05 significance level that two data rows do not have the same distribution.
The average score is better in KM3 group and the frequency diagram of Figure 2 shows the better result of KM3 group too.At KM3 group 35% of the students achieved the best result while in ABK 2007-2008 only 11% of them achieved it.However in ABK 2007-2008 group more results were between 61%-85% than in KM3 group.All in all we can say that the result of KM3 group is better.
According to the result of the independence test (Table IX.), with 95% probability the data rows are not independent, so the preparation method does influence the result.Although the Chi-square value is not a proper indicator for the strength of the relationship of two data rows, but it is well known, that its small value shows the independence of the data rows, while its higher value refers, as in our case, on that the data rows are not independent.All three contingency coefficients calculated by us differ from zero, which shows a medium weak relationship, so the data rows are not independent.Thus the result of the KM3 group is significantly better.
At the examination of the 20-20 questions exam test of the ABK 2009-2010 and KM3 group we found that the result of the KM3 group is better.According to analysis of the distribution the two data rows does not have the same distribution.The independence test shows that the two data rows at 0.05 significance level are not independent, on the other hand at 0.03 and smaller significance level they are independent.If the level of acceptance is between 1% and 3%, then the statistic practice rejects the null hypothesis.So the result of the second independence analysis confirms the result of the first independence test.Thus the mode of the learning influences the result.
According to Figure 3 showing the result by topics it can be seen that it is true for neither of the groups that all result are better than the other.Neither of the topics can be considered easy or difficult equally for both groups.
Thus we can say based on our analysis that the learning from exercise book proved more effective.VII.SURVEY At the end of the semesters we made a survey, in which beside other topics we asked the students how useful they found the preparation with the help of practice tests and with the use of the exercise books.In course of the survey we followed [8]'s guide.
We outlined the summary of answers of the ABK courses already in Table I.We asked all students of KM3 group to answer our questions.66% of the KM3 group' students took part in the survey, it is random who did not fill up the questionnaire thus we can consider the survey representative.From among the questions two are interesting for us:  How many times did they solve the exercises of the exercise book or rather how many per cent of the exercises did they solve more times?(Table XI.)  Which preparation method did they consider more useful, and which one would they recommend for the next college year students?(Table XII.)According to the data of Table XI.a quarter of the students did not use the exercise book.1/5 of the students solved all exercises of the exercise book, a quarter of them solved more than 75% of the exercises, while the other students attended to less than half of the exercises.The number of the solutions of exercises for the second time or several times is decreasing.Only few per cent of the students solved all exercises more times (4.3%).
According to data of the Table XII.more than half of the students consider both preparation manners useful.Among those students who marked only one of the preparation manner a little bit more students considered the exercise book useful.Two third of the students recommended both of the preparation manners for the students of the next colleges year.The proportion of those students is near alike, who suggest only one or other preparation manner for the future students.VIII.SUMMARY The analysis made based on basic statistic features indicated that the results of the students learning with the help of exercise book are better.The values of the contingency coefficients indicate that the method of the preparation defines the results on a medium level.All of the two independence analysis shows a significantly better result at the students learning from exercise book.There is not an essential difference between the advantages and disadvantages of the preparation methods, just like no difference in the level of support among students preferring preparation methods.Based on these we would suggest the use of the exercise book as a teacher, but we leave the decision to the students due to the different preferences of students.
For the assessment of the results of two subjects on one hand we compared the results of the exam tests of the ABK 2007-2008 and the KM3 courses, on the other hand we examined the results of 20-20 questions of the exam tests of the ABK 2009-2010 and the KM3 courses.At the ABK courses in 2007 and 2008 the exam tests were the same, and the students of the ABK 2009 and ABK 2010 had to solve the same exam tests, for this we contracted the results.

Figure 3 .
Figure 3. Results by topics

TABLE II .
DISTRIBUTION OF THE PRACTICE TESTS BY TOPIC OR PRINTED EXERCISE BOOK?WHICH INDIVIDUAL MODE OF PREPARATION IS MORE EFFECTIVE?

TABLE V .
AVERAGE TIMES AND SCORES IN EXAM TESTS PAPER ELECTRONIC TEST OR PRINTED EXERCISE BOOK?WHICH INDIVIDUAL MODE OF PREPARATION IS MORE EFFECTIVE?

TABLE VIII .
THE RESULT OF THE HOMOGENEITY TEST

TABLE XI .
THE RESULTS OF THE KM3 SURVEY I.

The use of exercise book or solving of the tests did support better your preparation? Which preparation method would you rec- ommend for the students of the next college year?
ELECTRONIC TEST OR PRINTED EXERCISE BOOK?WHICH INDIVIDUAL MODE OF PREPARATION IS MORE EFFECTIVE? PAPER