Using Open-Source Software for Sign Language Learning and Assessment

: This study reports on the use of an open-source software for sign language learning and (self-)assessment. A Yes/No vocabulary size test for Swiss German Sign Language (Deutschschweizerische Gebärdensprache, DSGS) was developed, targeting beginning adult learners. The Web-based test, which can be used for self-assessment or placement purposes, was administered to 20 DSGS adult learners of ages 24 to 55 (M = 39.3). The learners filled out a background questionnaire, took the Yes/No test tests, and filled out a feedback questionnaire. The comments provided by the learners about the suitability of the Web-based DSGS vocabulary self-assessment instrument provided concrete feedback towards improvement of the system. Switzerland Abstract — This study reports on the use of an open-source software for sign language learning and (self-) assessment. A Yes/No vocabulary size test for Swiss German Sign Language (Deutschschweizerische Gebärdensprache , DSGS) was developed, targeting beginning adult learners. The Web-based test, which can be used for self-assessment or placement purposes, was administered to 20 DSGS adult learners of ages 24 to 55 (M = 39.3). The learners filled out a background questionnaire, took the Yes/No test tests, and filled out a feedback questionnaire. The comments provided by the learners about the suitability of the Web-based DSGS vocabulary self-assessment instrument provided concrete feedback towards improvement of the


Introduction
Web-based learning systems [1] have become more and more widespread in secondary and tertiary education worldwide. This is also the case for language learning in particular [2] [3]. In the last 20 years, the Common European Framework of Reference for Languages (CEFR) [4] has had a considerable impact on language education and assessment in Europe. More recently, the CEFR has also found its way into the field of learning sign languages as a second language in tertiary education [5]. As a result of this, sign language instructors are interested in providing new learning opportunities to their students. The concept of proficiency for any language, spoken or signed, is made up of many different components and sub-components as defined in theoretical models like the Communicative Language Ability [6]. When developing new learning opportunities for students in the classroom, it is common to focus on selected sub-components, e.g., vocabulary knowledge. In the second language acqui-sition and assessment literature, vocabulary knowledge has been recognized as a crucial variable contributing to learners' overall language proficiency [7] [8].
The goal of the present paper is to report on the development of a Web-based vocabulary learning and assessment instrument for Swiss German Sign Language (Deutschschweizerische Gebärdensprache, DSGS) that relies on a self-report format (i.e., a Yes/No test) using the open-source software Lime Survey (https://www.limesurvey.org). The test can be used as both a self-assessment instrument and a placement or diagnostic test [8] in the context of sign language vocabulary learning. The paper at hand additionally presents feedback collected from learners following the application of the assessment instrument.
The remainder of this paper is organized as follows: Section 2 provides an overview of previous work, particularly Yes/No vocabulary size tests (Section 2.1) and Web-based instruments for sign language learning and assessment (Section 2.2). Section 3 introduces the DSGS YN test, reporting on its precise setup (Section 3.1), the participants chosen for an initial application of the test (Section 3.2), the testing procedure (Section 3.3), and the feedback questionnaire (Section 3.4). Section 4 reports on the results of the feedback questionnaire designed to gather insights, among other aspects, on the suitability of the Web-based assessment instrument. Section 5 finished with a discussion and conclusion.

2
Literature Review

Yes/No vocabulary size tests
Findings resulting from the application of Yes/No (YN) tests for spoken languages informed the development of the YN test for DSGS [9]. For example, the basic design of YN tests was adopted, consisting of a learner seeing a word and indicating whether he/she knows it [10]. In order to minimize the effect of guessing on the part of the learner, i.e., mitigating the risk of a test taker rating more words as familiar than he/she actually knows [8], [11] added pseudowords (phonological possible forms, e.g., for English foggy becomes wuggy; [12]) words to a YN test. The same approach was also used for the DSGS YN test, with nonsense signs, i.e., phonologically plausible forms of a sign language that bear no meaning in that language by manipulating the phonological parameters of that sign language [13]. These nonsense signs take the role of pseudowords. According to [10], there are no clear guidelines regarding the ratio of real words to pseudowords. For example, the ratios reported in different studies include 30:3 [14], 90:60 [15], 96:32 [16], 60:40 [17], and 40:20 [18]. For the current project, 25 nonsense signs were developed to supplement the existing 95 true test items.

Use of web-based instruments for sign language learning and assessment
In the past, Web-based sign language tests or learning tools were commonly purpose-built for a specific research project [19]. More recently, existing open-source software solutions for sign language learning and assessment have allowed for managing large numbers of videos and streaming larger videos. As a result, learning platforms like Moodle have begun to be used for sign language learning scenarios.

DSGS YN test
The format of a YN test lends itself to administration via a survey software tool, as the test is about investigating whether a learner knows a word/sign, which is equivalent to posing a yes/no question in a survey.
Lime Survey is an open-source software that has been developed for the purpose of conducting surveys, not specifically aiming at learning scenarios as in the case of this study. Since version 2.7, Lime Survey allows for more straightforward embedding of videos in different formats (e.g., .mp4 and webm format) directly through the graphical user interface. For the purpose of the current study, version 2.7. was used. Lime Survey was installed on the servers of the first author's university. The authors were the only persons who had access to the data.
The test implemented in Lime Survey consisted of a welcome message and an introduction in written German and DSGS. Subsequently, each item was presented with the question "Do you know the form and the meaning of the sign?", with "yes" and "no" as possible responses (omitting a response was not possible).
Since presentation of the items followed a uniform scheme, the items were inserted into Lime Survey by populating the underlying database (in this case, a PostgreSQL database), i.e., writing a Python script that automatically produced the corresponding insert statements for each item. The items were presented in a random order. A "continue" button allowed for proceeding to the next item (cf. Figure 1) and a "back" button for returning to the preceding item.
The response data was automatically sent to a secure server for storage. The test was self-administered, but the researcher was present during test taking in case of any technical issues. The results were later imported into SPSS for analysis.

Participants
The DSGS YN test was initially carried out with 20 adult learners of DSGS in 2017. Of the 20 participants, 15 were female and 5 males. The learners were between 24 and 55 (M = 39.3) years old at the time of testing. 19 of the 20 learners were hearing; one had a Cochlear implant but had acquired German as a first language and was also learning DSGS as an adult. The majority of the learners had a spoken language (e.g., a Swiss German dialect or Standard German; n = 18) as their L1. Two participants reported having grown up with two spoken languages. All participants had learned DSGS as adults (range: 18-53 years, M = 35.4).

Procedure
The initial study took place in a quiet room at the first author's workplace. The researcher was present throughout the session. The session, which took approximately one hour, included the following steps for the learners: • Receiving background information on the project, filling out the consent form, reading through the information sheets of the project • Filling out the background questionnaire online on a laptop • Taking the YN test online on a laptop • Taking a second test (a translation test) using a separate assessment instrument, the results of which are reported in [9] • Filling out the feedback questionnaire online on a laptop

Feedback questionnaire
The learners were asked to fill out a short online feedback questionnaire after using the YN test, also implemented in Lime Survey. The goal of this questionnaire was to gather general feedback on the YN test. Studies on test takers' feedback for spoken language tests informed the development of this questionnaire, for example, [24] and [25]. The learners were asked to indicate their agreement with the statements 1 to 4 below on a four-point Likert scale ranging from "strongly disagree" to "strongly agree", and to give one of the four possible responses with "negative", "rather negative", "rather positive", or "positive" to question 5: • The wording of the test instruction is clear. Test takers were given the possibility to leave a written comment. For the present paper, the optional free-text comment box is particularly relevant, as it provided an opportunity for learners to comment on the suitability of the Web-based assessment instrument.

Results of the Feedback Questionnaire
As part of the comment box, participants delivered a number of valuable comments on the format of the Web-based YN test that are capable of informing future Webbased assessment instrument development. The written German comments were translated into English by the first author.
For example, some of the participants left comments related to the suitability of the Web-based self-assessment system: "This self-assessment is an important instrument to gain insights into signs that one does not remember anymore" (ID11). Another participant (ID17) stated: "I missed the immediate objective response as to whether I remembered the sign correctly or not. It would have been good to have a German translation to be able to validate my yes/no decision." Still another participant's (ID23) comment went into the same direction: "I would have preferred to be able to not only indicate whether I know or do not know the sign but also write out the German translation" [and get immediate feedback on it; in square brackets added by the authors]. Two participants did not perceive the Web-based self-assessment vocabulary instruments as very positive. One of these two participants (ID16) stated "I perceived the self-assessment as being more exhausting [than the second test, a translation test which was shorter]", the other participant (ID21) emphasized the importance of the selected items to match the learners' signing skills. He or she found it very frustrating that he/she had to hit the "No" button, indicating he did not know the sign, continuously. One participant (ID13) provided a useful comment related to the technical side of the self-assessment system. He or she stated that "it would be good for the survey to automatically advance once I have selected 'yes' or 'no', i.e., to not have to hit the 'continue' button myself".

Discussion and Conclusion
Tests and assessment instruments can be evaluated from different perspectives, e.g., by collecting evidence of different validity arguments of test evaluation and use or collectively contributing to construct validity [26]. The focus of the present paper is different, it aimed to gather feedback on the Web-based self-assessment format.
Based on the learners' comments, some changes will be implemented into the Lime Survey-based self-assessment system: • Provide the option to see the German translation directly after indicating knowing or not knowing a sign.
• Provide the option of a report after completing the self-assessment entirely, aligned to the levels of the CEFR.
• Check if the number of items can be reduced while retaining sufficient information about the learners' DSGS vocabulary size. • Add information on the level of self-assessment (e.g., A1) in the beginning in order to avoid frustrating experience on the side of the learner.
The novelty of the work reported in this paper consists of using an existing opensource software as a basis for developing and evaluating a DSGS YN test for selfassessment purposes. A limitation of this study is that the sample is rather small and that feedback data on the Web-based format collected from the participants is rather scarce.
In the future, the self-assessment instrument will be used more regularly in DSGS learning contexts at the first author's university with the goal of gathering more information about the usefulness of the YN test as well as gaining deeper insight into its current Web-based format using Lime Survey.