Very Simple and Inexpensive System for Distance Learning with the Use of Chalkboard

Using technology in Covid-19 time's classroom is a given. Many teachers, however, prefer to use chalkboards both because they have difficulty adapting to new methods and because they believe teaching with a chalkboard is more effective. Writing with chalk on a blackboard makes it easier to control the pace of a lecture because it induces to talk while writing, thus giving to the students the time to assimilate new information and to take notes properly. To make it possible for teachers to continue lecturing with this method but in distance learning mode, as required by the anti-Covid-19 provisions, an easy and inexpensive automatic shooting system has been developed, that can be used by schools and universities with minimum effort. To take advantage of the developed software, it is sufficient to have a smartphone (or a webcam) and a PC, without the need for other expensive hardware. The method has been experimented and found to be very effective in our department with a number of colleagues completely relying on it to deliver classes. Keywords—Chalkboard; distance learning; image elaboration


Introduction
The spread of COVID-19 in 2020 caused a very hard situation and, at least until a vaccine is found, the best and only way to contain its effects has been to resort to social distancing measures [1,2] that are strongly affecting every aspect of our daily life, including education at all levels. COVID-19 has forced educational structures (schools, universities, academies) to resort to the use of technological aids for performing lessons [3,4]. To guarantee safety conditions, in fact, the lessons must be held, in many cases, in distance learning mode or, in the best of cases, in blended mode [5,6]. For remote lessons, the use of ready-made presentations to be shown on the PC and of graphic tablets is the most intuitive solution. Although, for example, the PowerPoint tool has caught on, for a few disciplines students still seem to prefer lessons on the blackboard [7] and many teachers do not want to give up the use of chalkboards, both because of the difficulties related to the use of systems with which they are not familiar, and because they consider the blackboard the most effective teaching method. This choice is independent of age or academic qualification [8]. It has been observed that students can learn better and more effectively through traditional face-to-face teaching methods and systems that aim to blend ICT and traditional method have been proposed [9]. Besides, it is one thing to choose to take advantage of new technologies for experimenting new teaching methods but it is quite another thing to be forced, often without technical and methodological training, to employ completely different approach in an emergency. Although in fact the platforms used for distance offer the possibility of working with virtual whiteboards, in order to use them it is necessary to have graphic tablets. These have disadvantages: the cheaper ones require "blind writing", being the teachers able to see only what it is written on the screen; those that allow to write on the screen do not always have good reaction times, so they are a good solution in case teachers have to write down something on readymade slides but become difficult to use when teachers have to do entire lessons by writing; solutions on paper (Bamboo Folio [10] for example) have a limited writing area; the solutions that offer better performance are too expensive and difficult to implement in a limited time as required by the need for immediate response to the pandemic [11,12]. Xiaohua Jin in [13] observes as conclusion of his study that teachers' innovation consciousness needs to be improved.
To help teachers who have difficulty adapting to new technologies but who are currently forced not to give up their preferred teaching method (chalk and blackboard), a software application (Auto-Director) has been developed that is freely available to schools and teaching institution, or even to single teachers, as an aid in shooting and distributing lessons held on the blackboard in real time. Although systems that allow to "digitize" writing on the blackboard (with the use of appropriate sensors) [14] have already been proposed, Auto-Director, having been developed as an emergency solution, aims at providing an extremely affordable and easy to implement solution.
The use of Auto-Director can be integrated into all the most used platforms for distance learning (Microsoft Teams, Zoom, Google Suite), thus allowing to exploit all the facilities of the latter, such as, for example, recording the lectures, to faces the students and talking to them, to interchange blackboard view with other views (programs working on a the PC) by switching the shared window. In such a context, the target was to obtain a "usable" solution, for both streaming or recorded lectures, with the lowest possible requirements in terms of hardware and software for the final user.
In developing the software, the objectives were: 1. Speed of development: The project has been all about solving an immediate problem with limited tools and exploiting every other software/hardware already available. 2. Affordability in terms of required hardware: Although the system can work with cameras or webcams (as long as capable of HD resolution), the cheaper device for shooting is the smartphone (with the capability for HD video recording) because it is expected to be, in most cases, already in possession of the teacher. The same is true for the required PC: the software performs reasonably well on a laptop with Windows 10 and a CPU-MARK of at least 2500. 3. Ease of use by the user: As long as lighting condition in the classroom are reasonably controlled and uniform over time, once some key parameters have been calibrated (by technical staff of the institution where the courses are held), the behav-ior of the system should be reasonably stable and consistent. The software is already configured so that external software, which may also include graphical interfaces for easy parameter modification, can be added without having to intervene on the main software. Some personalization may be required on a "per teacher" basis, in order to account for the fact that the teacher is left handed instead of right handed. In truth, the software, in the form of a Windows executable, can be run from a USB stick that each teacher may carry with himself/herself with his/her custom set up.
In the following sections we will discuss what the software does, what equipment is needed to make it work and we will illustrate how it was used and with what results in our department.

What is Auto Director
The greatest difficulty in shooting and transmitting lessons carried out on the blackboard in real time is related to the need to continuously change the shooting field according to the actions and movements of the teacher. At the same time, it is necessary to provide the possibility of a shooting field that includes the blackboard(s) present in the classroom as a whole. This field of view, however, is too wide for allowing whoever receives the video stream to clearly distinguish what is being written on the blackboard by the teacher. One could send the entire shooting field in high resolution and leave the student with the task of selecting (and zooming in) the region of the blackboard that is of interest at any moment, but this solution has two important contraindications: 1. The weight of an HD video broadcast is generally not sustainable in many situations (transmission side bandwidth limitations and receive side band/hardware limitation).
The student should constantly change the point of view (position and magnification of the region of the screen to be enlarged) and this would seriously interfere with the level of attention and student concentration.
The solution, in a context of unlimited technical and economic means, would be to resort to a camera operator (a "cameraman") and a "director" who constantly guides the cameraman by continually instructing him/her on the region to be framed (pan tilt and zoom instructions). Obviously, "cameraman" and "director" would be the same individual, but it is expected that in almost all cases it would not be possible to afford the cost (personnel time and/or money) required to have, besides the teacher, a second unit of staff to be employed full time for each lesson in progress. Over the last few years, "automatic" control systems have been introduced on the market using professional computer controlled PZT (Pan Zoom Tilt) cameras and algorithms [15]. Sufficiently sophisticated "computer vision" can independently determine the best shot corresponding to the situation recognized on the basis of what takes place on the scene. The cost of these systems is relatively high (from one to several thousand euros for each classroom) and in any case their acquisition and installation cannot occur in a very short time. The high level of sophistication of these systems, which justifies the cost, is mainly due to the fact that no limitation exists to the scene, behavior and clothing of the individuals filmed and the techniques of "Computer vision" are necessary to make decisions based on a scenario complex and only partially controlled.
In the case of a blackboard lesson, however, the situation is different and much simpler as it can be assumed that there is only one person present on the scene (the teacher). Furthermore, we can classify many of the teacher's activities on the blackboard as falling in one of the following cases: 1. The teacher is not present in the shooting area 2. The teacher faces the students and speaks to them 3. The teacher, still facing (at least partially) the students, indicates contents on the blackboard to his/her right or to his/her left 4. The teacher faces the blackboard and writes on the blackboard It is clear that not all possible situations fall into the previous four categories. For example, one can imagine that the teacher is writing on the blackboard with his/her right hand while pointing, with his/her left hand, a portion of the blackboard with contents related to what he is writing at that moment. Taking into account more situations than the ones listed above would have resulted in a much longer development time, and, besides, we judged that the four cases listed above are sufficient for reaching the goal for which the project was started in the first place.

How Auto Director Works
Auto-Director has been developed for Windows 10 (64 bit) assuming it to be the most widespread operating system used in schools and universities. Should it be required, the software can be prepared to work on Linux based systems. Indeed, only free software has been employed so that the software can be freely distributed for non-commercial use. We used the OpenCV library for image processing and MinGW-w64 for development in c/c++ [16,17].
The idea behind Auto-Director is relatively simple: we assume that a Full HD camera is available for recording the entire field of view that will be used by the teacher in talking to the students and writing onto the chalkboard. This Full HD video stream is used by Auto-Director to implement a virtual PZT camera by digital image processing, resulting in a video stream at lower resolution (960x540 pixels) with proper framing of the image (according to an automatic internal "directing" algorithm) so that the content of the lecture can be clearly viewed and understood by the students receiving the output video feed though one of the many applications used for sharing content over the internet.
The working hypothesis, in the development of the directing algorithm, was to assume that anyone of the four situations listed in the previous section can be traced from the sole estimate of the position of the teacher's right and left hand. To make it possible to estimate the position of the teacher's hands, the teacher must wear a green wristband on the right wrist and a red wristband on the left wrist. At the same time, it is necessary to ensure that the hue of all the other colors present on the scene, including the teacher's clothes, are as far away as possible from the hue of the employed wristbands. A possible example of wristbands suitable for operation with Auto-Director (obtained by cutting out colored cardboards) is shown in Figure 1. This element of "unnaturalness", which would certainly be unacceptable in other contexts, is the (small) price to pay to make it possible to create the automatic control system described in these pages.
In the following we will briefly illustrate the behaviour of the system in the different situations and we will indicate the way in which the teacher, in certain situations, can "guide", within certain limits, the automatic directing system.

The teacher is not present in the shooting area
This situation is identified by the fact that the system is unable to recognize the presence of coloured wristbands on the scene. In this case, the system shows the largest possible range. Note that the field shown in this case is the one that corresponds to the complete field of view of the shooting camera. (or smartphone) as shown in Figure 2. The teacher must position the camera in order to frame the maximum field of view that he intends to employ that, as illustrated in Figure 2, typically coincides with the full extension of the blackboard.
Note that the camera always frames and transmits the same shot to the PC. It is the Auto-Director software that creates the illusion of PZT framing by software. Employing full HD (1920x1080) resolution in indeed necessary so that with the preprogrammed highest level digital zoom, the resolution remains sufficient to distinguish what is being shown.
When the teacher is present on the stage, he/she, if facing the students, can induce the system to switch to full-field shooting mode by hiding the wristbands (for example by hiding both hands behind his/her back).

The teacher faces the students and talks to them
This situation is recognized by the system when, on the shooting scene, the green wristband is to the left of the red one and their position is "collected" with respect to the body. In this case AD increases the zoom level trying to keep the teacher centered with respect to the frame, with the lower border of the frame loosely aligned with the position of the wristbands, as in Figure 3. This means that if the teacher moves, AT follows his/her position continually. Note that to conceal the identity of the teacher, the face and his figure have been deliberately blurred in the image shown in Figure 3.  3. The teacher faces the students and talks to them. The zoom level is increased with respect to the case of Fig. 2 and AD tends to maintain teacher centered with respect to the frame, with the lower border of the frame aligned with the position of the wristbands.

The teacher faces the students and indicates something on the blackboard
This situation is recognized by the fact that one hand moves away from the other (typically one hand remains close to the body and the other one is used to indicate). In this case it is assumed that "the hand that is pointing" is the one corresponding to the wristband in higher position between the two. The shot is modified by trying to frame the region of the blackboard indicated by the teacher, following the position of the wristband of the hand that is indicating, as in Figure 4.

The teacher is facing the blackboard and writes on the blackboard
The recognition of this situation is based on the inversion, along the horizontal axis, of the relative position of the wristbands (first red, then green in moving from left to right along the blackboard). This is a somewhat critical situation because the system must be able to identify, even if only for a limited time, both wristbands when the teacher is facing the blackboard. In the system there are precautions to manage the situation in which, while writing on the blackboard, the wristbands (especially the one corresponding to the hand that is not used to write) are temporarily hidden behind the body of the teacher. As far as possible, however, the teacher should be "collaborative" and try to assume occasionally a position in which both wristbands are visible. When writing (with the right hand), the system further increases the zoom level and the frame is centered to the left of the hand so that what has already been written is easily visible ( Figure 5). The system can be informed of the fact that the teacher writes with the left hand (a specific parameter must be changed in the configuration file for this condition) and in this case, obviously, the hand that is being followed is the left one (red wristband) instead of the right one (green wristband).
The level of sophistication of the part of the code that deals with the direction (choice of framing and zoom levels) is, currently, moderate. The estimate of what is happening on the scene is nevertheless obtained by averaging over several successive frames and an internal "voting" system is implemented to establish if and how to switch to a new framing mode. This helps to avoid too many changes occurring in a short period of time since this would disturb the vision. The reactivity of the virtual camera in changing position (pan-tilt movements) can be changed by setting proper parameters in the configuration file. In general, a number of the parameters used to implement the direction algorithms are configurable in order to adapt, within certain limits, to the personal "style" of the individual teacher (if he often turns while writing, if he "gestures" more or less while speaking, etc.). Note that an initial "calibration" step may be required depending on the lighting conditions in the room (it would be advisable to always use artificial light so as to work with constant lighting) and the exact color hue of the wristbands employed if these are "personal" and not standardized.
Preparing the system before the start of the lesson requires some simple steps that are described in the following paragraphs.

Required Equipment (hardware)
Auto-Director is currently available as binary files for Windows 10 (64-bit). A PC or notebook with a CPU-Mark greater than 2500, 4GB of memory (better if 8 GB) is needed.
A tripod may be required to position the high resolution webcam or smartphone. The other required equipment differs depending on whether a webcam or a smartphone is used.
If the teacher uses a webcam, it must be capable of resolution (Full HD). Most available webcams can be easily mounted on a standard photographic tripod and some of them are also equipped with a microphone for audio recording. As it may be necessary to place the camera away from the PC, a good quality USB extension cable with a length of at least 2 meters could be required.
If the teacher intends to use a smartphone, he needs to check that the smartphone is capable of recording videos in Full HD resolution. It could be necessary to buy a special support for positioning the smartphone at the photo stand. Screw type rather than spring type are recommended, as the latter can easily damage the smartphone screen.
Finally, a green and a red wristband, as shown in figure 1 are required. They can be made of cloth or of cardboard. Shiny surfaces must be avoided: a surface that tends to diffuse the light is preferable. Color hues as close as possible to red (RGB = 255,0,0) and green (RGB = 0,255,0) are recommended. The Auto-Director system still has parameters that can be adjusted to compensate for small color deviations.
No additional equipment is required.

Required Equipment (software)
If the teacher intends to use a webcam, no auxiliary software is needed.
If the teacher intends to use a smartphone, he needs to install an App on it to enable the transmission of the video stream from the phone to PC or notebook. For example, with "IP Webcam" [18] the smartphone behaves like an wireless IP webcam, with "DroidCam (for Android)" [19] the smartphone can also acts as USB webcam (proper configuration of the smartphone is generally required to enable data connection through an USB cable).
If the teacher uses a smartphone, the audio source remains the PC (Auto-Director does not manage the audio from the Smartphone). In any case, the audio picked up by the phone used as a camera (and hence positioned away from the blackboard) can hardly be of good quality. Using the PC for audio streaming allows the teacher to use and connect more easily wireless microphones, if required. When the smartphone is used as a IP webcam, a possible drawback is the fact that the audio signal and the videos that are then transmitted (via platforms such as Skype or Teams) do not appear perfectly synchronized: the video stream lags slightly behind the audio stream. Although this "non-synchronization" is in many cases tolerable, it is possible to introduce an equalization by slightly delaying the audio stream. This can easily be achieved by free software to install on the PC such as "Equalizer APO" [20].
For this reason, the USB should be preferred also in the case of a smartphone used as a camera, with the additional advantage that the USB cable would also maintain the smartphone charged (otherwise, due to the relatively large power consumption when the smartphone is used to continuously filming in Full HD, a power bank may be needed).
During live sessions, the teacher can share the video output window provided by Auto-Director in the same way he would share any other content on the PC with any of the many sharing software used for remote teaching/learning. If the teacher desires to record the lesson off line, any software that allows to record a video in an active window (for example NCH suite [21]) can be employed.
No other auxiliary software is needed.

Configuration Parameters
At start up, AD asks to set up the parameters for the video input device. This is done by means of a quite simple GUI (Graphical User Interface) shown in Figure 6. The user must select whether a USB camera is used and, in this case, the internal number assigned to it by the Windows system. Typically, if the USB webcam used for AD is the only one connected to the system, this number is 0, but it can be different in case other video input devices are connected (for instance the internal webcam of a notebook). The best approach, in such cases, is to start the program with increasing numbers starting from 0, until the correct webcam is selected (as can be deduced by the output windows created by AD). In case an IP camera is used, the IP address of the camera must be input. Note that a way to connect a smartphone to the PC as a wireless IP camera can be to activate the "hotspot mobile" option on the Notebook, so that a separated network is used for the HD video feed from the smartphone to the PC and the main network interface of the notebook is used to connect to the internet.
While we have verified that a 720p resolution is in general unsatisfactory for the operation of AD, we have still maintained the option using an HD webcam (and not Full HD, corresponding to the 1080p option) so that an emergency system can be set up with even lower hardware requirements.
The software can be started in test mode, and this is useful because other windows are opened in which the wristbands are seen "flashing" if recognized and also the presence of spurious recognized areas can be detected. Parameters are available that can be changed in order to address the problem of color equalization for the correct recognition of the wristbands on the scene, to specify the size of the framing (zoom level) corresponding to each recognized situation (no wristbands detected, teacher standing and talking etc.), the value of parameters that influence the speed of the pantilt-zoom movements of the virtual camera etc. A few parameters are also used to tailor the computational cost of the elaboration in the case of low performance PCs. Clearly, reducing the elaboration speed has a consequence on the overall quality of the rendering, however there may be situations in which it is mandatory to lighten the load of the CPU in order to allow the PC to run other concurrent applications.
At start up, AD reads a configuration file to obtain the values of the configurable parameters.
In future versions, it will be possible to change some of the parameters while AD is running in order to rapidly verify the effect of the changes. This feature is not implemented for the general user as yet. Backdoors are provided for implementing this functionality, but it must be used by skilled technician only in order to avoid continuous system crashes.

Video Feed Elaboration and Related Parameters
Since AD performs digital zooming on the image, the incoming video feed must be characterized by high resolution so that small portions of the feed can be enlarged and send online with sufficient resolution. Nowadays, almost any non-entry level smartphone has the capability of recording video in FullHD and either via wifi or via USB cable the video feed can be transferred to a PC with sufficient quality and frame rate to allow acceptable streaming quality when AD is used.
As we have mentioned before, a Full-HD video stream cannot generally be sustained by students home connections, so that AD, while selecting the proper field of view and following the actions of the teacher, must reduce the data rate to be sent toward the remote user.
For this reason, the output video window created by AD, that can be shared with the students in remote locations, has a size of 960x540 pixels.
As far as the video feed elaboration is concerned, we set the target of reducing as much as possible the requirements in terms of PC elaboration performances as it cannot be expected that high performances PC are generally available in schools. Moreover, the PC, besides running AD, must also run the software used for remote conferencing. At the same time, the generated frame rate at the output must be sufficiently high (typically larger than 16 FPS) in order to obtain an acceptable user experience. In our experiments, a CPU with a CPU-Mark above 2500 is sufficient to obtain acceptable output video quality. Moreover, as we shall presently discuss, AD has the capabil-ity of automatically taking advantage of high performances hardware, should this be available.
The way in which video feed elaboration is performed within AD is illustrated in Figure 7. The Full HD incoming video feed is sent to the program section "camera operator" that performs digital Pan Zoom and Tilt (PZT) operations on the input feed in order to produce a lower resolution feed, with the same FPS, for the output window to be shared with students. The camera operator receives indications on the desired framing (centre and zoom level) from the program section that analyses the teacher position from the full HD (decimated) stream at the output of the frame decimator block. The analysis of each frame in order to recognize the position of the wristbands is, by far, the task with the highest computational cost. For this reason, the frame rate for analysis is reduced, while still remaining sufficient for following the changes of the scene due to the movements of the teacher.

Methodology and Discussion
The software Auto-Director has been tested and widely used in our Department. Seven classrooms have been equipped with webcams and PC with the software installed. Four of these classrooms are dedicated to lessons in blended mode with students of the first years of the degree courses in Engineering (about one hundred students for each course), three are dedicated to lessons held exclusively at a distance for the course years following the first. The software was developed and made available for the first semester of this academic year, so it has been used since last September. The (five) teachers of the first years that have used Auto-Director teach fundamental subjects (calculus, physics, chemistry), while the other five that have used (in the first semester) teach technical subjects (electronics, mechanics, geotechnics).
The teachers are very satisfied with the method because it offered the possibility to continue teaching as it has always been done, without perceiving any particular difficulty due to the need to respond quickly to the pandemic situation. The use of Auto-Director has been integrated in Microsoft Teams, thus performing all the lectures in streaming and contextually recording them. Regarding the way of movement of each teacher, no particular problems were encountered. In fact, the software is set up in such a way as to "intuit" what the teacher is doing or about to do and is therefore not sensitive to every small movement that would otherwise cause a sort of "madness" of the shooting. For teachers who move in a very frenetic way (one of the monitored cases) it is sufficient to act on the software setting parameters (distances between cuffs, speed and framing zoom) to obtain excellent results. The case was reported by students who, in the first lesson, found exaggerated movements of the shot. Once the appropriate parameters were set, the students declared themselves satisfied with the result.
At this point, the only trick that the all the teachers had to learn was to show both wristbands when they turn to write, so that the system can recognize the writing activity. This may require, for some teachers, to place the left hand (or the right hand for left-handers) in an unnatural position. However, this position should only be held for a brief initial moment. The software, in fact, as soon as it recognizes the writing position, keeps it until a noticeable change of position occurs. The teacher can then continue to write in the most natural position, possibly even hiding the wristband of the hand he/she is not using from the frame.
The students were satisfied. The students did not evidence any differences between the use of blackboard or virtual whiteboard. The advantage of the virtual whiteboard, from their point of view, is that at the end of the lesson the notes are immediately available because it is possible to save everything that is written on a file. However, as it is possible to record the lesson given with the classic blackboard and that students now prefer to learn from videos rather than written texts, the students are still satisfied. Instead, they express a clear preference for the use of the board (virtual or classic) over the use of already prepared slides. This is because there is a greater synchronization between the teacher who speaks and writes and the student who takes notes. With the presentations ready, teachers tend to "run" during the explanations, also skipping calculation steps that could create difficulties for students.
The main problem highlighted by the application of the method arises from the focus of the blackboard. Some students complained of a blurry vision of what was written on the board. The problem is due to the fact that low-cost webcams have been used for the implementation of the method, which are optimized for close-up shots. However, the problem can be solved with some tricks: by setting the webcam focus on fixed mode instead of auto and taking care that there are no objects interposed between the webcam and the blackboard; by avoiding writing with too small characters (but this should already be a trick adopted in large classrooms for 100-300 students); by arranging the framing well so that the blackboards are in the center to avoid edge effects. However, the focus problems were not highlighted in the case in which the teacher used the smartphone instead of the webcam, given the best shooting performance. Most teachers, however, prefer the simplicity of a system already assembled and ready to use rather than having to make the connections with their smartphone and therefore have adopted the precautions listed above.
A "desktop" version of Auto-Director has also been developed which allows to follow the teacher's handwriting on an A2 sheet of paper, shot from above. This solution is ideal for teachers who are forced to give lessons from their office or even from their own home. This second version, that works similarly, but not identically to the former version) is also being used in our Department by teachers of more advanced technical subjects. The students' feedback on the experience provided by Auto-Director on the received end, had been positive as well.

Conclusion
In this paper we wanted to share the distance teaching experience made in our department due to the COVID-19 emergency. In particular, we discussed how we addressed the need to continue teaching with the use of the chalkboard. The software developed and used at our university is available to any teacher who needs a fast and easy to implement solution for continuing to use the blackboard in remote learning environments.