Longitudinal Learning Outcomes from Engineering-Specific Adaptions of Hybrid Online Undergraduate Instruction

Hybrid online delivery, which is also referred to as mixed-mode delivery, utilizes a combination of online content and traditional face-to-face methods which may benefit significantly from specific delivery adaptations for undergraduate engi-neering curricula. Herein, a novel eight-step phased instructional flow with several targeted adaptations is used to accommodate the mixed-mode delivery of STEM curricula is evaluated with a longitudinal study of students afforded these adapta-tions versus those without them. This STEM Blended Delivery Protocol (STEM-BDP) emphasizes scaffolding of analytical procedures along with hands-on prob-lem solving throughout online and face-to-face components equally. Two high enrollment course case studies utilizing STEM-BDP are examined herein, includ-ing an Electrical and Computer Engineering required core undergraduate course and a Mechanical and Aerospace Engineering undergraduate course. The details of the STEM-BDP delivery strategies, learning activities, and student perceptions surveys are presented. Student-resolution longitudinal analysis within a controlled study using blinded evaluation indicates that over a five-year period, failure rates have decreased by 63% among students undergoing STEM-BDP while control and alternatives have not demonstrated similar improvements within the same degree programs. 
Given increasing enrollments within STEM curricula, it is sought to overcome challenges of conventional lecture-only delivery in high-enrollment courses.


Introduction
'Blended learning,' 'mixed-mode delivery,' 'flipped classroom,' and 'hybrid online and face-to-face instruction' are terms which are frequently used to refer to the interwoven conveyance of electronically-delivered and in-class learning modalities [1] [14] [16] [30] [45] [49]. Whereas there is considerable information in the literature on various flipped classroom approaches, this paper begins by identifying some open issues with respect to the use of blended delivery more specifically within STEM. Those are used to identify how the described approach fits into the larger body of work and the subsequent sections describe in detail what is novel about this approach. Case studies are elicited from both a required core undergraduate Electrical and Computer Engineering course (EEL3801: Computer Organization) and a core Mechanical Engineering course (EML4142: Heat Transfer I) in which the techniques of STEM-BDP were applied for multiple semesters.
Conventional instructional delivery relying upon live lecture, homework assignments, and synchronous in-class exams remains as the predominant delivery mode within undergraduate Science, Technology, Engineering, and Mathematics (STEM) programs [8]. Conventional lecture can offer advantages of simplicity of a low-tech broadcast mechanism for large class sizes and matches the expectation of some students to be lectured on the material, thus maintaining their status quo bias [39]. However, as enrollments grow, students in large classes may tend to lose concentration due to the crowded environment, and thus may hesitate to ask questions during class. Specifically, innovations are sought to advance the STEM Creativity, Design, and Soft (CDS) skills, whereas CDS skills are vital and differential factors for career success [28]. As identified in Table 1, conventional live lecture may not fully-engage these skills. Lecture delivery relies predominately on teacher-driven transfer of knowledge within a regimented classroom setting which offers limited opportunities to engage broader skills central to STEM and codified in ABET accreditation criteria. This has motivated research to sustain content engagement [6] and overcome live lecture's challenges at engaging critical thinking and soft skills within its classroom setting [28]. As a means to enable mastery learning, it is sought to utilize instructional technologies with alternative modes of delivery embracing active learning [20] and other pathways identified herein. At the other extreme, Massive Open Online Courses (MOOCs) exclusively utilize online delivery methods with a high reliance on self-paced learning via an asynchronous delivery mechanism and often at the expense of reduced engagement [32].
Strengths of MOOCs include very high instructor productivity, which can reach thousands of students and some peer-assessment is feasible albeit via asynchronous discussion mechanisms. While MOOCs excel at productivity, they allow few verifiable assessment mechanisms and the limitations of communication opportunities with instructors constrain both scalability and effectiveness for engineering curricula [36] [7] [31]. Challenges of MOOCs for teaching STEM include reduced retention [24], few opportunities for active engagement, and challenges with assessment arising from the lack of authentication wherein online-only grading may be difficult to realize meaningful assessments [17] [22].
Mixed-Mode delivery hybridizes these to utilize online knowledge acquisition followed by classroom-based instructional activities. In comparison to MOOCs or live lecture, the methods of mixed-delivery offer increased flexibility and scalability, while retaining valuable personal interaction and authentic peer learning opportunities. Videos improve comprehension and student enjoyment [43], and the in-class time is reallocated to active learning and productive activities [15] [29]. This can assist with struggles with engagement of millennials acclimated to learning from on-demand interactions, such as Internet searches and YouTube videos [6]. Meanwhile, a Blended delivery model enables compartmentalizing of the knowledge delivery from its individualized tailoring [15]. Additionally, blended curricula have been transforming education from a knowledge-based-transfer method to an active and collaborative model. Although a comprehensive meta-analysis study of the U.S. Department of Education concludes "on average, students in online learning conditions performed modestly better than those receiving face-to-face instruction" [34], it simultaneously imposes additional delivery challenges such as personalization of student interaction, soft skills development, and student plagiarism. The metanalysis study in [49] looked at the blended learning impacts on achievement of undergraduates and found a small summary effect relative to live lecture alone. Blended learning has been successful in many disciplines outside of engineering as a means of combining the desirable attributes of both modes in stages of an online component and face-to-face instruction.
However, relative to other disciplines, the adoption of blended delivery has lagged in STEM especially in engineering disciplines, which can be due in-part to some additional demands of their curricula. Some challenges codified in the literature that have proven challenging are addressed herein with specific interventions include: 1. students may be unprepared when they arrive to conduct the face-to-face component [29], 2. homework must be tailored to be effective [29], and 3. students may lack appropriate feedback [16] [19] [43].
Each of these is addressed herein with the pedagogical and/or instructional technology advances described in the subsequent sections. The overall objective of the manuscript is to explore how Mixed-Mode delivery can be adapted in several aspects to accommodate STEM curricula and to discuss the results of those adaptations on core undergraduate courses at a large state university. Herein, an adaptation of mixed-mode delivery is formalized as a multi-stage phased delivery approach called the STEM Blended Delivery Protocol (STEM-BDP). STEP-BDP places an increased emphasis on scaffolding procedures in the online component and active learning mechanisms in face-to-face components of the delivery. Results will be presented to provide the evidence of applicability of STEM-BDP to large-enrollment Electrical/Computer Engineering and Mechanical Engineering courses, associated challenges, tools, and suggestions for success.

2
Challenges facing blended delivery of STEM curricula

Elements in present in most blended delivery courses
Blended learning continues to be reported as a key issue in teaching and learning on EDUCAUSE Learning Initiative's annual survey of the higher education community [18]. In a 20-year longitudinal study at the institutional level [14] found that the University of Central Florida (UCF) experienced a consistent trend of blended courses outperforming fully online and traditional face-to-face modalities in terms of student success, withdrawal, and satisfaction [14]. The Blended Learning Toolkit developed by UCF and the American Association of State Colleges and Universities (AASCU) provides a set of guiding principles for blended course development including a focus on outcomes, student-student and student-instructor interactions, and setting clear student expectations, to name a few (UCF & AASCU). In a systematic review of the literature [4] found "The design of blended learning environments brings with it four key challenges: (1) incorporating flexibility, (2) stimulating interaction, (3) facilitating students' learning processes, and (4) fostering an affective learning climate" (p. 1).
Quality education is extremely difficult to define and measure, but several organizations have developed systems, tools, or rubrics to help guide designers [14]. One such tool is the OLC OSCQR Course Design Review Scorecard developed by Open SUNY [41], which is incorporated into the larger Online Learning Consortium's Quality Scorecard for Blended Learning programs [40]. The OLC OSCQR Course Design Review Scorecard promotes specific design elements related to course overview and information, course technology and tools, design and layout, content and activities, interaction, and assessment and feedback [40].
Independent of academic discipline, one of the greatest challenges of blended learning course design is balancing the online and face-to-face components so that they complement each other and finding the right blend that unifies the two modalities in a cohesive delivery of instruction [21] [23]. Ref. [21] developed a "comfort" model that explains how organization, communication, and support may be used to reduce students' vulnerabilities in a blended environment so that they are more likely to succeed. While organization, active learning, and interaction are key elements of good course design for any discipline, [25] highlighted that they are particularly important in the STEM fields when they reported "We show that a highly structured course design, based on daily and weekly practice with problem-solving, data analysis, and other higher-order cognitive skills, improved the performance of all students in a collegelevel introductory biology class and reduced the achievement gap between disadvantaged and nondisadvantaged students-without increased expenditures".

Need to convey complex systems in STEM curricula
Often in the STEM fields, instructors are challenged by the need to explain complex and intricate material. In the classroom, it is important to annotate static content electronically while explaining a concept or stepping through a problem. Similarly, when delivering content online, static text or images may not be sufficient. Animations, screen capture recordings, and narrated interactive notations may be used to explain complex concepts or procedures. Furthermore, traditional classroom teaching is often limited to information transfer, thus limiting students' engagement with the content, particularly in larger class sizes [27]. When information transfer is moved to the online environment, students can engage with the content at their own pace, which then frees up in-class time for more instructor interaction and problem-solving activities. For example, instructor generated videos allow instructors to provide their own explanation of complex topics just as they would have in lecture, but by providing them online students have the added benefit of pausing, rewinding, and replaying the initial delivery [5] [26]. Then classroom time may be used more efficiently for clarifying complex concepts and implementing active learning strategies, which are widely supported in STEM education [2] [20] [33] [42]. Engineering education is largely problem-based and project oriented [13] [35] [38] and student engagement is often attributed to success [37].

Diminished feasibility of online discussion groups
Another common challenge that blended learning may improve is building the students' sense of community [44]. The same challenge is amplified in the STEM fields where students have reported that collaborating with peers is a common success strategy [5]. Online discussion forums may be used to foster student-student interaction. While Ref. [47] found that students who participated in structured online discussions in a computer science course reported more positive attitudes than their peers who participated in unstructured discussion forums, others have experienced mixed results. For example, in a five-year study of blended learning courses in computer and information sciences, Ref. [12] found that instructors who used discussion forums to encourage student conversations around a topic did not see the results they had hoped for, but over time they adopted other forms of interaction strategies with better results. In a review of 29 studies, Ref. [3] found that social networking tools can be used to promote interpersonal communication for sharing and discussing ideas while synchronous cloud computing tools may be used to engage students in active learning experiences. The traditional online discussion feature offered in most learning management systems allows for student-to-student interaction but often falls short in providing an effective environment for collaborative problem solving.

2.4
Towards attaining an optimal modality blending for STEM curricula Ref. [23] talked about finding the "right blend" that maximize the affordances of each modality to meet the contextual student needs. Ultimately, the challenge is strategically connecting the online and face-to-face components in a comprehensive way that maximizes the benefits of each modality while providing a cohesive student experience. One approach that links the two is online content delivery followed by in-class concept clarification and practice. A successful blended design also requires a balanced assessment strategy within the integration of online and face-to-face modalities. For example, if students are expected to prepare online prior to each class meeting, the online activities should be assessed as well as the in-class participation so that class time does not slip back into traditional lecture to compensate for poor student preparation. Given these challenges, the authors explored the following guiding questions: 1. Which technology-enhanced learning methods can address these challenges effectively? 2. What is an effective way to properly structure a blended course design for high enrollment engineering courses?
This exploration has led to a set of vetted best practices for blended instruction in STEM courses similar to these two case studies which are identified in detail herein.

STEM blended delivery protocol (STEM-BDP)
An overview of the delivery mechanisms utilized in STEM-BDP is shown in Figure  1. The three main steps of this delivery method are Online Components, Face-to-Face Components and Assessment Components, respectively. The Assessment Component covers content of both the Online Component and the Face-to-Face Component. There is also overlap between them. Namely, two Face-to-Face activities which are the Motivational Quiz and the GLASS (Group Learning and Assessment at Significant Scale) Digitally-Mediated Team Learning Activity. These occur in-class using online participation mechanisms as described below within the context of the weekly sequence of activities listed as depicted in Figure 2.

Fig. 2.
Sequencing of activities for Blended Delivery whereby roughly one hour of activities numbered 1) and 2) occur online weekly and two hours occur face-to-face weekly for activities 3), 4), and 5).

Activity 1: Online knowledge acquisition
As consistent with mixed-mode delivery, each course module begins with an online activity to facilitate the knowledge acquisition phase. Consider a typical Engineering course which is enrolled as 4 credits, namely "4(3,3)" credit hour format whereby there are 3 hours of instruction and one 3-hour laboratory session each week, which follow an eight-activity sequence in STEM-BDP. First, students conduct approximately 1-hour of knowledge acquisition online, which substitutes for one of the three hours of classroom meetings. STEM-BDP advocates for the first pass of knowledge acquisition to occur outside of the classroom through fortified video content with dynamic highlighting, callouts, electronic pen, hotlinks and online activities as illustrated in detail in Section 4. Slides of the video content are also provided verbatim that match those used in the video, which are made available as a .pdf file. Students are assigned to annotate them with questions while viewing the fortified video to ask during face-to-face meeting as detailed below.

Activity 2: Online mechanisms to engage problem-based learning (PBL)
To reinforce the material presented in the fortified videos, problem-based learning is engaged next. The three modalities evaluated include Assigned Homework, Automated Systems, and the Study Set approach. Assigned homework is a conventional problem set which is collected and graded manually, but subject to collaboration, authentication, and integrity challenges. Automated PBL systems include online problem solving through publisher web-based systems such as McGraw Hill Connect, Adaptive Learning Systems such as RealizeIt!, or other Intelligent Tutoring System. For instance, homework assignments in EML4142 are delivered through McGraw-Hill Connect, which allows students to perform at their own pace during a specified window with unlimited attempts. Students can also revisit assignments throughout the semester to further reinforce their mastery of concepts. These unsupervised preliminary formative assignments help students acquire foundational knowledge while requiring minimal workload from faculty since these assignments are automatically graded. The third option, which is a hallmark novel activity in STEM-BDP, is the use of Study Sets in lieu of homework. Study Sets consist of five to seven worked problems relating to the content of each module. Each problem provides a clear statement of givens and soughts along with a detailed solution. Students will then obtain credit for demonstrating the skill via a lockdown proctored biweekly quiz/exam.

Activity 3: In-class individual motivational quiz
When students meet for the face-to-face component, they receive a quiz in the first 5 minutes of class. This fosters accountability to complete the online component prior to class. The authors and others using STEM-BDP at their large state university utilize Individual Motivational Quizzes to afford extra credit of 1 point on the upcoming exam which is out of 100 points. This helps students to be positively motivated through auto graded quizzes disbursed via the LMS, or by iClickers.

Activity 4: Face-to-face question-and-answer
After completing the Motivational Quiz, a Question-and-Answer session based on annotations of pdf slides is conducted. The instructor allows students to ask any questions for 40 minutes which reduces the visitation load during office hours. Namely, students' concerns are addressed via the use of a broadcast mode to address common questions, as described in detail in Section 4.

Activity 5: In-class problem solving of selected study set questions
After answering questions led by student inquiries, the instructor solves some archetypical Study Set questions in real-time to impart authentic problem-solving experience during face-to-face class-time. Supportive instructional technologies such as electronic pen are vital to annotate the previously disbursed problems and solutions, while solving them from scratch. Several examples are illustrated in Section 4 of this paper.

Activity 6: Virtualized active learning
Sixth, active learning is engaged via a Team Challenge problem during the last 40 minutes of each 2-hour class. Students are assigned automatically to virtual teams randomly via the LMS to solve Team Challenge questions together which are problembased learning. The virtual collaboration tools allow students to participate in teams insitu without requiring special furniture or moving chairs. Color-coding and Most Valuable Peer strategies have been developed by the authors to attain scalable, traceable, autograded quizzes for large enrollment of STEM curricula.

Activity 7: Proctored digitized quizzes and exams
Basing the course points on the proctored assessment avoids integrity vulnerabilities in classes with online components. It uses lockdown proctored biweekly quiz/exam which avoids integrity vulnerabilities common to online delivery methods. Since multiple choice can be restrictive, students' hand-written scratch worksheets composed during assessment are scanned-in. This is further explained herein within the Proctored Assessment Component in Section 6 of this paper.

Activity 8: Score clarification to foster metacognition
Score Clarification is a technique that motivates learners in a quest for partial credit to explain the problem-solving flow that they used in their formative assessment submissions from scanned-in scratch sheets. These elicit an explanation of the solution in their own words with first-line remediation by student tutors, with student follow-up to the instructor. This is further explained herein within the Proctored Assessment Component in Section 6 of this paper.

Online components
Online components evidently play a significant role in blended delivery. This section presents the method the authors have developed and applied to two pilot courses EEL3801: Computer Organization and EML4142: Heat Transfer 1, which span two disciplines of Computer Science and Mechanical Engineering, respectively. Multiple anonymous surveys have been administered each semester in both courses to collect student perceptions of the mixed-mode delivery mode. Over the years, the authors have continuously refined the method based on student feedback and put forth the practice which was widely praised by students and regarded as effective.

4.1
Course home page on LMS Figure 3 shows the course "Home" page on Canvas LMS.

Fig. 3.
Course "home" page on LMS featuring "Quick start" etc. in EML4142 As the default page students see while logging into the course, the Home page features the following components:  Figure 4, helps students to overview and organize course content from the start of the semester. It also clarifies course expectations by listing important hints students may follow for the semester. 3. "Facebook" links to the course Facebook page created by the instructors. The purpose is to set up a platform where students feel welcome and invited to share ideas and ask questions about the course. Students' feedback has indicated that they are more at ease posting on Facebook than on the LMS discussion board. 4. "Feedback on Performance" provides a histogram of scores, plus additional posttesting assistance, after each assessment, so students can be aware of their own performance relative to the class average as a whole. This will be elaborated in the Assessment Component. 5. "Testing Reference Sheet" links the equation sheets for each test of the semester, which will be provided in the testing center during quizzes and exams. Students are instructed to use the equation sheets to solve assigned problems so that they can become familiar with them while finding information quickly during tests. In the meantime, students also practice a needed career skill of referring to data sheets.

Content authoring and importance of video and lightboard-based technical material
The idea of blended delivery is to utilize substantial online activities to substitute for reduced classroom meetings. The quality of online lectures essentially decides the success of the blended delivery. While exploring effective approaches to conduct online activities in their courses in both Mechanical Engineering and Computer Sciences, the authors learnt that some common practice in other disciplines may not apply to STEM curriculum. For instance, face-to-face lectures may be sufficiently replaced by reading materials and discussion assignments in some other disciplines, but video lectures may still be highly beneficial in STEM due to the complex nature of subjects considering that videos may communicate with more clarity and impact than written words alone. Ideally, the shortest amount of time to explain the concept can be advantageous. Minivideos of less than 5 minutes are usually recommended in other disciplines. However short videos are unlikely to be adequate to cover engineering contents that are equivalent to face-to-face lecturers. Based on the authors' observation over the years, longer videos of half to an hour seem to be acceptable for engineering students in general as well. The authors learned that a well-defined clarification between online and face-toface activities in structure help students set expectations and minimize confusion. In the two pilot courses, video lectures focus on concept and theoretical knowledge, and face-to-face classes are dedicated to problem solving, with an emphasis on collaborative problem solving. For rather challenging topics, videos were also created on extra practice questions as supplemental resources.
Lightboard lecture videos. Figure 5 shows a snapshot of a lecturer video in EML4142 which was recorded using Lightboard technology and post-edited by Cam-tasia®. Lightboard is a piece of transparent glass board illustrated with LED lights, which allows presenters to face towards the camera while writing on the board at the same time. The course introduction video shown in Figure 5 was well received by students. A quality introduction video shows that the instructor is prepared and provide a welcome class environment. Lightboard allows students to view both the instructor and the content shown on board simultaneously which makes it a more engaging experience and hence yields agreeable course introductory videos. Figure 5 shows a screenshot of a video for extra practice questions where the instructor works through the problem to thoroughly explain the process. Lightboard also suits well for recording problem solving videos as it allows easy writing with colorful fluorescent marker that glows brightly on the board. Moreover, at the author's institution, the lightboard is facilitated by the Faculty Multimedia Center with all relevant devices such as camera and microphone ready to use, and hence requires zero setup work from the instructor. The instructor just needs to make an appointment, walk in, and start or stop the recording by pressing one button, which makes Lightboard an efficient tool for making short videos that requires more handwriting than PowerPoint slides.
Screencast Videos. Besides Lightboard, the authors in both pilot courses created the majority of their lecture videos using screencasting, which is a digital video and audio recording of what occurs on a presenter's computer screen. Screencasts can be made with a number of software products available, ranging from free downloadable programs with limited features to fee-based products offering more advanced options. The authors have used a rather affordable and user-friendly software "Camtasia" produced by TechSmith for recording screen and editing videos and they find it rather effective.
Comparing screencast and lightboard videos, screencast allows displaying more content on each screen since writing on a board with a marker naturally results in large fonts and the size of the lightboard is very limited. Frequent change of screen may disturb the lecture flow and negatively affect viewer experience and learning effectiveness. In contrast, screencast allows pre-prepared printed text and images as shown in Figure 6 and with the development of tablet technology nowadays, writing on a Tablet with a quality stylus could feel akin to their paper-and-pen counterpart. For most problem-based STEM content, high quality screencast videos perceived as most useful by students depended not only on thorough planning of the recorded content, but upon careful post-editing with callouts. Of course, any awkward pauses, misspoken words, or other unwanted portions should be removed to craft a focused video that uses students' time efficiently and sustains their retention. Furthermore, it is important to stress that rich annotations created by instructors during pre-and post-editing can help grab students' attention, significantly enhance video quality, result in deep impact, and make it a more fun experience. As shown in Figure 6, various annotation formats can be provided depending on the topic, including electronic pen annotation of equation derivation or problem being solved during recording, and text and graphic callouts, such as "text balloons" that provide hints, links, notes or typed-out questions. were males and 15% (n=44) were females. Approximately 57% of the participants were White (n=164), 21% (n=67) Hispanic. Seniors accounted for the majority 93% (n=271) of the participants. Almost all students were either Mechanical Engineering comprising 67% (n=193) or Aerospace Engineering majors comprising 33% (n=96). 87%, 252 out of 290 Agreed or Strongly Agreed that "Highlights/Callouts/Electronic annotations assist in following technical material on video presentations", while 10% being Neutral and only 3% of the students Disagreed. This data suggested that various annotations should be added during post-production despite the fact that a significant amount of work is required for this job. 79% of students Agreed or Strongly Agreed that "I can learn adequately (or better) at my own pace via video content", while 17% were Neutral and only 4% Disagreed. This suggested that course videos which were assigned for students to watch outside of classroom provides effective learning experience. It is interesting to note that only 49% of students Agreed or Strongly Agreed that "Having a small inset window of the professor speaking during the video is beneficial for my learning" while 29% being neutral, which suggests that instructors may not be necessarily turn on camera during recording. Creating a video without the camera on could make recording remarkably less stressful and demanding, since instructors can instead focus more on course content being delivered instead of their appearance.

4.3
Course modules Figure 7 shows a snapshot of a typical course module on a weekly basis in LMS canvas for EML4142, which clearly lists all the activities for that week. For instance, student can readily access the video for the module along with corresponding partiallyfilled class notes for that specific lecture video and can the practice questions for the in-class lecture for that week and important course reminders. Weekly modules organized in this consistent format keep students well informed about course progress and facilitate their success.

Project and learning resources
Online learning resources are in as shown in Figure 8 tensively used for design projects so the projects can be kept updated and draw student interests. For instance, in EEL3801, the instructor developed multiple design projects for students to conduct via in-class extra-credit team challenge problems. The link to online tutorials and other project information is provided for students to refer to. Face-to-face components vital to STEM-BDP

Motivational quiz submitted as individual work
As mentioned in Section 3, individual motivational quizzes are utilized to encourage students to complete the online component prior to face-to-face classes. For instance, the EEL3801 class meets weekly for 2 hours of face-to-face instruction, which begins with a 5-minute long motivational quiz delivered by the LMS using the students' own laptop or tablet PCs. Clones of question are used to decrease the impact of information sharing among students whereas lockdown browsers are not feasible. Moreover, questions asked are those not easily obtained via search engines, but rather refer to artifacts developed within the video content that is specific to the video itself.
In EML4142, the motivational quiz is delivered by iClicker Classroom Response System, which does not allow internet access and inherently avoids the needs to create clones of questions for faculty. In the same midterm anonymous survey as mentioned in Section 4.2.3, 82%, 267 out of 326 students, Agreed or Strongly Agreed that theiClicker quizzes offered motivation for them to watch the course videos prior to attending classes. Starting from Spring 2020, the instructor has transited the motivational quizzes polling system from iClicker to Canvas Quizzes to accommodate the demand of pure remote classes due to COVID19. Based on another anonymous survey conducted in Spring 2021 in EML4142, 71%, 228 out of 326 students, Agreed or Strongly Agreed that the canvas quizzes motivated them to watch the course videos prior to the synchronized classes via Zoom, which suggested that internet-based quizzes can also provide effective motivation for students to complete the viewing task.

Virtualized active learning with team challenge problems
Active learning can be especially effective within STEM curricula. It is ubiquitous in the case of three hours per week labs as separate meetings, and fundamental to building STEM practical skills from the theory covered in the course. With the availability of mixed-mode which moves lecturing to video, it is also possible to add more active learning exercises during the face-to-face component. Moreover, active learning is highly-synergistic with mixed-mode delivery because it is complimentary to online activities. Active learning during in-class meeting time can be vital for STEM problem solving, design, and team-based activities, which in the past the student had to undertake on their own. In fact, accreditation requirements for these skills have had little room in the curriculum for "functioning on multi-disciplinary teams" except for senior design capstone projects, so until arrival mixed-mode we have had little spare time nor opportunity to add it to the classroom. Now, the challenge becomes which pedagogies and technologies can best assist to deliver active learning effectively within face-toface time of mixed-mode courses. STEM-BDP attempts to address that need. To thrive, the foci need to include scalability within existing instructor and physical resources while achieving student traceability and authentic interaction mechanisms sufficient to guide and assist the activity. In the case of large enrollment STEM courses, this mandates observability by the instructor despite large class sizes and limited GTA availability. Here, automation is essential to make active learning feasible in UCF classrooms. This includes some level of auto-grading and good integration with the LMS.
The novelty of STEM-BDP is to apply Virtualized Active Learning weekly in the case of EEL3801 or biweekly in the case of EML4142. Namely, the authors developed the Group Learning and Assessment at Significant Scale (GLASS) approach to increase the scalability and efficacy of student design teams during group sessions [9]. GLASS allows the instructor to manage multiple design teams to conduct a weekly Challenge Problem during in-class time. Once the design team concurs that their results are complete, they submit their answers to the Learning Management System (LMS) for auto-grading and score-recording in the grade book. Credit is earned by correctly answering each designated question subpart, which provides partial credit. Throughout the team design activity, the instructor monitors the assignment progress online in real-time, including windows for each design team showing a solution draft as it is constructed, and providing feedback via each group's designated chat channel. LMS statistics are available in real-time for the autograded answer of the first design team having a correct solution, dubbed the Pioneer Group, which receives a bonus after its group leader presents their solution to the class.
Two more important factors are the instructor and student perspectives. The authors profoundly enjoy having a 30-minute active learning exercise at the end of every faceto-face meeting weekly. It's a rewarding experience to answer their questions before students get too far off track. The authors have three years of Qualtrics data that 80% of those students agree that the active learning exercises were worthwhile, more effective use of class time, and experience that is more enjoyable.
However, instructors shall not overlook that it is demanding work to prepare authentic, new, and perpetually-fresh active learning exercises and then to conduct them weekly. Some automation could manage the complexity of running that weekly while maintaining decent Student Perception of Instruction (SPI), handling the grading load, and sustaining the significant coordination and team grading challenges which may be overly burdensome to many faculty. The authors found that a balanced and effective protocol for active learning in the classroom is to have a maximum of 30 minutes at the end of each face-to-face time. Such can maximize adoption rates by faculty and student satisfaction greatly compared to more than 30 minutes per week of class time. Also, instructors may increase the amount and frequency of active learning gradually from semester to semester to alleviate the preparation and delivery load.
Simultaneously, as shown in Figure 9, the instructor is able to view the whiteboard windows of each design team, which can be displayed on a private screen or broadcast to the entire room. Here, the instructor can provide real-time guidance for a group via their chat channel, and then moving on to observe and assist the next group. Thus, GLASS makes problem-based learning tractable for groups of design teams in F2F sessions, while helping to coordinate and automate the logistic mechanisms, as well as providing new means for observing and guiding learning. Finally, the selected Pioneer Group is invited to present and defend their design to the rest of the class, while earning bonus credit for its group members. This further engages the technical communication soft-skills of the presenting design team and critical thinking skills of the other design teams, who comprise the audience. Overall, GLASS assists the instructor by increasing the observability of the solution process, providing instructional technology to guide learning while it is occurring, and providing traceability of student interactions that are valuable for after-action review to refine the content or pace of the course, and for review with individual students. After completion of the design team activity, an optional post-class activity to elicit follow-up at significant scale is afforded to students through an opportunity to create a discussion post or video blog, in order to elaborate on technical aspects outside of F2F time.

Proctored assessment component
This component utilizes the college-level Evaluation and Proficiency Center (EPC) which is depicted in Figure 10 [8] [10] [11] [46]. The EPC targets value-based instructional harvesting using a novel cost-saving educational infrastructure for both students and faculty. It recasts GTA and faculty roles of labor-intensive tasks towards high-gain learning activities such as: • exam preparation and secure exam delivery, • GTA-guided content tutoring, and • Score Clarification which is a post-test remediation based on scanned-in scratch sheets. Thus, the well-cited "Testing Effect" engages learners with retrieval practice through closed-book proctored quizzes interwoven with rapid tutored remediation. It pools together instructional and human resources (GTAs) from 29 courses across seven degree programs to achieve higher learning impact at reduced cost, via rapid student feedback and detailed statistics for instructors to tune their delivery. It has achieved learning benefits as depicted in Figure 11. It realizes new efficiencies of paperless delivery of 20,000+ exams using auto-grading, followed by 2,500+ tutoring sessions via existing GTA resources which are freed from grading to facilitate increased enrollments. Figure  11 shows student perceptions of EPC in the intervention-applied course EEL3801: Computer Organization which is a required undergraduate course for majors in electrical engineering and computer engineering at the University of Central Florida. The study was conducted via Qualtrics and open for a two-week duration as an IRBexempted educational study. The majority of respondents were positive, i.e. responded strongly agreed or agreed regarding the question if they deemed that the interventions applied were beneficial for their learning. It is seen their self-assessment of increased their understanding of the concepts at 60% favorable and nearly two-thirds indicated that GTA guided and self-paced access to exam results enhanced their comprehension of material. The majority agreed that EPC-based delivery was beneficial, e.g. 90% deemed that Study Sets followed by a computerized quiz in the EPC were more effective than traditional homework. Additionally, for STEM-BDP Activity 8: Score Clarification, 81% of respondents assessed the efficacy of Score Clarification to be favorable in the post-survey at the end of the course. Score Clarification is a cornerstone of posttest review in STEM-BDP that self-motivates students via partial credit to explain the problem-solving flow they used on scanned-in handwritten scratch worksheets with the pooled GTA tutors and the instructor's office hours gained. Thus, substantiating an improvement in efficacy while also raising efficiency. These preliminary results were encouraging to continue the intervention for multiple studies and obtain additional quantitative outcomes impact on learning via the longitudinal study described in figure  11. In the case study courses, the significant assessments including quizzes, midterm exams, and the final exam were delivered via the LMS Canvas in the EPC. Test Proctors in the EPC provide a turnkey service in a secure environment to prevent cheating/Googling solutions using IP restriction, camera/phone checks, and lockdown browsers. Various question type such as Multiple Choice, Multiple Answer, Multiple Dropdown, Formula Format, and Incremental Solution assessments were adapted to the assessment design [8]. The proctored formative and summative tests contribute to 76% of course grade. The authors carried out a crossover study that randomly-partitioned all enrolled students in a class into control and intervention cohorts to examine the effectiveness of computer-based assessment relative to paper-based assessment. It was found that wellformed and well-delivered CBAs can determine scores differing as little as 0.6% compared to paper-based assessment. This strong consistency demonstrated that CBA could result in scoring comparable to PBA and thus validated the feasibility of CBA [46]. Moreover, if the paper-based grading time which was eliminated is then reallocated for tutoring and Score Clarification, then higher learning outcomes than paper-based assessment are attainable without additional instructor resource. DeMara et al. discussed strategies they developed while applying computer-based assessment in a large enrollment engineering course [11]. Due to space constraints in this manuscript, the reader is referred to those references for supporting details.

Results
To evaluate the learning outcome and effectiveness of STEM-BDP, student-resolution longitudinal data was collected and analyzed. Specifically, the UCF Institutional Knowledge Management (IKM) office collected and analyzed the Drop-Failure-Withdrawal (DFW) rate in EEL4768: Computer Architecture which requires either of the co-listed pre-requisite courses EEL3801: Computer Organization or CDA3103: Computer Organization. Both EEL3801 and CDA3103 use the same textbook and cover identical material. However, the STEM-BDP approach was adopted by the authors has been utilized progressively since 2015, while in CDA3103 conventional classroom teaching continued to be conducted. Hence, among all students who enrolled in EEL4768, some students had completed the EEL3801 pre-requisite course using STEM-BDP methods, and the others had completed the CDA3103 pre-requisite course without the methods of STEM-BDP. Therefore, the group of students who took CDA3103 as a pre-requisite are considered as a control cohort. Meanwhile, the group of students who took EEL3801 as a pre-requisite comprise the intervention cohort. Furthermore, EEL4768 was delivered identically to both cohorts, and during all years taught by completely different instructor who had taught EEL3801 and CDA3103. Therefore, the only intervention involved is whether students were exposed to STEM-BDP or not, and their performance in EEL4768 was assessed by an impartial instructor who was not part of the control nor intervention delivery mechanisms. Figure 12 shows the DFW rates in EEL4768 by students' prerequisite course compared to corresponding overall trends across the college and university. Firstly, it is seen that EEL4768 is a challenging course by the metric of the proportion of students who need to repeat or withdraw, relative to mean rates in the College of Engineering and Computer Science, as well as the university on the whole. It can be seen that the DFW rates have decreased by 63% for the students undergoing STEM-BDP in the intervention cohort, while the DFW rates decreased by 27% for the control cohort when tracked longitudinally. In absence of other factors, the significant drop in DFW rates is attributed to the STEM-BDP the intervention group who carried with them more usable skills longitudinally as independently assessed in the follow-on course. Comparing to the overall DFW rates in the entire university, it can also be seen that the DFW rates of the intervention cohort decreased from 2.3-fold above the UCF mean value to 0.1-fold below the UCF mean value, which indicates that better learning outcomes have been received via the STEM-BDP approach and that it is possible to transform even challenging courses by improving the delivery and learning mechanisms via STEM-BDP. The UCF Institutional Knowledge Management (IKM) office also collected and studied students' average grade over the past four years in EML4143: Heat Transfer 2, which requires EML4142 as a prerequisite. As shown in Figure 13, the students who took EML4142 with the author who adopts the STEM-BDP approach received consistently higher average grade relative to the students who took EML4142 without STEM-BDP. These results from the Mechanical and Aerospace Engineering department also corroborate benefit of STEM-BDP methods on students' learning outcomes.  Figure 14 shows the overall DFW rate EEL4768 and another follow-on course in EEL4742: Embedded Systems where STEM-BDP was adopted in contrast to EEE3307: Electronics I where conventional delivery where its pre-requisite was supplemented starting in 2015 with an additional white-board-only problem solving in a concomitant fashion to address its comparably high DFW rate. It is be seen that in both courses with STEM-BDP, the DFW rates are reduced approximately by one quarter overall. Meanwhile, in the course without STEM-BDP even adding costly additional white-boardonly problem solving of two additional hours weekly, the DFW rate still increased by roughly one quarter over the same observational period within the same degree program of comparable course. This is also supportive of higher learning outcomes per resources utilized, i.e., instructor hours and classroom space, can result from the STEM-BDP approach.

Fig. 14.
Above: DFW rate in EEL4742 and EEL4768 with STEM-BDP. Below: EEE3307 is similarly advanced course in same degree program to without STEM-BDP but additional "chalk-and-talk" sessions added instead to its pre-requisite chain.
To gather student perceptions of STEM-BDP, anonymous surveys were administered both mid-semester and upon exit of EEL-3801: Computer Organization and EML-4142: Heat Transfer I courses. These surveys provide detailed information regarding student's view towards STEM-BDP. Additionally, we specify a part for student's comments on this particular blended delivery at the end of the survey where we receive a large number of positive feedback responses. Throughout these semesters using STEM-BDP, we have updated several aspects of the initial version of this method. Student's comments and survey results of each semester have been considered intensively to update this method in a way to make it more student-friendly and practical.
For instance, survey questions asked at the end of EEL-3801 in Fall 2018 semester are shown in Figure 15. It shows the results for the 99 respondents out of the 126 students who were enrolled. According to these results, the majority of the students have a positive outlook towards different phases of STEM-BDP. As shown in Figure 15 (b), 72% of the students wished that more courses offered Mixed-Mode delivery options besides lecture-only format. Results indicated that 75% of students Agreed or Strongly Agreed that "My ability to apply engineering skills, design components, and function on multidisciplinary teams has been increased more so than via traditional lecture-based format", while only 5% Disagreed. Similarly, in Figure 15 (d), 85% of the students Strongly Agreed/Agreed that electronically-mediated groups can be beneficial in large enrollment classes. This number is especially encouraging as no student disagreed with the statement. Similar results from EML4142 were obtained with a larger enrollment. These positive feedbacks show the importance of updating the teaching methods with respect to the current digitalized learning environments. The use of videos and different complimentary features, allows students to have more fun and get more enthusiastic thorough the entire semester. This method has exhibited higher interest for learning in students and consequently higher achievements compared to the other methods. We have observations for STEM-BDP: 1. After the initial active learning activity which may be considered a training or induction step, then Digitally-Mediated Team Learning exhibited somewhat higher achievement than Paper-Based Team Learning. 2. In all cases, EPC alleviated over 3 hours per week of non-learning gain tasks including photocopying, grading, paper distribution/collection, and gradebook entry/correction. 3. DMTL exhibits lower average time spent on answering the questions than PBTL.
4. On each quiz session, students can see/guess their score right after submitting their answers and do not have to wait until their papers are graded. This can help them to further discuss the answers with each other, which leads to a better learning process.

Conclusion
STEM-BDP provides STEM-specific tailoring of mixed-mode delivery with a special emphasis on scaffolding of analytical procedures in the online component and active learning in face-to-face component. Within the online component, fortified video delivery, Study Sets, and student annotations are emphasized. Within the face-to-face component of the delivery, motivational quizzes at the start of class plus virtualized active learning in the last 30 minutes are emphasized along with a traditional questionand-answer session and solving of worked examples. STEM-BDP delivery strategies, learning activities, and student perceptions surveys have been overwhelmingly favorable from both instructors and students.
As with any technology-enhanced delivery, time and effort is required to conduct the initial conversion. In the case of STEM-BDP, modularizing and splitting the online and F2F roles, drafting course weekly schedule, and creating website layout for the entire course may take a solid week of work. The time required to convert each module's content varies by topic, but screencasting slides with minimal edits/retakes can be completed in a couple of days per module. However, the most useful features such as callouts, links, highlights, and animations bring that number to one week or more. Motivational quizzes may be composed quickly in under an hour each. Active learning with GLASS may take a day initially to create the problems and solutions.
The authors' institution offers a course release to convert a traditional face-to-face class to mixed-mode to facilitate above efforts. Digitized exams for an entire course can be quite time consuming, so the authors' institution offers another course release to do so, and in some case the publisher's test banks provide a useful start. At the Authors' university, it was found that with two semester course release to provide the faculty with sufficient time to conduct the transformation, then conversion could be ready to deliver after that. So, a two semester course release can roughly quantify the minimum expectation as more updates will be made during the offering and subsequent semesters. Assessment digitization and mixed-mode delivery could occur in separate semesters.
According to anonymous surveys which were administered both mid-semester and upon exit of courses, the majority of the students have a positive feedback regarding various phases of STEM-BDP. Also, Student-resolution longitudinal analysis within a controlled study was conducted using blinded evaluation to evaluate the learning outcome and effectiveness of STEM-BDP, which indicated that the DFW rates have decreased by 63% for the students undergoing STEM-BDP in the intervention cohort, while the DFW rates decreased by 27% for the control cohort when tracked longitudinally. Furthermore, Students' mean course grades over the past four years in another large enrollment course were analyzed, which indicates that students who adopts the STEM-BDP approach received consistently higher average grade relative to the students without STEM-BDP.
With regards to inclusion of virtualized active learning, research was extended with an NSF grant for a 3-day Workshop in Digitally-Mediated Team Learning (https://www.digital-learning-teams.com/) in 2019 by the authors. It is significant to note that in utilizing virtualized active learning as a weekly activity in STEM-BDP, two important factors are the instructor and student perspectives. Instructors report enjoying having a 30-minute active learning exercise at the end of every face-to-face meeting. It is rewarding for instructors to be able resolve their students' questions before they get too far off track while helping them solve the problem at-hand. Students overwhelmingly agree that the active learning exercises were worthwhile, conducted more effective use of class time, and were even fun. Nonetheless, it is some additional work to prepare authentic, new, and perpetually-fresh active learning exercises. Some automation could manage the complexity of running that weekly while handling the grading load and sustaining the significant coordination and team grading challenges which may otherwise burden faculty. These are being addressed as on-going and future work. Wendy Howard is the Program Director of the Pegasus Innovation Lab (iLab) at the University of Central Florida, which is an incubator of experimental projects focused on digital learning innovations that can be developed and refined through rapid prototyping and then promoted throughout the university to maximize collective impact on student success at scale. With over twenty years of experience in both instructional design and teaching, her current research is focused on faculty development, collaborative online learning and internationalizing the curriculum through technology.