An Analysis of Engineering Educational Stand- ards and Outcomes Achieved by a Robotics Summer Camp Experience

Summer camps can be an effective method of encouraging children and teenagers to learn about technology and stimulate interest in engineering careers. These camps can provide opportunities beyond what a normal school can offer, and achieve specific educational goals that would be difficult to realize in a traditional classroom setting. A robotics summer camp curriculum was developed in accordance with the age specific standards from the National Council of Teachers of Mathematics (NCTM) and the National Science Education Standards (NSES). To link this to the engineering skills that are developed in college, the resultant student experience was evaluated in the context of the engineering outcomes for a four year engineering degree defined by the Accreditation Board of Engineering and Technology (ABET). The robotics summer camp was evaluated for its effectiveness building a foundation and supporting the skills that a student should develop in a college engineering program. Preand posttests were given and scored with a standard rubric. The questions were mapped to the specific ABET outcomes which best align with the goals of the summer camp. The test scores show the degree of improvement in various areas of engineering and problem solving. The resulting data shows the strengths of the summer camp and identifies areas that can be targeted for improvement, to make a larger impact on the attendees.


I. BACKGROUND
The development of curriculum for educational programs can be influenced by a number of sources. The goals of the program could be defined based on the funding source, the community, government departments of education (state and federal level), or by the educator who is designing the activity. Many educational organizations have published guidelines to help educators frame their curriculum in the context of skills and outcomes that a particular lesson or program would hope to achieve based on the subject matter. The recommendations take the form of state and national education standards. Three sources that provide pertinent standards for a robotics summer camp are the National Council of Teachers of Mathematics (NCTM), the National Science Education Standards (NSES), and the Accreditation Board of Engineering and Technology (ABET).
The NCTM is an organization that promotes mathematics education through support to teachers and pedagogical research [1]. They have developed mathematical standards for K-12 education. The NSES was drafted in 1996 by the National Committee on Science Education Standards and Assessment which was commissioned by the National Research Council and provides a set of standards inclusive of K-12 science [2].
ABET is an international nonprofit organization that certifies the quality of educational programs in applied science, computing, engineering, and engineering technology at colleges and universities. ABET developed a list of eleven engineering outcomes that a student should achieve in the course of a four year engineering degree. Even though these are undergraduate program outcomes, it is worthwhile to compare them to a grade school engineering experience, to measure contributions to the preparation of children for entering technical degree programs. The outcomes pertinent to this study relate to gaining STEM knowledge and developing problem solving skills using technology [3].
The robotics summer camps that are the subject of this paper have been held for several years for children in grades 6 through 8 in Western Pennsylvania. The principal intent of these camps is to foster an interest in science, technology, engineering and mathematics (STEM) among elementary and intermediate age students. These grades are targeted so that students who exhibit an interest can make middle school and high school curriculum choices in preparation for a college STEM degree. The nontraditional paradigm used is an inquiry based framework that is as much fun as it is educational. Students, or campers, meet in a semi-formal setting and work in teams to accomplish the tasks identified in the program curriculum. Structure is provided through the use of camp instructors and administrative aids. Campers are provided a Code of Conduct, to which they and their parents must agree. Disrespectful behavior is not tolerated. Students are encouraged to ask questions and to actively engage in the lessons to the extent they are comfortable.
The camp builds sequentially from material developed in preceding lessons. Beginning with an overview, students introduce themselves and complete a pre-camp assessment. The assessment is a two-part questionnaire intended to gauge the students' competencies at the start of camp and to determine the extent of learning during camp. A multiple choice inventory addressing basic math/science knowledge and Lego Mindstorm NXT-G specific skills is administered along with a short form assessment. This short form allows campers to expound more thoroughly on their thoughts with free response questions.
This particular camp was first designed around the Vex Robotics System. But for the past two summers, Vex was PAPER AN ANALYSIS OF ENGINEERING EDUCATIONAL STANDARDS AND OUTCOMES ACHIEVED BY A ROBOTICS SUMMER… replaced by Lego Mindstorm NXT 2.0. The use of the Lego Mindstorm system for the middle school robotics summer camp verses Vex or other robotics systems was made for a variety of reasons.
• Many students are already familiar with Lego assembly methods and schematics, some even have Mindstorm experience. This reduces the learning curve for robotic assembly. • Mindstorm software can be taught in a Labview or C++ based environment providing different levels of complexity, but also experience with software that mimics real engineering tools [4,5,6] and supports ABET engineering outcomes [7]. • The FIRST Lego League Robotics Competition [8] is open to students age 9-16, building interest for middle school students at robotics summer camps could encourage them to participate in or even start a robotics team at their school to compete in this challenge. • LEGO Mindstorm kits require no additional tooling or familiarity/skill with components. The pieces simply snap together and are designed for the target age of the campers (6 th -8 th grade). • Other educational outreach programs have successfully used this platform at this age level [9,10,11,12].
One challenge with a week-long engineering summer camp is the time it takes for the students to learn the technology and be able to use it in a meaningful way. One possible solution is to extend the length of the camp to multiple weeks. A more drawn out method would be to engage the student over the course of a year, meeting on weekends to teach the students basic concepts and give them time and access during the school year to play with the concepts they have learned and complete homework assignments or mini-projects. Then the summer camp becomes a capstone experience for the students to have a block of dedicated time to pull together all of the individual concepts they have learned into a final team project that would be more significant than a project done solely within the confines of one or more weeks of summer camp [13]. Unfortunately this approach requires additional resources, planning, and a much larger time commitment from the instructor and the students. For the summer camp analyzed here, two identical one-week sessions hosting 20 students each were offered, followed by a one week advanced level camp that was available to students from those two initial introductory sessions, prior summers, or students who already had basic exposure to the Lego Mindstorm robotics system and felt prepared for the advanced camp. The focus of the advanced class was the programming and control aspect of robotics. The curriculum and outcome of the advanced summer camp session will not be discussed in this paper.
It has been shown that students can relate and benefit from engineering activities that allow them to be involved in the design and development of toys [14]. Using the Lego Mindstorm kits, students could be led through a similar engineering design process, being given the challenge of creating a new "kit" that makes a particular robot, with step-by-step assembly and programming guides, such as the ones that come with standard Lego sets. The students would prototype and test an original design to complete specific task or set of tasks. Once the robot has demonstrated its ability to do these tasks the students can fully document the assembly, by doing "reverse engineering" taking the robot apart, recording each component and photographing it at various stages of disassembly. Reversing the disassembly process creates the assembly instructions. This could also be a lesson presented in context with the FIRST robotics program, to have the students document their design once it has been finalized for competition.

II. CURRICULUM
The camp curriculum is mapped to a four-day schedule and is taught at the introductory level. In developing the curriculum, it was assumed that participants would have no prior experience with either LEGO construction techniques or with NXT-G/LabView programming. One of the advantages of the LEGO Mindstorms is that it requires no additional tools for assembly, making familiarity with and training in the use of tools unnecessary. While the emphasis of the curriculum is on programming, students are engaged first in spatial relation exercises in preparation for assembly activities using LEGO Mindstorm kits. The ability to read, generate and understand technical drawings to build a model is a prerequisite to the development of a testable platform upon which the success and efficacy of the programming can be assessed. The first day's curriculum can be seen in Appendix A.
Care was taken to map the daily outcomes to accepted, national teaching standards in both mathematics and science. For example, the second activity of the camp involved the representation of a 3-dimensional object on a 2-dimensional sheet of paper. Students were provided instruction on the isometric and orthographic rendering of real objects, including the use of scale. Upon completion, students were asked to draw a randomly selected, but simple object (rectangular prism, cylinder, box, etc.). Once completed, the objects were collected and placed on a table. Next, the renderings were distributed so that each student got one created by another student. Finally, the students had to choose from the collection of all objects that object represented by the drawing they received. This activity addresses both the National Council of Teachers of Mathematics (NCTM) Standard with respect to Geometry and the National Science Education Standards (NSES) requirement with respect to inquiry. Specifically, it targets the desire to have 6-8 th grade students be able to "precisely described, classify, and understand the relationships among types of two and three dimensional objects." It further satisfies the Standard's requirement that students "specify location and describe spatial relationships using coordinate geometry and other representational systems" [15]. This is particularly evident by the students' use of graph paper to aid in their reproduction of the dimensional features of their respective objects. Finally, the exercise addresses the NSES requirement that students use mathematics in all aspects of scientific inquiry [16].
Daily activities for each of the four days are summarized in Appendix B: B. The chart lists the camp's daily activities and maps the specific national standards or engineering outcomes that each activity satisfies.
Camp officially opens after completion of the pre-camp assessment with a brief lecture on the nature of science and engineering, followed by a review of the working definition of a robot. Lecturing is aimed principally at providing a historical perspective on the scientific method and for drawing commonalities between it and the engi-PAPER AN ANALYSIS OF ENGINEERING EDUCATIONAL STANDARDS AND OUTCOMES ACHIEVED BY A ROBOTICS SUMMER… neering process. Students are taught that theses methodologies are a formalized means to minimize trial-and-error approaches to problem solving, thereby enhancing efficiency. For most individuals, this lecturing part of the itinerary is the least liked portion of the camp. It feels most like school. Fortunately, most also recognize the necessity of it and tolerate it to varying degrees. To address this issue of boredom and better fix student attention, we have included a number of video clips. These clips, review the notion of what it is to be a "robot" from the perspective of today's mass media in order to introduce and dispel preconceived misconceptions. We conclude with a working definition of a robot as a non-living entity that has the ability to sense its surroundings and act according to a predetermined plan.
After the video, students are provided a tutorial on the construction and importance of mathematical models. Campers engage in communicating technical specifications by creating and interpreting technical drawings. They are assessed, based on their ability to correctly identify orthographic translations from one perspective to another. Concluding the first day, campers, in pairs or groups of three, are provided assembly instructions for a "Castorbot" along with a box of parts. They are given one hour to complete the assembly, accessing parts from the box as dictated by the assembly manual. Instructions are organized in steps corresponding to major assemblies. Students are asked not to proceed to the second step until completion of the first step is reviewed with one of the instructors. Once approved to proceed, the group or team is allowed to continue construction unabated until finished or having solicited help.
On the second day, the curriculum shifts from mechanical lessons to programming for linkage between hardware and software. Specifically, the students begin the task of learning how to program their robot to complete simple behaviors such as moving forward a specified distance and turning right and left. The NXT-G software development tool used in the camp provides a graphical user interface, based on LabView, that minimizes the time required for students to acquire proficiency in its use. Unlike command-line interfaces, the graphical interface is intuitive and syntactically simple, allowing students to quickly apply it. Concurrent with the instruction of programming simple movements, campers are engaged in critical thinking exercises that compare and contrast the locomotion of their robots with their own bipedal movement. Biological analogs to the wheels, gears, and beams are highlighted. The complexity of human motion is stressed. Campers explore how something with wheels can turn. Most are surprised that turning can be accomplished by either stopping one wheel while rotating the other or by rotating both wheels at different speeds. Once this is understood, programming right, left, point or swing turns becomes second nature. Finally, students are shown how to calibrate the simple behavior of moving forward one wheel rotation and to use the information to write an algebraic expression relating the distance traveled to the number of wheel rotations. Similarly, campers calibrate the degree of body turn as a function of wheel rotation and are taught how this information can be used to calculate the number of wheel rotations required to achieve a desired body turn.

III. ASSESSMENT METHOD
A repeated-measures design was employed as the assessment tool to gage the extent to which students' knowledge of robotics concepts improved. Both a pre-test and a post-test, in the form of a multiple choice assessment (Appendix C) and free response assessment (Appendix D) designed by the creators of the summer camp curriculum were administered to all students. These tests presented the exact same items, in the exact order. The only difference between tests was the day it was given. Campers were advised that they would be completing a pre-and post-camp test and that their scores would not be applied to any ranking or grades. To eliminate students using results from the pre-test to guide their responses to questions on the post-test, scores and correct answers from the former were withheld from the participates. Failure of an individual to turn in either one or both forms resulted in their elimination from the sample set, so only 35 of the 40 participants are included. A paired t-test was used on the results of the individual pre-and post-tests to assess the difference in the means to determine whether the course was effective at fostering learning. Since the camp was run twice consecutively, the results for each week of the camp are analyzed separately, so that we are assessing the effectiveness of each offering of the camp as two separate sample groups.
The short form subjective assessment pre-test with open ended questions was administered as a targeted evaluation of the ABET engineering outcomes. A rubric (Appendix E) was devised prior to the start of the camps on how to score responses on a scale of 1 to 5, with 5 being an excellent answer, and was based on rubrics used for similar studies [17]. The same criteria for administering and assessing the multiple choice questions was also used for this tool.

IV. RESULTS
During the course of the four day summer camp offered in summer of 2012, students were introduced to the basic concept of robotics, learned to understand basic mechanical diagrams and visual programming languages, and apply basic Lego building methods to produce and program robots to solve specific problems. Students were able to succeed at making robots that were able to satisfy the various challenges and in the processes deepened their overall understanding of robotics and increased their technical problem solving skills, as shown by the pre-test and post-test.
The analysis of the pre-and post-test examines the differences in mean scores (M) for each class and also uses the standard deviation (SD) to calculate the threshold for significance using a t value at the appropriate number of degrees of freedom with a 0.05 p value to set the confidence limit. For the multiple choice portion of the test, camp participants during the first week increased their average test score, M = 8.5 points with SD = 12.0. The effect was statistically significant, t(17) = 3.05, p = 0.05. Likewise, the second week showed an increase in average scores, M = 12.9 points with SD = 8.7. Again, the effect was statistically significant, t(16) = 5.76, p = 0.05. So both weeks of the camp showed a significant improvement in test scores for their technical knowledge and problem solving skills. Scores are expected to be higher for the second week due to the familiarity of the instructors with PAPER AN ANALYSIS OF ENGINEERING EDUCATIONAL STANDARDS AND OUTCOMES ACHIEVED BY A ROBOTICS SUMMER… the material and comfort with the course. Figure 1 shows a histogram of test scores for the multiple choice pre-and post-test overall grades with a shift from a pre-test average of 62% to a post-test average of 76%, and a threefold increase in the students scoring 90% or above.
The short answer portion of the survey gave the students an open forum to respond to questions in their own words. The questions were designed to investigate how much the students knew and gained conceptually, versus the problem solving skills tested by the multiple choice portion. Figure 2 shows the breakdown of average scores by question, showing that the students already came into the camp with a fair amount of general knowledge about engineers, robots, and uses of robots in society. They performed well on questions that asked "what" things are, but seemed to struggle with questions asking "why" or "how" about engineering methods or processes. The largest gains are from questions relating to skill based tasks that involve describing a method or process to accomplish a goal with the robot. Reasonable gains are also seen in a couple of the computational problems (5 and 15), one of the hardest conceptual computational problems, translating rotational motion to linear motion, (4) had no significant change.
The short answer survey also showed strong gains from pre-to post-test. Based on a rubric of judging open ended question and answers, the overall understanding demonstrated by their responses shifted the mean score from 62% to 76%. Based on a paired t-test this is a statically significant gain in their understanding of engineering and robotics. The number of students scoring >90% tripled.
By mapping the ABET outcomes to the pre/post test questions, the areas of greatest impact can be measured by aggregating the improvement across all relevant questions relating to each outcome.
A pre-test and post-test score for each outcome was derived using the grading rubric found in Appendix E (examples of sample responses to the open ended questions in Appendix F) and the mapping of ABET outcomes to questions in Appendix G. For each student a pre-and post-test score is calculated for based on the questions that relate to each outcome. All 35 students were evaluated together for the outcomes assessment, so the degrees of freedom for the T test was 34. This data includes questions where students answered on the pre-test but then submitted no answer on the post-test. If this were to be omitted due to unanswered questions, then the difference between test scores would increase by 0.2-0.4 points per category. Even with non-responses factored in, the six outcome areas analyzed show statistically significant improvements based on the open response questions, as seen in Table 1. The 95% confidence level for significant improvement needs a p value below 0.05, and all results were an order of magnitude or more smaller.

V. CONCLUSIONS
This paper presents a methodology for hosting a one week robotics camp for middle school aged students and an assessment method that is based on both national standards for K-12 education but also looking ahead to measure improvement in outcomes defined for collegiate engineering programs. A multiple choice test showed that technical knowledge and problem solving ability pertaining to robotics increased by a statistically significant level.
The open ended questions asked of the students were mapped to a selection of ABET outcomes for a collegiate engineering program and the six areas measured all showed extremely significant improvement. A sample of the curriculum, survey instruments and scoring rubrics are included to show the level of comprehension attained by the students.   • Reason abstractly and quantitatively.

Average % improvement in test scores Average difference in pts between pre/post score t value p-value for Two tail t test, (DF = 34) 95%
• Model with mathematics. • Determine a theme or central idea of a text and analyze in detail its development over the course of the text, including how it emerges and is shaped and refined by specific details; provide an objective summary of the text • Determine the meaning of symbols, key terms, and other domain-specific words and phrases as they are used in a specific scientific or technical context relevant to grades 9-10 texts and topics Assessment Anchor/ Eligible Content HSA/REI.B3 CCRA/SL.1 CCRA/SL.2 • Solve linear equations and inequalities in one variable, including equations with coefficients represented by letters.
• Prepare for and participate effectively in a range of conversations and collaborations with diverse partners, building on others' ideas and expressing their own clearly and persuasively • Integrate and evaluate information presented in diverse media and formats, including visually, quantitatively, and orally. Objectives: The students will be able to: 1.
Construct mobile robot using visual/graphic instructions 4.
Plan/Conduct tests to characterize functional capabilities 5.
Write and integrate graphical programming Materials Course Manual (to be passed out), notebook, pen/pencil, Lego TM Mindstorms NXT 2.

12
Believes that working individually on a project is more beneficial and really does not provide any information about the effectiveness of teamwork and the importance of teamwork and how it relates to engineering.
Demonstrates the benefits of working as a team rather than individually. States a couple examples why it is beneficial and important to work cooperatively as a team. Talks about certain professional and ethical responsibilities dealing with teams.
Understands the importance of teams and teamwork in engineering. Identifies the ability to communicate effectively with a diverse group, distribute workload, master your tasks, and contemporary issues in the real world.

13
Student believes that trial and error is the best way to begin an experiment and offers no insight into any other types of solutions. Response is very general and lacks accuracy.
Presents some alternative ways to starting certain projects or experiments. Examples are very basic and not very descriptive. Briefly mentions the engineering process but does not elaborate beyond that.
Demonstrates knowledge of the engineering process and incorporates the process into how to begin and finish every engineering project. Applies knowledge of mathematics and engineering to illustrate other alternative ways to begin a project that are more accurate and precise.
14 Response is not accurate and the not very clear. Does not answer question fully and uses no concepts dealing with mathematics, science, or engineering.
Explains the situation very generically, with little to no mathematical terminology or engineering thought. Provides a correct way of explaining the question but answer lacks complexity and engineering mindset.
Ability to apply knowledge of mathematics, science, and engineering to adequately state how wheel rotation can determine how far an object has traveled. Uses complex mathematic terms and concepts to mathematically explain the scenario.