: A Way to Expose

— During interaction with learning robots the students are often faced with the challenge of understanding the robot intent and its practical realization. To answer this challenge, we propose a connected environment which integrates the robot, its digital twin and virtual sensors. We implemented a reinforcement learning scenario in which a humanoid robot learns to lift a weight of unknown mass through autonomous trial-and-error search. To expedite the process, trials of the physical robot are substituted by simulations with its digital twin. The optimal parameters of the robot posture for executing the weightlift-ing task, found by analysis of the virtual trials, are transmitted to the robot through internet communication. The approach exposes students to the concepts and technologies of machine learning, parametric design, digital prototyping and simulation, connectivity and internet of things. Pilot implementation of the approach indicates its potential for teaching freshman and HS students, and for teacher education.


Introduction
One of the main objectives of introducing the capability of learning in robot (especially in industrial applications) is to reduce the need for human guidance when performing robot tasks. Driven by training and experience, the learning robot can adapt to the environment and improve its performance. However, there are domains where intensive human interaction with learning robots is an important requirement. During robot programming and debugging there is a need for real-time monitoring and control of robot learning procedure including inputs, outputs and state information [1]. Another example of human guidance is robot learning by demonstration -direct or indirect [2,3]. The interaction can be especially intensive in human-guided training of robot reinforcement learning [4]. In educational robotics the ultimate objective of robot learning implementations is to educate students. Therefore, active participation of students in robot training is essential.
The two machine learning approaches, mainly applied to teach new skills to robots, are imitation learning and reinforcement learning [5,6]. In imitation learning, the robot records and imitates the target movement demonstrated by the instructor. In reinforcement learning (RL), the robot is not directly instructed, but through autonomous trial-and-error search, it determines and records the action which optimizes the performance criterion. The opportunities to teach robots new skills or adapt existing skills to new situations by using the RL approach have been intensively investigated, while one of the main challenges is to reduce the experimentation time and the wear and tear on the robot. A recommended method to cope with this challenge is by means of an empirical predictive model which is autonomously generated and used to guide robot actions [7].
Simulation in robotics is a software tool for design and testing robot behaviors on a virtual robot before implementing them on a real robot. Some of the benefits of using robotics simulations, listed in [8], directly relate to reinforcement learning. In particular, performing robot trials in a virtual environment allows experimental data to be generated faster, more easily, and in any desired quantity, thus significantly speed-up the learning process. Modern computer aided design systems provide means for creating virtual models which accurately resemble the geometric and mechanical characteristics of the real robots.
Cloud robotics is a method to enhance functionality of a robot by using remote computing resources of memory, computational power, and connectivity [9]. In robot learning, connection to the intended cloud platform enables the accumulation, storage, and processing of data of robot trials and other relevant information on the Web server, and transmit the data to the robot. The platform can serve a hub of the Internet of Things (IoT) network, through which robots can share between them the learned skills and communicate with other systems.
The goal of our research project is to propose and explore an approach in which the challenge of implementation of robot learning is used as a thread for teaching the discussed robot intelligence technologies to high school and first-year engineering students. In this approach, the student is assigned to implement a robot task in which the desired behavior cannot be pre-programmed, but has to be learned by the robot. In such a project the student teaches the robot to acquire the skill by implementing a reinforcement learning process supported by simulation modeling and cloud communication.
Our research is an ongoing multi-case study conducted at the Technion Center for Robotics and Digital Technology Education through collaboration with the PTC Israel Office. Participants are 1st and 2nd year students from MIT doing summer internship projects in our lab, high school (HS) students participating in our outreach activities, and Technion students studying technology education. We utilize robots constructed by students using the ROBOTIS Bioloid Premium kit (http://en.robotis.com) and software tools by PTC, namely, the 3D modeling system Creo Parametric, and the IoT platform ThingWorx.
Our project so far has passed three research phases. In the first phase, university students implemented a RL scenario in which a humanoid robot, through a series of trials, learns to adapt its body tilt angle for lifting different weights [10]. In the second phase, a group of high school students, mentored by a faculty staff member and our university students, constructed animal-like robots and implemented different RL scenarios, utilizing the approach tested in the first phase. The focus of this paper is the third phase, in which university students apply reinforcement learning, 3D modeling, and cloud communication in order to implement a scenario in which a humanoid robot learns to manipulate multiple joints to maintain its stability when lifting different weights.

Robot weightlifting task
Pick-and-place manipulations by fixed-base robots are widely explored and studied in industrial robotics. On the other hand, planning basic handling tasks such as weightlifting, to be executed by humanoid robots, is yet an evolving research topic [11,12]. In the weightlifting task, if the mass and size of the weight are known to the robot, then its posture can be controlled in the open loop. The control policy is to prevent the robot from falling down by maintaining its static and dynamic stability [9]. If the weight's mass and size are unknown, the close loop control is needed. Here, the control policy can be determined analytically, based on the dynamic model of the robot and data from force and torque sensors [11]. Rosenstein et al. [13] noted that analytic solutions for humanoid robot weightlifting can be complex. They proposed an alternative approach based on reinforcement learning through trial-and error.
Recently, performing weightlifting tasks by humanoid robots has become a challenge in educational robotics addressed to university and even high school students [14]. Michieletto et al. [15] used a weightlifting task as a challenge of their "Autonomous Robotics" course for MSc students majoring in computer science. The task was implemented trough robot learning from a human demonstrator with the aid of Microsoft Kinect. Weightlifting by a humanoid robot was also posed as a benchmark assignment of the robot competition FIRA HuroCup for university and school students [15,16]. The assignment was posed without reference to robot learning.
Our motivation to explore the educational challenge of humanoid robot weightlifting came from developing a fetch-and-carry robot for the RoboWaiter contest [17] in which we introduced the humanoid league [18,19]. In the contest assignment, the mass, size, and location of the weight were predetermined. Following the contest project, we turned to the new challenge of lifting a weight when its mass is unknown to the humanoid robot. As mentioned in the introduction, in the first phase of our project, undergraduate students constructed and programmed a humanoid robot that coped with the new challenge and learned to lift an unknown weight by a series of trials and errors [10]. The robot was built from the ROBOTIS Bioloid Premium kit and had 18 degrees of freedom, an accelerometer, a Bluetooth communication module, an IR sensor, and a sound sensor. The robot was programmed using RoboPlus software.
The reinforcement learning scenario was as follows: The robot is given an unknown weight while sitting down. The mass of the weight is estimated by measuring the angular velocity of the robot shoulder joints in the way described in [10]. Then, the robot performs weightlifting trials for different values of body tilt angles, each time attempting to stand up from the sitting position. The robot evaluates whether it succeeded or failed the task by determining if it remains standing or has toppled over, based on data provided by its accelerometer [10].
Because of the memory limitation of the robot controller, the robot can store results of a limited number of trials and only until the robot is powered off. Therefore, the empirical data acquired from the robot trials were stored on a local computer. The computer communicated with the robot via a Bluetooth interface supported by a Python script. Based on these data, the computer provided the robot with the tilt angle value suitable for successfully lifting the specific weight.
In the following section we will discuss the way in which the robot weightlifting task has been implemented in our current study.

Development of robot learning mechanisms
The developed robot learning environment is presented in Figure 1. It consists of three components: the robot, the simulator (digital twin) and the cloud (ThingWorx). The constructed robot is essentially the same as in the first stage of our project, but was upgraded by adding grippers to suit barbell lifting. The digital twin is a virtual counterpart of the robot created to test robot functioning in the simulation mode instead of testing the physical robot. The ThingWorx server is connected with the robot through the local computer used as a routing point. ThingWorx also receives and analyzes data from the simulator and sends recommendations for weightlifting posture to the robot upon request.

Construction and calibration of digital twin
The virtual robot was created using the Creo modeling software. We took the computer designed models of parts of the Bioloid premium kit from the ROBOTIS website and imported them to Creo. Using these parts, we assembled the virtual robot in the same order as the construction of the physical robot. Because the models of the robot parts on the website do not have assigned weights, we weighed the parts and added the mass properties to each of the parts in Creo. We also assigned to each joint of the virtual robot the same range of motion as of the corresponding joint of the physical robot.
After assembling the virtual robot, we calibrated the model to have the same balance characteristics as the physical robot. During this step, we modified several parts of the model, such as its motors, where we took into account its uneven distribution of mass. Then, we compared the balance characteristics of the physical robot and its digital twin. We tested the balance of the physical and virtual robots in the same posture shown in Figure 1, for different values of mass of the weight. For the virtual robot, the calculation was made using the "center of gravity analysis" and "sensitivity analysis" features of Creo. The maximal mass that each robot can hold in this posture without losing stability was determined. We calculated that the discrepancy between the physical robot and its digital twin was less than 3%.

Simulation analysis
The objective of the simulation analysis was to optimize reinforced learning of the physical robot. Two possible approaches to such analysis were considered: real-time on-line simulation and batch offline simulation. The latter approach was chosen as more simple and applicable to cloud-based learning. The approach implements massive multi-parametric analysis (virtual testing) of weightlifting by the digital twin to create a "space" of possible solutions and refine it into a "sub-space" of optimal solutions. Then the optimal solutions are stored on the IoT platform and used in on-line communication with the physical robot.
The problem for the simulation analysis was defined as follows: for the weight of a given mass, test the balance of the virtual robot in its various postures over the range of possible bending angles at the hip, knee, and ankle joints. The concept of a "virtual sensor" was utilized -using capabilities of Creo, we have attached a "sensor" to the center of gravity of the digital twin, to analyze the balance of the robot for different combinations of possible angles of the joints. The angles ranged within 100° for the hip and knee joints and within 90° for the ankle joint. We divided the ranges into intervals of 10°. This resulted in 10 angles for the hip joint, 10 angles for the knee joint, and 9 angles for the ankle joint, which means that there were a total of 900 robot postures to analyze for each value of mass of the weight.
The two parameters calculated for each posture of the digital twin were: !, the distance between the center of gravity and the center of the foot, and !, the robot's height. The distance ! is used to evaluate the robot stability, and the height ! is a parameter which characterizes the robot's posture when completing the task. Using "Design Study" capability of Creo, the simulator determined various combinations of angles that allow the robot to lift the given weight. By further analysis, the optimal combination (in which the robot is in balance, ! is maximal and ! is minimal for this !) was found. We note that the number of virtual trials can be reduced by using the Dynamic Analysis capability of Creo.

Cloud management of robot learning data
Each optimal solution, generated through simulation analysis for each mass value, contained three parameters (hip, knee and ankle angles). Database of these optimal solutions has been uploaded to a data table in IoT platform ThingWorx.
We defined a function in ThingWorx which, upon getting the weight value as input, utilizes the data table, and returns the corresponding angle values of the optimal solution. When the physical robot has to lift a weight, it first measures the weight mass, and sends its value to the ThingWorx server. In response, the robot receives the values of the three angles suggested based on the simulation analysis. Then the robot executes the lifting using those values.
To visualize the online communication between the robot and ThingWorx, we created a mashup web page which serves as a dashboard for displaying parameters of the robot weightlifting trial (the weight's mass and the three angles). The mashup is shown in Figure 2.  table in ThingWorx   4 Educational implementation As noted in the introduction, our research project explores an approach to teaching robot intelligence technologies to high school and first-year engineering students by engaging them in teaching robots to learn. The instructional design in the project is implemented by the authors of this paper. A faculty staff member develops instruction in robot construction and programming and conducts courses for school and freshman students.
Two more instructional designers are Technion mechanical engineering graduates working at PTC and pursuing an additional degree in science and technology educa-tion in our faculty. In the research project, which is part of their studies, they develop instructional units on 3D modeling and internet communication in robotics and mentor school and freshman students in these topics. The strategic planning, project coordination and supervision of the instructional designers are made by a faculty member who also guides school and freshman students in pedagogical concepts relevant to robot learning. Consultancy regarding the modeling and communication technologies and software systems was given by PTC.
Three first-year students majoring in mechanical engineering at MIT have performed robot learning projects in our lab, two students in 2015 and one in 2016. The learning activities in the 2015 project are discussed in [11]. In the 2016 project the student constructed and programmed the robot; created and calibrated the digital twin; programmed the balance analysis procedure for weightlifting; implemented virtual trials and transmitted the results to the cloud data table; presented her work at PTC and faculty seminars; and wrote a project report.
We implement the proposed approach by teaching intelligent technologies through outreach courses to students of a high school in Haifa that has recently established an engineering systems program. In the second stage of our research project, during the 2015-2016 academic year, we conducted pilot courses in 3D modeling and robotics to 11th graders. In the modeling course, the students learned to design and analyze computer models of robots using Creo.
In the robotics course, they constructed various robots using the Bioloid kit and implemented different scenarios of reinforcement learning. The students applied the knowledge acquired in our courses in the project developed for participation in the FIRST Robotics Competition. The school team participated in the 2016 international competition in St. Louis and won the Rookie All Star Award for "implementing the mission of FIRST to inspire students to learn more about science and technology."

Conclusion
The approach proposed in our research extends the scope of educational robotics, which traditionally focuses on practices with preprogramed robots. Results of our research indicate that the challenge of developing learning robots can engage novice engineering students in experiential learning of innovative concepts and technologies, such as machine learning, parametric design, digital prototyping and simulation, connectivity and internet of things. We found that those concepts and technologies are within the grasp of understanding of freshman and HS students.
The current stage of our research focuses on enriching student interaction with the learning robot using means of augmented reality (AR). We are developing a connected AR application (using ThingWorx Studio) which will allow to visually observe real-time data measured by physical and virtual sensors -directly overlaid over the physical robot. The student will interact with the robot through the AR application to take the control of robot motion. In this case the student will get an opportunity to view the motion as performed by the digital twin -before it is executed by the physical robot. At this stage we will practically implement our approach in an outreach course and we anticipate that the evaluation of this experience will lead to the development of strategies for learning with learning robots. Michael Reitman is VP of Engineering and R&D at PTC Inc. His professional responsibilities include development and delivery of engineering software as part of PTC Creo product family, and research projects related to new technologies combining CAD with IoT and AR.