Changing Culture in Robotics Classroom | NSF DRK12 Project 1418199

CCRC R&D Timeline

What are the major goals of the project?

The major goals of this project are to develop tools that answer the following questions:

  • What kinds of resources are useful for motivating and preparing teachers/students to teach/learn computational thinking and practices thinking through robotics education?
  • Where do teachers struggle most in teaching CS principles and what kinds of supports are needed to address these weaknesses?
  • Can virtual environments be used to significantly increase access to CS? And if yes, what level of computational thinking and practices can be taught and how can this be advanced?

2014-15 Development

Updated Three Different Robotic Curricula

The team added opportunities to use virtual simulation software with over 40 virtual programming challenges to the following curriculum, which is hosted online for free at:

Robot Programming Competition Simulations


The Beltway game is designed to be solved using only programming. The game enables teachers to foreground the following big ideas in CS principles: algorithms, abstraction, repurposing code, and creativity. It is designed in a way that encourages the development of functions, repurpose code, and pass parameters.

In order to make the game solvable in an autonomous only format (i.e., using code rather than human control), game developers took advantage of Unity's physics engine to help students solve the following mechanically difficult challenges in the game:

  • Grabbing blocks - the blocks are automatically grabbed when the robot's gripper is programmed to close and the gripper is in close proximity of the block
  • Stacking blocks - the blocks automatically stack when the robot's gripper is opened and is in close proximity above another block
  • Accumulated error - Accumulated error is a major problem with educational robots. If your robot is off one or two degrees on a turn or is a couple of millimeters short on a distance, that error compounds with each movement. The team added a programmable belt around the outside of the game that enables the programmer to re-establish the robots position in a known position regardless of the angle that you enter the belt from.

The Teacher's Guide for Beltway is found here:
Download Teacher's Guide for VEX IQ Highrise Beltway

2015 Robotic Virtual Summer of Learning Course


The course was a certification course that students can take using either the LEGO or VEX hardware platform and use LEGO's NXT-Graphical, LEGO's EV3-Graphical, ROBOTC Graphical, or ROBOTC Text programming languages. The course was highly Scaffolded, starting with robot math, simple movement, and an introduction to sensors and then covers conditionals, loops, functions, and passing parameters. The summer course offered an opportunity to both test our new lessons as well as new technology

Integrated a Mapped Badged Pathway into One Family of Robotic Activities

The mapped badged pathway that we are using to evaluate students' understanding continues to mature. As players play any of the games or complete challenges within the curriculum they are awarded badges that are now coordinated by underlying learning content, rather than a simple collection of stickers to earn. The system includes multiple levels of badges: activity badges, skill badges, uber skill badges, and certifications.

Activity Badges

The lowest level of badge. Each activity in the game or curriculum has been evaluated according to the computational principle that it teaches (i.e. sequencing, conditionals, structures). When students log into the system and complete various activities in the curriculum they are automatically awarded a digital badge that has values based on the complexity of the activity (i.e. 10 points, 15 points, etc.). All stakeholders can track students' progress

Skill Badges

The second level of badge in our system. Students earn skill badges by accruing the requisite number of points to earn that skill badge. There are multiple levels of skill badges: bronze - the student earns the points, silver - the student earns the points and passes a knowledge assessment test, gold - the student earns the points, passes the knowledge assessment test, and receives a teacher endorsement.

Uber Skill Badges

In order to qualify for an uber skill badge students have to earn multiple types of skill badges, pass an assessment test, and receive teacher endorsement (i.e. Introduction to Computational Thinking Badge)


In order to qualify to take the certification test students must earn the Uber Skill badge and receive teacher endorsement. The certification test includes both performance (you have to program things) and a knowledge assessment. The team is focusing on the VEX IQ technology. Innovation First and the REC Foundation have shown significant support for the project.

2015-16 Development

Developed Two Intermediate Level Robotic Curricula

The following games were developed by a combined effort of the Robotics Academy and Robomatter Incorporated. The games were promoted by the Doolittle Foundation and the Robotics Education and Competition Foundation.

Robot Programming Competition Simulations

The following games were developed by a combined effort of the Robotics Academy and Robomatter Incorporated. The games were promoted by the Doolittle Foundation and the Robotics Education and Competition Foundation.

2016 Summer of Learning Course


This Summer of Learning was both virtual and physical. The virtual portion was offered through The Computer Science STEM Network and the physical portion was offered at North Allegheny School District and included around 100 students. In this experiment we were interested in studying the motivational effects of our badging to certification technologies as well as the completion rate of the certification with live counselors verse online counselors.

Robotics Course Login Procedure for Data Collection

Teachers are required to create an account on the Computer Science STEM Network (CS2N), a Carnegie Mellon secure server. The teacher will follow the directions found at: to setup their online class roster. If they need help setting up the class roster, then they can contact CMRA using the help tab on the side of the screen.

Additional Teaching Resources Developed

To see the new additional resources the project has developed go to the Resources link at this website.

2016 Modified CS2N to Enable Better Access for Students and Teachers

For example, the installation process was greatly simplified to allow efficient downloading of programming software and multiple virtual worlds through one installer. In addition, the process of adding students into a course was greatly simplified. Teachers setting up CS2N courses are now able to upload an excel spreadsheet with student demographics on it when they create their classes.

Research Findings

We received commitments from two large school districts and one small district last fall. We provided training and resources for those districts since 2014 and conducted first round testing of our products with over 2000 students since the project began.

Validate Assumptions About Needs in Informal Robotics

We conducted interviews with the adult leaders, mentors, and instructors from robotics teams, community organizations with robotics programs, and local schools offering robotics as a class in the Pittsburgh area. In total, 35 interviews were completed. Of the 35 interviews, 23 were conducted with FIRST-related team coaches (12 FLL, 6 FRC and 5 FTC), 6 with VEX-related team coaches (4 VEX and 2 VEX IQ), and 10 were conducted with robotics teachers or local organizations not specifically focused on a specific competition. In the interview, participants were asked a series of questions in four subject areas: specifics of the program, approach, team structure/breakdown, and program goals. In the specifics category, the adults were asked about meeting time(s)/duration, the dropout rate and what motivates the students to join. In the approach section, they were asked about any curriculums put in place and whether or not there was a mentorship program. The team structure/breakdown questions pertained to what kind of program (robotics team/class), competitions preparation/number attended, level of importance for the team to place well, influences, demographics, academics, role of adults, underlying structure of team/class, if programming is used and if so, what language, and how the students who are involved in programming are supported. As to the program goals, the adults were asked about the specific learning goals for the program, whether or not learning how to program was important, what the students can expect to get out of the program, and what the adults wanted them to take away from being involved. Additionally, if the program was a robotics class in a school, they were asked whether or not it was a required class and if there was a minimum GPA. All programs were asked about academic levels (team members/students above/below average achievement, learning support, on the Autism Spectrum, ADHD), with a fair amount of respondents having students on the Autism Spectrum involved in their program.

In addition to the adult interviews, student survey data was collected at eight FIRST and VEX robotics competitions. In total, 502 surveys were completed by members of the various robotics teams in attendance. Of the completed surveys, from the FIRST-related competitions, 155 were FLL, 123 FRC, and 76 were FTC. From the VEX-related robotics events, 27 surveys were VEX and 43 were VEX IQ. Additionally, there were 78 surveys collected from a FLL/VEX hybrid event. In the survey, students were asked about their feelings on STEM coursework, robotics, team cooperation, experience in robotics competitions, participation in competition preparation, programming experience and future plans.

At seven of the eight robotics competitions attended, 139 robotic programs were collected from various teams. The programs were later evaluated in three subject areas: logical sophistication, code organization and additional features.

At the largest studied competition (FLL), we also collected data on collaboration quality to understand how this aspect of such a team oriented context influenced student engagement. The dataset included scores for actual performance (i.e., table scores) and scores judged during the competition (core values, project, and robot design). All 61 teams were being judged on these categories, with different groups of judges evaluated each team and their performance. Each team received three scores for core values, project, and robot design categories, with the scores for these three categories ranging from 1 (beginning) to 4 (exemplary). Additionally, we added a new measurement: each team was assessed in terms of collaboration quality while working on a hypothetical challenge. The minimum score for collaboration quality was 1 (minimal) to 3 (substantial).

Develop and Validate Curriculum Assessment Instruments

Two alternative forms of a multiple-choice assessment were developed to measure computational thinking. The assessments are embedded in scenarios and assess performance in areas such as: use of mathematics in algorithms, iteration, developing abstractions, selection using Boolean logic, sequencing, combining algorithms, generalizing algorithms, working with arrays, and documentation. The assessment consisted of 17 items and had acceptable internal reliability for such a short measure of so many complex constructs (alpha=.67).

A third form was created to allow for use in longitudinal studies across years, and an equating study was conducted.

A revised assessment was developed in year 2 based on emerging standards for computational thinking that goes beyond programming competencies.

Assess Impact of CCRC Curriculum Materials on Student Computational Thinking

Pre and post-tests were administered at five local area schools. Data was also collected via log files of relative amount of use of the CCRC materials.

Pre and post tests were also administered to teachers participating in weeklong summer PD workshops at CMU.

Significant Results

Regarding the nature of informal robotics competitions, most adult mentors claim to be very hands-off in regard to their approach to teaching programming. However, most organizations also say that it is important for the children to have some level of exposure to programming, but are not required to continue/further their understanding if they choose not to do so or are uninterested. Many adult mentors expected the students to gain a better understanding of programming, STEM topics, approaches to problem solving, and develop more confidence in their abilities to solve more complex problems. They also wanted the students to be able to better collaborate on a team, achieve a sense of accomplishment, and to form stronger relationships with their peers while simultaneously coming to better understand the role of robotics in society.

In follow-up interviews with some of the programs, all interviewees exhibited a pattern of describing their work in terms of educational relevance, suggesting that they see their work as most centrally educational in nature. However, high school programs tend to see their work as career-oriented (engineering and technician work, and soft skills associated with employment), while the elementary school program was more interested in building foundational thinking skills. This aligns with differences in team/classroom structures – high school programs divided labor along “departmental” lines that designated programming as a specialization opted-in to by only a small subset of team members, while the elementary school classroom used a peer-partners structure in which every student engaged with the programming tasks.

A final emerging theme from the interviews which may prove useful to our efforts is based in the framing that teams used to describe the relationship between programming and robotics – some programs saw programming as a way to enhance machines, while others saw the machines as physical embodiments to make coding tangible.

Our analysis of collaboration quality and robotics competition performance data focused on how well the collaboration quality scores predict team performance across different categories. Multiple regression results indicated that the level of collaboration quality among team members is significantly associated with the team performance in the robotics tournament across the board, including the teams’ programming performance, whether measured by official judge scores at the competition or by our own programming sophistication measure applied to obtained competition code.

Using participant-level survey data collected from multiple competition sites that spanned different age groups and robotics platforms, we ran a series of regression analyses to determine general trends in programming interest, while controlling for other likely interest correlates, like math, science, and general robotics. We further investigated whether effects were differential by age and/or gender. We also attempted to examine the effects of the type of programming involved in the competition (autonomous-only or mixed autonomous/teleoperator control), but our sample included only middle school autonomous and high school mixed-control competitions, so we were unable to differentiate whether certain effects were due to age, or due to competition type. Overall, we found that interest levels in CS are higher in middle school autonomous programming competitions than in high school mixed-control competitions, even when controlling for math, science, and general robotics interest. Boys and girls had similar levels of interest in programming in the middle school autonomous competitions, but programming interest levels for both were lower in the high school mixed-control competitions; furthermore, there was a significant differential effect of gender, such that girls’ interest rates declined to a greater extent between the two.

Turning to our evaluations of the CCRC curricular units being implemented at local schools, we begin with the logfile data. We found that these schools (who were implementing materials during technology education classrooms during one quarter of the school year) were generally only able to complete a small subset of the provided materials, basically the units emphasizing proportional reasoning (more mathematics than CS) and the units emphasizing basic movement (the earliest aspects of computational thinking). In future years, we are encouraging the schools to either start earlier in the year or distribute the larger curriculum across school years.

Despite not having completed the more advanced computational thinking materials, students at the middle school level showed substantial gains in computational thinking t(129)=6.0, p<.001, and effects were found on approximately one third of the items in the computational thinking assessment. Classrooms at some of the schools did not implement even the basic movement unit and showed no gains, ruling out any repeated testing effects as the source of pre-post gains in the focal test group.

In the second year of testing, students at the middle/early high school levels also showed significant pre-post gains in computational thinking, and again the strongest effects were found for those students completing more of the units.

However, we also found that many teachers were not able to get students through more than half of the units. Based on interviews and classroom observations, a number of causes of this slow pace were identified. For example, there were a few intermediate challenge activities that occupied many days of student work, and such activities could be simplified while still keeping the core objectives. Some of the units involved many robotic-centric components that did not introduce new computational thinking constructs and could be simplified. Also, the early modules were too simple/structured for some classes leading to engagement issues, so teachers inserted many fun extra challenges that then occupied a large amount of class time.

What is the impact on the development of the principal discipline(s) of the project?

In our testing, we are seeing positive results with our Expedition Atlantis Math Game, which covers proportional reasoning and teaches kids to program their robots using numbers. Robomatter continues to make significant improvements to the simulation environment, which makes it easier for teachers to use the technology to foreground mathematics. We have also seen significant gains in computational thinking through completion of the introductory curriculum.

We have not conducted a formal study on the use of virtual worlds by competition robot teams, but we have seen a significant jump in downloads of the VEX and FTC competition games and from that we can hypothesize that more competition teams are using the virtual environment to begin to develop code for their competition robots.

Additional Key Outcomes or Other Achievements

We have received strong positive feedback from all teachers involved in the project, and evidence of significant learning gains. A prior experiment showed that we could significantly reduce the time that it takes to teach students how to program robots by using Robot Virtual World simulation software, and the more recent data showed that robust learning gains were obtained whether physical or virtual robots were used. The Robotics Academy has modified its teacher professional development program by introducing the simulation software the first day of training. Teachers are finding that it is easier for them to iterate in the virtual environment until they get working code and then load the code to a physical robot and test it in the physical environment. We are conducting additional surveys with all teachers in our PD courses and will publish the results of our surveys this fall.

Teacher Training/Professional Development

We provided training via multiple settings for this project.
  • For competition teams each fall via online training. A partnership with the Robotics Education and Competition Foundations (REC) provides us with an organization that registers teams and then provided blogs that announce the training. The training is done by the Robotics Academy and enables us to promote the Modified Autonomous Competitions that we want students and teachers to participate in. We record all training webinars and make them available at:

  • Since the VEX IQ Highrise game had the best potential to meet our needs for the Modified Autonomous competition we marketed our programs more heavily towards that community. You can learn more about that training at:

  • For partnering schools. We offered training during the school year and during the summer. This year we have two new school districts that will work with us and their teachers are participating in weeklong summer training.

  • Through the Robotics Summer of Learning Project.