Idaho State University, industry partner House of Design receive $162K IGEM grant to develop application to connect robotics with augmented reality

This project will be supported by a $162,606 grant to ISU from the Idaho Global Entrepreneurial Mission (IGEM) Council, which is administered by the Idaho Department of Commerce. The grant to design and develop an augmented reality platform for robotic systems design and interaction (ARPRI) will run next year. House of Design does custom automation of ABB robots that are used in industrial settings.

Co-principal investigator Alba Perez, ISU associate professor and chair of mechanical engineering, said she envisions a programmer wearing Microsoft Hololens augmented reality glasses who will be able to see the real world, but then when they’re within a certain distance they can communicate wirelessly with the robot. The augmented reality glasses may feature menus off to the side that programmers can see and then select menu options, and it is possible the augmented reality system could use voice commands for communications between the programmers and the robots.

Omid Heidari, an ISU Ph.D. student who will be working on the project, said he enjoyed working with House of Design during an internship this summer and is looking forward to participating in this project.

“In normal situations the designers and programmers sit at their computers, write some programs, simulate them there and then upload them and then the robot does the task,” Heidari said. “So now we are thinking instead of using the computer and the software, use augmented reality glasses so that the programmer can go by the robot  and interact with the robot and can see a visual panel by the robot, for example, to see what is inside the robot, the programs that are running, and give it commands.”

The starting point of this project was during Heidari’s internship, where he helped design an augmented-reality application for mobile devices to interact with their robots.

“In this project, we will be expanding those preliminary results to a more complete platforms and using HoloLens,” Heidari said.

Perez said the first step in the project is put some sensors in the augmented reality glasses so the glasses can identify the robot. Then the researchers need to establish communications with the robot, then send information to the robot and then integrate all these processes. 

“We will try to develop a tool that will allow them  to communicate with the robots using  an augmented reality to send information to the robots  and receive information from the robots to make the programming and the setup very natural and easy,” Perez said.

The grant’s other principal investigator is mechanical engineering Professor Marco Schoen in the College of Science and Engineering. ISU will work with House of Design CEO Shane Dittrich. They’ll be completing the work with an additional ISU graduate student in the ISU College of Science and Engineering’s Measurement and Control Engineering Research Center. An industrial robot worth $35,000 to $40,000 is being provided to ISU for the project by House of Design.

“I am very excited about this opportunity to connect business and industry partners with our university researchers,” said President Kevin Satterlee, Idaho State University. “IGEM’s goal of fostering economic prosperity is a great benefit to the communities of Idaho, and Idaho State is pleased to be part of the process.”

Robotic systems are being deployed more and more in many industrial processes, and House of Design is a leader company in this field, which was named 2017 Innovative Company of the Year in Idaho, Perez said. This project has grown out of ISU’s expertise in developing an augmented reality system to assist with arm rehabilitation and Heidari’s internship with House of Design.

 "We expect House of Design to benefit from having a very natural application to interact with the robots,” Heidari said. “Looking at the future, applications in augmented reality may expand to many activities in industry, health care and the environment in  which robotics may also be involved"

The IGEM grant funds will go towards funding two full-time graduate students to work on the project for a year,

“It is great to have this grant,” Perez said. “These things have a ‘wow’ factor that is always there. We will start developing the platform as soon as the paperwork is done in early January.”