Engineering students create gesture-controlled piano

Two engineering students spent the summer researching and creating a piano that’s played without touching any keys.

“When people hear that it’s a piano that you can play without touching the instrument, just using hand gestures, they are very surprised and are very interested in our project,” said Qingbao Wang, a senior from China.

Wang and Will Dragon, a junior from Mount Vernon, Iowa, used their knowledge of music, their engineering know-how, and their computer science programming skills to complete the piano research project within the eight weeks of the Cornell Summer Research Institute (CSRI).

“We built an ultrasonic, gesture-controlled piano, fit with an LCD display and full-functioning gesture controls to change from notes to chords and also to change the voicing the instrument plays (like an electric keyboard) and shut the instrument off,” Dragon said. 

Dragon and Wang worked with Cornell College Assistant Professor of Engineering Niloofar Kamran to create the piano. The pair of researchers say the hardest part was making the computer programs talk to one another to actually produce the sound, but they worked together to figure out how to make it all come together.

“I think I had the dream team,” Kamran said. “They come up with the ideas, and they come up with the solutions. They are great.” 

The finished product was a piano enclosed in a wooden case and painted Cornell purple. A row of sensors lined the top of the piano.

“When you hold your hand above each ultrasonic sensor, an ultrasonic wave is emitted and bounces back to the receiver,” Dragon said. “That, then, is correlated to a distance value. The distance value is run through our code and then comes out as a note, given how far your hand is from that sensor. Then the note is sent to a program called FluidSynth which then turns that note back into an actual sound.”

The idea for the research came from an open-source project on instructables.com, and the students improved the idea, added more features to it, and built their own version from scratch. While they completed the project for CSRI, this research team has bigger dreams for their creation.

“We hope it could be used with many applications for those with disabilities as well as an opportunity for people to create music for those who can’t afford or don’t have space to have a piano,” Dragon said.

“It’s very good for the people who have weak muscles and is hard for them to press the actual key on the piano,” Wang added.

Wang and Dragon both have plans to go to graduate school for engineering and finding jobs that put their research skills to good use.

“This research gives them a lot of benefits, especially for going to grad school and even for work,” Kamran said. “Because at work they will be asked to solve a problem that exists. They don’t give you the solution and it’s not in a book. It’s always good to have that tool in hand to go investigate something and solve a problem. That’s very similar to a research project.” 

But looking back at their summer, they also just think about how much they enjoyed engineering an ultrasonic, gesture-controlled piano.

“It was a blast,” Dragon said. This project has been really fun.”