Cyth Systems (San Diego) has built a Guitar Hero playing robot that uses a camera to see and recognize the notes flying by and pneumatic fingers to depress the buttons with great precision. You have to see it to believe it...
"We have 5 groups of pixels on the screen corresponding to the 5 notes we might need to play," says Ivan Gagne, Systems Engineer with Cyth Systems. "We relied on the light intensity of the pixels in each of those groups. We couldn't rely on the color of the notes because during the game all the notes turn bluish white when you activate star power. We also couldn't rely on the shape of the notes because, at times, they turn into stars. If that light intensity value exceeds a predetermined threshold, we know we need to hold down that key and actuate the strum bar after a predetermined delay while the note travels down the screen to the bottom "play line". If a note was there for more than ~1/4 second the machine activates the whammy bar to score additional points on the sustained notes. There's another threshold where the intensity has to drop before it can consider the note "played" and therefore ready to start playing another note. The imaging, the region of interest, and the light intensity functions were all built into NI's vision toolkit for LabView."
"We have 5 groups of pixels on the screen corresponding to the 5 notes we might need to play," says Ivan Gagne, Systems Engineer with Cyth Systems. "We relied on the light intensity of the pixels in each of those groups. We couldn't rely on the color of the notes because during the game all the notes turn bluish white when you activate star power. We also couldn't rely on the shape of the notes because, at times, they turn into stars. If that light intensity value exceeds a predetermined threshold, we know we need to hold down that key and actuate the strum bar after a predetermined delay while the note travels down the screen to the bottom "play line". If a note was there for more than ~1/4 second the machine activates the whammy bar to score additional points on the sustained notes. There's another threshold where the intensity has to drop before it can consider the note "played" and therefore ready to start playing another note. The imaging, the region of interest, and the light intensity functions were all built into NI's vision toolkit for LabView."
