by michelle pan
My project is an interactive weather scene that can be controlled through hand gestures detected from a webcam. Up to two hands (one left and one right) can manipulate the objects on the screen at a time by grabbing, poking, and pinching the sun and clouds.
One person can use two hands to play around, or two people can use one hand each to interact with the piece together!
To build off of what I had made in the previous version of this project (weather whimsy), the first thing I wanted to do was to enable interaction with multiple hands.
At first I planned to allow up to four, which is the most that the Handsfree.js library I used supports. However, I found it hard to track the hands as they moved across frames which made the objects jump around too.
I settled on only detecting two hands, which allowed me to distinguish between them by a “handedness” marker.
I originally added cursor icons over the positions of the detected hands to help with debugging, and decided to keep them in the final product because of how useful they were. Instead of looking at the console logs, with one icon I could communicate information about which hands were being tracked, whether they were left or right, and which gestures were being detected.
In addition to adding a fun element to the piece, the icons also help users see what is happening in the logic behind the scenes, or understanding “what the machine thinks”.
default: open hand from cursor.in
grab: closed hand from cursor.in
poke: pointing hand from cursor.in
pinch: custom icons, modified from open hand icons by me!
As I was brainstorming new gestures to add beyond grabbing, my first idea was to create objects with gravity that could be held with a flat hand. However, after some initial testing, I realized that a flat hand is hard to detect on the camera because there aren’t many keypoints visible (see video below)