Our bodies are instruments, tools and limits to how we understand and exist in this world. Emotion and experiences are received and communicated through our physical being. Each sensation is first felt and interpreted by our body; it knows it more fully than words can express.
She Said was born of a personal interest in vision loss and the loss of human-to-human connection through a digital and virtual divide. It aims to capture the nuanced hand gestures in communication and provide an opportunity for the other senses to work together to gather information and meaning. Presented as an installation for one participant, each piece or sculpture is meant to be touched and explored. The touch generates audio recordings of the words and tone associated with each physicality.
Interaction
The installation begins in silence, with a projected video of the gesture above the associated gesture atop a plinth. As the participant approaches, the video disappears and they are invited to touch the sculptures. As the fingers glide along the edges, different audio clips are heard—a distinct piece of the sculpture corresponding to a distinct stage inthe movement and verbalization of the gesture.
System Parts
Drawing from architectural models, developing the gesture sculptures involved a layered topographic form made of soft balsa wood. The layers provided a direct correlation to the frame-by-frame capture of the filmed gesture.
As the sculptures are abstract in form, embedding an audio component reinforces the meaning behind the shape and allows a fuller expression. To highlight the different stages of movement, the areas of sensitivity on the sculpture are dispersed as such. This enables the participant to follow the verbalized concept with the feeling of the sculpture. The data flow includes the following:
- Copper tape lines the sculptures.
- The tape is connected to analog pins on the Arduino and data is interpreted using the Capacitive Sense library.
- Through serial communication, the Arduino sends the amount of resistance to Processing. Processing has loaded the audio files and is receiving serial data from the Arduino.
- Processing interprets the serial data and once it reaches a specific threshold (when the capacitive sensor has sensed human touch) it raises the volume of the corresponding audio. The audio cable is linked from the computer to a hidden speaker near the plinth.