Talk about what you've discovered by using ETC-- and post your high ranks!
Posts: 19
Joined: Mon Dec 29, 2008 3:28 am


Postby sam » Thu Nov 04, 2010 12:05 pm

What if three senses were combined? One to listen, one to establish some kind of frame of reference, and one to assign a function for what the pitch actually does. That sounds abstract.

For example, imagine every time a pitch was played, it is associated with a color and a part of the body. So the pitch C would flash the color green, and some device would vibrate the is wrapped around your ankle. The body could be divided up into 12 sections each with some kind of device that can either move, vibrate, change temperature.. whatever. In this way, each pitch could be associated with doing something, having a function. And maybe it could go both ways, where if someone wanted to hear the note C, they could touch their ankle.

Could this possibly help with the categorical perception problem?

Posts: 19
Joined: Mon Dec 29, 2008 3:28 am

Postby sam » Thu Nov 04, 2010 12:09 pm

One more idea. A while back I posted something about having to sing a certain pitch to accomplish some task. A variation on this could be having various locks that only open when a person can identify whether or not a melody or cluster of chords contains a certain pitch. So it would be like APA except that each pitch would have a different function. So instead of birds crossing the road for all pitches... identifying one pitch could unlock the refrigerator, one could open the car door, etc.

Site Admin
Posts: 1694
Joined: Tue Dec 14, 2004 12:09 pm
Location: Evanston, IL

Postby aruffo » Thu Nov 04, 2010 9:10 pm

I wonder what kind(s) of devices you'd have to rig up to make that work-?

Posts: 19
Joined: Mon Dec 29, 2008 3:28 am

Postby sam » Thu Nov 04, 2010 11:28 pm

Well it depends on if the music is audio or midi. If it is midi it is much easier.

For the body suit idea, what you need is something to listen to the incoming midi, make decisions based on what it hears, send out those decisions to some kind of controller, which would transmit signals to the body suit.

For this I would use pure data and an arduino microcontroller. Are you familiar with pure data and arduinos? If you aren't, pure data is a graphical music programming environment and and arduino is a DIY favorite microcontroller. Lots of libraries and code already exist, so usually you can put bits and pieces together to do what you want rather than starting from first principle.

So pure data listens to the midi, and libraries already exist that allow it to talk to the arduino. The arduino then talks to the suit and tells it what to do. The suit could be some kind of utility one piece, jumpsuit, and at that point you could use any vibrating device that takes power from the arduino. Not a simple task when it is all said and done, but not a monumental challenge either.

I am in school studying electrical engineering and I work in an electronics shop so I see this stuff all the time. I'm thinking about making the body suit idea a project for school.

The other idea wouldn't be that hard either, but it would need some kind of logic based input and locking mechanisms... much more complicated. Still possible but would require a lot more time.

Posts: 31
Joined: Thu Jan 07, 2010 11:55 am

Postby RockofStrength » Fri Nov 05, 2010 10:58 pm

This is called "multimodal integration" or "multisensory integration". Perhaps some useful info can be gleamed from the wikipedia entry.

Posts: 87
Joined: Wed May 23, 2007 7:55 am

Postby abminor » Mon Nov 08, 2010 5:22 am

Talking about multimodal sensory integration, you may want to check out:

It's an experiment where some peoples are asked to learn to recognize shome shapes by sound. (hearing shapes) They found that as long as the audio encoding uses a systematic approch, people are able to perfom this task. In the encoding they use the frequency of sound to represent the Y axis of the shape canvas, the X axis being represented as a mix between stereo panning and timeframe.

I personnally found this arcticle facinating as it cleary assigns a function to pitch frequency as a mean to represent an abstract concept such as a shape.

Return to “Your observations”

Who is online

Users browsing this forum: No registered users and 1 guest