Syn(es)thetic Reality: simulating synesthesia for the non-synesthetic

 

We generally understand and perceive our surroundings primarily through our five sensory organs – audition, sight, taste, touch and smell.

And we describe our reality in terms of these senses, and perhaps as a summation of these senses. An object is qualified by the color we see or the texture of sounds we hear. All these bits are added to form a description in our brains.

What would then the understanding of the world be if we were to perceive the world not as a sum but as a product of our senses? Can an object be described by the color we hear and the sounds we see?

This is the phenomenon of synesthesia and Syn(es)thetic Reality aims to create a synthetic experience of synesthesia.

What is Synesthesia?“Synesthesia is a rare experience where one property of a stimulus evokes a second experience not associated with the first.”[1]

In other words a stimulation of one sense causes an involuntary stimulation in another sense. And thus allowing some people to smell sounds and taste letters. The first sense is called the inducer and the second is called the concurrent. The inducers and concurrents are not limited to our external five sensory organs.

This leads to the categorisation of two types of synesthesia – associative and projective. Associative synesthesia is where the person has a feeling of the secondary sense. It is like an internal trigger rather than on the physical level. For example when a synesthete hears the waterfall, there is a strong association to the color purple, but they don’t actually see the color. This type is more common than the projective kind.

On the other hand projective synesthesia is where the concurrent is a physical phenomena. The purple color of the waterfall would actually be seen by the projective synesthete in some form.

Some people associate colors to numbers or sounds to shapes or smells to sounds, and similarly there are various types of synesthesia. An important thing to note with synesthesia is the fact that not everybody has the same associations even with the same type of synesthesia. Everyone has different connections and this makes it difficult to make comparisons between synesthetes or find common ground rendering synesthesia to be very subjective.

Chromesthesia is one of the many types of synesthesia particularly dealing with the sound to vision connection. Where people associate colors to sounds and vice versa. Parameters or features in sound such as amplitude frequency or texture could correlate to the hue, saturation or brightness in color.

[1] Banissy M.J., Jonas C. and Kadosh R.C. Synesthesia: An introduction

Technology in SensesTechnology gives us a common ground to simultaneously access and reconstruct our human senses. We have devices like microphones and cameras that create data out of the physical world. As well as screens and speakers to create physicality out of data.

And this provides for a standardized denomination across them, allowing us to interweave sensory connections.

With this information the project abstract is as follows:

Abstract
‘Syn(es)thetic Reality’ explores a new way of sensing the world by understanding sounds through colors. It looks to simulate projective chromesthesia, an experience of seeing colors involuntarily as a result of what is heard.

It is a web application for mobiles that are to be used as AR/VR devices. This format allows for the user to be immersed in a synthetic reality. By displaying audio based content over the camera feed, the users visual perception is altered to reflect his auditory perception. The basic principle is to correlate frequencies to hue and amplitude to saturation and visualise these colors creatively.

Through this new perception one is lost in a synesthetic reality.

Tools
WebAudio uses FFT algorithms to analyse incoming audio data into a frequency spectrum giving information about the amplitude for specific frequencies.

WebVR and Three.js allows to create virtual or augmented reality apps for the web. Three.js creates the content and WebVR renders it for a stereoscopic effect.

Design
The components I require are a microphone for the audio, a camera for the vision, and a display screen. These all come neatly packed into phones nowadays which allows also for a processing unit and a battery. Pairing the phone the google cardboard was the next step to use it as a VR device.

In terms of the platform for this application, the web allows easy dissemination of the project  as compared to other formats.

The idea behind the headset was to first of all avoid disturbing the ears and secondly make it lightweight. As for the viewer I have redesigned the Google Cardboard to fit the aesthetics of the headset.

Claims
In terms of the claims and what the project is dealing with, firstly would the use of this app aid in developing chromesthesia? Previous research has shown that by simulating Grapheme Color synesthesia, when people see letters in colors, the group started showing signs of independent associations.

Another idea being discussed is whether through synesthesia we change our understanding of the world. For example Plato’s allegory of the cave and the matrix have shown that reality exists beyond our experience of it and that through different experiences, reality changes or enhances.

Although there are studies in synesthesia, due to its subjectivity, scarcity and ignorance the studies are helpful only to a certain extent.

Syn(es)thetic Reality is a project of IAAC, Institute for Advanced Architecture of Catalonia developed at Masters in Advanced Interaction in 2019 by:
Student: Vinay Khare
Faculty: Luis Fraguada, Elizabeth Bigger
Support: Cristian Rizzuti