Localization of sound

Just note to self:

http://www.sdss.jhu.edu/~tamas/bolts/locator.html

for 3D-sound placement with 4 microphones.

Need better understanding of cross-correlation.

Maybe one could use really simple mics and place them on a table. It's almost only time-difference that matters. 2D-case is much simpler.

Lear synth conceptual sketch

Let me introduce the Lear Synth, a collaborative soundscape-synthesizer. The synthesizer is furniture sized, and have a lot of different cavities, with a slight similarity of an ear.

Loudspeakers and microphones are placed in the cavities and the sound is generated partly by the cavities shaping the sound, and partly the digital signal processing in the built-in computer.

The sound is affected and shaped by placing bodyparts or other objects in the cavities and thereby changing the geometry and resonance in the cavities.

The sound experience gets richer than with headphones or stereo speakers since the different sound sources are heard on their own.

Alternative ways of interacting with sound

When I was young, I tried to build a modular syntesizer, one there one could make any sound imaginable (for a really crazy modular, check out extensive interview with Joe Paradiso, MIT).

However, you want quality knobs, and they costs like 60 kr/each, complete with knobs, frontplates and so on. The whole case would be very expensive and actually, you need a lot of knobs. And the syntesizer would not start to be fun until you had at least a half dozen effects and generators. That would be a huge investment. And since I had some nice softwares at hand, the interaction part would be the more important than the sound genereating one (since that could be solved in my computer).

I actually sketched at least a hundred sketches of interaction possibilities, some optical, some knobs and others even more freaked out (inspired by the group 8Tunnel2, who made music by slicing broccoli with electronic knifes...).

The light interaction part was really fruitful. Light is easy to mix (an important property), and can be led in plastic fibers, leds and photosensors are indeed unexpensive. It can also have different colors, and be filtered through colored plastic. It's also visible, nondangerous and can be shut of by a hand or any non-transparent object. Building gain-like things would be very easy (especially compared to the knobs). Negative aspects is that it can not be "negative" in an easy way, you cannot show negative voltages with light (because that would be the negative absence of light). You need some kind of microphones to convert light to electricity and "sound" again (in my world, all sound is electric, loudspeakers are usually not that intresting to me).

However, I eventually realized that the ac in the walls would put a nice 50 Hz "light hum" through light bulbs in all the sounds if the box wheren't totally sealed. However, one could work with it bit more. I will explore it further.

Hypnosis 2000

Idag får det bli svenska.

När jag kom till lektionen, hade alla redan börjat. Jag tog min dator (hur skulle jag klara mig utan dator) och började pyssla med phidget-servona. Jag insåg att jag aldrig någonsin ägnat mig åt kontrollerade stegmotorer. Det är en ganska dyr alternativt krånglig teknik för hemmapulare, eftersom att man behöver både DACar och mikrokontrollers, och om man ska implementera dem från scratch behövs dessutom ett hum om reglerteknik och induktiva laster, vilket inte är så enkelt.

Nåväl, tack och lov fick jag igång ett java-program, som SimpleServo.java, som kunde ställa ut motorn och vrida den. Programmet krashade en smula då det försökte ställa motorn för långt.

Jag kom att tänka på att det vore roligt att bygga en hypnosmaskin, Austin Powers och sedan den absurda karaktären Percy Nilegård kom upp i mitt inre:

Och titta vad han har i handen! En hypnosmaskin! Man kunde helt enkelt göra liknande spiralmönstrade plattor och sätta på servomotorn. När någon kom i närheten kunde den börja snurra och datorn som styrde den kunde spela upp hypotiserande Sirensång och datorn kunde genom en lugn röst styra människor i sin omgivning. Bra idé!

Johan och Anna anslöt. Eftersom att datorn var min och ingen av dem kan Mac särskilt bra fortsatte jag att vara datoroperatör. Jag är ganska bra på det också, det är väl inget konstigt, men jag insåg att det var ganska svårt att programmera med som sitter bredvid. Ofta ville de kolla referenser, och så, när jag ville programmera. Nåja, det gick bra, även om det måste varit väldigt tråkigt för dem att titta på när andra jobbar.

Det gick i alla fall framåt. Vi diskuterade min idé, hypnotisören, och hur man skulle kunna kontrollera människor - till exempel genom att titta hur mycket de rörde sig genom att filma dem med en kamera. Om de rörde sig skulle de sägas åt att stå stilla. Datorn som befallare.

Interaktionsdesignern kom förbi och pratade om hjärnvågor och så, för att genom dem kunna beräkna vad en människa höll på med. Jag har inte gått vidare med den idén riktigt, eftersom att jag inte har någon bra idé på hur man skulle göra.

Ett problem var att vi både behövde se till att personen stod still framför kameran, men samtidigt inte "försvann" och maskinen fortsatte sin hypnos i onödan. En IR-sensor, värme eller en ljusstråle som bröts skulle kunna lösa problemet.

Diskussionsgenomgång med hela klassen. Intressant att höra de andras idéer, men extra roligt att höra att hypnosmaskinen var något helt nytt. Anna berättade om pendeln, vilket var en fullkomligt lysande idé. Anna var tvungen att gå på något ärende, och jag och Johan implementerade pendeln, med hjälp av ett bit sladd och en vikt.

Genom formeln 2*pi*sqrt(L/g) fick vi fram periodtiden, vilken vi implementerade i servot, så att den slog fram och tillbaka i rätt takt. Inget utjämnande, den går från 10° till 30° i dagsläget, vilket var tillräckligt bra för att vikten skulle börja pendla bra.

Intressant var när vi hängde upp pendeln i taket, och satte igång maskineriet. Folk tittade förbluffat på pendeln. Uppenbarligen fungerar maskinen bättre än vi vågade hoppas.

Vi hittade en algoritm för motion detection i processing på http://www.learningprocessing.com/examples/chapter-16/example-16-13/ som vi blev väldigt nöjda med, vi lade till en räknare som räknade alla svarta pixlar för varje bildruta, och det blev bara svarta pixlar när det var en stor förändring i bilden. Det blev tillräckligt bra. Genom en finurlig medelvärdesalgoritm kunde vi säga när personen rörde sig för mycket.

Det återstod bara att koppla ihop webkamera-applikationen (processing) och servokontrollen (java). Efter att försökt använda processing-kärnan i java men gett upp, lyckades vi, genom att Johan förstod att vi behövde fånga exceptions, och Anna som förstod att vi var tvugna att bygga allt som objekt och jag som förstod att vi behövde threads för att inte bli knäppa vid implementationen. Till slut fungerade det! Vi kopplade det till en början bara till att servot rörde sig när man rörde på sig i kameran, och bara det var en verkligt rolig applikation.

När man satt framför datorn kunde man alltså gunga och så började pendeln också gunga. Fantastiskt roligt. Så långt kom vi idag.

Här är några bilder:

Och en film:

Man ser i tur och ordning datorn med en skymt av motion-detection-programmet och sedan pendeln.

Här är källoden för programmet också (fungerar på macbook, annars måste man göra något åt kamera-raden, antagligen):

import processing.video.*;

import com.phidgets.*;

import com.phidgets.event.*;

float lengthofpendulum = 0.55; // meter!

int periodtime = int( 1000 * 2 * PI * sqrt ( lengthofpendulum / 9.82)); // milliseconds!

Capture video; // Variable for capture device

PImage prevFrame; // Previous Frame

float threshold = 70; //// How different must a pixel be to be a "motion" pixel

int movtresh = 50; //how many frames that need motion detected to be able to move

int movement=0; //increases when a videoframe have a lot of black dots (have been moved)

int blackthreshold = 4000; //how many black pixels are nescessary befor

// Create the object with the run() method

Runnable runnable = new PendelThread();

// Create the thread supplying it with the runnable object

Thread pendelthread = new Thread(runnable);

//Thread controlthread = new Thread(controller);

// Learning Processing

// Daniel Shiffman

// http://www.learningprocessing.com

// Example 16-13: Simple motion detection

void setup() {

size(320,240);

video = new Capture(this, width, height, 30);

// Create an empty image the same size as the video

prevFrame = createImage(video.width,video.height,RGB);

pendelthread.start();

}

void draw() {

// Capture video

if (video.available()) {

// Save previous frame for motion detection!!

prevFrame.copy(video,0,0,video.width,video.height,0,0,video.width,video.height); // Before we read the new frame, we always save the previous frame for comparison!

prevFrame.updatePixels();

video.read();

}

loadPixels();

video.loadPixels();

prevFrame.loadPixels();

long blackamt = 0; //reset the black-counter between every frameupdate

// Begin loop to walk through every pixel

for (int x = 0; x <>

for (int y = 0; y <>

int loc = x + y*video.width; // Step 1, what is the 1D pixel location

color current = video.pixels[loc]; // Step 2, what is the current color

color previous = prevFrame.pixels[loc]; // Step 3, what is the previous color

// Step 4, compare colors (previous vs. current)

float r1 = red(current);

float g1 = green(current);

float b1 = blue(current);

float r2 = red(previous);

float g2 = green(previous);

float b2 = blue(previous);

float diff = dist(r1,g1,b1,r2,g2,b2);

// Step 5, How different are the colors?

// If the color at that pixel has changed, then there is motion at that pixel.

if (diff > threshold) {

// If motion, display black

pixels[loc] = color(0);

blackamt++;

}

else {

// If not, display white

pixels[loc] = color(255);

}

}

if (blackamt > blackthreshold) {

movement++; //if we hade over

}

}

updatePixels();

//println(blackamt); //DEBUG: how many black pixels are there?

}

// Start the thread

class PendelThread implements Runnable {

void run() {

println("Pendel: Pendel starta");

// This method is called when the thread runs

//här ska koden ligga

try { //because listeners can throw phidgetexception, which must handled by processing. //simple solution

ServoPhidget servo;

System.out.println(Phidget.getLibraryVersion());

servo = new ServoPhidget();

servo.addAttachListener(new AttachListener() {

public void attached(AttachEvent ae) {

System.out.println("attachment of " + ae);

}

}

);

servo.addDetachListener(new DetachListener() {

public void detached(DetachEvent ae) {

System.out.println("detachment of " + ae);

}

}

);

servo.addErrorListener(new ErrorListener() {

public void error(ErrorEvent ee) {

System.out.println("error event for " + ee);

}

}

);

servo.addServoPositionChangeListener(new ServoPositionChangeListener()

{

public void servoPositionChanged(ServoPositionChangeEvent oe)

{

System.out.println(oe);

}

}

);

servo.openAny();

System.out.println("waiting for Servo attachment...");

servo.waitForAttachment();

System.out.println("Serial: " + servo.getSerialNumber());

System.out.println("Servos: " + servo.getMotorCount());

servo.setPosition(0,0);

Thread.sleep(1000);

int pos=10;

while (true) //isch ok

{

println("Gungar!");

if (pos==10 && movement > movtresh){

pos=30;

servo.setPosition(0,90);

}

else if (movement > movtresh) {

pos=10;

servo.setPosition(0,10);

}

movement = 0;

//servo.setPosition(0, i);

Thread.sleep(periodtime);

System.out.println("Position: " + servo.getPosition(0));

}

/*System.out.println("Outputting events. Input to stop.");

System.in.read();

servo.setPosition(0, 0);

System.out.print("closing...");

servo.close();

servo = null;

System.out.println(" ok");*/

}

catch (Exception e) {

println("fel!");

}

}

}

Regarding control of modern sounds

Electronically generated music have been around for more than a century. From Léon Theremin's hand played theremin, where the player controlls the sound by manipulating the capacitance between his body and the antenna of the instrument, resulting in beating between two radio frequency oscillators) to the most modern highly evolved musical hardware, ie Elektrons Machinedrum, and the totally spaced out Eigenharp and eventually the total transitation into the software domain, with for example Native Instruments Reaktor where soundcapabilities is way beyond imagination.

The possibilities for electronic music making have exploded during the last years. But still people pay excessively high prices for the actual music hardware. The sonicly quite limited Roland SH-101 is still selled for prices well over 6000 SEK (which is probably more than it costed originally). How come?

If you ever have had the pleasure of playing a subtractive analogue synthesizer, with all the knobs, arpeggiators and such thing (or a modern equivalent) you know you quickly gets totally stunned by the high degree of control you can have. The direct feedback from the sound must be experienced to be understood.

Which are the factors for this experience? I see several factors:

• the constraing of price have forced the designers to chose a simple, yet general architechture
• a few (not many) quirks in the architechture gives important character to the instrument, which are often discussed and loved among the users (some parameter that can be driven out of range, a special distortion when driven to loud, a sinus tone coming from the resonance filter).
  
• the parameters range and scales are carefully chosen, for good precision when using the knobs and other controllers
  
• the non-chaning spatial mapping to parameters in the sound gives the possibility to predict how a sound probably will sound, which is gotten from experience.
• A fairly straight forward architechture, with not too many uncontrolled feedback loops (this is not true for modulars).
• constraint in how many tones can be played, sometimes monophone
• a very responsive instrument ("infinitley fast", low latency)
  

During the 90ties (80-talet in swedish), the digital technology came and actually destroyed most of the usability in the previous analogue instruments. This was mainly because of cost and engineering effiency. Buttons are cheaper and even easier to program than knobs (which are very analogue). However, by reducing the previously very inituitive, and organic method of shaping sound, both the model (non-inituitive FM-synthesis, wavetables) and the interface (buttons and abstract parameters and choosable models) rendered the soundprogramming to some kind of black art. The sounds were also percepted as more harsh and metallic, which is to expect because of the richer and more complex overtones.

This can be heard in the songs as well - the arrangements are more complex (because of the increased amount of polyphony and different timbres played at once), but the sound went in a sense more strict and repeating, probably beacuse the sequencers of the time made it harder to get a less perfect sound. The use of samples gave a new touch to the music, but it was still often not as organic as with the old analogue synthesizers.

After around 1994 everything went crazy, and, among others, the swedish company Clavia introduced the Nord Lead (pictured above), a digital synthesizer with analogue knobs that worked as an analogue synthesizer, and the music production changed totally again, and techno and more esoteric electronic music like did get widely spread, like.

Artists like Aphex Twin made song like Isopropohlex (in Spotify playlist), which is has a very simple "melody", but all the sounds in the song evolve very intensive. This would have been impossible to do in the 90-ties without a very large and expensive analogue modularsynthesizer.

Do you remeber the Trance music from around 2002? This evolved from the gated patterned arpeggio of the Access Virus synthesizer, which eventuallty moved straight into the software realm. Nowadays the electronic music have diffused into most of modern radio productions (Röyksopp - Girl and the robot, among many others, just look at the swedish eurovision song contest).

It's stunning to see what deep impact the music hardware (and software) have on the music created. When new features arrive, they are of course used in new creative ways, and old ways of using the instruments can get out fashion quickly.

Here is a video of the japanse artist Denkitribe using two quite small instruments (the Korg ER-1 drum machine and the Clavia Micro Modular) for a quite catchy tune.

The basics, the history

My name is Linus Ericsson, I study Engineering Physics, Teknisk Fysik, although I have been fiddling around with computers and electronics from when I was about 9 years old. I sometimes wonder how it would be if I got into the physical interaction design course already then...

I won an iPod shuffle yesterday, in a short workshop at Excitera innovation challenge. It was something about P2P-networks between mobiles, and possible uses of them, I suggested that one could use it for transmitting subversive information among dissidents in totalarian states. The problem was not mine, since I lunched with Erik Jennische, who works as some kind of assistant to the policitican Madeleine Sjöstedt in the Stockholm City hall. Erik traveled to Cuba and handed out books about things like economic theory, free speach, democrazy and more, since the government keeps its power through restricting information. The books were passed on 100 times and more!

P2P network on the mobile phones would solve the problem of copying the information (a laser printer does not work on Cuba of mainly logistical reasons). There are still other problems as well. The government probably likes to forbid things with these features.

Technology changes societys. People with power but without technology always fears that.