Thursday, November 18, 2010

ColorChord Sound Lighting

About a year and a half ago, I wanted to put some LEDs in a clear guitar. This has been done before, but I wanted to give it a try and see if I could come up with something really cool. I bought a clear guitar, put USB controlled LEDs into it and plugged it up to my computer before I found out: No one has any tone-based algorithms to convert sound into color! What did I do? I invented ColorChord (TM).

ColorChord is the name for the technology that can select colors according to sound. Unlike most systems that look for a beat, or classify sound into various sound ranges, ColorChord is unique in that it looks at the sound exclusively on a chromatic scale. In my opinion, if you assign "A" to be purple, then A440 should be purple, A220 should be purple and so should all the other A's.

Many people have attempted to assign colors to sound in the past. Many of them were either synesthetic, had perfect pitch (or worse) both. Why is this worse? Because most people aren't synesthetic and don't have perfect pitch. Many of the systems that these people produced involved unusual color associations to sounds, or revolved around a given note always applying to a given color regardless of the key of the music it was used in. Many others had color systems that were unable to transition smoothly while increasing in pitch up a musical scale.

The concept of ColorChord here is that the primary color (most times Yellow) is applied to the note that the piece's key is. I.e. If you're playing many guitar songs, you're playing in the key of E. Therefore, the E note will be Yellow. If you're playing something in C major, the C note will become Yellow.

After much experimentation, some friends and I found that the chromatic color wheel applies surprisingly well to the chromatic scale. I have noticed that in some situations sliding the scale slightly seems to work better, but for the most part this seems generally applicable and is what is used in all of the videos.

There are some neat properties that immediately jump out of this. Chords make some semblance of artistic color sense. In this picture, it is a wheel turned to C major. A C Chord would produce [Yellow, Bluish purple, and Red] (in descending brightness) You'll notice that fifths produce nearly complementary? colors. This works out very well. After trying some other combinations this is the layout that I just couldn't do any better than. This is the color mode I use for all of the videos I'm releasing.

Speaking of which, here's another on Youtube that really shows off the complements.

Technically, this process is very complicated, and if people want the source code, I have no problem publishing it. But here's a rough overview. Just as a word of warning, everything has to happen with very low latency. This system takes sound in via a microphone and transforms it into colors in real time. More than 1/30th or 1/20th of a second and it starts to get annoying.

Step 1: Record the sound (512 to 1024 samples at a time) must be constantly streaming, minimise any buffering.
Step 2: Copy the sound to a buffer on the GPU. Because I'm using GLSL, this is just a 1D texture.
Step 3: Perform a windowed DFT, with the beginning of the sound niblet and ends tapering off to silence. This DFT cannot be implemented as an FFT. It needs to be in chromatic space. For example, on some of these demos, every octave may have 24-512 individual bins. The output is "folded." This means a C in one octave will contribute to the same bucket as a C in another octave. Several octaves may fall into the same bucket.
Step 4: Copy back to the CPU, and filter against previous samples. More filtering produces a less jittery result, but also adds latency in the response of the lights.
Step 5: Find all the peaks. These are the pitches within the octave that are loudest in the sample.
Step 6: Select the highest few peaks.
Step 7: Assign your lights to your peaks. Try to keep some inter-frame coherence here. Also, assign your lights based on the brightness of your peaks. Very high peaks may take over several lights (or all) while quieter peaks may only take on a few. If you're playing a chord, you should get roughly even distribution among all three peaks).
Step 8: Assign your peaks (or lights) the colors according to the wheel.
Step 9: Modify the brightness of the lights based on the strength of the peak. Don't worry here - percussion will simply raise the noise floor of the output because of the DFT we performed earlier, it will not show up.
Step 10: Output to the lights.

Additionally, if you're taking input via MIDI, this process is greatly simplified: the audio capture/DFT process turns into simple MIDI capture. I found that MIDI does work relatively well with this system but I don't have any videos on hand.

One point of future work I never got around to was automatically figuring out the key of a song. I tried neural networks, but that just didn't cut it. Oh well.

If you want to use this technology commercially or just for for a project, feel free to contact me.

Note that all of this material is patent pending.

Another few videos:

Wednesday, June 9, 2010

Comparison of different cloths for projector screen

I purchased a HD20 Optoma 1080p projector and fell in love with using it against the wall in my old dorm room. Recently, though, there have been more and more cases where I needed to use the projector somewhere where a wall was unavailable.

We ended up doing rock band in a number of unusual places, we we needed a solution.
This prompted the purchase of a sheet from target. This was fraught with problems. The sheet allowed too much light through from behind. When there wasn't much light, light would pass through the sheet, bounce off of whatever was behind it and then come back through the sheet causing "echoes" of sorts. The king size bed sheet technique failed miserably.

This first picture shows the echo effect, you can see it around the Pin Panic and ACK in Attack.

One fun thing that happened was we were able to play though the sheet. NOTE: This is actually TWO sheets. These king size sheets are ridiculously translucent.

Now that I have gotten around to it, I went to Jo Ann Fabrics and bought 1.5" of every white fabric that I thought would be reasonable.

These cloths include (in order of increasing appeal after tests)
  1. Wool Felt - 37" Wide, $9.99/yd, 2.0 oz for 3 sqft, capable of making a [37"x65"]; [75" or 6.25' diag.]; [17 sqft] screen @ $19 and 11 oz. - Fail immediately, produced a "lumpy" surface in terms of brightness.
  2. Polyester Felt - 72" Wide, $4.99/yd, 1.5 oz for 3 sqft, capable of making a [72"x128"]; [146" or 12.2' diag]; [64 sqft] screen @ $18 and 32 oz. - Fail immediately, produced a "lumpy" surface in terms of brightness.
  3. Polyester Fleece - 62" Wide, $7.99/yd, 3.0 oz for 3 sqft, capable of making a [62"x110"]; [126" or 10.5' diag]; [47.3 sqft] screen @ $25 and 47 oz. - Fail immediately, produced a blurry picture.
  4. Cotton Denim - 56" Wide, $9.99/yd, 3.5 oz for 3 sqft, capable of making a [56"x99"]; [114" or 9.5' diag]; [37.5 sqft] screen @ $28 and 44 oz. - Fail after a bit of consideration, produced unusual patterns because the ridges in the denim interacted with the pixels. It also wrinkles very easily.
Now, we move on to the winners.
  1. Polyester Pleather - 54" Wide, $8.99/yd, 2.0 oz for 3sqft, capable of making a [54"x96"]; [110" or 9.1' diag]; [36 sqft] screen @ $24 and 24 oz. - One side is blurry, the other is very shiny but sharp. This unfortunately allows the projector to reflect back in your face, so there's bright spots. This is usable for rear or front projection, but would not be my first choice. Does not allow much light through but you can still see it on the other side. This means echoes will be very minor. If this is used for rear projection, it will eat a lot of the light but will not have a bright spot where the projector is.
  2. Target 30% Cotton 70% Polyester - 56" Wide, $6.99/yd, 2.0 oz for 3 sqft, capable of making a [56"x99"]; [114" or 9.5' diag]; [37.5 sqft] screen @ $20 and 25 oz. - Probably best material if you still wanted rear projection to be possible. Both sides are relatively good and very similar. Picture quality almost as good as #1 and #2. This allows you to do rear projection. Front projection will have a worse echo problem then the pleather. I am considering this, since I may want to do rear projection. Also this will have a bright spot where the projector is.
  3. Multipurpose Cloth - 54" Wide, $19.99/yd, 4.5 oz for 3sqft, capable of making a [54"x96"]; [110" or 9.1' diag]; [36 sqft] screen @ $54 and 54 oz. - Completely opaque, no chance of echo. Sharp picture, almost the same as blackout cloth. Prohibitively expensive and very heavy. This could double as a thick blanket in an emergency. :-p
  4. ***Black Out Cloth (70% Polyester, 30% Cotton) - 54" Wide, $5.99/yd, 3.5 oz for 3sqft, capable of making a [54"x96"]; [110" or 9.1' diag]; [36 sqft] screen @ $16 and 42 oz. - Opaque, nice, sharp picture. Very similar to multipurpose cloth. Fabric does not breathe, would not use as a blanket. Does not seem very easy to clean either. BUT! It's CHEAP, OPAQUE and not too heavy! This is what I will be building my screen out of. There's a lot of guides that go about ways of doing this. If I find a particularly innovative way, I may post it.
I took a number of pictures to compare.

You can see the pictures over at

Some interesting pics are: [Poly Felt, Denim, Fleece, Target, Pleather, Wool Felt]

Comparison of the front and back side of the pleather:

You can see the problems with the denim and the moire effect

Just to make it completely obvious how bad the fleece is: Here is the fleece compared to the target.

Comparison from behind [Target-Blackout-Multicloth-Denim-Pleather]

You may not want to use Target if you are concerned about a bright spot coming through from the other side.

I cannot begin to stress how bad felt is at this: