Thursday, November 18, 2010

ColorChord Sound Lighting


About a year and a half ago, I wanted to put some LEDs in a clear guitar. This has been done before, but I wanted to give it a try and see if I could come up with something really cool. I bought a clear guitar, put USB controlled LEDs into it and plugged it up to my computer before I found out: No one has any tone-based algorithms to convert sound into color! What did I do? I invented ColorChord (TM).



ColorChord is the name for the technology that can select colors according to sound. Unlike most systems that look for a beat, or classify sound into various sound ranges, ColorChord is unique in that it looks at the sound exclusively on a chromatic scale. In my opinion, if you assign "A" to be purple, then A440 should be purple, A220 should be purple and so should all the other A's.

Many people have attempted to assign colors to sound in the past. Many of them were either synesthetic, had perfect pitch (or worse) both. Why is this worse? Because most people aren't synesthetic and don't have perfect pitch. Many of the systems that these people produced involved unusual color associations to sounds, or revolved around a given note always applying to a given color regardless of the key of the music it was used in. Many others had color systems that were unable to transition smoothly while increasing in pitch up a musical scale.

The concept of ColorChord here is that the primary color (most times Yellow) is applied to the note that the piece's key is. I.e. If you're playing many guitar songs, you're playing in the key of E. Therefore, the E note will be Yellow. If you're playing something in C major, the C note will become Yellow.

After much experimentation, some friends and I found that the chromatic color wheel applies surprisingly well to the chromatic scale. I have noticed that in some situations sliding the scale slightly seems to work better, but for the most part this seems generally applicable and is what is used in all of the videos.



There are some neat properties that immediately jump out of this. Chords make some semblance of artistic color sense. In this picture, it is a wheel turned to C major. A C Chord would produce [Yellow, Bluish purple, and Red] (in descending brightness) You'll notice that fifths produce nearly complementary? colors. This works out very well. After trying some other combinations this is the layout that I just couldn't do any better than. This is the color mode I use for all of the videos I'm releasing.

Speaking of which, here's another on Youtube that really shows off the complements.


Technically, this process is very complicated, and if people want the source code, I have no problem publishing it. But here's a rough overview. Just as a word of warning, everything has to happen with very low latency. This system takes sound in via a microphone and transforms it into colors in real time. More than 1/30th or 1/20th of a second and it starts to get annoying.

Step 1: Record the sound (512 to 1024 samples at a time) must be constantly streaming, minimise any buffering.
Step 2: Copy the sound to a buffer on the GPU. Because I'm using GLSL, this is just a 1D texture.
Step 3: Perform a windowed DFT, with the beginning of the sound niblet and ends tapering off to silence. This DFT cannot be implemented as an FFT. It needs to be in chromatic space. For example, on some of these demos, every octave may have 24-512 individual bins. The output is "folded." This means a C in one octave will contribute to the same bucket as a C in another octave. Several octaves may fall into the same bucket.
Step 4: Copy back to the CPU, and filter against previous samples. More filtering produces a less jittery result, but also adds latency in the response of the lights.
Step 5: Find all the peaks. These are the pitches within the octave that are loudest in the sample.
Step 6: Select the highest few peaks.
Step 7: Assign your lights to your peaks. Try to keep some inter-frame coherence here. Also, assign your lights based on the brightness of your peaks. Very high peaks may take over several lights (or all) while quieter peaks may only take on a few. If you're playing a chord, you should get roughly even distribution among all three peaks).
Step 8: Assign your peaks (or lights) the colors according to the wheel.
Step 9: Modify the brightness of the lights based on the strength of the peak. Don't worry here - percussion will simply raise the noise floor of the output because of the DFT we performed earlier, it will not show up.
Step 10: Output to the lights.

Additionally, if you're taking input via MIDI, this process is greatly simplified: the audio capture/DFT process turns into simple MIDI capture. I found that MIDI does work relatively well with this system but I don't have any videos on hand.

One point of future work I never got around to was automatically figuring out the key of a song. I tried neural networks, but that just didn't cut it. Oh well.

If you want to use this technology commercially or just for for a project, feel free to contact me.

Note that all of this material is patent pending.

Another few videos:





11 comments:

Tim said...

I would love to eventually use this software. I am planning to install LEDs in a clear guitar within the next 6 months and perform with it. Could you send it to me?

Charles said...

Sure. I am actually working on a few variants. One was done more recently for live music. I am trying to figure out a way of doing the work on an embedded system so it doesn't take a whole laptop, so we'll see where that goes. Right now, as it says, it needs a GPU to run the colors.

Austin said...

hey, I'm in a band that performs at local venues and is about to release a cd. I was wondering if it would be possible to get this software from you, and if it would work on a mac? thanks.
Austin

Charles said...

@Austin - No, it does not currently work on a mac. Also, it is very specialized to work with whatever system it's being used in at the moment. If you have/are a programmer, you should be able to get it to work on a mac and bend it to your needs.

The raw (pretty gnarly) code is at https://cnlohr.net/pubsvn/projects/voicevis/

You can try checking it out and building it on a mac. It currently on accepts the guitar I have and the DMX controller I have (For lighting).

Juris 3D said...

Amazing, i am a bit speechless. I am not into guitars, but i am big fan of making any kinds of color organ, light effects electronic projects since 80-ies. I wonder if there is any chance to get technical details about this project. Its just my hobby, my "soft spot". Thanks in advance!

Charles said...

@Juris - This is totally just a hobby, you are more than welcome to rifle through the code I posted and I might be able to provide help here and there.

David Labay said...

I have been looking into doing this for some time. I teach this relationship to my guitar students as a way of distinguishing musical notation. You have thought this all the way through but for someone like me I would like to demonstrate ColorChord to my students. Can you guide me through this? Any help would greatly be appreciated.

Charles said...

I'm terribly sorry it too so long, but I am up for helping with just about anything. I recommend contacting me via youtube, since I am much, much more responsive there.

Brandon Snell said...

Hi, where can I get the code to try this? looks awesome, and what program did you write the code in?

1reniernel said...

No offense.
When playing out of C chord and on your color wheel: why take blue for A if A is at lower frequency than C.
Surely the lowest frequency tone should be Red, which is the lowest frequency color. And the highest frequency tone must be violet. Why?
Simply to let the lights play by light frequencies corresponding to the music frequencies. Check out my project to try and make music visible for once and for all and for the deaf. Wanna collaborate?
https://sites.google.com/site/worldwishweb/music-color-chords

1reniernel said...

No offense.
When playing out of C chord and on your color wheel: why take blue for A if A is at lower frequency than C.
Surely the lowest frequency tone should be Red, which is the lowest frequency color. And the highest frequency tone must be violet. Why?
Simply to let the lights play by light frequencies corresponding to the music frequencies. Check out my project to try and make music visible for once and for all and for the deaf. Wanna collaborate?
https://sites.google.com/site/worldwishweb/music-color-chords