About a year and a half ago, I wanted to put some LEDs in a clear guitar. This has been done before, but I wanted to give it a try and see if I could come up with something really cool. I bought a clear guitar, put USB controlled LEDs into it and plugged it up to my computer before I found out: No one has any tone-based algorithms to convert sound into color! What did I do? I invented ColorChord (TM).
ColorChord is the name for the technology that can select colors according to sound. Unlike most systems that look for a beat, or classify sound into various sound ranges, ColorChord is unique in that it looks at the sound exclusively on a chromatic scale. In my opinion, if you assign "A" to be purple, then A440 should be purple, A220 should be purple and so should all the other A's.
Many people have attempted to assign colors to sound in the past. Many of them were either synesthetic, had perfect pitch (or worse) both. Why is this worse? Because most people aren't synesthetic and don't have perfect pitch. Many of the systems that these people produced involved unusual color associations to sounds, or revolved around a given note always applying to a given color regardless of the key of the music it was used in. Many others had color systems that were unable to transition smoothly while increasing in pitch up a musical scale.
The concept of ColorChord here is that the primary color (most times Yellow) is applied to the note that the piece's key is. I.e. If you're playing many guitar songs, you're playing in the key of E. Therefore, the E note will be Yellow. If you're playing something in C major, the C note will become Yellow.
After much experimentation, some friends and I found that the chromatic color wheel applies surprisingly well to the chromatic scale. I have noticed that in some situations sliding the scale slightly seems to work better, but for the most part this seems generally applicable and is what is used in all of the videos.
There are some neat properties that immediately jump out of this. Chords make some semblance of artistic color sense. In this picture, it is a wheel turned to C major. A C Chord would produce [Yellow, Bluish purple, and Red] (in descending brightness) You'll notice that fifths produce nearly complementary? colors. This works out very well. After trying some other combinations this is the layout that I just couldn't do any better than. This is the color mode I use for all of the videos I'm releasing.
Speaking of which, here's another on Youtube that really shows off the complements.
Technically, this process is very complicated, and if people want the source code, I have no problem publishing it. But here's a rough overview. Just as a word of warning, everything has to happen with very low latency. This system takes sound in via a microphone and transforms it into colors in real time. More than 1/30th or 1/20th of a second and it starts to get annoying.
Step 1: Record the sound (512 to 1024 samples at a time) must be constantly streaming, minimise any buffering.
Step 2: Copy the sound to a buffer on the GPU. Because I'm using GLSL, this is just a 1D texture.
Step 3: Perform a windowed DFT, with the beginning of the sound niblet and ends tapering off to silence. This DFT cannot be implemented as an FFT. It needs to be in chromatic space. For example, on some of these demos, every octave may have 24-512 individual bins. The output is "folded." This means a C in one octave will contribute to the same bucket as a C in another octave. Several octaves may fall into the same bucket.
Step 4: Copy back to the CPU, and filter against previous samples. More filtering produces a less jittery result, but also adds latency in the response of the lights.
Step 5: Find all the peaks. These are the pitches within the octave that are loudest in the sample.
Step 6: Select the highest few peaks.
Step 7: Assign your lights to your peaks. Try to keep some inter-frame coherence here. Also, assign your lights based on the brightness of your peaks. Very high peaks may take over several lights (or all) while quieter peaks may only take on a few. If you're playing a chord, you should get roughly even distribution among all three peaks).
Step 8: Assign your peaks (or lights) the colors according to the wheel.
Step 9: Modify the brightness of the lights based on the strength of the peak. Don't worry here - percussion will simply raise the noise floor of the output because of the DFT we performed earlier, it will not show up.
Step 10: Output to the lights.
Additionally, if you're taking input via MIDI, this process is greatly simplified: the audio capture/DFT process turns into simple MIDI capture. I found that MIDI does work relatively well with this system but I don't have any videos on hand.
One point of future work I never got around to was automatically figuring out the key of a song. I tried neural networks, but that just didn't cut it. Oh well.
If you want to use this technology commercially or just for for a project, feel free to contact me.
Note that all of this material is patent pending.
Another few videos:
