A Brief History of Synaesthesia and Music by Sean A. Day

This is a very good indepth article on the history of visuals and music

it discusses many of the composers who created rules for colour coding notes and keys such as Nikolai Rimsky-Korsakov who created this chart below

B major gloomy, dark blue with steel shine
Bb major darkish
A major clear, pink
Ab major greysh-vioket
G major brownish-gold, light
F# major green, clear (color of greenery)
F major green, clear (color of greenery)
E major blue, sapphire, bright
Eb major dark, gloomy, grey-bluish
D major daylight, yellowish, royal
Db major darkish, warm
C major white

http://www.thereminvox.com/article/articleprint/33/-1/5/

it also touches on more recent musicians who have explored this area, miles davies, Sofia Gubaidulina and Michael Torke, are but a few of the people who see this link.