Canvas to Composition: An Immersive Art-to-Music Experience

By A. Eric Heukeshoven

What if there was a way to transform colors, textures, and shapes from art into music? Sonification turns data into sound. While a soundscape is a composer’s interpretation of art, sonification creates music from data derived directly from the art itself.

Analyzing four artworks from the Minnesota Marine Art Museum’s collection, composer A. Eric Heukeshoven (American, b. 1956) assigned specific data points within each artwork sound properties, such as pitch, volume, and tone, to create original musical compositions. Inspired by the work of University of Toronto astrophysicist and musician Dr. Matt Russo, who pioneered the this technique to sonify images of space from the Hubble telescope, Heukeshoven, with support from a grant from the Minnesota State Arts Board, applied these techniques to four artworks from the Minnesota Marine Art Museum’s collection under the mentorship of Dr. Russo.

The sonifications were created using proprietary software written by Dr. Russo, Ableton Live 12, Logic Pro X,, Audacity, and Pixelmator Pro on an Apple MacBook Pro laptop. The visualizations were created by Dr. Matt Russo.

Sonifications Copyright © 2026 by A. Eric Heukeshoven (ASCAP) – All Rights Reserved.

Push boats and Pelicans by Adam Turman

Direction: Although the boat is moving to the right, the flight of the pelicans was the deciding factor for the music to move from left to right.

Scale: C Major Pentatonic (C-D-E-G-A-C)

Objects:

Clouds: Triads built from low to high. Bell with sustained sound

Pelican arc: Starting at the first pelican’s eye, the string sound “bends” downward following the second and third pelican eyes.

Boat outline: Low strings reminiscent of a tow boat’s chugging diesel engines

Waves: A low buzzy sound

Splash dots: The 243 dots are played by a midrange xylophone

Bluffs: Guitar sound with echoes

Empire Builder by Sonja Peterson

Direction: Many figures indicate right to left motion – Queen Isabella is pointing to the left - westward towards the New World, also the direction of the enslaved and the three ships – thus the music moves right to left.

Scale: Ab minor Pentatonic (Ab-Cb-Db-Eb-Gb-Ab)

Objects:

Top and bottom border (Moorish tiles): High (top) and low (bottom) pizzicato strings

Top two rows (Mayan carvings): Vertical lines sound as percussion (timbales and conga)

Fish: Reversed sustained string-like sounds

13 Pegs (which hold the artwork in place): Make a sound like ocean buoys.

Faces: Reverse shimmering sounds

Eyes: Bright bells

Queen Isabella (top right), Marie-Joseph Angelique (top center),
Cockacoesk (bottom left)

Structures: Gamelan-type sound

Castle (top right), Slaves (top center), Native American dancers (top left)

Ships: Submerged harp arpeggios

Ship 3 (bottom right), Ship 2 (bottom center), Ship 1 (bottom left)

The Rainbow by William Trost Richards

Direction: The motion of the waves dictated that the motion of the music flow from left to right.

Scale: G Major Pentatonic (G-A-B-D-E-G)

Objects:

Gull (bottom left) & distant ship (upper right): Glockenspiel (bright bell)

Sky: Evolving chord of notes A-E-G-B-D. The individual notes are filtered by the brightness of the sky: darker = lower frequencies, lighter = higher frequencies

Rainbow: The key focus of the painting is represented first by an ascending and then descending scale. Each note begins with a struck sound followed by a sustained note.

Waves: White noise (think static on the radio) volume and filtering is controlled by the shape of the waves. The result sounds remarkably authentic.

Stolen Moments by Mary Solberg

Direction: The subject faces left and is looking down. The direction of the music moves right to left with many of the objects represented by descending motion.

Scale: C Lydian (C-D-E-F#-G-A-B-C)

A system relating the “circle of fifths” in music to the standard color wheel assigns red to the key of C and green to key of F#. The C Lydian scale contains these two notes – aka the tritone.

Objects:

Bathing cap: Evolving strings play a chord – C-E-F#-A. The volume of the individual notes is controlled by the swirls in the cap – dark is softer, bright is louder.

Sparkles in the bathing cap: Photos do not capture the refraction of red and green “sparkles” in the bathing cap. I spent several hours mapping the “dots” to a photograph – 88 of them are represented in the sonification – there are probably many more! Red dots sound like a marimba played in open fifths with an echo. Green dots are portrayed by the sound of a toy piano.

Hair: Descending harp arpeggios represent the falling hair

Eyes: A fast, vibrating shimmering sound

Face outline: Ascending and descending scales on flute follow the contour of the face

Lips: A noisy shimmering sound

Shoulder lines: Descending low clarinet

Gold background: As the piece reaches its conclusion, a quartet of horns play a chord of G-A-D-F# in a lower octave. (G is the first note produced by a “red dot” at the beginning of the piece.) The volume control of the individual notes has bright areas indicating softer and dark mapped to louder volume. The chord crescendos and then fades away in the last few seconds of the piece.

Composer’s Notes

Sonification

One evening, I discovered the documentary “Listen to the Universe” on NASA TV. The program presented a fascinating story of how images from the Hubble and other space telescopes were being transformed into music via a process known as “sonification”. Sonification is not AI! I first learned this technique of creating music from data at the Workshop for Algorithmic Composition at the University of California-Santa Cruz in 2015. In the video, Dr. Matt Russo, an astrophysicist/guitarist from the University of Toronto explained how his software creates musical data from the space photos. The result was astonishing – allowing people to experience the images through sound as well, and in some cases in lieu of, sight. I immediately thought this technique could be applied to visual art and contacted Dr. Russo. To my delight, he agreed to mentor me if the project could be funded. With assistance from Scott Pollock, MMAM’s Executive Director at the time, I was awarded an Arts Experiences grant for 2026-2027 by the Minnesota State Arts Board.

Focused sound

Which brings me to the second personal experience. One of the things I treasure most about viewing art in a gallery is the solitude. Art museums are a sacred space. The question for this project was how to allow the listener to hear the sonifications without disturbing others around them. The answer came from a visit to the Guinness Brewery in Dublin, Ireland several years ago. At the beginning of the tour, visitors are presented with a series of what look like historical portraits. They are, in fact, short videos. When a visitor steps in front of the screen, the portrait comes to life and tells them about the person it portrays. Simple enough. However, I soon realized that people around me were not hearing the audio – just me. And so, I began to research “focused sound” or “directional” speakers to create an immersive environment for each of the sonifications. After reviewing several options, I chose an Audio Spotlight model 168ix speaker manufactured by Holosonics in western Massachusetts. 

Putting it together

By combining these two technologies the visitor experience is simple and organic. An area rug on the floor in front of each of the artworks indicates where the visitor should stand. When they move into the space a motion sensor in the speaker detects their presence and plays the sonification. The visitor hears the music while others in the gallery will not. In addition to the in-person experience, a QR code on a panel adjacent to the artwork takes visitors to a visualization on the MMAM website that overlays the artwork with subtle graphics highlighting the direction of the music and the elements of the artwork used to produce it. Visitors can revisit this page and, hopefully, share it with others.

I am deeply indebted to my mentor, and now friend, Dr. Matt Russo. This project would have never happened without his amazing software, patient teaching, and helpful suggestions. Thanks also to everyone at MMAM especially Scott Pollock for his creative vision, Dave Casey and Heather Casper for their early input, and the entire staff for their support and enthusiasm. To Sean Carolan at Holosonics, for his expertise in helping me wrap my brain around the focused sound technology. And to the three living artists, Adam Turman, Sonja Peterson, and Mary Solberg, for sharing insights about their works and allowing me to use their creativity to spark my own.

Biography

A. Eric Heukeshoven (Hoy-kess-hoh-ven) is a first-generation American – the son of a German immigrant father and Scandinavian mother. Heukeshoven holds a B.A. in Music Composition and Theory from the University of Minnesota-Minneapolis and an M.S. in Music Technology from the Jacobs School of Music at Indiana University. His composition mentors include Dominick Argento, Paul Fetler, Frank Bencriscutto, Tom Root, Erik Stokes, Sylvia Dyrhaug, David Cope, and Kevin Dobbe.

As a composer, educator, and music technologist, Heukeshoven creates works ranging from traditional to experimental. Where his instrumental and choral pieces reflect tradition, projects like “Time Machine” (2008), “In the Loop” (2017-2019), and “re-synth” (2024) push the boundaries of integrating traditional music with technology. In October 2026, four sonifications created by Heukeshoven of artworks from the permanent collection of the Minnesota Marine Art Museum (Winona, MN) will be unveiled as part of the museum’s Fall New Look Weekend.

Heukeshoven’s many instrumental and choral compositions are published by Carl Fisher, Score Exchange, and Swirly Music. His works are performed throughout the U.S. and Europe, and he is an in-demand jazz performer on keyboard and trombone as well as a composer/arranger. He is a member of ASCAP, American Composers Forum, American Guild of Organists, and Phi Mu Alpha Sinfonia. In 2023, Heukeshoven was inducted as a National Arts Associate in Sigma Alpha Iota where he also is member of the SAI Composers Bureau. In 2024, he received an “Artsie” award recognizing him a “Champion of the Arts” by the River Arts Alliance.

A former Assistant Professor of Music at Saint Mary’s University of Minnesota, Heukeshoven is now Director of Worship, Music & Arts at Central Lutheran Church in Winona, Minnesota and a member of the music faculty at Winona State University. He and his wife, Janet, have two sons, two grandchildren, and a yellow tabby cat.

We welcome any feedback on the sonifcation project. Please click below to take a brief survey.