Also known as interactive or dynamic music, adaptive music reacts to changes in gameplay state. It encompasses techniques such as re-sequencing of precomposed pieces, dynamic mixing, generative music where either notes or sounds are generated at runtime in software, or sequencing directly based on user input such as in Tetris Effect.
Horizontal and Vertical Adaption
There are two main types of adaptive music, typically referred to as horizontal re-sequencing or vertical remixing/re-orchestration.
Horizontal re-sequencing
Horizontal re-sequencing is when a game soundtrack is divided into segments that are jumped between based on the game state. 'Horizontal' refers to the musical timeline which is usually layed out horizontally in music software and musical notation.
Hard re-sequencing
Games from the 1980s with horizontal re-sequencing tended to do hard cuts between music parts, completely ignoring key and rhythm, such as in Frogger (1981) (see Hutchinson (2021)).
Re-sequencing with transitions
Lucasfilm Games refined horizontal re-sequencing with Monkey Island 2 (DOS) by creating a system for hand-written transitions between different music parts (see Bajakian, Land (2019), Collins (2016), Collins (2008)). This type of adaptive music has been the norm for most large video game productions in the 2010s-2020s.
Vertical remixing/re-orchestration
Vertical remixing is dynamically changing a piece’s instrumentation by attenuating musical layers based on gameplay state.
An simple example of this is the bongo track added in Super Mario World (1990) when you are riding a Yoshi. The musical piece continues playing without interruption, but the bongo track is muted depending on whether you're riding Yoshi or not.
A more sophisticated example can be seen in Banjo-Kazooie (1998), where several tracks are faded in or out depending on your location in the world.
In 140 (2013), music tracks are attenuated, muted/unmuted, as well as filtered based on gameplay progress. For the 4th boss fight, 20 loops are running simultaneously, most of them muted, and then dynamically attenuated, unmuted, and filtered depending on gameplay state.
Synchronized Scoring
In film, 'synchronized scoring' refers to scenes where music is synchronized with the action. If the synchronization is very tight, as in Disney's classic Mickey Mouse cartoon 'Steamboat Willie' from 1928, it is referred to as 'Mickey Mousing'.
The two approaches to creating such scenes are:
- syncing picture to music: The music is composed first, and the film is animated or edited to synchronize with the music. This was used in Disney's 'Fantasia', where animations were coreographed to fit classical music.
- syncing music to picture: The music is composed or edited to synchronized with an already recorded/animated film.
In games, synchronized scoring is also used, but different approaches must be used.
Syncing music to picture
In video games, music can be synced to animations by triggering music based on animation events in a similar fashion to how Foley sound effects are triggered. A famous example of this technique can be found in Dig Dug (1982), and an even more detailed implementation can be found in 'Desert Demolition Starring Road Runner and Wile E. Coyote' (Genesis 1995).
Syncing picture to music (not adaptive music)
Syncing picture to music in video games can be performed by adding metadata in the form of cue points to the music which then is used to trigger animations when they are reached. Note that this is not a form of adaptive music, but rather adaptive animation.
Rhythm games like Vib-Ribbon, Guitar Hero, the Rhythm Heaven series, Beat Saber, and Thumper all synchronize visuals to music in order to represent music-based gameplay in a visually unambigious way.
In 140 (2013), the gameplay adapts directly to the music, by moving level elements in time with rhythmic elements. At the same time, the music adapts to player progress using horizontal re-sequencing and vertical remixing.
Hi-Fi Rush (2023) also adapts the gameplay to the music, but in a fully 3D environment. The game also has rhythm game elements, such as attacking 'on the beat'.
Rayman Legends (2013) has music-based levels where animations are synchronized to a song that plays linearly. Super Mario Bros. Wonder (2023) has similar levels with musical elements such as singing enemies.
Generative Music/algorithmic music
Some adaptive music is generative, but not all, and not all generative music is adaptive. See Generative Music for more information.
References
Game Sound: An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design
Karen Collins (2008)
The MIT PressMickey Mousing
Various authors
WikipediaBeep: A Documentary History of Game Sound
Karen Collins (2016)Clint Bajakian and Michael Land from LucasArts games interview
Clint Bajakian, Michael Land (2019, LucasArts)
channel: Ehtonal Canada, interview for Beep movieSonic College talk '140' - slides
Jakob Schmid (2020)Adaptive Soundtracks in Games
Mark Brown
series: Game Maker's ToolkitIntroduction to Adaptive Music in Games
Simon Hutchinson (2021)
series: Listening to VideogamesHow Did They Do That - Banjo-Kazooie's Dynamic Music
Rob Wass (2015)
channel: ClassicGameJunkie
series: How Did They Do That