Sony patents AI soundtrack generator that adapts game music to your emotions

Sony has filed a patent for a soundtrack generator that matches music to a gamer’s emotions, sparking speculation that the tech could find its way into the PlayStation 5.

The patent, which was first spotted by Gaming Intel, describes an AI-powered system that adapts background music to game scenarios and a player’s activity.

The feature maps emotions such as tension, joy, and fear to musical components including rhythm, melodic structure, and harmonic density, based on published reviews and social media opinions. This data will train algorithms to play sounds that trigger the desired feelings.

According to the patent, composers would first create musical motifs for characters, activities, areas, and even player personalities:

Once the composer has created the motifs, they can be assigned to the elements/characters. This could be done by the composer or sound designers and can be changed as the game is developed or even dynamically inside the game after it has been released.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

The algorithms would then adjust the motifs to the gamer’s style of play, the time of day, state of the game, and the hours that they’ve been playing.

[Read: Nvidia teaches AI to create new version of Pac-Man just by watching gameplay]

It’s not entirely clear how this will work in practice. But the filing suggests that the music’s tempo could be increased if the player’s moving quickly, and then reduced when they’re less active.

Patents offer hints for future gaming

While the filing makes no specific mention of the PS5, its timing suggests that feature could be used in the console.

It joins a growing list of gaming-related patents recently filed by Sony. Others include a controller sensor that measures sweat and heart rates to understand how a player’s feeling and a companion robot that reacts to their emotions.

The new patent suggests that some of these could be combined with the music generator:

Players who have opted in can be tracked on social media, analysis about their personalities can be made based on their behavior and as more and more users take advantage of biometric devices which track them (electrodermal activity, pulse and respiration, body temperature, blood pressure, brain wave activity, genetic predispositions, etc.), environmental customization can be applied to musical environments.

The wackier ideas are less likely to get the green light, but the filings show that Sony sees a big future for real-time personalization in gaming.