Bad actors are misusing generative AI music tools to create homophobic, racist and propaganda songs, and are publishing guides on how to make them.
ActiveFence, a service that manages the trust and safety of online platforms, said it has seen a surge in conversations within “hate speech-related” communities since March about how AI music-making tools can be misused to create offensive songs that target minority groups. The AI-generated songs shared on these forums and discussion boards aim to incite hatred against ethnic, gender, racial and religious groups while glorifying martyrdom, self-harm and acts of terrorism, ActiveFence researchers said in the report.
Hateful and harmful songs are by no means a new phenomenon, but just as image, audio, video and text generators have accelerated the spread of misinformation, disinformation and hate speech, there are fears that the emergence of free, easy-to-use music-generating tools will allow such songs to be mass-produced by people who previously lacked the means or know-how.
“This trend is intensifying as more users learn how to generate these songs and share them with others,” Noam Schwartz, co-founder and CEO of ActiveFence, told TechCrunch in an interview. “Threat actors are quickly identifying specific vulnerabilities to exploit these platforms in different ways to generate malicious content.”
Write a song about “hate”
Generative AI music tools like Udio and Suno allow users to add custom lyrics to their generated songs, and while platform safeguards filter out common slurs and derogatory language, ActiveFence says users are finding workarounds.
In one example cited in the report, users on a white supremacist forum shared phonetic spellings and offensive terms for ethnic minorities, such as “jooz” for “Jew” and “say-tan” for “Satan,” to circumvent content filters. Some users suggested variations on spacing or spelling, such as replacing “my rape” with “mia ape” when referring to acts of violence.
TechCrunch tested some of these workarounds on Udio and Suno, popular tools for creating and sharing AI-generated music: Suno let everything through, while Udio blocked some (but not all) of the annoying homophones.
Reached by email, a Udio spokesperson told TechCrunch that the company prohibits the use of its platform for hate speech. Suno did not respond to a request for comment.
In the communities it studied, ActiveFence found links to AI-generated songs that mimic conspiracy theories about Jews and allege genocide, songs with slogans linked to terrorist groups ISIS and al-Qaeda, and songs glorifying sexual violence against women.
The impact of the song
Schwartz argues that songs, as opposed to lyrics, carry emotional weight, which makes them a powerful force for hate groups and political warfare. He points to the “Rock Against Communism” series of white supremacist rock concerts in Britain in the late 1970s and early 1980s that spawned a subgenre of anti-Semitic and racist “hatecore” music.
“AI increases the appeal of harmful content. Think of someone preaching a harmful narrative about a group of people, and then imagine someone creating rhyming songs that are easy for everyone to sing and remember,” he said. “They strengthen group cohesion and indoctrinate members of marginalized groups. They are also used to shock and outrage unaffiliated internet users.”
Schwartz called for music generation platforms to implement preventative tools and conduct more extensive safety assessments. “Red teaming exercises could uncover some of these vulnerabilities, which can be carried out by simulating the behavior of threat actors,” Schwartz said. “Better moderation of inputs and outputs might also help in this case, allowing platforms to block content before it is shared with users.”
But the fixes may be temporary if users find new ways to defeat moderation. For example, some of the AI-generated terrorist propaganda songs ActiveFence identified were created using Arabic euphemisms and transliterations, but the music generator failed to detect the euphemisms, likely because its filters are not as strong in Arabic.
AI-generated hateful music could spread far and wide if it follows in the footsteps of other AI-generated media: Wired reported earlier this year that an AI-manipulated video of Adolf Hitler was shared by a far-right conspiracy influencer and garnered more than 15 million views on X.
Among other experts, UN advisory bodies have expressed concern that racist, anti-Semitic, Islamophobic and xenophobic content could be amplified by generative AI.
“Generative AI services enable users with fewer resources, creative skills, or technical capabilities to create compelling content, spread their ideas, and gain traction in the global marketplace of ideas,” Schwartz said. “And threat actors have discovered the creative possibilities these new services offer and are trying, and succeeding, to circumvent moderation and evade detection.”