Experimenting at the edge of new technology, multidisciplinary artist and composer Holly Herndon has raised an AI, developed a digital twin Holly+, and is now breaking new ground building tools to help artists protect their IP in the rapidly developing AI landscape.  With a doctorate from Stanford's Center for Computer Research in Music and Acoustics researching “interplay between machine learning and the voice, and the implications of this technology for IP and vocal sovereignty,” and six years of hands-on experience developing her own vocal model, Holly is working to build a future where artists are empowered to work creatively with AI. The key? Consensual training data.

Holly’s voice model experiments began taking shape in 2016 when she started to notice a parallel between two projects she was simultaneously working on at the time: AI research and music ensemble composition for a new album. “I started to draw a through line between really early human technologies like vocal traditions and human coordination techniques and these really contemporary sophisticated technologies like machine learning systems.” Inspired by the observation, Holly and her partner and collaborator Mat Dryhurst set out on their initial experiment to raise an “AI baby” they named Spawn, fed on vocal training data.

“For the early version of Spawn we were already thinking about how the training data itself, much like a child, is a product of their environment–all of the input that you feed that child, like language and the various cultural inputs that you give to that child. In a similar way it’s also like raising a machine learning intelligence. And so we named that intelligence Spawn and took a kind of farm-to-table data approach where we weren’t using other people’s data without permission, instead we decided to create our own.”

Working with a hired ensemble in Berlin, Holly and Mat developed Spawn with training data they either owned or had explicit permission to use. The project culminated in Holly’s acclaimed 2019 album PROTO in which Spawn can be heard as a featured artist, making history as the first-ever record to utilize singing neural networks. “We worked with a group of people in Berlin, formed an ensemble and paid them for their time, and that’s the training data that we used to feed Spawn. So already the kind of embers were in place back then of understanding that the machine learning model is just so directly tied to the training data, and that it matters where that comes from.”

By 2021, the project developed into Holly+, a model developed in partnership with Never Before Heard Sounds, trained entirely using Holly’s voice. The first version of Holly+, available to the public to utilize at holly.plus, is a polyphonic, generative instrument and web interface that converts audio files into what Holly describes as a “kind of chorus of Holly’s singing the track back.” With the release of Holly+ also came the Holly+ DAO as a decentralized mechanism to steward Holly’s voice model. Holly+ DAO token holders vote on what new works to license or certify and mint in order to maintain the value and scarcity of official artworks created using Holly’s voice model. As a proof of concept, the DAO ran an NFT gallery auction on ZORA, approving official works based on submissions from the public splitting the profits between the artist, the DAO, and Holly.

“The Holly+ model creates a virtuous cycle,” writes Holly in her 2021 Mirror post announcing Holly+ and the Holly+ DAO. “I release tools to allow for the creative usage of my likeness, the best artworks and license opportunities are approved by DAO members, profit from those works will be shared amongst artists using the tools, DAO members, and a treasury to fund further development of the tools.”

https://holly.mirror.xyz/54ds2IiOnvthjGFkokFCoaI4EabytH9xjAYy1irHy94

Since then, the Holly+ tools have developed to include a groundbreaking new version of Holly+ –Holly’s “digital twin” capable of singing any song in any language as Holly in her natural sounding voice–as well as Spawning, an organization dedicated to finding solutions to keep AI data training consensual.

“Spawn, our little AI baby, kind of grew up and turned into a verb, which is why we use the term Spawning,” says Holly.

Spawning, the term and its associated organization, refers to the generative nature of AI and the data required to develop it. Having offered her IP to train Holly+, Holly explains spawning as similar to sampling, yet fundamentally different and requiring new tools and a new framework in order to protect and empower artists and copyright holders.

“A lot of people understand what sampling is, what it did for culture, and how it kind of opened up entirely new genres, but also how complicated it was in terms of IP and rights. It kind of required a new framework around it,” says Holly. “We see spawning as a kind of similarly disruptive but also exciting new ability that is fundamentally different from sampling. It’s not a 1 to 1 kind of mechanical reproduction of a sound from someone else kind of remixed into something new, but rather it’s the ability to generate material based on the logic of existing material.”

One tool Spawning has developed is called “Have I Been Trained”—where artists can see if their images are being used to train popular AI art models. The organization is in the process of developing additional tools to help artists opt-in or opt-out of having their data used in AI training. “We are trying to think about a new way to deal with IP and rights and remuneration and accreditation and in this kind of new paradigm where you have spawning along with sampling,” says Holly.

With the new version of Holly+, the vocal model so closely resembles Holly that a new set of possibilities and responsibilities are at play. “The next version of Holly+ we did is a very natural sounding version of my voice. With that comes somehow a different level of responsibility where it does actually sound like me, whereas the other one is a more processed version of the voice,” says Holly.

Holly demonstrated the new version of Holly+ during a TED Talk in Vancouver earlier this year stunning the crowd with a live performance from musician Pher, who sings first as himself (with a regular microphone) and then as Holly using a microphone with Holly+. Holly defines this phenomenon as identity-play or “creating with someone’s likeness or performing as someone else with their permission.”

Further showcasing Holly+’s capabilities, Holly and the Holly+ DAO have just released a video of Holly’s avatar performing a version of the Dolly Parton classic, Jolene generated by Holly+.

“The Holly+ project, a lot of it, of course, is just a creative experiment–a creative endeavor, and us trying to figure out what models work and what are the sticking points for other people to do this as well,” says Holly. “We can be really playful with our own IP, because I feel more comfortable doing that then with somebody else's IP. It’s not always clear how these things are going to work once they're out in the wild.”

These kinds of innovations and conversations bring up both excitement and skepticism as the AI landscape is still in its early formation, often creating more questions than answers. Holly doesn’t shy away from this reality, but instead chooses to find solutions along the way.

“A lot of the fear is valid. The deepfake thing is real and is concerning,” says Holly. “Being able to perform through other people’s likenesses is really exciting, of course, for me, but I can see how people can be really mistrustful of that. A lot of technical development hasn’t always had creators and IP rights holders’ best interests at heart, so I think a lot of people have been burned many times, and so they are rightly skeptical.”

Holly–who refers to the traditional music industry as a “dumpster fire,” is cautiously optimistic about the potential of the new internet on the grounds that it requires the community to be proactive in building a sustainable framework to protect creators.

“I do think that there has been an overemphasis on the deepfake narrative. If done in a consensual way, it could be exciting and cool, but it's all about having consent at the end of the day,” says Holly. “And honestly a lot of people we’ve interacted with in that space, whether they are a private company or research institute or a nonprofit, all want to have some way to have consensual data, it’s just really messy and difficult right now.”

The blockchain can play a vital role as far as the capability of tracking IP and creating a financial pathway back to copyright holders if stewarded correctly. Holly hopes builders will take note of web2’s mistakes and commit to building an ecosystem where artists' work is properly valued.

“One of the great promises of web3 was that we would move away from this kind of data mining, advertising, pyramid scheme web2 model, where creators are competing for likes in models that simply don't work,” says Holly. “When I get nervous is the conversation around NFT royalties—people coming out saying, ‘well you know we’re not going to respect royalties on our platform,’—I start to get reminded of how Spotify comes about.”

While Holly continues to build on the new frontier of AI art with the implications of technology on IP front and center in her work, she hopes builders in web3 will do the same.

“There are so many young people building web3, and I hope they will look at what went wrong with web2, and how so many creators were excited to leave web2—to join web3—and can honor that, and not be short-sighted to have short-term gain. Because if we don't build better ecosystems for creators then we’re going to be on to web4 or whatever’s next because it’s just not sustainable.”

With an intimate knowledge of the technology, algorithms, and systems that underpin creative production, Holly is always finding ways to inject humanity into her work and her vision for the future. Seizing the means and leaving a trail of heady, conceptual works and tools for others to follow suit, Holly is blazing the trail into the great unknown of art and tech’s potential while realizing her own.

****

By Wallace Morgan for “High Frequency,” NOISE’s weekly newsletter. Subscribe for more.

High Frequency | NOISE DAO | Substack

a newsletter covering the emerging web3 music ecosystem. Brought to you by NOISE, a DAO supporting music and technology. Click to read High Frequency, by NOISE DAO, a Substack publication with hundreds of subscribers.

https://noisedao.substack.com

Experimenting at the edge of new technology, multidisciplinary artist and composer Holly Herndon has raised an AI, developed a digital twin Holly+, and is now breaking new ground building tools to help artists protect their IP in the rapidly developing AI landscape.  With a doctorate from Stanford's Center for Computer Research in Music and Acoustics researching “interplay between machine learning and the voice, and the implications of this technology for IP and vocal sovereignty,” and six years of hands-on experience developing her own vocal model, Holly is working to build a future where artists are empowered to work creatively with AI. The key? Consensual training data.

Holly’s voice model experiments began taking shape in 2016 when she started to notice a parallel between two projects she was simultaneously working on at the time: AI research and music ensemble composition for a new album. “I started to draw a through line between really early human technologies like vocal traditions and human coordination techniques and these really contemporary sophisticated technologies like machine learning systems.” Inspired by the observation, Holly and her partner and collaborator Mat Dryhurst set out on their initial experiment to raise an “AI baby” they named Spawn, fed on vocal training data.

“For the early version of Spawn we were already thinking about how the training data itself, much like a child, is a product of their environment–all of the input that you feed that child, like language and the various cultural inputs that you give to that child. In a similar way it’s also like raising a machine learning intelligence. And so we named that intelligence Spawn and took a kind of farm-to-table data approach where we weren’t using other people’s data without permission, instead we decided to create our own.”

Working with a hired ensemble in Berlin, Holly and Mat developed Spawn with training data they either owned or had explicit permission to use. The project culminated in Holly’s acclaimed 2019 album PROTO in which Spawn can be heard as a featured artist, making history as the first-ever record to utilize singing neural networks. “We worked with a group of people in Berlin, formed an ensemble and paid them for their time, and that’s the training data that we used to feed Spawn. So already the kind of embers were in place back then of understanding that the machine learning model is just so directly tied to the training data, and that it matters where that comes from.”

By 2021, the project developed into Holly+, a model developed in partnership with Never Before Heard Sounds, trained entirely using Holly’s voice. The first version of Holly+, available to the public to utilize at holly.plus, is a polyphonic, generative instrument and web interface that converts audio files into what Holly describes as a “kind of chorus of Holly’s singing the track back.” With the release of Holly+ also came the Holly+ DAO as a decentralized mechanism to steward Holly’s voice model. Holly+ DAO token holders vote on what new works to license or certify and mint in order to maintain the value and scarcity of official artworks created using Holly’s voice model. As a proof of concept, the DAO ran an NFT gallery auction on ZORA, approving official works based on submissions from the public splitting the profits between the artist, the DAO, and Holly.

“The Holly+ model creates a virtuous cycle,” writes Holly in her 2021 Mirror post announcing Holly+ and the Holly+ DAO. “I release tools to allow for the creative usage of my likeness, the best artworks and license opportunities are approved by DAO members, profit from those works will be shared amongst artists using the tools, DAO members, and a treasury to fund further development of the tools.”

https://holly.mirror.xyz/54ds2IiOnvthjGFkokFCoaI4EabytH9xjAYy1irHy94

Since then, the Holly+ tools have developed to include a groundbreaking new version of Holly+ –Holly’s “digital twin” capable of singing any song in any language as Holly in her natural sounding voice–as well as Spawning, an organization dedicated to finding solutions to keep AI data training consensual.

“Spawn, our little AI baby, kind of grew up and turned into a verb, which is why we use the term Spawning,” says Holly.

Spawning, the term and its associated organization, refers to the generative nature of AI and the data required to develop it. Having offered her IP to train Holly+, Holly explains spawning as similar to sampling, yet fundamentally different and requiring new tools and a new framework in order to protect and empower artists and copyright holders.

“A lot of people understand what sampling is, what it did for culture, and how it kind of opened up entirely new genres, but also how complicated it was in terms of IP and rights. It kind of required a new framework around it,” says Holly. “We see spawning as a kind of similarly disruptive but also exciting new ability that is fundamentally different from sampling. It’s not a 1 to 1 kind of mechanical reproduction of a sound from someone else kind of remixed into something new, but rather it’s the ability to generate material based on the logic of existing material.”

One tool Spawning has developed is called “Have I Been Trained”—where artists can see if their images are being used to train popular AI art models. The organization is in the process of developing additional tools to help artists opt-in or opt-out of having their data used in AI training. “We are trying to think about a new way to deal with IP and rights and remuneration and accreditation and in this kind of new paradigm where you have spawning along with sampling,” says Holly.

With the new version of Holly+, the vocal model so closely resembles Holly that a new set of possibilities and responsibilities are at play. “The next version of Holly+ we did is a very natural sounding version of my voice. With that comes somehow a different level of responsibility where it does actually sound like me, whereas the other one is a more processed version of the voice,” says Holly.

Holly demonstrated the new version of Holly+ during a TED Talk in Vancouver earlier this year stunning the crowd with a live performance from musician Pher, who sings first as himself (with a regular microphone) and then as Holly using a microphone with Holly+. Holly defines this phenomenon as identity-play or “creating with someone’s likeness or performing as someone else with their permission.”

Further showcasing Holly+’s capabilities, Holly and the Holly+ DAO have just released a video of Holly’s avatar performing a version of the Dolly Parton classic, Jolene generated by Holly+.

“The Holly+ project, a lot of it, of course, is just a creative experiment–a creative endeavor, and us trying to figure out what models work and what are the sticking points for other people to do this as well,” says Holly. “We can be really playful with our own IP, because I feel more comfortable doing that then with somebody else's IP. It’s not always clear how these things are going to work once they're out in the wild.”

These kinds of innovations and conversations bring up both excitement and skepticism as the AI landscape is still in its early formation, often creating more questions than answers. Holly doesn’t shy away from this reality, but instead chooses to find solutions along the way.

“A lot of the fear is valid. The deepfake thing is real and is concerning,” says Holly. “Being able to perform through other people’s likenesses is really exciting, of course, for me, but I can see how people can be really mistrustful of that. A lot of technical development hasn’t always had creators and IP rights holders’ best interests at heart, so I think a lot of people have been burned many times, and so they are rightly skeptical.”

Holly–who refers to the traditional music industry as a “dumpster fire,” is cautiously optimistic about the potential of the new internet on the grounds that it requires the community to be proactive in building a sustainable framework to protect creators.

“I do think that there has been an overemphasis on the deepfake narrative. If done in a consensual way, it could be exciting and cool, but it's all about having consent at the end of the day,” says Holly. “And honestly a lot of people we’ve interacted with in that space, whether they are a private company or research institute or a nonprofit, all want to have some way to have consensual data, it’s just really messy and difficult right now.”

The blockchain can play a vital role as far as the capability of tracking IP and creating a financial pathway back to copyright holders if stewarded correctly. Holly hopes builders will take note of web2’s mistakes and commit to building an ecosystem where artists' work is properly valued.

“One of the great promises of web3 was that we would move away from this kind of data mining, advertising, pyramid scheme web2 model, where creators are competing for likes in models that simply don't work,” says Holly. “When I get nervous is the conversation around NFT royalties—people coming out saying, ‘well you know we’re not going to respect royalties on our platform,’—I start to get reminded of how Spotify comes about.”

While Holly continues to build on the new frontier of AI art with the implications of technology on IP front and center in her work, she hopes builders in web3 will do the same.

“There are so many young people building web3, and I hope they will look at what went wrong with web2, and how so many creators were excited to leave web2—to join web3—and can honor that, and not be short-sighted to have short-term gain. Because if we don't build better ecosystems for creators then we’re going to be on to web4 or whatever’s next because it’s just not sustainable.”

With an intimate knowledge of the technology, algorithms, and systems that underpin creative production, Holly is always finding ways to inject humanity into her work and her vision for the future. Seizing the means and leaving a trail of heady, conceptual works and tools for others to follow suit, Holly is blazing the trail into the great unknown of art and tech’s potential while realizing her own.

****

By Wallace Morgan for “High Frequency,” NOISE’s weekly newsletter. Subscribe for more.

High Frequency | NOISE DAO | Substack

a newsletter covering the emerging web3 music ecosystem. Brought to you by NOISE, a DAO supporting music and technology. Click to read High Frequency, by NOISE DAO, a Substack publication with hundreds of subscribers.

https://noisedao.substack.com