Study shows the possibilities of a future voice prosthesis for humans – sciencedaily
It’s possible to recreate a bird’s song by reading only its brain activity, shows an early proof-of-concept study from the University of California, San Diego. The researchers were able to replicate the intricate vocalizations of the songbird up to the pitch, volume and timbre of the original.
Posted on June 16 in Current biology, the study lays the foundation for building voice prostheses for people who have lost the ability to speak.
“The current state of the art in communication prostheses consists of implantable devices that allow you to generate textual output, writing at up to 20 wpm,” said lead author Timothy Gentner, professor. of Psychology and Neurobiology at UC San Diego. âNow imagine a voice prosthesis that allows you to communicate naturally with speech, saying out loud what you think almost as you think. This is our ultimate goal, and it is the next frontier in functional recovery. . “
The approach used by Gentner and his colleagues involves songbirds such as the zebra finch. The connection to vocal prostheses for humans may not be obvious, but in fact, songbird vocalizations are similar to human speech in a number of ways. They are complex and they are learned behaviors.
âIn the minds of many people, going from a model of a songbird to a system that will eventually enter humans is a pretty big evolutionary leap,â said Vikash Gilja, professor of electrical and computer engineering. at UC San Diego and co-author of the study. “But it’s a model that gives us complex behavior that we don’t have access to in typical primate models that are commonly used for neural prosthesis research.”
The research is a cross-collaborative effort between engineers and neuroscientists at UC San Diego, Gilja Laboratories and Gentner Laboratories working together to develop neural recording technologies and neural decoding strategies that leverage the expertise of two teams in neurobiological and behavioral experiments.
The team implanted silicon electrodes into adult male zebra finches and monitored the birds’ neuronal activity as they sang. Specifically, they recorded the electrical activity of several populations of neurons in the sensorimotor part of the brain that ultimately controls the muscles responsible for singing.
The researchers introduced neural recordings into machine learning algorithms. The idea was that these algorithms would be able to make computer-generated copies of actual zebra finches based solely on the neural activity of birds. But translating patterns of neural activity into patterns of sound is no easy task.
âThere are just too many neural models and too many sound models to find a single solution to directly map one signal to another,â Gentner said.
To accomplish this feat, the team used simple representations of the birds’ vocalization patterns. They are basically mathematical equations modeling the physical changes – that is, changes in pressure and tension – that occur in the finches’ vocal organ, called a syrinx, when they sing. The researchers then trained their algorithms to map neural activity directly to these representations.
This approach, according to the researchers, is more efficient than having to map neural activity to the songs themselves.
âIf you have to model every little nuance, every little detail of the underlying sound, then the mapping problem becomes much more difficult,â said Gilja. “By having this simple representation of the complex vocal behavior of songbirds, our system can learn more robust and generalizable mappings to a wider range of conditions and behaviors.”
The team’s next step is to demonstrate that their system can reconstruct birdsong from neural activity in real time.
Part of the challenge is that the vocal production of songbirds, like that of humans, involves not only the output of sound, but constant monitoring of the environment and constant monitoring of feedback. If you put headphones on humans, for example, and delay the time they hear their own voices, only disrupting time feedback, they will start to stutter. Birds do the same. They listen to their own song. They make adjustments based on what they just heard themselves sing and what they hope to sing next, Gentner explained. A successful voice prosthesis will ultimately have to operate on an equally fast and complex enough timescale to accommodate the entire feedback loop, including making adjustments for errors.
âWith our collaboration,â said Gentner, âwe are leveraging 40 years of bird research to build a voice prosthesis for humans – a device that wouldn’t just convert a person’s brain signals into a rudimentary set of whole words, but that would give them the ability to make any sound, and therefore any word, they can imagine, freeing them to communicate what they want. “