US scientist have tapped into the brain’s sound decoder and reconstructed speech from electrical signals recorded as subjects listened to speech.
Led by the University of California, Berkeley, the research involved placing two dimensional arrays of electrodes on the surface of the brain over the superior and middle temporal gyrus – areas close to the ear.
15 subjects were investigated, with electrode positioning piggy-backed on scheduled brain operations for epilepsy.
Recorded waveforms were fed into multiple sophisticated prototype speech reconstruction algorithms, and successful algorithms selected by measuring the mean-square error between the original speech and the reconstructed speech.
“We found that slow and intermediate temporal fluctuations, such as those corresponding to syllable rate, were accurately reconstructed using a linear model based on the auditory spectrogram. However, reconstruction of fast temporal fluctuations, such as syllable onsets and offsets, required a non-linear sound representation based on temporal modulation energy,” said the researchers in their paper on the on the peer-reviewed website PLoSbiology. “Once the model was fit to a training set, it could then be used to predict the spectro-temporal content of any arbitrary sound, including novel speech not used in training.”
It was found that signals from electrodes over the posterior superior temporal gyrus (pSTG) were needed to reconstruct speech.
A minimum of six electrodes were needed, then more electrodes improved reconstruction.
“pSTG is believed to participate in an intermediate stage of processing that extracts spectro-temporal features essential for auditory object recognition and discards nonessential acoustic features,” said the researchers in their paper.
Words could not be reconstructed when the subject mearly thought of the word rather than listened to it.