New technology that can read minds and turn thoughts into complete sentences with the help of AI is giving hope to people who can’t speak.
Researchers from the University of California say their technology is able to translate brain activity into English word by word with the help of machine learning.
The technology could revolutionise the way people who can’t speak or move are able to communicate, as it is more natural than existing tools, the team say.
It has an accuracy rate of 97 per cent – more than twice as high as other brain-signal decoding devices and works by mapping activity of neurons to words.
Translating neurons to words enables it to type word sequences on a computer interface in real time – which can then be read out by a synthetic voice.
Anatomical reconstructions of the four participants showing thoughts highlighted in various areas of the brain that were translated into English
The current process of giving people without a voice the ability to speak can be incredibly slow as it requires monitoring residual eye movement or muscle twitches.
They use these movements to mark letters that spell out words – but it is slow.
Researchers found that in many cases information needed to produce fluent speech is still in the brain and so this technology can help them express those words.
Corresponding author Dr Joseph Makin, of California University in San Francisco, said: ‘Average word error rates are as low as three per cent.’
The study was carried out on four epilepsy patients who had been fitted with brain implants to monitor their seizures.
Their neural activity was turned ‘word by word into an English sentence – in real time,’ said Dr Makin.
Previous techniques have had limited success – with efficiency far below that of natural speech.
In earlier studies using a similar technique they could only decode fragments of spoken words – or less than 40 percent of words in spoken phrases.
So Dr Makin and colleagues used artificial intelligence, or machine learning, to link the behaviour of brain cells directly to sentences.
In the study the participants read up to 50 simple sentences aloud while the electrodes recorded their neural activity.
Examples included ‘Tina Turner is a pop singer’, ‘the oasis was a mirage’, ‘part of the cake was eaten by the dog’, ‘how did the man get stuck in the tree’ and ‘the ladder was used to rescue the cat and the man.’
The brain signals were fed into a computer which created a representation of regularly occurring neural features.
‘These are likely to be related to repeated features of speech such as vowels, consonants or commands to parts of the mouth,’ explained Dr Makin.
Another deep learning technique, known as a recurrent neural network, then decoded them into sentences.
What’s more, the network was able to identify individual words – suggesting learning novel sentences from expanded vocabularies is possible.
In the last decade brain–machine interfaces (BMIs) have enabled some amount of motor function to be restored to paralysed patients.
Although this type of control can be used in conjunction with a virtual keyboard to produce text, even under ideal cursor control which is not currently achievable, the word rate would still be limited to that of typing with a single finger,’ Dr Makin said.
‘The alternative is the decoding of spoken, or attempted, speech, but up to now such BMIs have been limited either to isolated sounds or monosyllables – or, in the case of continuous speech on moderately sized vocabularies of about 100 words, to decoding correctly less than 40 per cent of words.
The software is able to translate brain waves line by line into words on a screen. They can then be read out using a synthetic voice. Stock image
‘In this study, we attempt to decode a single sentence at a time, as in most modern machine-translation algorithms, so in fact both tasks map to the same kind of output, a sequence of words corresponding to one sentence.
‘The inputs of the two tasks, on the other hand, are very different – neural signals and text.’
His team also found brain regions that strongly contributed to speech decoding were also involved in speech production and speech perception.
The approach decoded spoken sentences from one patient’s neural activity with an error rate similar to that of professional-level speech transcription, said Dr Makin.
Additionally, when the AI networks were pre-trained on neural activity and speech from one person before training on another participant, decoding results improved.
This suggests the approach may be transferable across people.
Further research is needed to fully investigate the potential and to increase decoding beyond the restricted language, Dr Makin added.
While mind reading technologies are primarily designed to help the sick, ethical concerns have been raised.
Some experts say they have the potential to be misused on the healthy and used to track people’s thoughts – and relay them back to governments or companies.
The research has been published in the journal Nature Neuroscience.