The Future of Lip Reading

Will your next hearing aids have cameras? New research on lip reading by artificial intelligence suggests that, and more is on the way.

“Read my lips.”  That’s a lot easier said than done.  It’s a difficult skill to master in part because only about 30% of speech is considered “visible”. Even the best lip readers can only understand somewhere between 40% and 60%, and those figures are open to question.

Put another way, it means that about half the time they are wrong.  A point illustrated by an episode of “Seinfeld” where Jerry is dating a deaf woman who relies on lip reading.  He asks her out, and offers to pick her up, “How about six?”.  She looks angry, offended and then leaves.  Jerry discovers later that she thought he had said “sex” instead of “six”.

Now comes news that the University of Oxford in partnership with Google’s DeepMind artificial intelligence program has come up with a system that may help clear up the confusion.

Their AI system was taught to lip read using some 5,000 hours of BBC television clips.  The system scanned people’s lips learning to read them, and it got better and better at it.  In fact, the AI correctly “read” about 47% of what was being said without making a mistake.  By comparison human lip readers barely managed to get 12% right.

You can try it yourself.  Here is one of the silent BBC clips.

Courtesy: Oxford
Courtesy: Oxford

The AI scanned inside the red square and produced these captions.

Courtesy: Oxford
Courtesy: Oxford

It’s a breakthrough that opens up some intriguing possibilities according to another team of researchers at Oxford who are working on a similar system called LipNet.

‘Machine lip readers have enormous practical potential, with applications in improved hearing aids, silent dictation in public spaces, covert conversations, speech recognition in noisy environments, biometric identification, and silent-movie processing.’ -LipNet Research Report

By parsing that statement you can imagine a few scenarios:  One day you may be able to look at your phone and mouth a command to Siri without speaking.  Or you might point your phone’s camera at someone in a noisy room, and have what they are saying dictated directly into your hearing aids via bluetooth.

More ominously it may offer new secret surveillance tools that can “listen” in on distant conversations.  Combine that with facial recognition software and you have a great plot twist in a spy thriller.

In the meantime, a few tips on lip reading.  Actually, the correct term these days is “speech reading” because it involves reading not just lips but facial expressions and gestures.

Anyone with hearing loss is already something of a speech reader since your brain is constantly searching for clues about what is being said.  Your may notice how much easier it is to understand someone if they are facing you directly and in a well lit space.

So to a large extent it’s intuitive but it’s also a skill that can be improved. For online training, try Lipreading.org.

 

 

Apple’s Agenda

Your hearing aids and your phone. Apple and Android are making the connection.

On October 27, 2016 Apple threw the switch on its new “accessibility” website.  In fact, it was the first item on the agenda for CEO Tim Cook’s presentation.  Of particular interest to those of us with hearing loss is the section on bluetooth connectivity between iphones and hearing aids.

This technology has proven indispensable to me.

(For more on how it’s changed my life see my recent review of the Oticon OPN)

Iphones have built-in software that uses bluetooth to send sound directly to your hearing aids. The sounds around you are then automatically lowered, and the sound from your phone is adjusted by your hearing aids to deliver the audio it has been programmed for to compensate for your hearing loss.

The result is much clearer sounding phone calls, music and alerts.

At last count, about 44 models of hearing aids from various manufacturers have bluetooth capability and are compatible with iphone.

Hearing aid makers also offer their own free iphone apps which let you control your hearing aid settings, such as the volume, using your phone.  You can also check your battery levels.

One very cool feature is called “listen live”, which pipes the sound from the phone’s mic into your hearing aids.  That can be a handy option as this new video from Apple demonstrates.

Apple “Live Listen” video

For more on how it all works check out:

Connecting Hearing Aids

For those of you with Android phones, the story is a little more complicated since there is such a great variety of manufacturers.  The best advice is to check out the website of your hearing aid maker and see if their app is compatible with your phone.

 

Steampunk Hearing Aid Art

“Steampunk is a joyous fantasy of the past, allowing us to revel in a nostalgia for what never was. – George Mann

Steampunk is a genre of art that’s best described as a fusion of 19th Century technology, retro-futuristic design, Victorian fashion, science fiction and history that never was.

"The Telectroscope" at London's City Hall
“The Telectroscope” at London’s City Hall

Examples of its influence can be seen in movies such as “The League of Extraordinary Gentlemen”, “Mad Max” and “The Wild, Wild West”, and in art installations such as Paul St. George’s “Telectroscope”.

 

Paris Metro Station
Paris Metro Station

Steampunk also inspires architecture as passengers on the Paris Metro may notice.  And in the fashion world steampunk has emerged as a kind of gothic-punk blend.

A few artists and craftspeople have produced whimsical steampunk hearing aids.  With a few rare exceptions they don’t actually work so they’re more form than function.

Here’s a sampling:

  • "I made this out of a bunch of watch parts my grandfather had (he was a watchmaker)" - Wednesday Apple

A Deaf Man in Paris

A rendezvous with one of the world’s leading geneticists who first identified the hereditary causes of hearing loss.

My French is bad, and my hearing is even worse.  Two good reasons why I didn’t understand what the waiter was asking. So I fessed up.

A cliche shot of me
A cliche shot of me

“Pardonnez-moi, je suis un peu sourd.”  (Pardon me, I am a little deaf)  Second time around I understood, mainly because he was pointing at my empty glass. He was asking if I wanted another.  “Oui”.

Of course the fine wine is a great reason to visit Paris but I had another mission.

Continue reading “A Deaf Man in Paris”