Hear with Your Eyes: A Review of Live Captioning Glasses and the Vision of Making Conversation Easier for All

Have you ever struggled to communicate in social situations—bars, restaurants, work meetings—due to hearing loss? Wouldn't it be wonderful if real life came with captions, so you could see a transcript of conversations in real-time right before your eyes? Well, this is no longer a futuristic notion, but rather something that is being made possible with augmented reality (AR) glasses.

In today's world, there are automated captions everywhere. We see them on YouTube, Social media platforms, and video communication services like Google Meet, Zoom, and FaceTime. Closed captioning can make a huge difference for people with hearing loss who want to follow a TV show or understand a video call.

AR is an enhanced version of a real-life environment using digital visual elements, sounds, or other sensory stimuli. Live-captioning glasses use AR to project onto your glasses, inserting a discreet captioning service into your real life.

AR captioning glasses can assist you in your everyday communications, meaning you no longer need to rely on body language and speechreading to fill in any missing keywords in conversation and can enjoy communication with relative ease. You simply pop on the captioning glasses and can “see” the conversion happening in your field of vision—just like watching television with captions.

Although captioning glasses have been around for a few years now, initial prototypes were aesthetically “chunky” and would not have been comfortable to wear for long periods. Increasingly, they are becoming more lightweight, and some even boast new innovative features beyond captioning. This means they can be worn for a whole host of different situations, such as conversations, the cinema, theater, lectures, meetings, or when you’re at home watching TV.

As you'll see below, several captioning glasses are currently "under development," and some you can try today. Here's a quick summary table of some of the best, most-promising technology companies when envisioning the new world of captioned glasses, followed by more detailed information about each product.

We'll also continue to update this page as new companies and models in the market appear—so bookmark this page!

Hearing Tracker Audiologist Matthew Allsop provides a review of XRAI Glass and its app. Closed captions are available on this video. If you are using a mobile phone, please enable captions clicking on the gear icon.

XRAI Glass and App

XRAI Glass is a U.S.-based company founded by two British men, Dan Scarfe and Mitchell Feldman. The XRAI name (pronounced x-ray) refers to XR (mixed reality) and AI (artificial intelligence). The idea for the XRAI Glass AR smart glasses originated with an interaction between Scarfe and his 96-year-old grandfather struggling to communicate across the dinner table one Christmas Day due to hearing loss. Scarfe commenced his mission to “Break down communication barriers and promote communication between everyone.”

XRAI software works both with the company's glasses and without, meaning you can benefit from live captioning of speech either through captioning glasses that use the XRAI software or simply just using the XRAI app on your phone or tablet, which you can download for free. The first edition of the XRAI software launched in November 2022 supported the XRAI Glass captioning glasses. Now, the most recent second edition of the software supports three styles of glasses:

1. XREAL Air (adjustable for comfort)
2. TCL NXTWEAR S (featuring removable shades), and
3. Rokid Max (with myopia/nearsightedness adjustment for eyeglass wearers).

The directional microphones on these glasses pick up the voice of the person in front of you and process the information with minimal delay. A live transcript of the conversation is then projected on the inside of the glasses in the center of the screen, which only you can see. If multiple people are speaking at the same time, the AI built into the app will recognize the different voices and label them separately on the screen.

XRAI Glass works with Lensology to ensure all XRAI-driven glasses are prescription lens compatible. You simply tell Lensology the style of glasses you have and send them your prescription; they will mail you an insert to go inside the glasses.

The glasses need to be plugged into a smartphone to work, which means you also need the XRAI Glass app, which is available for Android and iOS. The app is subscription-based with three options: the free Essentials and the Premium and Ultimate which offer wider access to additional features.

Life. Subtitled. A XRAI-glass produced video explaining their product.

XRAI software has the following additional functions:

• Translation: Additional AI providers, including Microsoft and Deepgram, now supplement Amazon and can be selected by users to provide accurate subtitles and translation into 76 languages and 140 dialects, with and without an internet connection.
• Conversation recorder: The app records your conversations, which you can access at your convenience.  XRAI says it has no access to this data, and users have full control and responsibility over how it is used or shared.
• Personal Assistant: XRAI Glass features a ChatGPT-driven personal assistant, allowing the user to ask complex questions with the answer appearing as closed captions.

Though the current models of glasses that are compatible with XRAI Glass require a wire connection to your phone, in the next couple of months the wireless RayNeo X2 glasses are due to be released. These will have the battery and processor built into the glasses.

XRAI Glass has won various awards including: "Technology For Good Award" - Global Business Tech Awards (2023), "Startup Tech Company of the Year" - National Technology Awards (2023), "Best Assistive Technology App" - Assistive Technology Awards (2023), and “Smart Tech Award - Highly Commended” - Disability Smart Tech Awards (2023).

Vuzix Ultralite smart glasses (for OEMs)

Vuzix, an American multinational technology company headquartered in Rochester, NY, is a leading supplier of wearable display technology, virtual reality (VR), and AR. Vuzix has developed an original equipment manufacturer (OEM) platform, meaning that other companies can buy it and use it in their own products.

Vuzix CEO Paul Travers says the company intends to “make the dream of sleek and fashionable smart glasses come true for the broader market." He adds that the new technology platform will be offered to third-party OEMs to deliver a range of new applications: “The Vuzix Ultralite OEM solution will enable leading consumer technology firms to accelerate deployment of AR smart glasses solutions to the masses."

At last year’s Consumer Electronics Show (CES 2023), they showcased their Vuzix Ultralite smart glasses platform. The glasses are designed to be a comprehensive wireless solution that uses Android and iOS mobile phone information for everything from live captioning of speech to other digital AR applications, including language translation, visual/contextual search, text notifications, and instructions. A developer’s kit will be available to bring more capabilities, including step count and heart rate, so that you can track your health throughout the day.

The glasses have advanced monocular waveguide optics and tiny display monitors (the size of a pencil eraser) that work together to create a clear transparent image over the wearer’s real-world view. They are said to have enough battery power to run for two days on a single charge. And, weighing in at less than 1.5 ounces, they are smaller and lighter than previous AR glasses. Vuzix are also prescription-ready and highly customizable boasting “virtually endless configuration options around size, color, form factor and optical setup to set your solution apart.”

When you put the glasses on, they make a connection to your phone via Bluetooth, meaning you no longer have to take your phone out of your pocket or purse. All important information from your smartphone is displayed on the display screen of the glasses. The idea is that you’ll be able to keep your head up and hands-free throughout the day.

The Company has won CES awards for innovation for the years 2005 to 2023 and several wireless technology innovation awards among others.

Vuzix Ultralite glasses are currently not yet available on the market, but you can contact Vuzix for more information on their Vuzix Ultralite OEM platform (also see XanderGlasses below).

XanderGlasses

Xander is a team of audio experts, engineers, and researchers, led by co-founders Alex Westner and Marilyn Morgan Westner. Xander uses AR to enhance in-person conversations to create deeper connections. They have partnered with Vuzix, using the Vuzix Shield model for their first product: Xander captioning glasses designed for people with hearing loss.

After earning an MS from the MIT Media Lab for acoustics and audio processing, Alex Westner spent 20 years in the audio industry, working with industry leaders Gibson, Mitsubishi Research Labs, and iZotope. Diagnosed with a vision condition, he became keenly interested in sensory substitution and “the most profound human problem related to audio”: helping people who have trouble hearing. XanderGlasses emerged as an assistive device to help people with hearing loss better understand sounds and speech.

YouTube video of XanderGlasses in action from Xander cofounder Alex Westner.

XanderGlasses feature an ergonomic design and a cushioned fit. They are simple to use and can be worn straight out of the box—meaning you needn't pair them to your smartphone or install an app. Simply push a button, put them on, and multiple noise-canceling microphones around the frames pick up clear speech from the conversation around you and send speech-bubble captions directly to the lens.

In noisy environments, such as crowded restaurants, Xander claims that their glasses deliver an average of 90% accuracy in captions. Accuracy is even higher in less noisy settings.

The glasses charge via a USB-C port and the estimated battery life is 6-7 hours of conversation time.

XanderGlasses use a hybrid cloud/offline technology. When the glasses are connected to their cloud service via WiFi, they can offer the highest accuracy. However, if you don’t want to connect to WiFi, or are in an environment where you don’t have a strong connection, the glasses will instead use an offline, built-in speech-to-text model that still reportedly delivers high accuracy. Whether online or offline, XanderGlasses do not store your conversations, adhering to SOC 2 standard compliance policies.

The company is currently working on a translation feature, which will be available in the future.

The glasses feature rugged lenses that meet ANSI Z87.1 and EN166/170 safety criteria in case you need to wear them in environments that require safety glasses. You can also get prescription inserts, and clip-on sunglasses will be available for outdoor use. Since the glasses are software-driven, they will continue to be updated and improved over time.

XanderGlasses currently don’t have an exact price point, but the company estimates the price will be comparable to an average pair of prescription hearing aids. The company states that accessibility for all—regardless of income or wealth—is important to them, and they are currently working with organizations and agencies that can help people pay for their products. They’re also working with US veterans’ organizations and audiologists to test and refine the XanderGlasses while helping to restore hard-of-hearing veterans' sense of belonging with families, friends, and communities.

XanderGlasses will be launching exclusively in the Boston area on August 15, 2023. It will be a local event where people are invited to try out and purchase the glasses. All being well, they aim to roll out XanderGlasses in North America by early October and eventually overseas.

As well as receiving a Catalyst Award from the National Academy of Medicine, Xander won the Eureka Park Accessibility Contest organized by the CTA Foundation at CES 2023.

Though they are currently not available to buy, Xander has working prototypes. While they work to refine their design, you can join a waiting list to keep updated on their progress.

TranscribeGlass

Madhav Lavakare from New Delhi, India, came up with the idea of captioning glasses when a friend dropped out of school because he couldn’t afford accessibility accommodations. Lavakare set out to find a solution.

The focus for TranscribeGlass is on low cost in addition to comfort, making it an accessible and affordable assistive technology device for people with hearing loss or difficulties understanding spoken communication. TranscribeGlass is not actually a pair of glasses, but rather a heads-up display that snaps onto your glasses' frames, or a pair of empty frames (provided in the box) if you are not a glasses-wearer.

TranscribeGlass-produced demonstration from YouTube.

The device works by connecting to a companion mobile app, which will be available on both iOS and Android, and from there you can select your favorite speech-to-text software or live captioning source. The captions are then sent to the TranscribeGlass heads-up display via Bluetooth Low Energy. The captions are overlaid onto the user’s vision of the surrounding real-world environment in real-time, and you can adjust the caption font size and position. You can also use the app to view the transcript history. 

Lavakare and his team are busy refining and improving the design of TranscribeGlass. The target weight for the device is under 1 oz and the target price is $95. 

TranscribeGlass Beta is the first limited release of TranscribeGlass and is priced at $55. There is a beta waitlist where early adopters can reserve the device on the website and agree to provide feedback as testers. Feedback from testers will be integral in finding bugs with the product and mobile app and improving the design. 

TranscribeGlass was "Highly Commended" in the Tech4Good Awards 2021 in the Accessibility category, and won the AssisTech Foundation (ATF) Award for "Best AT Start Up for Innovation in 2021”.

CEO Tom Pritsky and intern Desiree recently participated in the 2023 Hearing Loss Association of America (HLAA) convention, where they showcased TranscribeGlass and demonstrated how it works.

LEION Hey AR glasses

Beijing LLVision Technology Co., Ltd has created the Leion Hey AR Glasses to enhance your everyday experiences by providing real-time transcription and translation capabilities, as well as integrated chat functionality. These glasses, designed with the hard of hearing in mind, are wireless all-in-one AR & AI communication enhancers and boast live voice transcription in milliseconds. They are compatible with iOS and Android.

Leion Hey AR Glasses use dual microphones for noise cancellation, which can be targeted for radio reception. In a 70 dB noise environment, the accuracy rate of sound transcription at 5 meters can exceed 85%, and the audio content is converted into text and shown within 500 milliseconds, says the company.

They are lightweight, coming in at 2.8 ounces, so you can wear them for extended periods without discomfort. Integrating a Binocular Light Waveguide Fusion Display, they provide a clear vision of everything that is before your eyes without causing any dizziness. They are also weather resistant so you can wear them in all conditions without worrying about something breaking.

When using the Leion Hey Glasses for the first time, you need to connect them through the companion app to your phone via Bluetooth and WiFi. Next, you register as a user, go through basic settings, and pair your glasses with your device. This only needs to be done once, as the glasses remember the settings for future use. You can control all the functions from the app and switch between them.

Using AR and AI, these glasses bring you three key functions:

• Conversation mode: Here you can find the live captioning feature, which as expected, turns speech into text that is displayed on the screen. These smart glasses also keep a record of the conversations you have with people and the texts that you dictate.
• Translate mode: Leion Hey Smart Glasses are programmed to recognize and translate more than 90+ languages. If you want to use the translation feature, you go into the app and select the language in which the person is speaking. It will translate the speech and convert it to text on the screen in front of you on the inside of the glasses.
• ChatGTP: The ChatGTP function means these glasses act as an AI assistant—you can ask ChatGTP questions and receive the answer on the screen.

The Leion Hey AR Glasses charge through a USB-type C-cable, providing up to 2 hours with full usage, and have 4 styles of sunglasses attachments that connect magnetically to the front of the glasses, so you can wear them in bright conditions. You can also go to an optician to customize your glasses with prescription lenses.

For more information about the Leion Hey AR glasses, visit their Kickstarter campaign.

Will Google (re)launch its AR glasses?

Google had been developing Google Glass smart glasses for several years, and in 2014 a public retail version was released. Google Glass offered a hands-free means of viewing content and performing tasks such as taking phone calls, viewing photos, and checking messages. The technology required lengthy battery life, improved image-recognition capabilities, and a lot of data. Eventually, as of March 15, 2023, they stopped selling the device.

But there could be new developments on the horizon. In May 2022, at the Google I/O developer conference, Google CEO Sundar Pichai announced new prototype AR glasses with a mission to break down communication barriers by creating “subtitles for the world.” These glasses come with Google Translate built-in, and a company video shows AR glasses translating languages and captioning speech in real time. Though this product isn’t quite ready yet, it’s definitely one to watch!

A YouTube video produced by Google that features glasses that use Google Translate. Closed captions are available on this video. If you are using a mobile phone, please enable captions clicking on the gear icon.

Captioning apps

While we are talking about glasses that can provide real-time captioning, it seems relevant to mention some popular captioning apps you can use on your smartphone. And, since some smart glasses such as Vuzix and TranscribeGlass work hand-in-glove with your smartphone and/or your favorite speech-to-text app, the software on your phone may have a direct impact on the captioning capabilities of your AR glasses.

Various smartphone apps can convert speech to text quickly and efficiently, meaning you can benefit from live captioning when you’re on the go.

Check out:

• XRAI Glass (Android and Apple IOs)
• NALscribe: live captions (Apple iOS only)
• Google Live Caption-AI Voice to Text (Android only)
• Google Live Transcribe and Notification (Android only)
• Ava (Android and Apple)
• Hearing Helper - Live Captions (Apple iOS only)
• Otter: Transcribe Voice Notes (Android and Apple iOS)
• Speech to Text Transcribe: Android and Apple iOS

The future of captioning glasses

It’s safe to say we will see more of this type of technology appearing on the market and going mainstream. With AI programs such as OpenAI and Visual ChatGTP growing quickly in capacity, there will no doubt be more demand for wearable AI-compatible solutions in the future. And, with multiple third parties developing applications for Vuzix and other micro-display platform technologies, we can expect to see more competition in the market.

“I expect captioning glasses will continue to evolve physically to address the different needs, usage goals, and individual sensitivities of users,” predicts Xander's Westner.