Google Can Now Describe and Answer Questions About Images
Google has announced a series of new features that are designed to support users with disabilities, including an expansion to Lookout which can now describe the contents of a photo and answer questions about it to support those who are blind or have low-vision.
Lookout isn’t a new application — it was originally launched in 2019 and was designed to support those with vision impairments — and it uses artificial intelligence (AI) to, as Google describes, help people accomplish everyday tasks “like sorting mail and putting away groceries.”
Now for a select group of users at launch, Lookout is gaining the ability called “image question and answer” which allows the software to process an image and describe to a user. Users can then pose additional questions about the image which the AI can answer.
At present, Google uses what are known as “alt text” as a key to describing the contents of an image to those with vision impairments. But alt text is not necessarily always used by everyone who uploads an image and even then, alt text descriptions may be inadequate.
“In fact, a Carnegie Mellon study in 2019 found that of 1.09 million tweets, only .01% contained alt text added by content creators, which meant that over 99% of those images were not easily accessible to people who have blindness. Now, AI is helping make images more accessible,” Google says.
Lookout doesn’t rely on alt text or captions at all.
“Now, whether or not images have no captions or alt text, Lookout can process the image and provide a description of it — then people can use their voice or type to ask questions and have a more detailed understanding of what’s in an image. This feature is powered by an advanced visual language model developed by Google DeepMind,” Google explains.
Google says that the rollout of this feature follows months of internal testing with people who are blind or experience poor vision, and the company says it is working with the Royal National Institute of Blind People (RNIB) to invite a limited number of people to test out this feature now. It plans to make it available to even more people “soon.”
Image credits: Google