Cell phone cameras are both hardware and software, and Glass wants to improve both. But while wild anamorphic lenses are creeping onto the market, the company (which is running on $9.3 million in new funding) is using AI to significantly improve image quality without introducing strange AI upscaling artifacts. Released camera upgrade.
GlassAI is a pure software approach to image enhancement called a Neural Image Signal Processor (ISP). ISP essentially takes the raw sensor output (which is often flat, noisy, and distorted) and turns it into the crisp, colorful images we see.
Like phone makers like Apple and Google, ISPs are becoming increasingly complex, compositing multiple exposures, quickly detecting and sharpening faces, adjusting for small movements, and more. Many also include some form of machine learning or AI, but be careful. Using AI to generate details can lead to hallucinations and artifacts as the system attempts to create visual information where no visual information exists. Such “super-resolution” models can be useful in the field, but must be carefully monitored.
Glass manufactures both a complete camera system based on an unusual diamond-shaped front element and the ISP that backs it up. And while the former is working towards a market presence with some upcoming devices, the latter has turned out to be a product worth selling in its own right.
“Our inpainting network corrects optical aberrations and sensor issues while efficiently removing noise, and outperforms traditional image signal processing pipelines in fine texture recovery.” Founder Tom Bishop explained in a news release.
Concept animation showing the process of moving from RAW to Glass processed images.
The word “recovery” is important. Details are not just created; extracted From raw images. Depending on how your camera stack is already working, you may know that certain artifacts, angles, or noise patterns can be reliably resolved or exploited. Learning how to convert these implicit details into real details, or how to combine details from multiple exposures, is an important part of the computational photography stack. Co-founder and CEO Ziv Attar says the company's Neural ISP is better than any other ISP in the industry.
He pointed out that even Apple doesn't have a complete neural image stack and only uses it in specific situations where it's needed, and the results are (in his opinion) not great. He gave an example where Apple's neural ISP couldn't interpret text correctly, and Glass did a much better job.
Photo provided by Ziv Attar. It shows an iPhone 15 Pro Max zoomed 5x and a glass-processed version of his RAW image from the phone.
“If Apple isn't producing decent results, I think it's reasonable to think that's a difficult problem to solve,” he said. “How you train it is more important than the actual stack. We have a very unique method that was developed for anamorphic lens systems and is efficient for any camera. Basically, We have a training lab that includes robotic systems and optical correction systems that allow us to characterize lens aberrations in a very comprehensive way and train networks to fundamentally reverse optical distortions.”
As an example, we shared a case study in which we had DXO evaluate the Moto Edge 40's camera, then install GlassAI and evaluate it again. All Glass-processed images are noticeably improved, and in some cases dramatically so.
Image credits: Glass/DXO
At low light levels, the built-in ISP struggles to distinguish fine lines, textures, and facial details in night mode. With GlassAI, even half the exposure time is as sharp as a tack.
You can peek at the pixels in some of Glass' test photos by toggling between the raw and final photos.
Companies that build cell phones and cameras have to spend a lot of time tweaking their ISPs to make sure the sensors, lenses, and other components all work together properly to deliver the best possible images. But it seems like Glass' one-size-fits-all process might do a better job in a fraction of the time.
“Once you get a new type of device, the time it takes to train the shippable software can vary from a few hours to a few days. For reference, cell phone manufacturers have large teams working together to improve image quality. We spend months tuning our processes. Our process is fully automated, so we can support multiple devices in a matter of days,” said Attar.
Neural ISP is also end-to-end. This, in this context, means processing directly from the sensor RAW to the final image, without the need for additional processes such as denoising or sharpening.
Left: RAW, right: glass processing.
When I asked, Attar was careful to distinguish their work from super-resolution AI services that take a finished image and scale it up. These often end up creating detail where they see fit rather than “recovering” detail, but this process can sometimes produce undesirable results. Glass uses AI, but it's not generative like many image-related AIs.
Today marks the public availability of this product, likely after a long period of testing with our partners. If you're building an Android smartphone, you might want to at least give it a try.
However, on the hardware front, the phone with the strange diamond-shaped anamorphic camera will have to wait until the manufacturer is ready to reveal it to the public.
While Glass is developing its technology and testing customers, it's also busy raising capital. The company just completed a $9.3 million “expansion seed,” which I put in quotes because the seed round was in 2021. The new funding was led by GV, with participation from Future Ventures, Abstract Ventures, and LDV Capital.