George Barbastathis

Can you picture etchings on a transparent object, in a completely dark room? Granted, it's hard to just imagine this object in your mind's eye, and you wouldn't actually see anything at all inside such a room. 

But a camera can produce a credible image, given sufficient computational help, says George Barbastathis, MIT professor of mechanical engineering, whose experiments have achieved this surprising result. 

Barbastathis collaborates with colleagues in academia and industry on computational imaging, which combines optical engineering with machine learning and other advanced algorithms to solve such near-impossible imaging problems.  

Some of these techniques appear headed for roles as varied as manufacturing automotive parts, counting cells for medical treatments, and scrutinizing new artificial materials. 

"If our friends in industry have an imaging problem, especially an imaging problem that seems to have no solution, they should come to us!" he says.

Machine learning for images 

Many AI technologies such as machine learning first demonstrated their potential against challenges in imaging. "In fact, many of the mathematical methods used in machine learning actually originated in imaging science in the 1960s," Barbastathis remarks. 

Today, machine learning and other forms of highly sophisticated algorithms are embedded in many imaging applications, including the facial recognition capabilities that are featured in hundreds of millions of cell phones. 

In most of these commercialized AI apps, such as facial recognition, the job of the computer is to interpret the image. "But many imaging systems cannot, on their own, produce very good images," Barbastathis says. His lab often focuses on overcoming that problem of improving images. 

Many of these puzzles are brought up by imaging systems that lack a lens. If a system would need a very large and heavy lens, "you can consider getting rid of the lens and trying to compensate by computation," Barbastathis says. This is no easy task, but we now have the sheer computing power to model the very complex operations of the lenses, he says, and algorithms for machine learning and related mathematical operations can perform very elaborate image creation. 

To achieve the clear reproduction of transparent etchings in a dark room, which was presented in a 2018 Physical Review Letters paper, he and his team combined machine learning with an algorithm based on physics. 

While solving such problems may seem impossible or even a little crazy, it's doable for two reasons, he says. 

First, the sensitive electronics in the camera do capture a tiny amount of light in that very dark room, although the images are extremely noisy and the pixels don't appear to fit any pattern at all. Second, even though these scattered pixels “appear random to your eyes, and might even appear random to a traditional algorithm, there are mathematical relationships that are satisfied between those neighboring pixels," Barbastathis says. "We can design algorithms that can exploit this information and form an image." 

He and his collaborators also have presented a striking conceptual solution for a quite different problem: visualizing integrated circuits in extreme close-up, using the Chicago-based Argonne National Lab’s synchrotron, which acts as an extremely sophisticated X-ray source. 

"It's very, very difficult to see what's inside an integrated circuit; the features are incredibly small," Barbastathis says. "But we have shown using machine learning that you can actually make a map of what's inside the integrated circuit very efficiently and very accurately." 

“Synchrotrons are gigantic instruments, the size of a football field, and obviously very expensive to run and maintain," he says. Making this approach practical for chip inspection will require vastly less expensive devices; some preliminary evidence suggests that sources that fit in a room eventually may be possible. 

Industrial imaging 

On the commercial front, Barbastathis has developed imaging algorithms for Weichai, an automobile parts manufacturer based in China. Some of these imaging tools aid quality control in manufacturing, while others can screen pictures of damaged cars for fraudulent insurance claims. 

Another manufacturing project is part of an umbrella program between the MIT School of Engineering and Takeda Pharmaceuticals. Takeda needs to maintain very tight control of humidity in making pills, and Barbastathis and colleagues are working with drug company engineers to optically characterize the drying process. This technology may soon be adopted in production. 

Barbastathis also collaborates with Krystyn Van Vliet, professor of biological engineering, to examine the growth of stem cells that eventually may be used in regenerative medicine. 

Measuring how quickly biological cell populations grow is crucial, but that's hard to do because cells are almost always transparent. Measurement traditionally is performed either by physically sampling the cells or by genetically modifying the cells to make them fluorescent, and neither approach is at all ideal for actual medical use. 

But each cell's nucleus sticks up a little, just enough to be detected by clever computational imaging. "Simply by counting the nuclei of the cells, we can tell our colleagues how fast their cultures grow," he says. This program is being run with several pharmaceutical companies that are interested in cell manufacturing. 

Additionally, Barbastathis collaborates with Nicholas Fang, professor of mechanical engineering, in developing artificial materials through stereolithography, a form of 3D printing using photochemical processes that requires close interaction between material science and imaging. “We have to pattern the materials in very, very demanding ways," he says. "Imaging them is challenging too, because these are very tiny and not necessarily friendly to light." 

SMART work 

Barbastathis is active in the Singapore/MIT Alliance for Research and Technology (SMART) program. In SMART, "groups of professors that typically come from many departments get together, we identify a very important societal application, we apply for funding, and then we do the research in both Singapore and MIT," he explains. 

One field of study is precision agriculture. Singapore is tiny and densely populated, and imports most of its produce. The country would like to strengthen its options in urban farms, installed on high-rise buildings. Barbastathis and his SMART co-workers are partnering with agricultural firms to investigate ways to best monitor the health of growing crops. 

As at MIT, his Singapore collaborations cover a broad spectrum. In one project, his lab is helping to study the origins of glaucoma, a major cause of blindness. In another, his group is contributing to efforts to characterize the health of coral reefs, perhaps by combining optical and ultrasound surveys. 

Applying fundamental findings 

Surprisingly often, Barbastathis says, a research question that appears very fundamental and curiosity-driven suddenly spins up an idea that may have commercial implications. That's one reason he encourages his students to do both fundamental and applied work. 

Corporate partnerships are most likely to succeed when partners can fully provide the parameters and the context for their imaging challenges, he emphasizes. If MIT researchers know the exact challenges corporations want to overcome, "sometimes we can pull off these little miracles," he says. "We cannot solve all of these problems, but maybe we can solve some." 

Often, Barbastathis adds, it pays off for companies to take a little more risk, even addressing imaging problems that may seem insurmountable. 

"An argument can be made that the really risky stuff should be funded by the government, but I don't think that's always the case," he comments. "In many cases it pays off for industry to take more risk, for the very simple reason that they will be immediately ready to reap the reward."