When you look at a scenic mountain photo typically everything in the distance is in sharp focus. But this scene might be even more captivating if something striking were in the foreground, perhaps a field of wild flowers in peak bloom. The problem is if the flowers are close to the lens relative to the mountains it’s impossible for all elements in the photo to be in perfect focus — if the flowers are sharp, the distant mountains will be blurry and vice versa.
“This limitation is determined by the physics of optics,” explains Peter van Beek, a professor in the David R. Cheriton School of Computer Science. “The camera simply can’t take the picture you want, where everything is in focus, at least not in a single image.”
However, computerized image processing can. If a photographer takes many pictures, each one with a slightly different plane of focus, they can be merged computationally afterwards to create the seemingly impossible — a photo where everything is in sharp focus, from the closest flower to mountain peaks miles away.
“If you’re taking a picture of a field of flowers with mountains in the distance, you focus on the flowers and take a shot, then you focus out a tiny bit and take another shot, and so on until you’ve taken enough pictures that across the set of photos everything in the scene is in focus,” van Beek said.
The computational process to combine the set of images into a single picture is known as photo stacking. Many photo-editing applications such as Photoshop can blend a series of images into a single composite photo, but to do this several pictures are required and they need to be selected carefully one by one.
“The difficulty in doing this manually is that it’s slow and people tend to take far too many pictures,” he said. “Or, conversely, sometimes the photographer is hurried and misses taking a few pictures that are critical for the composite image to be in focus. Taking too many pictures is time consuming and taking too few means the resulting image won’t be sharp.”
Many computer scientists have investigated this problem and have concentrated largely on developing techniques that improve combining the set of images into a single photo. Van Beek and his team of undergraduate research assistants — David Choi, Aliya Pazylbekova and Wuhan Zhou — have instead put their effort toward improving image capturing and selection processes. They have developed an automated method to take the fewest number of images necessary for the set of pictures to be merged into a single all-in-focus photo (see Improved image selection for focus stacking in digital photography for a description of their research).
“You specify what you want in focus, press the shutter, then the algorithm selects the images,” van Beek explained.
Relative to methods other researchers have developed for automated focus stacking, the algorithm van Beek and his students created has as high or better focus accuracy but importantly requires significantly fewer images, an advantage to photographers in the field.
“Our algorithm needs 4.5 times fewer images on average — a reduction that would significantly reduce the time taken to capture the set of pictures,” van Beek said, adding that it would also be a benefit when the images are focus stacked in a computer afterwards.
“Our algorithm reduced post-processing times to focus stack images when using Photoshop by up to ten-fold. With our method, the time to create a focus stacked image ranged from 30 seconds to 3 minutes. Previous methods required up to 30 minutes.”
Focus stacking allows a photographer to create images that would be impossible to capture using conventional photographic methods and equipment — pictures where every element the photographer wants in focus is perfectly sharp. But van Beek says focus stacking is just the beginning of the potential that computational methods can bring to photography.
“Say you go Rome at the height of tourist season and the Colosseum is teeming with tourists and you want to get a picture of that iconic structure but without all the people,” he said. “It’s theoretically possible that the people could be removed computationally. If a series of pictures are taken, the background — in this case, the Colosseum — would be stationary. An algorithm could be developed to detect what’s changing in the series of images — the tourists walking about — and remove those elements from the photo, creating a picture of just the Colosseum.”
The top photograph of flowers is a single image with a narrow plane of focus where only the orange flower at the bottom centre is in sharp focus.
The bottom photo is an all-in-focus image created by focus stacking a series of images, each with a slightly different plane of focus, using the algorithm developed by Professor Peter van Beek and undergraduate research assistants, David Choi, Aliya Pazylbekova and Wuhan Zhou.
