The Art of Stitching Spherical Content
By Alexandre Jenny, Senior Director of Immersive Media Solutions
When it comes to capturing in spherical/360-degrees, one of the keys to delivering truly immersive content, as well as the ability to do OverCapture, lies in stitching. Stitching, put simply, is the process of combining multiple images with overlapping fields of view. The process that’s required to arrive at the perfect stitch, however, is very complex.
Whether stitching for immersive/VR content or for content on a standard flatscreen utilizing OverCapture, the image output must be clear, undistorted, and seamless. In both cases (immersive and OverCapture), if the stitch isn’t right, it is immediately noticeable to the viewer and instead of immersing them in the scene of a story, they are distracted by the imperfections of the image. This is why we spend so much time and energy on getting this right.
In order to create a seamless 360-degree image, it is essential that the entire space around any given scene is captured. The first key feature of GoPro’s new spherical camera, GoPro Fusion, is its offset lenses. The lenses are offset on each side to ensure that, together, all 360 degrees are captured in any given scene. While a back-to-back lens orientation would be straight forward, it increases the disparity resulting in stitching issues. The offset lens design used in Fusion allows the use of the best lenses, which are typically a little longer, while reducing the image disparity between the lenses. This orientation of the lenses is more effective if we consider the stitch as a part of the equation. Of course, each lens on Fusion captures >180 degrees in field of view. The combination of that offset and >180 degree field of view makes it possible to create a high quality, immersive spherical image that is seamless.
Once you have captured all 360-degrees of a scene, you must now bring the two images together. This is where stitching comes in. Think of two pieces of fabric. In order to connect them you must stitch them together. As the thread brings the two pieces together, you transform the two pieces into one. While you now have one piece of fabric, there is a seam that joins the two together. In a high quality, stitched image, that seam is invisible. Visible stitch lines can interrupt the immersive experience— and that is why our team has labored to create the most advanced stitching technology called D.WARP that not only joins the two image files, it composes them into a seamless, unified whole…and that means an immersive experience.
Introducing D.WARP: While stitching the images together is a complex process, making them appear as an accurate representation of a 360 scene is an even more nuanced operation. It’s more than just lining up each image, pixel by pixel— this requires several advanced algorithms to account for a number of variations that could disrupt the immersive experience. D.WARP is one of them. Since Fusion has two lenses, one on each side, the camera produces two video files, one corresponding to each lens, from 2 slightly different viewpoints. The resulting files are fisheye representations of the captured scene again from 2 slightly different viewpoints. Our proprietary D.WARP algorithm transforms these fisheye images into the spherical space, by understanding depth to minimizing visual artifacts coming for viewpoint difference. This is all done dynamically and in a time coherent manner which is key for video. D.WARP, this dynamic content aware stitching is the first of its kind and is on the forefront of 360 stitching.
What about Blending? Once the image warping has been corrected with D.WARP , the Blending Algorithm is applied to merge several source pixels together creating a more equalized image. This stage is about adjusting for exposure and contrast differences. D.WARP and good blending algorithm together are important to produce a seamless output.
As the industry advances and viewers crave new ways to experience content and tell stories, we strive to push the envelope and give them the tools they need to relive their favorite memories and make that experience feel as real as possible. Our goal has always been to make the process from shoot, to stitch, to share, seamless… and I’m excited to say with GoPro Fusion we’ve done just that.