Hey everyone,
for the past few months we have been working on a new product, and today we’re finally ready to show it to you.
As you probably know, at NeuralCam we have a photo NightMode app that’s recognized as the leading night mode solution for iPhones without a built-in photo Night Mode. Ever since the launch of our night mode app, a question kept popping up from our users: “can you build a night mode app for video?”, they asked.
We’ve pondered this for a while. Unlike photography, there is no obvious straightforward way to build night mode for video. In the case of photos, night mode usually means taking multiple shots with relatively long exposure and combining them into one high quality image. Then brightening and post-processing it for a final look.
Video is a different beast: you can’t just shoot multiple frames and combine them because you’re limited by both the maximum exposure time a frame can have and by the maximum processing time per frame (milliseconds instead of seconds in the case of real-time video).
Clearly, a very different approach is needed. We’ve explored everything from classic to AI-powered video processing techniques and we think we found the right mix of tech that results in much improved low light videos. After months of researching, hacking and developing, we’re proud to present NeuralCam NightVideo, our (and perhaps the world’s first) Video Night Mode app.
Night Mode Video
So how can we build Video NightMode? Where do we get started and why can’t we just take a normal video and brighten it up? Well, because it would look really bad.
The biggest problem in low light videography (and photography for that matter) is the presence of noise. There’s just too little amount of light hitting the image sensor and this lack of information results in a flickering mess of pixels with different light intensities and colors appearing especially in the darker areas of the scene - not a pleasant view.
In the case of the photography night mode, this noise is reduced by the mentioned frame merging process - which basically results in more information being added to the target frame. However, with video we don’t have the luxury of multiple frames and processing for seconds to do the processing so we’ll clearly have to find a very different approach.
Capturing and AI frame-boosting
The first thing we’re doing is to try to capture the most light inside each frame, which naturally results in less noisy frames.
As a standard practice, videos are shot at a shutter speed of no more than 1/24 to keep moving objects sharp and the video smooth. But this also means that this is the maximum amount of time that light can reach the sensor.
We’ve looked very closely at this problem and figured that there might be a way to let more light in while maintaining the smooth look of the video, namely a combination of longer exposure times and frame interpolation. This way we can use exposure times longer than 1/24 depending on the amount of available light, and then in order to keep the video looking smooth, we generate additional frames using a proprietary Machine Learning powered Video Interpolation algorithm that we’ve built especially for this purpose.
The result is a smooth video with a tiny amount of blur - depending on the used settings - and generally a nice “night video” look.
AI Powered Video Denoising
Doing the capturing part this way we’ve got higher quality frames, but these generally still have a lot of noise so what we’ll do next is to try to get rid of that.
It turns out that classic denoising methods aren’t good enough for this case, they usually blur the whole image, including edges and are helpless at the noise levels present in low light shots taken with a smartphone camera. Also, they’re quite slow as well.
To solve this problem we’ve built a Machine Learning-based denoising algorithm. Specially developed for the iPhone camera, it removes even the heaviest noise and does so selectively and adaptively only on the areas of the image where noise is present. The result is a much cleaner look in the bright areas of the video and a massive reduction of the ugly flickering effect in the darker parts of the image.
Brightening
As a final step after the capturing, interpolation and denoising, it’s finally possible to brighten up the image and get a good looking, smooth video as the result. The amount of brightening applied depends on the selected mode.
Different Video Night Modes
As a final step after the capturing, interpolation and denoising, it’s finally possible to brighten up the image and get a good looking, smooth video as the result. The amount of brightening applied depends on the selected mode.
In order to cover as many shooting situations as possible we’ve come up with 5 different capture modes:
Day:
The first capturing mode is not even a night mode at all. It is targeted at lower light day shots, or scenes with high dynamic range where the results of the default iPhone camera start to get too noisy. This mode contains no brightening, but because of the use of our AI denoising algorithm, resulting videos have a cleaner, more pleasant look. It’s especially useful when used with the ultra wide camera, in the case of which noise can appear even in medium-lit indoor situations.
Dusk:
This mode is designed for semi-low light. It is similar to Day mode, shot at a high frame rate to maintain sharp subjects in high movement situations, but in addition to denoising it also includes a bit of additional brightening.
Night:
The “Night” mode is the default mode for shooting night videos. It uses a slightly longer exposure time, that combined with our denoising and frame interpolation algorithms generates a bright, clean and smooth night video. This is the mode to use for most night scenes.
Night+:
A further enhanced mode for really dark scenes. In this mode, videos have an additional amount of blur but provide a brighter night video with the same clean noise-less look. In most of these situations the built-in camera can't produce usable results.
Time-lapse:
A Night Time-lapse mode, similar to the one on the new iPhone 12 default camera - now made available by NeuralCam NightVideo for older iPhones too. This mode doesn’t have sound and can handle even lower light situations, which combined with a faster playback, generates the specific night mode time-lapse look.
*Gentle Light:
Each of the modes can be combined with the “gentle light” feature, where the app turns on the flashlight, but at one of the lowest levels in order to keep the resulting video still natural and avoid “blinding” people. Gentle light can be used in totally dark scenes with subjects that are close enough to be illuminated by a small amount of light.
Real-time on-device processing
All this processing runs in real-time on the iPhone, making full use of the power the NeuralEngine provides. This real-time processing means that we can provide the same instant video capturing experience you’re used to with your default camera - tap to capture and everything is displayed and processed on the fly. Since this level of video processing requires quite some processing power, NightVideo is only available on devices with the A12 NeuralEngine (iPhone XR and later) and iOS 14.
With real-time processing and modes covering everything from slightly low light settings to really dark scenes, NeuralCam NightVideo is the first app to provide Night Mode for Video.
You can purchase the NeuralCam NightVideo app from the App Store, using the link below. Make sure to send us all your feedback over at hello@neural.cam - we'd love to hear from you.