Hey everyone,
Today is a big day for NeuralCam. We have three announcements to make: some news about NeuralCam as a company, a major update for our NightMode app and a brand new Smart Camera app.
NeuralCam - building Smart Camera Apps
Let’s start by talking a bit about cameras and AI.
At NeuralCam, we think that one of the biggest leaps in technology over the past years has been the advances Deep Learning brought in the area of Computer Vision. It gives computers the ability to see - to understand patterns, images and video similarly to how people do.
Ultimately, we think this technology will give birth to a new category of cameras - Smart Cameras - that will have a profound impact on every activity that involves capturing and processing images.
We see this change similar to the transformation from classic cellular phones to the modern smartphone we witnessed 10 years ago. We believe that martphones themselves seem to be the ideal smart camera platform: they have high quality cameras and enough processing power to run modern AI algorithms on-device.
So, we can all have our own personal smart cameras in our pockets, all that’s needed is the right ML software. This ML software is what we call Smart Camera Apps, and these are the type of apps we’re building at NeuralCam.
Whenever we see a visual task we might say “there’s a Smart Cam App for that”. We’ve been doing a lot of research and prototyping and we’re more and more confident that smart camera apps will play a key role in every industry, from Education through Healthcare to Entertainment, and we can’t wait to contribute something in each of these areas.
However, the first category that is naturally most likely to change is almost analogous to the camera itself: photography and videography. Thus, this is what we’ve decided to start with, building AI powered Photo & Video apps over the next few years.
Our first smart photo app we’ve launched is the NeuralCam NightMode app, which brought Night Mode to all iPhones and all iPhone camera lenses. But, NeuralCam is more than the Night Mode app. We think of NeuralCam as a range of smart camera apps in different categories for various use cases.
So, one of the things we did is update our brand to reflect this change. This is the first thing we’d like to show you today, a fresh NeuralCam:
And this is the new NeuralCam NightMode:
Ok, so that’s the story of NeuralCam - the company. The second thing we want to show you is an update to our NightMode app.
NeuralCam NightMode
We launched our first smart camera app, NeuralCam NightMode, about a year ago, last August. It brought NightMode to the iPhone and the response from the users and critics was amazing.
We’ve been covered in many major press publications, got lots of love letters from users, and been recognized as the best NightMode app out there.
While Apple has launched a built-in night mode on the iPhone 11 in the meantime, NeuralCam continues to be the NightMode solution for devices or iPhone cameras without a native NightMode - like the newly launched iPhone SE or even the UltraWide & Selfie cameras on the iPhone 11 Pro.
Since its launch, people have used NeuralCam NightMode to take more than 10M low light photos.
Now, with a major update, we have made our NightMode app even better: we've made some major improvements to our algorithm so that it supports higher resolutions on older devices too; we’ve updated the UI to be in-line with the new NeuralCam brand and we’ve officially added iPad support.
This is how the update NeuralCam NightMode app looks like on an iPad:
NeuralCam Live
The popularity of video calls has exploded lately. In the current global context, most of us use video calls a lot for both getting our job done or to keep in touch with loved ones. For something used as often, we really need to have the best possible tools.
Looking at the things out there, we realized that there’s a weak link in all this video call experience, and it’s the webcams.
We’ve got laptop webcams (quite bad), dedicated webcams (a bit better) or apps that let you use your iPhone as a webcam (best quality). However, these products really only differ from one another in the resolution of the video they provide.
The problem with webcams is that they’re mostly the same as they were 10 years ago, they do the same thing only slightly better. In the meantime, this cool new technology called Deep Learning came along which could add a whole new dimension to webcams and possibly change the whole video call experience - but nobody uses it.
It turns out that one of the main reasons for Deep Learning not being used with webcams is that no webcam has the processing power required to run the required ML algorithms. Except there is one device which, when converted to a webcam, could do all this: the iPhone in our pocket.
It’s got one of the best cameras in the smartphone industry and with its fast processors and dedicated machine learning hardware, it is capable of running even video processing machine learning algorithms in real-time. It just needs the right SmartCam App software to do all this.
We've built this software - we call it NeuralCam Live and it turns the iPhone into the world’s smartest webcam.
Ok, so how are we going to do that?
Technology
First, we need the software that connects the iPhone to the laptop and makes it appear just like a regular webcam when connected through a USB cable.
This software has two modules: the first one is an iOS app that provides the webcam functionality, while the second one is a plugin that’s installed on the laptop to make the iOS app appear as a webcam. Once you install both of these components, all you have to do is select the NeuralCam Live Camera in Zoom, Google Meet or your video call platform of choice.
So, that was quite easy, now comes the fun part.
The second component we need is the machine learning software to make our smart webcam... well, smart. We call it the LiveEngine
Machine learning, as a generic technology, can help us in two ways when it comes to webcams: we can use it to recognize things in the video feed or generate new video content, improved in some way.
We’ve developed multiple machine learning modules, some proprietary, some built on top of existing CoreML models and we’ve grouped them in three different machine learning frameworks that cover both video generation and analysis: the Style Framework, the Enhance Framework and the Gesture Framework. Let’s take a quick look at each of these.
The Style Framework is a deep learning-based machine learning framework that edits video similarly to a professional colorist.
Its Neural Nets were trained on images edited by hand by professionals in a range of different editing styles. The results are quite amazing, generating pro-looking videosf, very close in terms of style to the professionally edited frames it was trained on and it does all this in real-time at 30fps locally, on the user’s iPhone.
The Enhance Framework contains our set of machine learning models that can enhance a video in a bunch of different ways.
Our real-time video denoising and brightening models in this category achieve levels of quality and performance that are much better than traditional algorithms and make features such as video low light enhancement or face smoothing possible.
The Recognition Framework is the part of our LiveEngine that is meant to make our smart camera apps understand what they “see”.
It uses a combination of machine learning models provided by the iPhone SDK and adds a number of proprietary elements to detect faces, user gestures or other different patterns in the live video feed. In the app, we can use this type of technology to blur unwanted gestures or guide you to set up an ideal environment for a call.
Together, all these elements form what we call the LiveEngine - the ever-evolving software module that makes NeuralCam Live a smart webcam.
Features
Now, let’s take a look at the actual product.
The iPhone app exposes all of the functionality of the smart iPhone webcam in a simple layout, most of the features can be accessed by just tapping a button on the screen. It also shows a nice preview of the current video feed.
So, using all this technology, what more can a smart webcam do compared to a traditional webcam?
In terms of the outcome they provide to the user, we’ve realized that most of the AI features we’ve built fit nicely in two categories: making you look better and helping you avoid embarrassing mistakes.
Let’s take a look at these features one-by-one:
Face & Image Quality Meter
The Face & Image Quality Meter detects the amount of face lighting, type of face position and background and displays an overall Face Image Quality score.
It helps you set up an optimal video-call environment and helps you avoid making typical mistakes, such as sitting with your back towards a bright window or in an unfavorable low angle.
Pro Video Styling
Using the Style Framework, NeuralCam Live edits the video feed like a professional colorist would, in a range of different styles. The end-result is a more impactful look, allowing you to choose the style that best fits your mood, the type of call or the environment you're in.
Low-light Mode
It wouldn't be a NeuralCam product without a Night Mode, right?
NeuralCam Live’s Low Light Mode brightens up your video with a similar (though a lot more lightweight) technique used in our Night Mode photo app. It allows you to look good in video calls even in low light settings, a gentle light from your monitor is usually enough to provide a nice video feed.
And just in case you need to light up things even more, you can use the face-following Ring Light feature, which allows you to make video calls even in pitch dark rooms.
Gesture Guard
Detects and blurs different types of gestures that don't have their place in a video call.
Currently, it has two main components: detecting and blurring face touches (scratching face, picking nose, yawning, etc.) and detecting nudity. These help you to not get caught off guard.
Head-bubble Mode
It’s a more interesting way to hide your background.
We think current background segmentation used by most video call apps looks bad, really bad. Using face tracking, we put you in a "head bubble" and hide everything around it. It hides a cluttered background or your 3-year-old storming into the room.
NeuralCam Live - Free and Plus
The NeuralCam Live App is free to use with most of the features mentioned above, including 3 free styles.
We’ve also got a Plus plan - for our most engaged users - which contains a growing number of additional styles and the Low Light Mode, allowing you to obtain a high-quality video feed in poor lighting conditions. The Plus plan costs $4.99/mo or $29.99/yr.
Windows support & SDK
With NeuralCam Live, this is just the beginning. We have a long roadmap on how to make it even smarter. For now, we only support Macs but we're already working on adding support for Windows.
We’re also working on an iOS SDK that would allow third party video call or streaming apps to inject some smartness into their live videos too. Contact us if you’re interested in finding out more.
NeuralCam Live & Smart Webcams
Overall, having a smart webcam that helps you set up a good environment for your calls, edits your image for you to look more professional and makes sure you don’t do any silly gestures while in a call, creates a new kind of video call experience. Once you try it, you’ll never want to go back to a traditional webcam anymore.
You can download the NeuralCam Live app from the App Store, using the link below. Make sure to send us all your feedback over at hello@neural.cam - we'd love to hear from you.