How Samsung is using AI software to improve the camera game on its smartphones
Samsung has applied around 60 new AI models run by the neural processing unit (NPU) to optimize the camera functions of the Galaxy S22 Ultra smartphone, a company executive said.
This has enabled the South Korean tech giant to deliver camera experiences that can satisfy casual users with the best possible photographs and professional users with RAW files equivalent to those taken on DSLR cameras, said Joshua Sungdae Cho, vice president and head of visual software R&D at Samsung’s MX Business, in an interview with ZDNet.
“We first applied NPUs to our smartphones three years ago,” Cho said. “At the time, these NPUs were running around 10 AI models. On the Galaxy S22 Ultra, there are now 60 AI models. Basically, the NPU is involved in almost every function of the cameras.”
AI models are decision-making algorithms that are trained through a set of data to apply to certain tasks, while NPUs are chips designed specifically to process such algorithms.
A particular AI model used by Samsung for the Galaxy S22 Ultra is for portrait mode, in which recognizing the depth between the person and the background to separate them – also known as segmentation – is crucial. .
“It’s hard to judge depth with cameras alone. Some cameras use depth sensors, but instead we asked AI models to study a huge amount of depth-related imagery with various backgrounds and objects,” Cho said.
“As we have a diverse range of cameras on the S22, we also have the AI model which trains using two cameras, and it studies images with wiring, cups, glasses in a variety of conditions “said the vice president.
Zoom and night photography
Galaxy S22 Ultra cameras support 10x optical zoom and 100x digital zoom. It’s not the first model to do so as Samsung first introduced 100x digital zoom two years ago with the Galaxy S20 Ultra, although reception was mixed at the time.
But with the development of the new AI models, coupled with the computing power offered by the 4nm mobile processor on the latest Galaxy S22 Ultra, Samsung was confident that the latest iteration of the 100x digital zoom will impress users, said the vice-president.
“Processing time is critical for a feature like 100x zoom and we have the compute resources to handle it. Some of the work is done by the NPU and some by the GPU. manageable because the zoom is essentially a specific area of the sensor – for example, a 12MP area area on a 108MP sensor, so a zoom slam gets a lot of input from that smaller 12MP area.
“For images taken above 30x digital zoom, the camera takes between ten and twenty background images in a single instant. It then proceeds to summarize the multiple images. -processing to determine which details it wants to improve,” Cho mentioned.
The Galaxy S22 Ultra’s 108MP camera sensor can also pack up to nine pixels into one to better absorb light at night. This technology is coupled with artificial intelligence models trained to reduce noise and ensure images have “the feel of night” while brightness is increased, the VP said.
“The signal-to-noise ratio is severely affected when it is dark, so we came up with a new AI model to reduce the noise. We trained this AI model on various photographs taken at night, from those taken by a DSLR camera to synthesized images we created that intentionally increased brightness or noise.”
Samsung has also changed the way cameras approach night photography in general: “In previous Galaxy S phones with a 108MP camera, we reduced their pixel count to 12MP and then scaled the picture at 108MP. For Galaxy S22 Ultra, when photos are taken at night, the photo is taken by the 108MP camera and 12MP camera. Raw image detail is provided by the 12MP camera, while fine detail is provided by the 108MP.
Close to DSLR upgrade
All three Galaxy S22 series models offer auto and pro modes on its standard camera app as well as the downloadable Expert RAW app which provides multi-frame RAW files that can be edited with Adobe Lightroom.
“We really wanted the Galaxy S22 series cameras to cover the casual user and the professional user,” Cho said. “Customers today are very familiar with smartphone cameras, so with auto mode, which caters to the majority of users, we have Scene Optimizer enabled by default to give the best colors for scenes commonly taken in photos. We also wanted the screen to be clear without icons, while camera values are now displayed digitally. We wanted the user interface to be as simple as possible.
While the Pro mode can take RAW files, the Expert RAW app offers 16-bit multi-frame RAW files.
“We made full use of the NPU and AI models for 16-bit RAW files. A snapshot will collect twenty shots of the background, and users can use up to four cameras for the Ultra model simultaneously,” said the vice- President.
DSLR cameras have large sensors that allow them to take RAW images without noise, but so far for smartphone cameras this has been difficult to emulate in low light conditions as they contain sensors smaller. But with the computing power offered by the NPU on the Galaxy S22 series, their 16-bit RAW files match those taken by DSLR cameras, according to Cho.
“Professionals today want RAW files on their smartphones. A smartphone is a very handy tool for them because one smartphone can give them four lenses in the palm of their hands. With DSLR cameras, you have to carry all four lenses. We really wanted to give this group the same pairing experience on our smartphones as on a DSLR. Every function of a DSLR is offered on the Galaxy S22 series, including post-editing.
Finding the best camera setup
Each year, the processing power of NPUs is doubled and this trend is expected to continue for at least the next five years, said the vice president of Samsung.
NPUs today can handle 16-bit arithmetic and floating-point arithmetic, with the speed and accuracy of performing it expected to continue to increase in the coming years, Cho said.
Coupled with this increase in computing resources, the evolution of computational photography – or techniques for capturing and processing digital images – cameras on smartphones are expected to continue to improve.
“The most discussed topic in computational photography is multi-camera technology. Researchers and companies are always researching what is the best camera setup. We could add more cameras or fewer cameras. We are always looking for the optimal multi-camera setup,” the said VP said.
“Cameras are the centerpiece of smartphones today and we expect this trend to continue over the next decade. While we currently apply 60 AI models, we expect all Camera functions, which are handled by hundreds of algorithms, are handled by the NPU.
“16-bit RAW files currently have limitations in terms of resolution. This will require continuous improvement. For 4K and 8K video, to improve resolution in real time, we need even more computing power than today. It will take two to three years from now.”