The V4L2 (Video for Linux 2) API has lengthy provided an open supply different to proprietary digital camera/laptop interfaces, but it surely’s starting to indicate its age. On the Embedded Linux Convention Europe in October, the V4L2 challenge unveiled a successor referred to as libcamera. V4L2 co-creator and prolific Linux kernel contributor Laurent Pinchart outlined the early-stage libcamera challenge in a presentation referred to as “Why Embedded Cameras are Tough, and How one can Make Them Simple.”
V4l and V4L2 have been developed when camera-enabled embedded techniques have been far easier. “Perhaps you had a digital camera sensor related to a SoC, with possibly a scaler, and all the things was uncovered by way of the API,” mentioned Pinchart, who runs an embedded Linux companies referred to as Concepts on Board and is at present working for Renesas. “However when grew to become extra advanced, we disposed of the standard mannequin. As a substitute of exposing a digital camera as a single system with a single API, we let userspace dive into the system and expose the expertise to supply extra fine-grained management.”
These enhancements have been extensively documented, enabling skilled builders implement extra use circumstances than earlier than. But, the spec positioned a lot of the burden of controlling the advanced API on builders, with few assets out there to ease the training curve. In different phrases, “V4L2 grew to become extra advanced for userspace,” defined Pinchart.
The challenge deliberate so as to add a layer referred to as libv4l to handle this. The libv4l userspace library was designed to imitate the V4L2 kernel API and expose it to apps “so it may very well be fully clear in monitoring the code to libc,” mentioned Pinchart. “The plan was to have system particular plugins supplied by the seller and it will all be a part of the libv4l file, but it surely by no means occurred. Even when it had, it will not have been sufficient.”
Libcamera, which Pinchart describes as “not solely a digital camera library however a full digital camera stack in consumer house,” goals to ease embedded digital camera utility improvement, bettering each on V4L2 and libv4l. The core piece is a libcamera framework, written in C++, that exposes kernel driver APIs to userspace. On prime of the framework are optionally available language bindings for languages comparable to C.
The following layer up is a libcamera utility layer that interprets to current digital camera APIs, together with V4L2, Gstreamer, and the Android Digicam Framework, which Pinchart mentioned wouldn’t include the same old vendor particular Android HAL code. As for V4L2, “we are going to try to keep up compatibility as a finest effort, however we gained’t implement each function,” mentioned Pinchart. There will even be a local libcamera app format, in addition to plans to assist Chrome OS.
Libcamera retains the kernel stage hidden from the higher layers. The framework is constructed across the idea of a digital camera system, “which is what you’ll anticipate from a digital camera as an finish consumer,” mentioned Pinchart. “We are going to wish to implement every digital camera’s capabilities, and we’ll even have an idea of profiles, which is a better view of options. For instance, you could possibly select a video or point-and-shoot profile.”
Libcamera will assist a number of video streams from a single digital camera. “In videoconferencing, for instance, you may want a unique decision and stream than what you encode over the community,” mentioned Pinchart. “Chances are you’ll wish to show the dwell stream on the display screen and, on the similar time, seize stills or file video, maybe at totally different resolutions.”
Per-frame controls and a 3A API
One main new function is per-frame controls. “Cameras present controls for issues like video stabilization, flash, or publicity time which can change beneath totally different lighting circumstances,” mentioned Pinchart. “V4L2 helps most of those controls however with one massive limitation. Since you’re capturing a video stream with one body after one other, if you wish to improve publicity time you by no means know exactly at what body that may take impact. If you wish to take a nonetheless picture seize with flash, you don’t wish to activate a flash and obtain a picture that’s both earlier than or after the flash.”
With libcamera’s per-frame controls, you may be extra exact. “If you wish to make sure you all the time have the appropriate brightness and publicity time, it’s good to management these options in a approach that’s tied to the video stream,” defined Pinchart. “With per-frame controls you may modify all of the frames which are being captured in a approach that’s synchronized with the stream.”
Libcamera additionally provides a novel strategy to a given digital camera’s 3A controls, comparable to auto publicity, autofocus, and auto white stability. To offer a 3A management loop, “you may have a easy implementation with 100 traces of code that will provide you with barely usable outcomes or an implementation primarily based on two or three years of improvement by system distributors the place they actually attempt to optimize the picture high quality,” mentioned Pinchart. As a result of most SoC distributors refuse to launch the 3A algorithms that run of their ISPs with an open supply license, “we wish to create a framework and ecosystem wherein open supply re-implementations of proprietary 3A algorithms can be potential,” mentioned Pinchart.
Libcamera will present a 3A API that may translate between customary digital camera code and a vendor particular part. “The digital camera wants to speak with kernel drivers, which is a safety danger if the picture processing code is closed supply,” mentioned Pinchart. “You’re working untrusted 3A vendor code, and even when they’re not doing one thing behind your again, it may be hacked. So we would like to have the ability to isolate the closed supply part and make it function inside a sandbox. The API may be marshaled and unmarshaled over IPC. We will restrict the system calls which are out there and forestall the sandboxed part from immediately accessing the kernel driver. Sandboxing will make sure that all of the controls must undergo our API.”
The 3A API mixed with libcamera’s sandboxing strategy, could encourage extra SoC distributors to additional expose their ISPs simply as some are have begun to open up their GPUs. “We wish the distributors to publish open supply digital camera drivers that expose and doc each management on the system,” he mentioned. “If you end up interacting with a digital camera, a big a part of that code is system agnostic. Distributors implement a totally closed supply digital camera HAL and provide their very own buffer administration and reminiscence location and different duties that don’t add any worth. It’s a waste of assets. We wish as a lot code as potential that may be reused and shared with distributors.”
Pinchart went on to explain libcamera’s cam system supervisor, which can assist sizzling plugging and unplugging of cameras. He additionally defined libcamera’s pipeline handler, which controls reminiscence buffering and communications between MIPI-CSI or different digital camera receiver interfaces and the digital camera’s ISP.
“Our pipeline handler takes care of the small print so the appliance doesn’t need to,” mentioned Pinchart. “It handles scheduling, configuration, sign routing, the variety of streams, and finding and passing buffers.” The pipeline handler is versatile sufficient to assist an ISP with an built-in CSI receiver (and and not using a buffer pool) or different difficult ISPs that may have a direct pipeline to reminiscence.
Watch Pinchart’s total ELC speak under: