Repository navigation
feat(q25): front and rear cameras - #53
Draft
patrickjquinn wants to merge 30 commits into
Draft
patrickjquinn wants to merge 30 commits into
patrickjquinn wants to merge 30 commits into
Conversation
The kernel config leaves CONFIG_MEDIA_SUPPORT unset and nothing built in depends on the media core, so the camera work can load it as modules against the running 7.0.0-q25-cfi-test kernel without a new kernel. build.sh copies the kernel's drivers/media/mc, v4l2-core and common/videobuf2 and builds them as one M= set: the kbuild symbols (MEDIA_SUPPORT, MEDIA_CONTROLLER, VIDEO_DEV, VIDEO_V4L2_I2C, V4L2_ASYNC, V4L2_FWNODE, V4L2_CCI(_I2C), VIDEOBUF2_CORE/V4L2/MEMOPS/DMA_CONTIG) are forced on the make line as for netfilter, and media/Kbuild gives the C code and headers the matching -D flags (MEDIA_CONTROLLER, VIDEO_V4L2_SUBDEV_API, VIDEO_V4L2_I2C, V4L2_CCI_I2C) as audio/ does for ASoC. struct video_device and v4l2_subdev change layout with MEDIA_CONTROLLER, so every module that includes <media/*.h> has to build in this set. The modules are staged in OUT/camera and kept out of the shipped set and its kocmp comparison until the camera milestones pass. W=1 is clean, modpost resolves every symbol and each module's vermagic is "7.0.0-q25-cfi-test SMP preempt aarch64".
…ivers MT6789 (Q25) and MT8189 (Genio 720) are the same ISP6s generation and share the CAMSV block, the SENINF register family and the CD-PHY 2.x receiver. These are Genio's mainline-style V4L2 drivers for that raw path, imported unmodified as the base for the Q25 port; the MT6789 changes follow as a separate patch so they stay reviewable against upstream. Source: https://gitlab.com/mediatek/aiot/bsp/linux branch mtk-v6.6, SRCREV a246e0c68c39cef5c935b5c202c881cc37a191b9 (the kernel of IoT Yocto rity-scarthgap-v26.0, linux-mtk_6.6.bb), byte-identical copies: drivers/media/platform/mediatek/isp/isp_6s/seninf/* -> camera/seninf/ drivers/media/platform/mediatek/isp/isp_6s/camsv/* -> camera/camsv/ drivers/phy/mediatek/phy-mtk-mipi-csi-2-1.c -> camera/phy/ drivers/phy/mediatek/phy-mtk-mipi-csi-2-1-rx-reg.h -> camera/phy/ Documentation/devicetree/bindings/media/ mediatek,mt8189-seninf-core.yaml -> camera/bindings/ mediatek,mt8189-camsv.yaml -> camera/bindings/ The C sources are GPL-2.0 (SPDX headers kept, MODULE_LICENSE("GPL")), the bindings GPL-2.0-only OR BSD-2-Clause. phy-mtk-io.h is not imported: the kernel's copy is identical and build.sh takes it from KSRC.
camera/0001-mt6789-port.patch, applied by build.sh to the verbatim Genio import, so the whole MT6789 delta stays one reviewable diff. MT6789 support: - New compatibles mediatek,mt6789-seninf-core, -seninf, -csi-rx and -camsv beside the MT8189 ones; MODULE_DEVICE_TABLE dropped so nothing autoloads before the TPG bench passes. - mtk_csi_phy_mt6789 ops: 6 SENINF, 8 muxes, 13 cam-muxes, and the test pattern generator at +0x600 per SENINF page (a new tg_offset field; the MT8189 data keeps +0xF00). The ops code now uses the SoC's ops chosen at probe instead of always mtk_csi_phy_2_1. SENINF_NUM grows to 8 and the cam-mux enum to 13; TOP_MUX_SENINF2/3 and TOP_MUX_CAMTG3 are optional clocks. - The cam-mux CTRL/NEXT_CTRL/CHK_CTL switches stopped at mux 7; they are now computed (4-bit selector per byte, 0x10 per CHK block) as in the vendor seninf_impl.c. - seninf takes the cam-mux from the CAMSV node's mediatek,cammux-id (camsv2 = 5 on the Q25 bench) and falls back to Genio's name mapping. - CAMSV: no IOMMU on the Q25, so the iommu_present() defer is gone, the DMA mask is 32 bits (IMGO_BASE_ADDR is 32-bit) and, given a "larb" reg, stream-on clears MMU_EN on larb13 ports 6-8, which firmware and every CAM power-up leave set. - Unpacked RAW10/12 with standard Bayer fourccs replaces the MTISP packed ones mainline lacks. RAW10 is offered as 14-bit fourccs first (the Q25 test pattern lands 14-bit LSB-aligned) and as 10-bit ones. - Test pattern menu value 3 selects the TESTMDL generator; before, no value reached set_test_model(). Linux 7.0: void .remove, min_queued_buffers, no vb2 wait_prepare/finish, init_cfg moved to internal_ops.init_state, v4l2_subdev_state_get_format, get_frame_interval as an active-state pad op, const of_phandle_args in the PHY xlate, the subdev name size, and the of.h/of_platform.h includes. Genio bugs fixed: CAMSV setup returned with irqlock held and a PM reference leaked on reset timeout; stop_streaming returned with op_lock held; stream_off returned with streaming_protect held; seninf runtime resume returned with core->mutex held; set_test_model and the sensor stream-on leaked PM/clock references on errors; the A/B CSI port number formula; the PHY port struct duplicated in seninf (now one shared header); get_mbus_format_by_dt() truncated bus codes to u8; get_vcinfo swapped width and height; set_idle forgot the cam-mux so a second stream ran to mux 0xff; vb2 max segment size was cleared on the wrong device; %llx on u32 register reads. The ops functions only used through the ops table are static. W=1 (clang 22, LLVM=1) is clean and modpost resolves every symbol. checkpatch --strict on this patch reports only the MAINTAINERS notice. clang-tidy (bugprone, clang-analyzer) reports nothing on the patch's own lines; what remains is unchanged Genio code or container_of/list macro false positives.
…dia core
build.sh copies camera/{seninf,camsv,phy} into the media set, takes
phy-mtk-io.h from KSRC, applies 0001-mt6789-port.patch and forces
MTK_SENINF6S, MTK_CAMSV6S and PHY_MTK_MIPI_CSI_2_1 to m, so the drivers
share the media core's -D flags and Module.symvers. They are staged in
OUT/camera with the media core, outside the shipped set.
q25-camera.dtso is the runtime overlay for the test-pattern bench:
SENINF on CSI port 0 with no sensor endpoint, CAMSV2 on cam-mux 5 with
larb13 mapped for the MMU_EN clear, and the CSI0/CSI1 D-PHYs (CSI1
disabled). Both power domains point at q25-cam-pd's domain 12 on the SPM
power-controller (base phandle 0x1e); clocks are mainline topckgen and
camsys indices on the base phandles 0x12 and 0x1a. build.sh compiles it
with the kernel's dtc into OUT/camera/q25-camera.dtbo; dtc's
phandle-property checks are off because the providers live in the base.
It merges cleanly onto image/evidence/live.dts with fdtoverlay.
7.0's media_entity_pads_init() rejects a pad that is not exactly one of SINK or SOURCE; Genio set the flags after the call, which 6.6 tolerated, so camsv2 failed to probe with -EINVAL. Set them first, for the subdev and the video node. Also drops a stray .checkpatch-camelcase.git. file the patch had picked up. Verified on the Q25: the graph seninf-0 -> camsv2 -> /dev/video0 registers, and five test-pattern frames captured with v4l2-ctl are byte-identical to the raw-register bench frame.
The boot DTB has no SENINF/CAMSV nodes and the kernel has no userspace overlay interface. q25-cam-dt (enable=1, loaded by hand) applies q25-camera.dtbo from the firmware path with of_overlay_fdt_apply(); a debugfs switch removes and re-applies it, since modules cannot unload. Verified on the Q25: the seninf-top, camsv2 and CSI PHY nodes appear under /proc/device-tree/soc and the camera drivers bind to them.
The Q25's seven camera rails come from a Will Semi WL2868C on i2c5 (0x2f, enable GPIO15), and mainline has no driver for it. Carry Qianhui Xu's v4 regulator driver unmodified. Its register map and 8 mV steps match the stock vendor driver, and the vendor's front-camera settings decode to the same 1.2/2.8/1.8 V. The module is loaded by hand and is not shipped yet.
Build mainline imx111 in the media set and describe the front camera in the runtime overlay: the WL2868C's LDO2/LDO5/LDO7 at the stock 1.2/2.8/ 1.8 V, reset on GPIO16, a 24 MHz CAMTG clock on CMMCLK0, two data lanes at 542.4 MHz into seninf-0 port 0, i2c4 enabled. The module's PDN pin (GPIO17) is a fixed regulator on the IO rail, as Genio does for its sensors' PDN. dtc's gpios check joins the other raw-phandle checks that the overlay cannot satisfy. Verified on the Q25: imx111 reads chip ID 0x0111 and binds to seninf-0; LDO7 is moved from its 2.8 V reset value to 1.8 V before it is enabled; 1640x1232 RAW10 streams to /dev/video0 at 30.6 fps with no CSI-2 CRC, ECC or FIFO errors, and SENINF counts exactly 1640x1232 per frame.
imx111 labels unflipped output SGBRG10, but the sensor's own solid-colour test pattern puts red at (0,0): RGGB unflipped, GRBG with hflip, GBRG with vflip, BGGR with both. The driver's no-flip/vflip and hflip/hvflip entries are swapped. The stock MTK driver agrees independently: it runs the sensor HV-mirrored and declares RAW_B (BGGR). SENINF counts exactly the mode's lines and CAMSV grabs from (0,0), so no line is lost on our side. Patch the format table and the default code. Verified on the Q25 for all four flip states, and a real scene renders with the right colours.
The port kept 10-bit fourccs for unpacked RAW10 in case a sensor's data arrived unshifted. It does not: the IMX111's samples are multiples of 16 (black level 1024 = 64 << 4), as with the SENINF test pattern, so a 10-bit fourcc would mislabel them. Drop those aliases. Verified on the Q25: RG14/GR14/GB14/BG14 capture from the IMX111 and the TPG; the 10-bit fourccs are no longer listed.
imx111_set_format() took data_depth from the format stored by the previous call, so the sensor streamed whichever depth was set before the current one. libcamera tries an 8-bit format before the 10-bit one it keeps, so the IMX111 sent RAW8 (CSI data type 0x2a) into a RAW10 pipeline and no frame ever reached CAMSV. Take it from the requested code. build.sh now applies every patch in camera/i2c/. Verified on the Q25: libcamera's cam streams raw 1640x1232 at 30.6 fps.
CAMSV takes a new buffer only on PASS1_DON, and PASS1_DON never fires without a buffer programmed. libcamera calls STREAMON before its first QBUF, so the stream started empty and stayed empty (IMGO_BASE 0, SOF only). Programming the buffer at QBUF instead, mid-frame, raised an IMGO DMA error. Hold start_streaming until a buffer is queued, as v4l2-ctl's order already did. Also re-read the DMA address at every prepare: an imported DMABUF can change between QBUFs. Verified on the Q25: libcamera raw and software-ISP streams run at 30.6 fps; the v4l2-ctl test pattern is unchanged.
The SENINF muxes pass lines in whole groups of 8 pixels: in the IMX111's 820x614 mode the CSI packets carry 820 pixels but the mux measures 816, and CAMSV, programmed for 820, never completes a frame. Round try_fmt widths down to 8 and enumerate the frame sizes stepwise with that step, so userspace can tell such modes apart. Verified on the Q25: 820x614 now fails at configuration instead of hanging; libcamera (with its tryPipeline size check) picks the 1280x720 mode for the viewfinder; 1280x720, 1640x924, 1640x1232 and the test pattern still capture.
The camera path needs the CAM MTCMOS, and nothing on the Q25 can power it: q25-mfg-only-v2, q25-audio-pd and q25-conn-pd each answer for their own index only. q25-cam-pd adds one more provider on the SPM power-controller node that resolves index 12 alone, found under DISP. Data and sequence are the stock mtk-scpsys-mt6789 ones: ctl 0x35c, sta bit 23 in 0x16c/0x170, SRAM pdn 8 / ack 12, infracfg bus protect BIT(0)|BIT(2) then BIT(1)|BIT(3) through 0x2d4/0x2d8, sta 0x2ec, clear ack ignored; basic clock cam_sel, subsys clocks camsys LARB13/LARB14. CAM is a child of DISP, which the firmware leaves on for the panel. DISP's PWR_CON and bus-protect bits are never written; every power on re-checks that DISP is on. The boot-time pm-domain patch hard-crashed the Q25, so this follows the GPU and CONN providers: enable=0 by default, no device table, no exit, init refuses a CAM that is on or half-on, DISP off, or RAWA/RAWB on. Each step logs PWR_CON and both status bits; a status timeout withdraws the request and a failed bus-protect step undoes the steps already set. debugfs test_power runs the sequence without a consumer and refuses, in both directions, while genpd has consumers, so a consumer's clocks never outlive CAM. Host build only, not loaded on the phone: W=1 and checkpatch --strict are clean, clang-tidy (bugprone, clang-analyzer) reports only the PTR_ERR/ssize_t narrowing it also reports for q25-conn-pd, and modpost resolves every symbol against the 7.0.0-q25-cfi-test tree. build.sh builds it but keeps it out of modules/ until it ships.
Alpine's libcamera (0.7.2) cannot drive the Q25's front camera: its software ISP has no 14-bit unpacked RAW, which MediaTek's CAMSV writes, and the simple pipeline does not enable the ISP on mtk-seninf. Package 0.8.0 under the same name, following Alpine's recipe, with six patches written for upstream: 14-bit unpacked Bayer in the CPU and GPU debayer, a GPU debayer sampling fix for scaled output, the software ISP on mtk-seninf, an IMX111 sensor helper, skipping capture sizes the video node rejects, and completing requests on stop when the IPA owes metadata. qcam, the Python bindings and the docs are left out, and so is the virtual test camera: it needs libyuv, which Alpine ships as headers only. The Android layer is off for the same reason. The unversioned libcamera.so stays in -dev. Verified on the Q25: the APKs built with abuild in the phone's Alpine snapshot (no warnings), unpacked on the phone, run the Camera app's preview and photos, and cam's raw and viewfinder captures.
The port dropped seninf's, CAMSV's and the CSI PHY's MODULE_DEVICE_TABLE so nothing would bind before the capture path was proven. It is now (test pattern, IMX111, libcamera), and their compatibles only match the camera overlay's nodes, so restore the tables: udev loads the drivers when q25-cam-dt adds the nodes. Verified on the Q25: after a reboot with only q25-cam-pd and q25-cam-dt loaded explicitly, seninf, camsv, the PHY, the WL2868C and the IMX111 load and bind by 9 s.
Add the front camera's set to q25-kernel-modules under extra/q25-camera: the CAM power domain and overlay loader, the WL2868C PMIC, the V4L2 media core (less v4l2-dv-timings, which nothing needs), the SENINF, CAMSV and CSI PHY drivers and the IMX111, and q25-camera.dtbo in /usr/lib/firmware. build.sh now puts them with the other shipped modules, so its rebuild comparison covers them too. Verified: a rebuild matches the shipped modules' code and data (113 of 113), and on the Q25 the r20 APK unpacks, depmod finds the camera's OF aliases, and the camera comes up after a reboot.
Load q25-cam-pd and q25-cam-dt with enable=1 at boot: the CAM power domain first, then the overlay whose nodes autoload the rest. Depend on libcamera and its GStreamer element, through which the Camera app gets the camera. Verified on the Q25 after a reboot with nothing loaded by hand: cam lists the front camera and streams at 30.6 fps, and the Camera app launched from the shell shows the live preview and saves a JPEG.
Qt 6.11's GStreamer image capture probes the frame before its jpegenc branch and writes that buffer to the file, so captureToFile() saved raw RGBA under a .jpg name. Capture to a QImage instead and write the JPEG with QImageWriter on the thread pool; photoSaved and errorOccurred keep their meaning, and the FFmpeg backend takes the same path. Verified on the Q25 front camera: the shutter saves a 1276x720 JPEG (FF D8 header) that opens with the right picture.
The camera permission bound nothing into the sandbox, and apps used FFmpeg's multimedia backend, whose camera is plain V4L2: on a phone with a media-controller ISP like the Q25 there was no camera at all. Apps with "camera" now get QtMultimedia's GStreamer backend (libcamerasrc), the media graph's nodes (/dev/media*, video*, v4l-subdev*), udmabuf for the software ISP's output and /run/udev. They start in their own sandbox rather than a spare, as WebEngine apps do, since a spare has neither. The compositor keeps an app's own QT_MEDIA_BACKEND instead of forcing FFmpeg. Verified on the Q25: the Camera app shows the front camera's live preview from the shell's launch, and its sandbox holds exactly those nodes.
The APKBUILD copied every app's C++ library into /usr/lib although CMake already installs it beside its QML plugin, which finds it through $ORIGIN. The copy put the Camera app's libcamera.so at the path of libcamera's own development symlink, so libcamera-dev could no longer be installed alongside marathon-shell. Drop it. Verified on the Q25 with /usr/lib/libcamera.so and /usr/lib/libcalculator.so removed: Camera (own sandbox) and Calculator (adopted spare) launch and work; every app's library sits beside its plugin.
QtMultimedia's GStreamer backend asks the device monitor for hidden devices too, so besides libcamera's camera it listed the V4L2 provider's devices, which libcamera's provider hides: on the Q25 that is CAMSV's media-controller capture node, offered as a second camera that cannot be opened on its own. Camera apps get their cameras through libcamera, so their sandbox masks GStreamer's V4L2 plugin. Verified on the Q25: QMediaDevices::videoInputs() lists one camera in place of two.
The recorder used Qt's default format, HEVC, whose software encoder managed about 5 fps at 1276x720 on the Q25 (420 frames in 80 s). Record H.264 in MP4: openh264 keeps up with the camera. Verified on the Q25: an 11.3 s recording holds 339 frames (29.9 fps) and plays back with the right picture.
The Q25's rear S5KJN1 had no gain model, so the software ISP's AGC left the image dark. The helper gives it the driver's linear gain and the sensor's black level.
CAMSV's coherent allocations stop at 4 MiB, short of the front and rear sensors' full-resolution frames. The new board DT patch reserves 256 MiB below 4 GiB for it under a fixed phandle that the camera overlay names. The Q25 boots the same node added to its current DTB.
dma_alloc_coherent() stops at 4 MiB without CMA, short of the IMX111's 8 MP and the S5KJN1's 12.5 MP frames. CAMSV now attaches the pool its memory-region names, when there is one, and keeps the default allocator otherwise. Tested on the Q25 with the board DT's pool: both sensors streamed full-resolution frames.
The rebuild check stripped each new module's shipped counterpart and stopped at the first module that has never shipped. It now names it as new and goes on.
The rear camera is a Samsung S5KJN1 on i2c8 at 0x10 (chip ID 0x38e1), four lanes at 700 MHz into CSI port 1, clocked from CAMTG2. The overlay adds the port, the sensor and its WL2868C rails; CAMSV2 takes either port, one camera at a time, and names the board DT's camera pool for full-resolution frames. The build adds the kernel's s5kjn1 driver with a fix for its crop rectangle, which reported the mode width as its height. Tested on the Q25: 12.5 MP raw frames at 30 fps, and libcamera lists the front and back cameras and streams 1920x1080 from the back one at 30 fps through the software ISP.
q25-kernel-modules r21 adds s5kjn1 and replaces mtk-camsv6s and the camera overlay with the builds that take the rear port and the camera pool. The device package's modules-load note now names both cameras.
patrickjquinn
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #52. Brings up the Zinwa Q25's cameras, the front Sony IMX111 and the rear Samsung S5KJN1, end to end: kernel drivers, libcamera, the shell's sandbox and the Camera app.
Kernel (out-of-tree modules, shipped in
q25-kernel-modulesr21)q25-cam-pd(CAM power domain),q25-cam-dt(appliesq25-camera.dtboat runtime; the boot DTB has no camera nodes), the posted WL2868C PMIC driver (v4, unmodified), and mainlineimx111ands5kjn1.kernel/0010, fixed phandle 0x93) that CAMSV allocates from, so full-resolution frames (8 MP front, 12.5 MP rear) fit; the Q25 boots this DTB as its default.media_entity_pads_init(7.0), CAMSV starting before a buffer is queued, CAMSV's 8-pixel width step, unpacked RAW10 being 14-bit, twoimx111bugs (Bayer order off by one row; bit depth taken from the previous format), ands5kjn1reporting its crop height as the mode width.device-zinwa-q25-marathonr36 loads the power domain and overlay at boot; the drivers autoload from the overlay's nodes.libcamera 0.8.0 (
packaging/packages/libcamera, replaces Alpine's 0.7.2): seven patches written for upstream: 14-bit unpacked Bayer in the software ISP (CPU and GPU), a GPU debayer fix for scaled output, the software ISP on mtk-seninf, IMX111 and S5KJN1 helpers, skipping capture sizes the video node rejects, and completing requests on stop when the IPA owes metadata.Shell and app
camerapermission get QtMultimedia's GStreamer backend (libcamerasrc), the media nodes and udmabuf in their sandbox, and their own sandbox instead of a spare.captureToFile()writes the raw frame./usr/lib(the Camera app'slibcamera.sotook libcamera's path).Validation (Q25)
Test pattern through V4L2: bit-exact frames, 500 start/stop cycles and a 15,000-frame stream without drops or errors.
IMX111: 1640x1232 at 30.6 fps with no CSI errors; Bayer order checked with the sensor's solid-colour pattern in all four flip states.
Packages built with abuild in the phone's Alpine snapshot and unpacked on the phone: after a reboot with nothing loaded by hand the camera comes up by 9 s,
camstreams at 30.6 fps, and the Camera app launched from the shell shows the live preview and saves JPEGs.A rebuild of all 113 shipped modules matches their code and data (
kocmp.py). clippy, rustfmt and clang-format are clean.Rear: 4080x3072 raw frames at the sensor's 30 fps; libcamera lists both cameras and streams 1920x1080 from the rear at 30 fps with working exposure. After unpacking r21, libcamera r1 and the device package r36 and rebooting, both cameras come up at boot with the pool attached, and the Camera app previews.
Not yet: full-resolution stills (libcamera merges the sensor size ranges wrongly, so photos are at preview size), audio in videos (Qt's recorder stalls with audio), preview rotation, the S5KJN1's focus motor, and removing the overlay at runtime (a null dereference in SENINF unbind).