Application Test - Object Segmentation Detection (YOLOv5-Seg)

Run Talowe
talowe_yolov5_seg_camon CAM126B-P: the camera performs real-time 80-class COCO instance segmentation (boxes + colored masks), draws them on the picture, and pushes H.264 RTP to the computer. Openseg.sdpin VLC to view the picture.Source: https://github.com/hbt021211-coder/talowe-rv1126b-aidemo
Inference: RKNN | Capture: V4L2
/dev/video12NV12 1920x1080| Encoding: MPP | Stream receive: UDP5000
Talowe's original project uses OpenCV + GStreamer. The factory Buildroot IPC does not have these two libraries.
The current Demo has been changed to V4L2 + RGA + MPP; mask scaling no longer depends on the system OpenCV.
Encoding uses a direct NV12 copy from the camera, and boxes and semi-transparent masks are drawn on NV12, avoiding RGB→NV12 causing VLC to be all black.
The procedure on this page is written for rkipc, /dev/video12, S99rkipc).
- For flashing, see CAM126B-P Buildroot IPC (
Rockchip-RV1126B-Buildroot-IPC) in Upgrade Firmware Reflashing is not required : if the board already has this IPC firmware, demonstrate directly- If it is currently Debian / another image, first flash Buildroot IPC according to that page, then follow the steps below
For IPC streaming, account and password, see Firmware Usage - IPC Streaming.
This Demo shares camera /dev/video12 and UDP 5000 with Human Pose, YOLO11, Helmet, Garbage Classification, Fire and Smoke, and Mask, and cannot run simultaneously.
run_seg.sh stops the other Demos / rkipc first.
Demonstration (see the effect directly)
Environment as the three machines below. If the computer IP is not 10.0.0.49, change the script parameter to your IP.
| Role | Address | Description |
|---|---|---|
| Stream-receiving computer (Mac) | 10.0.0.49 | Install VLC, open seg.sdp |
| CAM126B-P | ssh root@10.0.2.0 | Default password see Firmware Usage |
| Program directory | /root/rknn_yolov5_seg_cam | Already deployed package |
1. Open VLC on the computer first
Copy seg.sdp to the computer, e.g. ~/seg.sdp. The content is fixed as follows (port 5000, do not change to the board IP):
v=0
o=- 0 0 IN IP4 0.0.0.0
s=Seg
c=IN IP4 0.0.0.0
t=0 0
m=video 5000 RTP/AVP 96
a=rtpmap:96 H264/90000
The package also contains a play.sdp with the same content (s=Seg). For this page use seg.sdp; do not mix it with other Demos' sdp.
open -a VLC ~/seg.sdp
Or VLC → File → Open File → select seg.sdp.
The first time it may ask the local firewall to allow UDP 5000; click Allow.
VLC will stay black waiting for the stream; this is normal. Start the board side in the next step.
2. Start streaming on the board
ssh root@10.0.2.0
cd /root/rknn_yolov5_seg_cam
./run_seg.sh 10.0.0.49
run_seg.sh will: stop other Demos / rkipc → open /dev/video12 → infer and draw masks / boxes → push H.264 to 10.0.0.49:5000.
When you see logs like the following, the stream is out:
v4l2 opened /dev/video12 1920x1080 stride=1920 nbufs=4
mpp rtp -> 10.0.0.49:5000 1920x1080
Pipeline initialized. Starting inference, RTP 10.0.0.49:5000 ...
[SEG] person | box:( 412 , 201)( 890 , 980) | size: 478*779 | conf: 87.32%
VLC should show a live picture: semi-transparent colored masks of object outlines, detection boxes, class names, and FPS in the upper-left corner. COCO objects such as people, cups, and keyboards will be segmented when they enter the frame. Segmentation is heavier than pure detection, so the frame rate will be lower than YOLO11.
Manual equivalent command (when not using the script):
/etc/init.d/S99rkipc stop
cd /root/rknn_yolov5_seg_cam
export LD_LIBRARY_PATH=./lib:$LD_LIBRARY_PATH
./rknn_yolov5_seg_cam ./model/yolov5_seg.rknn /dev/video12 10.0.0.49
The third parameter must be the 10.0.2.0.
3. After the demonstration, restore IPC
After stopping the Demo with Ctrl + C:
killall rknn_yolov5_seg_cam rkaiq_3A_server 2>/dev/null
/etc/init.d/S99rkipc start
The web page / RTSP (rtsp://10.0.2.0:554/live/0) will recover.
The Demo and rkipc cannot occupy the camera at the same time.
Board-side package contents
root@rv1126bp-buildroot:~/rknn_yolov5_seg_cam# ls
lib model play.sdp rknn_yolov5_seg_cam run_seg.sh seg.sdp
| File | Purpose |
|---|---|
rknn_yolov5_seg_cam | Main program |
run_seg.sh | Stop rkipc and start; parameter is the computer IP |
seg.sdp | Copy to the computer for VLC (recommended) |
play.sdp | Same content as seg.sdp, s=Seg |
model/yolov5_seg.rknn | Segmentation model (about 8.6 MB, 80 COCO classes) |
model/coco_80_labels_list.txt | Class names |
lib/librknnrt.so librga.so librockchip_mpp.so | NPU / RGA / encoding |
Startup parameters:
./rknn_yolov5_seg_cam <model_path> <camera_index> <target_ipv4>
<camera_index>: ISP main path; on this board it is/dev/video12(rkisp_mainpathinv4l2-ctl --list-devices)<target_ipv4>: LAN IP of the stream-receiving computer; port is hardcoded 5000
Thresholds are in postprocess.h: BOX_THRESH default 0.25, NMS_THRESH default 0.45.
Compile from source and deploy
You can skip this section if you only want to demonstrate. Follow the steps below when changing code or changing boards.
Machines
| Role | Example |
|---|---|
| Local machine | ~/SDK/talowe-rv1126b-aidemo (Mac only edits code and rsyncs) |
| Build server | ssh roy@10.0.0.18 (x86_64 Ubuntu, cross-compile aarch64) |
| Board | ssh root@10.0.2.0 |
Use the distribution cross-compiler; no need to build the full SDK host:
# build server (root)
apt-get install -y gcc-aarch64-linux-gnu g++-aarch64-linux-gnu cmake
Sync source
3rdparty/opencv is about 525 MB and is not used by this Demo; do not sync it.
ssh roy@10.0.0.18 'mkdir -p ~/talowe-rv1126b-aidemo/talowe_yolov5_seg_cam'
rsync -av --progress \
--exclude '.git' \
--exclude 'cpp/3rdparty/opencv' \
--exclude 'cpp/build' \
--exclude 'cpp/install' \
~/SDK/talowe-rv1126b-aidemo/talowe_yolov5_seg_cam/ \
roy@10.0.0.18:~/talowe-rv1126b-aidemo/talowe_yolov5_seg_cam/
Confirm model/yolov5_seg.rknn is about 8.6 MB and already in the project model/.
Cross-compile
ssh roy@10.0.0.18
cd ~/talowe-rv1126b-aidemo/talowe_yolov5_seg_cam/cpp
chmod +x build-linux.sh
./build-linux.sh
Output:
~/talowe-rv1126b-aidemo/talowe_yolov5_seg_cam/cpp/install/rv1126b_linux_aarch64/rknn_yolov5_seg_cam/
# lib model play.sdp rknn_yolov5_seg_cam run_seg.sh seg.sdp
CMakeLists.txt / build-linux.sh use aarch64-linux-gnu-gcc and link against the board's existing librockchip_mpp (a copy of the so is included in the package).
Push to the board
The board's Buildroot usually has no rsync; create a tar on the build machine and extract it:
ssh roy@10.0.0.18 \
'tar czf - -C ~/talowe-rv1126b-aidemo/talowe_yolov5_seg_cam/cpp/install/rv1126b_linux_aarch64 rknn_yolov5_seg_cam' \
| ssh root@10.0.2.0 'tar xzf - -C /root'
You can also use this when USB is connected via adb:
adb push rknn_yolov5_seg_cam /root/
Then go back to steps 1 and 2 in Demonstration above.
Data path
IMX / Sensor
→ rkisp_mainpath /dev/video12 (NV12 1920x1080)
→ copy one NV12 frame for encoding (masks / boxes / text)
→ RGA convert to RGB888
→ RKNN YOLOv5-Seg (80 COCO classes + proto mask)
→ overlay semi-transparent masks / boxes / class names / FPS on NV12
→ MPP H.264 (each IDR carries SPS/PPS)
→ RTP PT=96 UDP → computer:5000
→ VLC opens seg.sdp
The resolution and port are written in cpp/main.cc, cpp/run_seg.sh, seg.sdp / play.sdp; when changing 1080p / 5000, change all three places.
FAQ
| Symptom | Handling |
|---|---|
| VLC stays black | Close VLC first, reopen seg.sdp, then start the board side; the third parameter must be the computer IP; allow UDP 5000 locally |
Camera open failed / VIDIOC_S_FMT failed | Other Demos / rkipc are still running. ./run_seg.sh stops them |
| All black after opening the camera | ISP 3A is not up. The script starts rkaiq_3A_server /etc/iqfiles; confirm the module is inserted |
| Picture present but no masks / boxes | The object must fully enter the frame; thresholds are in postprocess.h BOX_THRESH (default 0.25) |
Class names are all null / garbled | You must start from the package root directory; post-processing reads ./model/coco_80_labels_list.txt |
| Frame rate noticeably lower than the detection Demo | Segmentation post-processing is heavier; this is normal |
| Web page / RTSP gone after demonstration | rkipc not restored: killall rknn_yolov5_seg_cam then /etc/init.d/S99rkipc start |
Build missing aarch64-linux-gnu-gcc | On the build server apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu |
The RKNN model can not be found | yolov5_seg.rknn was not placed in the project model/ before compiling |
Camera node confirmation:
v4l2-ctl --list-devices
rkisp_mainpath (platform:rkisp-vir0):
/dev/video12
Differences from the detection Demo
| YOLO11 | This page YOLOv5-Seg | |
|---|---|---|
| Project | talowe_yolo11_detection_cam | talowe_yolov5_seg_cam |
| Model | yolo11.rknn ~4.2 MB | yolov5_seg.rknn ~8.6 MB |
| Output | Boxes + COCO class names | Boxes + colored masks + COCO class names |
| Stream receive | VLC yolo11.sdp | VLC seg.sdp |
| With rkipc | Must stop | Must stop |
Download source code and prebuilt packages
Link: https://pan.baidu.com/s/15rNLzfb12Af8Sh_DX-KVBw?pwd=fanc
Extraction code: fanc
The cloud drive contains two archives:
| Package | Purpose |
|---|---|
talowe-rv1126b-aidemo | Source code, cross-compile yourself |
talowe-rv1126b-aidemo-app | Already compiled board-side programs (aarch64); copy to /root/ on the board and they run |
Notes for sending to customers / using yourself:
run_*.shpushes H.264 to10.0.0.49:5000by default. This is an example computer IP and must be changed to the stream-receiving computer's own LAN IP:./run_xxx.sh <computer IP>.- Firmware must be CAM126B-P Buildroot IPC. The Demo and
rkipcshare/dev/video12and cannot run simultaneously; the script stopsrkipcfirst. - Open VLC on the computer with the sdp corresponding to this page (this page is
seg.sdp, port 5000); do not mix it with other Demos' sdp. - In
talowe-rv1126b-aidemo-app, please ignorerknn_yolov5_demo(leftover from the old image inference). For live camera detection userknn_yolov5_detection_cam. - Most Demos in the source have been changed to V4L2 + RGA + MPP; no need to include the entire
3rdparty/opencv(about 11 × 525 MB). Only PP-OCR cross-compilation still needs the in-project OpenCV static library.
Summary
- Open
seg.sdpin VLC on the computer (wait for UDP 5000) - On the board
cd /root/rknn_yolov5_seg_cam && ./run_seg.sh 10.0.0.49 - After screenshot / demonstration
killall rknn_yolov5_seg_cam,/etc/init.d/S99rkipc start