GStreamer
← RZ-G
Application Note by Renesas
This User's Manual (Application Note) explains GStreamer and the custom plug-ins created by Renesas.
Usage examples are also included.
- RZ/G2L Group and RZ/V2L Group Linux Interface Specification GStreamer User's Manual
- https://www.renesas.com/us/en/document/mas/rzg2l-group-and-rzv2l-group-linux-interface-specification-gstreamer-users-manual-software?utm_nooverride=1&language=en
Examples for RZ/G2L
Down Scaling using vspmfilter
- Below table show range of support:
Input resolution | Output scale down ratio | |
Min | 128x72 | 1 |
Max | 1920x1080 | 15 |
Following are sample commands for Up/Down Scaling with h264 Elementary Stream file.
Case of Down Scaling
gst-launch-1.0 filesrc location=./<FullHDsize_h264_file> ! h264parse ! queue ! omxh264dec ! queue ! vspmfilter outbuf-alloc=true ! video/x-raw, format=BGRA, width=1280, height=720 ! waylandsink
Case of Down Scaling (without color format conversion)
gst-launch-1.0 filesrc location=./<FullHDsize_h264_file> ! h264parse ! queue ! omxh264dec ! queue ! vspmfilter outbuf-alloc=true ! video/x-raw, width=1280, height=720 ! waylandsink
Case of Down Scaling (without color format conversion – use dmabuf)
gst-launch-1.0 filesrc location=./<FullHDsize_h264_file> ! h264parse ! queue ! omxh264dec ! queue ! vspmfilter dmabuf-use=true ! video/x-raw, width=1280, height=720 ! waylandsink
Color format conversion using vspmfilter
- Below table show color format conversion using vspmfilter:
No. | Supported color format |
1 | BGRA |
2 | BGRx |
3 | RGB16 |
4 | YUY2 |
5 | NV12 |
Following are sample commands for converting color format. Please set the output format which begin supported following “format=”.
Case of BGRA output
gst-launch-1.0 filesrc location=filename.mp4 ! qtdemux ! queue ! h264parse ! omxh264dec ! queue ! vspmfilter outbuf-alloc=true ! video/x-raw, format=BGRA ! waylandsink
Camera Setup
MIPI Camera Setup (OV5645 camera)
Copy below script in a file say v412-init.sh and run the script first before using camera.
#!/bin/sh cru=$(cat /sys/class/video4linux/video*/name | grep "CRU") csi2=$(cat /sys/class/video4linux/v4l-subdev*/name | grep "csi2") # Please choose one of a following resolution then comment out the rest. ov5645_res=1280x960 #ov5645_res=1920x1080 if [ -z "$cru" ] then echo "No CRU video device founds" else media-ctl -d /dev/media0 -r if [ -z "$csi2" ] then echo "No MIPI CSI2 sub video device founds" else media-ctl -d /dev/media0 -l "'rzg2l_csi2 10830400.csi2':1 -> 'CRU output':0 [1]" media-ctl -d /dev/media0 -V "'rzg2l_csi2 10830400.csi2':1 [fmt:UYVY8_2X8/$ov5645_res field:none]" media-ctl -d /dev/media0 -V "'ov5645 0-003c':0 [fmt:UYVY8_2X8/$ov5645_res field:none]" echo "Link CRU/CSI2 to ov5645 0-003c with format UYVY8_2X8 and resolution $ov5645_res" fi fi
USB Camera Setup
$ rmmod uvcvideo $ insmod /lib/modules/5.10.83-cip1-yocto-standard/kernel/drivers/media/usb/uvc/uvcvideo.ko allocators=1
MIPI Camera Examples
Capture from camera and display
gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=1280,height=960 ! waylandsink
Capture and Scaling and Conversion using vspmfilter
gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=1280,height=960 ! vspmfilter outbuf-alloc=true ! video/x-raw, format=BGRA, width=640, height=480 ! waylandsink
v4l2src resolution must be specified to get the scaling working.
H.264 encode and save to a MP4 container file (using omxh264enc)
gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 ! video/x-raw,width=1280,height=960 ! vspmfilter dmabuf-use=true ! video/x-raw,format=NV12 ! omxh264enc control-rate=2 target-bitrate=10485760 interval_intraframes=14 periodicty-idr=2 ! video/x-h264,profile=\(string\)high,level=\(string\)4.2 ! filesink location=output.mp4
H.264 decode stream (using omxh264dec)
gst-launch-1.0 filesrc location=output.mp4 ! h264parse ! omxh264dec ! waylandsink
USB Camera Examples
Encode an .264 stream from camera (using omxh264enc)
gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 ! video/x-raw,width=1280,height=960 ! vspmfilter dmabuf-use=true ! video/x-raw,format=NV12 ! omxh264enc control-rate=2 target-bitrate=10485760 interval_intraframes=14 periodicty-idr=2 ! video/x-h264,profile=\(string\)high,level=\(string\)4.2 ! filesink location=output_xsga.264
Decode (using omxh264dec)
gst-launch-1.0 filesrc location=output_xsga.264 ! h264parse ! omxh264dec ! waylandsink
Stream H.264 video via Ethernet
Create .sdp file for VLC to receive stream using rtsp
Let create a file named test.sdp and copy the below content in the file
NOTE: The IP address is the IP4 is the IP address of the stream sender
m=video 9001 RTP/AVP 96, 9001 is the port number which needs to be matched with the sender port
v=0 i=RZ Board Demo c=IN IP4 192.168.86.57 s=ESP H264 STREAM m=video 9001 RTP/AVP 96 a=rtpmap:96 H264/90000
Save the file and launch the VLC application by clicking test.sdp file to receive the stream.
Stream H.264 compressed test-pattern video via Ethernet to VLC media player
NOTE: host ip address is the ip address of PC running VLC application gst-launch-1.0 -v videotestsrc ! queue ! videoconvert ! queue ! video/x-raw, width=320, height=240 ! queue ! omxh264enc ! queue ! rtph264pay ! queue ! udpsink host= 192.168.86.36 port=9001
Send h.264 compressed Camera video over Ethernet to the VLC media player
gst-launch-1.0 --gst-debug=3 v4l2src device=/dev/video1 ! video/x-raw, width=1280, height=720 ! videoconvert \ ! video/x-raw, format=NV12 ! omxh264enc control-rate=2 target-bitrate=10485760 interval_intraframes=14 periodicty-idr=2 use-dmabuf=false \ ! video/x-h264, profile=\(string\)high,level=\(string\)4.2 ! h264parse ! rtph264pay ! queue ! udpsink host=192.168.86.36 port=9001
Send h.264 compressed MP4 video over Ethernet to the VLC media player
gst-launch-1.0 -v filesrc location=sintel_trailer-1080p.mp4 ! decodebin ! omxh264enc ! rtph264pay ! udpsink host=192.168.86.36 port=9001
Send and receive h264 video between the devices over Ethernet using gstreamer
Sender: gst-launch-1.0 -v filesrc location=sintel_trailer-1080p.mp4 ! decodebin ! omxh264enc ! rtph264pay ! udpsink host=192.168.86.36 port=9001
Receiver: gst-launch-1.0 -v udpsrc port=9001 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! videoconvert ! autovideosink
Examples for RZV2L
Camera Setup
This section describes how to setup the RZV2L Camera module OV5645.
Camera Setup VGA
media-ctl -d /dev/media0 -r ;
media-ctl -d /dev/media0 -l "'rzg2l_csi2 10830400.csi2':1 -> 'CRU output':0 [1]" ;
media-ctl -d /dev/media0 -V "'rzg2l_csi2 10830400.csi2':1 [fmt:UYVY8_2X8/640x480 field:none]" ;
media-ctl -d /dev/media0 -V "'ov5645 0-003c':0 [fmt:UYVY8_2X8/640x480 field:none]" ;
gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! waylandsink
Camera Setup 720p
media-ctl -d /dev/media0 -r ;
media-ctl -d /dev/media0 -l "'rzg2l_csi2 10830400.csi2':1 -> 'CRU output':0 [1]" ;
media-ctl -d /dev/media0 -V "'rzg2l_csi2 10830400.csi2':1 [fmt:UYVY8_2X8/1280x720 field:none]" ;
media-ctl -d /dev/media0 -V "'ov5645 0-003c':0 [fmt:UYVY8_2X8/1280x720 field:none]" ;
gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! waylandsink
Down Scaling using vspmfilter
Down Scale Camera 720p video
gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! vspmfilter dmabuf-use=true ! video/x-raw, width=640, height=480 ! waylandsink
Down Scale H264 Video File
gst-launch-1.0 filesrc location=./in720p.mp4 ! qtdemux ! queue ! h264parse ! queue ! omxh264dec ! queue ! ! vspmfilter dmabuf-use=true ! video/x-raw, width=640, height=480 !
waylandsink
Create H264 Video File
This script can be run on the RZV2L
gst-launch-1.0 videotestsrc ! omxh264enc ! filesink location=./testvideo.h264
Stream H264 Video Over IP
RZV2L(Source) Device send h264 stream using RTP:
Set the IP address to the PC or device that receives the video
gst-launch-1.0 -v v4l2src device=/dev/video0 ! videoconvert ! vspmfilter dmabuf-use=true ! video/x-raw, width=640, height=480 ! queue ! omxh264enc ! queue ! rtph264pay ! queue ! udpsink host=<PC IP Address> port=5000
PC that receives h264 rtp stream:
gst-launch-1.0 -v udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! videoconvert ! autovideosink
PC receives H264 RTP Stream with VLC Player:
1) Create a SDP script file ( i.e. test.sdp ).
2) Add the following to the file. The variable IP_ADDR_HOST_VID needs to be changed to the device (RZV2L) that is hosting the video.
c=IN IP4 IP_ADDR_HOST_VID
m=video 5000 RTP/AVP 96
a=rtpmap:96 H264/90000
3) Run the following Command
vlc --avcodec-hw=vaapi test.sdp
RZ GStreamer C Level API Demos
Here is a an example of how to use the GStreamer C Code API for the RZV and RZG. The demo utilizes the RZ H.264 and VSPM library.
Examples for RZ/G2UL
Compared to the big brothers the RZ/G2UL does not feature a GPU nor a HW video encoder / decoder. Nevertheless gstreamer can still be used, mainly using SW plugins and the CPU and, of course, the performance is limited due to this.
Camera
MIPI Camera Setup (OV5645 camera)
Since many VLP releases now, v412-init.sh
is installed by default on the /home/root folder. It has to be run before the following commands can be given.
MIPI Camera Examples
Capture from camera and display
gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=ARGB,width=1280,height=960 ! vspmfilter outbuf-alloc=true ! waylandsink
Capture and Scaling and Conversion using vspmfilter
gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=ARGB,width=1280,height=960 ! vspmfilter outbuf-alloc=true ! video/x-raw, format=BGRA, width=640, height=480 ! waylandsink
Video decoding
As mentioned there's no HW video decoder, so SW decoding gstreamer plugins have to be used. Low resolution videos are normally played ok, medium / high resolution may be challenging.
VP8 decoding
gst-launch-1.0 filesrc location=./[VP8 encoded file].webm ! matroskademux ! avdec_vp8 ! videoconvert ! waylandsink
VP9 decoding
gst-launch-1.0 filesrc location=./[VP9 encoded file].webm ! matroskademux ! avdec_vp9 ! videoconvert ! waylandsink
H.264 decoding
gst-launch-1.0 filesrc location=./[H.264 encoded file].mp4 ! qtdemux ! h264parse ! avdec_h264 ! videoconvert ! waylandsink
H.264 decoding and down scaling
gst-launch-1.0 filesrc location=./[H.264 encoded file].mp4 ! qtdemux ! h264parse ! avdec_h264 ! videoscale ! videoconvert ! video/x-raw, width=640, height=480 ! waylandsink