RZ-G/RZG GStreamer: Difference between revisions

From Renesas.info
 
(3 intermediate revisions by one other user not shown)
Line 16: Line 16:
* https://www.renesas.com/us/en/document/mas/rzg2l-group-and-rzv2l-group-linux-interface-specification-gstreamer-users-manual-software?utm_nooverride=1&language=en
* https://www.renesas.com/us/en/document/mas/rzg2l-group-and-rzv2l-group-linux-interface-specification-gstreamer-users-manual-software?utm_nooverride=1&language=en


= Examples for RZ/G2L =
= Sample Code by Renesas =
Sample code for using GStreamer on Renesas RZ devices are stored in '''github'''.


== Down Scaling using vspmfilter ==
* https://github.com/renesas-rz/rz_gstreamer_sample_code


* Below table show range of support:
= Examples for RZ/G2L-V2L =
{| class="wikitable"
For most common use cases, refer to the gstreamer user's manual linked above. Below are reported other specific cases not covered in that document.
|+ style="caption-side:bottom;"|''Supported solution range''
|
| Input resolution
| Output scale down ratio
|-
| Min
| 128x72
| 1
|-
| Max
| 1920x1080
| 15
|}


Following are sample commands for Up/Down Scaling with h264 Elementary Stream file.
== USB Camera Examples ==
===Case of Down Scaling===
<pre>
gst-launch-1.0 filesrc location=./<FullHDsize_h264_file> ! h264parse ! queue ! omxh264dec ! queue ! vspmfilter outbuf-alloc=true ! video/x-raw, format=BGRA, width=1280, height=720 ! waylandsink
</pre>
===Case of Down Scaling (without color format conversion)===
<pre>
gst-launch-1.0 filesrc location=./<FullHDsize_h264_file> ! h264parse ! queue ! omxh264dec ! queue ! vspmfilter outbuf-alloc=true ! video/x-raw, width=1280, height=720 ! waylandsink
</pre>
===Case of Down Scaling (without color format conversion – use dmabuf)===
<pre>
gst-launch-1.0 filesrc location=./<FullHDsize_h264_file> ! h264parse ! queue ! omxh264dec ! queue ! vspmfilter dmabuf-use=true ! video/x-raw, width=1280, height=720 ! waylandsink
</pre>
 
== Color format conversion using vspmfilter ==
* Below table show color format conversion using vspmfilter:
 
{| class="wikitable"
|+ style="caption-side:bottom;"|''Supported color format''
| No.
| Supported color format
|-
| 1
| BGRA
|-
| 2
| BGRx
|-
| 3
| RGB16
|-
| 4
| YUY2
|-
| 5
| NV12
|}
 
Following are sample commands for converting color format. Please set the output format which begin supported following “format=”.
===Case of BGRA output===
<pre>
gst-launch-1.0 filesrc location=filename.mp4 ! qtdemux ! queue ! h264parse ! omxh264dec ! queue ! vspmfilter outbuf-alloc=true ! video/x-raw, format=BGRA ! waylandsink
</pre>
== Camera Setup ==
=== MIPI Camera Setup (OV5645 camera) ===
Copy below script in a file say v412-init.sh and run the script first before using camera.
<pre>
#!/bin/sh
 
cru=$(cat /sys/class/video4linux/video*/name | grep "CRU")
csi2=$(cat /sys/class/video4linux/v4l-subdev*/name | grep "csi2")
 
# Please choose one of a following resolution then comment out the rest.
ov5645_res=1280x960
#ov5645_res=1920x1080
 
if [ -z "$cru" ]
then
        echo "No CRU video device founds"
else
        media-ctl -d /dev/media0 -r
        if [ -z "$csi2" ]
        then
                echo "No MIPI CSI2 sub video device founds"
        else
                media-ctl -d /dev/media0 -l "'rzg2l_csi2 10830400.csi2':1 -> 'CRU output':0 [1]"
                media-ctl -d /dev/media0 -V "'rzg2l_csi2 10830400.csi2':1 [fmt:UYVY8_2X8/$ov5645_res field:none]"
                media-ctl -d /dev/media0 -V "'ov5645 0-003c':0 [fmt:UYVY8_2X8/$ov5645_res field:none]"
                echo "Link CRU/CSI2 to ov5645 0-003c with format UYVY8_2X8 and resolution $ov5645_res"
        fi
fi
</pre>
 
=== USB Camera Setup ===
<pre>
$ rmmod uvcvideo
$ insmod /lib/modules/5.10.83-cip1-yocto-standard/kernel/drivers/media/usb/uvc/uvcvideo.ko allocators=1
</pre>
 
=== MIPI Camera Examples ===
==== Capture from camera and display ====
<pre>
gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=1280,height=960 ! waylandsink
</pre>
==== Capture and Scaling and Conversion using vspmfilter ====
<pre>
gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=1280,height=960 ! vspmfilter outbuf-alloc=true ! video/x-raw, format=BGRA, width=640, height=480 !  waylandsink
</pre>v4l2src resolution must be specified to get the scaling working.
 
====H.264 encode and save to a MP4 container file (using omxh264enc)====
<pre>
gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 ! video/x-raw,width=1280,height=960 ! vspmfilter dmabuf-use=true ! video/x-raw,format=NV12 ! omxh264enc control-rate=2 target-bitrate=10485760 interval_intraframes=14 periodicty-idr=2 ! video/x-h264,profile=\(string\)high,level=\(string\)4.2 ! filesink location=output.mp4
</pre>
====H.264 decode stream (using omxh264dec)====
<pre>
gst-launch-1.0 filesrc location=output.mp4 ! h264parse ! omxh264dec ! waylandsink
</pre>
 
=== USB Camera Examples ===
====Encode an .264 stream from camera (using omxh264enc)====
====Encode an .264 stream from camera (using omxh264enc)====
<pre>
<pre>
gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 ! video/x-raw,width=1280,height=960 ! vspmfilter dmabuf-use=true ! video/x-raw,format=NV12 ! omxh264enc control-rate=2 target-bitrate=10485760 interval_intraframes=14 periodicty-idr=2 ! video/x-h264,profile=\(string\)high,level=\(string\)4.2 ! filesink location=output_xsga.264
gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 ! video/x-raw,width=1280,height=960 ! vspmfilter dmabuf-use=true ! video/x-raw,format=NV12 ! omxh264enc control-rate=2 target-bitrate=10485760 interval_intraframes=14 periodicty-idr=2 ! video/x-h264,profile=\(string\)high,level=\(string\)4.2 ! filesink location=output_xsga.264
</pre>
</pre>
====Decode (using omxh264dec)====
====Stream H.264 video via Ethernet====
<pre>
gst-launch-1.0 filesrc location=output_xsga.264 ! h264parse ! omxh264dec ! waylandsink
</pre>
=== Stream H.264 video via Ethernet===
====Create .sdp file for VLC to receive stream using rtsp====
====Create .sdp file for VLC to receive stream using rtsp====
Let create a file named test.sdp and copy the below content in the file<br>
Let create a file named test.sdp and copy the below content in the file<br>
Line 174: Line 60:
</pre>
</pre>


====Send and receive h264 video between the devices over Ethernet using gstreamer====
= RZ GStreamer C Level API Demos =
<pre>
Here is a an example of how to use the GStreamer C Code API for the RZV and RZG. The demo utilizes the RZ H.264 and VSPM library.
Sender:
gst-launch-1.0 -v filesrc location=sintel_trailer-1080p.mp4 ! decodebin ! omxh264enc ! rtph264pay ! udpsink host=192.168.86.36 port=9001
</pre>
<pre>
Receiver:
gst-launch-1.0 -v udpsrc port=9001 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! videoconvert ! autovideosink
</pre>


= Examples for RZV2L =
[https://github.com/renesas-rz/rz_gstreamer_demos RZ GStreamer Demo]


== Camera Setup ==
= Examples for RZ/G2UL =
This section describes how to setup the RZV2L Camera module OV5645.  
Compared to the big brothers the RZ/G2UL does not feature a GPU nor a HW video encoder / decoder. Nevertheless gstreamer can still be used, mainly using SW plugins and the CPU and, of course, the performance is limited due to this.


=== Camera Setup VGA ===
== Camera ==
<code>media-ctl -d /dev/media0 -r ;</code>
=== MIPI Camera Setup (OV5645 camera) ===
Since many VLP releases now, <code>v412-init.sh</code> is installed by default on the /home/root folder. It has to be run before the following commands can be given.


<code>media-ctl -d /dev/media0 -l "'rzg2l_csi2 10830400.csi2':1 -> 'CRU output':0 [1]" ;</code>
=== MIPI Camera Examples ===


<code>media-ctl -d /dev/media0 -V "'rzg2l_csi2 10830400.csi2':1 [fmt:UYVY8_2X8/640x480 field:none]" ;</code>
==== Capture from camera and display ====
<pre>
gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=ARGB,width=1280,height=960 ! vspmfilter outbuf-alloc=true ! waylandsink
</pre>


<code>media-ctl -d /dev/media0 -V "'ov5645 0-003c':0 [fmt:UYVY8_2X8/640x480 field:none]" ;</code>
==== Capture and Scaling and Conversion using vspmfilter ====
<pre>
gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=ARGB,width=1280,height=960 ! vspmfilter outbuf-alloc=true ! video/x-raw, format=BGRA, width=640, height=480 ! waylandsink
</pre>


<code>gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! waylandsink</code>
== Video decoding ==
As mentioned there's no HW video decoder, so SW decoding gstreamer plugins have to be used. Low resolution videos are normally played ok, medium / high resolution may be challenging.


=== Camera Setup 720p ===
=== VP8 decoding ===
<code>media-ctl -d /dev/media0 -r ;</code>
<pre>
gst-launch-1.0 filesrc location=./[VP8 encoded file].webm ! matroskademux ! avdec_vp8 ! videoconvert ! waylandsink
</pre>


<code>media-ctl -d /dev/media0 -l "'rzg2l_csi2 10830400.csi2':1 -> 'CRU output':0 [1]" ;</code>
=== VP9 decoding ===
<pre>
gst-launch-1.0 filesrc location=./[VP9 encoded file].webm ! matroskademux ! avdec_vp9 ! videoconvert ! waylandsink
</pre>


<code>media-ctl -d /dev/media0 -V "'rzg2l_csi2 10830400.csi2':1 [fmt:UYVY8_2X8/1280x720 field:none]" ;</code>
=== H.264 decoding ===
 
<pre>
<code>media-ctl -d /dev/media0 -V "'ov5645 0-003c':0 [fmt:UYVY8_2X8/1280x720 field:none]" ;</code>
gst-launch-1.0 filesrc location=./[H.264 encoded file].mp4 ! qtdemux ! h264parse ! avdec_h264 ! videoconvert ! waylandsink
</pre>


<code>gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! waylandsink</code>
=== H.264 decoding and down scaling ===
 
gst-launch-1.0 filesrc location=./[H.264 encoded file].mp4 ! qtdemux ! h264parse ! avdec_h264 ! videoscale ! videoconvert ! video/x-raw, width=640, height=480 ! waylandsink
== Down Scaling using vspmfilter ==
 
=== Down Scale Camera 720p video ===
<code>gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! vspmfilter dmabuf-use=true ! video/x-raw, width=640, height=480 ! waylandsink</code>
 
=== Down Scale H264 Video File ===
gst-launch-1.0 filesrc location=./in720p.mp4 ! qtdemux ! queue ! h264parse ! queue ! omxh264dec ! queue ! <code>! vspmfilter dmabuf-use=true ! video/x-raw, width=640, height=480 !</code>waylandsink
 
=== Create H264 Video File ===
This script can be run on the RZV2L
 
<code>gst-launch-1.0 videotestsrc ! omxh264enc ! filesink location=./testvideo.h264</code>
 
== Stream H264 Video Over IP ==
 
=== RZV2L(Source)  Device send h264 stream using RTP: ===
'''''Set the IP address to the PC or device that receives the video'''''
 
<code>gst-launch-1.0 -v v4l2src device=/dev/video0 ! videoconvert ! vspmfilter dmabuf-use=true ! video/x-raw, width=640, height=480 ! queue ! omxh264enc ! queue ! rtph264pay ! queue ! udpsink host=<PC IP Address> port=5000</code>
 
=== PC that receives h264 rtp stream: ===
<code>gst-launch-1.0 -v udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! videoconvert ! autovideosink</code>
 
=== PC receives H264 RTP Stream with VLC Player: ===
1) Create a SDP script file ( i.e. test.sdp ).
 
2) Add the following to the file. The variable <code>IP_ADDR_HOST_VID needs to be changed to the device (RZV2L) that is hosting the video.</code>
 
<code>c=IN IP4 IP_ADDR_HOST_VID</code>
 
<code>m=video 5000 RTP/AVP 96</code>
 
<code>a=rtpmap:96 H264/90000</code>
 
3) Run the following Command
 
<code>vlc --avcodec-hw=vaapi test.sdp</code>
==RZ GStreamer C Level API Demos==
Here is a an example of how to use the GStreamer C Code API for the RZV and RZG. The demo utilizes the RZ H.264 and VSPM library.
 
[https://github.com/renesas-rz/rz_gstreamer_demos RZ GStreamer Demo]

Latest revision as of 13:20, 6 December 2023

← RZ-G



Application Note by Renesas

This User's Manual (Application Note) explains GStreamer and the custom plug-ins created by Renesas.

Usage examples are also included.

Sample Code by Renesas

Sample code for using GStreamer on Renesas RZ devices are stored in github.

Examples for RZ/G2L-V2L

For most common use cases, refer to the gstreamer user's manual linked above. Below are reported other specific cases not covered in that document.

USB Camera Examples

Encode an .264 stream from camera (using omxh264enc)

gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 ! video/x-raw,width=1280,height=960 ! vspmfilter dmabuf-use=true ! video/x-raw,format=NV12 ! omxh264enc control-rate=2 target-bitrate=10485760 interval_intraframes=14 periodicty-idr=2 ! video/x-h264,profile=\(string\)high,level=\(string\)4.2 ! filesink location=output_xsga.264

Stream H.264 video via Ethernet

Create .sdp file for VLC to receive stream using rtsp

Let create a file named test.sdp and copy the below content in the file
NOTE: The IP address is the IP4 is the IP address of the stream sender
m=video 9001 RTP/AVP 96, 9001 is the port number which needs to be matched with the sender port 

v=0
i=RZ Board Demo
c=IN IP4 192.168.86.57
s=ESP H264 STREAM
m=video 9001 RTP/AVP 96
a=rtpmap:96 H264/90000

Save the file and launch the VLC application by clicking test.sdp file to receive the stream.

Stream H.264 compressed test-pattern video via Ethernet to VLC media player

NOTE: host ip address is the ip address of PC running VLC application
gst-launch-1.0 -v videotestsrc ! queue ! videoconvert ! queue ! video/x-raw, width=320, height=240 ! queue ! omxh264enc ! queue ! rtph264pay ! queue ! udpsink host= 192.168.86.36 port=9001

Send h.264 compressed Camera video over Ethernet to the VLC media player

gst-launch-1.0 --gst-debug=3 v4l2src device=/dev/video1 ! video/x-raw, width=1280, height=720 ! videoconvert \
! video/x-raw, format=NV12 ! omxh264enc control-rate=2 target-bitrate=10485760 interval_intraframes=14 periodicty-idr=2 use-dmabuf=false \
! video/x-h264, profile=\(string\)high,level=\(string\)4.2 ! h264parse ! rtph264pay ! queue ! udpsink host=192.168.86.36 port=9001

Send h.264 compressed MP4 video over Ethernet to the VLC media player

 gst-launch-1.0 -v filesrc location=sintel_trailer-1080p.mp4 ! decodebin ! omxh264enc ! rtph264pay ! udpsink host=192.168.86.36 port=9001

RZ GStreamer C Level API Demos

Here is a an example of how to use the GStreamer C Code API for the RZV and RZG. The demo utilizes the RZ H.264 and VSPM library.

RZ GStreamer Demo

Examples for RZ/G2UL

Compared to the big brothers the RZ/G2UL does not feature a GPU nor a HW video encoder / decoder. Nevertheless gstreamer can still be used, mainly using SW plugins and the CPU and, of course, the performance is limited due to this.

Camera

MIPI Camera Setup (OV5645 camera)

Since many VLP releases now, v412-init.sh is installed by default on the /home/root folder. It has to be run before the following commands can be given.

MIPI Camera Examples

Capture from camera and display

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=ARGB,width=1280,height=960 ! vspmfilter outbuf-alloc=true ! waylandsink

Capture and Scaling and Conversion using vspmfilter

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=ARGB,width=1280,height=960 ! vspmfilter outbuf-alloc=true ! video/x-raw, format=BGRA, width=640, height=480 ! waylandsink

Video decoding

As mentioned there's no HW video decoder, so SW decoding gstreamer plugins have to be used. Low resolution videos are normally played ok, medium / high resolution may be challenging.

VP8 decoding

gst-launch-1.0 filesrc location=./[VP8 encoded file].webm ! matroskademux ! avdec_vp8 ! videoconvert ! waylandsink

VP9 decoding

gst-launch-1.0 filesrc location=./[VP9 encoded file].webm ! matroskademux ! avdec_vp9 ! videoconvert ! waylandsink

H.264 decoding

gst-launch-1.0 filesrc location=./[H.264 encoded file].mp4 ! qtdemux ! h264parse ! avdec_h264 ! videoconvert ! waylandsink

H.264 decoding and down scaling

gst-launch-1.0 filesrc location=./[H.264 encoded file].mp4 ! qtdemux ! h264parse ! avdec_h264 ! videoscale ! videoconvert ! video/x-raw, width=640, height=480 ! waylandsink
Retrieved from "https://renesas.info/w/index.php?title=RZ-G/RZG_GStreamer&oldid=2592"