Accelerated GStreamer User Guide
Accelerated GStreamer User Guide
Accelerated GStreamer User Guide
ACCELERATED GSTREAMER
USER GUIDE
Release 32.1
DOCUMENT CHANGE HISTORY
DA_07303-4.0
Version Date Authors Description of Change
v1.0 01 May 2015 NVIDIA Initial release.
v1.1 30 Jun 2015 mzensius Added rotation and scaling commands, other
new content.
v1.2 03 Nov 2015 emilyh Changes for 23.1
v1.3 19 Nov 2015 mzensius Added note for display export.
Updated gst-nvivafilter sample pipelines.
v1.4 17 Dec 2015 hlang Updated steps to build gstreamer manually.
v1.5 08 Jan 2016 kstone Added nvvidconv interpolation method.
v1.5 29 Jan 2016 hlang Additional syntax changes for 23.2 release
v2.0 11 May 2016 mzensius Minor change to nvgstcapture options.
v3.0 11 Aug 2016 mzensius Versioned for 24.2 release. GStreamer-0.10
content removed. Also Adds Video Cropping
example, interpolation methods for video
scaling, EGLStream producer example, and an
EGL Image transform example.
v3.1 06 Oct 2016 mzensius Minor updates to video encoder features.
v3.1.1 21 Nov 2016 mzensius Changed title of document.
V3.2 12 Jan 2017 mzensius Adds H.264/H.265 encoder documentation.
Also corrects the GStreamer-1.0 installation
procedure.
3.2 03 Mar 2017 hlang Update date/moniker for L4T 27.1 release. No
other updates.
3.3 13 Jul 2017 mzensius Minor edit to command syntax, and update of
date/moniker for L4T 28.1 release.
3.4 01 Dec 2017 mzensius Includes support for NVIDIA® Jetson™ TX1,
previously documented elsewhere. Also
includes Overlay Sink information, and
formatting enhancements.
This document is a user guide for the GStreamer version 1.0 based accelerated solution
included in NVIDIA® Tegra® Linux Driver Package (L4T) for NVIDIA® Jetson AGX
Xavier™ devices.
To install GStreamer-1.0
Install GStreamer-1.0 on the platform with the following commands:
gst-inspect-1.0 --version
Note The gst-omx plugin is deprecated in Linux for Tegra (L4T) Release 32.1.
Use the gst-v4l2 plugin instead.
GStreamer version 1.0 includes the following EGL image video sink:
Video Sink Description
EGL/GLES videosink element, both the X11 and
nveglglessink
Wayland backends
nv3dsink EGL/GLES videosink element
GStreamer version 1.0 includes the following libjpeg based JPEG image video
encode/decode plugins:
JPEG Description
nvjpegenc JPEG encoder element
nvjpegdec JPEG decoder element
Note Execute this command on the target before starting the video
decode pipeline using gst-launch or nvgstplayer.
export DISPLAY=:0
Start the X server with xinit &, if it is not already running.
DECODE EXAMPLES
The examples in this section show how you can perform audio and video decode with
GStreamer.
Note To route audio over HDMI, set the alsasink property device as
follows:
hw:Tegra,3
Note For decode use cases with low memory allocation requirements (e.g. on
Jetson Nano), use the enable-low-outbuffer property of the gst-
omx decoder plugin.
For an example, see the pipeline below.
Note To enable max perf mode, use the disable-dvfs property of the gst-
omx decoder plugin. Expect increased power consumption in max perf
mode.
For an example, see the pipeline below.
Note If the primary display is NOT used to render video, use the display-
id property of nvoverlaysink.
For example, refer to the pipeline below.
ENCODE EXAMPLES
The examples in this section show how you can perform audio and video encode with
GStreamer.
gst-launch-1.0 audiotestsrc ! \
'audio/x-raw, format=(string)S16LE,
layout=(string)interleaved, rate=(int)44100, channels=(int)2' ! \
voaacenc ! qtmux ! filesink location=test.mp4 -e
gst-launch-1.0 audiotestsrc ! \
'audio/x-raw, format=(string)S16LE, layout=(string)interleaved, \
rate=(int)16000, channels=(int)1' ! voamrwbenc ! qtmux ! \
filesink location=test.mp4 -e
gst-launch-1.0 videotestsrc ! \
'video/x-raw, format=(string)I420, width=(int)640, \
height=(int)480' ! omxh264enc ! \
'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! \
qtmux ! filesink location=test.mp4 -e
gst-launch-1.0 videotestsrc ! \
'video/x-raw, format=(string)I420, width=(int)640, \
height=(int)480' ! omxh265enc ! filesink location=test.h265 -e
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! \
nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)I420_10LE' !
omxh265enc ! matroskamux ! filesink location=test_10bit.mkv -e
gst-launch-1.0 videotestsrc ! \
'video/x-raw, format=(string)I420, width=(int)640, \
height=(int)480' ! omxvp8enc ! matroskamux ! \
filesink location=test.mkv -e
VP9 Encode (NVIDIA Accelerated, Supported with Jetson TX2 and Jetson
AGX Xavier)
gst-launch-1.0 videotestsrc ! \
'video/x-raw, format=(string)I420, width=(int)640, \
height=(int)480' ! omxvp9enc ! matroskamux ! \
filesink location=test.mkv -e
gst-launch-1.0 videotestsrc ! \
'video/x-raw, format=(string)I420, width=(int)640, \
height=(int)480' ! avenc_mpeg4 ! qtmux ! \
filesink location=test.mp4 -e
gst-launch-1.0 videotestsrc ! \
'video/x-raw, format=(string)I420, width=(int)704, \
height=(int)576' ! avenc_h263 ! qtmux ! filesink location=test.mp4 -e
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! nvv4l2h264enc ! \
bitrate-8000000 ! h264parse ! qtmux ! filesink \
location=<filename_h264.mp4> -e
Note To enable max perf mode, use the maxperf-enable property of the
gst-v4l2 encoder plugin. Expect increased power consumption in
max perf mode.
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! nvv4l2h264enc \
maxperf-enable=1 bitrate=8000000 ! h264parse ! qtmux ! filesink \
location=<filename_h264.mp4> -e
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! nvv4l2h265enc \
bitrate=8000000 ! h265parse ! qtmux ! filesink \
location=<filename_h265.mp4> -e
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! nvv4l2vp9enc \
bitrate=8000000 ! matroskamux ! filesink \
location=<filename_vp9.mkv> -e
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! nvv4l2vp9enc \
enable-headers=1 bitrate=8000000 ! filesink \
location=<filename_vp9.vp9> -e
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! nvv4l2vp8enc \
bitrate=8000000 ! matroskamux ! filesink \
location=<filename_vp8.mkv> -e
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! nvv4l2vp8enc \
enable-headers=1 bitrate=8000000 ! filesink \
location=<filename_vp8.vp8> -e
Image Encode
It takes effect only in variable bit rate(control-rate=1) mode. By default, the value is
configured as (1.2*bitrate).
"<I_range>:<P_range>:<B_range>"
The range of B frames does not take effect if the number of B frames is 0.
Set Profile
Set Level
From omxh264enc, the following levels are supported: 1, 1b, 1.2, 1.3, 2, 2.1,
2.2, 3, 3.1, 3.2, 4, 4.1, 4.2, 5, 5.1, and 5.2.
From omxh265enc, the following levels are supported: main1, main2, main2.1,
main3, main3.1, main4, main4.1, main5, high1, high2, high2.1,
high3, high3.1, high4, high4.1, and high5.
If enabled, a sequence parameter set (SPS) and a picture parameter set (PPS) are inserted
before each IDR frame in the H.264/H.265 stream.
Two-pass CBR must be enabled along with constant bit rate (control-rate=2).
If the buffer size of decoder or network bandwidth is limited, configuring virtual buffer
size can cause video stream generation to correspond to the limitations according to the
following formula:
Stringent Bitrate must be enabled along with constant bit rate (control-rate=2), two-pass
CBR being enabled, and virtual buffer size being set.
The range of B frames does not take effect if the number of B frames is 0.
Mode Description
0 DisablePreset
1 UltraFastPreset
FastPreset
Only Integer Pixel (integer-pel) block motion is
2 estimated. For I/P macroblock mode decisions, only
Intra 16×16 cost is compared with intermode costs.
Supports intra 16×16 and intra 4×4 modes.
MediumPreset
Supports up to Half Pixel (half-pel) block motion
3 estimation. For I/P macroblock mode decisions, only
Intra 16×16 cost is compared with intermode costs.
Supports intra 16×16 and intra 4×4 modes.
SlowPreset
Supports up to Quarter Pixel (Qpel) block motion
4 estimation. For I/P macroblock mode decisions, intra
4×4 as well as intra 16×16 cost is compared with
intermode costs. Supports intra 16×16 and intra 4×4
modes.
If enabled, a sequence parameter set (SPS) and a picture parameter set (PPS) are inserted
before each IDR frame in the H.264 stream.
Two-pass CBR must be enabled along with constant bit rate (control-rate=1).
Note For multi-instance encode with two-pass CBR enabled, enable max perf
mode by using the maxperf-enable property of the gst-v4l2 encoder
to achieve best performance. Expect increased power consumption in
max perf mode.
This property sets the number of B frames between two reference frames.
nvgstcapture-1.0 --help
The nvgstcapture-1.0 application uses the v4l2src plugin to capture still images
and video.
nvgstplayer-1.0 --help
Video can be output to HD displays using the HDMI connector on the platform. The
GStreamer-1.0 application supports currently the following video sinks:
Note The nvoverlaysink plugin is deprecated in L4T Release 32.1. Use the
nvdrmvideosink plugin for development.
This nvgstplayer-1.0 application supports specific window position and dimensions for
windowed playback:
Ubuntu 16.04 does not support the Wayland display server. That is, there is no UI
support to switch to Wayland from Xorg. You must start the Wayland server (Weston)
using the target’s shell before performing any Weston based operation.
To start Weston:
The following steps are required before you first run the GStreamer pipeline with the
Wayland backend. They are not required on subsequent runs.
unset DISPLAY
mkdir /tmp/xdg
chmod 700 /tmp/xdg
Use the following command to start the GStreamer pipeline using nveglglesink with
the Wayland backend:
Properties
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! \
'video/x-raw(memory:NVMM), format=(string)I420’ ! omxh264enc ! \
qtmux ! filesink location=test.mp4 -e
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! \
'video/x-raw(memory:NVMM), format=(string)I420’ ! nvv4l2h264enc ! \
h264parse ! qtmux ! filesink location=test.mp4 -e
gst-launch-1.0 videotestsrc ! \
'video/x-raw, format=(string)I420, width=(int)1280, \
height=(int)720' ! nvvidconv ! \
'video/x-raw(memory:NVMM), width=(int)640, height=(int)480, \
format=(string)I420' ! omxh264enc ! \
'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! \
qtmux ! filesink location=test.mp4 -e
gst-launch-1.0 videotestsrc ! \
'video/x-raw, format=(string)I420, width=(int)1280, \
height=(int)720' ! nvvidconv ! \
'video/x-raw(memory:NVMM), width=(int)640, height=(int)480, \
format=(string)I420' ! nvv4l2h264enc ! \
'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! \
qtmux ! filesink location=test.mp4 -e
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! \
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! \
'video/x-raw(memory:NVMM), width=(int)640, height=(int)480, \
format=(string)NV12’ ! nvv4l2h264enc ! h264parse ! \
qtmux ! filesink location=test.mp4 -e
To crop video
Using the gst-omx decoder:
gst-videocuda
This GStreamer-1.0 plugin performs CUDA post-processing operations on decoder-
provided EGL images and render video using nveglglessink.
The following are sample pipeline creation and application usage commands.
gst-nvivafilter
This NVIDIA proprietary GStreamer-1.0 plugin performs pre/post and CUDA post-
processing operations on CSI camera captured or decoded frames, and renders video
using overlay video sink or video encode.
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, \
format=(string)NV12, framerate=(fraction)30/1' ! \
nvivafilter cuda-process=true \
customer-lib-name="libnvsample_cudaprocess.so" ! \
'video/x-raw(memory:NVMM), format=(string)NV12' ! nvoverlaysink -e
The following table shows the supported values for the nvvidconv flip-method
property.
gst-launch-1.0 nvarguscamerasrc! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! \
'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
gst-launch-1.0 nvarguscamerasrc ! \
'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, \
format=(string)NV12, framerate=(fraction)30/1' ! \
nvvidconv flip-method=3 ! 'video/x-raw(memory:NVMM), \
width=(int)480, height=(int)640, format=(string)I420' ! \
nvoverlaysink -e
Prerequisites
Install the following dependent GStreamer package.
Clear the registry cache file, in case there is an issue with gst-inspect-1.0
nvcompositor.
$ rm .cache/gstreamer-1.0/registry.aarch64.bin
gst-launch-1.0 nvcompositor \
name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 \
sink_0::height=1080 sink_1::xpos=0 sink_1::ypos=0 \
sink_1::width=1600 sink_1::height=1024 sink_2::xpos=0 \
sink_2::ypos=0 sink_2::width=1366 sink_2::height=768 \
sink_3::xpos=0 sink_3::ypos=0 sink_3::width=1024 \
sink_3::height=576 ! nvoverlaysink display-id=1 \
filesrc location=<filename_h264_1080p_30fps.mp4> ! qtdemux ! \
h264parse ! omxh264dec ! comp. filesrc \
location=< filename_h265_1080p_30fps.mp4> ! qtdemux ! h265parse ! \
omxh265dec ! comp. filesrc \
location=< filename_vp8_1080p_30fps.webm> matroskademux ! \
omxvp8dec ! \
comp. filesrc location=<filename_vp9_1080p_30fps.webm> ! \
matroskademux ! omxvp9dec ! comp. -e
The following table shows the supported values for the nvvidconv
interpolation-method property.
nvgstcapture-1.0 --camsrc=3
This release contains the gst-install script to install a specific GStreamer version.
This section provides a procedure for building current versions of GStreamer.
Where <install_path> is the location where you are installing GStreamer and
<version> is the GStreamer version. For example:
export LD_LIBRARY_PATH=<install_path>/lib/aarch64-linux-gnu
export PATH=<install_path>/bin:$PATH
Where <install_path> is the location where you are installing GStreamer. For
example:
export LD_LIBRARY_PATH=/home/ubuntu/gst-1.15.2/lib/aarch64-linux-gnu
export PATH=/home/ubuntu/gst-1.15.2/bin:$PATH
http://gstreamer.freedesktop.org/src/
The following are the files you need from version 1.15.2:
● gstreamer-1.15.2.tar.xz
● gst-plugins-base-1.15.2.tar.xz
● gst-plugins-good-1.15.2.tar.xz
● gst-plugins-bad-1.15.2.tar.xz
● gst-plugins-ugly-1.15.2.tar.xz
export PKG_CONFIG_PATH=/home/ubuntu/gst_1.15.2/out/lib/pkgconfig
7. Build gstreamer (in this example, gstreamer-1.15.2) with the following commands:
./configure --prefix=/home/ubuntu/gst_1.15.2/out
make
make install
make
make install
12. Set the LD_LIBRARY_PATH environment variable with the following command:
export LD_LIBRARY_PATH=/home/ubuntu/gst_1.15.2/out/lib/
13. Copy the nvidia gstreamer-1.0 libraries to the gst_1.15.2 plugin directory
using the following command:
cd /usr/lib/aarch64-linux-gnu/gstreamer-1.0/
cp libgstnv* libgstomx.so \
~/gst_1.15.2/out/lib/gstreamer-1.0/
libgstnvarguscamera.so
libgstnvcompositor.so
libgstnvdrmvideosink.so
libgstnveglglessink.so
libgstnveglstreamsrc.so
libgstnvegltransform.so
libgstnvivafilter.so
libgstnvjpeg.so
libgstnvtee.so
libgstnvvidconv.so
libgstnvvideo4linux2.so
libgstnvvideocuda.so
libgstnvvideosink.so
libgstnvvideosinks.so
libgstomx.so
NOTES
The nvgstcapture-1.0 application generates image and video output files in the same
directory as the application itself.
Filenames for image and video content are in the formats, respectively:
● nvcamtest_<pid>_<sensor_id>_<counter>.jpg
● nvcamtest_<pid>_<sensor_id>_<counter>.mp4
Where:
Switch Description
-d <seconds> Play duration of the segment in media in seconds.
--duration <seconds>
--no-sync Disable AV sync.
--disable-dpms Unconditionally disable DPMS/ScreenBlanking during
operation; re-enable on exit.
--stealth Operate in stealth mode, staying alive even when no
media is playing.
--bg Operate in background mode, ignoring keyboard input.
--use-playbin Use Playbin.
--no-audio Disable audio.
--no-video Disable video.
--disable-anative Disable native audio rendering.
--disable-vnative Disable native video rendering.
--use-buffering Use buffering.
-l <percent> Low threshold for buffering to start, in percent.
--low-percent <percent>
-j <percent> High threshold for buffering to finish, in percent.
--high-percent <percent>
--loop-forever Play the URI(s) in an endless loop.
-t <seconds> Maximum time in queue, in seconds (0=automatic).
--max-size-time <seconds>
-y <n> Maximum amount of memory in the queue, in bytes
--max-size-bytes <n> (0=automatic).
Switch Description
--svd (=) chain for video decoding.
--sad (=) chain for audio decoding.
--svc (=) chain for video postprocessing.
--sac (=) chain for audio postprocessing.
--svs (=) chain for video rendering.
--sas (=) chain for audio rendering.
--shttp (=) chain for http source.
--srtsp (=) chain for rtsp source.
--sudp (=) chain for udp source.
--sfsrc (=) chain for file source.
Description
Command Example
p Pause playback.
r Start/resume playback.
z Stop playback.
i:<uri> Enter a single URI.
CSI CAMERAS
NVIDIA® Jetson Nano™, NVIDIA® Jetson AGX Xavier™, and NVIDIA® Jetson™
TX2 can capture camera images via CSI interface.
Jetson Nano, Jetson AGX Xavier, and Jetson TX2 all support both YUV and RAW
Bayer capture data.
GStreamer supports simultaneous capture from multiple CSI cameras. Suppoert is
validated using the nvgstcapture application.
Capture is validated for SDR, PWL HDR and DOL HDR modes for various sensors
using the nvgstcapture application.
http://www.logitech.com/en-in/product/hd-pro-webcam-c920
http://www.e-consystems.com/UltraHD-USB-Camera.asp
● USB 3.0
● UVC compliant
● 3840 x 2160 at 30 FPS | 4224 x 3156 at 13 FPS
● Purpose - Embedded Navigation
● Test using the nvgstcapture app.
● Issues encountered:
― FPS cannot be fixed. Changes based on exposure.
― FPS cannot be changed. Needs payment to vendor to get the support added
to their firmware.
Trademarks
NVIDIA, the NVIDIA logo, CUDA, Jetson, Nano, AGX Xavier, TX2, TX1, and Tegra are trademarks or registered
trademarks of NVIDIA Corporation in the United States and other countries. Other company and product names
may be trademarks of the respective companies with which they are associated.
Copyright© 2015−2019 NVIDIA Corporation. All rights reserved.
www.nvidia.com