"Fossies" - the Fresh Open Source Software Archive

Member "gst-plugins-good-1.20.3/gst/rtp/README" (15 Jun 2022, 17868 Bytes) of package /linux/misc/gst-plugins-good-1.20.3.tar.xz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file.

    1 This directory contains some RTP payloaders/depayloaders for different payload
    2 types. Use one payloader/depayloder pair per payload. If several payloads can be
    3 payloaded/depayloaded by the same element, make different copies of it, one for
    4 each payload.
    5 
    6 The application/x-rtp mime type
    7 -------------------------------
    8 
    9 For valid RTP packets encapsulated in GstBuffers, we use the caps with
   10 mime type application/x-rtp.
   11 
   12 The following fields can or must (*) be specified in the structure:
   13 
   14  * media: (String) [ "audio", "video", "application", "data", "control" ]
   15      Defined in RFC 2327 in the SDP media announcement field.
   16      Converted to lower case.
   17 
   18  * payload: (int) [0, 127]
   19      For audio and video, these will normally be a media payload type as 
   20      defined in the RTP Audio/Video Profile. For dynamically allocated 
   21      payload types, this value will be >= 96 and the encoding-name must be
   22      set.
   23 
   24  * clock-rate: (int) [0 - MAXINT]
   25     The RTP clock rate. 
   26 
   27    encoding-name: (String) ANY
   28      typically second part of the mime type. ex. MP4V-ES. only required if
   29      payload type >= 96. Converted to upper case.
   30 
   31    encoding-params: (String) ANY
   32      extra encoding parameters (as in the SDP a=rtpmap: field). only required
   33      if different from the default of the encoding-name.
   34      Converted to lower-case.
   35      
   36    ssrc: (uint) [0 - MAXINT]
   37     The ssrc value currently in use. (default = the SSRC of the first RTP
   38     packet)
   39 
   40    timestamp-offset: (uint) [0 - MAXINT]
   41     The RTP time representing time npt-start. (default = rtptime of first RTP
   42     packet).
   43 
   44    seqnum-offset: (uint) [0 - MAXINT]
   45     The RTP sequence number representing the first rtp packet. When this
   46     parameter is given, all sequence numbers below this seqnum should be
   47     ignored. (default = seqnum of first RTP packet).
   48 
   49    npt-start: (uint64) [0 - MAXINT]
   50     The Normal Play Time for clock-base. This is the position in the stream and
   51     is between 0 and the duration of the stream. This value is expressed in
   52     nanoseconds GstClockTime. (default = 0)
   53 
   54    npt-stop: (uint64) [0 - MAXINT] 
   55     The last position in the stream. This value is expressed in nanoseconds
   56     GstClockTime. (default = -1, stop unknown)
   57 
   58    play-speed: (gdouble) [-MIN - MAX]
   59     The intended playback speed of the stream. The client is delivered data at
   60     the adjusted speed. The client should adjust its playback speed with this
   61     value and thus corresponds to the GStreamer rate field in the NEWSEGMENT
   62     event. (default = 1.0)
   63     
   64    play-scale: (gdouble) [-MIN - MAX]
   65     The rate already applied to the stream. The client is delivered a stream
   66     that is scaled by this amount. This value is used to adjust position
   67     reporting and corresponds to the GStream applied-rate field in the
   68     NEWSEGMENT event. (default = 1.0)
   69 
   70    maxptime: (uint) [0, MAX]
   71     The maxptime as defined in RFC 4566, this defines the maximum size of a
   72     packet. It overrides the max-ptime property of payloaders.
   73 
   74    Optional parameters as key/value pairs, media type specific. The value type
   75    should be of type G_TYPE_STRING. The key is converted to lower-case. The
   76    value is left in its original case.
   77    A parameter with no value is converted to <param>=1.
   78 
   79  Example:
   80 
   81   "application/x-rtp",
   82       "media", G_TYPE_STRING, "audio",		-.
   83       "payload", G_TYPE_INT, 96,                 | - required
   84       "clock-rate", G_TYPE_INT, 8000,           -'
   85       "encoding-name", G_TYPE_STRING, "AMR",    -. - required since payload >= 96
   86       "encoding-params", G_TYPE_STRING, "1",	-' - optional param for AMR
   87       "octet-align", G_TYPE_STRING, "1",	-.
   88       "crc", G_TYPE_STRING, "0",                 |
   89       "robust-sorting", G_TYPE_STRING, "0",      |  AMR specific params.
   90       "interleaving", G_TYPE_STRING, "0",       -'
   91   
   92  Mapping of caps to and from SDP fields:
   93 
   94    m=<media> <udp port> RTP/AVP <payload>       -] media and payload from caps
   95    a=rtpmap:<payload> <encoding-name>/<clock-rate>[/<encoding-params>]
   96               -> when <payload> >= 96
   97    a=fmtp:<payload> <param>=<value>;...
   98 
   99  For above caps:
  100 
  101    m=audio <udp port> RTP/AVP 96
  102    a=rtpmap:96 AMR/8000/1
  103    a=fmtp:96 octet-align=1;crc=0;robust-sorting=0;interleaving=0
  104 
  105  Attributes are converted as follows:
  106 
  107   IANA registered attribute names are prepended with 'a-' before putting them in
  108   the caps. Unregistered keys (starting with 'x-') are copied directly into the
  109   caps.
  110 
  111  in RTSP, the SSRC is also sent.
  112 
  113  The optional parameters in the SDP fields are case insensitive. In the caps we
  114  always use the lowercase names so that the SDP -> caps mapping remains
  115  possible.
  116 
  117  Mapping of caps to NEWSEGMENT:
  118 
  119   rate:         <play-speed>
  120   applied-rate: <play-scale>
  121   format:       GST_FORMAT_TIME
  122   start:        <clock-base> * GST_SECOND / <clock-rate>
  123   stop:         if <ntp-stop> != -1
  124                   <npt-stop> - <npt-start> + start
  125 		else 
  126 		  -1
  127   time:         <npt-start>
  128 
  129 
  130 Timestamping
  131 ------------
  132 
  133 RTP in GStreamer uses a combination of the RTP timestamps and GStreamer buffer
  134 timestamps to ensure proper synchronisation at the sender and the receiver end.
  135 
  136 In RTP applications, the synchronisation is most complex at the receiver side.
  137 
  138 At the sender side, the RTP timestamps are generated in the payloaders based on
  139 GStreamer timestamps. At the receiver, GStreamer timestamps are reconstructed
  140 from the RTP timestamps and the GStreamer timestamps in the jitterbuffer. This
  141 process is explained in more detail below.
  142 
  143 = synchronisation at the sender
  144 
  145 Individual streams at the sender are synchronised using GStreamer timestamps.
  146 The payloader at the sender will convert the GStreamer timestamp into an RTP
  147 timestamp using the following formula:
  148 
  149    RTP = ((RT - RT-base) * clock-rate / GST_SECOND) + RTP-offset
  150 
  151   RTP:        the RTP timestamp for the stream. This value is truncated to
  152               32 bits.
  153   RT:         the GStreamer running time corresponding to the timestamp of the
  154               packet to payload
  155   RT-base:    the GStreamer running time of the first packet encoded
  156   clock-rate: the clock-rate of the stream
  157   RTP-offset: a random RTP offset
  158 
  159 The RTP timestamp corresponding to RT-base is the clock-base (see caps above). 
  160 
  161 In addition to setting an RTP timestamp in the RTP packet, the payloader is also
  162 responsible for putting the GStreamer timestamp on the resulting output buffer.
  163 This timestamp is used for further synchronisation at the sender pipeline, such
  164 as for sending out the packet on the network.
  165 
  166 Notice that the absolute timing information is lost; if the sender is sending
  167 multiple streams, the RTP timestamps in the packets do not contain enough
  168 information to synchronize them in the receiver. The receiver can however use
  169 the RTP timestamps to reconstruct the timing of the stream as it was created by
  170 the sender according to the sender's clock.
  171 
  172 Because the payloaded packet contains both an RTP timestamp and a GStreamer
  173 timestamp, it is possible for an RTP session manager to derive the relation
  174 between the RTP and GST timestamps. This information is used by a session
  175 manager to create SR reports. The NTP time in the report will contain the
  176 running time converted to NTP time and the corresponding RTP timestamp.
  177 
  178 Note that at the sender side, the RTP and GStreamer timestamp both increment at
  179 the same rate, the sender rate. This rate depends on the global pipeline clock
  180 of the sender. 
  181 
  182 Some pipelines to illustrate the process:
  183 
  184     gst-launch-1.0 v4l2src ! videoconvert ! avenc_h263p ! rtph263ppay ! udpsink
  185 
  186   v4l2src puts a GStreamer timestamp on the video frames base on the current
  187   running_time. The encoder encodes and passed the timestamp on. The payloader
  188   generates an RTP timestamp using the above formula and puts it in the RTP
  189   packet. It also copies the incoming GStreamer timestamp on the output RTP
  190   packet. udpsink synchronizes on the gstreamer timestamp before pushing out the
  191   packet. 
  192 
  193 
  194 = synchronisation at the receiver
  195 
  196 The receiver is responsible for timestamping the received RTP packet with the
  197 running_time of the clock at the time the packet was received. This GStreamer
  198 timestamp reflects the receiver rate and depends on the global pipeline clock of
  199 the receiver. The gstreamer timestamp of the received RTP packet contains a
  200 certain amount of jitter introduced by the network.
  201 
  202 The most simple option for the receiver is to depayload the RTP packet and play
  203 it back as soon as possible, this is with the timestamp when it was received
  204 from the network. For the above sender pipeline this would be done with the
  205 following pipeline:
  206 
  207     gst-launch-1.0 udpsrc caps="application/x-rtp, media=(string)video,
  208       clock-rate=(int)90000, encoding-name=(string)H263-1998" ! rtph263pdepay !
  209       avdec_h263 ! autovideosink
  210 
  211 It is important that the depayloader copies the incoming GStreamer timestamp
  212 directly to the depayloaded output buffer. It should never attempt to perform
  213 any logic with the RTP timestamp, this task is for the jitterbuffer as we will
  214 see next.
  215 
  216 The above pipeline does not attempt to deal with reordered packets or network
  217 jitter, which could result in jerky playback in the case of high jitter or
  218 corrupted video in the case of packet loss or reordering. This functionality is
  219 performed by the gstrtpjitterbuffer in GStreamer.
  220 
  221 The task of the gstrtpjitterbuffer element is to:
  222 
  223  - deal with reordered packets based on the seqnum
  224  - calculate the drift between the sender and receiver clocks using the
  225    GStreamer timestamps (receiver clock rate) and RTP timestamps (sender clock
  226    rate).
  227 
  228 To deal with reordered packet, the jitterbuffer holds on to the received RTP
  229 packets in a queue for a configurable amount of time, called the latency.
  230 
  231 The jitterbuffer also eliminates network jitter and then tracks the drift
  232 between the local clock (as expressed in the GStreamer timestamps) and the
  233 remote clock (as expressed in the RTP timestamps). It will remove the jitter
  234 and will apply the drift correction to the GStreamer timestamp before pushing
  235 the buffer downstream. The result is that the depayloader receives a smoothed
  236 GStreamer timestamp on the RTP packet, which is copied to the depayloaded data.
  237 
  238 The following pipeline illustrates a receiver with a jitterbuffer.
  239 
  240     gst-launch-1.0 udpsrc caps="application/x-rtp, media=(string)video,
  241       clock-rate=(int)90000, encoding-name=(string)H263-1998" !
  242       rtpjitterbuffer latency=100 ! rtph263pdepay !  avdec_h263 ! autovideosink
  243 
  244 The latency property on the jitterbuffer controls the amount of delay (in
  245 milliseconds) to apply to the outgoing packets. A higher latency will produce
  246 smoother playback in networks with high jitter but cause a higher latency.
  247 Choosing a good value for the latency is a tradeoff between the quality and
  248 latency. The better the network, the lower the latency can be set.
  249 
  250 
  251 usage with UDP
  252 --------------
  253 
  254 To correctly and completely use the RTP payloaders on the sender and the
  255 receiver you need to write an application. It is not possible to write a full
  256 blown RTP server with a single gst-launch-1.0 line.
  257 
  258 That said, it is possible to do something functional with a few gst-launch
  259 lines. The biggest problem when constructing a correct gst-launch-1.0 line lies on
  260 the receiver end. 
  261 
  262 The receiver needs to know about the type of the RTP data along with a set of
  263 RTP configuration parameters. This information is usually transmitted to the
  264 client using some sort of session description language (SDP) over some reliable
  265 channel (HTTP/RTSP/...).  
  266 
  267 All of the required parameters to connect and use the RTP session on the
  268 server can be found in the caps on the server end. The client receives this
  269 information in some way (caps are converted to and from SDP, as explained above,
  270 for example).
  271 
  272 Some gst-launch-1.0 lines:
  273 
  274   gst-launch-1.0 -v videotestsrc ! videoconvert ! avenc_h263p ! rtph263ppay ! udpsink
  275 
  276    Setting pipeline to PAUSED ...
  277    /pipeline0/videotestsrc0.src: caps = video/x-raw, format=(string)I420,
  278    width=(int)320, height=(int)240, framerate=(fraction)30/1
  279    Pipeline is PREROLLING ...
  280    ....
  281    /pipeline0/udpsink0.sink: caps = application/x-rtp, media=(string)video,
  282    payload=(int)96, clock-rate=(int)90000, encoding-name=(string)H263-1998,
  283    ssrc=(guint)527842345, clock-base=(guint)1150776941, seqnum-base=(guint)30982
  284    ....
  285    Pipeline is PREROLLED ...
  286    Setting pipeline to PLAYING ...
  287    New clock: GstSystemClock
  288 
  289  Write down the caps on the udpsink and set them as the caps of the UDP 
  290  receiver:
  291 
  292   gst-launch-1.0 -v udpsrc caps="application/x-rtp, media=(string)video,
  293   payload=(int)96, clock-rate=(int)90000, encoding-name=(string)H263-1998,
  294   ssrc=(guint)527842345, clock-base=(guint)1150776941, seqnum-base=(guint)30982"
  295   ! rtph263pdepay ! avdec_h263 ! autovideosink
  296 
  297  The receiver now displays an h263 image. Since there is no jitterbuffer in the
  298  pipeline, frames will be displayed at the time when they are received. This can
  299  result in jerky playback in the case of high network jitter or corrupted video
  300  when packets are dropped or reordered.
  301 
  302  Stream a quicktime file with mpeg4 video and AAC audio on port 5000 and port
  303  5002.
  304 
  305   gst-launch-1.0 -v filesrc location=~/data/sincity.mp4 ! qtdemux name=d ! queue ! rtpmp4vpay ! udpsink port=5000
  306                          d. ! queue ! rtpmp4gpay ! udpsink port=5002
  307     ....
  308     /pipeline0/udpsink0.sink: caps = application/x-rtp, media=(string)video,
  309     payload=(int)96, clock-rate=(int)90000, encoding-name=(string)MP4V-ES,
  310     ssrc=(guint)1162703703, clock-base=(guint)816135835, seqnum-base=(guint)9294,
  311     profile-level-id=(string)3, config=(string)000001b003000001b50900000100000001200086c5d4c307d314043c1463000001b25876694430303334
  312     /pipeline0/udpsink1.sink: caps = application/x-rtp, media=(string)audio,
  313     payload=(int)96, clock-rate=(int)44100, encoding-name=(string)MPEG4-GENERIC,
  314     ssrc=(guint)3246149898, clock-base=(guint)4134514058, seqnum-base=(guint)57633,
  315     encoding-params=(string)2, streamtype=(string)5, profile-level-id=(string)1,
  316     mode=(string)aac-hbr, config=(string)1210, sizelength=(string)13,
  317     indexlength=(string)3, indexdeltalength=(string)3
  318     ....
  319 
  320  Again copy the caps on both sinks to the receiver launch line
  321 
  322     gst-launch-1.0
  323      udpsrc port=5000 caps="application/x-rtp, media=(string)video, payload=(int)96,
  324       clock-rate=(int)90000, encoding-name=(string)MP4V-ES, ssrc=(guint)1162703703,
  325       clock-base=(guint)816135835, seqnum-base=(guint)9294, profile-level-id=(string)3,
  326       config=(string)000001b003000001b50900000100000001200086c5d4c307d314043c1463000001b25876694430303334"
  327       ! rtpmp4vdepay ! ffdec_mpeg4 ! autovideosink sync=false
  328      udpsrc port=5002 caps="application/x-rtp, media=(string)audio, payload=(int)96,
  329       clock-rate=(int)44100, encoding-name=(string)MPEG4-GENERIC, ssrc=(guint)3246149898,
  330       clock-base=(guint)4134514058, seqnum-base=(guint)57633, encoding-params=(string)2,
  331       streamtype=(string)5, profile-level-id=(string)1, mode=(string)aac-hbr,
  332       config=(string)1210, sizelength=(string)13, indexlength=(string)3,
  333       indexdeltalength=(string)3" 
  334       ! rtpmp4gdepay ! faad ! alsasink sync=false
  335 
  336  The caps on the udpsinks can be retrieved when the server pipeline prerolled to
  337  PAUSED.
  338 
  339  The above pipeline sets sync=false on the audio and video sink which means that
  340  no synchronisation will be performed in the sinks, they play the data when it
  341  arrives. If you want to enable synchronisation in the sinks it is highly
  342  recommended to use a gstrtpjitterbuffer after the udpsrc elements. 
  343  
  344  Even when sync is enabled, the two different streams will not play synchronised
  345  against each other because the receiver does not have enough information to
  346  perform this task. For this you need to add the rtpbin element in both the
  347  sender and receiver pipeline and use additional sources and sinks to transmit
  348  RTCP packets used for inter-stream synchronisation.
  349 
  350  The caps on the receiver side can be set on the UDP source elements when the
  351  pipeline went to PAUSED. In that state no data is received from the UDP sources
  352  as they are live sources and only produce data in PLAYING.
  353 
  354 
  355 Relevant RFCs
  356 -------------
  357 
  358 3550 RTP: A Transport Protocol for Real-Time Applications. ( 1889 Obsolete )
  359 
  360 2198 RTP Payload for Redundant Audio Data.
  361 3119 A More Loss-Tolerant RTP Payload Format for MP3 Audio.
  362 
  363 2793 RTP Payload for Text Conversation.
  364 
  365 2032 RTP Payload Format for H.261 Video Streams.
  366 2190 RTP Payload Format for H.263 Video Streams.
  367 2250 RTP Payload Format for MPEG1/MPEG2 Video.
  368 2343 RTP Payload Format for Bundled MPEG.
  369 2429 RTP Payload Format for the 1998 Version of ITU-T Rec. H.263 Video
  370 2431 RTP Payload Format for BT.656 Video Encoding.
  371 2435 RTP Payload Format for JPEG-compressed Video.
  372 3016 RTP Payload Format for MPEG-4 Audio/Visual Streams.
  373 3047 RTP Payload Format for ITU-T Recommendation G.722.1.
  374 3189 RTP Payload Format for DV (IEC 61834) Video.
  375 3190 RTP Payload Format for 12-bit DAT Audio and 20- and 24-bit Linear Sampled Audio.
  376 3389 Real-time Transport Protocol (RTP) Payload for Comfort Noise (CN)
  377 2733 An RTP Payload Format for Generic Forward Error Correction.
  378 2833 RTP Payload for DTMF Digits, Telephony Tones and Telephony
  379      Signals.
  380 2862 RTP Payload Format for Real-Time Pointers.
  381 3351 RTP Profile for Audio and Video Conferences with Minimal Control. ( 1890 Obsolete )
  382 3555 MIME Type Registration of RTP Payload Formats.
  383 
  384 2508 Compressing IP/UDP/RTP Headers for Low-Speed Serial Links.
  385 1305 Network Time Protocol (Version 3) Specification, Implementation and Analysis.
  386 3339 Date and Time on the Internet: Timestamps.
  387 2246 The TLS Protocol Version 1.0
  388 3546 Transport Layer Security (TLS) Extensions. ( Updates 2246 )
  389 
  390 do we care?
  391 -----------
  392 
  393 2029 RTP Payload Format of Sun's CellB Video Encoding.
  394 
  395 useful
  396 ------
  397 
  398 http://www.iana.org/assignments/rtp-parameters