"Fossies" - the Fresh Open Source Software Archive

Member "ffmpeg-4.0.1/doc/filter_design.txt" (20 Apr 2018, 11510 Bytes) of package /linux/misc/ffmpeg-4.0.1.tar.xz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. See also the last Fossies "Diffs" side-by-side code changes report for "filter_design.txt": 3.3.4_vs_3.4.

    1 Filter design
    2 =============
    3 
    4 This document explains guidelines that should be observed (or ignored with
    5 good reason) when writing filters for libavfilter.
    6 
    7 In this document, the word “frame” indicates either a video frame or a group
    8 of audio samples, as stored in an AVFrame structure.
    9 
   10 
   11 Format negotiation
   12 ==================
   13 
   14   The query_formats method should set, for each input and each output links,
   15   the list of supported formats.
   16 
   17   For video links, that means pixel format. For audio links, that means
   18   channel layout, sample format (the sample packing is implied by the sample
   19   format) and sample rate.
   20 
   21   The lists are not just lists, they are references to shared objects. When
   22   the negotiation mechanism computes the intersection of the formats
   23   supported at each end of a link, all references to both lists are replaced
   24   with a reference to the intersection. And when a single format is
   25   eventually chosen for a link amongst the remaining list, again, all
   26   references to the list are updated.
   27 
   28   That means that if a filter requires that its input and output have the
   29   same format amongst a supported list, all it has to do is use a reference
   30   to the same list of formats.
   31 
   32   query_formats can leave some formats unset and return AVERROR(EAGAIN) to
   33   cause the negotiation mechanism to try again later. That can be used by
   34   filters with complex requirements to use the format negotiated on one link
   35   to set the formats supported on another.
   36 
   37 
   38 Frame references ownership and permissions
   39 ==========================================
   40 
   41   Principle
   42   ---------
   43 
   44     Audio and video data are voluminous; the frame and frame reference
   45     mechanism is intended to avoid, as much as possible, expensive copies of
   46     that data while still allowing the filters to produce correct results.
   47 
   48     The data is stored in buffers represented by AVFrame structures.
   49     Several references can point to the same frame buffer; the buffer is
   50     automatically deallocated once all corresponding references have been
   51     destroyed.
   52 
   53     The characteristics of the data (resolution, sample rate, etc.) are
   54     stored in the reference; different references for the same buffer can
   55     show different characteristics. In particular, a video reference can
   56     point to only a part of a video buffer.
   57 
   58     A reference is usually obtained as input to the filter_frame method or
   59     requested using the ff_get_video_buffer or ff_get_audio_buffer
   60     functions. A new reference on an existing buffer can be created with
   61     av_frame_ref(). A reference is destroyed using
   62     the av_frame_free() function.
   63 
   64   Reference ownership
   65   -------------------
   66 
   67     At any time, a reference “belongs” to a particular piece of code,
   68     usually a filter. With a few caveats that will be explained below, only
   69     that piece of code is allowed to access it. It is also responsible for
   70     destroying it, although this is sometimes done automatically (see the
   71     section on link reference fields).
   72 
   73     Here are the (fairly obvious) rules for reference ownership:
   74 
   75     * A reference received by the filter_frame method belongs to the
   76       corresponding filter.
   77 
   78     * A reference passed to ff_filter_frame is given away and must no longer
   79       be used.
   80 
   81     * A reference created with av_frame_ref() belongs to the code that
   82       created it.
   83 
   84     * A reference obtained with ff_get_video_buffer or ff_get_audio_buffer
   85       belongs to the code that requested it.
   86 
   87     * A reference given as return value by the get_video_buffer or
   88       get_audio_buffer method is given away and must no longer be used.
   89 
   90   Link reference fields
   91   ---------------------
   92 
   93     The AVFilterLink structure has a few AVFrame fields.
   94 
   95     partial_buf is used by libavfilter internally and must not be accessed
   96     by filters.
   97 
   98     fifo contains frames queued in the filter's input. They belong to the
   99     framework until they are taken by the filter.
  100 
  101   Reference permissions
  102   ---------------------
  103 
  104     Since the same frame data can be shared by several frames, modifying may
  105     have unintended consequences. A frame is considered writable if only one
  106     reference to it exists. The code owning that reference it then allowed
  107     to modify the data.
  108 
  109     A filter can check if a frame is writable by using the
  110     av_frame_is_writable() function.
  111 
  112     A filter can ensure that a frame is writable at some point of the code
  113     by using the ff_inlink_make_frame_writable() function. It will duplicate
  114     the frame if needed.
  115 
  116     A filter can ensure that the frame passed to the filter_frame() callback
  117     is writable by setting the needs_writable flag on the corresponding
  118     input pad. It does not apply to the activate() callback.
  119 
  120 
  121 Frame scheduling
  122 ================
  123 
  124   The purpose of these rules is to ensure that frames flow in the filter
  125   graph without getting stuck and accumulating somewhere.
  126 
  127   Simple filters that output one frame for each input frame should not have
  128   to worry about it.
  129 
  130   There are two design for filters: one using the filter_frame() and
  131   request_frame() callbacks and the other using the activate() callback.
  132 
  133   The design using filter_frame() and request_frame() is legacy, but it is
  134   suitable for filters that have a single input and process one frame at a
  135   time. New filters with several inputs, that treat several frames at a time
  136   or that require a special treatment at EOF should probably use the design
  137   using activate().
  138 
  139   activate
  140   --------
  141 
  142     This method is called when something must be done in a filter; the
  143     definition of that "something" depends on the semantic of the filter.
  144 
  145     The callback must examine the status of the filter's links and proceed
  146     accordingly.
  147 
  148     The status of output links is stored in the frame_wanted_out, status_in
  149     and status_out fields and tested by the ff_outlink_frame_wanted()
  150     function. If this function returns true, then the processing requires a
  151     frame on this link and the filter is expected to make efforts in that
  152     direction.
  153 
  154     The status of input links is stored by the status_in, fifo and
  155     status_out fields; they must not be accessed directly. The fifo field
  156     contains the frames that are queued in the input for processing by the
  157     filter. The status_in and status_out fields contains the queued status
  158     (EOF or error) of the link; status_in is a status change that must be
  159     taken into account after all frames in fifo have been processed;
  160     status_out is the status that have been taken into account, it is final
  161     when it is not 0.
  162 
  163     The typical task of an activate callback is to first check the backward
  164     status of output links, and if relevant forward it to the corresponding
  165     input. Then, if relevant, for each input link: test the availability of
  166     frames in fifo and process them;  if no frame is available, test and
  167     acknowledge a change of status using ff_inlink_acknowledge_status(); and
  168     forward the result (frame or status change) to the corresponding input.
  169     If nothing is possible, test the status of outputs and forward it to the
  170     corresponding input(s). If still not possible, return FFERROR_NOT_READY.
  171 
  172     If the filters stores internally one or a few frame for some input, it
  173     can consider them to be part of the FIFO and delay acknowledging a
  174     status change accordingly.
  175 
  176     Example code:
  177 
  178     ret = ff_outlink_get_status(outlink);
  179     if (ret) {
  180         ff_inlink_set_status(inlink, ret);
  181         return 0;
  182     }
  183     if (priv->next_frame) {
  184         /* use it */
  185         return 0;
  186     }
  187     ret = ff_inlink_consume_frame(inlink, &frame);
  188     if (ret < 0)
  189         return ret;
  190     if (ret) {
  191         /* use it */
  192         return 0;
  193     }
  194     ret = ff_inlink_acknowledge_status(inlink, &status, &pts);
  195     if (ret) {
  196         /* flush */
  197         ff_outlink_set_status(outlink, status, pts);
  198         return 0;
  199     }
  200     if (ff_outlink_frame_wanted(outlink)) {
  201         ff_inlink_request_frame(inlink);
  202         return 0;
  203     }
  204     return FFERROR_NOT_READY;
  205 
  206     The exact code depends on how similar the /* use it */ blocks are and
  207     how related they are to the /* flush */ block, and needs to apply these
  208     operations to the correct inlink or outlink if there are several.
  209 
  210     Macros are available to factor that when no extra processing is needed:
  211 
  212     FF_FILTER_FORWARD_STATUS_BACK(outlink, inlink);
  213     FF_FILTER_FORWARD_STATUS_ALL(outlink, filter);
  214     FF_FILTER_FORWARD_STATUS(inlink, outlink);
  215     FF_FILTER_FORWARD_STATUS_ALL(inlink, filter);
  216     FF_FILTER_FORWARD_WANTED(outlink, inlink);
  217 
  218   filter_frame
  219   ------------
  220 
  221     For filters that do not use the activate() callback, this method is
  222     called when a frame is pushed to the filter's input. It can be called at
  223     any time except in a reentrant way.
  224 
  225     If the input frame is enough to produce output, then the filter should
  226     push the output frames on the output link immediately.
  227 
  228     As an exception to the previous rule, if the input frame is enough to
  229     produce several output frames, then the filter needs output only at
  230     least one per link. The additional frames can be left buffered in the
  231     filter; these buffered frames must be flushed immediately if a new input
  232     produces new output.
  233 
  234     (Example: frame rate-doubling filter: filter_frame must (1) flush the
  235     second copy of the previous frame, if it is still there, (2) push the
  236     first copy of the incoming frame, (3) keep the second copy for later.)
  237 
  238     If the input frame is not enough to produce output, the filter must not
  239     call request_frame to get more. It must just process the frame or queue
  240     it. The task of requesting more frames is left to the filter's
  241     request_frame method or the application.
  242 
  243     If a filter has several inputs, the filter must be ready for frames
  244     arriving randomly on any input. Therefore, any filter with several inputs
  245     will most likely require some kind of queuing mechanism. It is perfectly
  246     acceptable to have a limited queue and to drop frames when the inputs
  247     are too unbalanced.
  248 
  249   request_frame
  250   -------------
  251 
  252     For filters that do not use the activate() callback, this method is
  253     called when a frame is wanted on an output.
  254 
  255     For a source, it should directly call filter_frame on the corresponding
  256     output.
  257 
  258     For a filter, if there are queued frames already ready, one of these
  259     frames should be pushed. If not, the filter should request a frame on
  260     one of its inputs, repeatedly until at least one frame has been pushed.
  261 
  262     Return values:
  263     if request_frame could produce a frame, or at least make progress
  264     towards producing a frame, it should return 0;
  265     if it could not for temporary reasons, it should return AVERROR(EAGAIN);
  266     if it could not because there are no more frames, it should return
  267     AVERROR_EOF.
  268 
  269     The typical implementation of request_frame for a filter with several
  270     inputs will look like that:
  271 
  272         if (frames_queued) {
  273             push_one_frame();
  274             return 0;
  275         }
  276         input = input_where_a_frame_is_most_needed();
  277         ret = ff_request_frame(input);
  278         if (ret == AVERROR_EOF) {
  279             process_eof_on_input();
  280         } else if (ret < 0) {
  281             return ret;
  282         }
  283         return 0;
  284 
  285     Note that, except for filters that can have queued frames and sources,
  286     request_frame does not push frames: it requests them to its input, and
  287     as a reaction, the filter_frame method possibly will be called and do
  288     the work.