GStreamer 中的 GstVideoDecoder 基类旨在为实现视频解码器提供一个框架。它定义了一套规则和规范，用于指导基类与其派生子类（具体的视频解码器）之间如何交互与协作。
/**
* SECTION:gstvideodecoder
* @title: GstVideoDecoder
* @short_description: Base class for video decoders
*
* This base class is for video decoders turning encoded data into raw video
* frames.
*
* The GstVideoDecoder base class and derived subclasses should cooperate as
* follows:
*
* ## Configuration
*
* * Initially, GstVideoDecoder calls @start when the decoder element
* is activated, which allows the subclass to perform any global setup.
*
* * GstVideoDecoder calls @set_format to inform the subclass of caps
* describing input video data that it is about to receive, including
* possibly configuration data.
* While unlikely, it might be called more than once, if changing input
* parameters require reconfiguration.
*
* * Incoming data buffers are processed as needed, described in Data
* Processing below.
*
* * GstVideoDecoder calls @stop at end of all processing.
*
* ## Data processing
*
* * The base class gathers input data, and optionally allows subclass
* to parse this into subsequently manageable chunks, typically
* corresponding to and referred to as 'frames'.
*
* * Each input frame is provided in turn to the subclass' @handle_frame
* callback.
* * When the subclass enables the subframe mode with `gst_video_decoder_set_subframe_mode`,
* the base class will provide to the subclass the same input frame with
* different input buffers to the subclass @handle_frame
* callback. During this call, the subclass needs to take
* ownership of the input_buffer as @GstVideoCodecFrame.input_buffer
* will have been changed before the next subframe buffer is received.
* The subclass will call `gst_video_decoder_have_last_subframe`
* when a new input frame can be created by the base class.
* Every subframe will share the same @GstVideoCodecFrame.output_buffer
* to write the decoding result. The subclass is responsible to protect
* its access.
*
* * If codec processing results in decoded data, the subclass should call
* @gst_video_decoder_finish_frame to have decoded data pushed
* downstream. In subframe mode
* the subclass should call @gst_video_decoder_finish_subframe until the
* last subframe where it should call @gst_video_decoder_finish_frame.
* The subclass can detect the last subframe using GST_VIDEO_BUFFER_FLAG_MARKER
* on buffers or using its own logic to collect the subframes.
* In case of decoding failure, the subclass must call
* @gst_video_decoder_drop_frame or @gst_video_decoder_drop_subframe,
* to allow the base class to do timestamp and offset tracking, and possibly
* to requeue the frame for a later attempt in the case of reverse playback.
*
* ## Shutdown phase
*
* * The GstVideoDecoder class calls @stop to inform the subclass that data
* parsing will be stopped.
*
* ## Additional Notes
*
* * Seeking/Flushing
*
* * When the pipeline is seeked or otherwise flushed, the subclass is
* informed via a call to its @reset callback, with the hard parameter
* set to true. This indicates the subclass should drop any internal data
* queues and timestamps and prepare for a fresh set of buffers to arrive
* for parsing and decoding.
*
* * End Of Stream
*
* * At end-of-stream, the subclass @parse function may be called some final
* times with the at_eos parameter set to true, indicating that the element
* should not expect any more data to be arriving, and it should parse and
* remaining frames and call gst_video_decoder_have_frame() if possible.
*
* The subclass is responsible for providing pad template caps for
* source and sink pads. The pads need to be named "sink" and "src". It also
* needs to provide information about the output caps, when they are known.
* This may be when the base class calls the subclass' @set_format function,
* though it might be during decoding, before calling
* @gst_video_decoder_finish_frame. This is done via
* @gst_video_decoder_set_output_state
*
* The subclass is also responsible for providing (presentation) timestamps
* (likely based on corresponding input ones). If that is not applicable
* or possible, the base class provides limited framerate based interpolation.
*
* Similarly, the base class provides some limited (legacy) seeking support
* if specifically requested by the subclass, as full-fledged support
* should rather be left to upstream demuxer, parser or alike. This simple
* approach caters for seeking and duration reporting using estimated input
* bitrates. To enable it, a subclass should call
* @gst_video_decoder_set_estimate_rate to enable handling of incoming
* byte-streams.
*
* The base class provides some support for reverse playback, in particular
* in case incoming data is not packetized or upstream does not provide
* fragments on keyframe boundaries. However, the subclass should then be
* prepared for the parsing and frame processing stage to occur separately
* (in normal forward processing, the latter immediately follows the former),
* The subclass also needs to ensure the parsing stage properly marks
* keyframes, unless it knows the upstream elements will do so properly for
* incoming data.
*
* The bare minimum that a functional subclass needs to implement is:
*
* * Provide pad templates
* * Inform the base class of output caps via
* @gst_video_decoder_set_output_state
*
* * Parse input data, if it is not considered packetized from upstream
* Data will be provided to @parse which should invoke
* @gst_video_decoder_add_to_frame and @gst_video_decoder_have_frame to
* separate the data belonging to each video frame.
*
* * Accept data in @handle_frame and provide decoded results to
* @gst_video_decoder_finish_frame, or call @gst_video_decoder_drop_frame.
*/GstVideoDecoderClass定义的虚函数：
cpp
/**
* GstVideoDecoderClass:
* @open: Optional.
* Called when the element changes to GST_STATE_READY.
* Allows opening external resources.
* @close: Optional.
* Called when the element changes to GST_STATE_NULL.
* Allows closing external resources.
* @start: Optional.
* Called when the element starts processing.
* Allows opening external resources.
* @stop: Optional.
* Called when the element stops processing.
* Allows closing external resources.
* @set_format: Notifies subclass of incoming data format (caps).
* @parse: Required for non-packetized input.
* Allows chopping incoming data into manageable units (frames)
* for subsequent decoding.
* @reset: Optional.
* Allows subclass (decoder) to perform post-seek semantics reset.
* Deprecated.
* @handle_frame: Provides input data frame to subclass. In subframe mode, the subclass needs
* to take ownership of @GstVideoCodecFrame.input_buffer as it will be modified
* by the base class on the next subframe buffer receiving.
* @finish: Optional.
* Called to request subclass to dispatch any pending remaining
* data at EOS. Sub-classes can refuse to decode new data after.
* @drain: Optional.
* Called to request subclass to decode any data it can at this
* point, but that more data may arrive after. (e.g. at segment end).
* Sub-classes should be prepared to handle new data afterward,
* or seamless segment processing will break. Since: 1.6
* @sink_event: Optional.
* Event handler on the sink pad. This function should return
* TRUE if the event was handled and should be discarded
* (i.e. not unref'ed).
* Subclasses should chain up to the parent implementation to
* invoke the default handler.
* @src_event: Optional.
* Event handler on the source pad. This function should return
* TRUE if the event was handled and should be discarded
* (i.e. not unref'ed).
* Subclasses should chain up to the parent implementation to
* invoke the default handler.
* @negotiate: Optional.
* Negotiate with downstream and configure buffer pools, etc.
* Subclasses should chain up to the parent implementation to
* invoke the default handler.
* @decide_allocation: Optional.
* Setup the allocation parameters for allocating output
* buffers. The passed in query contains the result of the
* downstream allocation query.
* Subclasses should chain up to the parent implementation to
* invoke the default handler.
* @propose_allocation: Optional.
* Propose buffer allocation parameters for upstream elements.
* Subclasses should chain up to the parent implementation to
* invoke the default handler.
* @flush: Optional.
* Flush all remaining data from the decoder without
* pushing it downstream. Since: 1.2
* @sink_query: Optional.
* Query handler on the sink pad. This function should
* return TRUE if the query could be performed. Subclasses
* should chain up to the parent implementation to invoke the
* default handler. Since: 1.4
* @src_query: Optional.
* Query handler on the source pad. This function should
* return TRUE if the query could be performed. Subclasses
* should chain up to the parent implementation to invoke the
* default handler. Since: 1.4
* @getcaps: Optional.
* Allows for a custom sink getcaps implementation.
* If not implemented, default returns
* gst_video_decoder_proxy_getcaps
* applied to sink template caps.
* @transform_meta: Optional. Transform the metadata on the input buffer to the
* output buffer. By default this method is copies all meta without
* tags and meta with only the "video" tag. subclasses can
* implement this method and return %TRUE if the metadata is to be
* copied. Since: 1.6
*
* Subclasses can override any of the available virtual methods or not, as
* needed. At minimum @handle_frame needs to be overridden, and @set_format
* and likely as well. If non-packetized input is supported or expected,
* @parse needs to be overridden as well.
*/如注释所说子类至少需要重写handle_frame和set_format，如果non-packetized要重写parse方法。
本文以jpegdec插件为例，重写的关键函数：
vdec_class->start = gst_jpeg_dec_start;
vdec_class->stop = gst_jpeg_dec_stop;
vdec_class->flush = gst_jpeg_dec_flush;
vdec_class->parse = gst_jpeg_dec_parse;
vdec_class->set_format = gst_jpeg_dec_set_format;
vdec_class->handle_frame = gst_jpeg_dec_handle_frame;
vdec_class->decide_allocation = gst_jpeg_dec_decide_allocation;
vdec_class->sink_event = gst_jpeg_dec_sink_event;: Notifies subclass of incoming data format (caps)
set_format调用在gst_video_decoder_sink_event_default->gst_video_decoder_setcaps
cpp
case GST_EVENT_CAPS:
{
GstCaps *caps;
gst_event_parse_caps (event, &caps);
ret = gst_video_decoder_setcaps (decoder, caps);
gst_event_unref (event);
event = NULL;
break;
}
cpp
static gboolean
gst_video_decoder_setcaps (GstVideoDecoder * decoder, GstCaps * caps)
{
GstVideoDecoderClass *decoder_class;
GstVideoCodecState *state;
gboolean ret = TRUE;
decoder_class = GST_VIDEO_DECODER_GET_CLASS (decoder);
GST_DEBUG_OBJECT (decoder, "setcaps %" GST_PTR_FORMAT, caps);
GST_VIDEO_DECODER_STREAM_LOCK (decoder);
if (decoder->priv->input_state) {
GST_DEBUG_OBJECT (decoder,
"Checking if caps changed old %" GST_PTR_FORMAT " new %" GST_PTR_FORMAT,
decoder->priv->input_state->caps, caps);
if (gst_caps_is_equal (decoder->priv->input_state->caps, caps))
goto caps_not_changed;
}
state = _new_input_state (caps);
if (G_UNLIKELY (state == NULL))
goto parse_fail;
if (decoder_class->set_format)
ret = decoder_class->set_format (decoder, state);
if (!ret)
goto refused_format;
if (decoder->priv->input_state)
gst_video_codec_state_unref (decoder->priv->input_state);
decoder->priv->input_state = state;
GST_VIDEO_DECODER_STREAM_UNLOCK (decoder);
return ret;
}
cpp
/**
* GstVideoCodecState:
* @info: The #GstVideoInfo describing the stream
* @caps: The #GstCaps used in the caps negotiation of the pad.
* @codec_data: a #GstBuffer corresponding to the
* 'codec_data' field of a stream, or NULL.
* @allocation_caps: The #GstCaps for allocation query and pool
* negotiation. Since: 1.10
* @mastering_display_info: Mastering display color volume information
* (HDR metadata) for the stream. Since: 1.20
* @content_light_level: Content light level information for the stream.
* Since: 1.20
*
* Structure representing the state of an incoming or outgoing video
* stream for encoders and decoders.
*
* Decoders and encoders will receive such a state through their
* respective @set_format vmethods.
*
* Decoders and encoders can set the downstream state, by using the
* gst_video_decoder_set_output_state() or
* gst_video_encoder_set_output_state() methods.
*/
/**
* GstVideoCodecState.mastering_display_info:
*
* Mastering display color volume information (HDR metadata) for the stream.
*
* Since: 1.20
*/
/**
* GstVideoCodecState.content_light_level:
*
* Content light level information for the stream.
*
* Since: 1.20
*/
struct _GstVideoCodecState
{
/*< private >*/
gint ref_count;
/*< public >*/
GstVideoInfo info;
GstCaps *caps;
GstBuffer *codec_data;
GstCaps *allocation_caps;
GstVideoMasteringDisplayInfo *mastering_display_info;
GstVideoContentLightLevel *content_light_level;
/*< private >*/
gpointer padding[GST_PADDING_LARGE - 3];
};上游插件通过GST_EVENT_CAPS事件设置caps，然后保存在了GstVideoCodecState中。
继续看gst_video_decoder_change_state
cpp
switch (transition) {
case GST_STATE_CHANGE_NULL_TO_READY:
/* open device/library if needed */
if (decoder_class->open && !decoder_class->open (decoder))
goto open_failed;
break;
case GST_STATE_CHANGE_READY_TO_PAUSED:
GST_VIDEO_DECODER_STREAM_LOCK (decoder);
gst_video_decoder_reset (decoder, TRUE, TRUE);
GST_VIDEO_DECODER_STREAM_UNLOCK (decoder);
/* Initialize device/library if needed */
if (decoder_class->start && !decoder_class->start (decoder))
goto start_failed;
break;
default:
break;
}会依次调用子类的decoder_class->open和decoder_class->start。jpegdec没有实现open
cpp
static gboolean
gst_jpeg_dec_start (GstVideoDecoder * bdec)
{
GstJpegDec *dec = (GstJpegDec *) bdec;
#ifdef JCS_EXTENSIONS
dec->format_convert = FALSE;
#endif
dec->saw_header = FALSE;
dec->parse_entropy_len = 0;
dec->parse_resync = FALSE;
gst_video_decoder_set_packetized (bdec, FALSE);
return TRUE;
}上游插件push数据调用chain函数
cpp
static GstFlowReturn
gst_video_decoder_chain (GstPad * pad, GstObject * parent, GstBuffer * buf)
{
GstVideoDecoder *decoder;
GstFlowReturn ret = GST_FLOW_OK;
decoder = GST_VIDEO_DECODER (parent);
if (G_UNLIKELY (!decoder->priv->input_state && decoder->priv->needs_format))
goto not_negotiated;
GST_LOG_OBJECT (decoder,
"chain PTS %" GST_TIME_FORMAT ", DTS %" GST_TIME_FORMAT " duration %"
GST_TIME_FORMAT " size %" G_GSIZE_FORMAT " flags %x",
GST_TIME_ARGS (GST_BUFFER_PTS (buf)),
GST_TIME_ARGS (GST_BUFFER_DTS (buf)),
GST_TIME_ARGS (GST_BUFFER_DURATION (buf)),
gst_buffer_get_size (buf), GST_BUFFER_FLAGS (buf));
GST_VIDEO_DECODER_STREAM_LOCK (decoder);
/* NOTE:
* requiring the pad to be negotiated makes it impossible to use
* oggdemux or filesrc ! decoder */
if (decoder->input_segment.format == GST_FORMAT_UNDEFINED) {
GstEvent *event;
GstSegment *segment = &decoder->input_segment;
GST_WARNING_OBJECT (decoder,
"Received buffer without a new-segment. "
"Assuming timestamps start from 0.");
gst_segment_init (segment, GST_FORMAT_TIME);
event = gst_event_new_segment (segment);
decoder->priv->current_frame_events =
g_list_prepend (decoder->priv->current_frame_events, event);
}
decoder->priv->had_input_data = TRUE;
if (decoder->input_segment.rate > 0.0)
ret = gst_video_decoder_chain_forward (decoder, buf, FALSE);
else
ret = gst_video_decoder_chain_reverse (decoder, buf);
GST_VIDEO_DECODER_STREAM_UNLOCK (decoder);
return ret;
/* ERRORS */
not_negotiated:
{
GST_ELEMENT_ERROR (decoder, CORE, NEGOTIATION, (NULL),
("decoder not initialized"));
gst_buffer_unref (buf);
return GST_FLOW_NOT_NEGOTIATED;
}
}chain函数中根据decoder->input_segment.rate 调用
cpp
if (decoder->input_segment.rate > 0.0)
ret = gst_video_decoder_chain_forward (decoder, buf, FALSE);
else
ret = gst_video_decoder_chain_reverse (decoder, buf);重点看下gst_video_decoder_chain_forward
cpp
static GstFlowReturn
gst_video_decoder_chain_forward (GstVideoDecoder * decoder,
GstBuffer * buf, gboolean at_eos)
{
GstVideoDecoderPrivate *priv;
GstVideoDecoderClass *klass;
GstFlowReturn ret = GST_FLOW_OK;
klass = GST_VIDEO_DECODER_GET_CLASS (decoder);
priv = decoder->priv;
g_return_val_if_fail (priv->packetized || klass->parse, GST_FLOW_ERROR);
/* Draining on DISCONT is handled in chain_reverse() for reverse playback,
* and this function would only be called to get everything collected GOP
* by GOP in the parse_gather list */
if (decoder->input_segment.rate > 0.0 && GST_BUFFER_IS_DISCONT (buf)
&& (decoder->input_segment.flags & GST_SEEK_FLAG_TRICKMODE_KEY_UNITS))
ret = gst_video_decoder_drain_out (decoder, FALSE);
if (priv->current_frame == NULL)
priv->current_frame = gst_video_decoder_new_frame (decoder);
if (!priv->packetized)
gst_video_decoder_add_buffer_info (decoder, buf);
priv->input_offset += gst_buffer_get_size (buf);
if (priv->packetized) {
GstVideoCodecFrame *frame;
gboolean was_keyframe = FALSE;
frame = priv->current_frame;
frame->abidata.ABI.num_subframes++;
if (gst_video_decoder_get_subframe_mode (decoder)) {
/* End the frame if the marker flag is set */
if (!GST_BUFFER_FLAG_IS_SET (buf, GST_VIDEO_BUFFER_FLAG_MARKER)
&& (decoder->input_segment.rate > 0.0))
priv->current_frame = gst_video_codec_frame_ref (frame);
else
priv->current_frame = NULL;
} else {
priv->current_frame = frame;
}
if (!GST_BUFFER_FLAG_IS_SET (buf, GST_BUFFER_FLAG_DELTA_UNIT)) {
was_keyframe = TRUE;
GST_DEBUG_OBJECT (decoder, "Marking current_frame as sync point");
GST_VIDEO_CODEC_FRAME_SET_SYNC_POINT (frame);
}
if (frame->input_buffer) {
gst_video_decoder_copy_metas (decoder, frame, frame->input_buffer, buf);
gst_buffer_unref (frame->input_buffer);
}
frame->input_buffer = buf;
if (decoder->input_segment.rate < 0.0) {
priv->parse_gather = g_list_prepend (priv->parse_gather, frame);
priv->current_frame = NULL;
} else {
ret = gst_video_decoder_decode_frame (decoder, frame);
if (!gst_video_decoder_get_subframe_mode (decoder))
priv->current_frame = NULL;
}
/* If in trick mode and it was a keyframe, drain decoder to avoid extra
* latency. Only do this for forwards playback as reverse playback handles
* draining on keyframes in flush_parse(), and would otherwise call back
* from drain_out() to here causing an infinite loop.
* Also this function is only called for reverse playback to gather frames
* GOP by GOP, and does not do any actual decoding. That would be done by
* flush_decode() */
if (ret == GST_FLOW_OK && was_keyframe && decoder->input_segment.rate > 0.0
&& (decoder->input_segment.flags & GST_SEEK_FLAG_TRICKMODE_KEY_UNITS))
ret = gst_video_decoder_drain_out (decoder, FALSE);
} else {
gst_adapter_push (priv->input_adapter, buf);
ret = gst_video_decoder_parse_available (decoder, at_eos, TRUE);
}
if (ret == GST_VIDEO_DECODER_FLOW_NEED_DATA)
return GST_FLOW_OK;
return ret;
}关键流程：
（1）如果priv->packetized == false,会走直接走gst_video_decoder_decode_frame，然后调用子类的decoder_class->handle_frame函数
（2） 如果priv->packetized == true,会走gst_adapter_push数据推送到input_adapter，并且调用
gst_video_decoder_parse_available，继而调用子类的decoder_class->parse函数进行数据解析。
jpegdec的gst_jpeg_dec_handle_frame，完成了数据的解码处理。
cpp
static GstFlowReturn
gst_jpeg_dec_handle_frame (GstVideoDecoder * bdec, GstVideoCodecFrame * frame)
{
GstFlowReturn ret = GST_FLOW_OK;
GstJpegDec *dec = (GstJpegDec *) bdec;
GstVideoFrame vframe;
gint num_fields; /* number of fields (1 or 2) */
gint output_height; /* height of output image (one or two fields) */
gint height; /* height of current frame (whole image or a field) */
gint width;
guint code;
gboolean need_unmap = TRUE;
GstVideoCodecState *state = NULL;
gboolean release_frame = TRUE;
gboolean has_eoi;
guint8 *data;
gsize nbytes;
if (!gst_buffer_map (frame->input_buffer, &dec->current_frame_map,
GST_MAP_READ))
goto map_failed;
data = dec->current_frame_map.data;
nbytes = dec->current_frame_map.size;
if (nbytes < 2)
goto need_more_data;
has_eoi = ((data[nbytes - 2] == 0xff) && (data[nbytes - 1] == 0xd9));
/* some cameras fail to send an end-of-image marker (EOI),
* add it if that is the case. */
if (!has_eoi) {
GstMapInfo map;
GstBuffer *eoibuf = gst_buffer_new_and_alloc (2);
/* unmap, will add EOI and remap at the end */
gst_buffer_unmap (frame->input_buffer, &dec->current_frame_map);
gst_buffer_map (eoibuf, &map, GST_MAP_WRITE);
map.data[0] = 0xff;
map.data[1] = 0xd9;
gst_buffer_unmap (eoibuf, &map);
/* append to input buffer, and remap */
frame->input_buffer = gst_buffer_append (frame->input_buffer, eoibuf);
gst_buffer_map (frame->input_buffer, &dec->current_frame_map, GST_MAP_READ);
GST_DEBUG ("fixup EOI marker added");
}
dec->current_frame = frame;
dec->cinfo.src->next_input_byte = dec->current_frame_map.data;
dec->cinfo.src->bytes_in_buffer = dec->current_frame_map.size;
if (setjmp (dec->jerr.setjmp_buffer)) {
code = dec->jerr.pub.msg_code;
if (code == JERR_INPUT_EOF) {
GST_DEBUG ("jpeg input EOF error, we probably need more data");
goto need_more_data;
}
goto decode_error;
}
/* read header and check values */
ret = gst_jpeg_dec_prepare_decode (dec);
if (G_UNLIKELY (ret == GST_FLOW_ERROR))
goto done;
width = dec->cinfo.output_width;
height = dec->cinfo.output_height;
/* is it interlaced MJPEG? (we really don't want to scan the jpeg data
* to see if there are two SOF markers in the packet to detect this) */
if (gst_video_decoder_get_packetized (bdec) &&
dec->input_state &&
dec->input_state->info.height > height &&
dec->input_state->info.height <= (height * 2)
&& dec->input_state->info.width == width) {
GST_LOG_OBJECT (dec,
"looks like an interlaced image: "
"input width/height of %dx%d with JPEG frame width/height of %dx%d",
dec->input_state->info.width, dec->input_state->info.height, width,
height);
output_height = dec->input_state->info.height;
height = dec->input_state->info.height / 2;
num_fields = 2;
GST_LOG_OBJECT (dec, "field height=%d", height);
} else {
output_height = height;
num_fields = 1;
}
gst_jpeg_dec_negotiate (dec, width, output_height,
dec->cinfo.jpeg_color_space, num_fields == 2);
state = gst_video_decoder_get_output_state (bdec);
ret = gst_video_decoder_allocate_output_frame (bdec, frame);
if (G_UNLIKELY (ret != GST_FLOW_OK))
goto alloc_failed;
if (!gst_video_frame_map (&vframe, &state->info, frame->output_buffer,
GST_MAP_READWRITE))
goto alloc_failed;
if (setjmp (dec->jerr.setjmp_buffer)) {
code = dec->jerr.pub.msg_code;
gst_video_frame_unmap (&vframe);
goto decode_error;
}
GST_LOG_OBJECT (dec, "width %d, height %d, fields %d", width, output_height,
num_fields);
ret = gst_jpeg_dec_decode (dec, &vframe, width, height, 1, num_fields);
if (G_UNLIKELY (ret != GST_FLOW_OK)) {
gst_video_frame_unmap (&vframe);
goto decode_failed;
}
if (setjmp (dec->jerr.setjmp_buffer)) {
code = dec->jerr.pub.msg_code;
gst_video_frame_unmap (&vframe);
goto decode_error;
}
/* decode second field if there is one */
if (num_fields == 2) {
GstVideoFormat field2_format;
/* Checked above before setting num_fields to 2 */
g_assert (dec->input_state != NULL);
/* skip any chunk or padding bytes before the next SOI marker; both fields
* are in one single buffer here, so direct access should be fine here */
while (dec->jsrc.pub.bytes_in_buffer > 2 &&
GST_READ_UINT16_BE (dec->jsrc.pub.next_input_byte) != 0xffd8) {
--dec->jsrc.pub.bytes_in_buffer;
++dec->jsrc.pub.next_input_byte;
}
if (gst_jpeg_dec_prepare_decode (dec) != GST_FLOW_OK) {
GST_WARNING_OBJECT (dec, "problem reading jpeg header of 2nd field");
/* FIXME: post a warning message here? */
gst_video_frame_unmap (&vframe);
goto decode_failed;
}
/* check if format has changed for the second field */
#ifdef JCS_EXTENSIONS
if (dec->format_convert) {
field2_format = dec->format;
} else
#endif
{
switch (dec->cinfo.jpeg_color_space) {
case JCS_RGB:
field2_format = GST_VIDEO_FORMAT_RGB;
break;
case JCS_GRAYSCALE:
field2_format = GST_VIDEO_FORMAT_GRAY8;
break;
default:
field2_format = GST_VIDEO_FORMAT_I420;
break;
}
}
GST_LOG_OBJECT (dec,
"got for second field of interlaced image: "
"input width/height of %dx%d with JPEG frame width/height of %dx%d",
dec->input_state->info.width, dec->input_state->info.height,
dec->cinfo.output_width, dec->cinfo.output_height);
if (dec->cinfo.output_width != GST_VIDEO_INFO_WIDTH (&state->info) ||
GST_VIDEO_INFO_HEIGHT (&state->info) <= dec->cinfo.output_height ||
GST_VIDEO_INFO_HEIGHT (&state->info) > (dec->cinfo.output_height * 2) ||
field2_format != GST_VIDEO_INFO_FORMAT (&state->info)) {
GST_WARNING_OBJECT (dec, "second field has different format than first");
gst_video_frame_unmap (&vframe);
goto decode_failed;
}
ret = gst_jpeg_dec_decode (dec, &vframe, width, height, 2, 2);
if (G_UNLIKELY (ret != GST_FLOW_OK)) {
gst_video_frame_unmap (&vframe);
goto decode_failed;
}
}
gst_video_frame_unmap (&vframe);
gst_buffer_unmap (frame->input_buffer, &dec->current_frame_map);
ret = gst_video_decoder_finish_frame (bdec, frame);
release_frame = FALSE;
need_unmap = FALSE;
done:
exit:
if (need_unmap)
gst_buffer_unmap (frame->input_buffer, &dec->current_frame_map);
if (release_frame)
gst_video_decoder_release_frame (bdec, frame);
if (state)
gst_video_codec_state_unref (state);
return ret;
/* special cases */
need_more_data:
{
GST_LOG_OBJECT (dec, "we need more data");
ret = GST_FLOW_OK;
goto exit;
}
/* ERRORS */
map_failed:
{
GST_ELEMENT_ERROR (dec, RESOURCE, READ, (_("Failed to read memory")),
("gst_buffer_map() failed for READ access"));
ret = GST_FLOW_ERROR;
goto exit;
}
decode_error:
{
gchar err_msg[JMSG_LENGTH_MAX];
dec->jerr.pub.format_message ((j_common_ptr) (&dec->cinfo), err_msg);
GST_VIDEO_DECODER_ERROR (dec, 1, STREAM, DECODE,
(_("Failed to decode JPEG image")), ("Decode error #%u: %s", code,
err_msg), ret);
gst_buffer_unmap (frame->input_buffer, &dec->current_frame_map);
gst_video_decoder_drop_frame (bdec, frame);
release_frame = FALSE;
need_unmap = FALSE;
jpeg_abort_decompress (&dec->cinfo);
goto done;
}
decode_failed:
{
/* already posted an error message */
goto done;
}
alloc_failed:
{
const gchar *reason;
reason = gst_flow_get_name (ret);
GST_DEBUG_OBJECT (dec, "failed to alloc buffer, reason %s", reason);
/* Reset for next time */
jpeg_abort_decompress (&dec->cinfo);
if (ret != GST_FLOW_EOS && ret != GST_FLOW_FLUSHING &&
ret != GST_FLOW_NOT_LINKED) {
GST_VIDEO_DECODER_ERROR (dec, 1, STREAM, DECODE,
(_("Failed to decode JPEG image")),
("Buffer allocation failed, reason: %s", reason), ret);
jpeg_abort_decompress (&dec->cinfo);
}
goto exit;
}
}而priv->packetized流会走流程2也就是parse，parse会不断解析数据，直到解析到完整的一帧数据后，然后会走到have_full_frame流程。
cpp
static GstFlowReturn
gst_jpeg_dec_parse (GstVideoDecoder * bdec, GstVideoCodecFrame * frame,
GstAdapter * adapter, gboolean at_eos)
{
guint size;
gint toadd = 0;
gboolean resync;
gint offset = 0, noffset;
GstJpegDec *dec = (GstJpegDec *) bdec;
GST_VIDEO_CODEC_FRAME_SET_SYNC_POINT (frame);
/* FIXME : The overhead of using scan_uint32 is massive */
size = gst_adapter_available (adapter);
GST_DEBUG ("Parsing jpeg image data (%u bytes)", size);
if (at_eos) {
GST_DEBUG ("Flushing all data out");
toadd = size;
/* If we have leftover data, throw it away */
if (!dec->saw_header)
goto drop_frame;
goto have_full_frame;
}
if (size < 8)
goto need_more_data;
if (!dec->saw_header) {
gint ret;
/* we expect at least 4 bytes, first of which start marker */
ret =
gst_adapter_masked_scan_uint32 (adapter, 0xffff0000, 0xffd80000, 0,
size - 4);
GST_DEBUG ("ret:%d", ret);
if (ret < 0)
goto need_more_data;
if (ret) {
gst_adapter_flush (adapter, ret);
size -= ret;
}
dec->saw_header = TRUE;
}
while (1) {
guint frame_len;
guint32 value;
GST_DEBUG ("offset:%d, size:%d", offset, size);
noffset =
gst_adapter_masked_scan_uint32_peek (adapter, 0x0000ff00, 0x0000ff00,
offset, size - offset, &value);
/* lost sync if 0xff marker not where expected */
if ((resync = (noffset != offset))) {
GST_DEBUG ("Lost sync at 0x%08x, resyncing", offset + 2);
}
/* may have marker, but could have been resyncng */
resync = resync || dec->parse_resync;
/* Skip over extra 0xff */
while ((noffset >= 0) && ((value & 0xff) == 0xff)) {
noffset++;
noffset =
gst_adapter_masked_scan_uint32_peek (adapter, 0x0000ff00, 0x0000ff00,
noffset, size - noffset, &value);
}
/* enough bytes left for marker? (we need 0xNN after the 0xff) */
if (noffset < 0) {
GST_DEBUG ("at end of input and no EOI marker found, need more data");
goto need_more_data;
}
/* now lock on the marker we found */
offset = noffset;
value = value & 0xff;
if (value == 0xd9) {
GST_DEBUG ("0x%08x: EOI marker", offset + 2);
/* clear parse state */
dec->saw_header = FALSE;
dec->parse_resync = FALSE;
toadd = offset + 4;
goto have_full_frame;
}
if (value == 0xd8) {
GST_DEBUG ("0x%08x: SOI marker before EOI marker", offset + 2);
/* clear parse state */
dec->saw_header = FALSE;
dec->parse_resync = FALSE;
toadd = offset;
goto have_full_frame;
}
if (value >= 0xd0 && value <= 0xd7)
frame_len = 0;
else {
/* peek tag and subsequent length */
if (offset + 2 + 4 > size)
goto need_more_data;
else
gst_adapter_masked_scan_uint32_peek (adapter, 0x0, 0x0, offset + 2, 4,
&frame_len);
frame_len = frame_len & 0xffff;
}
GST_DEBUG ("0x%08x: tag %02x, frame_len=%u", offset + 2, value, frame_len);
/* the frame length includes the 2 bytes for the length; here we want at
* least 2 more bytes at the end for an end marker */
if (offset + 2 + 2 + frame_len + 2 > size) {
goto need_more_data;
}
if (gst_jpeg_dec_parse_tag_has_entropy_segment (value)) {
guint eseglen = dec->parse_entropy_len;
GST_DEBUG ("0x%08x: finding entropy segment length (eseglen:%d)",
offset + 2, eseglen);
if (size < offset + 2 + frame_len + eseglen)
goto need_more_data;
noffset = offset + 2 + frame_len + dec->parse_entropy_len;
while (1) {
GST_DEBUG ("noffset:%d, size:%d, size - noffset:%d",
noffset, size, size - noffset);
noffset = gst_adapter_masked_scan_uint32_peek (adapter, 0x0000ff00,
0x0000ff00, noffset, size - noffset, &value);
if (noffset < 0) {
/* need more data */
dec->parse_entropy_len = size - offset - 4 - frame_len - 2;
goto need_more_data;
}
if ((value & 0xff) != 0x00) {
eseglen = noffset - offset - frame_len - 2;
break;
}
noffset++;
}
dec->parse_entropy_len = 0;
frame_len += eseglen;
GST_DEBUG ("entropy segment length=%u => frame_len=%u", eseglen,
frame_len);
}
if (resync) {
/* check if we will still be in sync if we interpret
* this as a sync point and skip this frame */
noffset = offset + frame_len + 2;
noffset = gst_adapter_masked_scan_uint32 (adapter, 0x0000ff00, 0x0000ff00,
noffset, 4);
if (noffset < 0) {
/* ignore and continue resyncing until we hit the end
* of our data or find a sync point that looks okay */
offset++;
continue;
}
GST_DEBUG ("found sync at 0x%x", offset + 2);
}
/* Add current data to output buffer */
toadd += frame_len + 2;
offset += frame_len + 2;
}
need_more_data:
if (toadd)
gst_video_decoder_add_to_frame (bdec, toadd);
return GST_VIDEO_DECODER_FLOW_NEED_DATA;
have_full_frame:
if (toadd)
gst_video_decoder_add_to_frame (bdec, toadd);
GST_VIDEO_CODEC_FRAME_SET_SYNC_POINT (frame);
return gst_video_decoder_have_frame (bdec);
drop_frame:
gst_adapter_flush (adapter, size);
return GST_FLOW_OK;
}其中gst_video_decoder_add_to_frame会将数据push到priv->output_adapter中。还会调用
gst_video_decoder_have_frame，其中关键流程会调用gst_video_decoder_decode_frame继而调用到decoder_class->handle_frame，完成解码。
cpp
/* In reverse playback, just capture and queue frames for later processing */
if (decoder->input_segment.rate < 0.0) {
priv->parse_gather =
g_list_prepend (priv->parse_gather, priv->current_frame);
priv->current_frame = NULL;
} else {
GstVideoCodecFrame *frame = priv->current_frame;
frame->abidata.ABI.num_subframes++;
/* In subframe mode, we keep a ref for ourselves
* as this frame will be kept during the data collection
* in parsed mode. The frame reference will be released by
* finish_(sub)frame or drop_(sub)frame.*/
if (gst_video_decoder_get_subframe_mode (decoder))
gst_video_codec_frame_ref (priv->current_frame);
else
priv->current_frame = NULL;
/* Decode the frame, which gives away our ref */
ret = gst_video_decoder_decode_frame (decoder, frame);
}packetized流，也就是上游送下的数据，并不是可以直接解码的原始数据流，所以必须通过parse从数据中解析出来可解码的原始数据。
/**
* gst_video_decoder_set_packetized:
* @decoder: a #GstVideoDecoder
* @packetized: whether the input data should be considered as packetized.
*
* Allows baseclass to consider input data as packetized or not. If the
* input is packetized, then the @parse method will not be called.
*/
void
gst_video_decoder_set_packetized (GstVideoDecoder * decoder,
gboolean packetized)
{
decoder->priv->packetized = packetized;
}总结：
GstVideoDecoder流程不是很复杂，主要通过set_format设置caps，通过handle_frame我对数据进行解码。