Recover from non-monotonic presentationTimeUs caused by naively-muxed B-frame content

[2bitoperations]

Fixes #3347

Problem

On Google TV Streamer 4K (MediaTek MT8696, c2.mtk.hevc.decoder),
decoding specific HEVC Main10 content causes the codec to return
output buffers with non-monotonic presentationTimeUs, while the
order in which those buffers are released remains correct — confirmed
by comparing each buffer's release-order rank against its
timestamp-sorted rank: the two never diverge by more than a few
positions, and the timestamp values themselves are individually
correct for the frame they belong to.

Reproduced independently of ExoPlayer, using a minimal test harness
built directly on AMediaCodec/AMediaExtractor (no ExoPlayer, no
Java MediaCodec) against the same file: identical non-monotonic
timestamp pattern. This confirms the defect is in the decoder/HAL
output, not in ExoPlayer's handling of it.

MediaCodecRenderer processes output buffers in raw release order
but uses the codec-reported presentationTimeUs directly for the
render/drop decision in MediaCodecVideoRenderer. When a buffer's
timestamp is behind the previous one, it's treated as arriving too
late and dropped. Measured on affected content: ~30% of decoded video
frames dropped, sustained throughout playback, reproduced in the
Media3 demo app as well as Plex, Jellyfin, and other ExoPlayer-based
players on the same device/file.

Fix

Re-derive each output buffer's presentationTimeUs from its release
order and the stream's known frame duration (from Format.frameRate),
instead of trusting the codec-reported value. Buffer release order is
left untouched — buffers are not held or reordered.

Testing

Verified on the affected device (Google TV Streamer 4K) against three
files that reliably reproduce the issue on unpatched main,
including a 90-second real-world clip: dropped-frame count goes from
~30% to 0, and the fix was visually confirmed on-device — direct
observation of smooth playback, not just log metrics. HOWEVER, with this approach, video and audio lose sync.

An earlier approach that buffered and released output buffers in
sorted-timestamp order was also tried and rejected: it also eliminated
logged drops, but produced visible playback artifacts on-device. That
result is why this PR does not reorder buffers.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[2bitoperations]

Pushed a fix for a second bug: audio/video would slowly drift out of
sync over long playback. Cause was an accumulating rounding error in
the per-frame timestamp math — invisible on a 90s test clip, ~58ms by
the end of a 2-hour movie. Fixed by rounding only the final offset,
not the per-frame duration.

Tested in two real players on the affected hardware (Jellyfin Android
TV, and Plezy). 40 minutes of real playback, no drift, no stutter.

[FongMi]

Thanks for the detailed investigation and for providing a legal,
reproducible sample. I agree that changing the late-frame threshold
does not address the underlying problem.

However, I don't think the current evidence isolates this as a
MediaTek decoder/HAL defect yet. The repro MP4 itself appears to have
a questionable presentation timeline for reordered frames.

I downloaded tos_x265_3.3_final.mp4 and inspected it with ffprobe.
All 360 video packets have PTS == DTS, even though the HEVC stream
uses B-frames and B-pyramid. For example:

packet PTS/DTS:
-0.080000 / -0.080000
-0.041667 / -0.041667
 0.000000 /  0.000000
 0.041667 /  0.041667
...

After decoding, the propagated frame PTS becomes non-monotonic:

frame PTS:
0.041667
0.083333
0.000000
0.125000
0.250000
0.291667
0.208333
...

while FFmpeg's best_effort_timestamp reconstructs approximately:

0.041667
0.083333
0.125000
0.166667
0.250000
0.291667
0.333333
...

This looks consistent with missing or incorrect composition offsets
in the MP4, possibly introduced by the raw-HEVC stream-copy muxing
pipeline in the repro README.

Android documents BufferInfo.presentationTimeUs as being derived
from the timestamp queued with the corresponding input buffer:

https://developer.android.com/reference/android/media/MediaCodec.BufferInfo#presentationTimeUs

Therefore, reproducing the same output timestamp sequence with
AMediaCodec/AMediaExtractor proves that ExoPlayer did not introduce
the sequence, but it does not by itself prove that the decoder/HAL is
behaving incorrectly. It may simply be propagating input timestamps
that do not describe the HEVC presentation order correctly.

Could we rule this out before changing MediaCodecRenderer?

1. Generate the same encoded content in a container with correct
   composition timestamps and verify that some packet PTS values
   differ from DTS where reordering requires it.
2. Log each timestamp queued into MediaCodec and its corresponding
   output BufferInfo.presentationTimeUs.
3. Compare packet PTS/DTS for the affected x265 3.3 file and the
   unaffected x265 4.1 file.

There is also a separate scope concern with the current patch: despite
being motivated by one MTK decoder, it rewrites timestamps for every
video decoder whenever Format.frameRate is known. This would replace
valid VFR timing, intentional gaps, duplicate frames, and format
transitions with a synthetic CFR timeline. The frame duration is also
not reset when the input frame rate changes.

The patch explains why this particular CFR sample becomes smooth, but
I think the source timeline needs to be ruled out before treating it
as a general Media3 renderer fix. If a player-side workaround is still
needed afterward, it should probably be gated to the affected codec
and device, activated only after detecting the known timestamp pattern,
and kept out of the generic MediaCodecRenderer path.

[2bitoperations]

Summary

MANY thanks for your helpful reply. We looked into all three of your suggestions. Short version: I no longer think this is a
MediaTek decoder defect. It's a bug in the reproduction repo's own muxing pipeline (which reflects bugs in the muxing pipelines of a bunch of files running around,) and as far as
we can tell it's independent of x265 version, encoder options (beyond whether B-frames are used
at all), and source content.

1. Composition timestamps: correctly-muxed vs. naively-muxed content

You were right about the repro file. Reproduced your ffprobe findings exactly on
tos_x265_3.3_final.mp4: every packet has PTS == DTS (0/360), and the propagated frame PTS is
non-monotonic in the pattern you posted, while best_effort_timestamp reconstructs a clean
sequence.

The repro README's mux step (ffmpeg -fflags +genpts -r 24 -i raw.hevc ... -c:v copy)
stream-copies a detached raw HEVC stream, which carries no composition-time
information of its own. +genpts fills the gap by assigning PTS = DTS = packet_index / fps —
decode order — which is only correct when there's no B-frame reordering to begin with. Minimal
confirmation, x265 4.1, defaults, nothing else changed:

ffmpeg -f lavfi -i "testsrc2=size=640x360:rate=24:duration=15" -f yuv4mpegpipe src.y4m

# encode straight to MP4 — libavformat assigns real composition offsets in-process
ffmpeg -i src.y4m -c:v libx265 -tag:v hvc1 good.mp4

# encode to a detached raw stream, then remux exactly like the repro README does
ffmpeg -i src.y4m -c:v libx265 -f hevc raw.hevc
ffmpeg -fflags +genpts -r 24 -i raw.hevc -c:v copy -tag:v hvc1 bad.mp4

good.mp4:  PTS != DTS on 270/360 packets  (correctly timed)
bad.mp4:   PTS != DTS on   0/360 packets  (every packet's PTS is just its DTS)

Same encoder, same options, same source — only the mux step differs.

Confirmed on-device too, on a build with none of our patches applied: correctly-muxed B-frame
content plays clean; content muxed the repro's way drops ~30% of frames, consistently, across
every file we've tried.

2. x265 3.3 vs. current x265 4.1

Re-encoded the same Tears of Steel source with current x265 (4.1+1-1d117be), options matched as
closely as possible, muxed with the unmodified repro pipeline. Result: indistinguishable from
the 3.3 encode — 0/360 PTS != DTS, same broken pattern.

This briefly looked like it contradicted our own earlier finding that x265 4.1 content didn't
reproduce the bug, so we ran it down rather than gloss over it. The earlier "unaffected" 4.1 file
has real composition offsets (PTS != DTS on 243/360 packets) — it was muxed correctly, almost
certainly via a single-step ffmpeg -c:v libx265 ... output.mp4 encode, not the raw-stream-then-
remux process the repro pipeline uses. That two-step process isn't optional for x265 3.3: its CLI
has no muxer at all (-o is raw-bitstream-only). Confirmed both ends on-device: the original file
still plays with zero dropped frames; a freshly built, zero-options x265 4.1 file put through the
naive remux drops 682/2280 frames (29.9%). Same decoder, same device, same encoder version — only
the mux method differs.

So: not a contradiction, and not an x265-version-specific bug. Encoder version is irrelevant to
the trigger; whether the file went through a proper muxer is the entire story.

3. Logging queued input vs. dequeued output timestamps

Patch (diagnostic-only, no behavior change):
6fc70188e49667a894b44ada761193e28ebda8b9
on our fork, based on the clean release branch (i.e. without our reorder/relabel patch from this
PR applied — this measures the raw, unpatched decoder). Logs the timestamp on every
queueInputBuffer/queueSecureInputBuffer call and every buffer dequeued from
drainOutputBuffer. Built, installed on the real device, captured against
tos_x265_3.3_final.mp4 and the 4.1 re-encode:

File	queued (IN)	IN non-monotonic	dequeued (OUT)	OUT non-monotonic
3.x repro	291	0	276	117
4.x re-encode	284	0	270	114

Queued-input order is perfectly monotonic in both — a direct echo of the container's decode-order
timestamps. Dequeued-output order is non-monotonic in both, at essentially the rate the container
alone predicted, with matching non-monotonic transition offsets between the two files.

The key check: every single dequeued output timestamp, in both files, is one of the exact
values that was queued as input — zero exceptions. c2.mtk.hevc.decoder never emits a
timestamp it wasn't given. Direct on-device confirmation of the presentationTimeUs contract you
cited — the decoder echoes back exactly what it was handed; the non-monotonicity is already
present in what it was handed.

Where this leaves things

I don't think this is evidence of a MediaTek decoder defect. However, what it does look like is
that files encoded with the x265 3.x series are likely to actually hit this in practice: its CLI
has no muxer of its own, so any pipeline built around it needs a separate remux step, and if that
step doesn't compute real composition offsets — ours doesn't, and we don't know what tooling
popularly shared media pipelines typically used either — this is exactly the result.

VLC plays these incorrectly-timestamped files just fine, so I want to dig into why. My guess is
it comes down to VLC's "if it's only a little late, just display it right now" approach versus
shouldDropOutputBuffer, which is closer to "if the presentation timestamp says it's late, chuck
that frame right in the garbage." I'll look into how VLC actually handles this (and crucially, how it is not overly permissive, as you have cautioned against) and follow up.

[2bitoperations]

Why VLC doesn't stutter on this content

Instrumented VLC itself (not just read its source) and ran it on-device against the same repro file. It receives equally or more disordered decoder output than ExoPlayer from c2.mtk.hevc.decoder — 52.2% non-monotonic steps in the raw echoed presentationTimeUs, vs. ~42% measured earlier from ExoPlayer, same 1×–4× frame-period jump signature. So it's not decoder behavior, and it's not a threading/queuing difference between the two clients.

VLC just never lets that value reach anything that schedules a frame. Its MP4 demuxer refuses to invent a PTS for a video track when the file has no ctts box — true of any file built via the raw-ES + -fflags +genpts pipeline this whole thread has been using:

// mp4.c, https://code.videolan.org/videolan/vlc/-/blob/ac6c2a405d652b5576128ceb9fec2c342f0e83ec/modules/demux/mp4/mp4.c#L1391-L1396
if( MP4_TrackGetPTSDelta( p_demux, tk, &i_delta ) )
    p_block->i_pts = p_block->i_dts + i_delta;
else if( tk->fmt.i_cat != VIDEO_ES )
    p_block->i_pts = p_block->i_dts;
else
    p_block->i_pts = VLC_TICK_INVALID;   // video track, no ctts: don't guess

That VLC_TICK_INVALID trips a fallback in the MediaCodec wrapper that relabels every output picture by decode-order index instead of trusting the (disordered) decoder echo — confirmed on-device: 0/2280 non-monotonic after the fallback, vs. the 52.2% it replaced.

androidx/media3's MP4 extractor has no "PTS unknown" state. When ctts is absent it silently produces PTS = DTS, because the offset term it would add just stays at its zero default:

// BoxParser.java, https://github.com/androidx/media/blob/5fb306449733dd71595700c1227ad6087578c559/libraries/extractor/src/main/java/androidx/media3/extractor/mp4/BoxParser.java#L516
int timestampOffset = 0;
...
// https://github.com/androidx/media/blob/5fb306449733dd71595700c1227ad6087578c559/libraries/extractor/src/main/java/androidx/media3/extractor/mp4/BoxParser.java#L627
timestamps[i] = timestampTimeUnits + timestampOffset;  // ctts == null → always 0 → PTS = DTS

That definite-but-wrong PTS is exactly what shouldDropOutputBuffer (MediaCodecVideoRenderer.java:2166) later measures against the audio clock and drops.

So: not a MediaTek defect, not an ExoPlayer bug in the usual sense — one demuxer treats a missing ctts as "unknown," the other treats it as "zero," and that's the entire difference in outcome for this class of file.

Next: I'm going to look at overhauling this to give media3 a first-class notion of "missing PTS," and only fall back to today's spray-and-pray frame-dropping behavior in that specific case.