Difference between revisions of "FATE failures"

From MultimediaWiki
Jump to navigation Jump to search
m
 
(21 intermediate revisions by 3 users not shown)
Line 5: Line 5:
=== H.264 ===
=== H.264 ===


MAIN/CABREF3_Sand_D.264 ... JF ( yuv420p @  352x288) not equal
tandberg 4/5 have large codec delays because of misordered frames, and the delay jumps randomly within the file. We don't preroll well enough to know the correct delay in advance and so at each point where we return a frame too far ahead in the future, we drop the preceeding frames once we get around to decoding them because we've already passed that point in units returned to the application. The correct solution for these would be to increase delay dynamically whenever we encounter a gap until either we're at max-delay, or until we've resolved the gap. almost certainly, max-delay should be settable in user applications so we can disable it for live/low-latency purposes. Q: should we return old frames also? The user may want to reorder himself (although this doesn't make much sense). Fate currently uses -strict 1 as a workaround for these samples, which simply forces codec-delay to 16. This is not the correct solution.
MAIN/CAFI1_SVA_C.264 ... JF ( yuv420p @  720x480) not equal
  MAIN/MR4_TANDBERG_C.264 ... JF ( yuv420p @  176x144 25 fps) not equal
MAIN/CAPAMA3_Sand_F.264 ... JF ( yuv420p @  352x288) not equal
  MAIN/MR5_TANDBERG_C.264 ... JF ( yuv420p @  176x144 /  25 fps) not equal
  MAIN/CVFI1_SVA_C.264 ... JF ( yuv420p @  720x480) not equal
 
MAIN/CVFI2_SVA_C.264 ... JF ( yuv420p @ 720x480) not equal
=== Other samples from the H264 conformance suite that don't play back correctly ===
  MAIN/FM1_FT_E.264 ... JF ( yuv420p @  176x144) not equal
 
  MAIN/FM2_SVA_C.264 ... JF ( yuv420p @  176x144) not equal
FMO:
MAIN/HCBP1_HHI_A.264 ... JF ( yuv420p @ 352x288) not equal
  MAIN/FM1_FT_E.264 ... JF ( yuv420p @  176x144 /  25 fps) not equal
  MAIN/HCBP2_HHI_A.264 ... JF ( yuv420p @  352x288) not equal
  MAIN/FM2_SVA_C.264 ... JF ( yuv420p @  176x144 25 fps) not equal
MAIN/HCMP1_HHI_A.264 ... JF ( yuv420p @  352x288) not equal
  MAIN/FM1_BT_B.h264 ... JF ( yuv420p @  176x144 / 25 fps) not equal
MAIN/MR2_TANDBERG_E.264 ... JF ( yuv420p @  176x144) not equal
 
MAIN/MR3_TANDBERG_B.264 ... JF ( yuv420p @  176x144) not equal
For this one, we output only 12 of a few hundred-or-so frames, unanalyzed beyond that:
MAIN/MR4_TANDBERG_C.264 ... JF ( yuv420p @  176x144) not equal
  MAIN/sp1_bt_a.h264 ... JF ( yuv420p @  176x144 /  25 fps) not equal
  MAIN/MR5_TANDBERG_C.264 ... JF ( yuv420p @  176x144) not equal
 
MAIN/camp_mot_fld0_full.26l ... JF ( yuv420p @ 720x480) not equal
Looks like garbage, unanalyzed beyond that:
  MAIN/camp_mot_picaff0_full.26l ... JF ( yuv420p @  720x480) not equal
  MAIN/H26L/BitstreamExchange/sp2_bt_b.h264 ... JF ( yuv420p @  176x144 /  25 fps) not equal
  MAIN/cvmp_mot_fld0_full_B.26l ... JF ( yuv420p @  720x480) not equal
 
MAIN/CVMP_MOT_FLD_L30_B.26l ... JF ( yuv420p @ 720x480) not equal
Looks like CAVLC-444 isn't completely implemented:
  MAIN/CVMP_MOT_FRM_L31_B.26l ... JF ( yuv420p @  720x480) not equal
  PP/PPCV444I-7.264 ... JF ( yuv420p @  48x80 25 fps) not equal
MAIN/cvmp_mot_picaff0_full_B.26l ... JF ( yuv420p @  720x480) not equal
  PP/PPCV444I4_Mitsubishi_A.264 ... JF ( yuv420p @  48x48 25 fps) not equal
  MAIN/FM1_BT_B.h264 ... JF ( yuv420p @  176x144) not equal
  PP/PPCV444I5_Mitsubishi_A.264 ... JF ( 50 fps 25 tbr 25 fps) not equal
  MAIN/H26L/BitstreamExchange/sp2_bt_b.h264 ... JF ( yuv420p @ 176x144) not equal
  PP/PPCV444I6_Mitsubishi_A.264 ... JF ( yuv420p @  48x48 25 fps) not equal
MAIN/MR6_BT_B.h264 ... JF ( yuv420p @  192x128) not equal
  PP/Professional_profiles/PPCV444I1_Thomson_A.bits ... JF ( 50 fps 25 tbr 25 fps) not equal
MAIN/MR8_BT_B.h264 ... JF ( yuv420p @  192x128) not equal
  PP/Professional_profiles/PPCV444I2_Thomson_A.bits ... JF ( 50 fps 25 tbr 25 fps) not equal
  MAIN/sp1_bt_a.h264 ... JF ( yuv420p @  176x144) not equal
  PP/Professional_profiles/PPCV444I3_Thomson_A.bits ... JF ( 50 fps 25 tbr 25 fps) not equal
MAIN/CABA3_Sony_C.jsv ... JF ( yuv420p @ 176x144) not equal
 
  MAIN/CABACI3_Sony_B.jsv ... JF ( yuv420p @  176x144) not equal
Missing 14-bits support:
MAIN/CABAST3_Sony_E.jsv ... JF ( yuv420p @  352x288) not equal
  PP/PPH444I4_Mitsubishi_A.264 ... JF ( 25 fps 25 tbr 25 fps) not equal
MAIN/CABASTBR3_Sony_B.jsv ... JF ( yuv420p @  352x288) not equal
  PP/PPH444I5_Mitsubishi_A.264 ... JF ( 25 fps 25 tbr 25 fps) not equal
  MAIN/CACQP3_Sony_D.jsv ... JF ( yuv420p @  176x144) not equal
  PP/PPH444I6_Mitsubishi_A.264 ... JF ( 25 fps 25 tbr / 25 fps) not equal
MAIN/CAMACI3_Sony_C.jsv ... JF ( yuv420p @  192x128) not equal
  PP/PPH444P6_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr /  25 fps) not equal
MAIN/CAMASL3_Sony_B.jsv ... JF ( yuv420p @  192x128) not equal
  PP/PPH444P7_Mitsubishi_A.264 ... JF ( 25 fps 25 tbr 25 fps) not equal
MAIN/CANL3_Sony_C.jsv ... JF ( yuv420p @  176x144) not equal
  PP/PPH444P8_Mitsubishi_A.264 ... JF ( 25 fps 25 tbr 25 fps) not equal
MAIN/CAPM3_Sony_D.jsv ... JF ( yuv420p @ 176x144) not equal
  PP/PPH444P9_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr / 25 fps) not equal
  MAIN/CVBS3_Sony_C.jsv ... JF ( yuv420p @  176x144) not equal
  PP/Professional_profiles/PPH444I1_Thomson_A.bits ... JF ( 25 fps @  25 tbr / 25 fps) not equal
MAIN/CVFC1_Sony_C.jsv ... JF ( yuv420p @  338x274) not equal
  PP/Professional_profiles/PPH444I2_Thomson_A.bits ... JF ( 25 fps @  25 tbr / 25 fps) not equal
MAIN/CVFI1_Sony_D.jsv ... JF ( yuv420p @  720x480) not equal
  PP/Professional_profiles/PPH444I3_Thomson_A.bits ... JF ( 25 fps @  25 tbr / 25 fps) not equal
  MAIN/CVFI2_Sony_H.jsv ... JF ( yuv420p @  720x480) not equal
  PP/Professional_profiles/PPH444P1_Thomson_A.bits ... JF ( 25 fps @  25 tbr /  25 fps) not equal
MAIN/CVMAPAQP3_Sony_E.jsv ... JF ( yuv420p @  720x480) not equal
  PP/Professional_profiles/PPH444P2_Thomson_A.bits ... JF ( 25 fps @  25 tbr /  25 fps) not equal
MAIN/CVMAQP3_Sony_D.jsv ... JF ( yuv420p @  192x128) not equal
  PP/Professional_profiles/PPH444P3_Thomson_A.bits ... JF ( 25 fps @  25 tbr /  25  fps) not equal
MAIN/CVNLFI1_Sony_C.jsv ... JF ( yuv420p @  720x480) not equal
  PP/Professional_profiles/PPH444P4_Thomson_A.bits ... JF ( 25 fps @  25 tbr /  25  fps) not equal
MAIN/CVNLFI2_Sony_H.jsv ... JF ( yuv420p @  720x480) not equal
  PP/Professional_profiles/PPH444P5_Thomson_A.bits ... JF ( 25 fps @  25 tbr /  25  fps) not equal
MAIN/CVSE2_Sony_B.jsv ... JF ( yuv420p @  176x144) not equal
 
MAIN/CVSE3_Sony_H.jsv ... JF ( yuv420p @  176x144) not equal
FMO + 14-bits:
MAIN/CVSEFDFT3_Sony_E.jsv ... JF ( yuv420p @  176x144) not equal
  PP/PPH444I-7.264 ... JF ( 25 fps @  25 tbr /  25  fps) not equal
MAIN/FI1_Sony_E.jsv ... JF ( yuv420p @  192x128) not equal
  PP/PPH444P-10.264 ... JF ( 25 fps @  25 tbr / 25 fps) not equal
MAIN/Sharp_MP_Field_1_B.jvt ... JF ( yuv420p @  720x480) not equal
 
MAIN/Sharp_MP_Field_2_B.jvt ... JF ( yuv420p @  720x480) not equal
[[Category:FATE]]
MAIN/Sharp_MP_Field_3_B.jvt ... JF ( yuv420p @  720x480) not equal
FR/freh10.264 ... JF ( yuv420p @  352x288) not equal
FR/freh11.264 ... JF ( yuv420p @ 352x288) not equal
  FR/freh4.264 ... JF ( yuv420p @  352x288) not equal
  FR/freh6.264 ... JF ( yuv420p @  352x288) not equal
FR/Freh7_B.264 ... JF ( yuv420p @ 352x288) not equal
  FR/FREXT1_TANDBERG_A.264 ... JF ( yuv420p @  352x288) not equal
  FR/FREXT2_TANDBERG_A.264 ... JF ( yuv420p 352x288) not equal
FR/FREXT3_TANDBERG_A.264 ... JF ( yuv420p @ 352x288) not equal
  FR/FRExt_MMCO4_Sony_B/FRExt_MMCO4_Sony_B.264 ... JF ( yuv420p @  352x288) not equal
  FR/HCHP1_HHI_B.264 ... JF ( yuv420p @  352x288) not equal
FR/HCHP2_HHI_A.264 ... JF ( yuv420p @ 352x288) not equal
  FR/HCHP3_HHI_A.264 ... JF ( yuv420p @  720x576) not equal
  FR/Hi422FR10_SONY_B.264 ... JF ( yuv420p @  352x288) not equal
FR/Hi422FR11_SONY_B.264 ... JF ( yuv420p 352x288) not equal
FR/Hi422FR12_SONY_B.264 ... JF ( yuv420p @ 352x288) not equal
  FR/Hi422FR13_SONY_B.264 ... JF ( yuv420p10le @  352x288) not equal
  FR/Hi422FR14_SONY_B.264 ... JF ( yuv420p10le @  352x288) not equal
FR/Hi422FR15_SONY_B.264 ... JF ( yuv420p10le 352x288) not equal
FR/Hi422FREXT16_SONY_A.264 ... JF./script.sh: line 26: 45000 Segmentation fault ${FFMPEG} -i ${file} -vcodec rawvideo -f framecrc -y ${file}.ffcrc >&/dev/null ( yuv420p10le @ 1920x1088) not equal
  FR/Hi422FREXT17_SONY_A.264 ... JF./script.sh: line 26: 45019 Segmentation fault ${FFMPEG} -i ${file} -vcodec rawvideo -f framecrc -y ${file}.ffcrc >&/dev/null ( yuv420p10le @  1920x1088) not equal
  FR/Hi422FREXT18_SONY_A.264 ... JF ( yuv420p10le @  720x480) not equal
FR/Hi422FREXT19_SONY_A.264 ... JF ( yuv420p10le 720x480) not equal
FR/HPCAFL_BRCM_C.264 ... JF ( yuv420p @ 352x288) not equal
  FR/HPCAFLNL_BRCM_C.264 ... JF ( yuv420p @  352x288) not equal
FR/HPCAMAPALQ_BRCM_B.264 ... JF ( yuv420p @  352x288) not equal
FR/HPCVFL_BRCM_A.264 ... JF ( yuv420p @  352x288) not equal
  FR/HPCVFLNL_BRCM_A.264 ... JF ( yuv420p 352x288) not equal
FR/HVLCFI0_Sony_B/HVLCFI0_Sony_B.264 ... JF ( yuv420p @ 720x480) not equal
  FR/HVLCMFF0_Sony_B/HVLCMFF0_Sony_B.264 ... JF ( yuv420p @  720x480) not equal
  FR/HVLCPFF0_Sony_B/HVLCPFF0_Sony_B.264 ... JF ( yuv420p 720x480) not equal
FR/FRExt2_Panasonic.avc ... JF ( yuv420p @ 352x288) not equal
  FR/Hi422FR1_SONY_A.jsv ... JF ( yuv420p @  352x288) not equal
FR/Hi422FR2_SONY_A.jsv ... JF ( yuv420p @  352x288) not equal
FR/Hi422FR3_SONY_A.jsv ... JF ( yuv420p @  352x288) not equal
FR/Hi422FR4_SONY_A.jsv ... JF ( yuv420p @  352x288) not equal
FR/Hi422FR6_SONY_A.jsv ... JF ( yuv420p10le @  352x288) not equal
FR/Hi422FR7_SONY_A.jsv ... JF ( yuv420p10le @  352x288) not equal
FR/Hi422FR8_SONY_A.jsv ... JF ( yuv420p10le @  352x288) not equal
FR/Hi422FR9_SONY_A.jsv ... JF ( yuv420p10le @  352x288) not equal
  PP/PPCV444I-7.264 ... JF ( yuv420p 48x80) not equal
  PP/PPCV444I4_Mitsubishi_A.264 ... JF ( yuv420p @ 48x48) not equal
  PP/PPCV444I5_Mitsubishi_A.264 ... JF ( 50 fps @  25 tbr) not equal
PP/PPCV444I6_Mitsubishi_A.264 ... JF ( yuv420p @ 48x48) not equal
  PP/PPH422I1_Panasonic_A.264 ... JF ( yuv420p10le @  1280x720) not equal
  PP/PPH422I2_Panasonic_A.264 ... JF ( yuv420p10le 1280x720) not equal
PP/PPH422I3_Panasonic_A.264 ... JF ( yuv420p10le @ 1280x720) not equal
  PP/PPH422I4_Panasonic_A.264 ... JF ( yuv420p10le @  1920x1072) not equal
  PP/PPH422I5_Panasonic_A.264 ... JF ( yuv420p10le 1920x1072) not equal
PP/PPH422I6_Panasonic_A.264 ... JF ( yuv420p10le @ 1920x1072) not equal
  PP/PPH422I7_Panasonic_A.264 ... JF ( yuv420p10le @  1920x1072) not equal
  PP/PPH444I-7.264 ... JF ( 25 fps @  25 tbr) not equal
  PP/PPH444I4_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr) not equal
  PP/PPH444I5_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr) not equal
  PP/PPH444I6_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr) not equal
  PP/PPH444P-10.264 ... JF ( 25 fps @  25 tbr) not equal
  PP/PPH444P6_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr) not equal
  PP/PPH444P7_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr) not equal
  PP/PPH444P8_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr) not equal
  PP/PPH444P9_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr) not equal
PP/PPCV444I1_2008.bits ... JF ( 50 fps @ 25 tbr) not equal
  PP/PPCV444I2_2008.bits ... JF ( 50 fps @  25 tbr) not equal
PP/PPCV444I3_2008.bits ... JF ( 50 fps @ 25 tbr) not equal
  PP/PPH444I1_2008.bits ... JF ( 25 fps @  25 tbr) not equal
  PP/PPH444P1_2008.bits ... JF ( 25 fps @  25 tbr) not equal
  PP/PPH444P2_2008.bits ... JF ( 25 fps @  25 tbr) not equal
  PP/PPH444P3_2008.bits ... JF ( 25 fps @  25 tbr) not equal
  PP/PPH444P4_2008.bits ... JF ( 25 fps @  25 tbr) not equal
  PP/PPH444P5_2008.bits ... JF ( 25 fps @  25 tbr) not equal

Latest revision as of 08:49, 24 April 2013

Although FATE can appear green, in fact many of the samples in FATE are not bit-exact.

The following samples are not bit-exact.

H.264

tandberg 4/5 have large codec delays because of misordered frames, and the delay jumps randomly within the file. We don't preroll well enough to know the correct delay in advance and so at each point where we return a frame too far ahead in the future, we drop the preceeding frames once we get around to decoding them because we've already passed that point in units returned to the application. The correct solution for these would be to increase delay dynamically whenever we encounter a gap until either we're at max-delay, or until we've resolved the gap. almost certainly, max-delay should be settable in user applications so we can disable it for live/low-latency purposes. Q: should we return old frames also? The user may want to reorder himself (although this doesn't make much sense). Fate currently uses -strict 1 as a workaround for these samples, which simply forces codec-delay to 16. This is not the correct solution.

MAIN/MR4_TANDBERG_C.264 ... JF ( yuv420p @  176x144 /  25  fps) not equal
MAIN/MR5_TANDBERG_C.264 ... JF ( yuv420p @  176x144 /  25  fps) not equal

Other samples from the H264 conformance suite that don't play back correctly

FMO:

MAIN/FM1_FT_E.264 ... JF ( yuv420p @  176x144 /  25  fps) not equal
MAIN/FM2_SVA_C.264 ... JF ( yuv420p @  176x144 /  25  fps) not equal
MAIN/FM1_BT_B.h264 ... JF ( yuv420p @  176x144 /  25  fps) not equal

For this one, we output only 12 of a few hundred-or-so frames, unanalyzed beyond that:

MAIN/sp1_bt_a.h264 ... JF ( yuv420p @  176x144 /  25  fps) not equal

Looks like garbage, unanalyzed beyond that:

MAIN/H26L/BitstreamExchange/sp2_bt_b.h264 ... JF ( yuv420p @  176x144 /  25  fps) not equal

Looks like CAVLC-444 isn't completely implemented:

PP/PPCV444I-7.264 ... JF ( yuv420p @  48x80 /  25  fps) not equal
PP/PPCV444I4_Mitsubishi_A.264 ... JF ( yuv420p @  48x48 /  25  fps) not equal
PP/PPCV444I5_Mitsubishi_A.264 ... JF ( 50 fps @  25 tbr /  25  fps) not equal
PP/PPCV444I6_Mitsubishi_A.264 ... JF ( yuv420p @  48x48 /  25  fps) not equal
PP/Professional_profiles/PPCV444I1_Thomson_A.bits ... JF ( 50 fps @  25 tbr /  25  fps) not equal
PP/Professional_profiles/PPCV444I2_Thomson_A.bits ... JF ( 50 fps @  25 tbr /  25  fps) not equal
PP/Professional_profiles/PPCV444I3_Thomson_A.bits ... JF ( 50 fps @  25 tbr /  25  fps) not equal

Missing 14-bits support:

PP/PPH444I4_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/PPH444I5_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/PPH444I6_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/PPH444P6_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/PPH444P7_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/PPH444P8_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/PPH444P9_Mitsubishi_A.264 ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/Professional_profiles/PPH444I1_Thomson_A.bits ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/Professional_profiles/PPH444I2_Thomson_A.bits ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/Professional_profiles/PPH444I3_Thomson_A.bits ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/Professional_profiles/PPH444P1_Thomson_A.bits ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/Professional_profiles/PPH444P2_Thomson_A.bits ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/Professional_profiles/PPH444P3_Thomson_A.bits ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/Professional_profiles/PPH444P4_Thomson_A.bits ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/Professional_profiles/PPH444P5_Thomson_A.bits ... JF ( 25 fps @  25 tbr /  25  fps) not equal

FMO + 14-bits:

PP/PPH444I-7.264 ... JF ( 25 fps @  25 tbr /  25  fps) not equal
PP/PPH444P-10.264 ... JF ( 25 fps @  25 tbr /  25  fps) not equal