Compression standards and quality go hand in hand

Introduction

When I described the MPEG workflow in How does MPEG actually work? I highlighted the role of quality assessment across the entire MPEG standard life cycle: at the time of issuing a Call for Evidence (CfE) or a Call for Proposals (CfP), carrying out Core Experiments (CE) or executing Verification Tests.

We should consider, however, that in 30 years the coverage of the word “media” has changed substantially.

Originally (1989-90) the media types tested were Standard Definition (SD) 2D rectangular video and stereo audio. Today the video data types include also High Definition (HD), Ultra High Definition (UHD), Stereoscopic Video, High Dynamic Range (HDR) and Wide Colour Gamut (WCG), and Omnidirectional (Video 360).

The video information can be 2D, but also multiview and 3D: stereoscopic, 3 degrees of freedom + (3DoF+), 6 degrees of freedom (6DoF) and various forms of light field. Audio has evolved to different forms of Multichannel Audio and 6DoF. Recently Point Clouds were added to the media types for which MPEG has applied subjective quality measurements to develop compression standards.

In this article I would like to go inside the work that comes together with subjectively assessing the quality of media compression.

Preparing for the tests

Even before MPEG decides to issue a CfE or CfP for compression of some type of media content, viewing or listening to content may take place to appreciate the value of a proposal. When a Call is issued MPEG has already reached a pretty clear understanding of the use cases and requirements (at the time of a CfE) or the final version of them (at the time of a CfP).

The first step is the availability of appropriate test sequences. Sequences may already be in the MPEG Content Repository or are spontaneously offered by members or are obtained from industry representatives by issuing a Call for Content.

Selection of test sequences is a critical step because we need sequences that are suitable for the media type and representative of the use cases, and in a number that allows us to carry out meaningful and realistic tests.

By the CfE or CfP time, MPEG has also decided what is the standard against which responses to the CfE or CfP should be tested. For example, in the case of HEVC, the comparison was with AVC and, in the case of VVC, the comparison was with HEVC. In the case of Internet Video Coding (IVC) the comparison was with AVC. When such a reference standard does not exist (this was the case for, e.g., all layers of MPEG-1 Audio and Point Cloud Compression) the codec built during the exploratory phase that groups together state of the art tools is used.

Once the test sequences have been selected, the experts in charge of the reference software are asked to run the reference software encoder and produce the “anchors”, i.e. the video sequences encoded according to the “old” standard. The anchors are made available in an FTP site so that anybody intending to respond to the CfE or CfP can download them.

The set up used to generate the anchors are documented in “configuration files” for each class of submission. In the case of video these are (SD/HD/UHD, HDR, 360º) and design conditions (low delay or random access). Obviously, in order to have comparable data, all proponents must use the same configuration files when they encode the test sequences using their technology.

As logistic considerations play a key role in the preparation of quality tests, would-be proponents must submit formal statements of intention to respond to the CfE or CfP to the Requirements group chair (currently Jörn Ostermann) and Test group chair (currently Vittorio Baroncini), and the chair of the relevant technology group, 2 months before the submission deadline.

At the meeting before the responses to the CfP/CfE are due, an ad hoc group (AHG) is established with the task of promoting awareness of the Call among the industry, to carry out the tests, draft a report and submit conclusions on the quality tests to the following MPEG meeting.

Carrying out the tests

The actual tests are entirely carried out by the AHG under the leadership of AHG chairs (typically the Test chair and a representative of the relevant technology group).

Proponents send on hard disk drives or on an FTP site their files containing encoded data to the Test Chair by the deadline specified in the CfE/CfP.

When all drives are received the Test chair performs the following tasks

  1. Acquire special hardware and displays for the tests (if needed)
  2. Verify that the submitted files are all on disk and readable
  3. Assign submitted files to independent test labs (sometimes even 10 test labs are concurrently involved is a test run)
  4. Make copies and distribute the relevant files to the test labs
  5. Specify the tests or provide the scripts for the test labs to carry out.

The test labs carry out a first run of the tests and provide their results for the Test chair to verify. If necessary, the Test chair requests another test run or even visits the test labs to make sure that the tests will run properly.

When this “tuning” phase has been successfully executed, all test labs run the entire set of tests assigned to them using test subjects. Tens of “non-expert” subjects may be involved for several days.

Here is a sample of what it means to attend subjective quality tests

Test report

Tests results undergo a critical revision according to the following steps

  1. The Test chair collects all results from all test labs performs a statistical analysis of the data, prepares and submits a final report to the AHG
  2. The report id discussed in the AHG and may be revised depending on the discussions
  3. The AHG draws and submits its conclusions to MPEG along with the report
  4. Report and conclusions are added to all the material submitted by proponents
  5. The Requirements and the technology group in charge of the media type evaluate the material and rank the proposals. Because of the sensitivity of some of the data all the material is not made public.

This signals the end of the competition phase and the beginning of the collaboration phase.

Other media types

In general the process above has been described having specifically rectangular 2D or 360º video in mind. Most of the process applies to other media types with some specific actions to be made for each of them, e.g.

  1. 3DTV: for the purpose of 3D HEVC tests, polarised glasses as in 3d movies and autostereoscopic displays were used;
  2. 3DoF+: a common synthesiser will be used in the upcoming 3D0F+ tests to synthesise views that are not available at the decoder;
  3. Audio: in general subjects need to be carefully trained for the specific tests;
  4. Point Clouds: videos generated by a common presentation engine consisting of point clouds animated by a script (by rotating the object and seeing it from different viewpoints) were tested for quality as if they were natural videos (NB: there were no established method to assess the quality of point clouds before. It was demonstrated that the subjective tests converged to the same results as the objective measurements).

Verification tests

Verification tests are executed with a similar process. Test sequences are selected and compressed by experts running reference software for the “old” and the “new” standard. Subjective tests are carried out as done in CfE/CfP subjective tests. Test results are made public to provide the industry with guidance on the performance of the new standard. See as examples the Verification Test Report for HDR/WCG Video Coding Using HEVC Main 10 Profile and the MPEG-H 3D Audio Verification Test Report.

Conclusions

Quality tests play an enabling role in all phases of development of a media compression standard. For 30 years MPEG has succeeded in mobilising – on a voluntary basis – the necessary organisational and human resources to perform this critical task.

I hope that, with this post, I have opened a window to an aspect of MPEG life that is instrumental to offer industry the best technology so that users can have the best media experience.

Posts in this thread (in bold this post)

Digging deeper in the MPEG work

Introduction

In How does MPEG actually work? I described the “MPEG standards work flow” from a proposed an idea to the release of the corresponding standard and its verification. I also highlighted the key roleplayed by MPEG experts as the real makers of MPEG standards.

In this article I would like to introduce the role played by the peculiar MPEG organisation and some of its members in facilitating the work of MPEG experts.

The MPEG membership

MPEG operates in the framework of the International Organization for Standardization (ISO). To be entitled to attend and actively participate in the MPEG work, experts or their companies must be members of one of the 162 national standards organisations who are members of ISO. Respondents to an MPEG Call for Evidence (CfE) or Call for proposals (CfP) are allowed to attend a meeting where their proposal is presented for the first time. To continue attending, however, proponents are required to become formal members (and they have better to do so if they want to defend their proposals).

The MPEG organisation

The figure below depicts the conceptual MPEG organisation as of today. In its 30 years MPEG has created and disbanded several groups created for specific needs. More about the history of the MPEG groups and their chairs can be found here.

 

New ideas are typically submitted to the Requirements group. If they are considered worth of further explorations they are typically discussed in ad hoc groups (AHG) after the meeting. As explained in How does MPEG actually work? the AHG will report its findings at the following MPEG meeting. After a few iterations, MPEG will produce a Call for Evidence (CfE) or Call for Proposals (CfP) with due publicity in the Press Release. At that time. the AHG is charged with the task to disseminate the CfE or CfP, prepare for the logistics of the tests and perform a first assessment of the responses.

This does not mean that the Requirements group is no longer involved in this standard because the group typically continues the development of Use Cases and Requirements while the technical work progresses. When necessary, requirements are reviewed with the appropriate technology groups and may give rise to new CfPs whose outcome is fed into the work of the relevant technology groups after an assessment.

Today the task of the Test group is no longer confined to the assessment of the quality of proposals at CfE and CfP time. Designing and executing appropriate quality tests in support to Core Experiments has become the norm especially in the Video group.

If a Call requests Proposals, the Requirements group, in conjunction with the Test group and possibly one or more technology group, reviews the result of the AHG and makes a final assessment. If the technologies submitted are judged to be sufficient to initiate the development of a standard, the activity is tranferred to that/those group(s) for standard development.

Current MPEG Chairs are

Requirements Jörn Ostermann Leibniz Univ. Hannover
Systems Youngkwon Lim Samsung
Video Lu Yu, Jens-Rainer Ohm, Gary Sullivan Zhejiang Univ., RWTH Aachen, Microsoft
Audio Schuyler Quackenbush Audio Research Labs
3DGC Marius Preda Institut Mines Télécom
Test Vittorio Baroncini GBTech
Communication Kyuheon Kim Kyunghee Univ.

The chairs come from a variety of countries (CN, DE. FR, IT, KR, US) and organisations (small/medium size and large companies, and Universities).

Joint meetings

A key MPEG feature is the immediate availability of the necessary technical expertise to discuss matters that cross organisational boundaries. The speed of development and quality of MPEG standards would hardly be possible if MPEG did not have the ability to timely deploy the necessary expertise to address multi-faceted issues.

Let’s take as an example this video where the basketball player is represented as a compressed dynamic point cloud in a 360ᵒ video. The matter is raised at a meeting, discussed at a Chairs meeting where the need for a joint meeting is identified. This is proposed to the MPEG plenary and held with the participation of Requirements, Systems and 3D Graphics Coding experts. The outcome of such a meeting may be anything from the acknowledgement that “we don’t know well enough yet”, to the identification of technologies that can simply be developed as a collaborative effort of MPEG experts or require a CfP.

As a typical example let’s consider some of the joint meetings held at MPEG 124 (20 in total):

  1. Anchors for 3DoF+; Participants from Video and Test meet to select the anchors (reference sequences) to be used in the 3DoF+ CfP (Monday, 15:00 to 16:00)
  2. 3DoF+ CfP; Participants from Requirements, Video and Test meet to discuss the text of the CfP that was published at MPEG 124 (Tuesday, 09:00 to 10:00)
  3. 3DoF+ CfP & viewing; Participants from Video and Test meet to review the text of and view the content for the CfP (Wednesday, 11:30 to 12:30)
  4. 3DoF+ viewing; Participants from Video and Test meet to view 3DoF+ content for the CfP (Thursday, 12:00 to 13:00)
  5. Systems aspects for Point Cloud Compression; Participants from 3DG and Systems meet to discuss the topic intruduced in this session (Tuesday, 09:00 to 10:00 and Thursday, 12:00 to 13:00)
  6. MPEG-I Scene graph; Participants from Requirements, Systems and Audio meet to discuss Audio needs for a scene description technology (Thursday, 09:00 to 10:00)
  7. Neural Network Compression CfE results; Participants from Requirements and Video meet to discuss the CfE results and decide whether there is room for a CfP (Tuesday, 16:00 to 17:00)

Meeting documents

How are documents fed into the MPEG process? This is not a marginal aspect if we consider that the number of submissions uploaded can easily cross 1000 documents. In an age of (almost) pervasive broadband internet, it is hard to imagine how MPEG could operate in a totally “paper-based” fashion when its membership was already around 300. In October 1995 Pete Schirling (then with IBM) made available to MPEG a document management system that allowed MPEG members to upload their documents and download those uploaded by the other members. Wo Chang (NIST) in 2000 and Christian Tulvan (Institut Mines Télécom) in 2005 have taken over the system that ensures the high efficiency of MPEG operation.

Recently ISO has developed its own document management system for use by all entities in ISO. However, MPEG has been exempted from using it because the traffic generated by uploading and downloading MPEG document in the weeks around an MPEG meeting would bring the system down. Incidentally, a requirement for hosting an MPEG meeting is the availability of internet access at 1 Gbit/s.

Meeting information

For MPEG experts (and those who do not attend yet) a place with so many “hotbeds” discussing requirements, and assessing, integrating and testing media technologies is exciting, but how can MPEG experts know what happens where and when at a meeting? Christian, the designer and maintainer of the MPEG document management system, has come to help again and designed a system that answers those questions. MPEG members can have a full view of which meetings are held where and when to discuss which topics or documents. The design is responsive so MPEG experts can get the information from their smartphones.

The figure shows a how the service looked like at MPEG 124 (October 2018). The page all meetings of all groups, but filtered views are possible. By clicking on each meeting details (room, documents etc.) can be obtained. 

Mission control

Another major development made by Christian provides the chairs with a series of functionalities to manage the MPEG workplan and timeline. Some of these, like the one below, are open to MPEG members.

The figure shows the MPEG timeline for the portion related to the MPEG-I, MPEG-CICP and MPEG-G standards. On the top there are icons to create new activities, show different table previews, provide project-wise views and more.

Another remarkable functionality is the creation of the document that collects all the results from a meeting. Group chairs enter their own results independently and the system organises them by standards for approval at the Friday plenary. This has made the review of decisions shorter and allowed MPEG to slash the time taken by plenaries.

MPEG assets

MPEG defines as assets the data associated with the development of standards: URIs, Schemas, Content, Software and Conformance testing data.

The first two are publicly available here and here.

By Content assets we mean the collection of all test data (content) used for CfEs and CfPs, and in subsequent Core Experiments. MPEG relies on the good will of companies interested in a standard to provide relevant test data. These are typically licensed for exclusive use in MPEG standardisation because they typically have high value, e.g. from the content and technology viewpoints. Content owners license their content directly to individual MPEG experts.

Software assets is the collection of all software developed or under development as reference software of MPEG standards. With the approach taken by MPEG to develop most of its standards from reference software and give it a normative status, its importance can hardly be overstated.

Conformance testing data is the collection of all data that can be used to test an implementation for conformance to an MPEG standard, published or under development. The process of developing conformance testing suites is painstaking and depends on the good will of MPEG members and their companies. On the other hand conformance testing suites are vital for the creation of ecosystems of interoperable implementations.

Content, Software and Conformance testing data for most data used in the development of MPEG standards in the last 30 years are stored in the Media Repository, Software Repository and Conformance Testing Repository hosted by Institut Mines Télécom. We are talking of a total of several TeraBytes of data.

Conclusions

This article can only give hints at the level of commitment and dedication of so many MPEG members – with the support of their companies. Of course, MPEG standards are produced by MPEG experts, but the MPEG process works because so many other people and organisations contribute, some in a visible and some others in a less visible form to make MPEG what it is.

 

Posts in this thread (in bold this post)