Day: January 4, 2020

The driver of future MPEG standards

Introduction

It is not by chance that the MPEG Future Manifesto has the following action points at the top of its list (italics are mine)

  • Support and expand the academic and research community which provides the life blood of MPEG standards;
  • Enhance the value of intellectual property that make MPEG standards unique while facilitating their use.

Why are these two action points so important? Because the main ingredients of the successful MPEG recipe have been so far

  1. Developing the best performing standards by accessing the best technologies produced by academia and research;
  2. Augmenting the value of the intellectual property provided, thanks to its adoption in standards deployed to billions of users;
  3. Relying on facilitated access to essential patents required to implement MPEG standards.

So as to avoid any misunderstanding, I should explicitly state that MPEG decisions only affect item 1. of the list above because MPEG decisions are made solely on the basis of the technical merits of the technologies proposed and adopted after a scrutiny that follows an established process.

Item 2. is a just consequence of decisions made by MPEG. The “just” hides the fierce discussions that happen when a decision to accept or reject a proposed technology is made.

Item 3. is entirely in the hands of the market. Users wishing to acquire the right to use the essential patent of the standards may decide to use the services offered by patent pools or proceed in other ways.

In a string of articles A crisis, the causes and a solution, Can MPEG overcome its Video “crisis”?, Business model based ISO/IEC standards, IP counting or revenue counting? and Matching technology supply with demand I have

  1. Analysed the changes that have affected the ideal model of the numbered list above by the past quarter-of-a-century application;
  2. Highlighted the risks on the adequacy of the model entailed by the progressive change of the environment;
  3. Ventured to propose ideas for improving the situation.

The very fact that myarticles have generated significant controversy and that there is a continuous flow of readers of the articles, long time after they have been published, implies that indeed there is one (or more) issue with the model as it is applied today.

In this article I intend to review the situation trying to identify the issues that MPEG Future could address in implementing its Manifesto.

How MPEG works

The figure below claims to represent with some accuracy the way MPEG acquires the necessary technologies to develop a standard and how implementors of a standard can acquire the necessary rights.

When MPEG intends to develop a standard, it solicits requirements from the industry. Three segments of the industry can typically provide requirements: eventual users of the standard, implementers of the standard and providers of technology to develop the standard. Note that a specific company may belong to one or more than one industry, in the sense that one department of a company may belong to one industry and another department may belong to a different one.

MPEG receives (step 1)), assesses, refines and harmonises the requirements received. When these have reached sufficient maturity, a Call for Proposals (CfP) is issued (step 2). Companies belonging to the technology industry submit Proposals in response to the CfP (step 3). MPEG assesses the coding tools contained in the proposals and assigns them to specific Competence Centres, say, audio, video, 3D graphics, systems etc. (step 4) who adapt and refine the tools (possibly interacting between them) and adds the selected ones to those in the MPEG Toolkit (step 5) where all the technologies adopted in past standards are contained.

When the development of the standard is completed (step 6), companies claiming rights to the tools included in the standard may decide to join one or more than one patent pool or not to join any patent pool at all. Users of the standard (e.g. implementers and service providers) get use rights from patent holders (step 7) for use in their products, services or applications (step 8).

Albeit MPEG has no role in the last step, the perceived performance of MPEG as a provider of industry standards is highly dependent on how this step unfolds.

The troubles with the situation

What has been described is the result of a natural evolution over the past quarter of a century. In MPEG-2 and MPEG-4 AVC times there was a single patent pool with a limited number of companies not joining the pool. Today, we have the example of MPEG-H HEVC for which there are ~45 known patent holders, 3 known patent pools and a number of companies who have not joined any patent pool. Patent pool members make up ~ 2/3 of patent holders, while ~ 1/3 does not belong to any patent pool.

An assessment

There is no consensus in the industry that the situation described is problematic. Roughly speaking, technology companies say that the situation is just fine, while the client and implementation companies usually complain that getting a licence is “cumbersome”, “difficult” or ”expensive”.

Save for MPEG-1 or MPEG-2 Video, MPEG Video standards always had a competition. Some of those who intended to use the standard for internet streaming complained they were discouraged from using MPEG-4 Visual by some licence terms. Many turned to Real Network’s Real Video and Microsoft’s Windows Media Video. MPEG-4 AVC had competition from Google’s VP8 and VP9.

An industry forum called Alliance for Open Media (AOM) has released the specification of a video coding standard called AV1, that I call, but maybe I should not, “royalty free”. AV1 has an improved performance compared to HEVC but is less performing that Versatile Video Coding (VVC). Use of AV1 is getting significant traction in the industry for applications that used to be the purview of MPEG standards.

MPEG reacts

In the last 10 years, MPEG has tried to provide accessible solutions with its three projects for Option 1 video coding standards (Option 1 could, but should not, be considered as equivalent to “royalty free”).

All three projects have pracyically failed:

  • WebVC, the “baseline” profile of AVC, was dead at birth because some patent holders confirmed the Option 2 patent declaration (“my patents can be used, but at RAND conditions”) they had made for AVC.
  • Internet Video coding (IVC) is a published ISO standard with an attractive performance (comparable to AVC). However, it has the Damocles sword of some Option 2 declarations that do not provide any detail on what the infringing technologies are claimed to be. MPEG has declared its willingness to amend the IVC standard if appropriate information will be provided, but so far no information has been received.
  • Video Coding for Browsers (VCB) did not even reach publication because a company made an Option 3 patent declaration (“my patents may not be used for the standard”). Today it would no longer be possible to do so without identifying the infringed technologies. However, at the time the standard was developed, this was possible and MPEG could not remove the infringing technologies that had to be found in an initial list of 50 or so patents with hundreds of claims.

MPEG is currently developing MPEG-5 part 1 Essential Video Coding (EVC). The EVC CfP requested two-layer solutions where each layer had a nominal performance target with respect to AVC and HEVC. This helped the formation of a draft standard, to be promoted to FDIS in April 2020, that is expected to have a significant less complex IP context. The constarint of the CfP, however, is paid by lesser performance than VVC, a standard to be approved as an FDIS in July 2020. What will be the performance cost of the project design is not known today as verification tests for EVC and VVC have not been done yet.

Is this enough?

I cannot claim that what MPEG has done is the solution to what some perceive as a problem. The MPEG Option 1 video coding standards, for one reason or another, have failed. Without additional changes to the rules, it is hard to see how MPEG can offer an Option 1 Video Coding standard (but probably any other internally-developed media-related standard).

There is reason to believe that a standard with more performance than AV1, even if it has less performance than VVC but has a simplified access to the necessary IP will be a competitive proposal, but the jury of the EVC standard is still out and AOM may issue a more performing AV2 specification

The evolutions described above have not reached the endpoint, but MPEG needs a stable and shared approach to standardisation in an increasingly sophisticated IP environment. This is why MPEG Future has put this topic at the top of its priorities.

Conclusions

I think that MPEG Future should

  1. Make an objective analysis of the “difficulties” the current situation creates to users of MPEG standards;
  2. In case the existence of “difficulties” is confirmed, review the current rules and identify how they can be changed to achieve measurable results (rules have been changed once, so they could be changed a second time);
  3. Act to effect the identified changes to the rules.

Caius Julius Caesar said “Caesar’s wife must be above suspicions”.

Caius Julius Caesar said “Caesar’s wife must be above suspicions”. The guide to carry out the steps should not deny a priori the existence of a problem and not be suspected to be influenced by “the competition”.

Posts in this thread