A future without MPEG

You are currently viewing A future without MPEG

A future without MPEG

Table of contents

  1. Introduction
  2. The situation today
  3. The competitors
  4. What is/will be available
  5. What the market offers
  6. Licensing of other media coding standards
  7. My views on the next steps of media coding standards
  8. A disclosure
  9. My forecast
  10. So, what will happen next?
  11. And now?

Introduction

My plan this week was to talk about the prospects of MPEG for the next few years. But there is a big news: MPEG passed away on 2020/06/02T16:30 CEST. The agony was long, but the result was predictable. The causes have partly to do with the nature of ISO whose main feature I describe in This is ISO – A feudal organisation.

What used to be the reference standards group driving close to 2% of the Gross World Product, i.e. 1.5 T$, and affecting the daily lives of billions of people is now reduced to splinters. The radical innovation of putting in one committee all media components, copied by most standards bodies and companies, is gone. Now, ISO media standards are treated separately. Godspeed, when someone will need to put the pieces together.

The passing away of MPEG is another hurdle added to a big hurdle on which I have published several articles: A crisis, the causes and a solution, Can MPEG overcome its Video “crisis”?, Business model based ISO/IEC standards, and IP counting or revenue counting?

After publishing those articles, I was forced to silence, but now it is time to speak again, this time loud and clear.

I need to make 3 strong statements:

  1. Those who have created a new intellectual object have to right to exploit it
  2. Technology innovation is well rewarded by a system that identifies, determines and protects intellectual property (IP)
  3. Those who make available their IP to a technical specification declared standard by an official standards body have additional benefits compared to those who do not. They should have more obligations as well.

The situation today

I will not spend much time recalling how companies who had their IP in MPEG standards have handsomely been rewarded. According to a knowledgeable person, whose statement I could not get confirmed, MPEG-2 patent holders shared revenues of 1 B$ p.a. during the validity of the MPEG-2 standard. Most IP holders have reinvested in new technologies that have feed the MPEG virtuous cycle of more than 180 standards including 6 video coding standards and 6 audio coding standards.

Unfortunately, only in fairy tales the story ends with “…and they all lived happily ever after”. MPEG did not live happily after MPEG-2 because it saw that most MPEG-2 IP holders held IP on the following (video) coding standards and had difficulty adjusting to the internet video distribution paradigm. Adding to that, the number of IP holders on HEVC has skyrocketed to ~45, 2/3 of which belong to one of the 3 existing patent pools and 1/3 belong to none. It should be no surprise that the HEVC standard has some use in broadcasting, but its use on the web is estimated to be at 12%. If one considers that broadcasting is a rich but declining market and video on the web is constantly rising, one understands that ISO standards will be gradually relegated to a more and more marginal market.

The competitors

When the MPEG-4 Visual licence sentenced the death penalty to the MPEG-4 Visual standard, competitors popped up. Four famous names are Real Networks, Microsoft, On 2 and Google. The fact that private companies could assemble a video codec of commercial value signaled that video coding was a maturing technology and that “anybody” could assemble a reasonably performing solution.

With many video coding patents reaching 20 years after filing, it was reasonable for MPEG to try and assemble a video coding specification that could be used without a licence or, if possible, with a free licence. The rationale for MPEG to do this was: do you want to let your customers be served by a competitor – and not see them back – or do you want to offer a solution that responds to the customer’s needs?

MPEG tried 3 different routes to achieve the goal

  1. Define a royalty free AVC baseline (called WebVC). Some AVC patent holders confirmed their royalty bearing patent declarations
  2. Develop a royalty free standard from scratch (called Internet Video Coding or IVC). Declarations from some companies that they held patents were received
  3. Develop a royalty free standard starting from an existing solution (called Video Coding for Browsers or VCB). A declaration from a company that it held many patents was received.

What did we learn from the 3 experiences? Of course that a company should not be forced to give away a patent for free (experience #1) but also that ISO rules allow a company to prevent a no-licence/free licence standard from happening by simply making a cautionary “I may have patents that I am willing to licence” declaration.

What is/will be available

A group of companies came up with an original proposal of making a standard that would avoid the uncontrolled flow of IP, sometimes of dubious value, with the attached burden of agreeing to a license. The proposal requested

  1. A two-layer coding scheme.
    1. The base layer must contain technology that is more than 20 years old or is accessible at no cost and provide a significant improvement on AVC.
    2. The second layer should provide significant improvement over HEVC
  2. Encouragement to declare that licencing terms will be published within two years after FDIS approval.

The Standard, called Essential Video Coding (EVC), has reached FDIS in April 2020. Final performance data are not available yet, however, it is expected that EVC base layer will be some 20% better than AVC and full EVC will be 30% better than HEVC. Licensing terms are not known yet.

In 4 weeks (July 2020) it is expected that Versatile Video Coding (VVC), the latest video coding standard, will reach FDIS.

To complete the line up of video coding standards, Low Complexity Enhancement Video Coding (LCEVC) is expected to reach FDIS. In appropriate conditions the combination of LCEVC and a generation N codec should provide a quality comparable to the quality from a generation N+1 codec.

What the market offers

The market has not been idle. In 2015 an initial group of companies (Amazon, Cisco, Google, Intel, Microsoft, Mozilla, Netflix) have created Alliance for Open Media (AOM), initially targeting video. The AOM about page mentions 4 main features of the first specification AV1:

  1. Royalty-free ecosystem
  2. Patent review process and legal defense fund
  3. Cutting edge technologies
  4. Collaborative open source development.

Licensing of other media coding standards

The information provided in this section is believed to be accurate. If you intend to make a business decision, however, please seek advice from counsel.

Video

No licensing of MPEG-1 Video is known. Still the standard was widely used.

MPEG LA developed the MPEG-2 Video licence. MPEG-2 was widely used.

MPEG-4 Visual had a hard life because there was an unsuitable licence for streaming, the main target of the standard.

MPEG-4 AVC is a very successful standard that can proudly bear the “generic” attribute because it is used for broadcasting and online streaming as well.

After 7 years, MPEG-H HEVC patent holders could not get their acts together and propose a decently unified licence. HEVC is used in broadcasting, however, use for streaming is limited at best. For years an MPEG officer tried to convince MPEG that HEVC was doing well in the market.

In 2017 I took the initiative of proposing to bring the situation to the attention of JTC 1. A document was prepared and agreed. I went to JTC 1 and presented it. However, a couple of countries objected on procedural grounds. Today, ISO is still officially unaware of the patent problem because of that opposition.

Audio

MP3 had a very simple and effective licence that allowed the standard to be widely adopted. The number of hardware and software MP3 devices is counted by the billions.

AAC has a patent pool with an effective licence as shown by the number of hardware and software AAC devices estimated to be 10 billion units.

MPEG-H 3D Audio has been around for a few years and adopted for ATSC 3.0. However, there is no known licence available. Recently a profile that is expected to contain IP just from one company has been started. There is hope that a licence for 3D Audio will be possible.

3D graphics

Past standards in this area did not have much traction. The opposite is expected for point cloud compression (PCC). The standard will become FDIS in a month time (July 2020). The number of patent holders is expected to be significantly less than HEVC.

My views on the next steps of video coding standards

The void left by HEVC has been filled by AOM with their AV1 specification which is widely used for streaming.

EVC is promising because it provides a quality that is comparable with or better than AV1, although less than VVC. EVC may have a chance if a licence will be published. However, this has not happened yet.

Most likely the number of VVC patent holders is much larger than HEVC’s. A Media Coding Industry Forum (MC-IF) was established in 2018 with the goal

“to further the adoption of MPEG Standards, initially focusing on VVC (Versatile Video Coding), by establishing them as well accepted and widely used standards for the benefit of consumers and the industry”

So far, however, no concrete results have been made known outside, if we leave aside the events organised at conferences. MC-IF has 31 members, 7 of which are licensing entities (i.e. a little less than ¼ of all members). The “industry” members account for just ½ of the HEVC patent holders. In these conditions, it is hard to believe that VVC will fare better than HEVC. It could very well fare worse because VVC adoption in broadcasting will take years, if ever.

The success of EVC could help VVC succeed. Seeing the threat of EVC (and AV1), VVC patent holders could get their act together and provide a decent licence. While I would welcome such a development, I consider it as having low likelihood.

A decent licence for LCEVC could be a game changer and another threat to VVC that could mean additional pressure on VVC patent holders to provide a single and decent licence.

A disclosure

I can finally disclose my strategy to save what used to be MPEG Video coding from disappearance:

  1. HEVC and VVC will go nowhere if left by themselves
  2. EVC will be a mezzanine standard between HEVC and VVC
  3. LCEVC on top of EVC can compete with VVC in terms of quality, in appropriate conditions
  4. If EVC will succeed, and patent holders will feel the competition and agree to a decent licence, VVC can have a chance.

Few people, if any, have understood that my goal was never to promote the success of EVC per se. By pushing EVC, I intended to create internal competition and thus promote the success of VVC. If that will not happen, loyal MPEG customers still have a chance to buy the only competitive MPEG video codec available – EVC. By adding LCEVC, they can get state of the art performance.

My forecast

My forecast is grey for the following reasons

  1. There is no longer a united MPEG, actually, MPEG does not exist anymore. That was inevitable, it was just waiting for people to do it because MPEG, with all its influence on the industry, was like a free city in the Middle Ages, ready to fall under the attack of powerful armies. As the Romance of Three Kingdoms (三國演義) said 700 years ago: 話說天下大勢．分久必合，合久必分 (in the world things divided for a long time shall unite, things united for a long time shall divide), there  will be division for a long time, say, around 30 years, the time MPEG has operated united.
  2. The putative head of SC 29 has not shown a minimum of vision for the future. This is a death knell because digital media is developing at an aggressive rate in a mature market. If ISO waits for matters to consolidate in the market, and then develop a standard, it will lose its time with the standard. MPEG has gambled for 30 years, sometimes winning, sometimes losing. What matters, however, is that the industry has not been left without standard technologies for its necessities.
  3. Most importantly, the market has come up with a solution – AV1 by the Alliance for Open Media – that is capable to provide decently performing specifications that are royalty free and promises a defence to an attacked member (with strings attached, but everybody looks at the next quarter, and does not consider the implications of a choice 5 years from now).

So, what will happen next?

If EVC/LCEVC will fly

There is a hope that VVC will also fly. Patent holders will get the expected flow of royalties and may feel assured that supporting the model still pays off and may be willing to further invest in the environment. As this, however, will only be seen in 2-3 years, patent holders will have to give the environment credit for another 2-3 years.

If EVC/LCEVC will not fly

I will consider five cases

  1. A miracle happens. VVC patent holders get their acts together and provide a decent licence. However, miracles are not very common because they indicate sainthood.
  2. NPEs/Universities/Industrial research try to keep the environment alive by pretending that industry will use the “standards” produced. This will last for some time until higher-level  management realise they are wasting time and money.
  3. The environment becomes a special “conference” (not a standards meeting) assessing “papers” (not contributions) to produce “documents” (not standards). The value of a “document” is in the “technology screening process” that assesses the “value” of technologies, so that those who want to make a specification (e.g. AOM) know that they are good screened technologies. This may save the NPE business model.
  4. The environment is in charge of maintaining the large number of standards. This may become difficult because the environment will not be attractive enough to experts who are ready to perform their duty maintaining old standards in exchange for some fun in new standards (that will no longer be there).
  5. The environment becomes the place where Publicly Available Specifications (PAS) suppliers, e.g. AOM, send their specifications to be converted to ISO standards. This will be the final sanction of the end of The MPEG exception. I hope not to be in this world anymore to witness such a degradation if that happens .

And now

Something is going to happen…

See the Russian translation Будущее без MPEG

Posts in this thread

Posts in the previous thread

List of all related articles