How to make standards adopted by industry

Introduction

There are many definitions of standard. In the Webster’s you find a definition of standard as “Something that is established by authority, custom or general consent as a model or example to be followed”, an oldish definition that thinks that people must be directed to their good. In the Encyclopaedia Britannica you find “(A technical specification) that permits large production runs of component parts that are readily fitted to other parts without adjustment”, a definition driven by the idea that manufacturing is helped by the availability of different but compatible suppliers. Closer to my view of standard is another Webster’s definition “a conspicuous object (as a banner) formerly carried at the top of a pole and used to mark a rallying point especially in battle or to serve as an emblem” driven by the idea that everybody can develop a standard but its adoption depends on how satisfactory the proposed standard is to its intended users.

In many cases a standard is the result of the effort spent by a group of people who believe their interests are best served by agreeing to do certain things in an certain way. Agreeing on a standard may require a big effort (in MPEG developing a standard may cost tens of millions of USD to participating companies), but that is nothing compared to the effort required by convincing “other people” that the standard is what they need.

In this article I will present some of the efforts that MPEG has done over its 30+ years to convince “other people” that MPEG standards are what they need.

Convincing other people to adopt a standard is a process

If you think that convincing other industries that, when an MPEG standard, you need a good a marketing effort to get it adopted, you are missing the point. Anybody can put together a decent technical standard. Convincing other industries is a process that accompanies the development of the standard, starting from the moment the idea of a new standard takes shape.

In the early 1990’s all instances of the broadcasting industry – terrestrial, satellite and cable – were technically convinced that digital television was superior to analogue television. There were two problems, however. The first problem was that in some countries the industry espoused digital as an ally why in other countries the industry rejected it as a threat. The second problem was that there were solutions here and there and some attempts at developing standards, but solutions were proprietary and attempts at standards often fraught with rivalries. MPEG had achieved some notoriety with its first (MPEG-1) standard but had to acquire a new credibility vis-à-vis an industry that, already at that time, was worth more that 100 B$ p.a. and was understandably cautious with its decisions.

MPEG succeeded to convince the broadcasting industry, even the reluctant segments of it, namely the European terrestrial broadcasting industry. The deal was to offer its Requirements group as the place where the individual industry segments could express their needs and see them influence the technical developments. Unlike the approach of other bodies where often there is a coalition of interests blocking the requests from other groups based on the mantra “I cannot support this because my business is negatively affected”, MPEG took the opposite approach. All requests were discussed to understand whether they were new or could be folded in previous requests. The space of technical solutions was partitioned in profiles and levels to accommodate requests without negatively affecting others. Finally, when the MPEG-2 standards was completed, MPEG carried out Verification Tests and showed that 6 Mbit/s yielded “composite quality” of standard definition TV and 8 Mbit/s yielded “component quality”.

Credibility is not granted for ever

In the mid-1990’s MPEG had achieved the impossible. It had brought together all segments of the television industry, the package media industry included, and was addressing the studio needs that it satisfied with its 4:2:2 profile. MPEG, however, did not intend to be just the technical arm of the television industry (which, by the way, included audio as well). MPEG intended to fully execute the mission implied by its title “Coding of moving pictures and audio” which meant the “information representation layer” for whatever application domain.

In the mid-1990’s the role that internet would play in the media distribution media was not clear at all and so was the role that mobile networks would play. It was clear, however, that other delivery mechanisms would play a role. These mechanisms were characterised by “low bitrate”, “best effort” etc.

In hindsight trying to extend the hard-won role in the broadcasting industry to this unknown land was a very bold move. That field was antithetic to what MPEG had done so far, namely high quality and guaranteed (to some extent) delivery. Emblematic was the hot discussion around the MPEG-2 transport packetisation that was opposed by old style experts accustomed to rely on frame structure. More important was the fact that new industries, represented by the ICT (Information and Communication Technologies) acronym would play a major role.

MPEG made a big effort to adapt to the new environment. For instance it developed the software copyright disclaimer. The disclaimer eventually became a modified BSD – Berkeley Software Distribution licence, where the modification is contained in an explicit disclaimer that software copyright release does not imply release of patents. Another effort was to develop the file format which became the cornerstone on which the MPEG role in the ICT world was built.

A track record of collaborations

In 30+ years of standards development MPEG has established cooperation with many standards bodies and industry fora. In this chapter I will review some of the most outstanding and fruitful collaborations.

Broadcasting

MPEG has developed standards for broadcasting since its early days (DAB – Digital Audio Broadcasting was one application driving MPEG-1 Audio), Broadcasting continues to be a major customer to this dau. An indicative list of standards groups and industry fora MPEG interacts with is ABU – Asia-Pacific Broadcasting Union, ATSC – Advanced Television Systems Committee, Inc., DVB – Digital Video Broadcasting, EBU – European Broadcasting Union, ITU-R SG 6 – Broadcasting Service (terrestrial and satellite), ITU-T SG 9 – Television and sound transmission, SCTE – Society of Cable Telecommunications Engineers and DTG – Digital TV Group.

ATSC has adopted MPEG-2 Video, AVC and HEVC. In addition to these standards, DVB has also adopted MPEG-1 and MPEG-2 Audio and AAC. MPEG has referenced a DVB specification for its Media Orchestration standard (MPEG-B part 13). ITU-R and SCTE have adopted several MPEG standards.

Telecommunication

MPEG-1 was driven by the idea of interactive audio-visual services at a bitrate that telcos used to call as primary rate (1.5/2 Mbit/s) that were expected to be offered by ADSL – Asymmetric Digital Subscriber Line. Intense interaction with that industry began with MPEG-2 Video which is common text, which means that MPEG-2 Video is verbatim the same as H.222 and H.262. The tight collaboration of MPEG with ITU-T SG 16 – Multimedia services and systems continued with AVC and HEVC, and continues with VVC. The 3 standards are “aligned text” which means that the standards are technically equivalent but not editorially the same. Other related standard such as MPEG-C Part 7 – Supplemental enhancement information messages for coded video bitstreams, and MPEG-CICP Part 2 – Video and Part 4 – Usage of video signal type code points are also aligned text. MPEG is liaising with ITU-T SG 12 – Performance, QoS and QoE

MPEG has an ongoing intense collaboration with 3GPP – the Third Generation Partnership Project, an international organisation issuing standard for the mobile industry. 3GPP has adopted many MPEG standards such as AVC, HEVC, AAC, MP4 File Formay and DASH. MPEG is also liaising with ETSI – European Telecommunication Standards Institute.

Other media-related areas

The world of media is quite articulated and MPEG takes care of establishing contacts, developing standards for or using standards from different environments.

In the area of audio, MPEG has a long-standing liaison with AES – Audio Engineering Society and with SMPTE. MPEG is referencing several SMPTE standards, e.g. those related to HDR – High Dynamic Range.

AVS – Audio and Video Coding Standard Workgroup of China is a group developing audio-visual compression standards for the Chinese market. MPEG has a liaison with AVS.

Immersive media is the future but it is unclear what the future will exactly be. MPEG has developed ARAF – Augmented Reality Application Format and has developed ISO/IEC 21858 – Information model for mixed and augmented reality (MAR) contents jointly with SC 24 – Computer graphics, image processing and environmental data representation.

Fonts

Since the early 2000, MPEG has taken over the baton of the Open Type specification. Open Type was an open specification originally developed by Adobe, Apple and Microsoft. MPEG-OFF – Open Font Format is a standard that is universa;;y used wherever there are displays that are exprected to present fonts.

MPEG is liaising with SC 34 – Document Description and Processing Languages on the matter of fonts.

Information transport

When it developed MPEG-2 Video, MPEG had already the experience of the transport standard that it had developed for MPEG-1. MPEG-2 broadcasting applications could not rely on the assumption that the communication channel was error-free and MPEG had to develop a new standard that it called MPEG-2 Transport Stream (MPEG-2 Systems also defines another transport called MPEG-2 Program Stream aking to MPEG-1 Systems). MPEG-2 Systems is one of the most successful MPEG standards as it is used by broadcasting in all forms (ATSC, DVB, BDA and, before that DVD – Digital Versatile Disc etc.) and is used as a package in IPTV.

ATSC has adopted the full Audio-Video-Systems package offered by MPEG-H. The MPEG-H Systems layer is called MPEG Media Transport (MMT).

The Common Media Application Format (CMAF) is another successful transport standard.

transport standardMPEG has developed the MPEG-2 transport standard for sequences of JPEG 2000 and JPEG XS images.

Another successful media transport standard is MPEG-DASH. This has been adopted by 3GPP, ATSC, DVB and others.

Manufacturing industry

In addition to DAB, MPEG-1 was driven by the idea of a standard for audio-visual applications on CD – compact disc. This eventually was adopted by the industry under the name of Video CD. It was also driven by the idea of a new digital audio distribution format on CC – compact cassette. The CE – Consumer Electronic industry has tight contacts with MPEG via IEC TC 100 –  Audio, Video and Multimedia Systems and Equipment. MPEG is also working with CTA – Consumer Technology Association. MPEG liaises with BDA – Blu-ray Disc Association and BDA has adopted several MPEG standards such as AVC and HEVC.

Genomics

MPEG has developed the 3 parts of MPEG-G – Genomic Information Representation jointly with WG 5 – Data processing and integration of TC 276 – Biotechnology. It is at the last stages of  approval of Parts 4 and 5 and is developing, again jointly with TC 276/WG 5, Part 6 – Genomic Annotation Representation.

In addition to TC 276, MPEG is liaising with TC 215 – Health Informatics and with GA4GH –Global Alliance for Genomics and Health.

Internet of Things

Internet of Things per se is no business for MPEG because SC 41 – Internet of Things is in charge of standardisation in this area. MPEG liaises with SC 41.

MPEG has identified a specific instance of Internet of Things that it calls Internet of Media Things. This considers the specific but important case of a thing that is a camera or a microphone, a display or a loudspeaker, a unit capable of analysing the media content etc. Part 1 of MPEG-IoMT – Architecture is an instance of the general IoT Architecture developed by SC 41.

Artificial Intelligence

Artificial Intelligence per se is no business for MPEG because SC 42 – Artificial Intelligence is in charge of it. MPEG liaises with SC 41.

MPEG has used an AI technology – Neural Networks – for MPEG-7 Part 15 – Compact Descriptors for Video Analysis (CDVA), is working on MPEG-NNR, Part 17 of MPEG-7 – Compression of neural networks for multimedia content description and analysis. MPEG also plans to make intense use of neural networks for its future Video Coding for Machines standard.

MPEG is also investigating the connections between its Network-Based Media Processing (NBMP) standard, released as FDIS at the January 2020 meeting, and Big Media. Again, MPEG has no business in Big Media, an area of work for SC 42, but NBMP is likely to become an instance of the general Big Media Reference Model developed by SC 42.

Transportation

Data compression seems to have little to do with transportation, but this area of endeavour is more and more influenced by technologies mastered by MPEG. For instance, 3GPP is considering V2X (Vehicle-to-everything) communication, where information moves from a vehicle to any entity that may have a relationship with the vehicle, and vice versa. Specific forms of communication are: V2I (vehicle-to-infrastructure), V2N (vehicle-to-network), V2V (vehicle-to-vehicle), V2P (vehicle-to-pedestrian), V2D (vehicle-to-device) and V2G (vehicle-to-grid).

Audio-visual information is clearly a major user of any such communication forms and MPEG standards are and will be more and more the main sources. Two examples are G-PCC – Geometry-based Point Cloud Compression and VCM – Video Coding for Machines.

MPEG is liaising with ISO TC 22 – Road Vehicles and TC 204 – Intelligent Transport Systems.

Conclusions

Standards lubricates our complex society and allo it to function and make progress. Developing standards is easy, but making sure that standards are adopted is difficult.

MPEG has been successful with the latter because it takes a holistic, end-to-ed approach to standardisation where its partners and customers – standards bodies and industry fora – are parts of standard development.

Posts in this thread

 

MPEG status report (Jan 2020)

Introduction

In the week of the 13th of January, the Free University of Brussels has hosted the 129th MPEG meeting . Two days (11-12) were dedicated to some 15 ad hoc group meetings and 6 days (7-12)to meetings of JVET, the joint MPEG-SG 16 group tasked to develop the VVC standard.

In this status report I will highlight some of the most relevant topics on which progress was made. The figure below captures the essence of the MPEG work plan as it resulted from the meeting.

Versatile Video Coding (VVC)

VVC (part 3 of MPEG-I) is being balloted and the ballot results are expected to be received at the July meeting so that MPEG can approve VVC as FDIS.

MPEG is now working on two related standards that are important for practical deployment: Carriage in MPEG-2 TS (Amendment 2 of MPEG-2 Systems) and Carriage in ISOBMFF (Amendment 2 of MPEG-4 part 15), both expected to be approved in January 2021.

Another activity around VVC is called Multi-Decoder Video Interface for Immersive Media (part 13 of MPEG-I). This aims to support the flexible use of media decoders, for example decoding only a subset of a single elementary stream. This feature is required for processing immersive media composed of a large number of elementary streams.

Essential Video Coding (EVC)

EVC (part 1 of MPEG-5) addresses the needs that have become apparent in some use cases, such as video streaming, where existing ISO video coding standards have not been as widely adopted as might be expected from their purely technical characteristics. EVC is still under ballot and results are expected to become available at the April 2020 meeting (MPEG 130).

The group in charge of EVC has started considering Carriage of EVC in MPEG Systems.

Low Complexity Enhancement Video Coding (LCEVC)

LCEVC will provide a standardised video coding solution that leverages other video codecs in a manner that improves video compression efficiency while maintaining or lowering the overall encoding and decoding complexity. LCEVC will reach DIS in April 2020.

MPEG Immersive Video (MIV) and Video-based Point Cloud Compression (V-PCC)

Part 12 of MPEG-I Immersive Video shares with Part 5 of MPEG-I Video-based Point Cloud Coding (V-PCC) the notion of projecting a 3D scene to a series of planes, compressing the 2-D visual information on the planes with off-the-shelf video compression standards and providing a means to communicate how a 3D renderer can use the information contained in the atlases (in the case of MIV) and the patches (in the case of PCC). Outstanding convergence of the two approaches has been reached.

V-PCC will reach FDIS in April 2020 and MIV in January 2021. Both will have extensions, the latter to enable the ambitious, but needed 6 degrees of freedom (6DoF) where user can move in 6 directions.

The MPEG-4 File Format is being extended to include V-PCC and G-PCC data.

Video Coding for Machines (VCM)

VCM is an exploration on a new type of video coding designed to provide efficient representation of video information where the user is not human but a machine, with possible support of viewing by humans. Possible use cases for VCM are video surveillance, intelligent transportation, automatic driving and smart cities.

MPEG has produced a Draft Call for Evidence designed to acquire information on the feasibility of a Video Coding for Machines standard. For this purpose MPEG has published a Call for Test Data for Video Coding for Machines. Test data will be used to assess the responses to the Call for Evidence.

Neural Network-based Audio-Visual Compression

VVC and EVC will support the media industry by providing more compression for transmission and storage. They are both the current endpoints of a compression scheme that dates back to the mid-1980’s. Similarly MPEG-H 3D Audio is the current endpoint of the compression scheme initiated in 1997 with MPEG-2 AAC.

Today, as a result of the demonstration provided in recent years that neural networks can outperform other “traditional” algorithms in selected areas, many laboratories are carrying out significant research on the use of neural networks for coding of audio and visual signals as well as point clouds.

MPEG is calling its members to provide information on this new area of endeavour.

MPEG Immersive Audio

MPEG has produced a Draft CfP for Immersive Audio. The actual CfP will be issued in April 2020 and submissions are requested for July 2020. FDIS is planned for January 2022.

Neural Network Compression for Multimedia

Neyral Networks are used for muktimedia applications, such as speech understanding and image recognition. Industry, however, is coming to the conclusion that the IT infrastructure can very well not be able to cope with the growth of users and that in many cases intelligence is best distributed to the edge. As the size of some of these networks is hundreds of GBytes or even TBytes, compression of neural networks can support distribution of intelligemce to potentially millions of devices. See the figure below for a view of NBMP.

MPEG is progressing its work on the Compression of neural networks for multimedia content description and analysis standard. This is expected to reach CD status in January 2021.

Network-Based Media Processing (NBMP)

NBMP reached FDIS in January 2020. The standard defines a framework for content and service providers to describe, deploy, and control media processing. The framework includes an abstraction layer deployable on top of existing commercial cloud platforms and able to be integrated with 5G core and edge computing. The NBMP workflow manager enables composition of multiple media processing tasks to process incoming media and metadata from a media source and to produce processed media streams and metadata that are ready for distribution to media sinks.

MPEG is exploring how NBMP can become an instance of the Big Media reference model developed by SC 42 Artificial Intelligence.

Compression of Genomic Annotations

At the January 2020 meeting MPEG received 7 submissions in response to the joint Call for Proposals that MPEG and ISO TC 276/WG 5 on the efficient representation of annotations to sequencing data resulting from the analysis pipelines MPEG meeting in Brussels. MPEG has started working on a set of core experiments with the goal to integrate the proposed technologies into a single standard specification capable of satisfying all identified requirements and support rich varieties of queries.

FDIS is expected to be reached in January 2021.

MPEG and 5G

MPEG compression standards are mostly designed to represent information in an abstract way. However, the great success of MPEG standards is also due to the effort MPEG spent in providing the means to convey compressed information. 5G is being deployed and MPEG is investigating if and how its standards can be affected by 5G/

MPEG-21 Contracts to Smart Contracts

Blockchains offer an attractive way to execute electronic contracts. The problem is that there are many blockchains each with their own way of expressing the terms of a contract. MPEG considers that MPEG-21 can be the intermediate language in which smart contracts for different blockchains can be expressed.

One application can be for the following use case. There is no way to deduce from a smart contract the clauses that the smart contract contains. Publishing the human readable contract alleviates the concern, but does not ensure that the clauses of the human readable contract correspond to the clauses of the smart contract.

The figure below describes how the other party of the smart contract can know the clauses of the smart contract in a human readable form.

Posts in this thread

 

MPEG, 5 years from now

MPEG will soon be 32. It has produced many important standards that have changed the industry and the lives of billions of people. What will MPEG be 5 years from now?

MPEG is strong

The MPEG scope is digital media: analysis, compression, transport and consumption of digital moving pictures and audio for broadcast, broadband, mobile and physical distribution. MPEG standards have an extremely wide range of customers belonging to all industries that need digital audio and video packaged for their needs. Because of this, the MPEG brand is universally recognised, even by the public at large.

It should also be noted that MPEG has started applying its expertise to compression and transport of other data. MPEG has started working on compression of DNA reads from high speed sequencing machines some 5 years ago. The field is very active and there is no doubt that in five years it will continue to be so.

Another strength is the MPEG organisation, a flat structure based on interacting units belonging to technology-centred competence centres coordinated by a Technical Coordination Committee. It is the result of a 30-year long natural process driven by the specific needs of the group. MPEG has a large and competent membership with a full ecosystem of experts contributing to the development of technologies needed by MPEG standards. It also has a consolidated and experienced leadership team.

A third strength is the vast network of client communities, typically represented by their standards organisations and/or industry fora. This network is the result of an effective communication policy that includes liaisons letters, press communiqués released at every meeting documenting the progress of work, the MPEG website providing general information, workshop on existing or planned standards, white papers describing the purpose of approved standards, video tutorials and more.

A fourth strength is the MPEG business model. MPEG develops the best standards using the best available technologies provided by industry and academia. Good standards remunerate good IP whose holders may choose to use the revenues to develop more good IP for future standards.

MPEG has weaknesses of its own

At a speech I gave at the 100th MPEG meeting held in Geneva in 2012, I said that the MPEG area of work was so wide, so important for humans and with so many opportunities that, if in 100 years MPEG did not exist any longer, it would not be because the MPEG mission had been exhausted, but because the MPEG leaders of the time had not been able to manage the work of the group properly.

I confirm what I said, but it is a reality that digital media is a maturing business area. The most critical needs, e.g. digital television and media on mobile, have been satisfied. More needs exist but with many unknowns. MPEG is populated by excellent experts and, in the past, their expertise could cope with the need to convert to digital or mobile experiences which users were already accustomed to. Today, however, we are facing the problem that available and possible technologies may be used to offer new products, services or applications. Unfortunately we do not know what are the new products, services or applications using those technologies.

A good example is provided by the 27 year old MP3 standard. It took several years before the excellent MP3 technology found the use we all know. Other MPEG standards have not been so lucky.

The weakness is then that MPEG is a technical group almost without participation of people who can contribute the information that, merged with the existing technical information, can guide the group to develop the right standards.

MPEG has become the reference group for digital media compression and transport. As I have written several times, MPEG standards enable products, services and applications that are worth more than 1.5 T$ p.a. MPEG – a a working group – has achieved such an importance when there are Technical Committees whose economic importance is orders of magnitude less than that of the MPEG working group. So far this did not make much difference because MPEG’s client industries were keen on getting good technologies for their affiliates. In the new competitive environment the MPEG lack of status matters.

Finally MPEG suffers from its very success. Its business model has been the engine of MPEG success, but its unreviewed business model matters and more.

MPEG is threatened

The fact that digital media is maturing has creates another weakness: there is increasing competition in digital media standards. MPEG standards remain the best, but there are other groups developing competing standards with features that are at odds with the MPEG business model. Because of this the share of the client industry adopting MPEG standards is no longer as high as it used to be.

So far MPEG standards have dominated the broadcasting and mobile domains and had to face competition in the “internet” domain. MPEG’s dominance is threatened by the fact that some industries may abandon MPEG standards not because they are best but because they cannot compete on other features that other groups provide. MPEG standards may become less relevant if MPEG stays exclusively with its traditional business model.

MPEG standards are synonymous of globalisation. The established broadcast business was local by definition, but MPEG standards provided an enabling compression technology ready for global adoption. The mobile industry started as local but was easily conquered by the benefits offered by the global MPEG standards. The threat today is that globalisation may no longer be MPEG’s ally because in the new environment the global MPEG brand may be replaced by some local brands.

Opportunities

Borrowing from and extending the actions identified by the MPEG Future Manifesto, MPEG has the following opportunities

  1. Enhance its links with the research and academic community to stimulate more research on technologies that are relevant to MPEG’s standardisation scope
  2. Review its business model, analyse its continuing effectiveness and relevance to industry, identify the need for changes and, if any identified, take action to implement them
  3. Step up communication with its client industries to identify any early needs for standards and obtain requirements
  4. Inject “market-awareness” into MPEG to sharpen the target of MPEG standards
  5. Enhance the value of MPEG standards by strengthening and expanding collaboration with other standards committees, e.g. by developing joint standards
  6. Increase promotion of MPEG standards with its client industries
  7. Dedicate sufficient resources to cover other areas in need of compressions by leveraging MPEG technologies and expertise
  8. Preserve and refine the organisation of MPEG.

Conclusions

Barring exogenous forces, in 5 years MPEG will still be around. However, the next 5 years will not be an easy ride. This does not mean that MPEG will not be able to deliver the results that its customers – industry and end users – expect.

Posts in this thread

 

The driver of future MPEG standards

Introduction

It is not by chance that the MPEG Future Manifesto has the following action points at the top of its list (italics are mine)

  • Support and expand the academic and research community which provides the life blood of MPEG standards;
  • Enhance the value of intellectual property that make MPEG standards unique while facilitating their use.

Why are these two action points so important? Because the main ingredients of the successful MPEG recipe have been so far

  1. Developing the best performing standards by accessing the best technologies produced by academia and research;
  2. Augmenting the value of the intellectual property provided, thanks to its adoption in standards deployed to billions of users;
  3. Relying on facilitated access to essential patents required to implement MPEG standards.

So as to avoid any misunderstanding, I should explicitly state that MPEG decisions only affect item 1. of the list above because MPEG decisions are made solely on the basis of the technical merits of the technologies proposed and adopted after a scrutiny that follows an established process.

Item 2. is a just consequence of decisions made by MPEG. The “just” hides the fierce discussions that happen when a decision to accept or reject a proposed technology is made.

Item 3. is entirely in the hands of the market. Users wishing to acquire the right to use the essential patent of the standards may decide to use the services offered by patent pools or proceed in other ways.

In a string of articles A crisis, the causes and a solution, Can MPEG overcome its Video “crisis”?, Business model based ISO/IEC standards, IP counting or revenue counting? and Matching technology supply with demand I have

  1. Analysed the changes that have affected the ideal model of the numbered list above by the past quarter-of-a-century application;
  2. Highlighted the risks on the adequacy of the model entailed by the progressive change of the environment;
  3. Ventured to propose ideas for improving the situation.

The very fact that myarticles have generated significant controversy and that there is a continuous flow of readers of the articles, long time after they have been published, implies that indeed there is one (or more) issue with the model as it is applied today.

In this article I intend to review the situation trying to identify the issues that MPEG Future could address in implementing its Manifesto.

How MPEG works

The figure below claims to represent with some accuracy the way MPEG acquires the necessary technologies to develop a standard and how implementors of a standard can acquire the necessary rights.

When MPEG intends to develop a standard, it solicits requirements from the industry. Three segments of the industry can typically provide requirements: eventual users of the standard, implementers of the standard and providers of technology to develop the standard. Note that a specific company may belong to one or more than one industry, in the sense that one department of a company may belong to one industry and another department may belong to a different one.

MPEG receives (step 1)), assesses, refines and harmonises the requirements received. When these have reached sufficient maturity, a Call for Proposals (CfP) is issued (step 2). Companies belonging to the technology industry submit Proposals in response to the CfP (step 3). MPEG assesses the coding tools contained in the proposals and assigns them to specific Competence Centres, say, audio, video, 3D graphics, systems etc. (step 4) who adapt and refine the tools (possibly interacting between them) and adds the selected ones to those in the MPEG Toolkit (step 5) where all the technologies adopted in past standards are contained.

When the development of the standard is completed (step 6), companies claiming rights to the tools included in the standard may decide to join one or more than one patent pool or not to join any patent pool at all. Users of the standard (e.g. implementers and service providers) get use rights from patent holders (step 7) for use in their products, services or applications (step 8).

Albeit MPEG has no role in the last step, the perceived performance of MPEG as a provider of industry standards is highly dependent on how this step unfolds.

The troubles with the situation

What has been described is the result of a natural evolution over the past quarter of a century. In MPEG-2 and MPEG-4 AVC times there was a single patent pool with a limited number of companies not joining the pool. Today, we have the example of MPEG-H HEVC for which there are ~45 known patent holders, 3 known patent pools and a number of companies who have not joined any patent pool. Patent pool members make up ~ 2/3 of patent holders, while ~ 1/3 does not belong to any patent pool.

An assessment

There is no consensus in the industry that the situation described is problematic. Roughly speaking, technology companies say that the situation is just fine, while the client and implementation companies usually complain that getting a licence is “cumbersome”, “difficult” or ”expensive”.

Save for MPEG-1 or MPEG-2 Video, MPEG Video standards always had a competition. Some of those who intended to use the standard for internet streaming complained they were discouraged from using MPEG-4 Visual by some licence terms. Many turned to Real Network’s Real Video and Microsoft’s Windows Media Video. MPEG-4 AVC had competition from Google’s VP8 and VP9.

An industry forum called Alliance for Open Media (AOM) has released the specification of a video coding standard called AV1, that I call, but maybe I should not, “royalty free”. AV1 has an improved performance compared to HEVC but is less performing that Versatile Video Coding (VVC). Use of AV1 is getting significant traction in the industry for applications that used to be the purview of MPEG standards.

MPEG reacts

In the last 10 years, MPEG has tried to provide accessible solutions with its three projects for Option 1 video coding standards (Option 1 could, but should not, be considered as equivalent to “royalty free”).

All three projects have pracyically failed:

  • WebVC, the “baseline” profile of AVC, was dead at birth because some patent holders confirmed the Option 2 patent declaration (“my patents can be used, but at RAND conditions”) they had made for AVC.
  • Internet Video coding (IVC) is a published ISO standard with an attractive performance (comparable to AVC). However, it has the Damocles sword of some Option 2 declarations that do not provide any detail on what the infringing technologies are claimed to be. MPEG has declared its willingness to amend the IVC standard if appropriate information will be provided, but so far no information has been received.
  • Video Coding for Browsers (VCB) did not even reach publication because a company made an Option 3 patent declaration (“my patents may not be used for the standard”). Today it would no longer be possible to do so without identifying the infringed technologies. However, at the time the standard was developed, this was possible and MPEG could not remove the infringing technologies that had to be found in an initial list of 50 or so patents with hundreds of claims.

MPEG is currently developing MPEG-5 part 1 Essential Video Coding (EVC). The EVC CfP requested two-layer solutions where each layer had a nominal performance target with respect to AVC and HEVC. This helped the formation of a draft standard, to be promoted to FDIS in April 2020, that is expected to have a significant less complex IP context. The constarint of the CfP, however, is paid by lesser performance than VVC, a standard to be approved as an FDIS in July 2020. What will be the performance cost of the project design is not known today as verification tests for EVC and VVC have not been done yet.

Is this enough?

I cannot claim that what MPEG has done is the solution to what some perceive as a problem. The MPEG Option 1 video coding standards, for one reason or another, have failed. Without additional changes to the rules, it is hard to see how MPEG can offer an Option 1 Video Coding standard (but probably any other internally-developed media-related standard).

There is reason to believe that a standard with more performance than AV1, even if it has less performance than VVC but has a simplified access to the necessary IP will be a competitive proposal, but the jury of the EVC standard is still out and AOM may issue a more performing AV2 specification

The evolutions described above have not reached the endpoint, but MPEG needs a stable and shared approach to standardisation in an increasingly sophisticated IP environment. This is why MPEG Future has put this topic at the top of its priorities.

Conclusions

I think that MPEG Future should

  1. Make an objective analysis of the “difficulties” the current situation creates to users of MPEG standards;
  2. In case the existence of “difficulties” is confirmed, review the current rules and identify how they can be changed to achieve measurable results (rules have been changed once, so they could be changed a second time);
  3. Act to effect the identified changes to the rules.

Caius Julius Caesar said “Caesar’s wife must be above suspicions”.

Caius Julius Caesar said “Caesar’s wife must be above suspicions”. The guide to carry out the steps should not deny a priori the existence of a problem and not be suspected to be influenced by “the competition”.

Posts in this thread