Can you “clone” MPEG?

Can you “clone” MPEG?

Introduction

The publication of A vision made real – Past, present and future of MPEG has triggered many reactions coming from people in other industries. They are facing the same problem that MPEG started facing 31 years ago when it wanted to create digital media standards that were industry-agnostic and with a global scope.

My answer to this people has been that, indeed, the MPEG model can be exported to other domains where industries with different backgrounds, technologies, stakes and business models want to join forces and develop standards that enable new businesses.

But it won’t be like a piece of cake. Try to inject a new philosophy of work in an environment with its own history, people, relationships and the task is close to impossible. This is not because of technical reasons but because it is a matter of philosophy. MPEG had an easy (so to speak) task because it was a new group without history, with new people with a limited degree of relationships.

In this article I will examine some of the main aspects that one needs to care about to “clone” MPEG to make standards that are industry-agnostic, with a global scope and are intended to enable new businesses.

Going from here to there

The first task is to define what the common project is about. Clarifying the purpose of an undertaking is a good practice that should apply to any human endeavour. This good practice is even more necessary when a group of like-minded people work on a common project – a standard. When the standard is not designed by and for a single industry but by and for many, keeping this rule is vital for the success of the effort. When the standard involves disparate technologies, whose practitioners are not even accustomed to talk to one another, complying with this rule is a prerequisite.

When MPEG was established (late 1980’s) many industries, regions and countries had realised that the state of digital technologies had made enough progress to enable a switch from analogue to digital. Several companies had developed prototypes, regional initiatives were attempting to develop formats for specific countries and industries, some companies were planning products and some standards organisations were actually developing standards for their industries.

MPEG jumped in the scene at a time the different trials had not had the time to solidify. Therefore, it had a unique opportunity to execute its plan of an epochal analogue-to-digital transition of media.

Similarly, industries who wish to create standards that are industry agnostic need to understand what is exactly the world that they want to establish and share a common vision of it.

By the way, you need a business model

Of course, I am not talking about how a hypothetical group working for industry-agnostic standards is going to make money. I am talking about how companies participating in this common effort can develop standards that bring actual benefits to each of them in spite of their differences, market positioning etc.

In the case of MPEG the definition of the business model had to take into account the fact that industry and acad­emia had worked on video compression technologies for some 3 decades filing many patents (at that time they could already be counted by the thousands) which covered a wide range of basic video coding aspects. Video coding standard that are loosely called “royalty free” (in ISO language, for which only Option 1 patent declar­ations are made) were certainly possible but would probably have been unattractive because of their low performance compared with the state-of-the-art codecs.

Therefore, MPEG decided that it would develop standards with the best performance, without consideration of the IPR involved. Patent holders would get royalties from the use of MPEG standards widely adopted by the market. If a patent holder did not want to allow that to happen, they could make an Option 3 declaration and MPEG would remove the infringing tech­nologies.

The MPEG business model is certainly not a prerequisite for developing industry agnostic standards, but it has worked well for the industry. More than that, most patent holders have been and keep on re-investing the royalties they get from existing standards in more technologies for future standards. The MPEG “business model” has created a standard-producing machine (MPEG) that feeds itself with new technologies.

Standards must be industry-friendly

A primary – and obvious – goal of the effort is that the standards produced by the collaborating industries should serve the needs of the participating industries.

The following points describe the three main MPEG targets:

  1. Display formats: Since the appearance of television cameras and displays in the 1920’s, industry and govern­ments have created tens of television formats, mostly around the basic NTSC, PAL and SECAM families. Already in the late 1960’s, when the Picturephone service was deployed, the tradition was hard to die: AT&T invented a new 267-line format, with no obvious connection with any of the existing video formats. As MPEG wanted to serve all markets, it decided that it would just support any display format, leaving display formats outside MPEG standards.
  2. Serving one without encumbering others. An industry may like the idea of sharing the cost of an enabling technology but not at the cost of compromising their individual needs. MPEG standards share some basic technologies but provide the necessary flexibility to its many different users with the notion of Prof­iles (subsets of general interoperability) and Levels (grades of performance within a Profile).
  3. Standards apply only to decoders; encoders are only implicitly defined, and their implementation leaves ample margins of freedom. By restricting standardisation to the decoding functionality, MPEG extends the life of its standards and, at the same time, allows industry players to compete on the basis of their constantly improved encoders.

The conclusion is that standards are great because they enable interoperability but should leave meaningful room to individual participants to exercise their business. Even better, as MPEG did with standards defining only the encoder, this opens the way to research-enabled competition.

Standards for the market, not the other way around

Before MPEG, a company with a success­ful product would try to get a “standard” stamp on it, share the technology with its competitors and enjoy the economic benefits of its “standard” technology.

This process may still be in place for some industries but is not an option when different industries team up to define common industry-agnostic standards with a global scope.

Under the MPEG regime, companies do not wait for the market to decide which technology wins, an outcome that very often has little to do with the value of the technol­ogy or the product but wait for the “best stan­dard” to be developed based on a set of technologies each of which is collectively selected based on a priori defined criteria. Then the technology package – the standard – developed by MPEG is taken over by the industry.

In a multi-industry environment, standards must anticipate the future. The alternative is to stop making standards because if the body waits until market needs are clear, the market is already full of incompatible solutions and there is no room left for standards, certainly not industry-agnostic and with a global scope.

Anticipating market needs is in the DNA of MPEG standards. With each of its standards MPEG is betting that a certain standard technology will be adopted. This explains why some MPEG standards are extremely successful and other less so.

Integrated standards as toolkits

Today’s systems comprise many functions. Some users of the standards are keen to have the complete package of functions, while others want to keep the freedom to cherry-pick other solutions that hopefully fit in the package as shown in the figure.

If interfaces are kept, say the one between System B and System C, the complete system continues to work. Depending on the specific case, however, the level of performance (not the functionality) of the entire system may change and a degree of interoperability may be lost.

Most MPEG standards are composed of the 3 key elements – audio, video and systems – that make an audio-visual system and some, such as MPEG-4 and MPEG-I, even include 3D Graphic information and the way to combine all the media. However, the standards allow maximum usage flexibility:

  1. A standard can be directly used as complete solutions, e.g. like in VCD where Systems, Video and Audio are used
  2. The components of the standard can be used individually, e.g. like in ATSC A/53 where Systems and Video are from MPEG, and Audio is from and external source
  3. The standard does not specify a technology but only an interface to different implementations of the technology, e.g. like in the case of MPEG-I, for which MPEG will likely not standardise a Scene Description technology but just indicate how externally defined technologies can be plugged into the system
  4. A standard does not specify the solution but only the components of a solution, e.g. like in the case of Reconfigurable Video Coding (RVC) where a non-standard video codec can be assembled using an MPEG standard.

A multi industry effort must satisfy the needs of all customers, even those who do not want to use its standards in their entirety but other specifications as well.

Compete and collaborate

Competition is the engine of progress, but standards are the result of a collaboration. How to combine competition and collaboration?

MPEG favours competition to the maximum extent possible. This is achieved by calling for solutions that respondents must comprehensively describe, i.e. without black boxes, in order to qualify for consideration. MPEG experts, including other proponents, assess the merit of prop­osed technologies.

Exte­nding competition beyond a certain point, however, is counterproductive and prevents the group from reaching the goal with the best results.

MPEG develops and uses software platforms that assemble the candidate components selected by its experts – called Test Models – as the platforms where participants can work on improving the different areas of the Test Models.

Core Experiments is the tool that allows experts to improve the Test Model by adding step by step the software that implements the accepted technologies. A Core Experiment is “a technical experiment where the alter­nat­ives considered are fully documented as part of the test model, ensuring that the results of independent experimenters are consistent”.

MPEG mission is to provide the best standards to industry via competition, but MPEG standards should not be shielded from competition Probably the earliest example of application of this principle is provided by MPEG-2 part 3 (Audio). When backward compatibility requirements did not allow the standard to yield a performance comparable to that of algorithms not constrained by compatibility, MPEG issued a Call for Proposals and developed MPEG-2 part 7 (Advanced Audio Codec). Later the algorithms evolved and became the now ubiquitous MPEG-4 AAC. Had MPEG not made this decision, probably we would still have MP3 everywhere, but no other MPEG Audio standards. The latest example is Essential Video Coding (EVC), a standard not designed to offer the best performance, but a good performance with good licensability prospects.

Working on generic standards means that reasonable requests – say, the best unconstrained multichannel audio quality – cannot be dismissed. MPEG tried to achieve that with the technology it was working on – backward-compatible multichannel audio coding – and failed. The only way to respond to the request was to work on a new – competing – technology.

One step at a time

An obvious principle, but it is better to keep it in mind, is that one must fine tune the engine first before engaging in a car race. If in 1988 the newly born MPEG had proposed itself as the developer of an ambitious generic digital media technology standard applicable to all indus­tries on a global scale, the proposal would have been seen as far-fetched and most likely the initiative would have gone no­where.

Instead, MPEG started with a moderately ambitious project: a video coding standard for interactive applications on digital storage media (CD-ROM) at a rather low bitrate (1.5 Mbit/s) targeting the market covered by the video cassette (VHS/Beta) with the addition of interactivity.

Moving one step at a time has been MPEG policy for MPEG-1 and all its subsequent standards and so should do any effort comparable to MPEG’s.

Separate wheat from chaff

In human societies parliaments make laws and tribunals decide if a specific human action conforms to the law. In certain regulated environments (e.g. terrestrial broadcasting in many countries) there are standards and entities (authorised test laboratories) who decide whether a specific implementation conforms to the standard. MPEG has neither but, in keeping with its “industry-neutral” mission, it provides the technical means – namely, tools for conformance assessment, e.g. bitstreams and reference software – for industries to use in case they want to establish authorised test laboratories for their own purposes.

Providing the tools for testing the standard is vital in a multi-industry environment. The ecosystem is owned by all and should not be polluted by non-conforming implementations.

Technology is always on the move

The Greek philosopher Heraclitus is reported to have said: τὰ πάντα ῥεῖ καὶ οὐδὲν μένει (every­thing flows and nothing stays). The fate of any technology field today is that technologies not only do not stay but move fast and actually accelerate.

MPEG is well aware that the technology landscape is constantly changing, and this awareness informs its standards. Until HEVC – one can even say, including the upcoming Versatile Video Coding (VVC) standard – video meant a rectangular area (in MPEG-4, a flat area of any shape, in HEVC it can be a video projected on a sphere). The birth of immersive visual experiences is not without pain, but they are happening, and MPEG must be ready with solutions that take this basic assumption into account. This means that, in the technology scenario that is taking shape, the MPEG role of “anticipatory standards” is ever more important and challenging to achieve.

This has happened for most of its video and audio compression standards. A paradigmatic case of a standard addressing a change of context is MPEG Media Transport (MMT) that MPEG designed having in mind a broadcasting system for which the layer below it is IP, unlike MPEG-2 Transport Stream, originally designed for a digitised analogue channel (but also used for trans­port over IP as in IPTV).

Research for standards

The wild pace of technology progress requires an engine capable to constantly feed new technologies.

MPEG is not in the research business. However, without a world of researchers working with MPEG in mind there, would be no MPEG. The MPEG work plan promotes corporate/academic research because it pushes com­panies to improve their technologies to enable them to make successful responses to Calls for Proposals.

One of the reasons of MPEG success, but also of some of its difficulties, is that MPEG standardisation is a process closer to research than to product design.

Roughly speaking, in the MPEG standardisation process, research happens in two phases:

  1. In companies, in preparation for Calls for Evidence (CfE) or Calls for Proposals (CfP), that MPEG calls competitive phase
  2. In MPEG in what is called collaborative phase, i.e. during the development of Core Exper­iments (of course this research phase is still done by the companies, but in the framework of an MPEG standard under development).

The MPEG collaborative phase offers another opportunity to do more research. This has apparently a more lim­ited scope, because it is in the context of optimising a subset of the entire scope of the standard, but the sum of many small optimisations can provide big gains in performance. The shortcoming of this process is the possible introduction of a large number of IP items for a gain that some may well consider not to justify the added IP onus to complexity. With its MPEG-5 EVC project, MPEG is trying to see if a suitably placed lower limit to performance improvements can help solve the problems identified in the HEVC standard.

Standards as enablers, not disablers

A standard intended for use by many industries cannot be “owned” by a specific industry. Therefore MPEG, keeping faith to its “generic standards” mission, tries to accommodate all legitimate functional requirements when it develops a new standard. MPEG assesses each requirement for its merit (value of functionality, cost of implementation, possibility to aggregate the functionality with others etc.). Profiles and Lev­els are then used to partition the application space in response to specific industry needs.

The same happens if an industry comes with a legitimate request to add a functionality to an existing standard. The decision to accept or reject a request is only driven by the value brought by the proposal, as substantiated by use cases, not because an industry gets an advantage, or another is penalised.

Conclusions

This article has given some hints drawn from MPEG’s 30 years long experience to those who intend to undertake an effort to develop standards for a multi-industry environment.

It is a significant but doable task if the effort is supported by new people without common history because a completely new philosophy of work must be adopted.

It is a close to impossible task if the effort is supported by people who already had a common history. This is true also in the case that the effort is about cloning MPEG to develop digital media standards.

Posts in this thread