The Mule, Foundation and MPEG

What do the three entities of the title have to do together?

The second entity is Isaac Asimov’s Foundation Trilogy,  the tale of an organisation, actually more than one, established by Hari Seldon, who had invented psychohistory. According to that fictional theory the responses of large human populations to certain stimuli will remain the same over time if conditions remain as planned. Then, according to Asimov, psychohistory can predict the main elements of the evolution of society over the centuries and the Foundation is the organisation created to make sure that the future remains as Hari Seldon had planned it.

The first element is the Mule, a character of the trilogy, a mutant that quickly conquers the Galactic Empire with the power of his mental capabilities. It is an element of that fictional society whose appearance Hari Seldon’s psychohistory could not predict. The Mule was not expected to appear, but did.

The third is the MPEG data compression – especially media – group I have been writing about for some time on this blog. a group whose appearance in the media industry could not be predicted because it was completely outside of the rules of that industry, maybe the best real-world equivalent of Hari Seldon’s psychohistory.

Which were those rules? At certain points in history, several discoveries were made that rendered a range of inventions possible. Very often the only merit of guy who made the invention was that he put together a process whose elements were either known or already “floating in the air”. Regularly the invention was patented and gave the inventor the right to exploit his invention for the number of years granted by the law of his country.

In spite of this often chaotic process, several media types converged to the same technology. The photographic industry settled on a limited number of film sizes and the cinematographic industry settled on a limited number of formats: number of frames per second and film sizes. The sound recorded on vinyl records that were played at a limited number of speeds. All this according  to a series of steps that could not individually be predicted, but whose general outcome could.

Use of magnetics and electronics allowed more effective recording and, more importantly, enabled the instantaneous transmission of sound and images to remote places. Here the chaos reigned supreme with a large and growing number of formats for sound and television, real time and stored. If there had been a Hari Seldon of the media industry he could have applied his psychohistory.

In the Media Empire yhe Foundation materialised as a growing number of standards organisations who Tried to keep some order in the field. Table 1 shows just those at the international level, but others popped up at regional, national and industry level.

Table  1 – Media-related standards committees (1980’s)

ITU-T Speech SG XV WP 1
Video SG XV WP 2
ITU-R Audio SG 10
Video SG 11
IEC Recording of audio SC 60 A
Recording of video SC 60 B
Audio-visual equipment TC 84
Receivers SC 12A and G
ISO Photography TC 42
Cinematography TC 36

In “Foundation”, Hari Seldon had anticipated a number of “crises”. In the Media Empire, too, one crisis was due, the advent of digital technologies. Normally, this crisis should have been absorbed by making some cosmetic changes while keeping the system unchanged.

This is not what happened in the Media Empire because The Mule appeared in the form of a wild group of experts banding together under the MPEG flag. In the early days their very existence was not even detected by the most sophisticated devices, but soon the Mule’s onslaught was unstoppable. In a sequence of strikes  the MPEG Mule conquered  the media Empire: interactive video on compact disc, portable music, digital audio broadcasting, digital televisions, audio and video on the internet, file format, common encryption, IP-based television, 3D Audio, streaming on the unreliable internet and more. Billions of people were lured, without complaint but with joy, into the new world.

The forces of the MPEG Mule have clearly triumphed over the forces of darkness and anarchy. The Mule – the ultimate outsider – has exploited the confusion and brought order to everybody’s satisfaction if not to the forces of the Foundation who have been .designing their comeback

What will then be the eventual fate of the MPEG Mule?

In the Foundation, the Mule is eventually wiped out, not because his powers disappear but because others learned some of the methods of the Mule and applied them for their own sake, i.e. to re-instate confusion.

In 30 years of MPEG, and counting? I somehow referred to the MPEG Mule when I wrote “Another thirty years await MPEG, if some mindless industry elements will not get in the way”.

We may be close to know the fate of the MPEG Mule.

Posts in this thread

Can we improve MPEG standards’ success rate?


I am very proud of the high scientific level of the work that MPEG does with its standards. I think that many universities would enhance the value and effectiveness of the education they provide if they took an MPEG standard, not necessarily a successful one, and have students redo the same process that led MPEG to develop that standard.

Because of their high scientific value so many MPEG standards have been wildly successful, as The impact of MPEG standards tells. More standards – more successes – more failures tells about the other side of the coin and argues that as much as certain products from companies are successful and others less so, some MPEG standards have a level of success that does not match the expectations of the experts who developed them.

What went wrong in those cases? My answer to this question is that not necessarily something went wrong and I will try and explain why.

Standards and market needs

MPEG is an excellent environment to develop standards because it has hundreds of the best experts in its field of endeavour. Several hundreds more are in the laboratories who do not, or only seldom attend MPEG meetings. MPEG experts make their proposals of new standards because someone in their companies has envisaged a business enabled by the idea they propose. MPEG is very receptive of new proposals and has a very good process to  develop requirements for a new standard. However, that assumes that the market needs that standard because those who develop the requirements are often the same technical people who will later develop the standard.

Let’s see how another successful environment developing specification – the Digital Video Broadcasting (DVB) project – handles the matter. A proposal for new work goes first to the Commercial Module. If the proposal is not accepted by the Commercial Module, it will not be considered by the Technical Module.

This is possibly a good arrangement for DVB, a group that works apparently in a similar area as MPEG. Similarities, however, may be misleading without considering the entire picture.

  1. MPEG addresses matters regarding the compressed digital representation of media. Therefore MPEG is a “horizontal” committee, while the DVB addresses the entire broadcasting protocol stack, from the physical layer – e.g. DVB-T (for terrestrial), DVB-S (for satellite) and DVB-C (for cable) up to the application layer (e.g. the Multimedia Home Platform or MHP).
  2. MPEG hosts experts from all industries with a stake in digital media while DVB members are predominantly companies interested in broadcasting applications. In all fairness . the separation of broadcasting from other ICT applications in an age of convergence is no longer what it used to be at the technical level, but the business context is different and this still matters.
  3. Some MPEG standards have the luxury of being “long term” (say, 5 years). Examples are the investigations in 3DoF+ and Point Cloud Compression, the former of which is on the launching pad to become a standards and the latter pretty close to the end. Both activities have been incubated for some years. DVB is more short-term oriented and does not want to invest resources in projects that may pay off only years from now.

MPEG already tried something

In 2015 I proposed to MPEG to establish a new activity with the preliminary name of “Industry liaison” and I even developed a concept of the new organisation of MPEG groups (Figure 1). My idea was that this activity could become a new MPEG group, if proved successful.

Figure 1 – Concept of MPEG organisation with industry liaison

Rob Koenen, then of TNO and José Alvarex, then of Huawei America (both tell you something about the dynamics of MPEG experts) were kind enough to take the challenge. They and a group of interested members created a presentation of MPEG activities.that was used at several exhibitions and conferences, e.g. NAB and SMPTE, to stimulate reactions from industry participants. The last slide of the presentation asked three questions:

  • Which needs do you see for media standardisation, between now and years out?
  • What MPEG standardisation roadmap would best meet your needs?
  • To accommodate your use cases, what should MPEG’s priorities be for the delivery of specific standards? For example, do you urgently need something that may enable basic functionality now, or can you wait for a more optimal solution to be released later?

That was a big effort and I am thankful to Rob and José for spending so much effort for the good of MPEG. Unfortunately the goal to create a momentum to the process of getting more information from industry about their needs was not achieved as a continuous and self-sustaining process.

This does not mean that the effort was not useful. Today Rob is in charge of keeping up to date the MPEG Standardisation Roadmap, a public document that provides a graphical representation of the entire MPEG work plan and its timeline, a very important function as Figure 2 related to March 2019 shows.

Figure 2 – MPEG Standardisation Roadmap

Let’s give it another try

Why did the proposal not work out as expected? One reason is intrinsic to the MPEG constitution. MPEG is a large community of experts, but there are very few if any market people, even from the larger companies. We have tried and reached out to the industry but we have been unable to get market people to join MPEG.

My assessment is that, as long as MPEG remains what it is, a wonderful forge of new standard technologies, it will be hard to achieve that goal.

In Which future for MPEG? I have proposed that MPEG become a Subcommittee (SC). This change of status is not intended to be a “promotion” but a better set up that allows MPEG to play a wider role than purely technical.

My proposal is reproduced here in an improved form as

Figure 3 – Proposed new structure of MPEG as an SC

The “Market needs” box represents the new attempt at getting market requirements into the MPEG process. However, it is quite different than Figure 1 because in that figure the Industry liaison function precedes Technical requirements, while here they work in parallel.

How should this new arrangement work?

Proposals for new work should be assessed both from their technical feasibility and their market value. The Technical requirements and Market needs Advisory Groups (AG) should make their assessments driven by “how much technology supports this proposal” and “how much market needs this proposal”, respectively. So, in a sense, the two groups will co-compete. In general Technical requirements will be for innovation, while Market needs will be for conservation. But it will not necessarily be always like that.

These groups should have, as all MPEG groups do have, joint meetings to align their views and they should, if they can, develop a single report. If that cannot happen, there will be two separate, possibly conflicting, reports. In both cases the SC plenary will asses the report(s), take responsibility and make a decision on the basis of a country-based vote.

What will be the meaning of this vote? It should certainly not be a replacement of the vote taken via ballot on a New Work Item Proposal (NWIP), but a vote on whether MPEG should acquire – via Calls for Proposals – the technologies that may be used to develop a standard. The actual NWIP vote will only take place when the Technical requirements AG and the appropriate WG(s) will agree that there is enough material to develop a standard.

So far I have talked of Market needs and Technical requirements having a passive role, i.e. waiting for proposals coming either from experts (as it has regularly happened so far) or from National Bodies.

The operation of the AGs, however, needs not be constrained to be in response to a proposal. They could very well, by their initiative or because they have been requested by the SC plenary, to engage in the study of new areas. Such studies should always be conducted from the point of view of what technology enables and what market needs, or what could be the reaction of the market to a certain standard technology.

The mission of the Market needs and Technical requirements AGs should not be considered accomplished when the NWIP has been approved and the development of the standard starts. MPEG must have the means to review the steps that have brought to success or failure of a standard.

You should not expect that this post mortem analysis can be done 6 months after MPEG has released a standard for publication. It is not unusual that 6 months after release by MPEG a standard has not been published yet. It is true that the world a quarter of century ago moved more slowly that today, but MP3, released in 1992 became a product only 5 years later, MPEG-1 Video (with MP2) became a product at the end of last century. We are talking here of an infrastructure to be put in place now whose first results we can probably see in five years.


This proposal should not be seen as a roadblock to the capability to innovate that MPEG has shown to be capable of in the last 30 years.

It is usually said that if commercialisation costs 100, product development costs 10 and research costs 1. I add that, probably, an MPEG standard costs a company 0.1 (excluding research because that is already accounted for).

So this proposal is not about cost saving. It is about making an excellent and successful MPEG process better, without running the risk highlighted by Monsieur de Montesquieu: Le mieux est l’ennemi du bien (better is the enemy of good).

MPEG is known for its innovative standards, but it should be better known for its process innovation.

Posts in this thread

Can you “clone” MPEG?


The publication of A vision made real – Past, present and future of MPEG has triggered many reactions coming from people in other industries. They are facing the same problem that MPEG started facing 31 years ago when it wanted to create digital media standards that were industry-agnostic and with a global scope.

My answer to this people has been that, indeed, the MPEG model can be exported to other domains where industries with different backgrounds, technologies, stakes and business models want to join forces and develop standards that enable new businesses.

But it won’t be like a piece of cake. Try to inject a new philosophy of work in an environment with its own history, people, relationships and the task is close to impossible. This is not because of technical reasons but because it is a matter of philosophy. MPEG had an easy (so to speak) task because it was a new group without history, with new people with a limited degree of relationships.

In this article I will examine some of the main aspects that one needs to care about to “clone” MPEG to make standards that are industry-agnostic, with a global scope and are intended to enable new businesses.

Going from here to there

The first task is to define what the common project is about. Clarifying the purpose of an undertaking is a good practice that should apply to any human endeavour. This good practice is even more necessary when a group of like-minded people work on a common project – a standard. When the standard is not designed by and for a single industry but by and for many, keeping this rule is vital for the success of the effort. When the standard involves disparate technologies, whose practitioners are not even accustomed to talk to one another, complying with this rule is a prerequisite.

When MPEG was established (late 1980’s) many industries, regions and countries had realised that the state of digital technologies had made enough progress to enable a switch from analogue to digital. Several companies had developed prototypes, regional initiatives were attempting to develop formats for specific countries and industries, some companies were planning products and some standards organisations were actually developing standards for their industries.

MPEG jumped in the scene at a time the different trials had not had the time to solidify. Therefore, it had a unique opportunity to execute its plan of an epochal analogue-to-digital transition of media.

Similarly, industries who wish to create standards that are industry agnostic need to understand what is exactly the world that they want to establish and share a common vision of it.

By the way, you need a business model

Of course, I am not talking about how a hypothetical group working for industry-agnostic standards is going to make money. I am talking about how companies participating in this common effort can develop standards that bring actual benefits to each of them in spite of their differences, market positioning etc.

In the case of MPEG the definition of the business model had to take into account the fact that industry and acad­emia had worked on video compression technologies for some 3 decades filing many patents (at that time they could already be counted by the thousands) which covered a wide range of basic video coding aspects. Video coding standard that are loosely called “royalty free” (in ISO language, for which only Option 1 patent declar­ations are made) were certainly possible but would probably have been unattractive because of their low performance compared with the state-of-the-art codecs.

Therefore, MPEG decided that it would develop standards with the best performance, without consideration of the IPR involved. Patent holders would get royalties from the use of MPEG standards widely adopted by the market. If a patent holder did not want to allow that to happen, they could make an Option 3 declaration and MPEG would remove the infringing tech­nologies.

The MPEG business model is certainly not a prerequisite for developing industry agnostic standards, but it has worked well for the industry. More than that, most patent holders have been and keep on re-investing the royalties they get from existing standards in more technologies for future standards. The MPEG “business model” has created a standard-producing machine (MPEG) that feeds itself with new technologies.

Standards must be industry-friendly

A primary – and obvious – goal of the effort is that the standards produced by the collaborating industries should serve the needs of the participating industries.

The following points describe the three main MPEG targets:

  1. Display formats: Since the appearance of television cameras and displays in the 1920’s, industry and govern­ments have created tens of television formats, mostly around the basic NTSC, PAL and SECAM families. Already in the late 1960’s, when the Picturephone service was deployed, the tradition was hard to die: AT&T invented a new 267-line format, with no obvious connection with any of the existing video formats. As MPEG wanted to serve all markets, it decided that it would just support any display format, leaving display formats outside MPEG standards.
  2. Serving one without encumbering others. An industry may like the idea of sharing the cost of an enabling technology but not at the cost of compromising their individual needs. MPEG standards share some basic technologies but provide the necessary flexibility to its many different users with the notion of Prof­iles (subsets of general interoperability) and Levels (grades of performance within a Profile).
  3. Standards apply only to decoders; encoders are only implicitly defined, and their implementation leaves ample margins of freedom. By restricting standardisation to the decoding functionality, MPEG extends the life of its standards and, at the same time, allows industry players to compete on the basis of their constantly improved encoders.

The conclusion is that standards are great because they enable interoperability but should leave meaningful room to individual participants to exercise their business. Even better, as MPEG did with standards defining only the encoder, this opens the way to research-enabled competition.

Standards for the market, not the other way around

Before MPEG, a company with a success­ful product would try to get a “standard” stamp on it, share the technology with its competitors and enjoy the economic benefits of its “standard” technology.

This process may still be in place for some industries but is not an option when different industries team up to define common industry-agnostic standards with a global scope.

Under the MPEG regime, companies do not wait for the market to decide which technology wins, an outcome that very often has little to do with the value of the technol­ogy or the product but wait for the “best stan­dard” to be developed based on a set of technologies each of which is collectively selected based on a priori defined criteria. Then the technology package – the standard – developed by MPEG is taken over by the industry.

In a multi-industry environment, standards must anticipate the future. The alternative is to stop making standards because if the body waits until market needs are clear, the market is already full of incompatible solutions and there is no room left for standards, certainly not industry-agnostic and with a global scope.

Anticipating market needs is in the DNA of MPEG standards. With each of its standards MPEG is betting that a certain standard technology will be adopted. This explains why some MPEG standards are extremely successful and other less so.

Integrated standards as toolkits

Today’s systems comprise many functions. Some users of the standards are keen to have the complete package of functions, while others want to keep the freedom to cherry-pick other solutions that hopefully fit in the package as shown in the figure.

If interfaces are kept, say the one between System B and System C, the complete system continues to work. Depending on the specific case, however, the level of performance (not the functionality) of the entire system may change and a degree of interoperability may be lost.

Most MPEG standards are composed of the 3 key elements – audio, video and systems – that make an audio-visual system and some, such as MPEG-4 and MPEG-I, even include 3D Graphic information and the way to combine all the media. However, the standards allow maximum usage flexibility:

  1. A standard can be directly used as complete solutions, e.g. like in VCD where Systems, Video and Audio are used
  2. The components of the standard can be used individually, e.g. like in ATSC A/53 where Systems and Video are from MPEG, and Audio is from and external source
  3. The standard does not specify a technology but only an interface to different implementations of the technology, e.g. like in the case of MPEG-I, for which MPEG will likely not standardise a Scene Description technology but just indicate how externally defined technologies can be plugged into the system
  4. A standard does not specify the solution but only the components of a solution, e.g. like in the case of Reconfigurable Video Coding (RVC) where a non-standard video codec can be assembled using an MPEG standard.

A multi industry effort must satisfy the needs of all customers, even those who do not want to use its standards in their entirety but other specifications as well.

Compete and collaborate

Competition is the engine of progress, but standards are the result of a collaboration. How to combine competition and collaboration?

MPEG favours competition to the maximum extent possible. This is achieved by calling for solutions that respondents must comprehensively describe, i.e. without black boxes, in order to qualify for consideration. MPEG experts, including other proponents, assess the merit of prop­osed technologies.

Exte­nding competition beyond a certain point, however, is counterproductive and prevents the group from reaching the goal with the best results.

MPEG develops and uses software platforms that assemble the candidate components selected by its experts – called Test Models – as the platforms where participants can work on improving the different areas of the Test Models.

Core Experiments is the tool that allows experts to improve the Test Model by adding step by step the software that implements the accepted technologies. A Core Experiment is “a technical experiment where the alter­nat­ives considered are fully documented as part of the test model, ensuring that the results of independent experimenters are consistent”.

MPEG mission is to provide the best standards to industry via competition, but MPEG standards should not be shielded from competition Probably the earliest example of application of this principle is provided by MPEG-2 part 3 (Audio). When backward compatibility requirements did not allow the standard to yield a performance comparable to that of algorithms not constrained by compatibility, MPEG issued a Call for Proposals and developed MPEG-2 part 7 (Advanced Audio Codec). Later the algorithms evolved and became the now ubiquitous MPEG-4 AAC. Had MPEG not made this decision, probably we would still have MP3 everywhere, but no other MPEG Audio standards. The latest example is Essential Video Coding (EVC), a standard not designed to offer the best performance, but a good performance with good licensability prospects.

Working on generic standards means that reasonable requests – say, the best unconstrained multichannel audio quality – cannot be dismissed. MPEG tried to achieve that with the technology it was working on – backward-compatible multichannel audio coding – and failed. The only way to respond to the request was to work on a new – competing – technology.

One step at a time

An obvious principle, but it is better to keep it in mind, is that one must fine tune the engine first before engaging in a car race. If in 1988 the newly born MPEG had proposed itself as the developer of an ambitious generic digital media technology standard applicable to all indus­tries on a global scale, the proposal would have been seen as far-fetched and most likely the initiative would have gone no­where.

Instead, MPEG started with a moderately ambitious project: a video coding standard for interactive applications on digital storage media (CD-ROM) at a rather low bitrate (1.5 Mbit/s) targeting the market covered by the video cassette (VHS/Beta) with the addition of interactivity.

Moving one step at a time has been MPEG policy for MPEG-1 and all its subsequent standards and so should do any effort comparable to MPEG’s.

Separate wheat from chaff

In human societies parliaments make laws and tribunals decide if a specific human action conforms to the law. In certain regulated environments (e.g. terrestrial broadcasting in many countries) there are standards and entities (authorised test laboratories) who decide whether a specific implementation conforms to the standard. MPEG has neither but, in keeping with its “industry-neutral” mission, it provides the technical means – namely, tools for conformance assessment, e.g. bitstreams and reference software – for industries to use in case they want to establish authorised test laboratories for their own purposes.

Providing the tools for testing the standard is vital in a multi-industry environment. The ecosystem is owned by all and should not be polluted by non-conforming implementations.

Technology is always on the move

The Greek philosopher Heraclitus is reported to have said: τὰ πάντα ῥεῖ καὶ οὐδὲν μένει (every­thing flows and nothing stays). The fate of any technology field today is that technologies not only do not stay but move fast and actually accelerate.

MPEG is well aware that the technology landscape is constantly changing, and this awareness informs its standards. Until HEVC – one can even say, including the upcoming Versatile Video Coding (VVC) standard – video meant a rectangular area (in MPEG-4, a flat area of any shape, in HEVC it can be a video projected on a sphere). The birth of immersive visual experiences is not without pain, but they are happening, and MPEG must be ready with solutions that take this basic assumption into account. This means that, in the technology scenario that is taking shape, the MPEG role of “anticipatory standards” is ever more important and challenging to achieve.

This has happened for most of its video and audio compression standards. A paradigmatic case of a standard addressing a change of context is MPEG Media Transport (MMT) that MPEG designed having in mind a broadcasting system for which the layer below it is IP, unlike MPEG-2 Transport Stream, originally designed for a digitised analogue channel (but also used for trans­port over IP as in IPTV).

Research for standards

The wild pace of technology progress requires an engine capable to constantly feed new technologies.

MPEG is not in the research business. However, without a world of researchers working with MPEG in mind there, would be no MPEG. The MPEG work plan promotes corporate/academic research because it pushes com­panies to improve their technologies to enable them to make successful responses to Calls for Proposals.

One of the reasons of MPEG success, but also of some of its difficulties, is that MPEG standardisation is a process closer to research than to product design.

Roughly speaking, in the MPEG standardisation process, research happens in two phases:

  1. In companies, in preparation for Calls for Evidence (CfE) or Calls for Proposals (CfP), that MPEG calls competitive phase
  2. In MPEG in what is called collaborative phase, i.e. during the development of Core Exper­iments (of course this research phase is still done by the companies, but in the framework of an MPEG standard under development).

The MPEG collaborative phase offers another opportunity to do more research. This has apparently a more lim­ited scope, because it is in the context of optimising a subset of the entire scope of the standard, but the sum of many small optimisations can provide big gains in performance. The shortcoming of this process is the possible introduction of a large number of IP items for a gain that some may well consider not to justify the added IP onus to complexity. With its MPEG-5 EVC project, MPEG is trying to see if a suitably placed lower limit to performance improvements can help solve the problems identified in the HEVC standard.

Standards as enablers, not disablers

A standard intended for use by many industries cannot be “owned” by a specific industry. Therefore MPEG, keeping faith to its “generic standards” mission, tries to accommodate all legitimate functional requirements when it develops a new standard. MPEG assesses each requirement for its merit (value of functionality, cost of implementation, possibility to aggregate the functionality with others etc.). Profiles and Lev­els are then used to partition the application space in response to specific industry needs.

The same happens if an industry comes with a legitimate request to add a functionality to an existing standard. The decision to accept or reject a request is only driven by the value brought by the proposal, as substantiated by use cases, not because an industry gets an advantage, or another is penalised.


This article has given some hints drawn from MPEG’s 30 years long experience to those who intend to undertake an effort to develop standards for a multi-industry environment.

It is a significant but doable task if the effort is supported by new people without common history because a completely new philosophy of work must be adopted.

It is a close to impossible task if the effort is supported by people who already had a common history. This is true also in the case that the effort is about cloning MPEG to develop digital media standards.

Posts in this thread

Who “owns” MPEG?


The title of this article contains three elements: the verb own pre- and postfixed with a quotation mark which conveys the notion of having “a control of, a stake in or an influence on”; the acronym MPEG with its multiple meanings; and the pronoun who which represents several entities.

In this article I will try and clarify the nature of these three elements and draw some conclusions that may be useful for some impending decisions.

The different uses of the word “MPEG”

In MPEG: what it did, is doing, will do I have reported evidence that even the public at large widely knows the word “MPEG”. But what do they mean when they use it? Most often they intend some “media technologies” present in a device or used to create, store or transmit media content. Therefore, the word “MPEG” in this context is owned by all.

The domain is obviously connected to the word “MPEG”. As I have written in the same MPEG: what it did, is doing, will do, the domain is owned by an individual who is not using the domain, is not inclined to donate it and does not answer when asked how much will make him inclined. So is privately owned.

A small number of people use the word “MPEG” to indicate one of more of the ISO/IEC standards listed in ISO numbers of MPEG standards. In this context the ownership is clear because ISO and IEC own the copyright of the standard, as stated in the second page of all MPEG standards. But what about other rights?

A small but important number of people use the word MPEG to indicate the working group (WG) whose official name is ISO/ IEC JTC 1/SC 29/WG 11 (I still have to meet someone who is not in the ISO/IEC circle and knows the avronym). To understand what “ownership” means in this context we have to make a digression that I hope will be perceived as short.

The place of the MPEG WG in ISO

The MPEG WG is the green box in the ISO organisation chart of Figure 1. I know that I should also say something about IEC, but the explanation I am going to make will discourage a sufficient number of well-intentioned readers to read further and I do not want to risk losing them all.

Figure 1 – MPEG sits at the bottom of the ISO organisation

In the hierarchical organisation of Figure 1 there are plenty of opportunities to claim “ownership”.

  1. The General Assembly, made up of all standards organisations members of ISO called National Bodies (NB);
  2. The ISO Council, the core governance body made up of 20 NBs, the ISO Officers and the Chairs of 4 committees;
  3. The Technical Management Board (TMB), in charge of managing the structure and activities of the Technical Committee (TC), made up of a chair and 15 members from NBs;
  4. The Joint ISO/IEC Technical Committee 1 Information technology (JTC 1), the largest Techical Committee in ISO with 15 Advisory Groups, 2 Working Groups, and 22 Subcommittees (SC);
  5. The Subcommittee 29, in charge of Coding of audio, picture, multimedia and hypermedia information and has 2 Working Groups;
  6. WG 11 aka MPEG in charge of Moving Pictures and Audio.

“Ownership” of MPEG standards


The MPEG WG is populated by experts who are accredited by the ISO NBs they are members of. They are typically employes of a company or university but there are some students, consultants or simply individuals.

Experts attending MPEG produce standards using the process described in … ISO and IEC sell copies of the standards whose copyright they own. These can be purchased online from Some standards (e.g. those common with ITU) can be freely downloaded.

Patent holders

The buyer of an MPEG standard does not necessarily have the right to practice the standard because of rights to essential patents that may be owned by third parties, in most cases employers of MPEG experts. Therefore, these third parties also “own” the MPEG standard in which they have essential patents. Since the very beginning, MPEG standards a large number of patent holders and patent pools were established to license packages of patents considered essential to practice MPEG standards. Therefore, patent pools, too, have a sort of “ownership”.

A court of law may declare that a patent is essential to practice a standard in a jurisdiction. This kind of declaration is available for a limited number o patents (compared to the number of patents granted). Therefore, the identity of those who really “own” something in an MPEG standard is typically wrapped in mystery.

ISO and IEC (and ITU when relevant) keep patent declarations spontaneously made by those who believe they have rights in the standards. Patent declarations are also made by those who submit technologies for consideration by MPEG, either as part of responses to a Call for Proposal or as independent submissions (e.g. when submitting results of a Core Experiment). ISO and IEC simply record those declarations and take no stance regarding their validity. Therefore, the actual “ownership” claimed by those declarations or agreements to recognise them has to be determined outside of ISO.

Ownership of reference software

MPEG prides itself to have the policy to develop two “versions” of most of its standards. One version is expressed in natural language (English, later translated into French and Spanish). The second version is expressed in a computer language and is called the reference software of the standard. Both versions have normative status.

In the second half of the 1990’s, MPEG developed the MPEG-4 “copyright disclaimer” whereby users could get a free license from ISO/IEC to use and modify individual modules of the reference software for products claiming conformance to the specific MPEG standard. Of course, the copyright disclaimer included the usual disclaimer about third parties’ rights.

In the mid 2000s MPEG adopted a slightly modified version of the Berkeley Software Distribution (BSD) available as MXM Licence), a licence originally used to distribute a Unix-like operating system. The licence simply says that the reference software may be used for anything the uses wishes, with some obvious exceptions. MPEG has added the usual disclaimer (the “modification” above) about possible third-party rights.

Ownership of conformance testing suites

The notion of “ownership”, possibly a more complicated one, is also applicable to conformance testing suites. These consist of bitstreams designed and produced by MPEG experts to test particularly critical parts of a decoder. They have a normative value and are declared to conform to a specific MPEG standard. Bitstreams can be downloaded from the ISO web site and used to check whether a decoder implementation conforms to the standard.

Ownership of test sequences

There are other parties who have some “ownership” of MPEG standards, the companies (in some cases individuals) who have contributed content – images, audio, video and point clouds – used to carry out evaluations of submissions or verification tests. The very first test sequences used by MPEG were the so-called “CCIR sequences” donated by CCIR (now ITU-R) and used in MPEG-1 and other Calls for Proposals. For years the entire video coding community widely used “Table tennis”, “Mobile and calendar” etc. More recently, rights holders have begun to license content to individual MPEG experts for the purpose of developing MPEG standards.

ISO’s is a different organisation chart

Many companies have similar organisation charts with similar boxes as those of Figure 1. They are populated in the real world by employees and managers often with the support of secretariats. Unlike companies, however, ISO does not have its own employees populating the boxes located under the Technical Management Board. Who populates them?

Working Groups are populated by experts sent by companies and chaired by convenors, normally coming from industry. The role of convenors is to facilitate consensus and to detect achievement of consensus. As part of their duty to convene experts, convenors may need to organise the work. This gives convenors a limited level of “ownership”.

Subcommittees and Technical Committees are populated by delegations of National Bodies made up of NB officers often complemented by industry representatives. SC and TC secretariats have a level of “ownership” because the ISO/IEC Directives assign to them several important tasks such as the management of document balloting, e.g. proposals of new work items or approval of standards under development. Committee chairs, too, have a level of ownership.

Another level of “ownership” comes from the fact that a Convenor is nominated by the SC secretariat and elected by NB delegates in the SC. The SC Chair is elected by NB delegates in the SC and confirmed by NB delegates in the TC.

National Bodies “own” ISO

As we have seen, National Bodies permeate the ISO structure at all levels. Decisions at TC and SC level are made by NB votes. It is a one country, one vote systems but, sometimes, some votes weigh more than others. Votes are cast on the basis of the opinion formed within the NBs, ostensibly to further “national interests”.

National Bodies usually have a structure mirroring that of ISO. Industrial interests of national companies which are members of the National Body, often subsidiaries of multinational companies, intermingle to determine national positions on matters to be decided at the international level.


In this article I have tried to identify and describe the different forms of MPEG-generated “ownership”. Clearly the most important “ownership” is represented by MPEG standards because they generate three revenue streams: from products/services/applications enabled by MPEG standards (companies) to satisfy end user needs, from those who practice MPEG standards (patent or, more generally, intellectual property holders) and from the sale of standards (ISO/IEC and National Bodies).

Unfortunately, this clearcut logic is polluted by the different forms of “ownership” that I have described above, especially the desire of National Bodies to hold committee secretariats and nominate chairs. These may be nice-looking elements in National Body panoplies, but have nothing to do with the intensity of the three streams, the raison d’être of a standards organisation.

The intensity of revenue streams exclusively depends on the productivity of the “machine” (the committee) that creates the standards. “Owners” acting according to logic should leverage – not meddle with – a working group whose standards enable a global turnover of devices worth more than 1 trillion USD and services revenues of ~230 billion USD for pay-tv alone (data of 2018).

It will be interesting to see if the next decisions will follow a revenue stream maximisation logic, a curio collection logic or a survival logic.

Posts in this thread

Which future for MPEG


For three decades MPEG has designed multi-threaded work programs, cooperated with tens of organisations and developed close to 200 specifications and hundreds of amendments. MPEG standards have transformed the global media industry from analogue to digital, enabled participation of new industries in the media business, provided standards serving all industries without favours and ensured unrelenting expansion of the entire media business.

MPEG achievements look reassuring: in 2018 the global turnover of MPEG-enabled devices is worth more than 1 trillion USD per annum and the global Pay-Tv revenues are 228 billion USD per annum, without mentioning other industries.

Should MPEG rest and look forward to a bright future built on the successes achieved? It would be so easy to answer “tout va très bien madame la marquise” but my answer is no.

The MPEG landscape has changed

In the first 10 years of its existence MPEG ruled the field of audio and video coding. In the following 10 years some proprietary solutions popped up, but the field was still largely dominated by MPEG standards. In the last 10 years MPEG has seen proprietary solutions getting significant strength. Today large areas that used to be exclusively covered by MPEG standards are satisfied by other solutions and the path to satisfy the next needs is not at all clear.

We do not have the complete picture yet of the extent the market will turn its back to MPEG standards. Market will speak and market is right by definition. If we do not like it, it is just because we did not try hard enough.

The next few years will be very challenging. It will not be a time of “business as usual”. MPEG needs to rethink itself and take appropriate measures. This article lays down some ideas and presents a proposal.

MPEG is about compression

So far MPEG has produced 5 generations of video compression standards, each generation offering more compression and more features. More of the same is expected from the 6th generation (VVC). MPEG has produced an equivalent number of audio coding standards. Will industry keep on asking for more video compression? I would like to answer with a resolute yes as I believe that there will always be need for more compression of visual information, but not always and not necessarily of the “old way”. More importantly, the need will not always be satisfied by MPEG standards because “The Price Is Right” applies to compression standards, too.
The answer to the question “Do we need more audio compression?” is, at least in the current time frame, that the currently available compression engine (MPEG-H 3D Audio) is good enough but we need new standards for other non-compression features, e.g. 6 Degrees of Freedom (6DoF). In the future this trend will also apply to video, as 3DoF+ video moves in the same direction as 6DoF audio (see The MPEG drive to immersive visual experiences).
Point cloud compression definitely needs compression but the convergence between 3D visual information captured by means of traditional video capturing devices and Point Cloud is still a matter for investigation.

MPEG is also about systems aspects

Systems aspects have been the enabling factor of the success of many MPEG standards and the future will not make those aspect less, but more important. The trend toward immersive media will require an even deeper integration between compressed media and the systems aspects that permeate them.
This can be seen from the requirements that are being identified in an activity called “Immersive Media Access and Delivery” where four dimensions are identified:

  1. Time (the usual one)
  2. Space (how to retrieve just the media parts of interest)
  3. Quality (how to access portions of media with the quality desired)
  4. Object (how to access specific parts of specific objects of interest).

MPEG in not just about media

In the past 30 years MPEG has shown that it could address new domains of expertise and learn their language. The fact that today all digital media speak the same (technical) language, is also due to the efforts made by MPEG to understand the needs of different industries, convert them to requirements, develop the technologies and quantise the standards into profiles and levels. This workflow has been in operation for 27 years, starting from the moment MPEG invented profiles and levels, and consistently applied them to talk to different communities using the same language.
This does not mean that there are no challenges when we talk to a new industry. MPEG has spent more than 3 years talking to, and identifying and validating requirements with the genomic community before starting the development of the MPEG-G standard. This significant effort has paid off: three International Standards on Genomic Data Compression have been developed jointly with TC 276 and 3 more are in the pipeline.

Governance is important

MPEG achieved its results as a working group, i.e. as the lowest organisational unit in ISO/IEC that the ISO/IEC directives recommend to be “reasonably limited in size”. Rightly, the ISO/IEC Directives do not define the term “reasonable”, but in 1989 MPEG had already 100 members, in 1999 it had 300 members and in 2019 it has 1500 members 500 of which attend its quarterly meetings. For the work it has done, MPEG has been reasonably limited in size.

For 30 years MPEG has played a role much above its status and I do not think there should be complaints about the results. In normal conditions MPEG could continue to operate as a working group for another 30 years but, as I said above, these are not normal conditions.

MPEG should become a Subcommittee (SC). Why? Because an SC has a solid governance administered by delegates appointed by National Bodies under the leadership of a chair. On the other hand, the design of the organisation must be properly done, if we do not want to do more harm than good.

Design of the SC is important

To be successful, the organisation of the SC should be conservative because of the importance of the industries served by MPEG standards. Therefore, the organisation of the SC should leverage the existing successful MPEG organisation by retaining and strengthening:

  1. The existing well-honed and demonstrably effective MPEG organisation. MPEG has fed the global media industry with technology that has allowed its growth for the last 30 years. It would be irresponsible to do anything that jeopardises such a large industry, the millions of jobs that go with it and the billions of consumers.
  2. The MPEG collaborative stance with other bodies. The major reason of the success of MPEG standards is MPEG’s open collaboration stance with its many client industries as represented by their standard organisation or industry fora. Collaboration is a must for MPEG because compression is always part of a bigger system with many interactions with other components. However, what was good 30, 15 or even 5 years ago is not necessarily sufficient today.
  3. The match of new MPEG standards to market needs. MPEG has produced hugely successful standards. However, other standards are less so. This is inevitable for an organisation that develops anticipatory standards that sometimes target the next 5 years. MPEG’s ability to engage in standards that are better matches of market needs has to be enhanced because conditions have changed.
  4. The strong MPEG brand. The new organisation is an internal matter designed to give MPEG a better chance to face a complex situation and should not create confusion in the MPEG client industries.

Figure 1 represents the proposed organisation.

Figure 1 – Structure of Subcommittee “MPEG Coding of Moving Pictures and Audio”

Meeting design criteria

Criterion #1: the organisational chart of Figure 1 retains the current MPEG organisational structure where existing informal subgroups become (in italic the names of the existing MPEG entities):

Advisory Groups (AG) if they do not develop standards (orange blocks):

  1. Technical Requirements (Requirements)
  2. Liaison and Communication (Communication)
  3. Technical Coordination (Chairs meetings)

Working Groups (WG) if they develop standards (light green blocks):

  1. Systems Coding (Systems)
  2. Video Coding (Video)
  3. Audio Coding (Audio)
  4. 3D Graphics Coding (3D Graphics)
  5. Quality Assessment (Test)
  6. Genomic Data Coding (Genomic activity in Requirements)

Criterion #2: The chart preserves and extends MPEG’s collaboration stance: Joint Teams with ITU-T (JCT-VC and JVET). The new organisation will be able to establish JWGs to develop standards on well-identified common areas of standardisation, e.g. JPEG, SC 24, SC 41 (IoT), SC 42 (Artificial Intelligence), TC 276 (Bioinformatics).

Criterion #3: The SC will now be able to carry out activities with high strategic value by assigning to the Market Needs AG Technical Requirements AG the task to investigate 1) existing or new areas of work and 2) proposals for new areas of work. Both AGs will produce coordinated reports that will be used by the MPEG SC to make an informed decision on new work.

Criterion #4: The SC should be called “MPEG Coding of Moving Pictures and Audio” prefixing the word MPEG to the current title of ISO/IEC JTC 1/SC 29/WG 11 (MPEG).


In a few weeks industry will decide the future of MPEG.

Will industry decide to give itself a safe and prosperous path by adopting the organisation proposed in this article or will it opt for the Japanese saying “出る釘は打たれる” (The nail that sticks out gets hammered down)? Will industry allow the MPEG nail to keep on sticking out or will it hammer it down?

Stay tuned to this block for further news.

Posts in this thread

Why MPEG is part of ISO/IEC


In July 1987 the plan to create a group that would develop industry-neutral standards was formed. But problem to be tackled was that the MPEG “digital baseband” (see The discontinuity of digital technologies) had to based on international international standards because they had to have global validity.

The question then was: where should those standards be developed? The answer is provided by the following sections:

  1. Standards describes the 3 international standards organisations;
  2. ISO and IEC standards describes the ISO structure and the ISO/IEC standardisation process;
  3. A home for MPEG describes how an independent home for MPEG was found.


Standards have a special place in industry because they represent convergence points where the parties involved, who typically are in competition, find it convenient to agree on a single solution.

Standards bodies exists at the international level:

  1. International Telecommunication Union (ITU) for matters related to telecommunication and broadcasting
  2. International Electrotechnical Commission (IEC) for electrotechnical matters
  3. International Organisation for Standardisation (ISO) for everything else.


The International Telecommunication Union (ITU) is the result of the 1934 merge between the International Telegraph Convention of 1865 and the International Radiotelegraph Convention of 1906, and today is an agency of the United Nations. This is reflected in the two main branches of the ITU: ITU-T and ITU-R. The former deals with standards for global telecommunications excluding radio communication because this is the purview of ITU-R.


The International Electrotechnical Commission (IEC) is a not-for-profit organisation founded in 1906. It develops International Standards in the fields of electrotechnology, e.g. power gener­ation, transmission and distribution to home appliances and office equipment, semiconductors, fibre optics, batteries, solar energy, nanotechnology and marine energy.


The International Organization for Standardization (ISO) is an international non-governmental standard-setting organisation founded in 1947 and composed of representatives from various national standards organizations.

ISO is well known for its family of quality management systems standards (ISO 9000), environ­mental management standards (ISO 14000) and Information Security Management Systems standards (ISO 27000). There are more than 20,000 ISO published standards.

ISO is a huge organisation whose technical branch is structured, as is the IEC’s, in Technical Committees (TC). The first 3 active TCs are: TC 1 Screw threads, TC 2 Fasteners and TC 4 Rolling bearings. The last 3 TCs in order of establishment are TC 322 Sustainable finance, TC 323 Circular economy and TC 324 Sharing economy.

Between these two extremes there is a large number of TCs, e.g., TC 35 Paints and varnishes, TC 186 Cutlery and table and decorative metal hollow-ware, TC 249 Traditional Chinese med­icine, TC 282 Water reuse, TC 297 Waste collection and transportation management, etc.

Most TCs are organised in working groups (WG). They are tasked to develop standards while TCs retain key functions such as strategy and management. In quite a few cases the area of res­ponsibility is so broad that a horizontal organisation would not be functional. In this case a TC may decide to establish Subcommittees (SC) which include WGs tasked develop standards.

Figure 1 is an organigram of ISO.

Figure  1 – ISO governance structure

ISO and IEC standards

The development process

ISO and IEC share the standard development process which can be summarised as follows:

  1. Submission and balloting of a New Work Item Proposal (NWIP) of a new project meant to lead to an International Standard (IS) or Technical Report (TR). The former contains normative clauses, the latter is informative
  2. Development of a Working Draft (WD, possibly several versions of it
  3. Balloting of the Committee Draft (CD, when the WD has achieved sufficient maturity)
  4. Balloting of the Draft International Standard (DIS, after resolving comments made by National Bodies)
  5. Balloting of the Final Draft International Standard (FDIS, after resolving comments made by National Bodies)

The last ballot is yes/no. No comments allowed.

Amendments (AMD) extend a standard. The same steps as above are carried out with the names Proposed Draft Amendment (PDAM), Draft Amendment (DAM) and Final Draft Amendment (FDAM).

If an error is discovered, a Corrigendum (COR) is produced. This only goes through two stages: Draft Corrigendum (DCOR) and Corrigendum (COR).

A Technical Report, a document without normative clauses, goes through two stages of approval: Proposed Draft Technical Report (PDTR) and Technical Report (TR).


ISO/IEC mandates that in the development of stan­dards working groups operate based on consensus. This is defined as

General agreement characterised by the absence of sustained opposition to substantial issues by any important part of the concerned interests and by a process that involves seeking to take into account the views of all parties concerned and to reconcile any conflicting arguments.

NOTE — Consensus need not imply unanimity.

Patent policy

ISO, IEC and ITU share a common policy vis-à-vis patents in their standards. Using few im­precise but hopefully clear words (as opposed to many precise but unclear words), the policy is:

  1. It is good if a standard has no patents or if the patent holders allow use of their patents for free (with an “Option 1” declaration);
  2. It is accepted if a standard has patents, but the patents holders only allow use of their patents on fair and reasonable terms and non-discriminatory conditions (with an “Option 2” declaration);
  3. It is not permitted to have a standard with patents whose holders do not allow use of their patents (with an “Option 3” declaration).

A home for MPEG

When the MPEG idea took shape in July 1987, the selection of a home to implement the idea was the primary concern. The idea was spoilt for choices as shown by the list of international committees in  Table 1 that were created for various reasons – regulation or simply need for an independent technical reference – to cater to the needs of standards by the different industries.

Table  1 – Media-related standards committees (1980’s)

ITU-T Speech SG XV WP 1
Video SG XV WP 2
ITU-R Audio SG 10
Video SG 11
IEC Recording of audio SC 60 A
Recording of video SC 60 B
Audio-visual equipment TC 84
Receivers SC 12A and G
ISO Photography TC 42
Cinematography TC 36

Since MPEG was conceived to be industry-neutral, committees already developing standards in the “media” area were considered unsuitable because the represented “vested interests”. The choice fell on ISO TC 97 Data Processing who had SC 2 Character sets and Information Coding who included WG 8 Coding of Audio and Picture Information.

In 1987 ISO/TC 97 Data Processing merged with IEC/TC 83 Information technology equipment. The resulting (joint) technical committee was called ISO/IEC JTC 1 Information Technology. SC 2 with its WGs, including WG 8, became part of JTC 1. MPEG was established as an Experts Group on Moving Pictures of ISO/IEC JTC 1/SC 2/WG 8 in 1988.

Note that Experts Group is an organisational entity not officially recognised in the ISO organ­igram. In 1991 SC 2/WG 8 seceded from SC 2 and became SC 29. WG 8’s Moving Picture Experts Group (MPEG) became WG 11 Coding of audio, picture, multimedia and hypermedia information (but everybody in the industry, and even in the general public, calls it MPEG).

Posts in this thread

The discontinuity of digital technologies


Last week I published as an article of this blog the Executive summary of my book A vision made real – Past, present and future of MPEG. This time I publish as an article the first chapter of the book about the four aspects of the media distribution business and their enabling tech­nologies:

  1. Analogue media distribution describes the vertical businesses of analogue media distribution;
  2. Digitised media describes media digitisation and why it was largely irrelevant to distribution;
  3. Compressed digital media describes how industry tried to use compression for distribution;
  4. Digital technologies for media distribution describes the potential structural impact of compressed digital media for distribution.

Analogue media distribution

In the 1980’s media were analogue, the sole exception being music on compact disc (CD). Different industries were engaged in the business of distributing media: telecom­mun­ication companies distributed music, cable operators distributed television via cable, terrestrial and sat­ellite broadcasters did the same via terrestrial and satellite networks and different types of busin­esses distributed all sort of recorded media on physical support (film, laser discs, compact cas­set­te, VHS/Betamax cassette, etc.).

Even if the media content was exactly the same, say a movie, the baseband signals that represented the media content were all different and specific of the delivery media: film for the theatrical vision, television for the terre­strial or satellite network or for the cable, a different format for video cassette. Added to these technological differences caused by the physical nature of the delivery media, there were often substantial differences that depended on countries or manufacturers.

Figure 1 depicts the vertical businesses of the analogue world when media distribution was a collection of industry-dependent distribution systems each using their own technologies for the baseband signal. The figure is simplified because it does not take into ac­count the country- or region-based differences within each industry.

Figure  1 – Analogue media distribution

Digitised media

Since the 1930’s the telecom industry had investigated digitisation of signals (speech at that time). In the 1960’s technology could support digitisation and ITU created G.711, the standard for digital speech, i.e. analogue speech sampled at 8 kHz with a nonlinear 8 bits quantisation. For several decades digital speech only existed in the (fixed) network, but few were aware of it because the speech did not leave the network as bits.

It was necessary to wait until 1982 for Philips and Sony to develop the Compact Disc (CD) which carried digital stereo audio, specified in the “Red Book”: analogue stereo audio sampled at 44.1 kHz with 16 bits linear. It was a revolution because consumers could have an audio quality that did nor deteriorate with time.

In 1980 a digital video standard was issued by ITU-R.  The luminance and the two colour-differ­ence signals were sampled at 13.5 and 6.75 MHz, respectively, at 8 bits per sample yielding an exceedingly high bitrate of 216 Mbit/s. It was a major achievement, but digital television never left the studio if not as bulky magnetic tapes.

The network could carry 64 kbit/s of digital speech, but no consumer-level delivery media of that time could carry the 1.41 Mbit/s of digital audio and much less the 216 Mbit/s of digital video. Therefore, in the 1960s studies on compression of digitised media begun in earnest.

Compressed digital media

In the 1980’s compression research yielded its first fruits:

  1. In 1980 ITU approved Recommendation T.4: Standardization of Group 3 facsimile terminals for document transmission. In the following decades hundreds of million Group 3 facsimile devices were installed worldwide because, thanks to compression, transmission time of an A4 sheet was cut from 6 min (Group 1 facsim­ile), or 3 min (Group 2 facsimile) to about 1 min.
  2. In 1982 ITU approved H.100 (11/88) Visual telephone systems for transmission of videocon­ference at 1.5/2 Mbit/s. Analogue videoconferencing was not unknown at that time because several com­panies had trials, but many hoped that H.100 would enable diffused business communication.
  3. In 1984 ITU started the standardisation activity that would give rise to Recommendations H.261: Video codec for audio-visual services at p x 64 kbit/s approved in 1988.
  4. In the mid-1980s several CE laboratories were studying digital video recording for magnetic tapes. One example was the European Digital Video Recorder (DVS) project that people ex­pected would provide a higher-quality alternative to the analogue VHS or Betamax video­cassette recorder, as much as CDs were supposed to be a higher-quality alternative to LP records.
  5. Still in the area of recording, but for a radically new type of application – interactive video on compact disc – Philips and RCA were independently studying methods to encode video signals at bitrates of 1.41 Mbit/s (the output bitrate of CD).
  6. In the same years CMTT, a special Group of the ITU dealing with transmission of radio and television programs on telecommunication networks, had started working on a standard for transmission of digital television for “primary contribution” (i.e. transmission between stu­dios).
  7. In 1987 the Advisory Committee on Advanced Television Service was formed to devise a plan to introduce HDTV in the USA and Europe was doing the same with their HD-MAC project.
  8. At the end of the 1980’s RAI and Telettra had developed an HDTV codec for satellite broad­casting that was used for demonstrations during the Soccer World cup in 1990 and General Instrument had showed its Digicipher II system for terrestrial HDTV broadcasting in the band­width of 6 MHz used by American terrestrial television.

 Digital technologies for media distribution

The above shows how companies, industries and standards committees were jockeying for a pos­ition in the upcoming digital world. These disparate and often uncorrelated initiatives betrayed the mindset that guided them: future distribution of digital media would have an arrangement similar to the one sketched in Figure 1 for analogue media: the “baseband signal” of each delivery medium would be digital, thus using new technology, but different for each industry and possibly for each country/region.

In the analogue world these scattered roles and responsibilities were not particularly harmful be­cause the delivery media and the baseband signals were so different that unification had never been attempted. But in the digital world unification made a lot of sense.

MPEG was conceived as the organisation that would achieve unification and provide generic, i.e. domain-independent digital media compression. In other words, MPEG envisaged the completely different set up depicted in Figure 2.

Figure  2 – Digital media distribution (à la MPEG)

 In retrospect that was a daunting task. If its magnitude had been realised, it would probably never have started.

Posts in this thread

A vision made real – Past, present and future of MPEG

Why this book?

In a generation, life of the large majority of human beings is incredibly different than the life of the generation before. The ability to  communicate made possible by ubiquitous internet and to convey media content to others made possible by MPEG standards can probably be mentioned among the most important factors of change. However, unlike internet about which a lot has been written, little is known about the MPEG group besides its name.

This book wants to make up for this lack of information.

It will talk about the radical transformation that MPEG standards wrought to the media distribution business by replacing a multitude of technologies owned by different businesses with a single technology shared by all; the environment in which it operates; the radically new philosophy that underpins this transformation; the means devised to put the philosophy into practice; the industrial and economic impact of MPEG standards; what  new standards are being developed; and what is the future that the author conceives for MPEG as an organisation that plays such an important industrial and social role.

Bottom line, MPEG is about technology. Therefore, the book offers an overview of all MPEG standards and, in particular, videoaudiomedia qualitysystems and data. This is for those more (but not a lot more) technology-minded.

Important – there are short Conclusions worth reading.

Leonardo Chiariglione

Table of Contents of A vision made real – Past, present and future of MPEG

Introduction of A vision made real – Past, present and future of MPEG

The impact of MPEG standards

I suppose that few visitors of this blog need to be convinced that MPEG is important because they have some personal experience of the MPEG importance. Again, I suppose not all visitors have full visibility of all the application areas where MPEG is important.

This article describes different application domains showing how applications have benefited from MPEG standards. The list is not exhaustive and the order in which applications are presented follows approximately the time in which MPEG enabled the application.

Digital Television for distribution

MPEG-2 was the first integrated digital television standard first deployed in 1994, even before the MPEG-2 standard was approved. While most countries have adopted MPEG-2 Video for their terrestrial broadcasting services, with one notable major exception, countries have made different selections of for the audio component.

MPEG-2 Transport Stream is the Systems layer of Digital Television. The Systems layer can carry the “format identifier”. In case the media (audio or video) carried by the Systems layer are different from MPEG, the format identifier indicates which of the registered formats is being actually used.

Digital Television exploits Digital Storage Media Command and Control (DSM-CC) to set up a network connection (used by CATV services) and the carousel to send the content of a slowly changing information source that each receiver that happens to “tune-in” can acquire after some time.

MPEG-4 AVC has replaced MPEG-2 Video in many instances because of its superior compression performance. MPEG-H HEVC is also being used in different countries especially for Ultra High Definition (UHD) distribution. HEVC has the advantage of providing better compression that AVC. Additionally it supports High Dynamic Range (HDR) and Wider Colour Gamut (WCG).

MPEG-B Part 9 provides a specification for Common Encryption of MPEG-2 Transport Streams.

MPEG-H part 1 MPEG Media Transport (MMT), replaces the original MPEG-2 Transport Stream. MMT is part of the ATSC 3.0 specification.

Digital Audio Broadcasting

In the mid-1990’s different European countries began to launch Digital Audio Broadcasting services based on the specifications of the Eureka 147 (EU 147) research project. EU 147 used MPEG-1 Audio Layer II as compressed audio format, in addition to other EU 147-proper specifications. The specification were widely adopted in other countries outside of Europe promoted by the non-government organisation WorldDAB.

In 2006 the DAB+ specifications were released. DAB+ includes HE-AAC v2 and MPEG surround (MPEG-D Part 1).

Technologically connected to DAB for the transport layer, but addressing video (AVC), is the Digital Multimedia Broadcasting (DMB) system developed by Korea for video transmission on mobile handsets.

Other audio services, such as XM, use HE-AAC.

Digital Audio

MP3 (MPEG-1 Audio Layer III) brought a revolution in the music world because it triggered new ways to distribute and enjoy music content. MP3 players continued the revolution brought about by the Walkman. Different versions of AAC continued that trend and triggered the birth of music distribution over the internet. Today most music is distributed via the internet using MPEG standards.

Digital Video for package media distribution

Video Compact Disc (VCD)

The original target of MPEG-1 – interactive video on compact disc – did not happen but, especially in Far East markets, VCD was a big success – probably 1 billion devices sold – anticipating the coming of the more performing but more complex MPEG-2 based DVD. VCD used MPEG-1 Systems, Video and Audio Layer II.

Digital Versatile Disc (DVD)

The widely successful DVD specification used MPEG-2 Video, MPEG-2 Program Stream and a selection of audio codecs for different world regions.

Blu-ray Disc (BD)

The BD specification makes reference to AVC and to Multiview Video Coding. MPEG-2 TS is used instead of MPEG-2 PS. Apparently, no MPEG audio codecs are supported.

Ultra HD Blu-ray

The specification supports 4K UHD video encoded in HEVC with 10-bit High Dynamic Range and Wider Colour Gamut.

Digital video for the studio

MPEG was born to serve the “last mile” of video distribution, but some companies requested to make a version of MPEG-2 targeting studio use. This is the origin of the MPEG-2 4:2:2 profile which only supports intraframe coding and a higher number of bits per pixels.

All standards following MPEG-2, starting from MPEG-4 Visual, have had a few profiles dedicates to use in the studio.

Not strictly in the video coding area is the Audio-Visual Description Profile (AVDP), defined in MPEG-7 Part 9. AVDP was developed to facilitate the introduction of automatic information extraction tools in media production, through the definition of a common format for the exchange of the metadata they generate, e.g. shot/scene detection, face recognition/tracking, speech recognition, copy detection and summarisation, etc.

Digital video

Repeating the “MP3 use case for video” was the ambition of many. MPEG-4 Visual provided the standard technology for doing it. DivX (a company) took over the spec and triggered the birth of “DVD-to-video file” industry that attracted significant attention for some time.

Video distribution over the internet

MPEG-4 Visual was the first video coding standard designed to be “IT-friendly”. Some companies started plans to deliver video over the then internet then growing (in bitrate). Those plans suffered a deadly blow with the publication of the MPEG-4 Visual licensing terms with the “content fee” clause.

The more relaxed AVC licensing terms favoured the development of MPEG-standard based internet-based video distribution. Unfortunately, the years lost with the MPEG-4 Visual licensing terms gave time to alternative proprietary video codecs to consolidate their position in the market.

A similar story continues with HEVC whose licensing terms are of concern to many not for what they say, but for what some patent holders do not say (because they do not provide licensing terms).

Not strictly in the video coding area, but extremely important for video distribution over the internet, is Dynamic Adaptive Streaming for HTTP. DASH enables a client to request a server to send a video segment of the quality that can be streamed on the bandwidth available at a particular time, as measured by client.

In the same space MPEG produced the Common Media Applic­ation Format (CMAF) standard. Several technologies drawn from different MPEG standards are restricted and integrated to enable efficient delivery of large scale, possibly protected, video applications, e.g. streaming of televised events. CMAF Segments can be delivered once to edge servers in content delivery networks (CDN), then accessed from cache by streaming video players without additional network backbone traffic or transmission delay.

File Format

To be “IT-friendly” MPEG-4 needed a file format and this is exactly what MPEG has provided

The MP4 File Format, officially called ISO Base Media File Format (ISO BMFF), was the MPEG response to the need. It can be used for editing, HTTP streaming and broadcasting.

MP4 FF contains tracks for each media type (audio, video etc.), with additional information: a four-character the media type ‘name’ with all parameters needed by the media type decoder. “Track selection data” helps a decoder identify what aspect of a track can be used and to determine which alternatives are available.

An important support to the file format is the Common Encryption for files provided by MPEG-B Part 7.

Posts in this thread

Still more to say about MPEG standards

In Is there a logic in MPEG standards? and There is more to say about MPEG standards I have made an overview of the first 11 MPEG standards (white squares in Figure 1). In this article I would like to continue the overview and briefly present the remaining 11 MPEG standards, including those what are still being developed. Using the same convention as before those marked yellow indicate that no work was done on them for a few years

Figure 1 – The 22 MPEG standards. Those in colour are presented in this article


When MPEG begun the development of the Augmented Reality Application Format (ARAF) it also started a specification called Augmented Reality Reference Model. Later it became aware that SC 24 Computer graphics, image processing and environmental data representation was doing a similar work and joined forces to develop a standard called Mixed and Augmented Reality Reference Model with them.

In the Mixed and Augmented Reality (MAR) paradigm, representations of physical and computer mediated virtual objects are combined in various modalities. The MAR standard has been developed to enable

  1. The design of MAR applications or services. The designer may refer and select the needed components from those specified in the MAR model architecture taking into account the given application/service requirements.
  2. The development of a MAR business model. Value chain and actors are identified in the Reference Model and implementors may map them to their business models or invent new ones.
  3. The extension of existing or creation of new MAR standards. MAR is interdisciplinary and creates ample opportunities for extending existing technology solutions and standards.

MAR-RM and ARAF paradigmatically express the differences between MPEG standardisation and “regular” IT standardisation. MPEG defines interfaces and technologies while IT standardars typically defines architectures and reference models. This explains why the majority of patent declarations that ISO receives relate to MPEG standards. It is also worth noting that in the 6 years it took to develop the standard, MPEG developed 3 editions of its ARAF standard.

The Reference architecture of the MAR standard is depicted in the figure below.

Information from the real world is sensed and enters the MAR engine either directly or after being “understood”. The engine can also access media assets or external services. All information is processed by the engine which outputs the result of its processing and manages the interaction with the user.

Figure 2 – MAR Reference System Architecture

Based on this model, the standard elaborates the Entreprise Viewpoint with classes of actors, roles, business model, successful criteria, the Computational Viewpoint with functionalities at the component level and the Informational Viewpoint with data communication between components.

MM-RM is a one-part standard.


Multimedia service platform technologies (MPEG-M) specifies two main components of a multimedia device, called peer in MPEG-M.

As shown in Figure 3, the first component is API: High-Level API for applications and Low Level API for network, energy and security. 

Figure 3 – High Level and Low Level MPEG-M API

The second components is a middleware called MXM that relies specifically on MPEG multimedia technologies (Figure 4)

Figure 4 – The MXM architecture

The Middleware is composed of two types of engine. Technology Engines are used to call functionalities defined by MPEG standards such as creating or interpreting a licence attached to a content item. Protocol Engines are used to communicate with other peer, e.g. in case a peer does not have a particular Technology Engine that another peer has. For instance, a peer can use a Protocol Engine to call a licence server to get a licence to attach to a multimedia content item. The MPEG-M middleware has the ability to create chains of Technology Engines (Orchestration) or Protocol Engines (Aggregation).

MPEG-M is a 5-part standard

  • Part 1 – Architecture specifies the architecture, and High and Low level API of Figure 3
  • Part 2 – MPEG extensible middleware (MXM) API specifies the API of Figure 4
  • Part 3 – Conformance and reference software
  • Part 4 – Elementary services specifies the elementary services provided by the Protocol Engines
  • Part 5 – Service aggregation specifies how elementary services can be aggregated.


The development of the MPEG-U standards was motivated by the evolution of User Interfaces that integrate advanced rich media content such as 2D/3D, animations and video/audio clips and aggregate dedicated small applications called widgets. These are standalone applications embedded in a Web page and rely on Web technologies (HTML, CSS, JS) or equivalent.

With its MPEG-U standard, MPEG sought to have a common UI on different devices, e.g. TV, Phone, Desktop and Web page.

Therefore MPEG-U extends W3C recommendations to

  1. Cover non-Web domains (Home network, Mobile, Broadcast)
  2. Support MPEG media types (BIFS and LASeR) and transports (MP4 FF and MPEG-2 TS)
  3. Enable Widget Communications with restricted profiles (without scripting)

The MPEG-U architecture is depicted in Figure 5.

Figure 5 – MPEG-U Architecture

The normative behaviour of the Widget Manager includes the following elements of a widget

  1. Packaging formats
  2. Representation format (manifest)
  3. Life Cycle handling
  4. Communication handling
  5. Context and Mobility management
  6. Individual rendering (i.e. scene description normative behaviour)

Figure 6 depicts the operation of an MPEG-U widget for TV in a DLNA enviornment.

Figure 6 – MPEG-U for TV in a DLNA environment

MPEG-U is a 3-part standard

  • Part 1 – Widgets
  • Part 2 – Additional gestures and multimodal interaction
  • Part 3 – Conformance and reference software


High efficiency coding and media delivery in heterogeneous environments (MPEG-H) is an integrated standard that resumes the original MPEG “one and trine” Systems-Video-Audio standards approach. In the wake of those standards, the 3 parts can be and are actually used independently, e.g. in video streaming applications. On the other hand, ATSC have adopted the full Systems-Video-Audio triad with extensions of their own.

MPEG-H has 15 parts, as follows

  1. Part 1 – MPEG Media Transport (MMT) is the solution for the new world of broadcasting where delivery of content can take place over different channels each with different characteristics, e.g. one-way (traditional broadcasting) and two-way (the ever more pervasive broadband network). MMT assumes that the Internet Protocol is common to all channels.
  2. Part 2 – High Efficiency Video Coding (HEVC) is the latest approved MPEG video coding standard supporting a range of functionalities: scalability, multiview, from 4:2:0 to 4:4:4, up to 16 bits, Wider Colour Gamut and High Dynamic Range and Screen Content Coding
  3. Part 3 – 3D Audio il the latest approved audio coding standards supporting enhanced 3D audio experiences
  4. Parts 4, 5 and 6 Reference software for MMT, HEVC and 3D Audio
  5. Parts 7, 8, 9 Conformance testing for MMT, HEVC and 3D Audio
  6. Part 10 – MPEG Media Transport FEC Codes specifies several Forward Erroro Correcting Codes for use by MMT.
  7. Part 11 – MPEG Composition Information specifies an extention to HTML 5 for use with MMT
  8. Part 12 – Image File Format specifies a file format for individual images and image sequences
  9. Part 13 – MMT Implementation Guidelines collects useful guidelines for MMT use
  10. Parts 14 – Conversion and coding practices for high-dynamic-range and wide-colour-gamut video and 15 – Signalling, backward compatibility and display adaptation for HDR/WCG video are technical reports to guide users in supporting HDR/WCC,


Dynamic adaptive streaming over HTTP (DASH) is a suite of standards for the efficient and easy streaming of multimedia using available HTTP infrastructure (particularly servers and CDNs, but also proxies, caches, etc.). DASH was motivated by the popularity of HTTP streaming and the existence of different protocols used in different streaming platforms, e.g. different manifest and segment formats.

By developing the DASH standard for HTTP streaming of multimedia content, MPEG has enabled a standard-based client to stream content from any standard-based server, thereby enabling interoperability between servers and clients of different vendors.

As depicted in Figure 7, the multimedia content is stored on an HTTP server in two components: 1) Media Presentation Description (MPD) which describes a manifest of the available content, its various alternatives, their URL addresses and other characteristics, and 2) Segments which contain the actual multimedia bitstreams in form of chunks, in single or multiple files.

Figure 7 – DASH model

Currently DASH is composed of 8 parts

  1. Part 1 – Media presentation description and segment formats specifies 1) the Media Presentation Description (MPD) which provides sufficient information for a DASH client to adaptive stream the content by downloading the media segments from a HTTP server, and 2) the segment formats which specify the formats of the entity body of the request response when issuing a HTTP GET request or a partial HTTP GET.
  2. Part 2 – Conformance and reference software the regular component of an MPEG standard
  3. Part 3 – Implementation guidelines provides guidance to implementors
  4. Part 4 – Segment encryption and authentication specifies encryption and authentication of DASH segments
  5. Part 5 – Server and Network Assisted DASH specifies asynchronous network-to-client and network-to-network communication of quality-related assisting information
  6. Part 6 – DASH with Server Push and WebSockets specified the carriage of MPEG-DASH media presentations over full duplex HTTP-compatible protocols, particularly HTTP/2 and WebSockets
  7. Part 7 – Delivery of CMAF content with DASH specifies how the content specified by the Common Media Application Format can be carried by DASH
  8. Part 8 – Session based DASH operation will specify a method for MPD to manage DASH sessions for the server to instruct the client about some operation continuously applied during the session.


Coded representation of immersive media (MPEG-I) represents the current MPEG effort to develop a suite of standards to support immersive media products, services and applications.

Currently MPEG-I has 11 parts but more parts are likely to be added.

  1. Part 1 – Immersive Media Architectures outlines possible architectures for immersive media services.
  2. Part 2 – Omnidirectional MediA Format specifies an application format that enables consumption of omnidirectional video (aka Video 360). Version 2 is under development
  3. Part 3 – Immersive Video Coding will specify the emerging Versatile Video Coding standard
  4. Part 4 – Immersive Audio Coding will specify metadata to enable enhanced immersive audio experiences compared to what is possible today with MPEG-H 3D Audio
  5. Part 5 – Video-based Point Cloud Compression will specify a standard to compress dense static and dynamic point clouds
  6. Part 6 – Immersive Media Metrics will specify different parameters useful for immersive media services and their measurability
  7. Part 7 – Immersive Media Metadata will specify systems, video and audio metadata for immersive experiences. One example is the current 3DoF+ Video activity
  8. Part 8 – Network-Based Media Processing will specify APIs to access remote media processing services
  9. Part 9 – Geometry-based Point Cloud Compression will specify a standard to compress sparse static and dynamic point clouds
  10. Part 10 – Carriage of Point Cloud Data will specify how to accommodate compressed point clouds in the MP4 File Format
  11. Part 11 – Implementation Guidelines for Network-based Media Processing is the usual collection of guidelines


Coding-Independent Code-Points (MPEG-CICP) is a collecion of code points that have been assemnled in single media- and technology-specific documents because they are not standard-specific.

Part 1 – Systems, Part 2 – Video and Part 3 – Audio collelct the respective code points and Part 4 – Usage of video signal type code points contains guidelines for their use


Genomic Information Representation (MPEG-G) is a suite of specifications developed jointly with TC 276 Biotechnology that allows to reduce the amount of information required to losslessly store and transmit DNA reads from high speed sequencing machines.

Figure 8 depicts the encoding process

An MPEG-G file can be created with the following sequence of operations:

  1. Put the reads in the input file (aligned or unaligned) in bins corresponding to segments of the reference genome
  2. Classify the reads in each bin in 6 classes: P (perfect match with the reference genome), M (reads with variants), etc.
  3. Convert the reads of each bin to a subset of 18 descriptors specific of the class: e.g., a class P descriptor is the start position of the read etc.
  4. Put the descriptors in the columns of a matrix
  5. Compress each descriptor column (MPEG-G uses the very efficient CABAC compressor already present in several video coding standards)
  6. Put compressed descriptors of a class of a bin in an Access Unit (AU) for a maximum of 6 AUs per bin

Figure 8 – MPEG-G compression

MPEGG-G currently includes 6 parts

  1. Part 1 – Transport and Storage of Genomic Information specifies the file and streaming formats
  2. Part 2 – Genomic Information Representation specified the algorithm to compress DNA reads from jigh speed sequencing machines
  3. Part 3 – Genomic information metadata and application programming interfaces (APIs) specifies metadat and API to access an MPEG-G file
  4. Part 4 – Reference Software and Part 5 – Conformance are the usual components of a standard
  5. Part 6 – Genomic Annotation Representation will specify how to compress annotations.


Internet of Media Things (MPEG-IoMT) is a suite of specifications:

  1. API to discover Media Things,
  2. Data formats and API to enable communication between Media Things.

A Media Thing (MThing) is the media “version” of IoT’s Things.

The IoMT reference model is represented in Figure 9

Figure 9: IoT in MPEG is for media – IoMT

Currently MPEG-IoMT includes 4 parts

  1. Part 1 – IoMT Architecture will specify the architecture
  2. Part 2 – IoMT Discovery and Communication API specifies Discovery and Communication API
  3. Part 3 – IoMT Media Data Formats and API specifies Media Data Formats and API
  4. Part 4 – Reference Software and Conformance is the usual part of MPEG stndards


General Video Coding (MPEG-5) is expected to contain video coding specifications. Currently two specifications are envisaged

  1. Part 1 – Essential Video Coding is expected to be the specification of a video codec with two layers. The first layer will provide a significant improvement over AVC but significantly less than HEVC and the second layer will provide a significant improvement over HEVC but significantly less than to VVC.
  2. Part 2 – Low Complexity Video Coding Enhancements is expected to be the specification of a data stream structure defined by two component streams, a base stream decodable by a hardware decoder, and an enhancement stream suitable for software processing implementation with sustainable power consumption. The enhancement stream will provide new features such as compression capability extension to existing codecs, lower encoding and decoding complexity, for on demand and live streaming applications. The LCEVC decoder is depicted in Figure 18.

Figure 18: Low Complexity Enhancement Video Coding

That’s all?

Well, yes, in terms of standards that have been developed, are being developed or being extended, or for which MPEG thinks that a standard should be developed. Well, no, because MPEG is a forge of ideas and new proposals may come at every meeting.

Currently MPEG is investigating the following topics

  1. In advance signalling of MPEG containers content is motivated by scenarios where the full content of a file is not available to a player but the player needs to take a decision to retrieve the file or not. Therefore the player needs to have sufficient information to determine if it can/cannot play the entire content or only a part.
  2. Data Compression continues the exploration in search for non typical media areas that can benefit from MPEG’s compression expertise. Currently MPEG is investigating Data compression for machine tools.
  3. MPEG-21 Based Smart Contracts investigates the benefits of converting MPEG-21 contract technologies, which can be human readable, to smart contracts for execution on blockchains.

Posts in this thread