MPAI – do we need it?


Sunday last week I launched the idea of MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – an organisation with the twofold goal of 1) developing Technical Specifications of coded representation of moving pictures, audio and data, especially using artificial intelligence and 2) bridging the gap between technical specifications and their practical use, especially using “framework licences”.

The response has been overwhelming, but some have asked me: “Why do we need MPAI?”. This is indeed a basic question and, in this article, I intend to provide my answer.

The first reason

As much as VC1, VP8/VP9 and AV1 were developed because MPEG and/or its ecosystem were not providing the solutions that the market demanded, MPAI responds to the need of industry to have usable standards that allow industry and consumers to benefit from technological progress.

The second reason

The body producing standards of such an industrial and social importance should be credible. MPEG is no more, and its unknown SC 29 replacement operates in ISO, a discredited environment because of its lack of governance. The very fact that a determined and connected group of ISO entities could hijack a successful group as MPEG, with its track record serving the industry, is a proof that, at the macro level, major decisions in ISO are made because some powers that be decide that certain things should go in a direction convenient to them. Then, at the micro level, common-sense decisions like preserving MPEG plenaries where the conclusions of different groups are integrated in a single whole are blocked because “they are not in the directives” (as if hijacking MPEG was in the directives).

The third reason

The standards produced by the body should be usable. I have already written that, about 15 years ago, at what was probably the pinnacle of MPEG success, I was already anticipating the evolution of the industry that we are witnessing today. However, my efforts to innovate the way MPEG developed standards were thwarted. I tried to bring the situation to the public attention (see for instance …). All in vain. The result has been that the two main components of MPEG-H, the latest integrated MPEG project – part 2 video (HEVC) and part 3 audio (3D Audio) – have miserably failed. The hope to see a decent licence for Part 3 video (VVC) of the next integrated MPEG project – MPEG-I – is in the mists of an unknown future and may well tread the same path.

It could well happen that, in a burst of pride, VVC patent holders will want to show that they can get their acts together and deliver a VVC licence, but who guarantees that, at the next standard, the same HEVC/3D Audio pantomime will not be on stage? Can the industry – and billions of consumers – continue to be the hostage of a handful of string pullers acting in the dark?

The fourth reason

We need a North Star guiding the industry in the years to come. Thirty-two years ago, the start of MPEG was a watershed. Digital technologies promised to provide more attractive moving pictures and audio, more conveniently and with more features to the many different and independent industries who were used to handle a host of incompatible audio-visual services. Having been present then and being present now, I can testify that MPEG has delivered much more than promised. By following the MPEG North Star, industry has got a unified technology platform on which different industries can build and extend their business.

MPAI is the new watershed. I don’t know if bigger or smaller than 32 years ago, probably bigger. Artificial Intelligence technologies demonstrate that it is possible to do better and more than traditional digital technologies. But there is a difference. In the last 32 years digital audio and video have offered wonders, still they kept the two information streams isolated from the rest of the information reaching the user. With artificial intelligence, audio and video have the potential to seamlessly integrate with the many other information types handled by a device on a unified technology platform. How? Leave it to digital media and artificial intelligence experts, which have started to become an integrated community, to open their respective domains to other technologies.

Forget the past

It would be nice – and many, I for one, would thank for it – if someone undertook to solve the open problems in the use of digital media standards past. I am afraid this is an intricate problem without a unified point from which one can attempt to find a solution.

But is that a worthwhile effort? One way or another, industry has interoperable audio-visual technologies for its current needs, some even say more than it needs.

What remains of the group that did the miracle in the past 32 years is paralysed and the organisation in which it used to operate is problem-ridden and discredited. I pity the hundreds of valuable experts who are forced to face unneeded troubles.

Look to the future

Let’s look to the future, because we can still give it the shape we want. This is what the MPAI statutes suggest when they define the MPAI purpose as developing technical specifications of coded representation of moving pictures, audio and data, especially using artificial intelligence.

The task for MPAI is to call the large community of researchers from industry and academia to reach the goal to develop standards that provide a quantum leap in user experience by doing better and offering more than done so far, and by achieving a deeper integration of information sources reaching the user.

I know that the technologies in our hands have the potential to reach the goal, but only a new organisation that has the spirit, the enthusiasm and the effectiveness of the old one to deliver on the new promises can actually reach the goal.

That is the ideal reason to create MPAI. A more prosaic but vital reason to do it is that standards should also be usable.

Posts in this thread

Posts in the previous thread

List of all related articles


New standards making for a new age

Problem statement: Making standards, especially communication standard, is one of the noblest activities that humans can perform for other humans. The MPEG group used to do that for media and other data. However, ISO, the body that hosted MPEG, suffers from several deficiencies, two of which are: fuzzy governance and ineffective handling of Intellectual Property Rights (IPR), the engine that ensures tech­nical innovation-based progress. The prospects of reforming ISO are low: installing good governance requires capable leadership and solving the IPR problem is an unre­warding endeavour. Actually, the only beneficiary of such endeavours is by and large only MPEG, whose standards collect ~57.5% of all patent declarations rece­ived by ISO.

Moving Picture, Audio and Data Coding by Artificial Intelligence – MPAI is a not-for-profit organisation that addresses the two deficiencies – governance and IPR handling – by building on and innovating MPEG’s experience and achiev­em­ents and by targeting the involvement of a large community of industry, research and acad­emic experts. MPAI’s gover­nance is clear and robust, and its specifications are developed using a process that is technically sound and designed to facil­it­ate practical use of IPR in MPAI specifications.

Mission: to promote the efficient use of Data by

  1. Developing Technical Specifications (TS) of
    1. Data coding, especially using new technologies such as Artificial Intelligence, and
    2. Technologies that facilitate integration of Data Compression components in Information and Communication Technology systems, and
  2. Bridging the gap between TSs and their practical use through the develop­ment of IPR Guidelines, such as Framework Licences and other instruments.

Data include, but are not restricted to, media, health, manufacturing, autom­ot­ive and generic data.

 Governance: The General Assembly (GA) elects the Board of Directors, establishes Develop­ment Committees (DC) tasked to develop specifications and approves their TSs. Each Member appoints an adequate number of representatives in DCs. Principal Members appoint one representative in the IPR Sup­port Advisory Committee (IPR SAC), tasked to develop Framework Licenses (FWL).

Process: before a new project starts (i.e. before a Call for Technologies is issued)

  1. The IPR SAC develops a FW) that lists the elements of the future licen­ce of the TS without any indication of cost. Examples of such possible elements could be: “royalty free profile” with a given performance level, possible “initial grace period” depending on market develop­ment, possible “content fees”, possible one or more annual “caps”, a possible given ratio of user devices generating human perceivable signals vs other user devices etc.
  2. The FWL does not contain actual values such royalty levels, dates, percentage values etc.
  3. The FWL is approved by a qualified majority of Principal Members participating in the project.
  4. Each Member participating in the project declares to be willing to make available a licence for its IP according to the FWL by a certain date, and to take a licence by a certain date if it will use the part of the TS that is covered by IP
  5. Each Member shall inform the Secretariat of the result of its best effort to identify IP that it believes is infringed by a TS that is being or has already been developed by a DC.

 Method of work: the GA develops, maintains and constantly updates a work plan on the basis of Members’ inputs and responses to Calls for Interest. The GA assigns the development of a TS to a DC. The DC typically issues Calls for Evidence and/or Calls for Technol­ogy. Anybody may answer Calls for Interest, Evidence or Technology. A non-Member whose contribution submitted in response to a Call for Technology is accepted, is requested to join MPAI. DCs develop TSs by consensus. If consensus is not reached on an issue, the chair may decide to bring the matter to the attention of the GA who decides by qualified majority vote. The DC shall docum­ent which (parts of) a contribution is adopted in the TS. See here for a detailed work flow.

Membership: companies and organisations, including universities, may become Principal or As­sociated Members at their choice. Applicants can become and then remain Members by paying yearly membership fees. Only Principal Members are allowed to vote. Associated Mem­bers may join DCs and contribute to the development of TSs.

Key documents: The text above is a summary description of MPAI. The Statutes, that include detailed Procedures of work should be con­sulted for precise information. See a summary here. The Statutes are being reviewed and will be made public shortly.

A novel approach: MPAI offers a novel approach to standardisation with the following features:

  1. MPAI intends to be a broad multidisciplinary and multi-stakeholder community.
  2. Low access threshold to participate in the development of Technical Specifications: most meetings are held by teleconference sup­ported by advanced ICT-based collaboration facilities.
  3. Facilitated participation of experts who have stayed away from formal standardisation because of cost and other concerns.
  4. Framework Licences, developed through a rigorous process, expedite the use of Technical Specifications covered by IP.
  5. Timely delivery of application-driven and technology-intensive specifications.
  6. Bottom-up governance in specification development.
  7. No external constraints on members when they decide about activities.

The MPAI web site is at As the TLD suggests, MPAI is a community. Therefore, comments from the community, in particular on Statutes and Operation, are welcome. Please send your comments to Leonardo, while the MPAI Secretariat is being established.

Posts in this thread

Posts in the previous thread

List of all related articles


The MPEG to Industry Hall of fame

At the suggestion of Steve Morgan, THE [RE]DESIGN GROUP, I initiate a new “MPEG to Industry” Hall of fame complementing the MPEG Hall of fame where I highlighted those who helped make MPEG what it eventually became, besides standards development.

Suggestions are open. If you want to make a nomination please send an email to Leonardo adding the name of the nominee and a brief text explaining the contribution of the nominee to convert one or more MPEG standards into products.

[2020/07/15, Steve Morgan]

The honorable Mr. Jerry Pierce was responsible for helping Matsushita (aka Panasonic) establish the Digital Video Compression Center (DVCC), which was Hollywood’s first and foremost DVD Mastering facility.  During it’s 7 years if operations, DVCC released over 87% of Hollywood’s “A” titles on DVD.

Posts in this thread

Posts in the previous thread

List of all related articles


This is ISO – An incompetent organisation

ISO is too important to leave it in the hand of people who are catapulted to Geneva from who knows where for who knows what alchemy to serve who knows what purposes.

Of course, the way an organisation elects to hire people is their business. However, the mission of that organisation must be fulfilled. The mission “to develop high quality voluntary International Standards which facilitate international exchange of goods and services, support sustainable and equitable economic growth, promote innovation and protect health, safety and the environment” cannot stop at putting in place a process prescribing how to move a document due to become a standard from one stage to another. I mean, that could have been the end point 73 years ago when ISO was established.

I do not know what is required for economic growth or protection of health, safety and the environment, but is innovation promoted just by managing the process of standards approval? In my opinion standard is synonymous of innovation, or we are talking of rubber stamping.

Of course, innovation has probably as many faces than there are industries, probably more. Therefore the point here is not about ISO looking for and hiring superhumans competent on everything and able to discover the enabling factors of innovation, but it is about hiring people who listens to the weak or strong signals coming from the field of standardisation.

In this article I will talk about how MPEG handled reference software for its standards, an issue that goes to the core of what is a media compression standard.

In 1990 Arian Koster proposed to develop a common reference software for MPEG-1. Internet may have developed on the principle of “Rough Consensus and Running Code”, but the world of video compression was born on what I would call “Rough Consensus and Running Hardware” where each active participant developed their own implementation of a commonly (roughly) agreed specification. Comparing results was not easy. In the COST 211 project satellite 2 Mbit/s satellite links were used to interconnect different hardware implementations. In MPEG-1 (but that was true of MPEG-2 as well) every active participant developed their own code and brought results of their simulations of Core Experiments. By proposing to create common code bases, Arian opened the doors of a new world to MPEG.

Arian’s proposal notwithstanding, there was not a lot of common code for MPEG-1 and MPEG-2, but in the mid 1990s his ideas were fully implemented for the MPEG-4 reference software – audio, video and systems. That was more than 10 years after Richard Stallman had launched the GNU Project. In a completely different setting, but with comparable motivation, MPEG made the decision to develop the reference software collaboratively because better software would be obtained, the scope of MPEG-4 was so large that no company could probably develop it all and a software implementation made available to the industry would accelerate adoption of the standard.

In those years, Mike Smith, the head of ISO’s Information Technology Task Force (ITTF), was of great help. He stated that ISO was only interested in “selling text”, not software, and allowed MPEG to develop what was called the MPEG-4 “copyright disclaimer” that contained the following elements

  1. The role of developer and contributors
  2. The status of the software as an implementation of the MPEG standard
  3. Free licence to use/modify the module in conforming products
  4. Warning to users that use of the software may infringe patents
  5. No liability for developers, contributors, companies and ISO/IEC for use/modify the software
  6. Original developer’s right to use, assign or donate the code to a third party.

For sure the MPEG-4 copyright disclaimer was rather disconnected from the GNU Public Licence (GPL), but it did serve MPEG purposes well. All MPEG reference software was made available on the ISO web site for free download. It is a known fact that many use MPEG reference software as the uncontroversial and unambiguous way to express the intention of the (textual) MPEG standard.

The copyright disclaimer was used for about 15 years, until large software companies in MPEG expressed their discomfort for it. At that time many companies were already using modified reference software in their products. This was allowed by the copyright disclaimer but handling software with different licences in their products was not desirable. MPEG then opted for a modified BSD licence. “Modification” meant adding a preamble to the BSD licence: “The copyright in this software is being made available under the BSD License, included below. This software may be subject to other third party and contributor rights, including patent rights, and no such rights are granted under this license”. MPEG called this the MXM licence because it was developed for and first applied to the MPEG Extensible Middleware standard.

One day, probably some 25 years after Arian’s proposal, the ISO attorney Holger Gehring realised that there was one, actually more issues with the reference software. As MPEG used to meet in Geneva every 9 months because of the video collaboration with ITU-T, the MPEG chairs had several sessions with him and his collaborators until late at night for a couple of MPEG meetings. We discussed that, for MPEG people, the textual version of standard was important, but the software version, at least for some types of standard, was more important and that the two had equal status, in the sense that textual version expressed normative clauses in a human language, while the software version expressed the same normative clauses in a machine language.

After reaching an agreement on this principle, the discussion moved to the licence of the software. I recalled the agreement with Mike Smith and that all MPEG reference software was posted on the ISO web site for free download and advocated the use of the MXM licence. This was agreed and I undertook to convince all MPEG subgroups to adopt it (some were still using the copyright disclaimer). With some difficulty I got the support of all subgroups to the agreement.

I communicated the agreement to the ISO attorney but got no acknowledgement. Later I learned that he had “left” ISO.

We kept our part of the agreement and released all reference software with the MXM licence. A little before I left ISO I learned that several standards with reference software in it had been withheld because the reference software issue had surfaced again, that they were discussing it and that they would inform us of the result…

ISO has to decide if it wants to “promote innovation” or if it wants to be feudal. People in the field have acquired a lot of competence, validated by some of the biggest software companies, that would allow ISO competently to address the issue of software copyright in standards.

Posts in this thread

Posts in the previous thread

List of all related articles


This is ISO – An obtuse organisation


This is the 4th episode of my series about the deficiencies of the organisation that calls itself International Organisation for Standardisation and claims “to develop high quality voluntary International Standards which facilitate international exchange of goods and services, support sustainable and equitable economic growth, promote innovation and protect health, safety and the environment”.

I do not intend to make comment on how the mission is achieved – that is for everybody to judge – but on what is further up and produces the claimed international standards.

In the first episode This is ISO – A feudal organisation I analysed how the ISO structure is permeated by an attitude befitting more the early Middle Ages than the end of the second decade of the third millennium. In the second episode This is ISO – A chaotic organisation I highlighted how standards making is negatively impacted by a work allocation that responds to a logic of power, not of efficiency. In the third episode This is ISO – A hypocritical organisation I described how the ISO/IEC directives make minute prescriptions on debatable or irrelevant issues, while big issues like governance are simply swept under the carpet.

In this episode I will analyse how ISO investments in Information Technology (IT), an indispensable tool in today’s standards making process, go awry causing harm to the very processes the IT investment is supposed to improve. This is a serious issue with two faces: thousands of people are forced to do useless work and National Body money is incompetently squandered.

Making standards editing digital

The idea of digitising the workflow that produces standards is sound. However, whether it makes also sense depends on how it is designed.

Most of the work MPEG was doing at the time of the MPEG-1 Committee Draft (1991) was already electronic, but few carried what were at that time heavy laptops. The MPEG-1 editors sent files to the secretariat, but I do not know if the secretariat forwarded files or paper to the ISO Central Secretariat (CS). Also, I do not know in which form ISO CS sent the drafts back to the editors for review. I would not be surprised if all correspondence were paper-based.

Two years later, at the meeting that approved the MPEG-2 CD (November 1993), all MPEG processes were electronic, but paper still ruled.  Notwithstanding the anti-environment record 1,000,000 photocopies, the event marked the beginning of a golden age that lasted about 20 years. Editors sent their MS Word files to the secretariat who reviewed and sent them to the ISO CS who would annotate the files received from the editors and send the annotated Word files back to the editors.

A golden age is called such because people can later refer to it as a time when everything ran so smoothly. In fact the standard development workflow was already digital. The dark age came when ISO thought they should “improve” it. They commissioned a system that converted word files into an SGML format. This would be the master file that would be annotated, converted to PDF, and sent to the editors. I imagine ISO CS thought this was a great idea, but the result was that editors could not access the SGML editor and even if they could, they would continue editing their Word files. The PDF caused all sorts of troubles, the most remarkable being the loss of all links from one section to another section of the file, sometimes running by the hundreds in a 500+ page document. The editors went crazy trying to recover the vital information lost.

Digital support to standards making

In November 1995 MPEG created a system that allowed members to upload their contributions to a remote server, and browse through and download selected documents. In the 3 weeks around an MPEG meeting there used to be very high peaks of traffic because hundreds of people would simultaneously download hundreds of MByte-size documents

It took many years for ISO to develop a system for uploading and downloading documents for its many committees. It is not known for what purposes the ISO system has been designed, maybe the secretariats who upload one file a day at best, and download two at most.

One day an ISO manager decided that MPEG, too, should use the ISO system. It took some time to convince him that, during an MPEG week, the ISO system might no longer be accessible by all ISO users just because of the MPEG traffic would cause denial of service. Obviously, a system can only be used for what it has been designed.

At one time the ISO CS wanted to know more about the MPEG system. A teleconference was held, and the functionalities of the system explained. ISO people said that they were redesigning a new system and that we would hear back from them. That never happened.


Precept #1 of a project to design a software system is to get the requirements from the people who will eventually use it. This rule is not in force in ISO because ISO is a feudal organisation where the duke decides, and the peasants comply.

We do not live in the year 800 BC. Obtuse decisions impact people who love what they do (for free) and expect that the system, if not help them, at least does not get in the way.

A disclaimer

I did write this in an earlier post, but it is important to repeat it here. ISO is an organisation with an important mission and most of the people from ISO CS that I have met were courteous, competent and ready to help. They were just trying to cope with the deficiencies of the system. ISO problems lie at the top echelons, at least in the current days.

Like the adage goes, a fish rots from the head.


Posts in this thread

Posts in the previous thread

List of all related articles