Category: Uncategorized

Some have it straight

Developing standards is one of the greatest endeavours humans can do for their fellow humans. A well conceived and rightly implemented standard is the lubricant that makes society more efficient. Communication standards are even more valuable because they enhance the most powerful feature of humans to do things with other humans in a coordinated way.

In the 20th century public authorities and entrepreneurs vied in inventing new ways to communicate within a community to the exclusion of others. My best example is ITU-R Recommendation 624 “Television systems” where minuscule and large countries alike compete in adding footnotes stating that they reserve the right to adopt different frequency tolerances and the like.

MPEG was born in 1988 in reaction to the balkanised state of analogue media standards and to the attempts by countries and companies to carry over the analogue media paradigm to digital media.

The rest is history. MPEG standards have universal coverage, a result achieved without the intervention of public authorities because companies were convinced that a global market was not only larger than the sum of individual markets but also more efficient. Consumers reaped the benefits.

The reality of today, however, is that the MPEG model is not adopted in all markets and there is an interest on the part of some powerful parties to make the MPEG model a temporary exception to the rule, starting with “Nothing will change”.

If this will happen, it will not be because people have lost faith in the MPEG model, but because of ferocious arm-twisting behind the scenes.

Some people have a straight back and some have not. Some others get it only too well. Others simply don’t get it.

Posts in this thread

The MPEG exception

In the past 18 months, I have provided ample information on how MPEG was established, how it works, what it has done and is doing, and why it is different than other committees. In the last article Why is there a battle around MPEG?, I have identified another point of divergence. Patent declarations that relate to MPEG standards are 57.5% of all patent declarations submitted to ISO. MPEG is special because the next ISO committee in terms of number of patent declarations submitted to ISO has 13% share. JTC 1 itself has 81% of all patent declarations submitted to ISO.

In this article I would like to explore another difference, the nature of MPEG standards vs other JTC 1 standards.

Let’s start from an extended version of the table where the % refers to patent declarations submitted to ISO for all JTC 1 standards.

Committee Title # Pat. Decl. %
MPEG Coding of Moving Pictures and Audio 1857 69.76
SC 31 Automatic identification and data capture techniques 399 14.99
SC 27 Information security, cybersecurity and privacy protection 125 4.70
SC 25 Interconnection of information technology equipment 92 3.46
JPEG Coding of digital representations of images 80 3.01
SC 17 Cards and security devices for personal identification 56 2.10
SC 23 Digitally Recorded Media for Information Interchange and Storage 22 0.83
SC 6 Telecommunications and information exchange between systems 22 0.83
SC 34 Document description and processing languages 3 0.11
SC 35 User interfaces 2 0.08
SC 37 Biometrics 2 0.08
SC 28 Office equipment 1 0.04
SC 36 Information technology for learning, education and training 1 0.04

We see that Data (MPEG, SC 31 and JPEG), Security (SC 27), Communication (SC 25 abd SC 6) and Physical Media (SC 17 and SC 23) have a number of patent declarations expressed with more than 1 decimal digit.

This should be no surprise. At the risk of being criticised, I would state that the first 8 committees are not typical Information Technology (IT) committees, but Information and Communication Technology (ICT) committees because they handle information that does not only reside in a device, and hence is not typically the object of a standards, but is expected to move between devices. For instance there is no single patent declaration for SC 24 Computer graphics, image processing and environmental data representation.

Indeed, save for the first 8 committees in the table, the main target for standardisation in the other committees is architectures, frameworks, principles, guidelines etc.

Some JTC 1 standards are indeed developed outside of JTC 1. There are many organisations who request, e.g., JTC 1 to become Suppliers of Publicly Available Specifications (PAS). Those organisations who are granted that status have a preferential channel to make their specifications International Standards.

The Virtual Reality Modeling Language (VRML) was a PAS. JTC 1/SC 24 took VRML97 and made it ISO/IEC 14772-1:1997 Information Technology — Computer Graphics and Image Processing — The Virtual Reality Modeling Language — Part 1: Functional Specification and UTF-8 Encoding.

The justification for a making an ISO standard from a PAS is that the PAS can be developed with greater freedom and at a higher speed. The ISO process puts an ISO stamp on the PAS without changing the specification. Again at the risk of being criticised, I think that standards are too important to make them international standards by letting them cut corners.

MPEG has received a limited number of PAS with a request to convert them to ISO standards, but done so for none. MPEG has issued calls to which  some organisations that had produced PAS’s have responded. MPEG has developed standards based on those responses. An example is MPEG-M to which the Digital Media Project has been a major contributor. This is fine. The way technology is developed should be unconstrained, but there should be a single route through which technologies become international standards.

The Alliance for Open Media (AOM) specification known as AV1 is a PAS. One day, but not today, AV1 may became an MPEG standard.

MPEG is an exception, the question is whether it will continue to be so.

Posts in this thread

Why is there a battle around MPEG?

View powerpoint presentation

MPEG is known for a variety of reasons. It is the group that carried over analogue television and made it digital, multiplying by orders of magnitude the number of channels, it opened the way to new business models for audio, it enabled carriage of digital media on the internet, added media to the mobile experience, is poised to make immersive media real and genomics affordable, and more.

MPEG achieved these goals – and keeps on setting, working on and achieving more goals – with technical specifications of minuscule material or immaterial things that have a big impact on devices and service delivering infrastructures that are worth billions of USD.

I never tire of saying that every year products that rely on MPEG standards to function or to be attractive to buying global customers are worth more than 1 trillion USD. Similarly for services, which are worth more than 500 billion USD p.a.

There is something emblematically immaterial that has driven the success of MPEG – Patents, the engine that has allowed the MPEG machine to pile up records.

When MPEG developed its first and second standards it could draw from decades of research in audio and video coding. MPEG-1 and MPEG-2 were extremely successful and patent holders were handsomely rewarded. This encouraged more researchers and companies to file more patents so that, when MPEG engaged in new generations of coding standards it could draw from a wider and fresher set of technologies.

The story is not without hiccups. Overall, however, it is a success story that puts MPEG on a different league than any other ISO committee. This is shown by the figure below

You see that patent declarations made to ISO by entities believing they hold patents relevant to MPEG standards are 57.5% of ALL patent declarations received by ISO. The next committee is JTC 1/SC 31 Automatic identification and data capture techniques which totals less than ¼ of MPEG patent declarations (and it is a Subcommittee). All other committees mentioned in the table are JTC 1 SCs. Other ISO TCs have typically a low or even zero number of patent declarations.

You can get the data yourself from the ISO website.

MPEG should stay as the defender of a “business model” that has offered a relentless expansion of the business to all facets of the media industry and to consumers the possibility to enjoy newer and fresher experiences.

Probably you can answer yourself why there is a battle around MPEG.

Posts in this thread

Strategic planning for MPEG

In Leonardo: who I am, what I did so far and what I plan to do next, I have given some information on myself and listed my challenges in standardisation. In this article I intend to provide some more details about the challenges of a strategic planning for an organisation as special as MPEG. This is the list

  1. Uphold the role of MPEG as the unmastered and non-discriminatory source of digital media standards
  2. Expand the use of MPEG standards to insufficiently reached market areas
  3. Keep MPEG constantly abreast of technology and market development
  4. Develop standards in emerging fields of MPEG’s traditional media domain
  5. Collaborate with other committees by developing joint standards when mutually convenient
  6. Cultivate new and enhanced relationships with industry, MPEG client industries and academia
  7. Enhance the MPEG image and standing across the industry
  8. Reach out to other domains for which data compression may be a business booster
  9. Establish an ongoing process to assess MPEG’s relevance in fulfilling its mission
  10. Defend the MPEG business model by adapting it to changing conditions
  11. Define and establish a management-level succession process based on community preferences.

Uphold the role of MPEG as the unmastered and non-discriminatory source of digital media standards

Most ISO committees are the direct expression of a particular industry. Most secretariats are the expressions of the policies of the country in which the secretariat is located. MPEG was established having the opposite in mind, namely a committee that would develop standards that were industry and country agnostic. The first standard MPEG-1 targeted products and services for the Telecom and Consumer Electronics (CE) industries, the second standard MPEG-2 added the broadcasting industry. The third standard MPEG-4 added the Information Technology (IT) and mobile industries.

For 30 years MPEG has been remarkably able to keep its standards country-, industry-, company-, individual-independent. What will happen if, say, Information Technology took control of MPEG? What will happen if a major country took control of MPEG? What will happen if an alien business model took control of MPEG? What will happen if a major company took control of MPEG?

These are very serious questions and I am not sure if the global industry and world population would like the answer to these questions.

There is a second aspect, somehow related with the above, that need to be made explicit. This is the need to keep MPEG as the non-discriminatory source of digital media standards. For someone who has operated in the unmastered world of MPEG for many years, an MPEG standard is the result of the efforts of those who wanted to participate in the endeavour that produced the standards. No one should be denied participation. This does not mean that anybody can actually add technology to a standard, because MPEG is an extremely competitive environment striving to produce the best. Therefore only the best technologies are selected.

Once the standard has been produced, it should be accessible to all those willing to pay for the benefit of using excellent technology.

The idea that only some qualify to contribute to the standard or to use the standard seems to take hold in some quarters. This is an idea that MPEG should fight. MPEG should be unmastered and not discriminate against anybody.

Expand the use of MPEG standards to insufficiently reached market areas

I am not negative to the fact that in some market areas MPEG standards are not used to the extent the quality of the technology would promise. The worst that can happen to an organisation is that it occupies all the space and its members are left without goals to achieve. For instance digital cinema does not use MPEG standards and production uses only partly MPEG standards. These, however, are not real concerns because they are markets where devices or applications are counted by the thousands.

The real concern is video distribution over the internet, because distribution is what MPEG is mostly about, but MPEG never had an easy life in this segment. In the early 2000 there were companies eager to try video and audio distribution on this new medium using MPEG-4 Visual. No way, the MPEG Visual licensing terms were disappointing. MPEG-4 AVC fared definitely better with business-friendly licensing terms and a single patent pool. AVC is universally used even though competitors like P8 and VP9 did emerge. The problem is that AVC is a 15+ year old technology and what was due to be its successor HEVC is definitely underperforming with reportedly only 12% market share. Some blame licensing terms and multiple patent pools and a large number of patent holder not belonging to any patent pools. MPEG has developed DASH and CMAF and can offer the full protocol stack with its media standards. The streaming market, however, is taking different directions..

Keep MPEG constantly abreast of technology and market development

There is not doubt that MPEG knows how to stay abreast of technology development. Almost all of its standards are anticipatory of products, services and applications to come. Currently MPEG is engaged in several technology-deep investigations. One is Video Coding for Machines (VCM). VCM is expected to be the standard that allows remotely placed cameras to extract machine-understandable information, compress it and send it to a machine to use. The problem is that machines can be good for routine work, but what if something happens remotely and a human must intervene? The same information designed for machines should be usable by a human, possibly with the best quality that regular video coding can yield. Another exploration is video coding based on deep neural networks (DNN). We already know that some tools used e.g. in VVC could be replaced with more efficient DNN-based tools. The idea is to investigate the improvement of DNN based video coding technologies – both traditional schemes with some tools replaced or end-to-end DNN coding systems – over existing MPEG standards.

This is all good and no one can blame MPEG for developing the standards that have brought industry convergence, catered to its continuous development and made media a daily continuous experience. However, MPEG, as a group, is not aware of what is happening in the market. This should change. The idea is to create a Market needs group, the counterpart of the technology driven Requirements group. By having these two groups interacting we can still rely on a technology-based forward looking group mitigated by a market-savvy group, as depicted in the figure below.

Develop standards in emerging fields of MPEG’s traditional media domain

This strategy element confirms what we have been saying all along. MPEG is well aware of what the world of digital media technology can offer because is members are for most part (~3/4) from industry and for a smaller part (~1/4) from academia. The core of MPEG customers is interested in media standards and MPEG has much to offer in terms of standards that are close to availability or still under investigation, e.g.

  • Video-based Point Cloud Compression (V-PCC)
  • Graphics-based Point Cloud Compression (G-PCC)
  • Neural Network Compression
  • MPEG Immersive Video (MIV)
  • MPEG Immersive Audio (MIA)
  • Session based DASH operation
  • MPEG-I Scene Descriptions
  • Low Complexity Video Coding Enhancements
  • Deep Neural Network (DNN) based Video Coding
  • Video Coding for Machines

And many other standards.

Collaborate with other committees by developing joint standards when mutually convenient

In companies and standards alike the NIH (Not-Invented Here) principle rules. This is understandable, because if you have been working in a field for long enough you think you know everything in your field. The problem is that in our age technology is fluid. Something that is developed for a purpose, might equally well be usable for another purpose. I mention a case that MPEG is aware of where technologies for video coding or media file format have been successfully adapted for use in compressing the reads from high speed DNA sequencing machines. MPEG should embrace collaboration with other groups even more than it has done so far. Three examples are Big Data, for which MPEG has already developed Network Based Media Processing, Artificial Intelligence for which MPEG is developing Neural Network Compression and investigating AI-based video coding and Internet of Things for which MPEG is working on extending its Internet of Media Things standard keeping an eye on Video Coding for Machines.

Cultivate new and enhanced relationships with industry, MPEG client industries and academia

MPEG has not been shy to make the first step approaching the industry. MPEG can wave the flag of being a committee that is agnostic to countries and industries, but this comes at a high cost because your work can fall in a vacuum.

For years MPEG has been the merchant who devised all sorts of tricks to get the attention of different industries through their fora. The efforts have been rewarded: MPEG regularly talks with the main bodies representing broadcasting, fixed and mobile telecommunications, and package media at the international and regional level.

MPEG has also aggressively sought to collaborate with other bodies in the development of common standards, the best examples being the common video coding standards developed with ITU-T. MPEG has developed standards on commission from other bodies such as JPEG and 3GPP.

MPEG can also boast to have an extended community of followers in industrial research and academia. Indeed ¼ of its members are from academia.

Is this enough? No, it used to be enough, but it is no longer sufficient. MPEG makes in advance investigations that should involve academia and MPEG needs in advance information on the technologies that generate the data that it will be asked to reduce to a manageable form. It is a new form of deeper liaison that should be developed and implemented. Then, of course, you never talk often enough with your customers…

Enhance the MPEG image and standing across the industry

MPEG has a unique brand and its logo is universally known, but the fact that there are market segments where MPEG don’t play the role they could is an indication that the MPEG imaged must be beefed up. Of course, I am not talking of a marketing exercise, but of something that goes at the roots of what MPEG is: the principles that guide its activity that have kept making bigger the lowest organisational unit in ISO; its business model that guarantees that there is always new technology coming in because there is old technology that is being remunerated; its structure that combines flexibility with identification of responsibility; uniquely competent and dedicated personnel; a modus operandi that is the result of decades of honing; the academic and industrial network that has MPEG as its hub; the experience accumulated over the years and the ability to adapt itself to new conditions; and its technical and market achievements.

MPEG must define this multi-threaded presence and task a revamped Liaisons and Communication group to establish and maintain it.

Reach out to other domains for which data compression may be a business booster

The proposal to identify data compression as one of the axes of the information society failed, but the idea that generated it is pretty much alive. Countless application domains generate data: media, industry 4.0, health, automotive to mention a few. Most of these data are simply not used or seldom used because there is no economic means transmit, store or process them. The MPEG-G standard proves that MPEG technologies can be used to make an application domain flooded with data more efficient.

Establish an ongoing process to assess MPEG’s relevance in fulfilling its mission

MPEG likes to boast the fact that it is a working groups but it has produced more standards than any other JTC 1 subcommitee. Great, one could say, but that statement could be compared to the statement of a book publisher who publishes more book than its competitors, but its books pile up in its warehouses.

Fortunately that is not the case. Products and services that use in a determinant way MPEG standard are worth 1.5 trillion dollars a year.

This does not mean that the efficiency of MPEG has reached its limits. Before I have mentioned the need to create a Market needs group to compensate the exclusive technology driver of technology in MPEG. That, however, is not sufficient. MPEG must be able to answer the question: why did a standard A on which so many efforts have spent did not have the adoption that it deserved? Was the idea wrong? Were the requirements wrong? Were the technologies used in the standard unsuitable? Was profiling wrong? Was it because there was another better solution on the market? If so why were we not able to add the right requirements? Was it because the licensing terms were rejected? And you can add more.

If we will be able to answer these and other questions we will probably not be able to save the standard, but we can probably inject into our standard development process feedbacks that we got from unsuccessful standards.

Defend the MPEG business model by adapting it to changing conditions

I continue to believe that the MPEG business model of making good standards that reward good technologies is the best in the context of a rapidly evolving technology scenario. It is not the only one, however. You can develop option 1 (a.k.a. royalty free) standards that only use 20+ year old technologies or you can develop royalty free standards by buying a set of patents that allow you to build a codec with a performance that is sufficient for your needs etc.

It is clear that the MPEG business model is under threat and that something must be done to defend the MPEG business model – the only one that can provide the best standard at a given time – against those threats.

There are probably several ways. One has been attempted with Essential Video Coding (EVC). Under changed conditions, it is the same approached used in MPEG-2 times. Backward Compatible (BC) MPEG-2 Audio was not faring well, so a Non-Backward Compatible (NBC) MPEG-2 standard was developed in the hope that those engaged in BC would try harder to counter NBC. Eventually BC people lost but MPEG gained because MPEG-2 AAC is the progenitor of the extremely successful MPEG-4 AAC standard.

EVC is clearly a competitor to HEVC. If HEVC will have a good licence EVC will fail, but if HEVC continues have a complex situation, EVC will win. In both cases MPEG will gain.

Define and establish a management-level succession process based on community preferences.

MPEG is not a hierarchical organisation. This does not mean that it does not need leaders. ISO gives the secretariat a lot of power in nominating leaders. This may be good in other areas but it is not the ideal solution for MPEG. Such an ideal solution may not be a plain election, but for sure the community should have a say in how leaders are appointed.

Posts in this thread

Leonardo: who I am, what I did so far and what I plan to do next

I have written this article because I thought that, at this particular juncture, it would be useful to tell more about myself. I assume I am not unknown to quite a few people, but I think that it could be useful to put in a succinct form some information that summarises the responses to the who and what in the title.

Not that such information is “hidden”. Over the years I did spend some time to document some of the endeavours I happened to be part of or initiated or managed. Although a little out of style, https://www.chiariglione.org/ can be taken as the starting point of a navigation, in particular my web page and Riding the Media Bits.

Leonardo, the man

I am an Italian citizen by birth and a convinced supporter of the vision of a united Europe since my high scool days. However, I like to think I am a citizen of the world. I live in Villar Dora, province of Turin. Google maps provides a view of my house with a glimpse of the vineyard surrounding it. Originally, the house was the summer house of my grandfather, a farmer. At that time all farmer houses in town had a vineyard grown to produce grapes to make wine. There are still some farmers in town, but no one is left with a vineyard. While I am no farmer, I still produce some wine for my consumption.

I am father of three kids who have left the nest some time ago. My kids have 3 children, two girls and one boy.

Leonardo’s education

I graduated in Electronic Engineering from the Polytechnic University of Turin. At that time Italy had just one academic degree, equivalent to today’s Master’s degree. Before my graduation, I applied for a Japanese government scholarship, then went to Japan and got a Doctor’s degree in Electrical Communication from the University of Tokyo. I am told I was the first Caucasian to get a Doctor’s degree from the University.

Leonardo’s work experiences

CSELT, at the time the research centre of what is called today TIM-Telecom Italia, hired me to do research in video coding for visual telephony. When I left the company in 2003, I was Director of the Multimedia Division with 30+ researchers. Throughout my career at CSELT, I was involved at different levels of responsibility – leader, participant or manager of resources – in some 60 collaborative research projects, mostly at European level. This table provides an overview of all the research projects I was involved in.

Since 2003 I am the CEO of CEDEO.net, a technology company in Turin developing advanced services and products. The main services are WimTV and Stream4U. The main product is TVBridge, based on MPEG-A’s Multimedia Linking Application Format. The main product under development is DNASearch, based on the MPEG-G standard. Since 2011, I am also the CEO of WimLabs, a sister company of CEDEO.net focused on the commercialisation of products developed and the provision of services enabled by CEDEO.net. The WimLabs website provides an overview of all products and services commercialised by WimLabs.

Leonardo’s standards experiences

Over the years I participated in various standards committees. At the European level I joined CEPT committees (before ETSI was established) and then the ETSI JTC where I helped to create the path for acceptance of the MPEG-2 standard in Europe, before DVB had even started.

At the international level, I joined ITU-T and ITU-R committees. In particular I joined the first few meetings of the Okubo group who developed H.261 recommendtaion. I attended ITU-R meetings especially to promote adoption of the MPEG-2 standard. I was/am the Head of the Italian Delegation to JTC 1, SC 2 and SC 29. I attended JTC 1 meetings on several occasions, starting from the time I lobbied to convince SC 2 and then JTC 1 to create SC 29. I am the Chairman of the Italian SC 29 committee.

I founded and held the rapporteur role of the Moving Picture Experts Group of SC 2/WG 8 (1988-90) and the convenor role of SC 29/WG 11 (1990-) under SC 2 and then SC 29.

I founded and chaired DAVIC – Digital Audio-Visual Council (1994-95), FIPA – Foundation for Intelligent Physical Agents (1996-1988), OPIMA – Open Platform Initiative for Multimedia Access (1997- 1999) and DMP – Digital Media Project (2003-2015). I was also the Executive Director of the Secure Digital Music Initiative (1999-2001), an industry forum with the mission to develop specifications for secure digital music delivery.

Leonardo’s academic experiences

These are my academic experiences

  1. In 1986 I founded the International Workshop on HDTV which I chaired until 1994. I was the editor of all Proceedings of the HDTV Workshop until 1994.
  2. In 1989 I founded and was the Editor-in-Chief of Signal Processing – Image Communication, a EURASIP journal, until 1999.
  3. In 2004-2005 I was Professor at the Information and Communication University in Daejeon, Korea.
  4. In 2011 I was the Editor of the book The MPEG Representation of Digital Media published by Springer Science.
  5. I have organised a total of 18 conferences related to digital media, including one Picture Coding Symposium and one Packet Video Workshop,
  6. I have written some 200 technical and strategy papers, mostly invited, some submitted to technical journals and some delivered at conferences.

In 2002 I received the Doctorate “honoris causa” from Instituto Superior Técnico (Lisbon) and the EURASIP Meritorious Service Award. I am Chevalier de l’Ordre des arts et des lettres (France) since 2003 and Membre de l’Académies des technologies (France) since 2008.

Leonardo’s awards

I have received several awards (the text of the awards can be found here):

  1. the Information and Communication Society of Japan Award (1995)
  2. the International Institute of Communications Award (1997)
  3. the Society of Photo-Optical and Instrumentation Engineers Award (1998)
  4. the Kilby Foundation Award (1998)
  5. the IEEE Masaru Ibuka Consumer Electronics Award (1999)
  6. the IBC John Tucker Award (1999)
  7. the Edward-Rhein Foundation Award (1999)
  8. the SMPTE James Leitch Gold Medal Award (2002)
  9. the NAB Award of Honor (2003)
  10. the Charles F. Jenkins Lifetime Achievement Award (2018).

I am Honorary member of SMPTE since 2014.

Leonardo’s vision

In 1986, when the CCIR failed to approve the HDTV recommendation, I called the technical and business world and established the International Workshop on HDTV. I chaired the Workshop until 1994. The Workshop attracted hundreds of participants and acted as a technical forum that enabled eventual convergence of diverse regional views on the next form of television experience.

In 1987 I proposed the creation of an experts group on moving picture coding. The original idea was stimulated by my years of research in video coding and telecom standardisation that did not have an impact on the market. I tried to find a way to make available video communication end device by blending the manufacturing capability of the consumer electronics industry with the infrastructure of the telecom industry and the content offer of the broadcasting industry. All this crossing what were at the time strong industry barriers. In 1988 this idea became the Moving Picture Experts Group (MPEG). Today MPEG has become the established source of digital media standards that have caused industry convergence and made digital media available to billions of people. MPEG has received several Emmy Awards.

In 1994 I launched the Digital Audio-Visual Council (DAVIC), an industry forum with the mission to promote the success of emerging digital audio-visual applications and services, by the timely availability of internationally agreed specifications of open interfaces and protocols that maximise interoperability across countries and applications/services. DAVIC reached a membership of more than 200 companies and developed two editions of its comprehensive specifications that were transposed to ISO/IEC standards (the ISO/IEC 16500 suite).

In 1996 I launched the Foundation for Intelligent Physical Agents (FIPA), a non-profit organisation with the mission to promote the development of specifications of generic agent technologies that maximise interoperability within and across agent-based applications. FIPA developed 2 editions of its specifications. An open source software implementation of the FIPA specification (JADE), developed by one of my collaborators at CSELT, is used in several application domains.

In 2003 I launched the Digital Media Project (DMP), a not-for-profit organisation with the mission to promote continuing successful development, deployment and use of Digital Media that respect the rights of creators and rights holders to exploit their works, the wish of end users to fully enjoy the benefits of digital media and the interests of various value-chain players to provide products and services. DMP developed 3 editions of its specifications which were implemented in open source software (Chillout). DMP members proposed and made major contributions to MPEG-A’s Media Streaming Application Format (ISO/IEC 23000-5) and to MPEG-M suite of standards (ISO/IEC 23006).

Leonardo’s challenges

I see my next challenges in two domains: in my companies and in standardisation.

The challenges for my companies are to make WimTV the natural choice for those who want to do business with video on the web, TVBridge the normal way for broadcasters to augment their viewers’ experience and DNASearch the indispensable companion of all humans following their lives from cradle to grave.

I am sure that I can meet the challenge because I can rely on the support of my family and CEDEO personnel.

My challenges in standardisation are to:

  1. Uphold the role of MPEG as the unmastered and non-discriminatory source of digital media standards
  2. Expand the use of MPEG standards to insufficiently reached market areas
  3. Keep MPEG constantly abreast of technology and market development
  4. Develop standards in emerging fields of MPEG’s traditional media domain
  5. Collaborate with other committees by developing joint standards when mutually convenient
  6. Cultivate new and enhanced relationships with industry, MPEG client industries and academia
  7. Enhance the MPEG image and standing across the industry
  8. Reach out to other domains for which data compression may be a business booster
  9. Establish an ongoing process to assess MPEG’s relevance in fulfilling its mission
  10. Defend the MPEG business model by adapting it to changing conditions
  11. Define and establish a management-level succession process based on community preferences.

I am sure these standardisation challenges can be met. My confidence comes from the fact that MPEG is a large organisation that relies on the best and most dedicated experts a standards committee can hope to attract, has a solid organisational structure, enjoys an excellent reputation, and boast a universally known brand and loyal following both in academy and industry.

MPEG Future has published a vision that includes standardisation goals and an assessment of MPEG’s organisational capabilities. Industry is actively reacting to MPEG Future’s vision.

Yes, there will be challenges. There have always been challenges in the last 32 years. The next may be the biggest ever, but I think I know how to face them, as a group.

Posts in this thread

The value of MPEG standards

Introduction

MPEG can certainly be proud of the valuable standards it has produced: MPEG-1, MPEG-2, MPEG-4 AVC, MPEG-4 File Format and, more recently HEVC, MMT, DASH and many more.

Why are MPEG standards valuable and what can we do to add more value to them?

Why are MPEG standards valuable?

There is no single answer to this question. So, let’s analyse the causes that make MPEG standards valuable. While we are at it. later we wik let’s also analyse why some standards. on which great hopes were laid, did not deliver.

MPEG-1 (Systems, Video and Audio layer II)

The standard was designed for what would be the first device handling digital media, was interactive and and would serve the telecom and Consumer Electronics (CE) markets. Eventually that device did not fly, but the technology that made it possible was later reconfigured to make a “lower class” DVD player. Success was incredible, with a total of 1 billion devices manufactured by many companies.

MP3 (MPEG-1 Audio layer III)

MP3 turned out to be an incredible success from a totally unexpected quarter. When MPEG-1 was approved in 1992, MP3 offered transparency at 192 kbit/s for stereo music, ¼ less than MPEG-1 Audio layer II at 256 kbit/s (MP2) and at ½ of the MPEG-1 Audio layer I bitrate of 384 kbit/s (MP1). Complexity, however, was daunting (at that time). While MP2 and MP1 were implemented in ICs for mass market products, in the first years MP3 was implemented with sparse logic for professional equipment. Until, I mean, the impossible happened: software implementing the MP3 decoder in real time on a PC became available, non-real time encoding software became available, ripping a track from a CD became an OS feature of the PC and RIO became the first portable MP3 player with the size of a deck of cards. Most importantly a US court ruled that the RIO player was an IT, not a CE device.

With MPEG-G MPEG is taking the shot by addressing a conceptually similar situation as music tracks on CD converted to compressed digital 20 years ago. MPEG-G can potentially make the output file of a sequenced DNA “liquid” because it reduces a file of a few un-processable terabytes to a few tens of processable gigabytes. The difference is that the file of a sequenced DNA is a professional object, even though it can be produced in billions of instances and thus it looks like a consumer object.

MPEG-2 (Systems, Video and Audio)

MPEG-2 was a planned success. The early 1990s was a case of Brownian motion where particles (companies) were scrambling to come out with a new generation of TV. Japan with its MUSE and Europe with its HD-MAC were analogue by definition, and the USA put some order in its industrial activities with the Grand Alliance. However, all manufacturing companies, already global at that time, pushed for a digital solution. MPEG, the new kid in the block, announced it would develop a digital television standard and did so. The world followed.

There is much to learn from MPEG-2 because it is a standard that addressed the common needs of broadcasting industries – terrestrial, satellite, cable and package media – that until that time did not have much in common, if not the television experience eventually delivered to the end user. MPEG-2 was the catalyst of media industry convergence.

The development of the MPEG-2 4:2:2 profile was a natural follow-up and one that was constantly implemented in the video coding standards following MPEG-2 (including MPEG-4, RGB etc.). The 4:2:2 profile was a step itoward another industry – production – that has never been entirely fascinated by the MPEG offers.

MPEG-4 Binary Format for Scenes

MPEG-4 was the project that was meant to provide digital audio and video for yet another customer, the IT industry, actually the ICT industry, because video – and audio – for mobile was an important target from early on. BIFS had as foundation the Virtual Reality Modelling Language (VRML), a specification of the computer graphics world. MPEG extended VRML to support efficient transmission and real time. BIFS did not fly, and VRML, too, did not fly. Fascination with XML prompted both MPEG and VRML to introduce XMT and X3D, respectively. But success did not come. A later specification developed by the Korean Digital Media Broadcasting (DMB) made use of a relatively small portion of BIFS for complementary low bitrate services to digital audio broadcasting.

BIFS is an example of how commonality of technology between two radically different industries can well lead to no synergy. The origins of computer graphics are rooted in professional users using packaged content. The intended users of BIFS were consumers using real-time content via the network. The battle for the introduction of the technology was fought independently (it could hardly have been otherwise). Unfortunately, it was lost – independently.

MPEG learned that you cannot develop a standard in a top down fashion. Vision is needed, but vision must constantly face reality.

MPEG-4 Visual and Advanced Video Coding

MPEG-4 Visual inherited the vision of 2D visual objects in a 3D space. MPEG-4 Visual was a standard that supported moving 2D object in addition to extend the performance of MPEG-2 Visual. Industry needed the standard to enable services on the web and mobile. The hopes of a licence of MPEG-4 Visual patents were dashed when the licence included a charge based on the time an MPEG-4 Visual decoder was used.

MPEG-4 Visual was the sacrificial lamb that allowed MPEG-4 AVC to get a better deal in the use of relevant patents. Higher compression and acceptable licencing terms made AVC the universal video codec that it still is today.

MPEG-4 Audio

The MPEG Audio fortunes have different stories to tell. MP3 was and is an outstanding success. MP2 has been a good success and MP1 had a short span of use in the Digita Compact Cassette. MPEG-2 Audio did not have many takers because the performance was not adequate in a multi-channel environment. MPEG-2 AAC is in use in some countries. More importantly, MPEG-2 AAC it paved the way to MPEG-4 AAC.

MPEG-4 AAC is an outstanding success, with probably 10 billion devices in current use for all sorts of applications. Why? One element is the performance of the technology, another is the choice made by Apple to base their web-based music distribution service on MPEG-4 AAC. Yet another is the continuous performance improvement that has delivered MPEG-4 HE-AAC v2.

Open Font Format

OFF is a great example of standard development not for an industry forum but for a group of companies who had developed a successful specification that they were no longer keen on maintaining. In a sense, still, that was a request from an industry, conveyed by a National Body, that MPEG gladly accepted. OFF is now an extremely successful standard. This success is not entirely of MPEG’s doing, but the amount of extension MPEG has added to the original OpenType specification is such that MPEG can say that OFF is an MPEG standard of great success.

MPEG Media Transport (MMT)

MMT is an example of a solution that answers the questions that many (terrestrial) broadcasters ask: how can I face the competition of services on the web? The answer could have been strengthening broadcast delivery with some magi. Instead MMT offered the means to move broadcasting services, if not to the web, at least to the transport technology that enables the web . In this way it becomes easy to integrate broadcasting services with interactive services. The standard is deployed in several important countries, but acceptance of MMT is not universal. Why? Broadcasting is a conservative world and highly influenced by political considerations. Interestingly, streaming services that use MMT can offer short delay. However, there is still a limited use of MMT for streaming services.

High Efficiency Video Coding

HEVC continues the tradition of MPEG Video compression standards in terms of technical achievements. The HEVC Verification Tests showed that on average HEVC delivers the same quality as AVC with 40% of the bitrate required by AVC. MPEG’s traditional customers have jumped on HEVC which is widely deployed in their devices and services. A concern is the low level of adoption in online services which is rated at ~12%.

3D Audio Coding

This standard, part of the MMT and HEVC package in MPEG-H has been adopted by several broadcasting agencies. Its use has been announced by other industry sources also for distribution of high quality immersive audio via the web.

DASH

DASH is a remarkable example of how MPEG is able to respond to the need of a client industry, in this particular case 3GPP, representing the mobile telecommunication industry. MPEG developed DASH with a close connection with 3GPP. Since the standard has been developed, DASH has seen a broad acceptance by the industry.

Adding value to MPEG standards

The incomplete summary above shows that MPEG can develop high quality standards, the best in terms of performance in its area of work. Its standards do respond to industry requirements thanks to MPEG’s network of liaisons with the relevant client industries.

Unfortunately, this is no longer sufficient today. The HEVC case shows that MPEG should not only be able to deliver high quality standards but also that its users can access the technology. MPEG has no place in discussions about licensing, but it can operate to create the condition for easier licensing to happen,

A possible way forward is shown by the recently released MPEG-5 Essential Video Coding (EVC) standard. EVC offers improved compression efficiency compared to existing video coding standards (HEVC and its competitors) and relies on the statements of all contributors to the standard who have committed to announce their licensing terms no later than two years after the FDIS publication date.

Another route that will add value to MPEG standards is the acknowledgment that media technologies get more and more intertwined with other technologies present in media devices or relied on by media services. MPEG should be open to embrace more collaborations with other bodies and to develop joint standards, Far from being a limitation of MPEG’s role, this is an enhancement because MPEG standards will be able to reach more communities and hence more customers.

Ditto for the trend that MPEG has initiated some years ago to develop compression standards for data other than media. The collaboration with ISP TC 276 Biotechnology in the area of compression of reads from high speed sequencing machines has produced 5 parts of the MPEG-G standard. MPEG-G shows that MPEG compression technologies, developed for media, can be adapted to compress other data as well.

Posts in this thread

Design and build the future of MPEG

Established in 1988, the Moving Picture Experts Group (MPEG) has produced some 180 standards in the area of efficient distribution on broadcast, broadband, mobile and physical media, and consum­ption of digital moving pictures and audio, both conventional and immersive, using analysis, compression and trans­port technologies. It has also produced 5 standards for efficient storage, processing and delivery of genomic data.

With its membership of ~1500 registered and ~600 attending experts, MPEG has produced more standards than any other JTC 1 subcommittee. The standards enable a manufacturing and service business worth more than 1.5 T$ p.a.

We expect that in the future industry will require more MPEG standards for the following areas

  1. Rectangular video coding. As ~80% of world communication traffic is video-based, new more efficient video coding standards will continue to be of high interest. These will likely incor­porate new video compression technologies, driven by extensive research both in algo­rithm and hardware domain, e.g. in the area of Machine Learning. The market will also demand standards with higher quality user experience such as high-fidelity near-live video coding.
  2. Audio coding. As for video, future audio coding standards could be in the wake of the MPEG Audio coding family, be Machine Learning based or a be mix of the two. High fidelity near live audio coding standards will also be required to provide new forms of user experience.
  3. Immersive media will require coding standards that allow a user to move unres­tricted in a 3D space, for Point Clouds for human and machine use, for realistic meshes and textures and for light fields. Immersive audio coding standards providing an even more intensely satisfying user experience than the MPEG-I Audio standard currently being developed will also be required.
  4. 3D scene representation standards will be needed to recreate realistic 3D scenes populated by media objects en­coded with other MPEG immersive media coding standards. It is expected that standards for effective human interaction with media will also be required.
  5. Quality of experience will likely differentiate the market of products and services through the use of sophis­ticated technologies that reduce the amount of information needed to transmit future media types. MPEG must support the manufacturing and service industries by offering standards with features beyond those of the currently available standards, such as machine analytics for quality measure.
  6. Video Coding for Machines (VCM) is a new area of endeavour primarily designed to serve remotely located machines and, potentially, human users as well. Audio Coding for Machines will likely be a necessary complement to VCM. Machine vision standards, e.g. for object classification, will continue the tradition of CDVS (image search) and CDVA (video analysis) standards and will mostly be based on similar technologies as VCM. Internet of Media Things standards will also serve as a framework for the above standards.
  7. Application formats will be required by industry to facilitate the use of the wide range of available and future MPEG basic media coding standards by configuring MPEG technologies to suit application needs.
  8. Transport of compressed media standards have always been enablers of the use of MPEG media coding standards and we expect the need for them will continue. Ways to cope with emerging constraints on video and audio delivery over internet, such as those witnessed during the Covid-19 emergency, should be considered. New networks, e.g. 5G and other future networks, will likely drive the requests and MPEG should actively work together with the relevant bodies.
  9. Compression of genomic data, initiated 5 years ago in collaboration with TC 276, has shown that MPEG technologies could be successfully applied to new non-media areas. Standards will continue to be produced to respond to this new client industry.
  10. MPEG has realised that Compression of other data is a broad new area that can benefit from its approach to standardisation in support to new industries such as Medical, Industry 4.0, Automotive etc. As MPEG does not obviously have domain specific competence, it plans to identify the needs and develop standards for these new areas jointly with the relevant committees.

The above is an ambitious program of work in an expanding area of standardisation that heavily depends on sophisticated and evolving technologies. MPEG is confident it will be able to execute this program because it has learnt how to foster and actually fostered innovation with its standard and because, since its early days, it has innovated the way a standards committee operates, and developed a set of procedures and operations that MPEG calls its “modus operandi”.

Below we will examine how MPEG’s “modus operandi” will be exercised and adapted to the new challenges of the work program.

No one controls MPEG Connection with industries
Connection with academia Technology-driven standards meeting market needs
Standards with and for other committees Plans for future common standards
Standard (co-)development Effectiveness of MPEG role
Role of ICT tools in standardisation  

No one controls MPEG. The key differentiating element is that no entity or industry “controls”  MPEG. At all levels, proposals follow a bottom-up path and are accepted, not because of their origin but because of their value, by a mostly flat internally dynamic organisation that fosters the constructive flow and interaction of ideas. The manag­ement monitors the activities of the different organisational units, identifies points of conver­gence, interaction or contention, and creates opportunities to resolve issues. This element of the modus operandi must be preserved.

Connection with industries. The success of MPEG standards is largely based on the establishment of efficient connections with the client industries. MPEG will continue to interact with them providing solutions to their cus­tomers’ needs. However, we expect that formal liaisons may soon become insufficient because MPEG standards heavily depend on rapidly evolving techno­logies, such as capture and presentation, which have a strong impact on its standards but have a scope outside of the MPEG purview. MPEG will need to identify and activ­ate relationships with the industries who are developing the new media technologies that will drive the development of its new standards. At the same time MPEG should enhance its relationships with the content industry that provides the ultimate user experience using MPEG standards, again using a network with varying capabilities to address devices that could have graded tiers of computational complexity.

Connection with academia. Another reason for the success of MPEG standards is MPEG’s strong links with academia and research organisations, an area that accounts for ~¼ of its total mem­bership and contributes to sustaining the vitality of its standards activities. MPEG has always conducted exploration on promising new technologies, before standardisation can begin. However, the number of directions that require explorations are growing fast and MPEG resources may not be sufficient in the future to assess the potential of all candidate technologies. MPEG will need to establish more organic links with academia and create more opportunities to stimulate, get input from and possibly collaborate with academia.

Technology-driven standards meeting market needs. MPEG’s modus operandi has been inspired by applications as supported by technology and we think that in the future this must continue. However, it is imperative that this be complemented by a form of internal competition with new MPEG members who are market-driven. The goal is to strike the right balance between tech­nol­ogy-push and market-pull, i.e. to be able to identify targets that are both technology-driven and attractive to the market. MPEG should create a market needs group who dialectically interacts with the requirem­ents group to fortify proposals for new work, keeping timely delivery of its standards as a high priority issue. MPEG should better assert its active presence on all aspects of the standardisation workflow: industry, market, research and academia.

Standards with and for other committees. MPEG’s modus operandi has allowed MPEG to manage the development of technology-heavy stan­dards, some of which have involved hundreds of person-years. Most standards have been developed with internal resources. However, other standards have been developed jointly with entities outside MPEG at different level of integration: e.g. with ITU-T SG 16, with SC 24 for augmented reality, with ISO TC 276 for genomics etc. Several MPEG standard extensions have been developed in response to the need of other committees (e.g. JPEG, 3GPP, SCTE etc.). This fluid interaction must be enhanced by broadening MPEG’s operation, as media technologies interact more and more with other technologies in products and services and use of compression spreads to more domains.

Plans for future common standards. We envisage the need for MPEG to continue to develop common standards with other committees, in partic­ular with ITU-T SG 16 on video coding and with TC 276 Biotechnologies on gen­omic data compression. Augmented Reality will provide opportunities to develop common standards with JTC 1/SC 24. We also see more work with other commi­ttees as an essential component of MPEG future. Examples are JTC 1/SC 41 on a new phase of Internet of Media Things (IoMT), JTC 1/SC 42 on Machine Learning for compression and Big Data, starting from MPEG’s Network-Based Media Processing (NBMP) standard, and TC 215 Health Informatics on data compression. In a not-so-distant future MPEG should seek new standardisation opportunities in the area of Industry 4.0 and automotive. It is an ambitious program that can be implemented using MPEG experience and representation.

Standard development models. MPEG has learned that effective standardisation can be achieved if experts work shoulder-to-shoulder. How can this be achieved with an expanded scope of work and increased number of collaboration parties? One answer lies in the experience MPEG has gathered from its long history of online collaborations and the fully online 130th meeting attended by ~600 experts. In normal conditions, a large part of MPEG meetings will continue to be held face-to-face. However, many standards can and should be developed jointly with other organisations mostly via online meetings. These can be easily integrated with the rest of the MPEG workflow because they deal with specific topics.

Effectiveness of MPEG role. MPEG will need to develop new metrics, beyond the number of standards produced (MPEG is already the largest producer of standards in JTC 1) and apply them in a regular fashion to revise the effectiveness of its role. While MPEG standards continue to have loyal customers, their importance in other domains is less significant. The past is often so sticky that MPEG must make sure that its past successes do not make it an “also-ran” in the future. This does not mean that MPEG must renege its practices, but that it must adapt its business model to the new challenges. MPEG needs to reassert its mission to develop the best standards satisfying agreed requir­ements, thus continuing to provide opportunities to remunerate good IP for good standards. At the same time, MPEG should develop a strategy to handle the growing competition to its standards. One way is by enhancing the role of reference software, as an equivalent normative version of MPEG standards. Another is to envisage conditions that may facilitate expeditious development of licensing terms by the market.

Role of ICT tools in standardisation. Many aspects of the MPEG modus operandi heavily rely on ICT support. In 1995 MPEG was probably the first committee of its size and structure to adopt electronic document upload and distribution. In the last 15 years MPEG had the privilege to benefit from the support of Institut Mines Telecom and especially Christian Tulvan who has continuously designed new solutions that allow MPEG to be a fluid and flat organisation where every member can contribute by knowing what happens where. MPEG expects to be able to rely on their unique support to implement more features that will enhance the quality and quantity of its standards through enhanced and qualified participation.

Posts in this thread

 

 

Another view at MPEG’s strengths and weaknesses

Introduction

In No one is perfect, but some are more accomplished than others I have started a 360 ⁰ analysis of MPEG strengths, weaknesses, opportunities and threats considering the Context in which MPEG operates, the Scope of MPEG standards and the Business model of the industries using MPEG standards. Subsequently, in More MPEG Strengths, Weaknesses, Opportunities and Threats I have considered the Membership, the organisational Structure and the Leadership.

In this article I would like to continue the SWOT analysis and talk about MPEG’s Client industries, Collaboration with other bodies and the Standards development process.

Client industries

Strengths

MPEG can claim as a strength something that others may consider a weakness. When MPEG was created, MPEG did not have a reference industry. It did create one little by little starting from Telecom and Consumer Electronics (CE), the first two industries members of which supported the MPEG-1 standard. Broadcasting became another client industry with MPEG-2 and IT another with MPEG-4. Industry participation followed with mobile telecom, VLSI and more.

Why do I call this a strength? Because MPEG did not have a “Master” dictating what it should do, but many masters. With this MPEG was able to develop standards that were abstracted from the specific needs of one industry and was able to concentrate on what was common to all industries served.

MPEG was always very aggressive in looking for customers. When you do something it is always better to talk with those who have clear ideas about what the something can do for them. This effort to reach out has paid because the standards were “better” (they provided the intended features) and the customers were ready to adopt them. 3GPP, ARIB, ATSC, BDA, CTA, DVB, ETSI, ITU-T/R, TTA are just dome of the organisation MPEG has developed standards for.

The result is that today MPEG has an array of industries following its lead.

Weaknesses

Success is a hard master. If you don’t pile up more successes than before you seem to fail.

The MPEG-4 Visual licence signalled to the IT world that the business model was still driven by the old paradigm and that IT companies needed to find something else more familiar to them. The web was the perfect place where this could happen and did happen.

Interestingly, the old categories of Telecom, CE, Broadcasting, IT gradually came to lose their value: Telecom had a lot to share with IT and needed CE, and CE continued to serve Broadcasting with its devices, but added mobile and morphed to something hard to distinguish from a manufacturer of IT devices.

MPEG had its hands largely tied in this global transformation and the result is that a significant proportion of what MPEG can claim to be its client industry is not (or no longer) following its lead.

Opportunities

Still MPEG is appreciated by many non-ISO industries, less so by organisationally closer ones, because it the organisation you are considered for your level, but in industry you are considered for what you are.

Therefore MPEG has the opportunity not only to recover old client industries, but also to acquire new ones.

Threats

The most challenging threat is the breaking up of the MPEG ecosystem into a set of more or less independent components. This would be a big loss because today industry knows they can come to the MPEG “emporium” and get the integrated technologies they need. With MPEG broken up in pieces there will be a strengthened competition from individual environments who can offer the same individual pieces as the broken up MPEG can offer.

The threat to MPEG can also come from industries abandoning MPEG not because its standards are the best, but because MPEG standards cannot compete on other features.

Another threat comes from the possibility for industry consortia to build their own specifications by aggregating MPEG building blocks rather than propose a new standard to MPEG.

Collaboration with other bodies

Strengths

MPEG has always had an open attitude to collaboration with other bodies operating in neighbouring fields. 3GPP, AES, IETF, ITU-T SG 16, Khronos, SCTE, SMPTE, VRIF, W3C are just some examples

We have had many collaborations with

  • ITU-T SG 16: MPEG-2 Systems and Video, MPEG-4 AVC, MPEG-H HEVC, MPEG-I VVC, and Coding Independent Code Points (CICP)
  • TC 276/WG 5: File Format, Genome Compression, Metadata and API and Reference Software and Conformance in a week time
  • SC 24: Augmented Reality Reference Model

MPEG has developed many standards for other committees

  • JPEG: developed the file format for JPEG and JPEG 2000, developed the MPEG-2 transport for JPEG 2000 and JPEG XL
  • 3GPP: DASH was originally suggested by 3GPP and developed in close consultation with this and other committees.

The broad scope of MPEG standards offers more opportunities to collaborate and provide better and better focused standards.

More about MPEG collaboration on standard development at Standards and collaboration,

Weaknesses

The lowly WG status reduces MPEG’s ability to deal with other bodies authoritatively.

Opportunities

MPEG is not part of those who think they can do everything that is useful by themselves. The time devices and services were controlled by a major technology are long gone. MPEG technologies are present in a large number of devices, but to be useful those devices need to integrate other key technologies as well.

Therefore more collaborations are vital to the success of MPEG standards:

  1. MPEG has a Internet of Media Things (IoMT) standard that may have dependencies on what KTC 1/SC 41 IoT is doing
  2. JTC 1/SC 42 is in charge of Artificial Intelligence (AI). Neural Networks are an important AI technology. MPEG is working on a Neural Network Compression standard
  3. MPEG has developed Network Based Media Processing, an implementation of the Big Data Reference Model developed by SC 42
  4. MPEG is extending the glTF™ (GL Transmission Format) 3D scene description specification developed by Khronos

Collaboration is a great tool to achieve a goal based on common interests with other committees.

Threats

Less collaborations is a threat because it means less opportunities to develop good standards that extend to other fields!

Standards development process

Strengths

The process to develop standards is a major MPEG strength. It is a thorough implementation of ISO/IEC process with major extensions added to seek technology with Calls for Evidence (CfE) and Call for Proposals (CfP), develop the standards with Core Experiments (CE) and prove value of standards with Verification Tests (VT).

The figure illustrates the major elements of the MPEG standard development process.

Some industry participantscomplain about the ISO process to develop and approve standards. For MPEG that process is an excellent tool that allows the development of better standards, not a bureaucratic imposition.

Weaknesses

In a positive context there is one weakness. A major value of MPEG standards lies in the fact that standards are typically multi-part and that the parts have been designed to interact with one another. MPEG has a lean organisation that promotes focus on technical matter and chairs coordination meetings that monitor progress of work and identify area of interaction or contention between different technical areas. The figure represents the interaction between and among different parts of a standard.

In spite of the essential role played by coordination, this function has an ad hoc nature, which is a weakness.

Another weakness is the fact that at meetings there are so many interesting things happening but not enough time to follow them all.

Opportunities

There are great opportunities to improve the MPEG standard development process. Institut Mines Télécom has lent its ICT support to MPEG and Christian Tulvan has done – and he is doing – a lot to provide more tools. MPEG is facing the current Covid-19 emergency with new ICT tools provided by Christian.

It would be great if the MPEG ICT tools could be integrated in the ISO IT platforms.

Threats

The big threat is the breaking up of the MPEG ecosystem.

Posts in this thread

An “unexpected” MPEG media type: fonts

Introduction

These days MPEG is adapting its working methods at physical meetings to cope with the constraints of the Covid-19 pandemic. But this is not the first time that MPEG does works, and excellent one, without the help of physical meetings. This article will talk about this first and indispensable attempt at developing standards outside of regular MPEG meetings.

Compression is a basic need

In the early 2000, MPEG received an unexpected, but in hindsight obvious proposal. MPEG-4 defines several compressed media types – audio, video and 3D graphics are obvious choices – but also the technology to compose the different media. Text information, presented in a particular font, is more and more an essential component of a multimedia presentation and the font data must be accessible with the text objects in the multimedia presentation. However, as thousands of fonts are available today for use by content authors, and only few fonts (or just one) could be available on devices that are used to consume media content – there is a need to compress and stream font information.
So MPEG began to work on the new “font” media type, and in 2003 produced the new standard: MPEG-4 Part 18 – Font compression and streaming, which relied on OpenType as input font data format.

Fonts need more from MPEG

Later MPEG received another proposal from Adobe and Microsoft, the original developers of OpenType, a format for scalable computer fonts. By that time, OpenType was already in wide use in the PC world, and was seen as an attractive technology for adoption by consumer electronic and mobile devices. The community using it kept on requesting support for new features, and many new usage scenarios have emerged. As MPEG knows only too well, responding to those requests from different constituencies required a significant investment of resources. Adobe and Microsoft asked MPEG if it could do that. I don’t have to tell you what was the response.

Not all technical communities are the same

A new problem arose, though. The overlap of the traditional MPEG membership with the OpenType community was minimal, save for Vladimir Levantovsky of Monotype who had brought the font compression and streaming proposal to MPEG. So a novel arrangement was devised. MPEG would create an ad hoc group that would do the work via online discussions, and once the group has reached a consensus, the result of the discussions, in the form of “report of the ad hoc group”, would be brought by Vladimir to MPEG. The report would be reviewed by the Systems group and the result integrated in the rest of MPEG work, as it is done with all other subgroups. The report would often be accompanied by additional material to produce draft standard that would be submitted by MPEG to National Body voting. Disposition of comments and the new version of the standard would be developed within the ad hoc group, and submitted to MPEG for action.
This working method has been able to draw from the vast and dispersed community of language and font experts, who would never have been willing or able to attend physical meetings, to develop, maintain and extend standard for 16 years, and with outstanding results. MPEG has produced four editions of the standard called Open Font Format (OFF). As always, each edition of the standard has been updated by several amendments (extensions to the standard) before a new edition was created.

What does OFF offer to the industry? 

High quality glyph scaling
A glyph is a unit of text display that provides visual representation for a character, such as the Latin letter a, the Greek letter  the Japanese hiragana letter あor the Chinese character 亞. In an output device, a scaled outline of a glyph is represented by dots (or pixels) in a process called rasterization. The approximation of a complex glyph outline with dots can lead to significant distortions, especially when the number of dots per unit length is low.

OFF fonts introduced programmatic approach – a technology called “hinting” deals with these inconsistencies by adding additional instructions encoded in the font. The OFF standard helped overcome the obstacles by making proprietary font hinting and other patented technologies easily accessible to implementers, and enabled multiple software and device vendors to obtain Fair, Reasonable and Non Discriminatory (FRAND) licences.

The figure illustrate the connection of the and new worlds, i.e. the transformation of analog to digital. The curved arrows on the right side of the picture illustrate the effects of applying hint instructions to glyph control points, and their interdependencies.

Leveraging Unicode

The Unicode Standard is a universal character encoding system that was developed by the Unicode Consortium to support the interchange, processing, and display of the written text in many world languages – both modern (i.e. used today) and historic (so that conversion of e.g. ancient books and document archives into digital format is possible).

The Unicode Standard follows a set of fundamental principles of separation of text content from text display. This goal is accomplished by introducing a universal repertoire of code-points for all characters, and by encoding plain text characters in their logical order, without regard for specific writing system, language, or text direction. As a result, the Unicode Standard enables text sequences to be editable, searchable, and machine-readable by a wide variety of applications.

In short, the Unicode encoding defines the semantics of text content and rules for its processing – to visualize a text and make it human readable we need fonts that bring all missing elements required for text display. Glyph outlines, character to glyph mapping, text metrics and language-specific layout features for dynamic text composition are only some (but not all) of the data elements encoded in an OFF font. It is the combination of font information with the Unicode compliant character encoding that makes it possible for all computer applications to “speak” in any of the world languages without effort!

Advanced typography

OFF fonts offer a number of key ingredients that enable advanced typographic features, including support for certain features (e.g. ligatures and contextual glyph substitution) that are mandatory for certain languages, and a number of discretionary features. These include support for stylistic alternates, colour fonts for emoji, and many other advanced features that are needed to support bi-directional text, complex scripts and writing systems, layout and composition of text when mixing characters from different languages … the list can go on!

Lately, the addition of the variable fonts made it possible to both improve the efficiency of encoding of font data (e.g. by replacing multi-font families with a single variable font that offers a variety of weight width and style choices), and introduce new dynamic typographic tools that revolutionized the way we use fonts on the web.

Font Embedding
As I mentioned earlier, thousands of fonts are available for use in media presentations, and making a particular font choice is solely the author’s right and responsibility. Media devices and applications must respect all authors’ decisions to ensure faithful content presentation. By embedding fonts in electronic documents and multimedia presentations, content creators are assured that text content will be reproduced exactly as intended by authors, preserving content appearance, look and feel, original layout, and language choices.

OFF supports many different ways for font embedding, including embedding font data in electronic documents, font data streaming and encoding as part of ISO Base Media File Format, and font linking on the web. OFF font technology standard developed by MPEG became a fundamental component for deployment of Web Open Font Format, which has facilitated adoption of OFF fonts in the web environment and is now widely supported in different browsers.

What is OFF being used for?
OFF has become the preferred font encoding solution for applications that demand advanced text and graphics capabilities. Fonts are a key component of any media presentation, including web content and applications, digital publishing, newscast, commercial broadcasting and advertisement, e-learning, games, interactive TV and Rich Media, multimedia messaging, sub-titling, Digital Cinema, document processing, etc.

It is fair to say that after many years of OFF development, the support for OFF and font technology in general became ubiquitous in all consumer electronic devices and applications, bringing tremendous benefits for font designers, authors, and users alike!

Acknowledgement

The review and substantial additions of Vladimir Levantosky of Monotype to this article are gratefully acknowledged. For 20 years Vladimir has dedicated his energies to coordinate the font community efforts, make font standards in MPEG and promote the standards to industry.

Thank you Vladimir!

Posts in this thread

MPEG and the future of visual information coding standards

Video in MPEG has a long history

MPEG started with the idea of compressing the 216 Mbit/s of standard definition video, and the associated 1.41 Mbit/s of stereo audio, for interactive video applications on Compact Disc (CD). That innovative medium of the early 1980s was capable to provide a sustained bitrate of 1.41 Mbit/s for about 1 hour. The bitrate was expected to accommodate both the video and audio information. At about the same time, some telco research laboratories were working on an oddly named technology called Asymmetric Digital Subscriber Line (ADSL), in other words, a modem for high-speed (at that time) transmission of ~1.5 Mbit/s for the “last mile”, but only from the telephone exchange to the subscriber’s network termination. In the other direction, only a few tens of kbit/s were supported.

Therefore, if we exclude a handful of forward-looking broadcasters, the MPEG-1 project was really a Consumer Electronics – Telco project.

Setting aside the eventual success of the MPEG-1 standard – Video CD (VCD) used MPEG-1 and 1 billion player were produced in total, hence a different goal than the original interactive video – MPEG-1 was a remarkable success for being the first toidentify an enticing business case for video (and audio) compression, and systems, on top of which tens of successful MPEG standards were built over the years.

This article has many links

In Forty years of video coding and counting I have recounted the full story of video coding and in The MPEG drive to immersive visual experiences I have focused on the efforts MPEG has made, since its early years, to provide standards for an extended 3D visual experience. In Quality, more quality and more more quality I have described how MPEG uses and innovates subjective quality assessment methodologies to develop and eventually certify the level of performance of a visual coding standard. In On the convergence of Video and 3D Graphics I have described the efforts MPEG is making to develop a unified framework that encompasses a set of video sources producing pixels and a set of sensors producing points. In More video with more features I described how MPEG has been able to support more video features in addition to basic compression.

Now to the question

Seeing all this, the obvious question a reader might ask could be: if MPEG has done so much in the area of visual information coding standards, does MPEG still have much to do in that space? A reader with only a superficial understanding of the force that drives MPEG should probably know the answer, but I am not going to give it right now. I will first argument what I see as the future of MPEG in this area.

I need to make a disclaimer first. The title of this article is “The future of visual information coding standards”, but I should restrict the scope to “dynamic (i.e. time dependent) visual information coding”. Indeed the coding of still pictures is a different field of emdeavour serving the needs of a different industry with a different business model. It should not be a surprise if the two JPEG standards – the original JPEG and JPEG 2000 – have both a baseline mode (the only one that is actually used) which is Option 1 (ISO/IEC/ITU language for “royalty free”). It should also be no surprise to see that, while it is conceivable to think of a standard for holographic still image coding,  holography is not even mentioned in this article.

There was always a need for new video codecs

Forty years of video coding and counting explains the incredible decade-long ride to develop video coding standards all based on the same basic ideas enriched at each generation, that will enable the industry to achieve a bitrate reduction of 1,000 from video PCM samples to compressed bitstream with the availability of the latest VVC  video compression standard, hopefully in the second half of 2020 (the incertainty is caused by the current Covid-19 pandemic which is taking its toll on MPEG as well).

The need for new and better compression standards, when technology makes it possible and the improvement over the latest existing standard justifies it, has been justified by the push toward higher resolution, colour, bit-per-pixel, dynamic range, viewing angle etc. video and the lagging availability of a correspondingly higher bitrate to the end user.

The push toward “higher everything” will continue, but will the bitrate made available to the end user continue to lag?

The safe answer is: it will depend. It is a matter of fact that bandwidth is not an asset uniformly available in the world. In the so-called advanced economies the introduction of fibre to the home or to the curb continues apace. The global 5G services market size is estimated to reach 45.7 B$ by 2020 and register a CAGR of 32.1% from 2021 to 2025 reaching ~184 B$. Note that 2025 is the time when MPEG should think seriously about a new video coding standard. The impact of the current pandemic could further accelerate 5G deployment.

More video and which codecs

The first question is whether there will be room for a new video coding standard. My answer is yes and for at least two reasons. The first is socio-economic: the amount of world population that is served by a limited amount of bandwidth will remain large while the desire to enjoy the same level of experience of the rest of the world will remain high. The second is technical: currently, efficient 3D video compression is largely dependent on efficient 2D video compression.

The second question is more tricky. Will this new (after VVC) 2D video compression standard still be another extension of the motion compensated prediction scheme? I am sure that the answer could be yes. The prowess of the MPEG community is such that another 50% improvement could well be provided. I am not sure that will happen, though. Machine learning applied to video coding is showing that significant improvements over state-of-the-art video compression can be obtained by replacing components of existing schemes with Neural Networks (NN), or even by defining entirely new NN-based architectures.

The latter approach has several aspects that make it desirable. The first is that a NN is trained for a certain purpose but you can always trained it better, possibly at the cost of making it heavier. Neural Network Compression (NNC), another standard MPEG is working on, could further extend the range of incrementally improving the performance of a video coding standard, without changing the standard, by making components of the standard downloadable. Another desirable aspect is that media devices will become more and more addicted to using Artificial Intelligence (AI)-inspired technologies. Therefore a NN-based video codec could simply be more attractive for a device implementor because the basic processing architectures are shared amongst a larger number of data types.

New types of video codec

There is another direction that needs to considered in this context and that is the large and growing quantity of data that are being and will be produced by connected vehicles, video surveillance, smart cities etc. In most cases today and more so in the future, it is out of question to have humans at the other side of the transmission channel watching what is being transmitted. More likely there will be machines that will monitor what happens. Therefore, the traditional video coding scenario that aims to achieve the best video/image under certain bit-rate constraints having humans as consumption targets is inefficient and unrealistic in terms of latency and scale when the consumption target is a machine.

Video Coding for Machines (VCM) is the title of an MPEG investigation that seeks to determine the requirement for this novel, but not entirely new video coding standard. Indeed, the technologies standardised by MPEG-7 – efficiently compressed image and video descriptors and description schemes – belong to the same category as VCM. It must be said, however, that 20 years have not passed in vain. It is expected that all descriptors will be the output of one or more NNs.

One important requirement is the fact that while millions of streams may be monitored by machines, some streams may need to be monitored by humans as well, possibly after having been alerted by a machine. Therefore VCM is linked to the potential new video coding I have talked about above. The question is whether VCM should be called HMVC (Human-Machine Video Coding) or there should be VCM (where the human part remains below threshold in terms of priority) and YAVC (Yet Another Video Coding, where the user is meant to be a human).

Immervice video codecs

The MPEG drive to immersive visual experiences shows that MPEG has always been fascinated by immersive video. The fascination is not fading away as shown by the fact that, in four months MPEG plans to release the Video-based Point Cloud Compression standard and in a year the MPEG Immersive Video standard.

These standards, however, are not the end points, but the starting points in the drive to more rewarding user experiences. Today we cannot say how and when MPEG standards will be able to provide full navigation in a virtual space. However, that remains the goal for MPEG. Reaching that goal will also depend on the appearance of new capture and display technologies.

Conclusions

The MPEG war machine will be capable to deliver the standards that will keep the industry busy developing the products and the service that will enhance the user experience. But we should not forget an important element: the need to adapt MPEG’s business model to the new age.

MPEG needs to adapt, not change its business model. If MPEG has been able to sustain the growth of the media industry, it is because it has provided opportunities to remunerate the good Intellectual Property that is part of its standards.

There are other business models appearing. The MPEG business model has shown its worth for the last 30 years. It can do the same for another 30 years if MPEG will be able to develop a strategy to face and overcome the competition to its standards.

Posts in this thread