Introduction
In my earlier post 30 years of MPEG, and counting? I brought evidence of what MPEG has done in the last 30 years to create the broadcast, recording, web and mobile digital media world that we know. This document tries to make the picture more complete by looking at the current main activities and deliveries in the next few months/years.
The MPEG work plan at a glance
The figure below shows the main standards developed or under development by MPEG in the 2017-2023 period organised in 3 main sections:
- Media Coding (e.g. MP3 and AVC)
- Systems and Tools (e.g. MPEG-2 TS and File Format)
- Beyond Media (currently Genome Compression and Neural Network Compression).
Navigating the MPEG work plan
In the Video coding area MPEG is currently handling 4 standards (MPEG-4, -H, -I and -CICP) and several Explorations.
MPEG-I ISO/IEC 23090 Coded representation of immersive media is the container of standards needed for the development of immersive media devices, applications and services.
MPEG is currently working on Part 3 Versatile Video Coding, the new video compression standard after HEVC. VVC is developed jointly with VCEG and is expected to reach FDIS stage in October 2020.
Explorations
- An exploration on a new video coding standard that combines coding efficiency (similar to that of HEVC), complexity (suitable for real time encoding and decoding) and usability (timely availability of licensing terms). In October 2018 a Call for Proposals was issued. Submissions are due in January 2019 and FDIS stage for the new standard is expected to be reached in January 2020.
- An exploration on a future standard that defines a data stream structure composed by two streams: a base stream decodable by a hardware decoder, and an enhancement stream suitable for software processing implementation with sustainable power consumption. This activity is supported by more than 30 major international players in the video distribution business. A Call for Proposal has been issued in October 2018. Submissions are due in March 2019 and FDIS stage is expected to be reached in April 2020.
- Several explorations on Immersive Video Coding
- 3DoF+ Visual: a Call for Proposal will be issued in January 2019. Submissions are due in March 2019 and the FDIS is planned for July 2020. The result of this activity is not meant to be a video coding standard but a set of metadata that can be used to provide a more realistic user experience in OMAF v2. Indeed, 3DoF+ Visual will be a part of MPEG-I part 7 Immersive Media Metadata. Note that 3 Degrees of Freedom (3DoF) means that a user can only make yaw, pitch, roll movements, but 3DoF+ means that the user can also displace the head to a limited extent.
- Several longer-term explorations on compression of 6DoF visual (Windowed-6DoF and Omnidirectional 6DoF) and Compression of Dense Representation of Light Fields. No firm timeline for standards in these areas has been set.
In the Audio coding area MPEG is handling 3 standards (MPEG-4, -D, and -I). Of particular relevance is the MPEG-I part 3 Immersive Audio activity. This is built upon MPEG-H 3D Audio – which already supports a 3DoF user experience – and will provide a 6DoF immersive audio VR experience. A Call for Proposal will be issued in March 2019. Submissions are expected in January 2020 and FDIS stage is expected to be reached in April 2021. As for 3DoF+ Visual, this standard will not be about compression, but about metadata.
In the 3D Graphics coding area MPEG is handling two parts of MPEG-I.
- Video-based Point Cloud Compression (V-PCC) for which FDIS stage is planned to be reached in October 2019. It must be noted that in July 2018 an activity was initiated to develop standard technology for integration of a 360 video and V-PCC objects.
- Geometry-based Point Cloud Compression (G-PCC) for which FDIS stage is planned to be reached in January 2020.
The two PCC standards employ different technologies and target different application areas: entertainment and automotive/unmanned aerial vehicles,
In the Font coding area MPEG is working on MPEG-4 part 22.
MPEG-4 ISO/IEC 14496 Coding of audio-visual objects is a 34-part standard that made possible large scale use of media on the fixed and mobile web.
Amendment 1 to Open Font Format will support complex layouts and new layout features. FDAM stage will be reached in April 2020.
MPEG-G ISO/IEC 23092 Genomic Information Representation is the standard developed in collaboration with TC 276 Biotechnology to compress files containing DNA reads from high speed sequencing machines.
In the Genome coding area MPEG plans to achieve FDIS stage for Part 2 Genomic Information Representation in January 2019. MPEG has started investigating additional genome coding areas that would benefit from standardisation.
Neural network compression is an exploration motivated by the increasing use of neural networks in many applications that require the deployment of a particular trained network instance potentially to a large number of devices, which may have limited processing power and memory.
In the Neural network coding area MPEG has issued a Call for Evidence in July 2018, assessed the responses received in October 2018 and collected evidence that justifies the Call for Proposals issued at the same October 2018 meeting. The goal of the Call is to make MPEG aware of technologies to reduce the size of trained neural networks. Responses are due in March 2019. As it is likely that in the future more Calls will be issued for other functionality (e.g., incremental representation), an expected time for FDIS has not been identified yet.
Media description is the goal of the MPEG-7 standard which contains technologies for describing media, e.g. for the purpose of searching media.
In the Media description area MPEG has completed Part 15 Compact descriptors for video analysis (CDVA) in October 2018. By exploiting the temporal redundancy of video, CDVA extracts a single compact descriptor that represents a video clip rather than individual frames, which was the goal of Compact Descriptors for Visual search (CDVS).
Work in this area continues to complete reference software and conformance.
In the System support area MPEG is working on MPEG-4, -B and -I. In MPEG-I MPEG is developing
- Part 6 – Immersive Media Metrics which specifies the metrics and measurement framework to enhance the immersive media quality and experiences. 3DoF+ Visual metadata will be one component of this standard
- Part 7 – Immersive Media Metadata which specifies common immersive media metadata focusing on immersive video (including 360° video), images, audio, and timed text.
Both parts are planned to reach FDIS stage in July 2020.
Intellectual Property Management and Protection
In the IPMP area MPEG is developing an amendment to support multiple keys per sample. FDAM stage is planned to be reached in March 2019. Note that IPMP is not about _defining_ but _employing_ security technologies to digital media.
In the Transport area MPEG is working on MPEG-2, -4, -B, -H, -DASH, -I, -G and Explorations.
MPEG-2 ISO/IEC 13818 Generic coding of moving pictures and associated audio information is the standard that enabled digital television.
Part 2 Systems continues to be an extremely lively area of work. After producing Edition 7, MPEG is working on two amendments to carry two different types of content
- JPEG XS (a JPEG standard for low latency applications)
- CMAF (an MPEG Application Format).
MPEG-4
Part 12 ISO Based Media File Format is another extremely lively area of work. Worth mentioning are two amendments
- Compact Sample-to-Group, new capabilities for tracks, and other improvements – has reached FDAM stage in October 2018
- Box relative data addressing – is expected to reach FDAM in March 2019.
The 7th Edition of the MP4 file format is awaiting publication.
MPEG-B ISO/IEC 23001 MPEG systems technologies is a collection of systems standards that are not specific to a given standard.
In MPEG-B MPEG is working on two new standards
- Part 14 Partial File Format will provide a standard mechanism to store HTTP entities and the partial file in broadcast applications for later cache population. The standard is planned to reach FDIS stage in April 2020.
- Part 15 Carriage of Web Resources in ISOBMFF will make it possible to enrich audio/video content, as well as audio-only content, with synchronised, animated, interactive web data, including overlays. The standard is planned to reach FDIS stage in January 2019.
MPEG-H ISO/IEC 23008 High efficiency coding and media delivery in heterogeneous environments is a 15-part standard for audio-visual compression and heterogeneous delivery.
Part 10 MPEG Media Transport FEC Codes is being enhanced by the Window-based FEC code an amendment. FDAM is expected to be reached in January 2020.
MPEG-DASH ISO/IEC 23009 Dynamic adaptive streaming over HTTP (DASH) is the standard for media delivery on unpredictable-bitrate delivery channels.
In MPEG-DASH MPEG is working on
- Part 1 Media presentation description and segment formats is being enhanced by the Device information and other extensions amendment. FDAM is planned to be reached in July 2019.
- Part 7 Delivery of CMAF content with DASH contains guidelines recommending some of the most popular delivery schemes for CMAF content using DASH. TR is planned to be reached in March 2019
MPEG-I
Part 2 Omnidirectional Media Format (OMAF) released in October 2017 is the first standard format for delivery of omnidirectional content. With OMAF 2nd Edition Interactivity support for OMAF, planned to reach FDIS in January 2020, MPEG is substantially extending the functionalities of OMAF. Future releases may add 3DoF+ functionalities.
MPEG-G
MPEG plans to achieve FDIS stage for Part 1 Transport and Storage of Genomic Information in January 2019.
MPEG-A ISO/IEC 23000 Multimedia Application Formats is a suite of standards for combinations of MPEG and other standards (when there are no suitable MPEG standard for the purpose).
MPEG is working on two new standards
- Part 21 Visual Identity Management Application Format will provide a framework for managing privacy of users appearing on pictures or videos shared among users. FDIS is expected to be reached in January 2019.
- Part 22 Multi-Image Application Format (MIAF) will enable precise interoperability points when creating, reading, parsing and decoding images embedded in HEIF (MPEG-H part 12).
Application Programming Interfaces
The Application Programming Interfaces area comprises standards developed in order to make possible effective use of some MPEG standards.
In the API area MPEG is working on MPEG-I, MPEG-G and MPEG-IoMT.
MPEG-I
MPEG is working on Part 8 Network-based Media Processing (NBMP), a framework that will allow users to describe media processing operations to be performed by the network. The standard is expected to reach FDIS stage in January 2020.
MPEG-G
MPEG is working on Part 3 Genomic Information Metadata and Application Programming Interfaces (APIs). The standard is expected to reach FDIS stage in March 2019.
MPEG-IoMT ISO/IEC 23093 Internet of Media Things is a suite of standards supporting the notion of Media Thing (MThing), i.e. a thing able to sense and/or act on physical or virtual objects, possibly connected to form complex distributed systems called Internet of Media Things (IoMT) where MThings interact between them and humans.
In MPEG-IoMT MPEG is working on
- Part 2 IoMT Discovery and Communication API
- Part 3 IoMT Media Data Formats and API.
Both expected to reach FDIS stage in March 2019.
Media Systems includes standards or Technical Reports targeting architectures and frameworks.
In Media Systems MPEG is working on Part 1 IoMT Architecture, expected to reach FDIS stage in March 2019. The architecture used in this standard is compatible with the IoT architecture developed by JTC 1/SC 41.
MPEG is working on the development of 10 standards for reference software of MPEG-4, -7, -B, -V, -H, -DASH, -G, -IoMT
MPEG is working on the development of 8 standards for conformance of MPEG-4, -7, -B, -V, -H, -DASH, -G, -IoMT.
Conclusions
Even this superficial overview should make evident to all the complexity of interactions in the MPEG ecosystem that has been ongoing for 30 years (note that the above only represents a part of what happened at the last meeting in Macau).
In 30 years of MPEG, and counting? I wrote “Another thirty years await MPEG, if some mindless industry elements will not get in the way”.
It might well have been a prophecy.
Posts in this thread (in bold this post)
- The MPEG ecosystem
- Why is MPEG successful?
- MPEG can also be green
- The life of an MPEG standard
- Genome is digital, and can be compressed
- Compression standards and quality go hand in hand
- Digging deeper in the MPEG work
- MPEG communicates
- How does MPEG actually work?
- Life inside MPEG
- Data Compression Technologies – A FAQ
- It worked twice and will work again
- Compression standards for the data industries
- 30 years of MPEG, and counting?
- The MPEG machine is ready to start (again)
- IP counting or revenue counting?
- Business model based ISO/IEC standards
- Can MPEG overcome its Video “crisis”?
- A crisis, the causes and a solution
- Compression – the technology for the digital age
- On my Charles F. Jenkins Lifetime Achievement Award
- Standards for the present and the future
