1. Overview of MPEG-4
Developments
Olivier Avaro
Chairman MPEG Systems
Deutsche Telekom - Berkom
Davic European Information Day
Geneva, 12 January 1999
Berkom
===!" §
2. Goals of the presentation
• Give a report of the status of MPEG-4 developments.
• Give a report of the status of the AIC Initiative.
• Some special highlights :
– Video and Systems tools,
– Carriage of MPEG-4 content over MPEG-2 and IP.
Berkom
===!" §
3. Messages
• MPEG-4 is not meant to replace existing architectures (ex :
MPEG-2 or IP based architecture).
• MPEG-4 augments existing architecture’s strong points with
new concepts (ex : Interactive Scene Description, new content
types, Java APIs).
• MPEG-4 extends existing architecture’s strong points to new
environments (ex : delivery independent representation of the
content).
Berkom
===!" §
4. Overview
• Introduction to MPEG-4 :
– History, Requirements and Architecture,
– MPEG-4 Video Version 1 tools,
– MPEG-4 Systems Version 1 tools.
• Carriage of MPEG-4 over MPEG-2 and IP.
• Introduction to the AIC Initiative :
– Objectives and architecture,
– Applications,
– Work plan.
• Annexes
– MPEG-4 Object Content Information (OCI)
– MPEG-4 Intellectual Property Management and Protection (IPMP)
Berkom
===!" §
5. A bit of history on MPEG-4
• Participants MPEG: Consumer Electronics, Computer
Industry, Telecom Operators, Academia.
• 1993 : started as ‘Very Low Bitrate Audio-visual Coding’.
• 1994: goal changed to ‘Coding of Audio-visual Objects ’.
• Now: Version 1 is ready; released for IS ballot at the end of
the month. Version 2 released for CD ballot in March.
‘TV/film’
AV-
data
From the MPEG-4
Call for Proposals
(drafted 1994) interactivity
Wireless
‘Computer’
‘Telecom’
Berkom
===!" §
6. Requirements
• Provide common technology for many types of services :
interactive, broadcast, conversational.
• Increase client-side and client-server interactivity.
• Integrate natural and synthetic content.
• Cover a wide range of access conditions.
• Provide serious solutions for the management and
protection of intellectual property.
Berkom
===!" §
7. MPEG-4 Principles
• Audio-visual scenes made of audio-visual objects composed
together according to a scene description :
– allows interaction with elements within the audio-visual scene,
– coding scheme can differ for individual objects,
– allows easy re-use of audio-visual content.
• Audio-visual objects can be of different nature :
– audio (single or multi-channel) or video (arbitrary shape or rectangular),
– natural (natural audio or video) or synthetic (text & graphics, animated faces,
synthetic music),
– 2D (Web like pages) or 3D (spatialized sound, 3D virtual world),
– streamed (video movie) or downloaded (audio jingle).
Berkom
===!" §
8. MPEG-4 Principles
• Scene Description provides :
– the spatial/temporal relationship between the audiovisual objects (2D, 3D,
mixed 2D and 3D scene description),
– the behavior and interactivity of the audio-visual objects and scenes,
– protocols to modify and animate the scene in time,
– a binary encoding for the scene.
• These principles are independent of the bit rate.
Berkom
===!" §
9. Architecture
Decoding
TransMux FlexMux Composition and
N Rendering
Primitive
e
t
AV Objects ...
w ... ...
o
r Ex: MPEG-2 Elementary
Transport Streams Scene Description Audiovisual Interactive Display and User
k Information Scene Interaction
Object Descriptor
Berkom
DAI
===!" §
10. Example of an MPEG-4
Audiovisual Scene (1)
2D Audio-visual scene 2D Audio-visual scene
Audio and Video + Scrolling Text Audio and Video + Still Images
and Still Images
Berkom
===!" §
11. Example of an MPEG-4
Audiovisual Scene (2)
3D Audio-visual scene
3D World + arbitrary shaped
video + still images + 3D
Objects
Berkom
===!" §
12. MPEG-4 Visual Version 1 Tools
• Natural video from 10 kbit/s - 10 Mbit/s
– Rectangular or arbitrary shape objects,
– Scalable,
– Interlaced and Progressive scan supported,
– ‘Sprites’ (e.g. backgrounds): send once, warp after.
• Computer-generated visual information
– facial animation,
– animated meshes with moving texture,
– animated text and graphics.
• All these objects can be synchronised together.
Berkom
===!" §
13. Example of MPEG-4 Video Objects
Arbitrary shape Animated Face
Rectangular shape video object
video object
Berkom
===!" §
14. Visual Version 2 Tools
• Body animation : to augment the face animation tool.
• Compression of 3D meshes with texture :
– games, higher quality virtual environments.
• Higher Quality Video
– currently under discussion.
• Ready end ‘99
• Note that V.2 is a compatible extension of V.1
Berkom
===!" §
15. MPEG-4 Systems Version 1Tools
• Decoder Model
– Predictable behavior of a decoder and decoded content,
– Management of decoder resources (buffer and memory),
– Tight synchronization of Audio, Video and synthetic elements.
• Object Descriptor Protocol
– Identification (location and stream type) of Elementary Streams
(content in the same scene can come from different sources),
– Association of Elementary Streams (alternative content, scalable
content, synchronization hierarchy),
– Association of the scene description and the streams,
– Object Content Information,
– Intellectual property protection and management.
Berkom
===!" §
16. MPEG-4 Systems Version 1Tools
• BIFS Scene Description
– VRML concepts : set of nodes to represent the primitive scene objects to
be composed, the scene graph constructs, the behavior and interactivity
+ Integration of streams,
+ 2D capabilities,
+ Integration of 2D and 3D,
+ Advanced Audio Features,
+ Timing model,
+ BIFS - Update protocol to modify the scene in time,
+ BIFS - Anim protocol to animate the scene in time,
+ Compression efficiency (BIFS is Binary Format for Scene).
Berkom
===!" §
17. Example of MPEG-4 Scenes (1)
2D Audio-visual scene
Audio and Video + Still Images
2D Audio-visual scene
Animated Text+ Video + Still Images
Berkom
===!" §
18. Example of MPEG-4 Scenes (2)
2D and 3D Graphics Primitive
Scene with Face Object
Web-like MPEG-4 Scene
Multi audio and video
MPEG-4 Scene with text
Berkom
===!" §
19. MPEG-4 Systems Version 2 Tools
• File format
– allows interchange and streaming of MPEG-4 content.
• Advanced BIFS
– New BIFS functionality : environmental spatialization, integration of
HTML and BIFS, chroma keying, application signaling,
– Advanced BIFS coding : PROTOs and ExternProtos, Integration of
Mesh, coding of MField.
• MPEG-J
– The MPEG-4 ‘Application Execution Engine’
– MPEG-4 specific API for interaction with the scene (complicated
content behavior), with the network / terminal / UI resources.
Berkom
===!" §
20. MPEG-J Architecture
MPEG-J Run Time Environment
Java MPEG-let
Terminal
Java Virtual Machine
DMIF Application Decoders Compositor
Interface
Berkom
===!" §
21. Profiles and Levels
• Most applications only need a part of the MPEG-4 tool set.
• Profiles define subsets useful for a large class of
applications/services.
• Types of Profiles:
– Scene description (e.g. behaviour),
– Object Descriptor (mainly timing models),
– Audio (natural and synthetic): types of objects,
– Visual (natural and synthetic): types of objects,
– Graphics.
• Levels limit the number of objects and complexity
• Profiles/Levels can be easily added when need arises
Berkom
===!" §
22. Profiles and Levels
MPEG-4 device
Scene
Description
Profiles
Main
speech simple 2D 3D
Audio
Audio
Profiles
core Core
hybrid 3D
FA 2D Object Descriptor
Profile
Visual Graphics
Profiles Profiles
Media Profiles
Berkom
===!" §
23. Carriage of MPEG-4 Content
• Transport of MPEG-4 content : left to other bodies
– Choose IP, MPEG-2 TS, ATM AAL5, …
• DMIF abstracts Application from delivery of data and provide
a uniform walkthrough to access content
– e.g. Broadcast, Internet protocol stack
• Create content once, play on any network configuration
Berkom
===!" §
24. Delivery Framework
SL-Packetized Streams Media
DAI
FlexMux FlexMux
FlexMux Streams
Protection sL Protection sL Delivery
(RTP) (PES) AAL2 H223 ….
UDP MPEG2 ATM PSTN
Mux sub Layer
IP TS
TransMux Streams
Berkom
===!" §
25. MPEG-4 over MPEG-2 and IP
• Technical issues to be addressed :
– Transport/Multiplexing/Packetization (ex : FlexMux, PES only or IETF
RTP multiplex),
– Synchronization (ex : Sync Layer, RTP sync, MPEG-2 sync),
– Identification and association of streams (ex : Object Descriptor, SDP).
• Solved by making tradeoff between :
– How far the integration should be done (ex : using MPEG-2 content or
RTP streams in MPEG-4 scenes, using the MPEG-4 content types in
MPEG-2 or RTP applications),
– How much the duplication of functionality can be reduced.
Berkom
===!" §
26. MPEG-4 over MPEG-2 and IP
• Examples
– Blind transmission of MPEG-4 Programs
• MPEG-4 content separate from others MPEG-2/RTP streams
– MPEG-4 Content (ES) in an MPEG-2 Systems /RTP application
• adding MPEG-4 content types to MPEG-2/RTP
– MPEG-4 ES + Systems in MPEG-2/RTP environment
• Adding MPEG-4 Interactive programming to MPEG-2/RTP
• MPEG-2/RTP ‘Objects’ may be used in the MPEG-4 application
• On-going activity within MPEG and IETF that should lead for
end 99 to :
– addenda to MPEG-2 and MPEG-4,
– IETF RFC.
Berkom
===!" §
27. AIC-I Charter
• Goal: Harmonize standards efforts and enable
interactive 2D, 3D and streaming content for commerce
and entertainment on set top devices.
• The Advanced Interactive Content Initiative (AIC Initiative)
applied to be an industry consortium operating under the
Industry Technical Agreement (ITA) of the International
Electro technical Commission (IEC).
• The purpose of the AIC Initiative is the development of
interactive content specifications for user devices to offer a
range of basic through advanced interactive applications using
3D as well as 2D content, in stored and streamed form.
Berkom
===!" §
28. AIC-I Applications
• Advanced Electronic Program Guide.
• Buy-me-button.
• Data Ticker.
• Program Enhancement.
• Interactive Home Shopping.
• Interactive Commercial.
• Interactive Entertainment.
• Demographic Programming.
Berkom
===!" §
29. AIC-I Principles
• Delivery in various transport environments
– including MPEG-2 and IP environments,
– and combinations of broadcast/interactive delivery.
• Presentation engine :
– One specification for integration of MPEG-4, ‘X-HTML’, VRML
content.
• Application execution engine
– Not the priority right now.
Berkom
===!" §
30. AIC-I Workplan
• 20th December: Delivery of Version 1.0.
• 8th-9th February: Delivery of Version 1.1.
• 21st-22nd of March: Field trial #1. Delivery of Version 1.2.
• May/June: Delivery of Version 1.3.
• 17th-18th of July: Delivery of Version 2.0.
• + Various demonstrations at NAB, IBC, CES ...
Berkom
===!" §
31. Conclusions
• First MPEG-4 applications will appear probably on the Internet
based on subsets of MPEG-4 integrated in the IP environment.
• In parallel, MPEG-4 can enhance MPEG-2 digital broadcast
with attractive and interactive content.
• Ongoing AIC-I activity, jointly with W3C, W3D, MPEG-4
will provide specification for advanced interactive content on
MPEG-2 and IP environments in an integrated way.
Berkom
===!" §
33. Credits
• The MPEG-4 Systems Sub-group.
• The European project MoMuSys.
• Material for the presentation
Text and pictures from slides courtesy of KPN Research
• Tape demonstration - shows possibilities
pictures courtesy of France Telecom CNET, Deutsche
Telekom, Instituto Superior Técnico
Berkom
===!" §
35. OCI - Object Content Information
• Waiting for MPEG-7 would be too long for some application,
MPEG-4 designed small, synchronised events containing
information descriptors.
• Carried in separate ES associated with an Object or in the ES
Descriptor of the Object.
• Types:
– content classification (entity + table number + info - open system! )
– rating, (entity + criteria + info)
– language (ISO 639)
– textual descriptors (language code + free text)
– keywords (ditto)
– creation of the content (who, date - no owner! )
Berkom
===!" §
36. IPMP - Intellectual Property
Management and Protection
• ‘IPMP’ is prerequisite for ‘publishing’ serious, valuable
content in digital form
– in any open environment - which nowadays even includes the CD!
• IPMP support integrated deeply into the MPEG-4 system
• Yet only the minimum is specified
– It makes no sense standardizing protection schemes, cryptography ,
etc.
– Only hooks are needed
• These hooks are already included in Version 1.
Berkom
===!" §
