How AI, OpenAI, and ChatGPT impact business and software.
Kurento cpmx
1. Real-time Multimedia Stream Processing
Developing rich multimedia applications with Kurento
The Stream oriented GE:
Developing rich multimedia applications with Kurento
igracia@naevatec.com
h0p://www.kurento.org
Open APIs for Open Minds
h"ps://join.me/kurentoCPMX
2. Mul8media
infrastructures
for
the
Future
Internet
Media
is
here
Media
got
there
Media
got
there
Analyze
Transform
Store
Transport
Enrich
Augment
Adapt
Sensors
Context
Events
Media
is
here
2
5. • Interoperable
media
exchange
(mul8plaUorm/mul8protocol)
• WebRTC,
RTP,
HTTP
(video
tag),
etc.
• Process
media
(Computer
vision,
augmented
reality,
media
indexing,
etc.)
• Media
and
metadata
recording
and
recovery
• Transform
and
adapt
media
(H.264,
H.263,
VP8,
Ogg,
and
others)
• Media
rou8ng
and
mixing
• Etc.
Mul8media
infrastructure
• REST
API
• JavaScript
API
• Java
API
APIs
• LGPL
2.1
Is
distributed
through
a
flexible
FOSS
license
What’s
Kurento
5
6. Kurento
Media
Server
(KMS):
the
nucleus
of
Kurento
• KMS
is
a
middleware
for
media
streams
– Receives
the
stream
– Process
the
stream
– Issues
the
stream
Send
Receive
Analyze
Augment
Enrich
Transform
Transcode
Record
Process
Replicate
Media
Source
Media
Sink
KMS
6
7. The
Media
API:
The
API
for
accessing
KMS
capabili8es
Send
Receive
Analyze
Augment
Enrich
Transform
Transcode
Record
Process
Replicate
Media
Source
Media
Sink
KMS
Java
Media
API
JavaScript
Media
API
REST
API
Applica8ons
define
the
processing
of
streams
geeng
through
KMS
7
8. Media
API:
Media
Elements
and
Media
Pipelines
Sink%SRC%
Sink%
SRC%
SRC%Sink%
Sink%
§ Media Element
• Provides a specific media
functionality
› Send/receive media
› Process media
› Transform media
• Exchange media through
› Sources
› Sinks
§ Media pipeline
• Chain of media elements
implementing the desired media
logic.
• The Media API provides the
capability of creating media
pipelines by joining media
elements of the toolbox
Media
Element
Sink
SRC
8
9. Media
API:
trivial
example
(Java)
MediaPipeline
mp
=
contentSession.getMediaPipelineFactory().create();
PlayerEndpoint
playerEndpoint
=
mp.newPlayerEndpoint(“file:///myFile.webm”).build();
H2pGetEndpoint
h"pEndpoint
=
mp.newH"pGetEndpoint().terminateOnEOS().build();
playerEndpoint.connect(h"pEndpoint);
h"pEndpoint.getUrl();
//URL
where
the
media
is
made
available
Media
Pipeline
(inside
KMS)
H"pGetEndpoint
Media
from
file
or
URI
HTTP
media
streaming
Sink
SRC
PlayerEndpoint
9
10. Is
that
enough?
Think
about
the
WWW
development
model
Process
WWW
request
-‐ DDBB
access
-‐ Authen8ca8on
-‐ XML
processing
-‐ Etc.
HTTP
request:
I
want
this
resource
HTTP
response:
The
resource
• Intui8on
behind
the
WWW
model
– Client
asks
what
it
wants
• GET
/this_resource
HTTP/1.1
– Server
side
APIs
execute
the
associated
processing
You
develop
this
using
…
• Servlet
• JSP
• ASP
• PHP
• Ruby
on
Rails
• CGI
• Etc.
10
11. We
need
an
equivalent
model:
the
role
of
the
Signaling
Plane
• We
need
“something”
with
the
capability
of
nego8a8ng
what,
when
and
how
media
is
exchanged
• A
protocol
for
“saying”
– I
want
“this
media”
…
• Iden8fica8on
of
the
media
to
exchange
– File
in
hard-‐drive,
IP
camera,
user,
etc.
– in
“this
way”
…
• Iden8fica8on
of
the
processing
of
media
– Augmented,
analyzed,
etc.
– with
“this
format”
…
• Quality
– Codec,
screen-‐size,
frame-‐rate,
etc.
– at
“this
moment”
• Stream
control
– Play,
stop,
start,
pause,
etc.
11
12. Don’t
get
it?
think
about
WWW
development
again
…
Process
WWW
request
-‐ DDBB
access
-‐ Authen8ca8on
-‐ XML
processing
-‐ Etc.
HTTP
request:
I
want
this
resource
HTTP
response:
The
resource
Process
media
request
-‐ Media
API
-‐ DDBB
access
-‐ Authen8ca8on
-‐ XML
processing
-‐ Etc.
Signaling
request:
I
want
this
media
Signaling
response:
The
media
is
here
This
is
how
you
create
WWW
applica8ons
(Servlets,
ASP,
PHP,
Rails,
etc.)
This
is
how
you
create
Kurento
enabled
applica8ons:
Mul8media
RTC
is
just
another
feature
of
your
applica8on
You,
as
a
programmer,
create
this
12
13. Abstrac8ng
the
signaling
plane:
The
Content
Handler
KMS
Sink
SRC
Sink
SRC
Sink
SRC
Sink
Media
API
REST
API
(Open
API
protocol)
The
Content
Handler
Equivalent
to
a
Servlet/ASP/PHP
script
Exposes
APIs
for
specifying:
-‐ when
receiving
“this
request”…
-‐ …
execute
“this
logic”
Developer
can
use
the
media
API
Code
building
the
media
pipeline
and
execu8ng
the
applica8on
logic
the
developer
wants
13
14. Kurento
Applica8on
Server:
The
container
of
Handlers
• Is
an
extension
of
Java
EE
technologies.
• Compa8ble
with
all
Java
EE
Servlet
containers
• Hold
Handlers
– Your
specific
applica8on
code
• Receives
signaling
requests:
– I
want
“this
media”
in
this
way
…
• Dispatches
request
to
the
appropriate
handler
– @annota8ons
based
mapping
• Generate
an
answer
showing
“where
and
how
to
find
the
media”
– URL
where
media
is
located
– Nego8ated
SDP
Java
EE
compaJble
container
HTTP
Servlet
SIP
Servlet
Web
services
Kurento
REST
API
Specific
handler
implementa8ons
Media
API
DD.BB.
Kurento
ApplicaJon
Server
(KAS)
Other
java
APIs.
14
15. Kurento
Architecture:
pueng
it
all
together
Kurento
Media
Server
(KMS)
Receive
Video
Augmented
Reality
Send
Video
Computer
Vision
Video
Playing
and
Recording
Java
EE
compaJble
container
HTTP
Servlet
SIP
Servlet
Web
services
Kurento
REST
API
Specific
handler
implementa8ons
Signaling
and
WWW
traffic
Media
Media
Signaling
and
WWW
traffic
Media
API
DD.BB.
Kurento
ApplicaJon
Server
(KAS)
Other
java
APIs.
15
16. Applica8on
execu8on
flow
Client
Code
Applica8on
Server
(KAS)
Media
Server
(KMS)
I
want
this
media
in
this
way
…
(JSON-‐RPC
request)
Commands
reques8ng
the
crea8on
of
a
pipeline
What
you
want
is
here
…
(JSON-‐RPC
answer)
Media
negoJaJon
phase
Media
exchange
phase
1
2
Specific
applica8on
logic
at
the
server-‐side
(Content
Handler)
Media
pipeline
creaJon
Media
exchange
between
client
and
server
16
17. Content
Handler:
trivial
example
//Specify
the
type
of
service
provided
by
this
handler:
Player,
Recorder,
WebRTC,
RTP,
etc.
@H2pPlayerService(path
=
"/player”)
//Mapping
of
handler
specified
in
path
public
class
MyPlayerHandler
extends
H"pPlayerHandler
{
@Override
public
void
onContentRequest(H"pPlayerSession
contentSession)
{
//Thie
client
wants
the
media
this
handler
provides
//Create
the
pipeline
for
providing
the
media
}
@Override
public
void
onContentStarted(H"pPlayerSession
contentSession)
{
//Media
started
flowing,
you
can
execute
addi8onal
ac8ons
}
@Override
Public
void
onSessionTerminated(H"pPlayerSession
contentSenssion){
//Media
exchange
termianted,
you
can
collect
your
resources
}
17
18. Let’s
develop
with
Kurento
• What
you
need
– A
Kurento
instance
• You
can
install
your
own
Kurento
instance
• You
can
launch
a
Kurento
instance
at
the
FI-‐LAB
(FI-‐WARE
project)
– h"p://lab.fi-‐ware.org
• Geeng
help
– FI-‐WARE
catalogue
• h"p://catalogue.fi-‐ware.org/enablers/stream-‐oriented-‐kurento
– Kurento
web
site
• h"p://www.kurento.org
– Kurento
mailing
list
• h"ps://groups.google.com/forum/#!forum/kurento
– Twi"er
• @kurentoms
19. • FI-‐LAB
– Working
instance
of
FI-‐WARE
enabling
free
experimenta8on
with
technology
– h"p://lab.fi-‐ware.org
• Crea8ng
a
Kurento
instance
from
an
image
– Use
latest
version
of
Kurento
images.
• Crea8ng
a
Kurento
instance
using
recipes
– Use
Ubuntu
13.10
clear
image
– Use
latest
version
of
Kurento
recipes
• Accessing
the
Kurento
demo
at
FI-‐LAB
– h"p://kurento.lab.fi-‐ware.org:8080/fi-‐lab-‐demo
19
Kurento
at
the
FI-‐LAB
20. Kurento
Hello
World:
Playing
a
file
with
an
HTML5
client
Media
Pipeline
H"pGetEndpoint
Media
from
file
or
URI
HTTP
media
streaming
Sink
SRC
PlayerEndpoint
Media
API
REST
API
(Open
API
protocol)
Handler
code
I
want
“this
media”
Media
is
“at
this
URL”
20
21. Playing
a
file
with
an
HTML5
client:
Handler
code
@H"pPlayerService(path
=
"/player”)
public
class
MyPlayerHandler
extends
H"pPlayerHandler
{
@Override
public
void
onContentRequest(H"pPlayerSession
contentSession)
throws
Excep8on
{
MediaPipeline
mp
=
contentSession.getMediaPipelineFactory().create();
contentSession.releaseOnTerminate(mp);
PlayerEndpoint
playerEndpoint
=
mp.newPlayerEndpoint(
"h2p://media.w3.org/2010/05/sintel/trailer.webm").build();
contentSession.setA"ribute("player",
playerEndpoint);
H2pGetEndpoint
h2pEndpoint
=
mp.newH2pGetEndpoint().terminateOnEOS().build();
playerEndpoint.connect(h2pEndpoint);
contentSession.start(h"pEndpoint);
}
@Override
public
void
onContentStarted(H"pPlayerSession
contentSession)
{
PlayerEndpoint
playerEndpoint
=
(PlayerEndpoint)
contentSession.getA"ribute("player");
playerEndpoint.play();
}
}
Source:
h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/
MyPlayerHandler.java
21
22. Playing
a
file
with
an
HTML5
client:
client
code
<!DOCTYPE
html>
<html>
<head>
<script
src="./js/kws-‐content-‐api.js"></script>
<script>
var
conn;
func8on
start()
{
var
op8ons
=
{
remoteVideoTag
:
"remoteVideo"
};
conn
=
new
kwsContentApi.KwsContentPlayer("./player",
op8ons);
}
func8on
terminate()
{
conn.terminate();
}
</script>
</head>
<body>
<bu"on
onclick="start();">Start</bu"on>
<bu"on
onclick="terminate();">Terminate</bu"on>
<br
/>
<video
id="remoteVideo"
autoplay></video>
</body>
</html>
Source:
h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/
webapp/player.html
22
23. Playing
a
file
with
an
HTML5
client:
See
the
example
working
h"ps://www.youtube.com/watch?v=n5BQlhYgGSo
23
24. Media
Pipeline
Adding
Computer
Vision
H"pGetEndpoint
Media
from
file
or
URI
HTTP
media
streaming
Sink
SRC
PlayerEndpoint
SRC
Sink
JackVaderFilter
Media
API
REST
API
(Open
API
protocol)
Handler
code
I
want
“this
media”
Media
is
“at
this
URL”
24
27. Adding
Computer
Vision:
See
the
example
working
h"ps://www.youtube.com/watch?v=yJAQs23eoXw
27
28. WebRTC
loopback
Kurento
Media
Server
WebRTC
Streaming
Media
API
JSON-‐RPC
API
Handler
code
I
want
“this
media”
(offer
SDP)
Media
is
“here”
(answer
SDP)
Sink
SRC
28
Kurento
ApplicaJon
Server
WebRtcEndpoint
Client
Browser
31. WebRTC
loopback:
see
the
example
working
h"ps://www.youtube.com/watch?v=HaVqO06uuNA
31
32. WebRTC
recording
Media
Pipeline
WebRTC
Streaming
Media
API
REST
API
(Open
API
protocol)
Handler
code
I
want
“this
media
(SDP)”
Media
is
“at
here
(SDP)”
Sink
SRC
Sink
Media
to
file
or
URI
32
33. WebRTC
recorder:
Handler
code
33
Source
h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/
tutorial/MyWebRtcRecorder.java
Source
of
handler
playing
the
recorded
video:
h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/
tutorial/MyPlayerRecording.java
@WebRtcContentService(path
=
"/webRtcRecorder")
public
class
MyWebRtcRecorder
extends
WebRtcContentHandler
{
@Override
public
void
onContentRequest(WebRtcContentSession
contentSession)
throws
Excep8on
{
MediaPipeline
mp
=
contentSession.getMediaPipelineFactory().create();
contentSession.releaseOnTerminate(mp);
WebRtcEndpoint
webRtcEndpoint
=
mp.newWebRtcEndpoint().build();
RecorderEndpoint
recorderEndpoint
=
mp.newRecorderEndpoint("file:///tmp/recording").build();
contentSession.setA"ribute("recorder",
recorderEndpoint);
webRtcEndpoint.connect(webRtcEndpoint);
webRtcEndpoint.connect(recorderEndpoint);
contentSession.start(webRtcEndpoint);
}
@Override
public
void
onContentStarted(WebRtcContentSession
contentSession)
{
RecorderEndpoint
recorderEndpoint
=
(RecorderEndpoint)
contentSession.getA"ribute("recorder");
recorderEndpoint.record();
}
}
33
43. WebRTC
game:
see
the
example
working
h"ps://www.youtube.com/watch?v=5eJRnwKxgbY
43
44. Complex
applica8on
44
Result:
h"ps://www.youtube.com/watch?v=PYCw9-‐4oWWA
Sink
SRC
Sink
Sink
SRC
Sink
SRC
Sink
SRC
Sink
SRC
WebRtcEndpoint
MirrorFilter
PointerDetectorFilter
ChromaFilter
FaceOverlayFilter
Media
Pipeline
• Beyond
video-‐conferencing:
receiving
events
from
the
media
stream
– Immersive
and
personalized
adver8sing
– Games
and
entertainment
Events
(bu"ons)
Game
or
adver8sement
logic
Change
face
Change
background
