Slides for my talk on building custom IPython kernels at PyCon Singapore 2015

1. Building a custom
kernel for IPython
Hari Allamraju
anarahari@gmail.com
https://github.com/hari-allamraju
remitplan.com
rememberthebudget.in

2. What this talk is about
• What is an IPython kernel
• How does an IPython kernel work
• How can you build a simple wrapper kernel
• Some pointers on how to build a full fledged native
kernel

3. Why build a kernel?
• IPython does a lot of neat magic which makes it
invaluable for Python development
• There are no such tools for other languages, or even
applications which could benefit from having a nice
interactive front end
• We are developers, we can build such tools and share
them
• It’s fun to take things apart and see how they work and
could be the first step to you contributing to IPython itself

4. IPython architecture
Distributed client and kernel components talking over zeromq; separation
of client and kernel means you can add any client to the same kernel

5. Notebook is just another client
This can be extended to any client in any language or
interface

6. The two types of kernels
Changes after the recent Jupyter project

7. IPython messaging
zeromq sockets; all clients connected to a kernel receive
all messages and share the kernel context

8. Messaging spec
• You can find more details here - https://ipython.org/ipython-
doc/dev/development/messaging.html
• 4 types of channels
• Shell: request reply channel where all execution happens
• IOPub: this is where the kernel broadcasts the side effects
of any request to all clients
• Stdin: this is where the kernel requests input from the clients
• Control: for shutdown, abort etc

9. {
# The message header contains a pair of unique identifiers for the
# originating session and the actual message id, in addition to the
# username for the process that generated the message. This is useful in
# collaborative settings where multiple users may be interacting with the
# same kernel simultaneously, so that frontends can label the various
# messages in a meaningful way.
'header' : {
'msg_id' : uuid,
'username' : str,
'session' : uuid,
# All recognized message type strings are listed below.
'msg_type' : str,
# the message protocol version
'version' : '5.0',
},
# In a chain of messages, the header from the parent is copied so that
# clients can track where messages come from.
'parent_header' : dict,
# Any metadata associated with the message.
'metadata' : dict,
# The actual content of the message must be a dict, whose structure
# depends on the message type.
'content' : dict,
}

10. Full native kernel
• You have to handle all the zeromq sockets and
communication mechanism
• You have to ensure that the IPython messages are
created and parsed correctly for each type of request -
if you don’t implement something then you need to
handle that gracefully
• You have the freedom to structure your code as you
see fit
• https://github.com/dsblank/simple_kernel

11. Wrapper kernels
• Use IPython machinery to start the kernel and the
various zeromq channels which will be used to
communicate with the clients
• Provide the ability to create and parse the messages
as per the IPython messaging spec
• Give you specific end points or methods to implement
without having to worry about how it will all fit together
• https://github.com/takluyver/bash_kernel

12. Our focus - Wrapper
kernels

13. The base class
• from IPython.kernel.zmq.kernelbase import Kernel
• Available from IPython 3
• Provides you all the scaffolding needed to build the
kernel

14. The main properties
implementation
implementation_version
language
language_version
banner
Information for Kernel info replies. ‘Implementation’ refers to the kernel (e.g. IPython), and ‘language’ refers to the language it
interprets (e.g. Python). The ‘banner’ is displayed to the user in console UIs before the first prompt. All of these values are strings.
language_info
Language information for Kernel info replies, in a dictionary. This should contain the key mimetype with the mimetype of code in the
target language (e.g. 'text/x-python'), and file_extension (e.g. 'py'). It may also contain keys codemirror_mode and pygments_lexer
if they need to differ from language.
Other keys may be added to this later.
https://ipython.org/ipython-doc/dev/development/messaging.html#msging-kernel-info

15. The main method
do_execute(code, silent, store_history=True, user_expressions=None, allow_stdin=False)
Execute user code.
Parameters:
code (str) – The code to be executed.
silent (bool) – Whether to display output.
store_history (bool) – Whether to record this code in history and increase the execution count. If silent is True, this is
implicitly False.
user_expressions (dict) – Mapping of names to expressions to evaluate after the code has run. You can ignore this if you
need to.
allow_stdin (bool) – Whether the frontend can provide input on request (e.g. for Python’s raw_input()).
Your method should return a dict containing the fields described in Execution results. To display output, it can send
messages using send_response(). See Messaging in IPython for details of the different message types.

16. Launching the kernel
if __name__ == '__main__':
from IPython.kernel.zmq.kernelapp import IPKernelApp
IPKernelApp.launch_instance(kernel_class=MyKernel)

17. Let’s see some code!
• Some simple examples - echo and nodejs (with
pexpect)
• https://github.com/supercoderz/redis_kernel
• This is a wrapper kernel, connects to redis and
executes redis commands
• Almost complete in terms of functionality

18. Let’s see some code!
• Code overview of redis kernel
• More methods - https://ipython.org/ipython-doc/
dev/development/wrapperkernels.html
• This might change with Jupyter, still have to test
redis_kernel with Jupyter

19. Thank you!
Any questions?
Or you can reach me at anarahari@gmail.com