In Search of the Perfect Global Interpreter Lock

In Search of the Perfect
Global Interpreter Lock
David Beazley
http://www.dabeaz.com
@dabeaz
October 15, 2011
Presented at RuPy 2011
Poznan, Poland

Copyright (C) 2010, David Beazley, http://www.dabeaz.com 1

Introduction

• As many programmers know, Python and Ruby
feature a Global Interpreter Lock (GIL)
• More precise: CPython and MRI
• It limits thread performance on multicore
• Theoretically restricts code to a single CPU


An Experiment
• Consider a trivial CPU-bound function
def countdown(n):
while n > 0:
n -= 1

• Run it once with a lot of work
COUNT = 100000000 # 100 million
countdown(COUNT)

• Now, divide the work across two threads
t1 = Thread(target=count,args=(COUNT//2,))
t2 = Thread(target=count,args=(COUNT//2,))
t1.start(); t2.start()
t1.join(); t2.join()


An Experiment
• Some Ruby
def countdown(n)
while n > 0
n -= 1
end
end

• Sequential
COUNT = 100000000 # 100 million
countdown(COUNT)

• Subdivided across threads
t1 = Thread.new { countdown(COUNT/2) }
t2 = Thread.new { countdown(COUNT/2) }
t1.join
t2.join


Expectations

• Sequential and threaded versions perform the
same amount of work (same # calculations)
• There is the GIL... so no parallelism
• Performance should be about the same


Results
• Ruby 1.9 on OS-X (4 cores)
Sequential : 2.46s
Threaded (2 threads) : 2.55s (~ same)


Results
Sequential : 2.46s

• Python 2.7
Sequential : 6.12s
Threaded (2 threads) : 9.28s (1.5x slower!)


Results
Sequential : 2.46s

• Python 2.7
Sequential : 6.12s

• Question: Why does it get slower in Python?


Results
• Ruby 1.9 on Windows Server 2008 (2 cores)
Sequential : 3.32s


Results
Sequential : 3.32s

• Python 2.7
Sequential : 6.9s


Results
Sequential : 3.32s

• Python 2.7
Sequential : 6.9s

• Why does it get that much slower on Windows?


Experiment: Messaging

• A request/reply server for size-preﬁxed messages
Client Server

• Each message: a size header + payload
• Similar: ZeroMQ

An Experiment: Messaging
• A simple test - message echo (pseudocode)
def client(nummsg,msg): def server():
while nummsg > 0: while True:
send(msg) msg = recv()
resp = recv() send(msg)
sleep(0.001)
nummsg -= 1


• A simple test - message echo (pseudocode)
def client(nummsg,msg): def server():
while nummsg > 0: while True:
send(msg) msg = recv()
resp = recv() send(msg)
sleep(0.001)
nummsg -= 1

• To be less evil, it's throttled (<1000 msg/sec)
• Not a messaging stress test

• A test: send/receive 1000 8K messages
• Scenario 1: Unloaded server
Client Server

• Scenario 2 : Server competing with one CPU-thread
CPU-Thread

Client Server


Results
• Messaging with no threads (OS-X, 4 cores)
C : 1.26s
Python 2.7 : 1.29s
Ruby 1.9 : 1.29s


Results
• Messaging with no threads (OS-X, 4 cores)
C : 1.26s
Python 2.7 : 1.29s
Ruby 1.9 : 1.29s

• Messaging with one CPU-bound thread*
C : 1.16s (~8% faster!?)
Python 2.7 : 12.3s (10x slower)
Ruby 1.9 : 42.0s (33x slower)

• Hmmm. Curious.
* On Ruby, the CPU-bound thread
was also given lower priority

Results
• Messaging with no threads (Linux, 8 CPUs)
C : 1.13s
Python 2.7 : 1.18s
Ruby 1.9 : 1.18s


Results
C : 1.13s
Python 2.7 : 1.18s
Ruby 1.9 : 1.18s

• Messaging with one CPU-bound thread
C : 1.11s (same)
Python 2.7 : 1.60s (1.4x slower) - better
Ruby 1.9 : 5839.4s (~5000x slower) - worse!


Results
C : 1.13s
Python 2.7 : 1.18s
Ruby 1.9 : 1.18s

C : 1.11s (same)
Python 2.7 : 1.60s (1.4x slower) - better
Ruby 1.9 : 5839.4s (~5000x slower) - worse!

• 5000x slower? Really? Why?


The Mystery Deepens
• Disable all but one CPU core
• CPU-bound threads (OS-X)
Python 2.7 (4 cores+hyperthreading) : 9.28s
Python 2.7 (1 core) : 7.9s (faster!)

Ruby 1.9 (4 cores+hyperthreading) : 42.0s
Ruby 1.9 (1 core) : 10.5s (much faster!)

• ?!?!?!?!?!?

Better is Worse
• Change software versions
• Let's upgrade to Python 3 (Linux)
Python 2.7 (Messaging) : 12.3s
Python 3.2 (Messaging) : 20.1s (1.6x slower)

• Let's downgrade to Ruby 1.8 (Linux)
Ruby 1.9 (Messaging) : 42.0
Ruby 1.8.7 (Messaging) : 10.0s (4x faster)

• So much for progress (sigh)

What's Happening?

• The GIL does far more than limit cores
• It can make performance much worse
• Better performance by turning off cores?
• 5000x performance hit on Linux?
• Why?


Why You Might Care
• Must you abandon Python/Ruby for concurrency?
• Having threads restricted to one CPU core might
be okay if it were sane
• Analogy: A multitasking operating system
(e.g., Linux) runs ﬁne on a single CPU
• Plus, threads get used a lot behind the scenes
(even in thread alternatives, e.g., async)


Why I Care

• It's an interesting little systems problem
• How do you make a better GIL?
• It's fun.


Some Background
• I have been discussing some of these issues
in the Python community since 2009

http://www.dabeaz.com/GIL

• I'm less familiar with Ruby, but I've looked at
its GIL implementation and experimented
• Very interested in commonalities/differences


A Tale of Two GILs


Thread Implementation

• System threads • System threads
(e.g., pthreads) (e.g., pthreads)

• Managed by OS • Managed by OS
• Concurrent • Concurrent
execution of the execution of the
Python interpreter Ruby VM
(written in C) (written in C)


Alas, the GIL

• Parallel execution is forbidden
• There is a "global interpreter lock"
• The GIL ensures that only one thread runs in
the interpreter at once
• Simpliﬁes many low-level details (memory
management, callouts to C extensions, etc.)


GIL Implementation
int gil_locked = 0; mutex_t gil;
mutex_t gil_mutex;
cond_t gil_cond; void gil_acquire() {
mutex_lock(gil);
void gil_acquire() { }
mutex_lock(gil_mutex); void gil_release() {
while (gil_locked) mutex_unlock(gil);
cond_wait(gil_cond); }
gil_locked = 1;
mutex_unlock(gil_mutex);
} Simple mutex lock
void gil_release() {
mutex_lock(gil_mutex);
gil_locked = 0;
cond_notify();
mutex_unlock(gil_mutex);
Condition variable
}

Thread Execution Model
• The GIL results in cooperative multitasking
block block block block block
Thread 1
run run

Thread 2 run
run run
Thread 3
release acquire release acquire
GIL GIL GIL GIL

• When a thread is running, it holds the GIL
• GIL released on blocking (e.g., I/O operations)

Threads for I/O

• For I/O it works great
• GIL is never held very long
• Most threads just sit around sleeping
• Life is good


Threads for Computation
• You may actually want to compute something!
• Fibonacci numbers
• Image/audio processing
• Parsing
• The CPU will be busy
• And it won't give up the GIL on its own

CPU-Bound Switching

• Releases and • Background thread
reacquires the GIL generates a timer
every 100 "ticks" interrupt every 10ms

• 1 Tick ~= 1 interpreter • GIL released and
instruction reacquired by current
thread on interrupt


Python Thread Switching
Run 100 Run 100 Run 100
ticks ticks ticks
CPU Bound
Thread e e e e e e
as uir
le q as uir
le q as uir
le q
re ac re ac re ac

• Every 100 VM instructions, GIL is dropped,
allowing other threads to run if they want
• Not time based--switching interval depends on
kind of instructions executed


Ruby Thread Switching
Timer Timer (10ms) Timer (10ms)
Thread

CPU Bound Run Run
Thread e e
as uir
e e
as uir
le q le q
re ac re ac

• Loosely mimics the time-slice of the OS
• Every 10ms, GIL is released/acquired


A Common Theme
• Both Python and Ruby have C code like this:
void execute() {
while (inst = next_instruction()) {
// Run the VM instruction
...
if (must_release_gil) {
GIL_release();
/* Other threads may run now */
GIL_acquire();
}
}
}

• Exact details vary, but concept is the same
• Each thread has periodic release/acquire in the
VM to allow other threads to run

Question
• What can go wrong with this bit of code?
GIL_release();
GIL_acquire();
}

• Short answer: Everything!


Pathology


Thread Switching
• Suppose you have two threads
Running
Thread 1

Thread 2 READY

• Thread 1 : Running
• Thread 2 : Ready (Waiting for GIL)

Thread Switching
• Easy case : Thread 1 performs I/O (read/write)
I/O
Running
Thread 1 BLOCKED
release
GIL
pthreads/OS

schedule
Running
Thread 2 READY

acquire GIL

• Thread 1 : Releases GIL and blocks for I/O
• Thread 2 : Gets scheduled, starts running

Thread Switching
• Tricky case : Thread 1 runs until preempted pt
m
ee
Running pr
Thread 1 ???
release
GIL
pthreads/OS Which thread runs?

Thread 2 READY ???


Thread Switching
• You might expect that Thread 2 will run pt
m
ee
pr
Running
Thread 1 READY
release
GIL
pthreads/OS
acquire
schedule
GIL
Running
Thread 2 READY

• But you assume the GIL plays nice...

Thread Switching
• What might actually happen on multicore pt
m
ee
pr
Running Running
Thread 1
release acquire
GIL GIL
pthreads/OS

schedule fails (GIL locked)

Thread 2 READY READY

• Both threads attempt to run simultaneously
• ... but only one will succeed (depends on timing)

Fallacy
• This code doesn't actually switch threads
GIL_release();
GIL_acquire();
}

• It might switch threads, but it depends
• What operating system
• # cores
• Lock scheduling policy (if any)

Fallacy
• This doesn't force switching (sleeping)
GIL_release();
sleep(0);
GIL_acquire();
}

• # cores

Fallacy
• Neither does this (calling the scheduler)
GIL_release();
sched_yield()
GIL_acquire();
}

• # cores

A Conflict
• There are conflicting goals
• Python/Ruby - wants to run on a single
CPU, but doesn't want to do thread
scheduling (i.e., let the OS do it).
• OS - "Oooh. Multiple cores."
Schedules as many runnable tasks as
possible at any instant
• Result: Threads fight with each other

Multicore GIL Battle
• Python 2.7 on OS-X (4 cores)
Sequential : 6.12s

pt pt pt
em em em
p re p re pr
e
100 ticks 100 ticks
Thread 1 ... READY

release acquire release acquire
pthreads/OS Eventually...
schedule fail schedule fail
run
Thread 2 READY READY READY

• Millions of failed GIL acquisitions

Multicore GIL Battle
• You can see it! (2 CPU-bound threads)
Why >100%?

• Comment: In Python, it's very rapid
• GIL is released every few microseconds!


I/O Handling
• If there is a CPU-bound thread, I/O bound
threads have a hard time getting the GIL
Thread 1 (CPU 1) Thread 2 (CPU 2)
run sleep
preempt Network Packet
run Acquire GIL (fails)
preempt
Might repeat
preempt
100s-1000s of times
preempt
Acquire GIL (success)
run


Messaging Pathology
• Messaging on Linux (8 Cores)
Ruby 1.9 (no threads) : 1.18s
Ruby 1.9 (1 CPU thread) : 5839.4s

• Locks in Linux have no fairness
• Consequence: Really hard to steal the GIL
• And Ruby only retries every 10ms


Let's Talk Fairness
• Fair-locking means that locks have some notion
of priorities, arrival order, queuing, etc.
running waiting
t0 Lock t1 t2 t3 t4 t5

release

running waiting
t1 Lock t2 t3 t4 t5 t0

• Releasing means you go to end of line

Effect of Fair-Locking
• Ruby 1.9 (multiple cores)
Messages + 1 CPU Thread (OS-X) : 42.0s
Messages + 1 CPU Thread (Linux) : 5839.4s

• Question: Which one uses fair locking?


Messages + 1 CPU Thread (OS-X) : 42.0s (Fair)

• Beneﬁt : I/O threads get their turn (yay!)



• Python 2.7 (multiple cores)
2 CPU-Bound Threads (OS-X) : 9.28s
2 CPU-Bound Threads (Windows) : 63.0s

• Question: Which one uses fair-locking?


• Python 2.7 (multiple cores)
2 CPU-Bound Threads (OS-X) : 9.28s
2 CPU-Bound Threads (Windows) : 63.0s (Fair)

• Problem: Too much context switching

Fair-Locking - Bah!
• In reality, you don't want fairness
• Messaging Revisited (OS X, 4 Cores)
Ruby 1.9 (No Threads) : 1.29s
Ruby 1.9 (1 CPU-Bound thread) : 42.0s (33x slower)

• Why is it still 33x slower?
• Answer: Fair locking! (and convoying)


Messaging Revisited
• Go back to the messaging server
def server():
while True:
msg = recv()
send(msg)


Messaging Revisited
• The actual implementation (size-preﬁxed messages)
def server():
while True:
size = recv(4)
msg = recv(size)
send(size)
send(msg)


Performance Explained
• What actually happens under the covers
def server():
while True:
GIL release size = recv(4)
GIL release
msg = recv(size)
GIL release
send(size)
GIL release
send(msg)

• Why? Each operation might block
• Catch: Passes control back to CPU-bound thread

Performance Illustrated
Timer 10ms 10ms 10ms 10ms 10ms
Thread
run
CPU Bound
Thread
run run run run run
I/O recv recv send send done
Thread
Data
Arrives

• Each message has 40ms response cycle
• 1000 messages x 40ms = 40s (42.0s measured)

Despair


A Solution?
Don't use threads!

• Yes, yes, everyone hates threads
• However, that's only because they're useful!
• Threads are used for all sorts of things
• Even if they're hidden behind the scenes


A Better Solution
Make the GIL better

• It's probably not going away (very difﬁcult)
• However, does it have to thrash wildly?
• Question: Can you do anything?


GIL Efforts in Python 3

• Python 3.2 has a new GIL implementation
• It's imperfect--in fact, it has a lot of problems
• However, people are experimenting with it


Python 3 GIL
• GIL acquisition now based on timeouts
running
Thread 1
drop_request release

5ms
running
Thread 2 IOWAIT READY
wait(gil, TIMEOUT) wait(gil, TIMEOUT)

data
arrives

• Involves waiting on a condition variable

Problem: Convoying
• CPU-bound threads signiﬁcantly degrade I/O
running running running
Thread 1
release
5ms 5ms 5ms
run run
Thread 2 READY READY READY

data data data
arrives arrives arrives

• This is the same problem as in Ruby
• Just a shorter time delay (5ms)

Problem: Convoying

• You can directly observe the delays (messaging)
Python/Ruby (No threads) : 1.29s (no delays)
Python 3.2 (1 Thread) : 20.1s (5ms delays)
Ruby 1.9 (1 Thread) : 42.0s (10ms delays)

• Still not great, but problem is understood


Promise


Priorities
• Best promise : Priority scheduling
• Earlier versions of Ruby had it
• It works (OS-X, 4 cores)
Ruby 1.9 (1 Thread) : 42.0s
Ruby 1.8.7 (1 Thread) : 40.2s
Ruby 1.8.7 (1 Thread, lower priority) : 10.0s

• Comment: Ruby-1.9 allows thread priorities to be
set in pthreads, but it doesn't seem to have much
(if any) effect

Priorities
• Experimental Python-3.2 with priority scheduler
• Also features immediate preemption
• Messages (OS X, 4 Cores)
Python 3.2 (No threads) : 1.29s
Python 3.2 (1 Thread) : 20.2s
Python 3.2+priorities (1 Thread) : 1.21s (faster?)

• That's a lot more promising!


New Problems
• Priorities bring new challenges
• Starvation
• Priority inversion
• Implementation complexity
• Do you have to write a full OS scheduler?
• Hopefully not, but it's an open question

Final Words

• Implementing a GIL is a lot trickier than it looks
• Even work with priorities has problems
• Good example of how multicore is diabolical


Thanks for Listening!

• I hope you learned at least one new thing
• I'm always interested in feedback
• Follow me on Twitter (@dabeaz)


In Search of the Perfect Global Interpreter Lock

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to In Search of the Perfect Global Interpreter Lock

Similar to In Search of the Perfect Global Interpreter Lock (20)

Recently uploaded

Recently uploaded (20)

In Search of the Perfect Global Interpreter Lock