SlideShare a Scribd company logo
1 of 30
Download to read offline
Python⾼级内存管理
- xiaorui.cc
Object-specific allocators
_____ ______ ______ ________
[ int ] [ dict ] [ list ] ... [ string ] Python core |
+3 | <----- Object-specific memory -----> | <-- Non-object memory --> |
_______________________________ | |
[ Python's object allocator ] | |
+2 | ####### Object memory ####### | <------ Internal buffers ------> |
______________________________________________________________ |
[ Python's raw memory allocator (PyMem_ API) ] |
+1 | <----- Python memory (under PyMem manager's control) ------> | |
__________________________________________________________________
[ Underlying general-purpose allocator (ex: C library malloc) ]
0 | <------ Virtual memory allocated for the python process -------> |
=========================================================================
_______________________________________________________________________
[ OS-specific Virtual Memory Manager (VMM) ]
-1 | <--- Kernel dynamic storage allocation & management (page-based) ---> |
__________________________________ __________________________________
[ ] [ ]
-2 | <-- Physical memory: ROM/RAM --> | | <-- Secondary storage (swap) --> |
* Request in bytes Size of allocated block Size class idx
* ----------------------------------------------------------------
* 1-8 8 0
* 9-16 16 1
* 17-24 24 2
* 25-32 32 3
* 33-40 40 4
* 41-48 48 5
* 49-56 56 6
* 57-64 64 7
* ... ... ...
* 497-504 504 62
* 505-512 512 63
*
*
*/
名词解释
process heap
Arenas
Pool
UsedPools
FreePools
method
posix malloc
python memory pool
object buffer pool
Arena
FeeePool
Pool
Pool
Pool
Pool
Headers
No BLock
Arena
malloc heap &
pool
Process
stack
heap
bss
init data
text
UserPool
1-8
… …
249 - 256
Pool
Free Block
Free Block
Use Block
userdpool design
UserdPools
1-8
9-16
17-24
…
…
249-256
Pool
Header
Free Block
Free Block
Header
分配
回收
…
Pool
Free Block
Free Block
Use Block
同⼀个Pool下Block⼀样长
单Pool为4kb
Block及Pool都为单链表
free pool desgin
FeeePool
Pool
Pool
Pool
…
Pool
Headers
No BLock
Pool
Headers
No BLock
Pool为4kb⼤小
Pool清理Headers
where store variable ?
run-time Stack
list
dict
int
heap
[1 ,2, 3]
{“n”: “1”}
1
why ?
In [1]: a = 123
In [2]: b = 123
In [3]: a is b
Out[3]: True
In [4]: a = 1000
In [5]: b = 1000
In [6]: a is b
Out[6]: False
In [7]: a = 'n'
In [8]: b = 'n'
In [9]: a is b
Out[9]: True
In [10]: a = "python"
In [11]: b = "python"
In [12]: a is b
Out[12]: True
why ?
In [1]: def go(var):
...: print id(var)
…:
In [2]: id(a)
Out[2]: 4401335072
In [3]: go(a)
4401335072
In [10]: a = b = 'nima'
In [11]: b = a
In [12]: a is b
Out[12]: True
In [13]: b = 'hehe'
In [14]: a is b
Out[14]: False
只有引用 ?
python objects stored in memory?
names
names
object
Python Has Names, Not Variables ! ! !
整数对象池
-5 -4 … 0 … 256 257
小整数 ⼤整数
… … -5 -4 … … 257 … …
var_1 var_2
… … -5 -4 … … 257 … …
var_3 var_4
the same addr !
not the same addr !
28 bytes
解释器初始化
整数对象池
Block List
Free List
PyIntBlock PyIntBlock
PyIntBlock PyIntBlock
不会归还给 Arena和os ! ! !
字符对象池
a b c d … … …
var_1 var_2
the same addr !
单个字符38 bytes
由解释器初始化
字符串对象池
aa en cao oh woyao buyao kuai feile
var_1
0 1 2 3 …
var_2
hash存储变量
共用地址
记录引用计数
ref
ref count
x = 300
y = x
z = [x, y]
X
ref += 1
300
y
Z
ref += 1
ref += 2
References -> 4 !
What does del do?
x = 300
y = x
del x
X
ref -= 1
300
y
References -> 1!The del statement doesn’t delete objects.
• removes that name as a reference to that object
• reduces the ref count by 1
ref count case
def go():
w = 300
go()
a = “fuc . ”
del a
b = “en, a”
b = None
ref count +1
w is out of scope; ref count -1
del a; ref count -1
重新赋值; ref count -1
cyclical refclass Node:
def __init__(self, va):
self.va = va
def next(self, next):
self.next = next
mid = Node(‘root’)
left = Node(‘left’)
right = Node(‘right’)
mid(left)
left.next(right)
right.next(left)
Mid
rightleft
if del mid node:
how ?
mark & sweep
gc root
b
a
w
c
K G
R
分代回收
node node node node node node node
node node node node node node
node node node node node node
可变 vs 不可变 (obj)
string
int
tuple
list
dict
container objects
a = [10, 10, 11]
b = a
PyListObject
Type list
rc 1
items
size
… …
PyObject
Type integer
rc 2
value 10
PyObject
Type integer
rc 1
value 11
10
10
11
copy.copy
a = [10, 10, [10, 11] ]
b = copy.copy(a)
PyListObject
Type list
rc 1
items
size
… …
PyObject
Type integer
rc 2
value 10
PyObject
Type integer
rc 1
value 11
10
10
ref
PyListObject
Type list
rc 1
items
size
… …
10
10
ref
PyListObject
10
11
copy.deepcopy
a = [10, [ 10, 11 ] ]
b = copy.deep(a)
PyListObject
Type list
rc 1
items
size
… …
PyObject
Type integer
rc 2
value 10
PyObject
Type integer
rc 1
value 11
10
ref
PyListObject
Type list
rc 1
items
size
… …
10
ref
PyListObject
10
11
PyListObject
10
11
diy gc
import gc
import sys
gc.set_debug(gc.DEBUG_STATS|gc.DEBUG_LEAK)
a=[]
b=[]
a.append(b)
print 'a refcount:',sys.getrefcount(a) # 2
print 'b refcount:',sys.getrefcount(b) # 3
del a
del b
print gc.collect() # 0
Garbage Collector Optimize
memory bound
可以降低threshold来时间换空间
cpu bound
提⾼threshold来空间换时间
暂停gc, 引⼊master worker设计
引用计数 跟 gil 的影响 ?
gc 是否是原⼦ ?
gc的 stop the world现象 ?
…
Q & A
“ END ”
– xiaorui.cc

More Related Content

What's hot

Groovy collection api
Groovy collection apiGroovy collection api
Groovy collection apitrygvea
 
Webinar: Replication and Replica Sets
Webinar: Replication and Replica SetsWebinar: Replication and Replica Sets
Webinar: Replication and Replica SetsMongoDB
 
Damn Fine CoffeeScript
Damn Fine CoffeeScriptDamn Fine CoffeeScript
Damn Fine CoffeeScriptniklal
 
CUDA First Programs: Computer Architecture CSE448 : UAA Alaska : Notes
CUDA First Programs: Computer Architecture CSE448 : UAA Alaska : NotesCUDA First Programs: Computer Architecture CSE448 : UAA Alaska : Notes
CUDA First Programs: Computer Architecture CSE448 : UAA Alaska : NotesSubhajit Sahu
 
Python For Data Science Cheat Sheet
Python For Data Science Cheat SheetPython For Data Science Cheat Sheet
Python For Data Science Cheat SheetKarlijn Willems
 
Advanced Data Visualization Examples with R-Part II
Advanced Data Visualization Examples with R-Part IIAdvanced Data Visualization Examples with R-Part II
Advanced Data Visualization Examples with R-Part IIDr. Volkan OBAN
 
Building Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at StripeBuilding Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at StripeMongoDB
 
Exploring slides
Exploring slidesExploring slides
Exploring slidesakaptur
 
Haskellで学ぶ関数型言語
Haskellで学ぶ関数型言語Haskellで学ぶ関数型言語
Haskellで学ぶ関数型言語ikdysfm
 
집단지성 프로그래밍 08-가격모델링
집단지성 프로그래밍 08-가격모델링집단지성 프로그래밍 08-가격모델링
집단지성 프로그래밍 08-가격모델링Kwang Woo NAM
 
밑바닥부터 시작하는 의료 AI
밑바닥부터 시작하는 의료 AI밑바닥부터 시작하는 의료 AI
밑바닥부터 시작하는 의료 AINAVER Engineering
 
Python postgre sql a wonderful wedding
Python postgre sql   a wonderful weddingPython postgre sql   a wonderful wedding
Python postgre sql a wonderful weddingStéphane Wirtel
 
Intoduction to dynamic memory allocation
Intoduction to dynamic memory allocationIntoduction to dynamic memory allocation
Intoduction to dynamic memory allocationUtsav276
 
Frsa
FrsaFrsa
Frsa_111
 
Pybelsberg — Constraint-based Programming in Python
Pybelsberg — Constraint-based Programming in PythonPybelsberg — Constraint-based Programming in Python
Pybelsberg — Constraint-based Programming in PythonChristoph Matthies
 
Python seaborn cheat_sheet
Python seaborn cheat_sheetPython seaborn cheat_sheet
Python seaborn cheat_sheetNishant Upadhyay
 

What's hot (20)

Groovy collection api
Groovy collection apiGroovy collection api
Groovy collection api
 
Webinar: Replication and Replica Sets
Webinar: Replication and Replica SetsWebinar: Replication and Replica Sets
Webinar: Replication and Replica Sets
 
Damn Fine CoffeeScript
Damn Fine CoffeeScriptDamn Fine CoffeeScript
Damn Fine CoffeeScript
 
CUDA First Programs: Computer Architecture CSE448 : UAA Alaska : Notes
CUDA First Programs: Computer Architecture CSE448 : UAA Alaska : NotesCUDA First Programs: Computer Architecture CSE448 : UAA Alaska : Notes
CUDA First Programs: Computer Architecture CSE448 : UAA Alaska : Notes
 
Python For Data Science Cheat Sheet
Python For Data Science Cheat SheetPython For Data Science Cheat Sheet
Python For Data Science Cheat Sheet
 
Advanced Data Visualization Examples with R-Part II
Advanced Data Visualization Examples with R-Part IIAdvanced Data Visualization Examples with R-Part II
Advanced Data Visualization Examples with R-Part II
 
Android Guava
Android GuavaAndroid Guava
Android Guava
 
Building Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at StripeBuilding Real Time Systems on MongoDB Using the Oplog at Stripe
Building Real Time Systems on MongoDB Using the Oplog at Stripe
 
Exploring slides
Exploring slidesExploring slides
Exploring slides
 
Haskellで学ぶ関数型言語
Haskellで学ぶ関数型言語Haskellで学ぶ関数型言語
Haskellで学ぶ関数型言語
 
Py3k
Py3kPy3k
Py3k
 
Elm: give it a try
Elm: give it a tryElm: give it a try
Elm: give it a try
 
집단지성 프로그래밍 08-가격모델링
집단지성 프로그래밍 08-가격모델링집단지성 프로그래밍 08-가격모델링
집단지성 프로그래밍 08-가격모델링
 
밑바닥부터 시작하는 의료 AI
밑바닥부터 시작하는 의료 AI밑바닥부터 시작하는 의료 AI
밑바닥부터 시작하는 의료 AI
 
Python postgre sql a wonderful wedding
Python postgre sql   a wonderful weddingPython postgre sql   a wonderful wedding
Python postgre sql a wonderful wedding
 
Intoduction to dynamic memory allocation
Intoduction to dynamic memory allocationIntoduction to dynamic memory allocation
Intoduction to dynamic memory allocation
 
Frsa
FrsaFrsa
Frsa
 
Pybelsberg — Constraint-based Programming in Python
Pybelsberg — Constraint-based Programming in PythonPybelsberg — Constraint-based Programming in Python
Pybelsberg — Constraint-based Programming in Python
 
Python seaborn cheat_sheet
Python seaborn cheat_sheetPython seaborn cheat_sheet
Python seaborn cheat_sheet
 
Introduzione a C#
Introduzione a C#Introduzione a C#
Introduzione a C#
 

Viewers also liked

cdn的那些事儿
cdn的那些事儿cdn的那些事儿
cdn的那些事儿rfyiamcool
 
Development with Qt for Windows CE
Development with Qt for Windows CEDevelopment with Qt for Windows CE
Development with Qt for Windows CEaccount inactive
 
领域驱动设计与模型驱动开发
领域驱动设计与模型驱动开发领域驱动设计与模型驱动开发
领域驱动设计与模型驱动开发Weijun Zhong
 
Enterprise Architecture Implementation And The Open Group Architecture Framew...
Enterprise Architecture Implementation And The Open Group Architecture Framew...Enterprise Architecture Implementation And The Open Group Architecture Framew...
Enterprise Architecture Implementation And The Open Group Architecture Framew...Alan McSweeney
 

Viewers also liked (6)

cdn的那些事儿
cdn的那些事儿cdn的那些事儿
cdn的那些事儿
 
Development with Qt for Windows CE
Development with Qt for Windows CEDevelopment with Qt for Windows CE
Development with Qt for Windows CE
 
领域驱动设计与模型驱动开发
领域驱动设计与模型驱动开发领域驱动设计与模型驱动开发
领域驱动设计与模型驱动开发
 
Enterprise Architecture Implementation And The Open Group Architecture Framew...
Enterprise Architecture Implementation And The Open Group Architecture Framew...Enterprise Architecture Implementation And The Open Group Architecture Framew...
Enterprise Architecture Implementation And The Open Group Architecture Framew...
 
Micro service
Micro serviceMicro service
Micro service
 
InnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick FiguresInnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick Figures
 

Similar to python高级内存管理

An overview of Python 2.7
An overview of Python 2.7An overview of Python 2.7
An overview of Python 2.7decoupled
 
A Few of My Favorite (Python) Things
A Few of My Favorite (Python) ThingsA Few of My Favorite (Python) Things
A Few of My Favorite (Python) ThingsMichael Pirnat
 
python-cheatsheets.pdf
python-cheatsheets.pdfpython-cheatsheets.pdf
python-cheatsheets.pdfKalyan969491
 
python-cheatsheets that will be for coders
python-cheatsheets that will be for coderspython-cheatsheets that will be for coders
python-cheatsheets that will be for coderssarafbisesh
 
Profiling and optimization
Profiling and optimizationProfiling and optimization
Profiling and optimizationg3_nittala
 
Обзор фреймворка Twisted
Обзор фреймворка TwistedОбзор фреймворка Twisted
Обзор фреймворка TwistedPython Meetup
 
Обзор фреймворка Twisted
Обзор фреймворка TwistedОбзор фреймворка Twisted
Обзор фреймворка TwistedMaxim Kulsha
 
Functional Programming inside OOP? It’s possible with Python
Functional Programming inside OOP? It’s possible with PythonFunctional Programming inside OOP? It’s possible with Python
Functional Programming inside OOP? It’s possible with PythonCarlos V.
 
Introduction to Python
Introduction to PythonIntroduction to Python
Introduction to PythonUC San Diego
 
PHP 7 – What changed internally? (Forum PHP 2015)
PHP 7 – What changed internally? (Forum PHP 2015)PHP 7 – What changed internally? (Forum PHP 2015)
PHP 7 – What changed internally? (Forum PHP 2015)Nikita Popov
 
PHP 7 – What changed internally? (PHP Barcelona 2015)
PHP 7 – What changed internally? (PHP Barcelona 2015)PHP 7 – What changed internally? (PHP Barcelona 2015)
PHP 7 – What changed internally? (PHP Barcelona 2015)Nikita Popov
 
Python 101 language features and functional programming
Python 101 language features and functional programmingPython 101 language features and functional programming
Python 101 language features and functional programmingLukasz Dynowski
 
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)MongoSF
 
Python fundamentals - basic | WeiYuan
Python fundamentals - basic | WeiYuanPython fundamentals - basic | WeiYuan
Python fundamentals - basic | WeiYuanWei-Yuan Chang
 
Replication and Replica Sets
Replication and Replica SetsReplication and Replica Sets
Replication and Replica SetsMongoDB
 
Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)
Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)
Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)James Clause
 
[1062BPY12001] Data analysis with R / week 2
[1062BPY12001] Data analysis with R / week 2[1062BPY12001] Data analysis with R / week 2
[1062BPY12001] Data analysis with R / week 2Kevin Chun-Hsien Hsu
 

Similar to python高级内存管理 (20)

An overview of Python 2.7
An overview of Python 2.7An overview of Python 2.7
An overview of Python 2.7
 
A tour of Python
A tour of PythonA tour of Python
A tour of Python
 
A Few of My Favorite (Python) Things
A Few of My Favorite (Python) ThingsA Few of My Favorite (Python) Things
A Few of My Favorite (Python) Things
 
python-cheatsheets.pdf
python-cheatsheets.pdfpython-cheatsheets.pdf
python-cheatsheets.pdf
 
python-cheatsheets that will be for coders
python-cheatsheets that will be for coderspython-cheatsheets that will be for coders
python-cheatsheets that will be for coders
 
Profiling and optimization
Profiling and optimizationProfiling and optimization
Profiling and optimization
 
Обзор фреймворка Twisted
Обзор фреймворка TwistedОбзор фреймворка Twisted
Обзор фреймворка Twisted
 
Обзор фреймворка Twisted
Обзор фреймворка TwistedОбзор фреймворка Twisted
Обзор фреймворка Twisted
 
Functional Programming inside OOP? It’s possible with Python
Functional Programming inside OOP? It’s possible with PythonFunctional Programming inside OOP? It’s possible with Python
Functional Programming inside OOP? It’s possible with Python
 
Introduction to Python
Introduction to PythonIntroduction to Python
Introduction to Python
 
PHP 7 – What changed internally? (Forum PHP 2015)
PHP 7 – What changed internally? (Forum PHP 2015)PHP 7 – What changed internally? (Forum PHP 2015)
PHP 7 – What changed internally? (Forum PHP 2015)
 
PHP 7 – What changed internally? (PHP Barcelona 2015)
PHP 7 – What changed internally? (PHP Barcelona 2015)PHP 7 – What changed internally? (PHP Barcelona 2015)
PHP 7 – What changed internally? (PHP Barcelona 2015)
 
Python 101 language features and functional programming
Python 101 language features and functional programmingPython 101 language features and functional programming
Python 101 language features and functional programming
 
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
 
Python fundamentals - basic | WeiYuan
Python fundamentals - basic | WeiYuanPython fundamentals - basic | WeiYuan
Python fundamentals - basic | WeiYuan
 
Intro to Python
Intro to PythonIntro to Python
Intro to Python
 
Ggplot2 v3
Ggplot2 v3Ggplot2 v3
Ggplot2 v3
 
Replication and Replica Sets
Replication and Replica SetsReplication and Replica Sets
Replication and Replica Sets
 
Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)
Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)
Advanced Dynamic Analysis for Leak Detection (Apple Internship 2008)
 
[1062BPY12001] Data analysis with R / week 2
[1062BPY12001] Data analysis with R / week 2[1062BPY12001] Data analysis with R / week 2
[1062BPY12001] Data analysis with R / week 2
 

More from rfyiamcool

Redis cluster那些事儿
Redis cluster那些事儿Redis cluster那些事儿
Redis cluster那些事儿rfyiamcool
 
Golang advance
Golang advanceGolang advance
Golang advancerfyiamcool
 
Golang 高性能实战
Golang 高性能实战Golang 高性能实战
Golang 高性能实战rfyiamcool
 
Mysql fast share
Mysql fast shareMysql fast share
Mysql fast sharerfyiamcool
 
分析mysql acid 设计实现
分析mysql acid 设计实现分析mysql acid 设计实现
分析mysql acid 设计实现rfyiamcool
 
大话redis设计实现
大话redis设计实现大话redis设计实现
大话redis设计实现rfyiamcool
 
async io frame
async io frameasync io frame
async io framerfyiamcool
 
异步io框架的实现
异步io框架的实现异步io框架的实现
异步io框架的实现rfyiamcool
 
美妙的多进程管理
美妙的多进程管理美妙的多进程管理
美妙的多进程管理rfyiamcool
 
聊聊我接触的集群管理
聊聊我接触的集群管理聊聊我接触的集群管理
聊聊我接触的集群管理rfyiamcool
 

More from rfyiamcool (12)

Redis cluster那些事儿
Redis cluster那些事儿Redis cluster那些事儿
Redis cluster那些事儿
 
Golang advance
Golang advanceGolang advance
Golang advance
 
Golang 高性能实战
Golang 高性能实战Golang 高性能实战
Golang 高性能实战
 
Mysql fast share
Mysql fast shareMysql fast share
Mysql fast share
 
分析mysql acid 设计实现
分析mysql acid 设计实现分析mysql acid 设计实现
分析mysql acid 设计实现
 
Raft
Raft Raft
Raft
 
大话redis设计实现
大话redis设计实现大话redis设计实现
大话redis设计实现
 
python gil
python gilpython gil
python gil
 
async io frame
async io frameasync io frame
async io frame
 
异步io框架的实现
异步io框架的实现异步io框架的实现
异步io框架的实现
 
美妙的多进程管理
美妙的多进程管理美妙的多进程管理
美妙的多进程管理
 
聊聊我接触的集群管理
聊聊我接触的集群管理聊聊我接触的集群管理
聊聊我接触的集群管理
 

Recently uploaded

Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldRoberto Pérez Alcolea
 
Key Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapKey Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapIshara Amarasekera
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxRTS corp
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shardsChristopher Curtin
 
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics
 
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...Bert Jan Schrijver
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolsosttopstonverter
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...OnePlan Solutions
 
Effort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsEffort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsDEEPRAJ PATHAK
 
Best Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh ITBest Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh ITmanoharjgpsolutions
 
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfEnhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfRTS corp
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdfSteve Caron
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogueitservices996
 
Mastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxMastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxAS Design & AST.
 
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfPros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfkalichargn70th171
 
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jGraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jNeo4j
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdfAndrey Devyatkin
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxSasikiranMarri
 
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...OnePlan Solutions
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsJean Silva
 

Recently uploaded (20)

Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository world
 
Key Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapKey Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery Roadmap
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptx
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards
 
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
 
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration tools
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
 
Effort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsEffort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software Projects
 
Best Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh ITBest Angular 17 Classroom & Online training - Naresh IT
Best Angular 17 Classroom & Online training - Naresh IT
 
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfEnhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogue
 
Mastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxMastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptx
 
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfPros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
 
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jGraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
 
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero results
 

python高级内存管理

  • 2. Object-specific allocators _____ ______ ______ ________ [ int ] [ dict ] [ list ] ... [ string ] Python core | +3 | <----- Object-specific memory -----> | <-- Non-object memory --> | _______________________________ | | [ Python's object allocator ] | | +2 | ####### Object memory ####### | <------ Internal buffers ------> | ______________________________________________________________ | [ Python's raw memory allocator (PyMem_ API) ] | +1 | <----- Python memory (under PyMem manager's control) ------> | | __________________________________________________________________ [ Underlying general-purpose allocator (ex: C library malloc) ] 0 | <------ Virtual memory allocated for the python process -------> | ========================================================================= _______________________________________________________________________ [ OS-specific Virtual Memory Manager (VMM) ] -1 | <--- Kernel dynamic storage allocation & management (page-based) ---> | __________________________________ __________________________________ [ ] [ ] -2 | <-- Physical memory: ROM/RAM --> | | <-- Secondary storage (swap) --> |
  • 3. * Request in bytes Size of allocated block Size class idx * ---------------------------------------------------------------- * 1-8 8 0 * 9-16 16 1 * 17-24 24 2 * 25-32 32 3 * 33-40 40 4 * 41-48 48 5 * 49-56 56 6 * 57-64 64 7 * ... ... ... * 497-504 504 62 * 505-512 512 63 * * */
  • 5. method posix malloc python memory pool object buffer pool
  • 6. Arena FeeePool Pool Pool Pool Pool Headers No BLock Arena malloc heap & pool Process stack heap bss init data text UserPool 1-8 … … 249 - 256 Pool Free Block Free Block Use Block
  • 7. userdpool design UserdPools 1-8 9-16 17-24 … … 249-256 Pool Header Free Block Free Block Header 分配 回收 … Pool Free Block Free Block Use Block 同⼀个Pool下Block⼀样长 单Pool为4kb Block及Pool都为单链表
  • 8. free pool desgin FeeePool Pool Pool Pool … Pool Headers No BLock Pool Headers No BLock Pool为4kb⼤小 Pool清理Headers
  • 9. where store variable ? run-time Stack list dict int heap [1 ,2, 3] {“n”: “1”} 1
  • 10. why ? In [1]: a = 123 In [2]: b = 123 In [3]: a is b Out[3]: True In [4]: a = 1000 In [5]: b = 1000 In [6]: a is b Out[6]: False In [7]: a = 'n' In [8]: b = 'n' In [9]: a is b Out[9]: True In [10]: a = "python" In [11]: b = "python" In [12]: a is b Out[12]: True
  • 11. why ? In [1]: def go(var): ...: print id(var) …: In [2]: id(a) Out[2]: 4401335072 In [3]: go(a) 4401335072 In [10]: a = b = 'nima' In [11]: b = a In [12]: a is b Out[12]: True In [13]: b = 'hehe' In [14]: a is b Out[14]: False 只有引用 ?
  • 12. python objects stored in memory? names names object Python Has Names, Not Variables ! ! !
  • 13. 整数对象池 -5 -4 … 0 … 256 257 小整数 ⼤整数 … … -5 -4 … … 257 … … var_1 var_2 … … -5 -4 … … 257 … … var_3 var_4 the same addr ! not the same addr ! 28 bytes 解释器初始化
  • 14. 整数对象池 Block List Free List PyIntBlock PyIntBlock PyIntBlock PyIntBlock 不会归还给 Arena和os ! ! !
  • 15. 字符对象池 a b c d … … … var_1 var_2 the same addr ! 单个字符38 bytes 由解释器初始化
  • 16. 字符串对象池 aa en cao oh woyao buyao kuai feile var_1 0 1 2 3 … var_2 hash存储变量 共用地址 记录引用计数 ref
  • 17. ref count x = 300 y = x z = [x, y] X ref += 1 300 y Z ref += 1 ref += 2 References -> 4 !
  • 18. What does del do? x = 300 y = x del x X ref -= 1 300 y References -> 1!The del statement doesn’t delete objects. • removes that name as a reference to that object • reduces the ref count by 1
  • 19. ref count case def go(): w = 300 go() a = “fuc . ” del a b = “en, a” b = None ref count +1 w is out of scope; ref count -1 del a; ref count -1 重新赋值; ref count -1
  • 20. cyclical refclass Node: def __init__(self, va): self.va = va def next(self, next): self.next = next mid = Node(‘root’) left = Node(‘left’) right = Node(‘right’) mid(left) left.next(right) right.next(left) Mid rightleft if del mid node: how ?
  • 21. mark & sweep gc root b a w c K G R
  • 22. 分代回收 node node node node node node node node node node node node node node node node node node node
  • 23. 可变 vs 不可变 (obj) string int tuple list dict
  • 24. container objects a = [10, 10, 11] b = a PyListObject Type list rc 1 items size … … PyObject Type integer rc 2 value 10 PyObject Type integer rc 1 value 11 10 10 11
  • 25. copy.copy a = [10, 10, [10, 11] ] b = copy.copy(a) PyListObject Type list rc 1 items size … … PyObject Type integer rc 2 value 10 PyObject Type integer rc 1 value 11 10 10 ref PyListObject Type list rc 1 items size … … 10 10 ref PyListObject 10 11
  • 26. copy.deepcopy a = [10, [ 10, 11 ] ] b = copy.deep(a) PyListObject Type list rc 1 items size … … PyObject Type integer rc 2 value 10 PyObject Type integer rc 1 value 11 10 ref PyListObject Type list rc 1 items size … … 10 ref PyListObject 10 11 PyListObject 10 11
  • 27. diy gc import gc import sys gc.set_debug(gc.DEBUG_STATS|gc.DEBUG_LEAK) a=[] b=[] a.append(b) print 'a refcount:',sys.getrefcount(a) # 2 print 'b refcount:',sys.getrefcount(b) # 3 del a del b print gc.collect() # 0
  • 28. Garbage Collector Optimize memory bound 可以降低threshold来时间换空间 cpu bound 提⾼threshold来空间换时间 暂停gc, 引⼊master worker设计
  • 29. 引用计数 跟 gil 的影响 ? gc 是否是原⼦ ? gc的 stop the world现象 ? … Q & A
  • 30. “ END ” – xiaorui.cc