BPStudy #42 MongoDB and MyBike.JP and Me

1. 2011/2/22

2. MongoDB MyBike MongoDB

3. MongoDB 1

6. Mitsukuni Sato a.k.a. @key3 • BeProud • • … • MyBike.JP • MongoDB : Python / Django / PHP /

11. BSON

12. BSON MongoDB JSON …

13. BSON MongoDB JSON … MongoDB shell version: 1.6.6-pre- connecting to: test > foo = { name : 'Mitsukuni Sato' } { "name" : "Mitsukuni Sato" } > db.people.save(foo) > db.people.find() { "_id" : ObjectId("4d63d9c48cbc8019697a25de"), "name" : "Mitsukuni Sato" }

17. db.collection_name.find( )

18. db.collection_name.find( ) > db.people.find({ name : "Mitsukuni Sato" }) { "_id" : ObjectId("4d63d9c48cbc8019697a25de"), "name" : "Mitsukuni Sato" }

21. db.collection_name.save(BSON )

22. db.collection_name.save(BSON ) > user = db.people.findOne({ name : "Mitsukuni Sato" }) { "_id" : ObjectId("4d63d9c48cbc8019697a25de"), "name" : "Mitsukuni Sato" } > user.age = 33 33 > db.people.save(user) > user = db.people.findOne({ name : "Mitsukuni Sato" }) { "_id" : ObjectId("4d63d9c48cbc8019697a25de"), "name" : "Mitsukuni Sato", "age" : 33 }

24. GridFS

30. 1:n master / slave

42. vs

43. vs HDD

44. vs HDD

45. vs HDD

46. vs HDD

47. GridFS 4MB

53. …

55. source code

56. source code pre-compiled binary

57. source code pre-compiled binary .deb (Debian / Ubuntu) RPM (CentOS / Fedora) MacOSX FreeBSD

59. MySQL

60. MySQL 1/10

61. MySQL 1/10

62. MySQL 1/10

64. mongodump & mongorestore mongostat mongo

68. compaction 1GB

73. ruby, python, C++, Java, Perl, C#

74. ruby, python, C++, Java, Perl, C# http://www.mongodb.org/display/DOCS/Drivers

75. ruby, python, C++, Java, Perl, C# http://www.mongodb.org/display/DOCS/Drivers

77. Python + Django

78. Python + Django django-mongokit

79. Python + Django django-mongokit django-mongodb-engine

80. Python + Django django-mongokit django-mongodb-engine django-storages (GridFS)

82. MySQL

83. MySQL

84. MySQL

85. MySQL

86. MySQL django-mongokit Model Document

87. MySQL + MongoDB

88. MySQL + MongoDB

89. MySQL + MongoDB

90. MySQL + MongoDB metadata object

91. MySQL + MongoDB metadata object django-mongodb-engine

93. Document GridFS django-mongokit django-mongodb-engine

94. Document GridFS django-mongokit django-mongodb-engine RDB

95. Document GridFS django-mongokit django-mongodb-engine RDB django-mongokit

96. Document GridFS django-mongokit django-mongodb-engine RDB django-mongokit

97. Document GridFS django-mongokit django-mongodb-engine RDB django-mongokit django-mongodb-engine

99. ManyToMany

100. ManyToMany tag=A & tag=B

107. 1.7.5 Write Ahead Log

112. …

120. 240,000 220,000 200,000 180,000 mongod 160,000 140,000 120,000 primary Y 100,000 80,000 60,000 40,000 20,000 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 X 240,000 220,000 200,000 180,000 160,000 mongod 140,000 bread/s 120,000 slave (1) 100,000 80,000 60,000 40,000 20,000 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 X 240,000 220,000 200,000 180,000 160,000 mongod 140,000 bread/s 120,000 100,000 slave (2) 80,000 60,000 40,000 20,000 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 X

121. 100 90 80 70 60 mongod (%) 50 primary 40 30 20 10 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800

122. 100 90 80 70 60 mongod (%) 50 primary 40 30 20 10 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800

123. 100 90 80 70 60 mongod (%) 50 primary 40 30 20 10 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 nice

124. 100 90 80 70 60 mongod (%) 50 primary 40 30 20 10 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 nice …

125. Mon Feb 21 06:55:00 [initandlisten] connection accepted from 127.0.0.1:59385 #25 Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig config object parses ok, 3 members specified Mon Feb 21 06:55:00 [conn25] replSet TODO : don't allow removal of a node until we handle it at the removed node end? Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig [2] Mon Feb 21 06:55:00 [conn25] replSet info saving a newer config version to local.system.replset Mon Feb 21 06:55:01 [conn25] replSet relinquishing primary state Mon Feb 21 06:55:01 [conn25] replSet RECOVERING Mon Feb 21 06:55:01 [conn25] replSet PRIMARY Mon Feb 21 06:55:01 [conn25] replSet replSetReconfig new config saved locally Mon Feb 21 06:55:01 [conn25] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "foo", version: 2, members: [ { _id: 0, host: "ip-10-118-13-29" }, { _id: 1, host: "ip-10-202-30-58" }, { _id: 2.0, host: "ip-10-195-11-64" } ] } } reslen:53 1036ms Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-195-11-64 is now down (or slow to respond) Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-118-13-29 is now up Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet ip-10-118-13-29 SECONDARY (snip) Mon Feb 21 07:01:30 [ReplSetHealthPollTask] replSet ip-10-195-11-64 SECONDARY

126. Mon Feb 21 06:55:00 [initandlisten] connection accepted from 127.0.0.1:59385 #25 Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig config object parses ok, 3 members specified Mon Feb 21 06:55:00 [conn25] replSet TODO : don't allow removal of a node until we handle it at the removed node end? Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig [2] Mon Feb 21 06:55:00 [conn25] replSet info saving a newer config version to local.system.replset Mon Feb 21 06:55:01 [conn25] replSet relinquishing primary state Mon Feb 21 06:55:01 [conn25] replSet RECOVERING Mon Feb 21 06:55:01 [conn25] replSet PRIMARY Mon Feb 21 06:55:01 [conn25] replSet replSetReconfig new config saved locally Mon Feb 21 06:55:01 [conn25] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "foo", version: 2, members: [ { _id: 0, host: "ip-10-118-13-29" }, { _id: 1, host: "ip-10-202-30-58" }, { _id: 2.0, host: "ip-10-195-11-64" } ] } } reslen:53 1036ms Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-195-11-64 is now down (or slow to respond) Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-118-13-29 is now up Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet ip-10-118-13-29 SECONDARY (snip) Mon Feb 21 07:01:30 [ReplSetHealthPollTask] replSet ip-10-195-11-64 SECONDARY 3GB

127. Mon Feb 21 06:55:00 [initandlisten] connection accepted from 127.0.0.1:59385 #25 Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig config object parses ok, 3 members specified Mon Feb 21 06:55:00 [conn25] replSet TODO : don't allow removal of a node until we handle it at the removed node end? Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig [2] Mon Feb 21 06:55:00 [conn25] replSet info saving a newer config version to local.system.replset Mon Feb 21 06:55:01 [conn25] replSet relinquishing primary state Mon Feb 21 06:55:01 [conn25] replSet RECOVERING Mon Feb 21 06:55:01 [conn25] replSet PRIMARY Mon Feb 21 06:55:01 [conn25] replSet replSetReconfig new config saved locally Mon Feb 21 06:55:01 [conn25] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "foo", version: 2, members: [ { _id: 0, host: "ip-10-118-13-29" }, { _id: 1, host: "ip-10-202-30-58" }, { _id: 2.0, host: "ip-10-195-11-64" } ] } } reslen:53 1036ms Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-195-11-64 is now down (or slow to respond) Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-118-13-29 is now up Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet ip-10-118-13-29 SECONDARY (snip) Mon Feb 21 07:01:30 [ReplSetHealthPollTask] replSet ip-10-195-11-64 SECONDARY 3GB 6 30

128. Mon Feb 21 06:55:00 [initandlisten] connection accepted from 127.0.0.1:59385 #25 Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig config object parses ok, 3 members specified Mon Feb 21 06:55:00 [conn25] replSet TODO : don't allow removal of a node until we handle it at the removed node end? Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig [2] Mon Feb 21 06:55:00 [conn25] replSet info saving a newer config version to local.system.replset Mon Feb 21 06:55:01 [conn25] replSet relinquishing primary state Mon Feb 21 06:55:01 [conn25] replSet RECOVERING Mon Feb 21 06:55:01 [conn25] replSet PRIMARY Mon Feb 21 06:55:01 [conn25] replSet replSetReconfig new config saved locally Mon Feb 21 06:55:01 [conn25] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "foo", version: 2, members: [ { _id: 0, host: "ip-10-118-13-29" }, { _id: 1, host: "ip-10-202-30-58" }, { _id: 2.0, host: "ip-10-195-11-64" } ] } } reslen:53 1036ms Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-195-11-64 is now down (or slow to respond) Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-118-13-29 is now up Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet ip-10-118-13-29 SECONDARY (snip) Mon Feb 21 07:01:30 [ReplSetHealthPollTask] replSet ip-10-195-11-64 SECONDARY 3GB 6 30

130. disk allocation

131. disk allocation pre allocation

132. disk allocation pre allocation dd zero padding OK

134. …

135. …

136. …

137. … 20

138. … 20

139. Amazon EC2 large MongoDB 1.6.5 16 connections from worker write 12000 qps select 100000qps

140. MyBike.JP

142. MyBike.JP

143. MyBike.JP http://mybike.jp

149. GridFS

150. GridFS

151. 7300 objects 1GB

155. Python

156. Python pymongo

157. Python pymongo

158. Python pymongo

159. Python pymongo …

164. nginx

165. nginx nginx-gridfs

166. nginx nginx-gridfs

169. GridFS

170. GridFS django-storages

175. malloc

176. malloc 32bit

177. malloc 32bit Wed Feb 9 16:52:09 [initandlisten] mmap() failed for /var/lib/mongodb/ $tmp_repairDatabase_0/mybike.4 len:536870912 errno:12 Cannot allocate memory Wed Feb 9 16:52:09 [initandlisten] mmap failed with out of memory, if you're using 32-bits, then you probably need to upgrade to 64 Wed Feb 9 16:52:09 [initandlisten] warning: exception cloning object in mybike.fs.chunks can't map file memory - mongo requires 64 bit build for larger datasets obj:{ _id: ObjectId('4d44a08c4ae613113700033f'), files_id: ObjectId('4d44a08c4ae613113700033e'), data: BinData, n: 0 } Wed Feb 9 16:52:09 [initandlisten] mmap failed with out of memory, if you're using 32-bits, then you probably need to upgrade to 64

179. 32bit

180. 32bit 32bit

181. 32bit 32bit 2GB

185. mongodb stop && mongodb start

186. mongodb stop && mongodb start

187. init script

188. init script 10gen Debian

189. init script 10gen Debian …

190. init script 10gen Debian … mongodb stop && mongodb start

191. init script 10gen Debian … mongodb stop && mongodb start stop

192. init script 10gen Debian … mongodb stop && mongodb start stop

194. Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog... Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets... Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator... Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files... Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525 Mon Feb 21 06:18:05 closeAllFiles() finished Mon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock... Mon Feb 21 06:18:05 dbexit: really exiting now

198. Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog... Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets... Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator... Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files... Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525 Mon Feb 21 06:18:05 closeAllFiles() finished Mon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock... Mon Feb 21 06:18:05 dbexit: really exiting now mongodb restart

199. Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog... Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets... Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator... Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files... Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525 Mon Feb 21 06:18:05 closeAllFiles() finished Mon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock... Mon Feb 21 06:18:05 dbexit: really exiting now mongodb restart

202. slave

203. slave primary

204. slave primary slave primary

205. slave primary slave primary primary / slave

206. QA

BPStudy #42 MongoDB and MyBike.JP and Me

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (13)

Similar to BPStudy #42 MongoDB and MyBike.JP and Me

Similar to BPStudy #42 MongoDB and MyBike.JP and Me (20)

Recently uploaded

Recently uploaded (20)

BPStudy #42 MongoDB and MyBike.JP and Me

Editor's Notes