Code4Lib JAPAN カンファレンス2014 発表

1. Code4Lib
JAPAN䜹䞁䝣䜯䝺䞁䝇2014
䠜⚟஭┴㪆Ụᕷ
2014ᖺ9᭶7᪥
䝻䜾䝕䞊䝍䛾ά⏝䛜㐍䜎䛺䛔⌧≧
䜢ᡴ◚䛩䜛䛯䜑䛻:
Wikipedia䝨䞊䝆㜀ぴ⤫ィ䝕䞊䝍䜢
㢟ᮦ䛸䛧䛯ヨ䜏䈊
Ύ⏣䚷㝧ྖ
䠄ᰴᘧ఍♫䝛䜽䝇䝖䠅

2. 䝴䞊䝄䞊䝻䜾ά⏝䛾ព⩏
๓ᥦ:
䝠䝖(䝃䞊䝡䝇ᥦ౪⪅䚸䝴䞊䝄䞊)䛾ពᛮỴ
ᐃ䛻ά䛛䛩
• 䝃䞊䝡䝇ᨵၿ䛾䝠䞁䝖䜢ᚓ䜛
– 䝴䞊䝄䞊䛾⾜ື䜢⌮ゎ䛩䜛䠄᫬㛫ᖏู䚸᭙᪥ู䚸
᭶ู䛾ഴྥ䛺䛹䠅
– 䝴䞊䝄䞊䜲䞁䝍䞊䝣䜵䞊䝇ୖ䛾ㄢ㢟䜢Ὑ䛔ฟ䛩
• ᪂䛯䛺౯್䜢⏕䜏ฟ䛩
– ಶู䝴䞊䝄䞊䛾₯ᅾⓗ䝙䞊䝈᥀䜚㉳䛣䛧
• ᥎⸀䠄ᅗ᭩䚸᝟ሗ䚸….䠅
• 䝃䝆䜵䝇䝖

3. 䝴䞊䝄䞊䝻䜾ά⏝䛻ᚲせ䛺䛣䛸
• ᝟ሗ䜲䞁䝣䝷
• 䝒䞊䝹฼⏝䛻㛵䛩䜛▱㆑
– Webୖ䛻㇏ᐩ䛻䛒䜛
• ᐇ㊶䞉⩦⇍䛩䜛ᶵ఍
– 䛩䛷䛻ά⏝䛥䜜䛶䛔䜛⤌⧊ෆ䛻䛿㇏ᐩ䛻䛒䜛
– ⤌⧊እ䛷䛿ᑡ䛺䛔

4. 䛹䛖䜔䛳䛶ᐇ㊶䞉⩦⇍䛩䜛䛛䠛
• 䜲䞁䝣䝷䚸䝒䞊䝹
– 䝟䝤䝸䝑䜽䜽䝷䜴䝗䛺䛹䛷ᐜ᫆䛻฼⏝ྍ⬟
– Web⏬㠃䛛䜙䜒᧯సྍ⬟
• 䝁䝭䝳䝙䝔䜱䚸ሙᡤ
– 䝝䞁䝈䜸䞁䛺䛹
• 䛚䜹䝛
• ඹ᭷䛷䛝䜛䝕䞊䝍

5. ᐇ㊶䞉⩦⇍䛻ồ䜑䜙䜜䜛䝕䞊䝍䛾ᛶ㉁
• 䝕䞊䝍㔞
– ⌧ሙ䛷ᛂ⏝䛩䜛䛯䜑
• 䝸䜰䝸䝔䜱䠄䝠䝖䜔♫఍䛾ື䛝䜢཯ᫎ䠅
– Ꮫ⩦ຠᯝ䜢㧗䜑䜛䛯䜑䛻䛿䚸䛂㠃ⓑ䛔䛃䛸ᛮ䛘䜛
⤖ᯝ䛜ᚓ䜙䜜䜛䛣䛸䛜኱஦
• 㩭ᗘ
– ពᛮỴᐃ䛻ά䛛䛩䛯䜑

6. 䝟䝤䝸䝑䜽䜽䝷䜴䝗
• ᚲせ䛻ᛂ䛨䛶฼⏝䛷䛝䜛䝁䞁䝢䝳䞊䝍䞊㈨※
䛾䝃䞊䝡䝇䠄ᚑ㔞ㄢ㔠ไ䠅
– 䝃䞊䝞䞊
– 䝇䝖䝺䞊䝆䠄グ᠈㡿ᇦ䠅
– 䝛䝑䝖䝽䞊䜽
• ୺せ䛺䝟䝤䝸䝑䜽䜽䝷䜴䝗䝃䞊䝡䝇
– Amazon
Web
Services
(AWS)
– MicrosoE
Windows
Azure
– Google
App
Engine
– 䛥䛟䜙䛾䜽䝷䜴䝗

7. AWS
Public
Data
Sets
hOp://aws.amazon.com/jp/public-­‐data-­‐sets/

8. AOL
Search
Query
Logs
(2006ᖺ)
hOp://www.gregsadetsky.com/aol-­‐data/

9. hOp://techcrunch.com/2006/08/06/aol-­‐proudly-­‐releases-­‐massive-­‐amounts-­‐of-­‐user-­‐search-­‐data/

10. hOps://search-­‐id.com/

11. 䝕䞊䝍ゎᯒ䛻䜘䜛ಶே≉ᐃ䛾䝸䝇䜽
• 䝬䝃䝏䝳䞊䝉䝑䝒ᕞ▱஦䛾་⒪グ㘓䛜බ㛤᝟ሗ
䛛䜙≉ᐃྍ⬟
– ཰㞟䛧䛯་⒪䝕䞊䝍䜢ຍᕤ䛧䛶බ㛤䛧䛶䛔䜛㻌ᕥ෇ෆ
– ୍᪉䚸㑅ᣲ䛾ᢞ⚊⪅ྡ⡙䛿බ㛤㻌ྑ෇ෆ
• ୧⪅䜢䛴䛝䛒䜟䛫䜛䛸
• 6
ே䛜▱஦䛸ྠ䛨⏕ᖺ᭶᪥
䚷䚷䛖䛱3
ே䛜⏨
䚷䚷䛖䛱1
ே䛜ྠ䛨㒑౽␒ྕ
• 1990ᖺ䛾㻌the US 1990 census data䛻䜘䜜䜀
– 87% 䛾ே䛜 (zipcode, ᛶู, ⏕ᖺ᭶᪥)䛻䜘䛳䛶୍ព≉ᐃྍ⬟

12. NII䝕䞊䝍䝉䝑䝖
hOp://www.nii.ac.jp/cscenter/idr/datalist.html

13. 䝴䞊䝄䞊䝻䜾ඹ᭷䛾⌧≧
• ୍⯡බ㛤䛥䜜䛶䛔䜛⏕䝻䜾䝕䞊䝍䛿䜋䜌Ꮡᅾ
䛧䛺䛔
– 䝥䝷䜲䝞䝅䞊౵ᐖ䛾䝸䝇䜽
– IP䜰䝗䝺䝇䛺䛹䜢๐㝖䛧䛯䛸䛧䛶䜒ಶே≉ᐃ䛾ྍ
⬟ᛶ䛜ṧ䜛
• ◊✲⏝㏵䛻㝈ᐃ䛧䛶ᥦ౪䛥䜜䛶䛔䜛䝻䜾䝕䞊
䝍䛿Ꮡᅾ䛩䜛
– ⛎ᐦಖᣢዎ⣙ᡭ⥆䛝䛜ᚲせ

14. Page
view
stascs
for
Wikimedia
projects
• Wikimedia඲ゝㄒ∧䛾䜰䜽䝉䝇ᩘ䝕䞊䝍
– 1᫬㛫䛤䛸䛾඲䝨䞊䝆䛾䜰䜽䝉䝇ᩘ
• ⏕䝻䜾䝕䞊䝍䛷䛿䛺䛔䛜䚸㞟ィ䛻ᚲせ䛺䝜䜴䝝䜴䛿⏕䝻䜾
䝕䞊䝍䛸ඹ㏻
• ᐇ㊶䞉⩦⇍䛻ᚲせ䛺ᛶ㉁䜢䠄䛒䜛⛬ᗘ䠅‶䛯䛩
– 䝕䞊䝍㔞:
1䞄᭶䛒䛯䜚⣙70Gbytes
(Gzipᅽ⦰)
– 䝸䜰䝸䝔䜱:
ୡ䛾୰䛾ື䛝䜢཯ᫎ䛧䛯䝕䞊䝍
– 㩭ᗘ:
30ศᚋ䛻䛿බ㛤
hOp://dumps.wikimedia.org/other/pagecounts-­‐raw/

15. ᢤ⢋
(Wikipedia᪥ᮏㄒ∧,
2014/09/01
09:00ࠥ10:00
JST)
ja
䝕䞁䜾⇕㻌㻌㻌㻌㻌㻌㻌㻌10745
554450816
ja
䝯䜲䞁䝨䞊䝆㻌㻌㻌㻌14093
438162758
ja
㛵ᮾ኱㟈⅏㻌㻌㻌㻌㻌㻌3114
281205408
ja
9᭶1᪥㻌㻌1833
165384461
ja
24᫬㛫䝔䝺䝡_䛂ឡ䛿ᆅ⌫䜢ᩆ䛖䛃㻌1723
137958282
ja
䜲䝏䝻䞊㻌㻌㻌㻌㻌㻌㻌㻌192
120358324
ja
ᮾ᪥ᮏ኱㟈⅏㻌㻌㻌㻌437
110995792
ja
ྜྷỌᑠⓒྜ㻌㻌㻌㻌㻌㻌2284
86710534
ja
䝪䝹䝅䜰䞉䝗䝹䝖䝮䞁䝖㻌㻌1388
75815693
ja
㯮⏣Ꮥ㧗㻌㻌㻌㻌㻌㻌㻌㻌1168
71345773
ja
ⰼᏊ䛸䜰䞁㻌㻌㻌㻌㻌㻌877
67992101
ja
ᇛᓥⱱ㻌㻌2197
55590788
ja
᪂ୡ⣖䜶䞂䜯䞁䝀䝸䜸䞁㻌㻌415
55537912
ja
㜵⅏䛾᪥㻌㻌㻌㻌㻌㻌㻌㻌4493
55510476
ja
ᑠῲඃᏊ㻌㻌㻌㻌㻌㻌㻌㻌2751
52931239
ja
ゟሗ_2014ᖺ㻌㻌㻌㻌㻌111
52488871
ja
㜰⚄䞉ῐ㊰኱㟈⅏㻌㻌㻌㻌㻌㻌㻌㻌285
52182898

17. ௒ᚋ䛾ᒎ㛤
• AWSୖ䛷䝕䞊䝍䝉䝑䝖䜢౑䛘䜛䜘䛖䛻䛩䜛
– Elasc
MapReduce
(Hadoop)
䛺䛹䛷㞟ィྍ⬟䛻
• 䝝䞁䝈䜸䞁⏝䛾䝁䞁䝔䞁䝒సᡂ
• Code4Lib䝁䝭䝳䝙䝔䜱䛷䛾䝝䞁䝈䜸䞁ᐇ᪋