5分で全然終わらなかったのでスライドを上げておきました

1. fluentdの設定が
よくわからない話
@harukasan
Sunday, June 2, 13

2. harukasan
・高専カンファレンス
・久留米高専/九工大/筑波大
・pixivのインフラ14ヶ月目
Sunday, June 2, 13

3. Sunday, June 2, 13

4. Sunday, June 2, 13

5. Ruby on Rails
Sunday, June 2, 13

6. Ruby on Rails
Capistrano
Sunday, June 2, 13

7. Ruby on Rails
Capistrano
Fluentd
Sunday, June 2, 13

8. fluentd
Sunday, June 2, 13

9. Front Access
App. Access
Activity - Sign-up, Sign-in, bookmark, ...
PHP/Rails error
JS(client-side) error
MySQL slow queries
Solr slow requests
IRC
Sunday, June 2, 13

10. capped
mongodb

APP/DB

in_tail
extended in_tail
out_mongo

out_file_alternative
JSON
Admin view


minutely/daily
batch
IRC
Google
Docs

MySQL
Sunday, June 2, 13

11. fluentd
Sunday, June 2, 13

12. fluentd
がよくわからなかった話
Sunday, June 2, 13

13. Input Plugin
Buffer Plugin
Output Plugin
Input data
(tag, data)
Output data
Fluentd Basic architecture
Sunday, June 2, 13

14. Configure
Sunday, June 2, 13

15. <source>
type tail
path /tmp/test.log
tag test.log
format json
</source>
<match test.log>
type mongo
host localhost
database test
collection log
buffer_chunk_limit 2m
buffer_queue_limit 256
flush_interval 1s
</match>
Sunday, June 2, 13

16. なんか動く
Sunday, June 2, 13

17. <source>
type tail
path /tmp/test.log
tag test.log
format json
</source>
<match test.log>
type mongo
host localhost
database test
collection log
buffer_chunk_limit 2m
buffer_queue_limit 256
flush_interval 1s
</match>
Sunday, June 2, 13

18. <source>
type tail
path /tmp/test.log
tag test.log
format json
</source>
<match test.log>
type mongo
host localhost
database test
collection log
buffer_chunk_limit 2m
buffer_queue_limit 256
flush_interval 1s
</match>
Sunday, June 2, 13

19. ようわからん
Sunday, June 2, 13

20. Sunday, June 2, 13

21. ようわからん
Sunday, June 2, 13

22. 質問
Sunday, June 2, 13

23. 難易度
英語 < Ruby
Sunday, June 2, 13

24. 難易度
Ruby < 英語
Sunday, June 2, 13

25. git clone
git@github.com:fluent/fluentd.git
Sunday, June 2, 13

26. Fluent::BufferedOutput
emit(tag,data)
emit(tag,data)
BufferedOutput
write(data)
BufferInput
someting(data)
push
pop
Sunday, June 2, 13

27. Fluent::BufferedOutput
emit(tag,data)
emit(tag,data)
BufferedOutput
write(data)
BufferInput
someting(data)
push
pop
Sunday, June 2, 13

28. Fluent::BasicBuffer#emit
def
emit(key,
data,
chain)
key
=
key.to_s
top
=
(@map[key]
||=
new_chunk(key))
if
top.size
+
data.bytesize
<=
@buffer_chunk_limit
chain.next
top
<<
data
return
false
elsif
@queue.size
>=
@buffer_queue_limit
raise
BufferQueueLimitError,
"queue
size
exceeds
limit"
end
nc
=
new_chunk(key)
Sunday, June 2, 13

29. Fluent::BasicBuffer#emit
def
emit(key,
data,
chain)
...
top
=
(@map[key]
||=
new_chunk(key))
if
top.size
+
data.bytesize
<=
@buffer_chunk_limit
...
top
<<
data
return
false
top=@map[key]
Sunday, June 2, 13

30. Fluent::BasicBuffer#emit
def
emit(key,
data,
chain)
...
top
=
(@map[key]
||=
new_chunk(key))
if
top.size
+
data.bytesize
<=
@buffer_chunk_limit
...
top
<<
data
return
false
top=@map[key]
emit(key,data)
Sunday, June 2, 13

31. Fluent::BasicBuffer#emit
def
emit(key,
data,
chain)
...
top
=
(@map[key]
||=
new_chunk(key))
if
top.size
+
data.bytesize
<=
@buffer_chunk_limit
...
top
<<
data
return
false
top=@map[key]
emit(key,data)
←chunk
Sunday, June 2, 13

32. Fluent::BasicBuffer#emit
def
emit(key,
data,
chain)
...
top
=
(@map[key]
||=
new_chunk(key))
if
top.size
+
data.bytesize
<=
@buffer_chunk_limit
...
top
<<
data
return
false
top=@map[key]
emit(key,data)
data
←chunk
Sunday, June 2, 13

33. Fluent::BasicBuffer#emit
def
emit(key,
data,
chain)
...
top
=
(@map[key]
||=
new_chunk(key))
if
top.size
+
data.bytesize
<=
@buffer_chunk_limit
...
top
<<
data
return
false
top=@map[key]
emit(key,data)
data
emit(key,data)
←chunk
Sunday, June 2, 13

34. Fluent::BasicBuffer#emit
def
emit(key,
data,
chain)
...
top
=
(@map[key]
||=
new_chunk(key))
if
top.size
+
data.bytesize
<=
@buffer_chunk_limit
...
top
<<
data
return
false
top=@map[key]
emit(key,data)
data
data
emit(key,data)
←chunk
Sunday, June 2, 13

35. Fluent::BasicBuffer#emit
def
emit(key,
data,
chain)
...
top
=
(@map[key]
||=
new_chunk(key))
if
top.size
+
data.bytesize
<=
@buffer_chunk_limit
...
top
<<
data
return
false
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data)
←chunk
Sunday, June 2, 13

36. Fluent::BasicBuffer#emit
end
nc
=
new_chunk(key)
ok
=
false
begin
nc
<<
data
chain.next
flush_trigger
=
false
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data) nc
Sunday, June 2, 13

37. Fluent::BasicBuffer#emit
end
nc
=
new_chunk(key)
ok
=
false
begin
nc
<<
data
chain.next
flush_trigger
=
false
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data) nc
data
Sunday, June 2, 13

38. Fluent::BasicBuffer#emit
flush_trigger
=
false
@queue.synchronize
{
enqueue(top)
flush_trigger
=
@queue.empty?
@queue
<<
top
@map[key]
=
nc
}
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data) nc
data
Sunday, June 2, 13

39. Fluent::BasicBuffer#emit
flush_trigger
=
false
@queue.synchronize
{
enqueue(top)
flush_trigger
=
@queue.empty?
@queue
<<
top
@map[key]
=
nc
}
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data) nc
data
@queue
Sunday, June 2, 13

40. Fluent::BasicBuffer#emit
flush_trigger
=
false
@queue.synchronize
{
enqueue(top)
flush_trigger
=
@queue.empty?
@queue
<<
top
@map[key]
=
nc
}
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data) nc
data
@queue
Sunday, June 2, 13

41. Fluent::BasicBuffer#emit
flush_trigger
=
false
@queue.synchronize
{
enqueue(top)
flush_trigger
=
@queue.empty?
@queue
<<
top
@map[key]
=
nc
}
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data)
data
@queue
Sunday, June 2, 13

42. Fluent::BasicBuffer#emit
flush_trigger
=
false
@queue.synchronize
{
enqueue(top)
flush_trigger
=
@queue.empty?
@queue
<<
top
@map[key]
=
nc
}
top=@map[key]
emit(key,data)
data
data
emit(key,data)
emit(key,data)
data
@queue
Sunday, June 2, 13

43. Fluent::BufferedOutput
emit(tag,data)
emit(tag,data)
BufferedOutput
write(data)
BufferInput
someting(data)
push
pop
Sunday, June 2, 13

44. Fluent::BasicBuffer#push
def
push(key)
top
=
@map[key]
if
!top
||
top.empty?
return
false
end
@queue.synchronize
do
enqueue(top)
@queue
<<
top
@map.delete(key)
end
return
true
end
Sunday, June 2, 13

45. Fluent::BasicBuffer#push
@queue.synchronize
do
enqueue(top)
@queue
<<
top
@map.delete(key)
end
return
true
end
top=@map[key]
data
data
data
@queue
Sunday, June 2, 13

46. Fluent::BasicBuffer#push
@queue.synchronize
do
enqueue(top)
@queue
<<
top
@map.delete(key)
end
return
true
end
data
data
data
@queue
Sunday, June 2, 13

47. Fluent::BufferedOutput
emit(tag,data)
emit(tag,data)
BufferedOutput
write(data)
BufferInput
someting(data)
push
pop
Sunday, June 2, 13

48. Fluent::BasicBuffer#pop
def
pop(out)
chunk
=
@queue.first
if
!chunk.empty?
write_chunk(chunk,
out)
end
@queue.delete_if
{|c|
c.object_id
==
chunk.object_id
}
chunk.purge
return
!@queue.empty?
end
Sunday, June 2, 13

49. Fluent::BasicBuffer#pop
data
data
data
@queue
def
pop(out)
chunk
=
@queue.first
if
!chunk.empty?
write_chunk(chunk,
out)
end
@queue.delete_if
{|c|
Sunday, June 2, 13

50. Fluent::BasicBuffer#pop
data
data
data
@queue
def
pop(out)
chunk
=
@queue.first
if
!chunk.empty?
write_chunk(chunk,
out)
end
@queue.delete_if
{|c|
Sunday, June 2, 13

51. Fluent::BufferedOutput
emit(tag,data)
emit(tag,data)
BufferedOutput
write(chunk)
BufferInput
someting(chunk)
push
pop
Sunday, June 2, 13

52. Configure parameters
Sunday, June 2, 13

53. <source>
type tail
path /tmp/test.log
tag test.log
format json
</source>
<match test.log>
type mongo
host localhost
database test
collection log
buffer_chunk_limit 2m
buffer_queue_limit 256
flush_interval 1s
</match>
Sunday, June 2, 13

54. buffer_chunk_limit
buffer chunkのサイズ制限
Sunday, June 2, 13

55. buffer_queue_limit
queueにはいるchunkの数
Sunday, June 2, 13

56. flush_interval
queueがpopされる間隔
Sunday, June 2, 13

57. まとめ
Sunday, June 2, 13

58. documentを書くのは面倒なので
documentを読みたい人はdocumentを書こう
documentがなくてもcodeが小さければ読める
よくわからなかったらcodeを読もう
Sunday, June 2, 13