This document describes a presentation about introducing black magic programming patterns in Ruby and their pragmatic uses. It provides an overview of Fluentd, including what it is, its versions, and the changes between versions 0.12 and 0.14. Specifically, it discusses how the plugin API was updated in version 0.14 to address problems with the version 0.12 API. It also explains how a compatibility layer was implemented to allow most existing 0.12 plugins to work without modification in 0.14.
6. Fluentd
• What is Fluentd?
• Open Source Log Collector
• Pluggable, Reliable, Less resource usage, Ease to use
• Versions of Fluentd
• v0.12: stable versions (2014/12 - Now)
• v0.14: versions for next stable (2016/05 - Now)
8. What's about this session?
• Introduce some patterns of "Black Magic"s (a.k.a. meta
programming) in Ruby
• Show you some PRAGMATIC use of Black Magics
10. Fluentd v0.14 API Update
• Everything changed :)
• Plugin namespace
• before: Fluent::* (Top level classes even for plugins!)
• after: Fluent::Plugin::*
• Plugin base class for common methods
• Inconsistent Output plugin hierarchy
• Plugin must call `super` in common methods
http://www.slideshare.net/tagomoris/fluentd-v014-plugin-api-details
11. Classes hierarchy (v0.12)
Fluent::Input F::Filter
F::Output
BufferedOutput
Object
Buffered
Time
Sliced
Multi
Output F::Buffer
F::Parser
F::Formatter
3rd party plugins
12. Classes hierarchy (v0.14)
F::P::Input F::P::Filter F::P::Output
Fluent::Plugin::Base
F::P::Buffer
F::P::Parser
F::P::Formatter
F::P::Storage
both of
buffered/non-buffered
F::P::
BareOutput
(not for 3rd party
plugins)
F::P::
MultiOutput
copy
roundrobin
13. diff v0.12 v0.14
F::P::Output
Fluent::Plugin::Base
both of
buffered/non-buffered
F::P::
BareOutput
(not for 3rd party
plugins)
F::P::
MultiOutput
copy
roundrobin
F::Output
BufferedOutput
Object
Buffered
Time
Sliced
Multi
Output
Super classes by
how to buffer data
All output plugins
are just "Output"
21. diff v0.12 v0.14
F::P::Output
Fluent::Plugin::Base
both of
buffered/non-buffered
F::P::
BareOutput
(not for 3rd party
plugins)
F::P::
MultiOutput
copy
roundrobin
F::Output
BufferedOutput
Object
Buffered
Time
Sliced
Multi
Output
Super classes by
how to buffer data
All output plugins
are just "Output"
28. Fluentd v0.12 Fluent::TimeSlicedOutput
class Fluent::Outputclass BufferedOutput
@buffer
MyOutput
class TimeSlicedOutput
OutputThread
#write(chunk)
@buffer calls #write in OutputThread
#write calls chunk.key
chunk
#pop
29. Fluentd v0.12 Fluent::ObjectBufferedOutput
class Fluent::Outputclass BufferedOutput
@buffer
MyOutput
class ObjectBufferedOutput
OutputThread
#write(chunk)
#write(chunk)
#write_object(chunk_key, chunk)
@buffer calls #write in OutputThread
chunk
#pop
30. Fluentd v0.12 API Problems
• Entry point method is implemented by Plugin subclasses
• Fluentd core cannot add any processes
• counting input events
• hook arguments/return values to update API
• Fluentd core didn't show fixed API
• Plugins have different call stacks
• It's not clear what should be implemented for authors
• It's not clear what interfaces are supported for
arguments/return values
35. Fluentd v0.14 Design Policy
• Separate entry points from implementations
• Methods in superclass control everything
• Do NOT override these methods!
• Methods in subclass do things only for themselves
• not for data flow, control flow nor others
• Plugins have simple/straightforward call stack
• Easy to understand/maintain
40. Double Decker Compat Layer?
• Existing plugins inherits Fluent::Output or others
• No more codes in Fluent top level :-(
• Separate code into Fluent::Compat
• and import it into Fluent top level
53. Fluent::Plugin::Outputclass MyOutput
@buffer
#write
#emit_events(tag, es)
When plugin overrides #emit
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk)
flush thread
#emit
1. #emit calls @buffer.emit with data to be written in buffer
0. plugin calls @buffer.extend to add #emit
2. @buffer.emit stores arguments into plugin's attribute
3. get stored data
4. call @buffer.write with data
61. For Example: shutdown compat plugins
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
It doesn't call "super"! We want to call this...
62. What We Want To Do:
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
1. call #shutdown anyway
0. Fluentd core calls #shutdown
2. call #shutdown? to check "super" is called or not
3. call #shutdown of superclass forcedly!
63. What We Want To Do:
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
How to make this point?
66. class A
#bar
class B
#bar
super
B.new.bar
module M
Wrapping Methods on a Class (2)
B.new.singleton_class
#bar
#bar
Using extend is powerful,
but it should be done for all instances
How about wrapping methods for
all instances of the class?
67. class A
#bar
class B
#bar
super
module M;def bar;super;end;end
B.prepend M
B.new.bar
module M
Wrapping Methods on a Class (3): Module#prepend
B.new.singleton_class
#bar
#bar
module M wraps B, and
M#bar is called at first
68. What We Want To Do:
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
THIS ONE !!!
69.
70. What We Got :-)
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
1. call #shutdown anyway
0. prepend CallSuperMixin at first
2. call #shutdown? to check "super" is called or not
3. if not, get method of superclass, bind self with it, then call it
Thank you @unak -san!
72. Testing: Capturing return values by Test Driver
OutputMyOutput
@buffer
#write
#emit_events
#format
#emit_buffered
Output Plugin Test Driver
Create plugin instances
Feed test data into plugin
73. Testing: Capturing return values by Test Driver
OutputMyOutput
@buffer
#write
#emit_events
#format
#emit_buffered
We want to assert this return value!
Output Plugin Test Driver
Feed test data into plugin
74. Using #prepend to capture return values
OutputMyOutput
@buffer
#write
#emit_events
#format
#emit_buffered
Output Plugin Test Driver
Feed test data into plugin
module
M
#format
Store return value of "super"
75. Using #prepend ... doesn't work :-(
OutputMyOutput
@buffer
#write
#emit_events
#format
#emit_buffered
Output Plugin Test Driver
Feed test data into plugin
#format
Test code sometimes overwrites methods for many reasons :P
singleton
class
#format
😱
module
M
78. class A
#bar
class B
#bar
super
module P
Another Study: How To Wrap Singleton Method?
B.new.singleton_class
#bar
b=B.new
b.singleton_class.module_eval{define_method(:bar){"1"}}
b.singleton_class.prepend P
b.bar
#bar
module M
#bar
Singleton class is a class,
so it can be prepended :)
It's actually done in Test Driver implementation...
79. Using #prepend on singleton_class: Yay!
OutputMyOutput
@buffer
#write
#emit_events
#format
#emit_buffered
Output Plugin Test Driver
Feed test data into plugin
#format
singleton
class
#format😃
module
M
Prepending modules on singleton_class overrides everything!