Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fighting API Compatibility On Fluentd Using "Black Magic"

6,837 views

Published on

YAPC::Asia Hachioji 2016 mid day 1

Published in: Software
  • Dating for everyone is here: ♥♥♥ http://bit.ly/2ZDZFYj ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Follow the link, new dating source: ♥♥♥ http://bit.ly/2ZDZFYj ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Fighting API Compatibility On Fluentd Using "Black Magic"

  1. 1. Fighting API Compatibility On Fluentd Using "Black Magic" Jul 2 2016 in YAPC::Asia Hachioji #yapc8oji Satoshi "Moris" Tagomori (@tagomoris)
  2. 2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, ... Treasure Data, Inc.
  3. 3. http://docs.fluentd.org/articles/logo
  4. 4. Fluentd v0.14 Release
  5. 5. Fluentd v0.14 API Update • Everything changed :) • Plugin namespace • before: Fluent::* (Top level classes even for plugins!) • after: Fluent::Plugin::* • Plugin base class for common methods • Inconsistent Output plugin hierarchy • Plugin must call `super` in common methods http://www.slideshare.net/tagomoris/fluentd-v014-plugin-api-details
  6. 6. Classes hierarchy (v0.12) Fluent::Input F::Filter F::Output BufferedOutput Object Buffered Time Sliced Multi Output F::Buffer F::Parser F::Formatter 3rd party plugins
  7. 7. Classes hierarchy (v0.14) F::P::Input F::P::Filter F::P::Output Fluent::Plugin::Base F::P::Buffer F::P::Parser F::P::Formatter F::P::Storage both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin
  8. 8. diff v0.12 v0.14 F::P::Output Fluent::Plugin::Base both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin F::Output BufferedOutput Object Buffered Time Sliced Multi Output Super classes by how to buffer data All output plugins are just "Output"
  9. 9. Basic Weapons: Class and Mixin in Ruby
  10. 10. Class and Subclass in Ruby class A #bar class B #bar super B.new.bar
  11. 11. class A #bar class B #bar super B.new.bar module M #bar Introducing Methods by Mixin
  12. 12. class A #bar class B #bar super B.new.bar module M #bar Singleton Class of Ruby #bar B.new.singleton_class
  13. 13. class A #bar class B #bar super b=B.new b.singleton_class.include M2 b.bar module M #bar Adding Methods on An Instance (1) B.new.singleton_class #bar M2 #bar
  14. 14. class A #bar class B #bar super b=B.new b.extend M2 b.bar module M #bar Adding Methods on An Instance (2) B.new.singleton_class #bar M2 #bar
  15. 15. Back to Fluentd code :)
  16. 16. diff v0.12 v0.14 F::P::Output Fluent::Plugin::Base both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin F::Output BufferedOutput Object Buffered Time Sliced Multi Output Super classes by how to buffer data All output plugins are just "Output"
  17. 17. Fluentd v0.12 Fluent::Output class Fluent::Output #emit(tag, es, chain) MyOutput Engine calls plugin.emit(tag, es, chain) @buffer
  18. 18. Fluentd v0.12 Fluent::BufferedOutput (1) class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #format(tag,time,record) #format_stream(tag,es)
  19. 19. Fluentd v0.12 Fluent::BufferedOutput (2) class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #format_stream(tag,es) #format_stream(tag,es) #format(tag,time,record)
  20. 20. Fluentd v0.12 Fluent::TimeSlicedOutput class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #emit(tag, es, chain) class TimeSlicedOutput #format(tag,time,record)
  21. 21. Fluentd v0.12 Fluent::ObjectBufferedOutput class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #emit(tag, es, chain) class ObjectBufferedOutput
  22. 22. Fluentd v0.12 Fluent::BufferedOutput class Fluent::Outputclass BufferedOutputMyOutput @buffer calls #write in OutputThread @buffer chunk #write(chunk) OutputThread #pop
  23. 23. Fluentd v0.12 Fluent::TimeSlicedOutput class Fluent::Outputclass BufferedOutput @buffer MyOutput class TimeSlicedOutput OutputThread #write(chunk) @buffer calls #write in OutputThread #write calls chunk.key chunk #pop
  24. 24. Fluentd v0.12 Fluent::ObjectBufferedOutput class Fluent::Outputclass BufferedOutput @buffer MyOutput class ObjectBufferedOutput OutputThread #write(chunk) #write(chunk) #write_object(chunk_key, chunk) @buffer calls #write in OutputThread chunk #pop
  25. 25. Fluentd v0.12 API Problems • Entry point method is implemented by Plugin subclasses • Fluentd core cannot add any processes • counting input events • hook arguments/return values to update API • Fluentd core didn't show fixed API • Plugins have different call stacks • It's not clear what should be implemented for authors • It's not clear what interfaces are supported for arguments/return values
  26. 26. How can we solve this problem?
  27. 27. Fluent::Plugin::Output (v0.14)
  28. 28. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput #process(tag, es) Engine calls plugin.emit_events(tag, es) @buffer #write #emit_events(tag, es) #format(tag, time, record) #write(chunk) #try_write(chunk) #emit_sync(tag, es) #emit_buffered(tag, es)
  29. 29. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput Output calls plugin.write (or try_write) @buffer chunk #write(chunk) #try_write(chunk) flush thread #process(tag, es) #format(tag, time, record)
  30. 30. Fluentd v0.14 Design Policy • Separate entry points from implementations • Methods in superclass control everything • Do NOT override these methods! • Methods in subclass do things only for themselves • not for data flow, control flow nor others • Plugins have simple/straightforward call stack • Easy to understand/maintain
  31. 31.
  32. 32. How about existing v0.12 plugins?
  33. 33. Requirement: (Almost) All Existing Plugins SHOULD
 Work Well WITHOUT ANY MODIFICATION
  34. 34. • Fluent::Compat namespace for compatibility layer v0.14 Plugins & Compat Layer F::P::Output F::P::Base v0.14 Plugins Fluent:: Compat:: Output F::C:: Buffered Output F::C:: TimeSliced Output F::C:: ObjectBuffered Output Fluent::Output F:: Buffered Output F:: TimeSliced Output F:: ObjectBuffered Output v0.12 Plugins
  35. 35. Double Decker Compat Layer? • Existing plugins inherits Fluent::Output or others • No more codes in Fluent top level :-( • Separate code into Fluent::Compat • and import it into Fluent top level
  36. 36. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput #process(tag, es) Engine calls plugin.emit_events(tag, es) @buffer #write #emit_events(tag, es) #format(tag, time, record) #write(chunk) #try_write(chunk) #emit_sync(tag, es) #emit_buffered(tag, es)
  37. 37. Fluentd v0.12 Fluent::BufferedOutput (2) class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #format_stream(tag,es) #format_stream(tag,es) #format(tag,time,record)
  38. 38. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) v0.12 Plugins via Compat Layer: Best case (virtual) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
  39. 39. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) v0.12 Plugins via Compat Layer: Best case (real) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
  40. 40. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #format_stream Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
  41. 41. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #format_stream Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread default implementation for calling "super"
  42. 42. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
  43. 43. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread When plugin overrides #emit
  44. 44. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread This call doesn't happen, in fact #emit doesn't return values! When plugin overrides #emit
  45. 45. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #emit calls @buffer.emit → NoMethodError !
  46. 46. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
  47. 47. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #emit 1. #emit calls @buffer.emit with data to be written in buffer 0. plugin calls @buffer.extend to add #emit 2. @buffer.emit stores arguments into plugin's attribute 3. get stored data 4. call @buffer.write with data
  48. 48. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
  49. 49. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Thinking about "chunk" instance ... Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #write may call "chunk.key", but v0.14 chunk doesn't have #key !
  50. 50. Fluent::Plugin::Outputclass MyOutput @buffer #write Compat::BufferedOutput #write(chunk) flush thread "chunk" has #metadata, and values of #key can be created via #metadata Let's "chunk.extend" ! Where to do so? ? Thinking about "chunk" instance ...
  51. 51. Fluent::Plugin::OutputMyOutput @buffer #write C::BufferedOutput #write(chunk) flush thread Thinking about "chunk" instance ... #write(chunk) BufferedChunkMixin plugin.extend BufferedChunkMixin in #configure
  52. 52. Similar hacks for TimeSlicedOutput and ObjectBufferedOutput ...
  53. 53. Controlling Plugin Lifecycle
  54. 54. Plugin Lifecycle Updated Methods(v0.12) • #configure • #start • #before_shutdown • #shutdown v0.12 Plugins often doesn't call "super"! Methods(v0.14) • #configure • #start • #stop • #before_shutdown • #shutdown • #after_shutdown • #close • #terminate In v0.14, these methods MUST call "super" • #configured? • #started? • #stopped? • #before_shutdown? • #shutdown? • #after_shutdown? • #closed? • #terminated?
  55. 55. For Example: shutdown compat plugins Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown It doesn't call "super"! We want to call this...
  56. 56. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown 1. call #shutdown anyway 0. Fluentd core calls #shutdown 2. call #shutdown? to check "super" is called or not 3. call #shutdown of superclass forcedly!
  57. 57. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown How to make this point?
  58. 58. More Weapon! Module#prepend
  59. 59. class A #bar class B #bar super B.new.bar Wrapping Methods on a Class (1) B.new.singleton_class #bar
  60. 60. class A #bar class B #bar super B.new.bar module M Wrapping Methods on a Class (2) B.new.singleton_class #bar #bar Using extend is powerful, but it should be done for all instances How about wrapping methods for all instances of the class?
  61. 61. class A #bar class B #bar super module M;def bar;super;end;end B.prepend M B.new.bar module M Wrapping Methods on a Class (3): Module#prepend B.new.singleton_class #bar #bar module M wraps B, and M#bar is called at first
  62. 62. class A #bar class B #bar super b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.bar Another Study: How To Wrap Singleton Method? B.new.singleton_class #bar
  63. 63. class A #bar class B #bar super module M Another Study: How To Wrap Singleton Method? B.new.singleton_class #bar Singleton class is a class, so it can be prepended :) b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.singleton_class.prepend M b.bar #bar It's actually done in Test Driver implementation...
  64. 64. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown THIS ONE !!!
  65. 65. What We Got :-) Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown 1. call #shutdown anyway 0. prepend CallSuperMixin at first 2. call #shutdown? to check "super" is called or not 3. if not, get method of superclass, bind self with it, then call it Thank you @unak -san!
  66. 66. IS BUILT ON A TOP OF BUNCH OF BLACK MAGICS :P
  67. 67. Do Whatever You Can For Users! It Makes Everyone Happier ... Except for Maintainers :(

×