SlideShare a Scribd company logo
1 of 59
Download to read offline
HIJACKING RUBY SYNTAX
IN RUBY
RubyKaigi 2018 Sendai #rubykaigi
2018/05/31 (Fri)
@joker1007 & @tagomoris
Self-Intro Joker
▸ id: joker1007
▸ Repro inc. CTO
▸ I’m familiar with Ruby/Rails/Fluentd/ECS/Presto.
▸ I ♥ “Jojo’s Bizarre Adventure” so much.
▸ I’m very happy to talk in Sendai (Moriocho)
Satoshi Tagomori (@tagomoris)
Fluentd, MessagePack-Ruby, Norikra, Woothee, ...
Treasure Data, Inc.
Hijacking Ruby Syntax in Ruby
Agenda
▸ Ruby Features
▸ Hacks
▸ finalist, overrider, abstriker
▸ binding_ninja
▸ with_resources, deferral
Ruby Features: Binding
Binding
▸ Context object, includes:
▸ Kernel#binding receiver
▸ local variables
▸ For template engines (?)
▸ Methods:
▸ #receiver, #eval,

#local_variables,

#local_variable_get,

#local_variable_defined?,

#local_variable_set
テキスト
テキスト
テキスト
Ruby Features: Binding
Binding#local_variable_set
▸ Method to
▸ add a variable only in a binding instance
▸ overwrite values of existing variables in original
context
Ruby Features: TracePoint
TracePoint
▸ Tracing events in VM
▸ For various events:
▸ :line , :raise
▸ :class , :end
▸ :call , :return , :c_call , :c_return , :b_call , :b_return
▸ :thread_begin , :thread_end , :fiber_switch
▸ "We can use TracePoint to gather information specifically
for exceptions:" (from Ruby doc)
▸ This is COMPLETELY WRONG statement...
Ruby Features: TracePoint
TracePoint Methods
▸ Methods:
▸ Control: disable, enable, enabled?
▸ Event what/where: event, defined_class, path, lineno
▸ Method names: method_id, callee_id
▸ Event special: raised_exception, return_value
▸ And: binding
▸ So what?
▸ We can use TracePoint 
▸ to gather information
▸ to overwrite everything
Ruby Features: TracePoint
Hijacking Ruby Syntax in Ruby
Hijacking Ruby Syntax in Ruby
テキスト
10min
Break (10min)
Ruby Features: Refinements
▸ Refinements provide a way to extend a class locally
▸ Useful use case. (Safety monkey patching)
Ruby Features: Refinements
Another use case
▸ Super private method
Ruby Features: Refinements
Hacks 1 Method modifiers
▸ final (https://github.com/joker1007/finalist)
▸ forbid method override
▸ override (https://github.com/joker1007/overrider)
▸ enforce method has super method
▸ abstract (https://github.com/joker1007/abstriker)
▸ enforce method override
These method modifiers work when Ruby defines class.
It is actually runtime, but in most case, before main logic.
Hacks: method modifiers
Sample code: finalist
Hacks: method modifiers
Sample code: overrider
Hacks: method modifiers
Sample code: abstriker
Hacks: method modifiers
How to implement method modifiers
I use so many hook methods.
`included`, `extended`, `method_added`, and `TracePoint`.
And I use `Ripper`.
In other words, I use the power of many black magics !!
Hacks: method modifiers
Ruby Features: Method Hooks
▸ Ruby has six method hooks
▸ Module#method_added
▸ Module#method_removed
▸ Module#method_undefined
▸ BasicObject#singleton_method_added
▸ BasicObject#singleton_method_removed
▸ BasicObject#singleton_method_undefined
Ruby Features: method hooks
Sample: #method_added
Ruby Features: method hooks
Sample code: #method_removed
Ruby Features: method hooks
Sample code: #method_undefined
Ruby Features: method hooks
Use case in finalist (Simplified sample)
Ruby Features: method hooks
call method_added
-> call verify_final_method
Usage: Method Hook
▸ Method Hook provides a way to implement declarations (like
protected, private)
▸ With class inheritance, Method Hook seems magic !!
▸ But it is not enough to implement “finalist” actually
▸ Ruby has so many cases of method definition
▸ `def` or `define_method`
▸ `include` module
▸ `extend`, `prepend
▸ Each case calls different hooks
Ruby Features: method hooks
“include” does not add method
▸ Because `include` insert module to method lookup
hierarchy, it does not touch method entry table.
Ruby Features: method hooks
“include” changes only chain of method discovering
Foo
Object
Bar
Insert module to hierarchy
It is different from to add method
Class#ancestors displays class-module hierarchy
Ruby Features: method hooks
Hooks in finalist gem
▸ `method_added` to detect override by subclass
▸ `singleton_method_added` to detect override class
method
▸ `included` to detect override by module include
▸ `extended` to detect override by module extend
And I talk about TracePoint too
overrider and abstriker uses TracePoint
▸ `inherited` and `included` to start `TracePoint``
call main logic
call main logic
Tracepoint in overrider, abstriker
Tracepoint in overrider, abstriker
Check abstract method existence in class hierarchy
Why use TracePoint?
▸ In order to verify method existence at the end of class
definition.
▸ Ruby interpreter needs to wait until the end of class
definition to know a method absence.
▸ override and abstract cannot detect violation just when
they are called.
▸ In ruby, The only way to detect the violation is
`TracePoint`.
Advanced TracePoint: Detect particular class end
Advanced Tracepoint
:end event cannot trace definition by `Class.new`.
Use :c_return and return_value to detect particular class end
Advanced TracePoint: Ripper combination
Advanced Tracepoint
Detect target Sexp node by TracePoint#linen
Sexp node type expresses the style of method cal
Ripper empowers TracePoint
▸ `Ripper.sexp` returns token position
▸ And TracePoint returns lineno an event occurs.
▸ Ripper provides more information about event context.
▸ ex. power_assert gem is implemented by this combination.
Advanced Tracepoint
Hacks 2 (binding_ninja)
▸ https://github.com/joker1007/binding_ninja
▸ It is method wrapper to pass binding of method caller
implicitly.
▸ I talked about it at Rubykaigi 2017 LT.
Hacks: binding_ninja
Sample code: binding_ninja
Hacks: binding_ninja
Pass binding of caller implicitly
How To Implement
▸ Binding is based on Ruby level control frame.
▸ When Ruby calls c-method, Ruby does not change Ruby level control
frame.
▸ It is core logic of binding_ninja
create binding in C level, binding keeps caller stack in Ruby level
Hacks: binding_ninja
Use case: binding_ninja
▸ I implement scala like
implicit parameter in Ruby.
▸ https://github.com/
joker1007/
implicit_parameter
Hacks: binding_ninja
Hacks: binding_ninja
Wrap method by `define_method`
And wrap method by binding_ninja
Black Magic is dangerous actually,
but it is very fun,
and it extends Ruby potential
These gems is for proof of concept.
But these work and decent
practical.
Ruby Quiz
Break (25-30min)
What the difference between:
- #undef_method
- #remove_method
RUBY QUIZ
class Foo
def foo
class Foo
def foo
class Bar < Foo
def foo
class Bar < Foo
def foo
Bar.new.foo()
RUBY QUIZ
class Foo
def foo
class Foo
def foo
class Bar < Foo
def foo
class Bar < Foo
Bar.new.foo()
undef_method(:foo)
NoMethodError
remove_method(:foo)
Hack: with_resources
Add "with" Statement in Ruby
▸ Safe resource allocate/release
▸ Ensure to release resources
▸ after a lexical scope
▸ in reverse order of allocation
▸ Idioms used very frequently
▸ Other languages:
▸ Java: try-with-resources
▸ Python: with
▸ C#: using
Hack: with_resources
Safe Resource Allocation/Release Statement in Ruby
▸ Open method with blocks
▸ File.open(path){|f| ... }
▸ Ruby way (?)
▸ More indentation
▸ Not implemented sometimes

(e.g., TCPSocket)
▸ Simple begin-ensure
▸ Anomaly cases can't be

handled
Hack: with_resources
with_resources.gem
▸ Safe resource allocation
▸ Top level "with"
▸ via Kernel refinement
▸ Resource allocation as lambda
▸ Multi statements to allocate resources
▸ to release first resource

if second resource allocation raises exception
▸ Block to define scope for resources
https://github.com/tagomoris/with_resources
Hack: with_resources
Implementing "with" in Ruby
▸ TracePoint
▸ "b_return": pass allocated resources to block arguments
▸ "line": identify allocated resources in lambda
▸ Binding
▸ detect newly defined

local variables

in allocation lambda
▸ Refinements
▸ introduce "with"

in top-level without side effects
https://github.com/tagomoris/with_resources
Hack: deferral
Alternative: defer?
▸ Multi-step resource

allocation in a method
▸ using "with"
▸ Nesting!
▸ not so bad
▸ many nesting looks

a bit messy :(
▸ Alternative?
▸ "defer" in golang
Hack: deferral
Making "defer" in Ruby
▸ Block based "defer"
▸ Should work
▸ Requires 1-level

nesting *always*
▸ Defer.start, end (+ nesting)

look too much (than golang)
▸ Abnormal case:

reassigning variables
Hack: deferral
deferral.gem
▸ Safe resource release
▸ Top level "defer"
▸ via Kernel refinements
▸ Deferred processing to release resources
▸ at the end of scope (method, block)
▸ or exception raised
Hack: deferral
Implementing "defer" in Ruby
▸ #defer
▸ Enable TracePoint if not yet
▸ Initialize internal stack frame
▸ TracePoint
▸ Monitor method call stack
▸ Get the snapshot of local variables in defer block
▸ Call release blocks at the end of scope
▸ Binding
▸ save/restore local variables of release block
▸ Refinements
▸ introduce "defer" in top-level without side effects
stack level 0
stack level 1
The Hard Thing
in Magical World:
Debugging!
"Are You Ready?
I'm OK." Blono Bucciarati
Thank You!
@joker1007 & @tagomoris

More Related Content

What's hot

The Future of library dependency management of Ruby
 The Future of library dependency management of Ruby The Future of library dependency management of Ruby
The Future of library dependency management of RubyHiroshi SHIBATA
 
Dependency Resolution with Standard Libraries
Dependency Resolution with Standard LibrariesDependency Resolution with Standard Libraries
Dependency Resolution with Standard LibrariesHiroshi SHIBATA
 
OSS Security the hard way
OSS Security the hard wayOSS Security the hard way
OSS Security the hard wayHiroshi SHIBATA
 
How to distribute Ruby to the world
How to distribute Ruby to the worldHow to distribute Ruby to the world
How to distribute Ruby to the worldHiroshi SHIBATA
 
How to Begin Developing Ruby Core
How to Begin Developing Ruby CoreHow to Begin Developing Ruby Core
How to Begin Developing Ruby CoreHiroshi SHIBATA
 
The Future of Dependency Management for Ruby
The Future of Dependency Management for RubyThe Future of Dependency Management for Ruby
The Future of Dependency Management for RubyHiroshi SHIBATA
 
How to distribute Ruby to the world
How to distribute Ruby to the worldHow to distribute Ruby to the world
How to distribute Ruby to the worldHiroshi SHIBATA
 
Gemification for Ruby 2.5/3.0
Gemification for Ruby 2.5/3.0Gemification for Ruby 2.5/3.0
Gemification for Ruby 2.5/3.0Hiroshi SHIBATA
 
How to test code with mruby
How to test code with mrubyHow to test code with mruby
How to test code with mrubyHiroshi SHIBATA
 
Roadmap for RubyGems 4 and Bundler 3
Roadmap for RubyGems 4 and Bundler 3Roadmap for RubyGems 4 and Bundler 3
Roadmap for RubyGems 4 and Bundler 3Hiroshi SHIBATA
 
The Future of library dependency manageement of Ruby
The Future of library dependency manageement of RubyThe Future of library dependency manageement of Ruby
The Future of library dependency manageement of RubyHiroshi SHIBATA
 
The secret of programming language development and future
The secret of programming  language development and futureThe secret of programming  language development and future
The secret of programming language development and futureHiroshi SHIBATA
 
The Future of Bundled Bundler
The Future of Bundled BundlerThe Future of Bundled Bundler
The Future of Bundled BundlerHiroshi SHIBATA
 
Gemification plan of Standard Library on Ruby
Gemification plan of Standard Library on RubyGemification plan of Standard Library on Ruby
Gemification plan of Standard Library on RubyHiroshi SHIBATA
 
Deploying Perl apps on dotCloud
Deploying Perl apps on dotCloudDeploying Perl apps on dotCloud
Deploying Perl apps on dotClouddaoswald
 
mruby で mackerel のプラグインを作るはなし
mruby で mackerel のプラグインを作るはなしmruby で mackerel のプラグインを作るはなし
mruby で mackerel のプラグインを作るはなしHiroshi SHIBATA
 
Gemification for Ruby 2.5/3.0
Gemification for Ruby 2.5/3.0Gemification for Ruby 2.5/3.0
Gemification for Ruby 2.5/3.0Hiroshi SHIBATA
 

What's hot (20)

Practical ngx_mruby
Practical ngx_mrubyPractical ngx_mruby
Practical ngx_mruby
 
The Future of library dependency management of Ruby
 The Future of library dependency management of Ruby The Future of library dependency management of Ruby
The Future of library dependency management of Ruby
 
Dependency Resolution with Standard Libraries
Dependency Resolution with Standard LibrariesDependency Resolution with Standard Libraries
Dependency Resolution with Standard Libraries
 
OSS Security the hard way
OSS Security the hard wayOSS Security the hard way
OSS Security the hard way
 
How to distribute Ruby to the world
How to distribute Ruby to the worldHow to distribute Ruby to the world
How to distribute Ruby to the world
 
How to Begin Developing Ruby Core
How to Begin Developing Ruby CoreHow to Begin Developing Ruby Core
How to Begin Developing Ruby Core
 
The Future of Dependency Management for Ruby
The Future of Dependency Management for RubyThe Future of Dependency Management for Ruby
The Future of Dependency Management for Ruby
 
How to distribute Ruby to the world
How to distribute Ruby to the worldHow to distribute Ruby to the world
How to distribute Ruby to the world
 
Gemification for Ruby 2.5/3.0
Gemification for Ruby 2.5/3.0Gemification for Ruby 2.5/3.0
Gemification for Ruby 2.5/3.0
 
How to test code with mruby
How to test code with mrubyHow to test code with mruby
How to test code with mruby
 
Roadmap for RubyGems 4 and Bundler 3
Roadmap for RubyGems 4 and Bundler 3Roadmap for RubyGems 4 and Bundler 3
Roadmap for RubyGems 4 and Bundler 3
 
The Future of library dependency manageement of Ruby
The Future of library dependency manageement of RubyThe Future of library dependency manageement of Ruby
The Future of library dependency manageement of Ruby
 
The secret of programming language development and future
The secret of programming  language development and futureThe secret of programming  language development and future
The secret of programming language development and future
 
Gems on Ruby
Gems on RubyGems on Ruby
Gems on Ruby
 
The Future of Bundled Bundler
The Future of Bundled BundlerThe Future of Bundled Bundler
The Future of Bundled Bundler
 
Gemification plan of Standard Library on Ruby
Gemification plan of Standard Library on RubyGemification plan of Standard Library on Ruby
Gemification plan of Standard Library on Ruby
 
Deploying Perl apps on dotCloud
Deploying Perl apps on dotCloudDeploying Perl apps on dotCloud
Deploying Perl apps on dotCloud
 
Debugging on rails
Debugging on railsDebugging on rails
Debugging on rails
 
mruby で mackerel のプラグインを作るはなし
mruby で mackerel のプラグインを作るはなしmruby で mackerel のプラグインを作るはなし
mruby で mackerel のプラグインを作るはなし
 
Gemification for Ruby 2.5/3.0
Gemification for Ruby 2.5/3.0Gemification for Ruby 2.5/3.0
Gemification for Ruby 2.5/3.0
 

Similar to Hijacking Ruby Syntax in Ruby

Ruby for C# Developers
Ruby for C# DevelopersRuby for C# Developers
Ruby for C# DevelopersCory Foy
 
Rapid Development with Ruby/JRuby and Rails
Rapid Development with Ruby/JRuby and RailsRapid Development with Ruby/JRuby and Rails
Rapid Development with Ruby/JRuby and Railselliando dias
 
Ruby for .NET developers
Ruby for .NET developersRuby for .NET developers
Ruby for .NET developersMax Titov
 
Debugging Rails 3 Applications
Debugging Rails 3 ApplicationsDebugging Rails 3 Applications
Debugging Rails 3 ApplicationsNathan Broadbent
 
Ruby Metaprogramming - OSCON 2008
Ruby Metaprogramming - OSCON 2008Ruby Metaprogramming - OSCON 2008
Ruby Metaprogramming - OSCON 2008Brian Sam-Bodden
 
Random Ruby Tips - Ruby Meetup 27 Jun 2018
Random Ruby Tips - Ruby Meetup 27 Jun 2018Random Ruby Tips - Ruby Meetup 27 Jun 2018
Random Ruby Tips - Ruby Meetup 27 Jun 2018Kenneth Teh
 
Hacking with ruby2ruby
Hacking with ruby2rubyHacking with ruby2ruby
Hacking with ruby2rubyMarc Chung
 
Ruby v cpp_preso
Ruby v cpp_presoRuby v cpp_preso
Ruby v cpp_presojessicard
 
TDD with PhpSpec - Lone Star PHP 2016
TDD with PhpSpec - Lone Star PHP 2016TDD with PhpSpec - Lone Star PHP 2016
TDD with PhpSpec - Lone Star PHP 2016CiaranMcNulty
 
Pentesting an unfriendly environment: bypassing (un)common defences and mate ...
Pentesting an unfriendly environment: bypassing (un)common defences and mate ...Pentesting an unfriendly environment: bypassing (un)common defences and mate ...
Pentesting an unfriendly environment: bypassing (un)common defences and mate ...Sandro Zaccarini
 
Pourquoi ruby et rails déchirent
Pourquoi ruby et rails déchirentPourquoi ruby et rails déchirent
Pourquoi ruby et rails déchirentNicolas Ledez
 
The details of CI/CD environment for Ruby
The details of CI/CD environment for RubyThe details of CI/CD environment for Ruby
The details of CI/CD environment for RubyHiroshi SHIBATA
 
Invokedynamic / JSR-292
Invokedynamic / JSR-292Invokedynamic / JSR-292
Invokedynamic / JSR-292ytoshima
 
Jruby a Pi and a database
Jruby a Pi and a databaseJruby a Pi and a database
Jruby a Pi and a databasePhilipp Fehre
 
Magic in ruby
Magic in rubyMagic in ruby
Magic in rubyDavid Lin
 

Similar to Hijacking Ruby Syntax in Ruby (20)

Ruby for C# Developers
Ruby for C# DevelopersRuby for C# Developers
Ruby for C# Developers
 
Rapid Development with Ruby/JRuby and Rails
Rapid Development with Ruby/JRuby and RailsRapid Development with Ruby/JRuby and Rails
Rapid Development with Ruby/JRuby and Rails
 
Low-Maintenance Perl
Low-Maintenance PerlLow-Maintenance Perl
Low-Maintenance Perl
 
Ruby for .NET developers
Ruby for .NET developersRuby for .NET developers
Ruby for .NET developers
 
Debugging Rails 3 Applications
Debugging Rails 3 ApplicationsDebugging Rails 3 Applications
Debugging Rails 3 Applications
 
Ruby
RubyRuby
Ruby
 
Ruby Metaprogramming - OSCON 2008
Ruby Metaprogramming - OSCON 2008Ruby Metaprogramming - OSCON 2008
Ruby Metaprogramming - OSCON 2008
 
Random Ruby Tips - Ruby Meetup 27 Jun 2018
Random Ruby Tips - Ruby Meetup 27 Jun 2018Random Ruby Tips - Ruby Meetup 27 Jun 2018
Random Ruby Tips - Ruby Meetup 27 Jun 2018
 
Ruby on Rails 3 Day BC
Ruby on Rails 3 Day BCRuby on Rails 3 Day BC
Ruby on Rails 3 Day BC
 
Hacking with ruby2ruby
Hacking with ruby2rubyHacking with ruby2ruby
Hacking with ruby2ruby
 
Ruby Hell Yeah
Ruby Hell YeahRuby Hell Yeah
Ruby Hell Yeah
 
Ruby v cpp_preso
Ruby v cpp_presoRuby v cpp_preso
Ruby v cpp_preso
 
TDD with PhpSpec - Lone Star PHP 2016
TDD with PhpSpec - Lone Star PHP 2016TDD with PhpSpec - Lone Star PHP 2016
TDD with PhpSpec - Lone Star PHP 2016
 
Pentesting an unfriendly environment: bypassing (un)common defences and mate ...
Pentesting an unfriendly environment: bypassing (un)common defences and mate ...Pentesting an unfriendly environment: bypassing (un)common defences and mate ...
Pentesting an unfriendly environment: bypassing (un)common defences and mate ...
 
Pourquoi ruby et rails déchirent
Pourquoi ruby et rails déchirentPourquoi ruby et rails déchirent
Pourquoi ruby et rails déchirent
 
The details of CI/CD environment for Ruby
The details of CI/CD environment for RubyThe details of CI/CD environment for Ruby
The details of CI/CD environment for Ruby
 
Invokedynamic / JSR-292
Invokedynamic / JSR-292Invokedynamic / JSR-292
Invokedynamic / JSR-292
 
Jruby a Pi and a database
Jruby a Pi and a databaseJruby a Pi and a database
Jruby a Pi and a database
 
PDF Ruby on Rails 3 Day BC
 PDF Ruby on Rails 3 Day BC PDF Ruby on Rails 3 Day BC
PDF Ruby on Rails 3 Day BC
 
Magic in ruby
Magic in rubyMagic in ruby
Magic in ruby
 

More from SATOSHI TAGOMORI

Ractor's speed is not light-speed
Ractor's speed is not light-speedRactor's speed is not light-speed
Ractor's speed is not light-speedSATOSHI TAGOMORI
 
Good Things and Hard Things of SaaS Development/Operations
Good Things and Hard Things of SaaS Development/OperationsGood Things and Hard Things of SaaS Development/Operations
Good Things and Hard Things of SaaS Development/OperationsSATOSHI TAGOMORI
 
Invitation to the dark side of Ruby
Invitation to the dark side of RubyInvitation to the dark side of Ruby
Invitation to the dark side of RubySATOSHI TAGOMORI
 
Make Your Ruby Script Confusing
Make Your Ruby Script ConfusingMake Your Ruby Script Confusing
Make Your Ruby Script ConfusingSATOSHI TAGOMORI
 
Lock, Concurrency and Throughput of Exclusive Operations
Lock, Concurrency and Throughput of Exclusive OperationsLock, Concurrency and Throughput of Exclusive Operations
Lock, Concurrency and Throughput of Exclusive OperationsSATOSHI TAGOMORI
 
Data Processing and Ruby in the World
Data Processing and Ruby in the WorldData Processing and Ruby in the World
Data Processing and Ruby in the WorldSATOSHI TAGOMORI
 
Planet-scale Data Ingestion Pipeline: Bigdam
Planet-scale Data Ingestion Pipeline: BigdamPlanet-scale Data Ingestion Pipeline: Bigdam
Planet-scale Data Ingestion Pipeline: BigdamSATOSHI TAGOMORI
 
Technologies, Data Analytics Service and Enterprise Business
Technologies, Data Analytics Service and Enterprise BusinessTechnologies, Data Analytics Service and Enterprise Business
Technologies, Data Analytics Service and Enterprise BusinessSATOSHI TAGOMORI
 
Ruby and Distributed Storage Systems
Ruby and Distributed Storage SystemsRuby and Distributed Storage Systems
Ruby and Distributed Storage SystemsSATOSHI TAGOMORI
 
Perfect Norikra 2nd Season
Perfect Norikra 2nd SeasonPerfect Norikra 2nd Season
Perfect Norikra 2nd SeasonSATOSHI TAGOMORI
 
To Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT ToTo Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT ToSATOSHI TAGOMORI
 
The Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersThe Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersSATOSHI TAGOMORI
 
How To Write Middleware In Ruby
How To Write Middleware In RubyHow To Write Middleware In Ruby
How To Write Middleware In RubySATOSHI TAGOMORI
 
Modern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real WorldModern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real WorldSATOSHI TAGOMORI
 
Open Source Software, Distributed Systems, Database as a Cloud Service
Open Source Software, Distributed Systems, Database as a Cloud ServiceOpen Source Software, Distributed Systems, Database as a Cloud Service
Open Source Software, Distributed Systems, Database as a Cloud ServiceSATOSHI TAGOMORI
 
Fluentd Overview, Now and Then
Fluentd Overview, Now and ThenFluentd Overview, Now and Then
Fluentd Overview, Now and ThenSATOSHI TAGOMORI
 
How to Make Norikra Perfect
How to Make Norikra PerfectHow to Make Norikra Perfect
How to Make Norikra PerfectSATOSHI TAGOMORI
 
Distributed Logging Architecture in Container Era
Distributed Logging Architecture in Container EraDistributed Logging Architecture in Container Era
Distributed Logging Architecture in Container EraSATOSHI TAGOMORI
 

More from SATOSHI TAGOMORI (20)

Ractor's speed is not light-speed
Ractor's speed is not light-speedRactor's speed is not light-speed
Ractor's speed is not light-speed
 
Good Things and Hard Things of SaaS Development/Operations
Good Things and Hard Things of SaaS Development/OperationsGood Things and Hard Things of SaaS Development/Operations
Good Things and Hard Things of SaaS Development/Operations
 
Maccro Strikes Back
Maccro Strikes BackMaccro Strikes Back
Maccro Strikes Back
 
Invitation to the dark side of Ruby
Invitation to the dark side of RubyInvitation to the dark side of Ruby
Invitation to the dark side of Ruby
 
Make Your Ruby Script Confusing
Make Your Ruby Script ConfusingMake Your Ruby Script Confusing
Make Your Ruby Script Confusing
 
Lock, Concurrency and Throughput of Exclusive Operations
Lock, Concurrency and Throughput of Exclusive OperationsLock, Concurrency and Throughput of Exclusive Operations
Lock, Concurrency and Throughput of Exclusive Operations
 
Data Processing and Ruby in the World
Data Processing and Ruby in the WorldData Processing and Ruby in the World
Data Processing and Ruby in the World
 
Planet-scale Data Ingestion Pipeline: Bigdam
Planet-scale Data Ingestion Pipeline: BigdamPlanet-scale Data Ingestion Pipeline: Bigdam
Planet-scale Data Ingestion Pipeline: Bigdam
 
Technologies, Data Analytics Service and Enterprise Business
Technologies, Data Analytics Service and Enterprise BusinessTechnologies, Data Analytics Service and Enterprise Business
Technologies, Data Analytics Service and Enterprise Business
 
Ruby and Distributed Storage Systems
Ruby and Distributed Storage SystemsRuby and Distributed Storage Systems
Ruby and Distributed Storage Systems
 
Perfect Norikra 2nd Season
Perfect Norikra 2nd SeasonPerfect Norikra 2nd Season
Perfect Norikra 2nd Season
 
Fluentd 101
Fluentd 101Fluentd 101
Fluentd 101
 
To Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT ToTo Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT To
 
The Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersThe Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and Containers
 
How To Write Middleware In Ruby
How To Write Middleware In RubyHow To Write Middleware In Ruby
How To Write Middleware In Ruby
 
Modern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real WorldModern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real World
 
Open Source Software, Distributed Systems, Database as a Cloud Service
Open Source Software, Distributed Systems, Database as a Cloud ServiceOpen Source Software, Distributed Systems, Database as a Cloud Service
Open Source Software, Distributed Systems, Database as a Cloud Service
 
Fluentd Overview, Now and Then
Fluentd Overview, Now and ThenFluentd Overview, Now and Then
Fluentd Overview, Now and Then
 
How to Make Norikra Perfect
How to Make Norikra PerfectHow to Make Norikra Perfect
How to Make Norikra Perfect
 
Distributed Logging Architecture in Container Era
Distributed Logging Architecture in Container EraDistributed Logging Architecture in Container Era
Distributed Logging Architecture in Container Era
 

Recently uploaded

Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9Jürgen Gutsch
 
Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!Neo4j
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 

Recently uploaded (20)

Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Program with GUTs
Program with GUTsProgram with GUTs
Program with GUTs
 
About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9
 
Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
Sustainable Web Design - Claire Thornewill
Sustainable Web Design - Claire ThornewillSustainable Web Design - Claire Thornewill
Sustainable Web Design - Claire Thornewill
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 

Hijacking Ruby Syntax in Ruby

  • 1. HIJACKING RUBY SYNTAX IN RUBY RubyKaigi 2018 Sendai #rubykaigi 2018/05/31 (Fri) @joker1007 & @tagomoris
  • 2. Self-Intro Joker ▸ id: joker1007 ▸ Repro inc. CTO ▸ I’m familiar with Ruby/Rails/Fluentd/ECS/Presto. ▸ I ♥ “Jojo’s Bizarre Adventure” so much. ▸ I’m very happy to talk in Sendai (Moriocho)
  • 3. Satoshi Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, Woothee, ... Treasure Data, Inc.
  • 5. Agenda ▸ Ruby Features ▸ Hacks ▸ finalist, overrider, abstriker ▸ binding_ninja ▸ with_resources, deferral
  • 6. Ruby Features: Binding Binding ▸ Context object, includes: ▸ Kernel#binding receiver ▸ local variables ▸ For template engines (?) ▸ Methods: ▸ #receiver, #eval,
 #local_variables,
 #local_variable_get,
 #local_variable_defined?,
 #local_variable_set
  • 10. Ruby Features: Binding Binding#local_variable_set ▸ Method to ▸ add a variable only in a binding instance ▸ overwrite values of existing variables in original context
  • 11. Ruby Features: TracePoint TracePoint ▸ Tracing events in VM ▸ For various events: ▸ :line , :raise ▸ :class , :end ▸ :call , :return , :c_call , :c_return , :b_call , :b_return ▸ :thread_begin , :thread_end , :fiber_switch ▸ "We can use TracePoint to gather information specifically for exceptions:" (from Ruby doc) ▸ This is COMPLETELY WRONG statement...
  • 12. Ruby Features: TracePoint TracePoint Methods ▸ Methods: ▸ Control: disable, enable, enabled? ▸ Event what/where: event, defined_class, path, lineno ▸ Method names: method_id, callee_id ▸ Event special: raised_exception, return_value ▸ And: binding ▸ So what? ▸ We can use TracePoint  ▸ to gather information ▸ to overwrite everything
  • 17. Ruby Features: Refinements ▸ Refinements provide a way to extend a class locally ▸ Useful use case. (Safety monkey patching) Ruby Features: Refinements
  • 18. Another use case ▸ Super private method Ruby Features: Refinements
  • 19. Hacks 1 Method modifiers ▸ final (https://github.com/joker1007/finalist) ▸ forbid method override ▸ override (https://github.com/joker1007/overrider) ▸ enforce method has super method ▸ abstract (https://github.com/joker1007/abstriker) ▸ enforce method override These method modifiers work when Ruby defines class. It is actually runtime, but in most case, before main logic. Hacks: method modifiers
  • 20. Sample code: finalist Hacks: method modifiers
  • 21. Sample code: overrider Hacks: method modifiers
  • 22. Sample code: abstriker Hacks: method modifiers
  • 23. How to implement method modifiers I use so many hook methods. `included`, `extended`, `method_added`, and `TracePoint`. And I use `Ripper`. In other words, I use the power of many black magics !! Hacks: method modifiers
  • 24. Ruby Features: Method Hooks ▸ Ruby has six method hooks ▸ Module#method_added ▸ Module#method_removed ▸ Module#method_undefined ▸ BasicObject#singleton_method_added ▸ BasicObject#singleton_method_removed ▸ BasicObject#singleton_method_undefined Ruby Features: method hooks
  • 26. Sample code: #method_removed Ruby Features: method hooks
  • 27. Sample code: #method_undefined Ruby Features: method hooks
  • 28. Use case in finalist (Simplified sample) Ruby Features: method hooks
  • 29. call method_added -> call verify_final_method
  • 30. Usage: Method Hook ▸ Method Hook provides a way to implement declarations (like protected, private) ▸ With class inheritance, Method Hook seems magic !! ▸ But it is not enough to implement “finalist” actually ▸ Ruby has so many cases of method definition ▸ `def` or `define_method` ▸ `include` module ▸ `extend`, `prepend ▸ Each case calls different hooks Ruby Features: method hooks
  • 31. “include” does not add method ▸ Because `include` insert module to method lookup hierarchy, it does not touch method entry table. Ruby Features: method hooks
  • 32. “include” changes only chain of method discovering Foo Object Bar Insert module to hierarchy It is different from to add method Class#ancestors displays class-module hierarchy Ruby Features: method hooks
  • 33. Hooks in finalist gem ▸ `method_added` to detect override by subclass ▸ `singleton_method_added` to detect override class method ▸ `included` to detect override by module include ▸ `extended` to detect override by module extend
  • 34. And I talk about TracePoint too
  • 35. overrider and abstriker uses TracePoint ▸ `inherited` and `included` to start `TracePoint`` call main logic call main logic Tracepoint in overrider, abstriker
  • 36. Tracepoint in overrider, abstriker Check abstract method existence in class hierarchy
  • 37. Why use TracePoint? ▸ In order to verify method existence at the end of class definition. ▸ Ruby interpreter needs to wait until the end of class definition to know a method absence. ▸ override and abstract cannot detect violation just when they are called. ▸ In ruby, The only way to detect the violation is `TracePoint`.
  • 38. Advanced TracePoint: Detect particular class end Advanced Tracepoint :end event cannot trace definition by `Class.new`. Use :c_return and return_value to detect particular class end
  • 39. Advanced TracePoint: Ripper combination Advanced Tracepoint Detect target Sexp node by TracePoint#linen Sexp node type expresses the style of method cal
  • 40. Ripper empowers TracePoint ▸ `Ripper.sexp` returns token position ▸ And TracePoint returns lineno an event occurs. ▸ Ripper provides more information about event context. ▸ ex. power_assert gem is implemented by this combination. Advanced Tracepoint
  • 41. Hacks 2 (binding_ninja) ▸ https://github.com/joker1007/binding_ninja ▸ It is method wrapper to pass binding of method caller implicitly. ▸ I talked about it at Rubykaigi 2017 LT. Hacks: binding_ninja
  • 42. Sample code: binding_ninja Hacks: binding_ninja Pass binding of caller implicitly
  • 43. How To Implement ▸ Binding is based on Ruby level control frame. ▸ When Ruby calls c-method, Ruby does not change Ruby level control frame. ▸ It is core logic of binding_ninja create binding in C level, binding keeps caller stack in Ruby level Hacks: binding_ninja
  • 44. Use case: binding_ninja ▸ I implement scala like implicit parameter in Ruby. ▸ https://github.com/ joker1007/ implicit_parameter Hacks: binding_ninja
  • 45. Hacks: binding_ninja Wrap method by `define_method` And wrap method by binding_ninja
  • 46. Black Magic is dangerous actually, but it is very fun, and it extends Ruby potential These gems is for proof of concept. But these work and decent practical.
  • 47. Ruby Quiz Break (25-30min) What the difference between: - #undef_method - #remove_method
  • 48. RUBY QUIZ class Foo def foo class Foo def foo class Bar < Foo def foo class Bar < Foo def foo Bar.new.foo()
  • 49. RUBY QUIZ class Foo def foo class Foo def foo class Bar < Foo def foo class Bar < Foo Bar.new.foo() undef_method(:foo) NoMethodError remove_method(:foo)
  • 50. Hack: with_resources Add "with" Statement in Ruby ▸ Safe resource allocate/release ▸ Ensure to release resources ▸ after a lexical scope ▸ in reverse order of allocation ▸ Idioms used very frequently ▸ Other languages: ▸ Java: try-with-resources ▸ Python: with ▸ C#: using
  • 51. Hack: with_resources Safe Resource Allocation/Release Statement in Ruby ▸ Open method with blocks ▸ File.open(path){|f| ... } ▸ Ruby way (?) ▸ More indentation ▸ Not implemented sometimes
 (e.g., TCPSocket) ▸ Simple begin-ensure ▸ Anomaly cases can't be
 handled
  • 52. Hack: with_resources with_resources.gem ▸ Safe resource allocation ▸ Top level "with" ▸ via Kernel refinement ▸ Resource allocation as lambda ▸ Multi statements to allocate resources ▸ to release first resource
 if second resource allocation raises exception ▸ Block to define scope for resources https://github.com/tagomoris/with_resources
  • 53. Hack: with_resources Implementing "with" in Ruby ▸ TracePoint ▸ "b_return": pass allocated resources to block arguments ▸ "line": identify allocated resources in lambda ▸ Binding ▸ detect newly defined
 local variables
 in allocation lambda ▸ Refinements ▸ introduce "with"
 in top-level without side effects https://github.com/tagomoris/with_resources
  • 54. Hack: deferral Alternative: defer? ▸ Multi-step resource
 allocation in a method ▸ using "with" ▸ Nesting! ▸ not so bad ▸ many nesting looks
 a bit messy :( ▸ Alternative? ▸ "defer" in golang
  • 55. Hack: deferral Making "defer" in Ruby ▸ Block based "defer" ▸ Should work ▸ Requires 1-level
 nesting *always* ▸ Defer.start, end (+ nesting)
 look too much (than golang) ▸ Abnormal case:
 reassigning variables
  • 56. Hack: deferral deferral.gem ▸ Safe resource release ▸ Top level "defer" ▸ via Kernel refinements ▸ Deferred processing to release resources ▸ at the end of scope (method, block) ▸ or exception raised
  • 57. Hack: deferral Implementing "defer" in Ruby ▸ #defer ▸ Enable TracePoint if not yet ▸ Initialize internal stack frame ▸ TracePoint ▸ Monitor method call stack ▸ Get the snapshot of local variables in defer block ▸ Call release blocks at the end of scope ▸ Binding ▸ save/restore local variables of release block ▸ Refinements ▸ introduce "defer" in top-level without side effects stack level 0 stack level 1
  • 58. The Hard Thing in Magical World: Debugging!
  • 59. "Are You Ready? I'm OK." Blono Bucciarati Thank You! @joker1007 & @tagomoris