SlideShare a Scribd company logo
1 of 65
Download to read offline
DEBUGGING
R U L E S A N D TO O L S
ian barber - http://phpir.com - @ianbarber
I did X
I wanted Y
Instead I got Z
http://www.debuggingrules.com
RULE 1
UNDERSTAND
THE SYSTEM
Bug
Wiki       Tracker   Frame
                     work




          RTFM       PHP
                     Doc



php.net
$field = $document->getField($fieldName);

if ($field->storeTermVector) {
  /**
   * @todo term vector storing support
   */
   require_once
         'Zend/Search/Lucene/Exception.php';
   throw new Zend_Search_Lucene_Exception
   ('Store term vector functionality is not
      supported yet.');
}
Komodo



Eclipse             Zend
                    Studio
          TOOLS

 PHP                 Net
Storm      Text     Beans
           Mate
RULE 2
MAKE IT
FAIL
Recreate And
Automate
PHP Unit
SimpleTest
Selenium
PHP Slim
JMeter
Ab
Time
    Load                 Timing

                                   User

Server
              CONTROL
 Env
              VARIABLES
                                  Request

     Client               DB
                         State
               Session
[root@localhost ~]# pear channel-discover pear.phpunit.de
[root@localhost ~]# pear install phpunit/PHPUnit

class Test_User extends
               PHPUnit_Framework_TestCase {
 public function testUpdate() {
   $r = ("a", "b");
   try {
     $u = User::fetch($this->newName);
     $result = $u->update($r);
   } catch (Id_Exception $e) {
     $this->fail($e->getMessage());
   }
   $this->assertEquals($r, $u->getRoles());
 }
}
http://seleniumhq.org/
class Example extends
           PHPUnit_Extensions_SeleniumTestCase {
  protected function setUp() {
    $this->setBrowser("*chrome");
    $this->setBrowserUrl("http://general.dev/");
  }

    public function testMyTestCase() {
      $this->open("/");
      $this->select("locale", "label=de");
      $this->waitForPageToLoad("30000");
      $this->assertTrue($this->isTextPresent(
                        "Datenbank-Einrichtung"));
    }
}
http://jakarta.apache.org/jmeter/
RULE 3
QUIT THINKING
AND LOOK
if($this->hasRoles) {
  $this->deleteRoles();
}

try {
  foreach($submittedRoles as $role) {
    $this->addRole($role);
  }
} catch(Identity_Exception $e) {
  $this->log->warn('Role add failed.');
  throw new Service_Exception_BadRequest(
                   'Cannot add user role');
}
Instrumentation:
Inspect or Report
Xdebug
      http://www.xdebug.org

[root@localhost ~]# pecl install xdebug

zend_extension=/usr/lib64/php/modules/xdebug.so
xdebug.profiler_enable=1
xdebug.profiler_output_dir=/tmp
Web Server
                                        PHP Script

Debugging Client                         XDebug



                   Debugging Commands
xdebug.remote_enable=1
xdebug.remote_port=9000
xdebug.remote_mode=req
xdebug.remote_host=192.168.192.1 # xdebug.remote_connect_back=1
[root@localhost ~]# lsof -p 4365
COMMAND   USER     FD    TYPE   DEVICE   NAME
httpd     apache   mem   REG    253,0    /lib64/libssl.so.0.9.8e
httpd     apache   mem   REG    253,0    /usr/lib64/php/modules/xdebug.so
httpd     apache   mem   REG    253,0    /usr/lib64/libsqlite3.so.0.8.6




[root@localhost ~]# strace -cp 4365
Process 6910 attached - interrupt to quit
Process 6910 detached
% time     seconds usecs/call      calls    errors           syscall
------ ----------- ----------- --------- ---------           -------------
 36.91    0.037307         466        80                     getdents
 30.07    0.030393         950        32         7           access
 17.04    0.017224         154       112        10           open
  5.08    0.005136          12       444         9           lstat
  3.87    0.003910          18       213         1           read
  3.04    0.003071          29       107                     close
  2.57    0.002598        1299         2                     writev
  0.78    0.000785           1       631                     stat
[root@localhost ~]# strace -Tp 4365
Process 6910 attached - interrupt to quit
epoll_wait(16, {{EPOLLIN, {u32=31389672, u64=47128707528680}}},
      2, 10000) = 1 <6.926140>
accept(4, {sa_family=AF_INET6, sin6_port=htons(64930), inet_pton
      (AF_INET6, "::ffff:192.168.192.1", &sin6_addr),
      sin6_flowinfo=0, sin6_scope_id=0}, [47128676139036]) = 17
      <0.000094>
fcntl(17, F_SETFL, O_RDWR|O_NONBLOCK)   = 0 <0.000035>
read(17,
  "POST / HTTP/1.1rnHost: general.d"...,
   8000) = 522 <0.000044>
stat("/mnt/hgfs/habari/htdocs/", {st_mode=S_IFDIR|0755,
      st_size=408, ...}) = 0 <0.000527>
open("/mnt/hgfs/habari/htdocs/.htaccess",
   O_RDONLY) = 18 <0.000457>
fcntl(17, F_SETFL, O_RDWR|O_NONBLOCK)   = 0 <0.000037>
[root@localhost ~]# strace -Tp 4365
Process 6910 attached - interrupt to quit
epoll_wait(16, {{EPOLLIN, {u32=31389672, u64=47128707528680}}},
      2, 10000) = 1 <6.926140>
accept(4, {sa_family=AF_INET6, sin6_port=htons(64930), inet_pton
      (AF_INET6, "::ffff:192.168.192.1", &sin6_addr),
      sin6_flowinfo=0, sin6_scope_id=0}, [47128676139036]) = 17
      <0.000094>
fcntl(17, F_SETFL, O_RDWR|O_NONBLOCK)   = 0 <0.000035>
read(17,
   "POST / HTTP/1.1rnHost: general.d"...,
    8000) = 522 <0.000044>
stat("/mnt/hgfs/habari/htdocs/", {st_mode=S_IFDIR|0755,
      st_size=408, ...}) = 0 <0.000527>
open("/mnt/hgfs/habari/htdocs/.htaccess",
    O_RDONLY) = 18 <0.000457>
Zend                    Drupal
    Log                    syslog
              sfLogger
                                    Kohana
                                     Log


error_log("Message for the PHP log")
error_log("Log to email", 1, "me@foo.com");
error_log("Log to a file",3, "/tmp/m.log");
error_log("Message for the SAPI log", 4);
MQ           Syslog   File



          LOGGING
          SYSTEM

Gearman     Database   Pipe
FirePHP
[root@localhost ~]# pear channel-discover pear.firephp.org
[root@localhost ~]# pear install firephp/FirePHPCore

require_once('FirePHPCore/FirePHP.class.php');
ob_start();
$firephp = FirePHP::getInstance(true);
doStuff($firephp);
$firephp->warn('Done Stuff');
$firephp->error('An error!');
$firephp->log($_GET);
ob_end_flush();

function doStuff($firephp) {
    $firephp->trace('Stuff Backtrace');
    echo "This page intentionally blank";
}
RULE 4
DIVIDE AND
CONQUER
<?xml version="1.0" encoding="UTF-8"?>
<phpunit bootstrap="./TestHelper.php"
colors="true">
 <testsuite name="Commonlib Tests">
  <directory>./library/Cl/</directory>
  <directory>./library/Sso/</directory>
 </testsuite>
<filter>
 <whitelist>
  <directory suffix=".php">../Sso/</directory>
  <directory suffix=".php">../Cl/</directory>
</whitelist>
</filter>
</phpunit>
Folder Root

          CL          SSO


      Auth       Client      Model    User


Adapter      Plugin   Exception Abstract


SSO          SFDC         Gearman
<?php

$worker = new GearmanWorker();
$worker->addServer();
$worker->addFunction(
            "allow_access", "allowAccess");
$worker->addFunction(
            "deny_access", "denyAccess");

while ($worker->work());
Git Bisect
git bisect start
git bisect bad
git bisect good <known good rev>

git bisect ?

git bisect reset
good                                      bad
                          1
 Request                                       View
                              2

  Front                           3
Controller                            Plugin


  Input                       Model
  Filter

             Controller
Firewall


  Cache


Web Server


 Database
Tamper
                          Data
   Curl


             Firewall
Wireshark
              Cache       Netcat


            Web Server
MySQL
Proxy                    Xdebug
             Database
Tamper Data
https://addons.mozilla.org/firefox/addon/966
Wireshark
http://www.wireshark.org/
[root@localhost ~]# { echo -ne "HTTP/1.0 200 OKrn
rn"; cat test.html; } | nc -l 8080
GET / HTTP/1.1
User-Agent: curl/7.19.7 (i386-apple-darwin10.0.0)
libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
Host: general.dev:8080
Accept: */*



[root@localhost ~]# curl http://general.dev:8080 -v
* About to connect() to general.dev port 8080 (#0)
*   Trying 192.168.192.129... connected
* Connected to general.dev (192.168.192.129) port
8080 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (i386-apple-darwin10.0.0)
libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
> Host: general.dev:8080
local log_file = 'mysql.log'
local fh = io.open(log_file, "a+")

function read_query( packet )
  if string.byte(packet) == proxy.COM_QUERY
  then
    local query = string.sub(packet, 2)
    fh:write( string.format("%sn",query))
    fh:flush()
  end
end


     MySQL Proxy
RULE 5
CHANGE ONE
THING AT
A TIME
[root@localhost ~]#: svn st
M       index.php


[root@localhost ~]#: git stash save
Saved working directory and index state WIP
on master: 0e22fdd Initial checkin
HEAD is now at 0e22fdd Initial checkin

[root@localhost ~]#: git status
# On branch master
nothing to commit (working directory clean)



      Source Control
[root@localhost ~]#: svn log .
-------------------------------------------
r4341 | rickc | 2010-09-24 03:01 | 3 lines

Don't test if Post::tags is empty before
getting tags. There may be a performance
hit for this. It can be dealt with if it is
noticable. Fixes ticket #1235. Props to ilo
for the patch.

 Note that Post still needs to be updated to
 use real Tags, not an array of slug/label
 pairs as the tags member.
--------------------------------------------

      Source Control
Known Good
[root@localhost ~]# diff -yw web.conf web2.conf
<VirtualHost *:80>           <VirtualHost *:80>
  ServerAdmin root@e.com      ServerAdmin root@e.com
  DocumentRoot /www/htdocs |   DocumentRoot /www/hdocs
  ServerName general.dev       ServerName general.dev
  ErrorLog logs/error_log      ErrorLog logs/error_log
</VirtualHost>               </VirtualHost>
RULE 6
KEEP AN
AUDIT TRAIL
Hot Key Log
http://www.blacktree.com/
http://do.davebsd.com/
https://launchpad.net/mb
http://simplenoteapp.com/
Backpack Journal
http://backpackit.com/
[root@localhost htdocs]# history 20
  365 ls /mnt/hgfs/habari/
  366 ls
  367 vi /etc/httpd/conf.d/web.conf
  368 setenforce 0
  369 /etc/init.d/httpd restart
  370 vi /var/log/httpd/error_log
  371 cd /mnt/hgfs/habari/
  372 ls
  373 cd htdocs/
  374 ls
  375 cd scripts/
  376 ls
  377 cd ..
  378 ls
  379 vi /etc/php.ini
RULE 7
CHECK THE
PLUG
Sources of Plugs
[root@localhost ~]# apachectl -M
Loaded Modules:
 core_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
          [root@localhost ~]# whoami
 auth_basic_module (shared)
           root
 auth_digest_module (shared)
          [root@localhost ~]# ifconfig
 authn_file_module (shared)
           eth0       Link encap:Ethernet HWad
 authn_alias_module (shared)
 authn_anon_module inet addr:192.168.192.129
                    (shared)
                   inet6 addr: fe80::20c:29ff:
authn_dbm_module (shared)
RULE 8
GET A FRESH
VIEW
“Grovelling is not
a substitute for
doing your
homework”
- ESR
http://www.catb.org/esr/faqs/smart-questions.html
RULE 9
IF YOU DIDN’T
FIX IT -
IT AIN’T FIXED
Five Whys
           The page is blank: Why?
      There is a fatal error: Why?
No mb_check_encoding(): Why?
   mbstring is not installed: Why?
   It isn’t part of the build: Why?
It was setup for another project.
The Rules
1. Understand The System
2. Make It Fail
3. Quit Thinking And Look
4. Divide And Conquer
5. Change One Thing At A Time
6. Keep An Audit Trail
7. Check The Plug
8. Get A Fresh View
9. If You Didn’t Fix It, It Ain’t Fixed
Questions

            ?
Image Credits
When men were men and wore white coats - by Jitze
http://www.flickr.com/photos/jitze1942/4292084185/
Vacuum Tube Etch-A-Sketch - by Jurvetson
http://www.flickr.com/photos/jurvetson/197768962/
Classic - by Oskay
http://www.flickr.com/photos/oskay/1364147095
Firmware Bug - by Oskay
http://www.flickr.com/photos/oskay/1364148351
Ready to hop - by Oskay
http://www.flickr.com/photos/oskay/1364153441/
Albino - by Oskay
http://www.flickr.com/photos/oskay/1364143833
Coming Through! - by Oskay
http://www.flickr.com/photos/oskay/1364147807
Tux the Pinguin - by Patrick van der Velden
From the photographer’s private collection
THANKANDTOOLS
DEBUGGING: RULES
                 YOU
   http://joind.in/3393

More Related Content

What's hot

Introdução ao Perl 6
Introdução ao Perl 6Introdução ao Perl 6
Introdução ao Perl 6garux
 
Security Meetup 22 октября. «Реверс-инжиниринг в Enterprise». Алексей Секрето...
Security Meetup 22 октября. «Реверс-инжиниринг в Enterprise». Алексей Секрето...Security Meetup 22 октября. «Реверс-инжиниринг в Enterprise». Алексей Секрето...
Security Meetup 22 октября. «Реверс-инжиниринг в Enterprise». Алексей Секрето...Mail.ru Group
 
News of the Symfony2 World
News of the Symfony2 WorldNews of the Symfony2 World
News of the Symfony2 WorldFabien Potencier
 
Advanced modulinos
Advanced modulinosAdvanced modulinos
Advanced modulinosbrian d foy
 
Créer une base NoSQL en 1 heure
Créer une base NoSQL en 1 heureCréer une base NoSQL en 1 heure
Créer une base NoSQL en 1 heureAmaury Bouchard
 
Xlab #1: Advantages of functional programming in Java 8
Xlab #1: Advantages of functional programming in Java 8Xlab #1: Advantages of functional programming in Java 8
Xlab #1: Advantages of functional programming in Java 8XSolve
 
The Magic Of Tie
The Magic Of TieThe Magic Of Tie
The Magic Of Tiebrian d foy
 
Advanced modulinos trial
Advanced modulinos trialAdvanced modulinos trial
Advanced modulinos trialbrian d foy
 
20 modules i haven't yet talked about
20 modules i haven't yet talked about20 modules i haven't yet talked about
20 modules i haven't yet talked aboutTatsuhiko Miyagawa
 
Introduction to CloudForecast / YAPC::Asia 2010 Tokyo
Introduction to CloudForecast / YAPC::Asia 2010 TokyoIntroduction to CloudForecast / YAPC::Asia 2010 Tokyo
Introduction to CloudForecast / YAPC::Asia 2010 TokyoMasahiro Nagano
 

What's hot (17)

Introdução ao Perl 6
Introdução ao Perl 6Introdução ao Perl 6
Introdução ao Perl 6
 
zinno
zinnozinno
zinno
 
Security Meetup 22 октября. «Реверс-инжиниринг в Enterprise». Алексей Секрето...
Security Meetup 22 октября. «Реверс-инжиниринг в Enterprise». Алексей Секрето...Security Meetup 22 октября. «Реверс-инжиниринг в Enterprise». Алексей Секрето...
Security Meetup 22 октября. «Реверс-инжиниринг в Enterprise». Алексей Секрето...
 
News of the Symfony2 World
News of the Symfony2 WorldNews of the Symfony2 World
News of the Symfony2 World
 
C99
C99C99
C99
 
Advanced modulinos
Advanced modulinosAdvanced modulinos
Advanced modulinos
 
Créer une base NoSQL en 1 heure
Créer une base NoSQL en 1 heureCréer une base NoSQL en 1 heure
Créer une base NoSQL en 1 heure
 
Xlab #1: Advantages of functional programming in Java 8
Xlab #1: Advantages of functional programming in Java 8Xlab #1: Advantages of functional programming in Java 8
Xlab #1: Advantages of functional programming in Java 8
 
C99[2]
C99[2]C99[2]
C99[2]
 
Php 101: PDO
Php 101: PDOPhp 101: PDO
Php 101: PDO
 
The Magic Of Tie
The Magic Of TieThe Magic Of Tie
The Magic Of Tie
 
Advanced modulinos trial
Advanced modulinos trialAdvanced modulinos trial
Advanced modulinos trial
 
Perl 6 by example
Perl 6 by examplePerl 6 by example
Perl 6 by example
 
20 modules i haven't yet talked about
20 modules i haven't yet talked about20 modules i haven't yet talked about
20 modules i haven't yet talked about
 
Cod
CodCod
Cod
 
Introduction to CloudForecast / YAPC::Asia 2010 Tokyo
Introduction to CloudForecast / YAPC::Asia 2010 TokyoIntroduction to CloudForecast / YAPC::Asia 2010 Tokyo
Introduction to CloudForecast / YAPC::Asia 2010 Tokyo
 
C99
C99C99
C99
 

Similar to DEBUGGING RULES AND TOOLS

Debugging: Rules & Tools
Debugging: Rules & ToolsDebugging: Rules & Tools
Debugging: Rules & ToolsIan Barber
 
Anatomy of a PHP Request ( UTOSC 2010 )
Anatomy of a PHP Request ( UTOSC 2010 )Anatomy of a PHP Request ( UTOSC 2010 )
Anatomy of a PHP Request ( UTOSC 2010 )Joseph Scott
 
Using ngx_lua in UPYUN
Using ngx_lua in UPYUNUsing ngx_lua in UPYUN
Using ngx_lua in UPYUNCong Zhang
 
Heavy Web Optimization: Backend
Heavy Web Optimization: BackendHeavy Web Optimization: Backend
Heavy Web Optimization: BackendVõ Duy Tuấn
 
The why and how of moving to PHP 5.4/5.5
The why and how of moving to PHP 5.4/5.5The why and how of moving to PHP 5.4/5.5
The why and how of moving to PHP 5.4/5.5Wim Godden
 
Facebook的缓存系统
Facebook的缓存系统Facebook的缓存系统
Facebook的缓存系统yiditushe
 
関西PHP勉強会 php5.4つまみぐい
関西PHP勉強会 php5.4つまみぐい関西PHP勉強会 php5.4つまみぐい
関西PHP勉強会 php5.4つまみぐいHisateru Tanaka
 
4069180 Caching Performance Lessons From Facebook
4069180 Caching Performance Lessons From Facebook4069180 Caching Performance Lessons From Facebook
4069180 Caching Performance Lessons From Facebookguoqing75
 
X64服务器 lnmp服务器部署标准 new
X64服务器 lnmp服务器部署标准 newX64服务器 lnmp服务器部署标准 new
X64服务器 lnmp服务器部署标准 newYiwei Ma
 
An introduction to PHP 5.4
An introduction to PHP 5.4An introduction to PHP 5.4
An introduction to PHP 5.4Giovanni Derks
 
Ch ch-changes cake php2
Ch ch-changes cake php2Ch ch-changes cake php2
Ch ch-changes cake php2markstory
 
Relayd: a load balancer for OpenBSD
Relayd: a load balancer for OpenBSD Relayd: a load balancer for OpenBSD
Relayd: a load balancer for OpenBSD Giovanni Bechis
 
PHP Backdoor: The rise of the vuln
PHP Backdoor: The rise of the vulnPHP Backdoor: The rise of the vuln
PHP Backdoor: The rise of the vulnSandro Zaccarini
 
Vladimir Vorontsov - Splitting, smuggling and cache poisoning come back
Vladimir Vorontsov - Splitting, smuggling and cache poisoning come backVladimir Vorontsov - Splitting, smuggling and cache poisoning come back
Vladimir Vorontsov - Splitting, smuggling and cache poisoning come backDefconRussia
 
The why and how of moving to php 5.4/5.5
The why and how of moving to php 5.4/5.5The why and how of moving to php 5.4/5.5
The why and how of moving to php 5.4/5.5Wim Godden
 
Getting started with TDD - Confoo 2014
Getting started with TDD - Confoo 2014Getting started with TDD - Confoo 2014
Getting started with TDD - Confoo 2014Eric Hogue
 
Tips
TipsTips
Tipsmclee
 
How to Design a Great API (using flask) [ploneconf2017]
How to Design a Great API (using flask) [ploneconf2017]How to Design a Great API (using flask) [ploneconf2017]
How to Design a Great API (using flask) [ploneconf2017]Devon Bernard
 

Similar to DEBUGGING RULES AND TOOLS (20)

Debugging: Rules & Tools
Debugging: Rules & ToolsDebugging: Rules & Tools
Debugging: Rules & Tools
 
Anatomy of a PHP Request ( UTOSC 2010 )
Anatomy of a PHP Request ( UTOSC 2010 )Anatomy of a PHP Request ( UTOSC 2010 )
Anatomy of a PHP Request ( UTOSC 2010 )
 
Using ngx_lua in UPYUN
Using ngx_lua in UPYUNUsing ngx_lua in UPYUN
Using ngx_lua in UPYUN
 
Heavy Web Optimization: Backend
Heavy Web Optimization: BackendHeavy Web Optimization: Backend
Heavy Web Optimization: Backend
 
The why and how of moving to PHP 5.4/5.5
The why and how of moving to PHP 5.4/5.5The why and how of moving to PHP 5.4/5.5
The why and how of moving to PHP 5.4/5.5
 
Facebook的缓存系统
Facebook的缓存系统Facebook的缓存系统
Facebook的缓存系统
 
関西PHP勉強会 php5.4つまみぐい
関西PHP勉強会 php5.4つまみぐい関西PHP勉強会 php5.4つまみぐい
関西PHP勉強会 php5.4つまみぐい
 
4069180 Caching Performance Lessons From Facebook
4069180 Caching Performance Lessons From Facebook4069180 Caching Performance Lessons From Facebook
4069180 Caching Performance Lessons From Facebook
 
X64服务器 lnmp服务器部署标准 new
X64服务器 lnmp服务器部署标准 newX64服务器 lnmp服务器部署标准 new
X64服务器 lnmp服务器部署标准 new
 
An introduction to PHP 5.4
An introduction to PHP 5.4An introduction to PHP 5.4
An introduction to PHP 5.4
 
Ch ch-changes cake php2
Ch ch-changes cake php2Ch ch-changes cake php2
Ch ch-changes cake php2
 
Relayd: a load balancer for OpenBSD
Relayd: a load balancer for OpenBSD Relayd: a load balancer for OpenBSD
Relayd: a load balancer for OpenBSD
 
PHP Backdoor: The rise of the vuln
PHP Backdoor: The rise of the vulnPHP Backdoor: The rise of the vuln
PHP Backdoor: The rise of the vuln
 
Vladimir Vorontsov - Splitting, smuggling and cache poisoning come back
Vladimir Vorontsov - Splitting, smuggling and cache poisoning come backVladimir Vorontsov - Splitting, smuggling and cache poisoning come back
Vladimir Vorontsov - Splitting, smuggling and cache poisoning come back
 
The why and how of moving to php 5.4/5.5
The why and how of moving to php 5.4/5.5The why and how of moving to php 5.4/5.5
The why and how of moving to php 5.4/5.5
 
Puppet Camp 2012
Puppet Camp 2012Puppet Camp 2012
Puppet Camp 2012
 
Php version 5
Php version 5Php version 5
Php version 5
 
Getting started with TDD - Confoo 2014
Getting started with TDD - Confoo 2014Getting started with TDD - Confoo 2014
Getting started with TDD - Confoo 2014
 
Tips
TipsTips
Tips
 
How to Design a Great API (using flask) [ploneconf2017]
How to Design a Great API (using flask) [ploneconf2017]How to Design a Great API (using flask) [ploneconf2017]
How to Design a Great API (using flask) [ploneconf2017]
 

More from Ian Barber

ZeroMQ Is The Answer: PHP Tek 11 Version
ZeroMQ Is The Answer: PHP Tek 11 VersionZeroMQ Is The Answer: PHP Tek 11 Version
ZeroMQ Is The Answer: PHP Tek 11 VersionIan Barber
 
Deployment Tactics
Deployment TacticsDeployment Tactics
Deployment TacticsIan Barber
 
In Search Of: Integrating Site Search (PHP Barcelona)
In Search Of: Integrating Site Search (PHP Barcelona)In Search Of: Integrating Site Search (PHP Barcelona)
In Search Of: Integrating Site Search (PHP Barcelona)Ian Barber
 
In Search Of... (Dutch PHP Conference 2010)
In Search Of... (Dutch PHP Conference 2010)In Search Of... (Dutch PHP Conference 2010)
In Search Of... (Dutch PHP Conference 2010)Ian Barber
 
In Search Of... integrating site search
In Search Of... integrating site search In Search Of... integrating site search
In Search Of... integrating site search Ian Barber
 
Document Classification In PHP - Slight Return
Document Classification In PHP - Slight ReturnDocument Classification In PHP - Slight Return
Document Classification In PHP - Slight ReturnIan Barber
 
Document Classification In PHP
Document Classification In PHPDocument Classification In PHP
Document Classification In PHPIan Barber
 

More from Ian Barber (7)

ZeroMQ Is The Answer: PHP Tek 11 Version
ZeroMQ Is The Answer: PHP Tek 11 VersionZeroMQ Is The Answer: PHP Tek 11 Version
ZeroMQ Is The Answer: PHP Tek 11 Version
 
Deployment Tactics
Deployment TacticsDeployment Tactics
Deployment Tactics
 
In Search Of: Integrating Site Search (PHP Barcelona)
In Search Of: Integrating Site Search (PHP Barcelona)In Search Of: Integrating Site Search (PHP Barcelona)
In Search Of: Integrating Site Search (PHP Barcelona)
 
In Search Of... (Dutch PHP Conference 2010)
In Search Of... (Dutch PHP Conference 2010)In Search Of... (Dutch PHP Conference 2010)
In Search Of... (Dutch PHP Conference 2010)
 
In Search Of... integrating site search
In Search Of... integrating site search In Search Of... integrating site search
In Search Of... integrating site search
 
Document Classification In PHP - Slight Return
Document Classification In PHP - Slight ReturnDocument Classification In PHP - Slight Return
Document Classification In PHP - Slight Return
 
Document Classification In PHP
Document Classification In PHPDocument Classification In PHP
Document Classification In PHP
 

Recently uploaded

The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPathCommunity
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructureitnewsafrica
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Nikki Chapple
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observabilityitnewsafrica
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationKnoldus Inc.
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 

Recently uploaded (20)

The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to Hero
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog Presentation
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 

DEBUGGING RULES AND TOOLS

  • 1. DEBUGGING R U L E S A N D TO O L S ian barber - http://phpir.com - @ianbarber
  • 2.
  • 3. I did X I wanted Y Instead I got Z
  • 6. Bug Wiki Tracker Frame work RTFM PHP Doc php.net
  • 7. $field = $document->getField($fieldName); if ($field->storeTermVector) { /** * @todo term vector storing support */ require_once 'Zend/Search/Lucene/Exception.php'; throw new Zend_Search_Lucene_Exception ('Store term vector functionality is not supported yet.'); }
  • 8. Komodo Eclipse Zend Studio TOOLS PHP Net Storm Text Beans Mate
  • 11. Time Load Timing User Server CONTROL Env VARIABLES Request Client DB State Session
  • 12. [root@localhost ~]# pear channel-discover pear.phpunit.de [root@localhost ~]# pear install phpunit/PHPUnit class Test_User extends PHPUnit_Framework_TestCase { public function testUpdate() { $r = ("a", "b"); try { $u = User::fetch($this->newName); $result = $u->update($r); } catch (Id_Exception $e) { $this->fail($e->getMessage()); } $this->assertEquals($r, $u->getRoles()); } }
  • 14. class Example extends PHPUnit_Extensions_SeleniumTestCase { protected function setUp() { $this->setBrowser("*chrome"); $this->setBrowserUrl("http://general.dev/"); } public function testMyTestCase() { $this->open("/"); $this->select("locale", "label=de"); $this->waitForPageToLoad("30000"); $this->assertTrue($this->isTextPresent( "Datenbank-Einrichtung")); } }
  • 17. if($this->hasRoles) { $this->deleteRoles(); } try { foreach($submittedRoles as $role) { $this->addRole($role); } } catch(Identity_Exception $e) { $this->log->warn('Role add failed.'); throw new Service_Exception_BadRequest( 'Cannot add user role'); }
  • 19. Xdebug http://www.xdebug.org [root@localhost ~]# pecl install xdebug zend_extension=/usr/lib64/php/modules/xdebug.so
  • 21.
  • 22. Web Server PHP Script Debugging Client XDebug Debugging Commands
  • 24.
  • 25.
  • 26. [root@localhost ~]# lsof -p 4365 COMMAND USER FD TYPE DEVICE NAME httpd apache mem REG 253,0 /lib64/libssl.so.0.9.8e httpd apache mem REG 253,0 /usr/lib64/php/modules/xdebug.so httpd apache mem REG 253,0 /usr/lib64/libsqlite3.so.0.8.6 [root@localhost ~]# strace -cp 4365 Process 6910 attached - interrupt to quit Process 6910 detached % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ------------- 36.91 0.037307 466 80 getdents 30.07 0.030393 950 32 7 access 17.04 0.017224 154 112 10 open 5.08 0.005136 12 444 9 lstat 3.87 0.003910 18 213 1 read 3.04 0.003071 29 107 close 2.57 0.002598 1299 2 writev 0.78 0.000785 1 631 stat
  • 27. [root@localhost ~]# strace -Tp 4365 Process 6910 attached - interrupt to quit epoll_wait(16, {{EPOLLIN, {u32=31389672, u64=47128707528680}}}, 2, 10000) = 1 <6.926140> accept(4, {sa_family=AF_INET6, sin6_port=htons(64930), inet_pton (AF_INET6, "::ffff:192.168.192.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [47128676139036]) = 17 <0.000094> fcntl(17, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000035> read(17, "POST / HTTP/1.1rnHost: general.d"..., 8000) = 522 <0.000044> stat("/mnt/hgfs/habari/htdocs/", {st_mode=S_IFDIR|0755, st_size=408, ...}) = 0 <0.000527> open("/mnt/hgfs/habari/htdocs/.htaccess", O_RDONLY) = 18 <0.000457> fcntl(17, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000037>
  • 28. [root@localhost ~]# strace -Tp 4365 Process 6910 attached - interrupt to quit epoll_wait(16, {{EPOLLIN, {u32=31389672, u64=47128707528680}}}, 2, 10000) = 1 <6.926140> accept(4, {sa_family=AF_INET6, sin6_port=htons(64930), inet_pton (AF_INET6, "::ffff:192.168.192.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [47128676139036]) = 17 <0.000094> fcntl(17, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000035> read(17, "POST / HTTP/1.1rnHost: general.d"..., 8000) = 522 <0.000044> stat("/mnt/hgfs/habari/htdocs/", {st_mode=S_IFDIR|0755, st_size=408, ...}) = 0 <0.000527> open("/mnt/hgfs/habari/htdocs/.htaccess", O_RDONLY) = 18 <0.000457>
  • 29. Zend Drupal Log syslog sfLogger Kohana Log error_log("Message for the PHP log") error_log("Log to email", 1, "me@foo.com"); error_log("Log to a file",3, "/tmp/m.log"); error_log("Message for the SAPI log", 4);
  • 30. MQ Syslog File LOGGING SYSTEM Gearman Database Pipe
  • 32. [root@localhost ~]# pear channel-discover pear.firephp.org [root@localhost ~]# pear install firephp/FirePHPCore require_once('FirePHPCore/FirePHP.class.php'); ob_start(); $firephp = FirePHP::getInstance(true); doStuff($firephp); $firephp->warn('Done Stuff'); $firephp->error('An error!'); $firephp->log($_GET); ob_end_flush(); function doStuff($firephp) { $firephp->trace('Stuff Backtrace'); echo "This page intentionally blank"; }
  • 34. <?xml version="1.0" encoding="UTF-8"?> <phpunit bootstrap="./TestHelper.php" colors="true"> <testsuite name="Commonlib Tests"> <directory>./library/Cl/</directory> <directory>./library/Sso/</directory> </testsuite> <filter> <whitelist> <directory suffix=".php">../Sso/</directory> <directory suffix=".php">../Cl/</directory> </whitelist> </filter> </phpunit>
  • 35. Folder Root CL SSO Auth Client Model User Adapter Plugin Exception Abstract SSO SFDC Gearman
  • 36. <?php $worker = new GearmanWorker(); $worker->addServer(); $worker->addFunction( "allow_access", "allowAccess"); $worker->addFunction( "deny_access", "denyAccess"); while ($worker->work());
  • 37. Git Bisect git bisect start git bisect bad git bisect good <known good rev> git bisect ? git bisect reset
  • 38. good bad 1 Request View 2 Front 3 Controller Plugin Input Model Filter Controller
  • 39. Firewall Cache Web Server Database
  • 40. Tamper Data Curl Firewall Wireshark Cache Netcat Web Server MySQL Proxy Xdebug Database
  • 43.
  • 44. [root@localhost ~]# { echo -ne "HTTP/1.0 200 OKrn rn"; cat test.html; } | nc -l 8080 GET / HTTP/1.1 User-Agent: curl/7.19.7 (i386-apple-darwin10.0.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 Host: general.dev:8080 Accept: */* [root@localhost ~]# curl http://general.dev:8080 -v * About to connect() to general.dev port 8080 (#0) * Trying 192.168.192.129... connected * Connected to general.dev (192.168.192.129) port 8080 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.19.7 (i386-apple-darwin10.0.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 > Host: general.dev:8080
  • 45. local log_file = 'mysql.log' local fh = io.open(log_file, "a+") function read_query( packet ) if string.byte(packet) == proxy.COM_QUERY then local query = string.sub(packet, 2) fh:write( string.format("%sn",query)) fh:flush() end end MySQL Proxy
  • 47. [root@localhost ~]#: svn st M index.php [root@localhost ~]#: git stash save Saved working directory and index state WIP on master: 0e22fdd Initial checkin HEAD is now at 0e22fdd Initial checkin [root@localhost ~]#: git status # On branch master nothing to commit (working directory clean) Source Control
  • 48. [root@localhost ~]#: svn log . ------------------------------------------- r4341 | rickc | 2010-09-24 03:01 | 3 lines Don't test if Post::tags is empty before getting tags. There may be a performance hit for this. It can be dealt with if it is noticable. Fixes ticket #1235. Props to ilo for the patch. Note that Post still needs to be updated to use real Tags, not an array of slug/label pairs as the tags member. -------------------------------------------- Source Control
  • 49. Known Good [root@localhost ~]# diff -yw web.conf web2.conf <VirtualHost *:80> <VirtualHost *:80> ServerAdmin root@e.com ServerAdmin root@e.com DocumentRoot /www/htdocs | DocumentRoot /www/hdocs ServerName general.dev ServerName general.dev ErrorLog logs/error_log ErrorLog logs/error_log </VirtualHost> </VirtualHost>
  • 53. [root@localhost htdocs]# history 20 365 ls /mnt/hgfs/habari/ 366 ls 367 vi /etc/httpd/conf.d/web.conf 368 setenforce 0 369 /etc/init.d/httpd restart 370 vi /var/log/httpd/error_log 371 cd /mnt/hgfs/habari/ 372 ls 373 cd htdocs/ 374 ls 375 cd scripts/ 376 ls 377 cd .. 378 ls 379 vi /etc/php.ini
  • 55. Sources of Plugs [root@localhost ~]# apachectl -M Loaded Modules: core_module (static) mpm_prefork_module (static) http_module (static) so_module (static) [root@localhost ~]# whoami auth_basic_module (shared) root auth_digest_module (shared) [root@localhost ~]# ifconfig authn_file_module (shared) eth0 Link encap:Ethernet HWad authn_alias_module (shared) authn_anon_module inet addr:192.168.192.129 (shared) inet6 addr: fe80::20c:29ff: authn_dbm_module (shared)
  • 56. RULE 8 GET A FRESH VIEW
  • 57. “Grovelling is not a substitute for doing your homework” - ESR http://www.catb.org/esr/faqs/smart-questions.html
  • 58.
  • 59.
  • 60. RULE 9 IF YOU DIDN’T FIX IT - IT AIN’T FIXED
  • 61. Five Whys The page is blank: Why? There is a fatal error: Why? No mb_check_encoding(): Why? mbstring is not installed: Why? It isn’t part of the build: Why? It was setup for another project.
  • 62. The Rules 1. Understand The System 2. Make It Fail 3. Quit Thinking And Look 4. Divide And Conquer 5. Change One Thing At A Time 6. Keep An Audit Trail 7. Check The Plug 8. Get A Fresh View 9. If You Didn’t Fix It, It Ain’t Fixed
  • 64. Image Credits When men were men and wore white coats - by Jitze http://www.flickr.com/photos/jitze1942/4292084185/ Vacuum Tube Etch-A-Sketch - by Jurvetson http://www.flickr.com/photos/jurvetson/197768962/ Classic - by Oskay http://www.flickr.com/photos/oskay/1364147095 Firmware Bug - by Oskay http://www.flickr.com/photos/oskay/1364148351 Ready to hop - by Oskay http://www.flickr.com/photos/oskay/1364153441/ Albino - by Oskay http://www.flickr.com/photos/oskay/1364143833 Coming Through! - by Oskay http://www.flickr.com/photos/oskay/1364147807 Tux the Pinguin - by Patrick van der Velden From the photographer’s private collection
  • 65. THANKANDTOOLS DEBUGGING: RULES YOU http://joind.in/3393