Caching and data analysis will move your Symfony2 application to the next level

Zzz and bla^3
will move your
to the next level.
???

I have not
invented anything
please
read carefully
the references
not yet 

I’m @liuggio
(aka Giulio De Donato)
I LOVE
and its dark passenger
fabpot

_ _ _ and _ _ _
will move your
to the next level.
???

_ _ _ and _ _ _
will move your
to the next level.
Symfony2
application

proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/photos/455.jpg HTTP/1.1"
200 986 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, like
Gecko) Version/3.1 Safari/525.13" "-"
proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/nav/tab_left_middle.gif
HTTP/1.1" 200 1020 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13
(KHTML, like Gecko) Version/3.1 Safari/525.13" "-"
dyn-530.optus.com.au - - [22/Apr/2009:18:52:51 +1200] "GET /images/photos/5332.jpg HTTP/1.1" 200
244 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322)" "-"
<?xml version="1.0"?>
<catalog>
<book id="bk101">
<author>Gambardella, Matthew</author>
<title>XML Developer's Guide</title>
<genre>Computer</genre>
<price>44.95</price>
<publish_date>2000-10-01</publish_date>dhcp-465.telstra.com.au - - [22/Apr/2009:18:52:51
@charset "UTF-8";html,body,div,span,applet,object,iframe,h1,h2,h3,h4,+1200] "GET /gallery.php?
section=entertainment HTTP/1.0" 200 58636 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en)
AppleWebKit/525.13 (KHTML, like Gecko) Version/3.1 Safari/525.13" "-"
proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /adclick.php?c=LME2 HTTP/1.1"
proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/ad/RH4Y.jpeg HTTP/1.1"
proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /images/favicon.ico HTTP/1.1" 200
2824 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, like
proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /adclick.php?c=GD52 HTTP/1.1"
In your company
every minute
data is generated
a lot of
differenttotally

dhcp-465.telstra.com.au - - [22/Apr/2009:18:52:51 +1200] "GET /gallery.php?section=entertainment
proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /stories.php?date=yesterday
proxy-435.dialup.xtra.co.nz - - [22/Apr/2009:18:52:51 +1200] "GET /latest.php HTTP/1.1" 200 991 "-"
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_4_11; en) AppleWebKit/525.13 (KHTML, like Gecko)
Version/3.1 Safari/525.13" "-"
proxy-622.dsl.clear.net.nz - - [22/Apr/2009:18:52:51 +1200] "GET /latest-news.rss HTTP/1.0" 200 7199
data
- Difficult to read
- Difficulto to measure
- Time consuming

data
Difference audences and destination
-business
-applications
-external services
-error logs ( js error, and application error)
- . . .
sucks

rocks
data
Difference audences and destination
-business
-applications
-external services
-error logs ( js error, and application error)
- . . .

What we do
with data?
_ _ _ _ _ !ignore
track

We track data
_ _ _ _ _ !ignore
then

… still 42.
and
bussiness
value?
and
bussiness
value?
4242
hitchhiker's guide to the galaxy

Our code generates business value when it runs,
not when we write code.
@coda

Store log and error data
Query on past events
Locate the bug
Know Servers/application
status: healthy/resources/
Responding as it should?
Face the reality of my
ecosystem
Understand why and
where the app. is slow
Analyze the behaviour
when it ‘s running
watch data
be alerted
fast website
Needs
data
…
BUSINESS
VALUE

Needs
BUSINESS
VALUE
logging
debugging
monitoring
profiling
metricking
optimizing
Actions
Store log and error data
Query on past events
Locate the bug
Know Servers/application
status: healthy/resources/
Responding as it should?
Face the reality of my
ecosystem
Understand why and
where the app. is slow
Analyze the behaviour
when it ‘s running
watch data
be alerted
fast website
…

BUSINESS
VALUE
logging
debugging
monitoring
profiling
metricking
optimizing
Actions
monolog, Logstash,
Graylog
Sentry / Graylog
Nagios, Cacti …
external service
XhProf
statsd-php-client,
StatsD, Graphite

It's not about the accuracy of estimating it's
about facing the reality of uncertainty.
@jimhighsmith

metrics
http://samate.nist.gov/index.php/Metrics_and_Measures.html
http://nvlpubs.nist.gov/nistpubs/jres/104/6/j46gra.pdf
The IEEE Standard Glossary of Software Engineering Terminology
also defines quality metric as :
(1) A quantitative measure of the degree to which
an item possesses a given quality attribute.
(2) A function whose inputs are software data and whose output is a
single numerical value that can be interpreted as the degree to which
the software possesses a given quality attribute.
* @param array $data
* @return float
function calculateUglyness(array $data);Ninja territory

measuring
http://samate.nist.gov/index.php/Metrics_and_Measures.html
HOW TO MEASURE ANYTHING - FINDING THE VALUE OF “INTANGIBLES” IN BUSINESS
second edition Hubbard
experimentally obtaining
one or more quantity values
that can reasonably be
attributed to a quantity.
reasonably => uncertainty reductionNinja territory

What metrics
means?
DUNNO
Merlin knows Yes Matrix I
saw the film

metrics
Visual tool to
better understand
reality

metrics
align technical and
business vision of
IT.

produces
graphs
DECISIONS
metrics

StatsD simple
— Node.JS daemon. Listens for message over UDP
simple daemon for easy stats aggregation.
Graphite rude <key> <numeric value> <timestamp>
Graphite stores numeric time-series data.
— Real-time graphing system
— Components:
— Carbon stores the data in Graphite’s specialized database.
— The data can then be visualized through graphite
web interface developed with django.
— Whisper allows carbon to write multiple data points.
metrics

Symfony2 integration:
there’s a bundle for it ‘liuggio/StatsDClientBundle’
— Amount of visitor
— Logged vs Anonymous user
— Doctrine collector
— Amount of PHP peak
— Execution Time
— Monolog
— Easily add a new metric:
$this->get(‘stats_d_client.factory')->increment(’upload.file');
metrics

Symfony2 integration PHP Peak
metrics
this is a
‘Timer’

Concepts
— Bucket
— Values
— Flush Interval
metrics
metrics
— Counter
— Gauge
— Timer
— Set
— Sample
Understand StatsD

GAUGES are arbitrary values, which can be recorded.
metrics
$factory = $this->get('stats_d_client.factory');
$data[] = $factory->gauge('cart.item.number', 30);
$this->get('stats_d_client.service’)->send($data);
cart.item.number|38
Result is the last one (per flush time)

COUNTS increment and decrement a value.
metrics
$data[] = $factory->increment('user.visitor');
The value stay in memory until the flush interval.

TIMERS collects numbers.
metrics
$data[] = $factory->timing(’php-peak', 500);
Operations [mean, upper, sum, lower, count, ...]
are already calculated 

Collect from different sources
# echo "deploy.web:1|c" | nc -w 1 -u graphite.com 8125

track
everythings that
moves
can
metrics

you have more
data than you think
and you need less
data than you think
MORELESS

new metrics
— Why do you want to know?
— How much do you know now?
— What is the value to additional
information

The time you spend for metrics has a cost,
and usually is more economical
than many managers assume.
maybe because
they didn’t
measure.
bazinga
metrics

produces
DECISIONS
fast
metricsgood

good and fast
DECISIONS
BUSINESS
help the

and _ _ _
will move your
to the next level.
metrics
???

Optimizing
Performance problems?
Enlarge
your
profiler

Great caching is like great sex.
It hides all the real problems
@vivekhaldar

Wikipedia: “a Cache ( /’kæ∫/ KASH[1]) is a
component that transparently stores data
so that future requests for that data can be
served faster.”
the cache miss
happens once for the
same response.
given the same request the Cache miss happens once.
given the same request the response is not ‘processed’ twice.
A Cache Hit happens if the Cache
contains the answer.
A Cache Miss happens if the Cache
doesn’t contain the answer.
The perfect cache when

— Application caching: app/cache with opcode
— Response caching and HTTP-CACHE
— Doctrine2 Query and Result Caching
— CDN and static files
— Other
CACHE + SF2

Terravison vendor folder is about 340 MB
The app/cache/prod folder is about 3000 files
1. Annotation (Entity/Controller/...)
2. Template
a. name => file
b. twig in php
3. Assetic (definition)
4. Url matcher, Url creator
5. Translations array key=>value
6. Doctrine Entity Proxy
./app/cache

Add your class to the services with the tag kernel.cache_warmer
Note: the CacheWarmerInterface is not properly the same as the snippet.
cache warmer
namespace SymfonyComponentHttpKernelCacheWarmer;
interface CacheWarmerInterface
/**
* Warms up the cache.
* @param string $cacheDir The cache directory
*/
public function warmUp($cacheDir);
public function isOptional()

Symfony2 is a web framework built from scratch around
the http specification
— a resource could be Fresh || Stale
— http-caching only on safe method
— validation || expiration
HTTP

HTTP headers
$response->setPublic() //default is private
$response->setMaxAge(600);
// Same as above but only for shared caches
$response->setSharedMaxAge(600);
$response->setETag(md5($response->getContent()));
$response->setLastModified($datetime);
if ($response->isNotModified($this->getRequest()))
...
$response->setVary(‘Accept-Encoding’);
symfony.com/doc/master/book/http_cache.html

The least
expensive
query is the
query you
never run.

DoctrineCommonCacheCache
(There’s not tagging)
doctrine cache
// Fetches an entry from the cache.
string function fetch($id);
// Tests if an entry exists in the cache.
bool function contains($id);
// Put data into the cache.
bool function save($id, $data, $lifeTime);
// Deletes a cache entry.
bool function delete($id);

MetaData Cache Annotation (OPCODE)
Query Cache DQL parsed into SQL
(OPCODE/REDIS/MEMCACHED)
Result Cache The query result
(REDIS/MEMCACHED/OPCODE)
doctrine caches

doctrine cache
Class ProductRepository
...
$query = $this->createQueryBuilder(‘p’)
->where(‘p.price > :price’)
->setParameter(‘price’, ‘19.99’)
->getQuery();
$products = $query->getResult();
$query->useResultCache(
true,
$lifetime,
__METHOD__ . serialize($query->getParameters())
);
$query->useQueryCache();

There are only two hard things in Computer Science
cache invalidation and naming things
Phil Karlton
There are only two hard things in Computer Science
cache invalidation, naming things and off-by-1
errors.
anonymous

Invalidation
Especially for http, but in general, you should not waste time to
invalidate a value, but you should invest resources to find the right
time of validity.
Cache miss storm
When a resource is no longer available (cache miss), and is required
by many clients simultaneously (storm), a congestion happens.
cache miss storm, cache stampede,
dog-piling, APC Cache slam
cache

Pre-Caching/Cacheback
Sometimes you need to have some data already available and you
don’t want that the first request wait.
Uniqueness of the cache key
Attention to the conflicts in the key definition, always put a prefix or
namespace such as language or role.
Http-Microcaching:
Save all the responses for 1||2 seconds
lot of examples on internet for Nginx
cache

and _ _ _
will move your
to the next level.
???

Performance
tips - rule #1
Mantainibility
win over
Performance

— The user should never wait
slow operation data in a queue (Redis/*MQ)
processed by a consumer (cli + supersivord):
— Image Manipulation
— File Creation (pdf)
— Sending email as spool
— I/O operation
— Use Session with Redis/Memcache
— Caution on what do you do in the loop && nested loop
easy …

… tips
— Must! OPCODE: APC using the apc.stat = 0
— $php composer.phar dump-autoload –optimize
— Twig.c php extension
— Assets
— Minify and combine all of your CSS and JS files
— Run all (or just some) of your CSS or JS files
through some sort of compiler,
such as LESS, SASS or CoffeeScript
— Run image optimizations on your images
http://symfony.com/doc/current/book/performance.html
https://speakerdeck.com/alexandresalome/symfony2-and-performances

the DIC of sf2.3 is lazy
A lazy loading value holder proxy is a virtual proxy that wraps and
lazily initializes a "real" instance of the proxied class.
<service id="foo" class="AcmeFoo" lazy="true" />
When do I use a lazy value holder?
-Your object takes a lot of time and memory to be initialized (with all
dependencies)
-Your object is not always used, and the instantiation overhead can be
avoided
github.com/Ocramius/ProxyManager/blob/master/docs/lazy-loading-value-holder.md
github.com/Ocramius/symfony-docs/blob/master/components/dependency_injection/lazy_services.rst

Cache Driver
# config_prod.yml
services:
cache:
class: DoctrineCommonCacheApcCache #
MemcacheCache
# config_dev.yml
services:
cache:
class: DoctrineCommonCacheArrayCache
if ($fooString = $this->get(‘cache’)->fetch(‘foo’)) {
$foo = unserialize($fooString);
} else {
// do something with $foo
$this->get(‘cache’)->save(‘foo’, serialize($foo),
$lifetime);
}
http://docs.doctrine-project.org/en/latest/reference/caching.html

DOCTRINE
Object relational mapper
(ORM) for PHP that sits on
top of a powerful database
abstraction layer (DBAL)

doctrine >=
2.4
New features:
https://speakerdeck.com/asm89/what-is-new-in-doctrine
it ill be faster ?
NO

doctrine >= 2.3
—2.4 flush multiples entities
$em->flush(array($entityA, $entityB);
—2.4 paginator
—2.4 master slave
—2.4 join to the interface
—2.3 You can override metadata
—2.3 Event listener in a separated class
/**
* @ORMManyToOne(targetEntity="Tvision...EntityStatusInterface")
**/

Optimize Doctrine
http://xkcd.com/1195/

Optimize Doctrine
https://doctrine-orm.readthedocs.org/en/latest/reference/unitofwork.html
step1:
deeply understand UOW
step2:
goto step1

$article = new Article();
$article->setHeadline('I love PHPday');
$em->persist($article); // the entity is managed
$em->flush(); // the entity is persisted
...
$article->setHeadline('I love JSday and PHPday');
$em->flush(); // the Headline is updated
on the persistence layer
$em->persist($article);

$article = $em->find('Article', 1234);
$article->setHeadline(’I love GRUSP');
$article2 = $em->find('Article', 1234);
echo $article2->getHeadline(); // I love GRUSP
if ($article === $article2) ---> true!!

Unit of work
Maintains a list of objects affected by a business transaction and
coordinates the writing out of changes and the resolution of concurrency
problems.
P of EAA page 184 Patterns of Enterprise Application Architecture
by Martin Fowler
UOW = keeps track
changes of Objects
TL;DR

Identity Map
The flush has a cost
— Number of managed Entities
— Tracking policy
— Type of entities
contains all the managed entities
On flush makes a property by property comparison for all the
entities managed.

tracking policy
Doctrine definition:
“is the process of determining what has changed in managed entities
since the last time they were synchronized with the database.”
— implicit tracking (default and simple)
— explicit tracking (suggested for lot of entities managed)
— notify tracking
/**
* @Entity
* @ChangeTrackingPolicy("DEFERRED_EXPLICIT”)
http://docs.doctrine-project.org/en/latest/reference/change-tracking-policies.html
Flush optimization #1

tracking policy
Deferred Implicit
UOW = DB
http://docs.doctrine-project.org/en/latest/reference/change-tracking-policies.html
Deferred Explicit
UOW != DB
— UOW = ’I love PUG Roma’
— DB is not updated.
Need the persist!
// $article is already managed or loaded
// we are changing the Headline
$article->setHeadline('I love PUG Roma');
$em->flush();
echo $article->getHeadline();// is ‘I love PUG Roma’

Read only entity
/**
* @Entity
* @READONLY
*/
Class Status {
$em->getUnitOfWork->markReadOnly($entity);
Temporarily mark entities as read only

Flush only on few
entities
$em->flush($entity)

Reducing the
flush cost
The flush has a cost
— Number of managed Entities (flushing only some)
— Tracking policy (explicit || implicit)
— Type of entities (read-only || temp. read-only)
Recap.

reference
$post = $postRepository->find($postId);
$comment->setPost($post);
...
// $postId comes from somewhere
$comment->setPost(
$em->getReference(‘AcmeBundleEntityPost’, $postId)
);
.

lazy associations
Association are LAZY by default
// Order
/**
* @ManyToOne(targetEntity=”Cart”, cascade={“all”},
fetch=”EAGER”)
*/
private $cart
<?php
$query = $em->createQuery(“SELECT o FROM Order o”);
$query->setFetchMode(“Order”, “Cart”, “LAZY”);
$query->execute();
Change the fetch mode

associations
What’s better than LAZY ?
/*
* @ORMManyToMany(targetEntity=”Trips”,
mappedBy=”rides”,fetch=”EXTRA_LAZY”)
*/
private $trips;
$turin->getTrips()->contains($turinCaselle);
$turin->getTrips()->count();
$turin->getTrips()->slice($offset, $length);

and _ _ _
will move your
to the next level.
Symfony2
application
Optimization

Knowing when optimization is premature
defines the differences the master engineer
and the apprentice
Theo Schlossnagle
Optimization
We’ve heard
this a million
times

When you discover the real problem, you done the half.
Problem are not all equals, and most are illusions.
Pieter Hintjens

Optimization
you are doing it wrong:
— Cache everything
— Guessing / action without measurement

and
will move your
to the next level.
Symfony2
application
Optimization

and
will move your
to the next level.
BUSINESS

and
will move your
to the next level.
metrics

and
will move your
to the next level.
metricsOptimization

and
will move your
to the next level.
metrics
Symfony2
application

Performance optimization could
Metrics should
Rationality must

@liuggio
please read the references

http://docs.doctrine-project.org/
http://nvlpubs.nist.gov/nistpubs/jres/104/6/j46gra.pdf
http://ilia.ws/files/phpuk_bottlenecks.pdf
http://www.slideshare.net/anton_chuvakin/log-mining-beyond-log-analysis
http://pivotallabs.com/139-metrics-metrics-everywhere/
http://www.hubbardresearch.com (HOW TO MEASURE ANYTHING - FINDING THE VALUE OF “INTANGIBLES” )
https://people.mozilla.com/~rmiller/heka-intro-2013-03/#/2
http://tools.ietf.org/html/draft-ietf-httpbis-p6-cache-18
https://developers.google.com/speed/docs/best-practices/caching
http://notmysock.org/blog/php/user-cache-timebomb.html
http://sonata-project.org/blog/2012/5/15/assetic-package-configuration
http://www.slideshare.net/jonkruger/advanced-objectorientedsolid-principles
http://slides.seld.be/?file=2011-10-20+High+Performance+Websites+with+Symfony2.html
http://stackoverflow.com/questions/8893081/how-to-cache-in-symfony-2
http://www.slideshare.net/postwait/scalable-internet-architecture
http://techportal.inviqa.com/2009/12/01/profiling-with-xhprof/
http://www.mysqlperformanceblog.com/2010/09/10/cache-miss-storm
http://www.slideshare.net/iamcal/scalable-web-architectures-common-patterns-ap...
http://blog.servergrove.com/2012/04/18/how-to-create-a-cache-warmer-in-symfony2/
http://www.slideshare.net/jwage/doctrine-intherealworldsf-live2011sanfran
http://www.slideshare.net/quipo/scalable-architectures-taming-the-twitter-firehose
https://github.com/liuggio/StatsDClientBundle
http://www.slideshare.net/quipo/monitoring-at-scale-intuitive-dashboard-design
http://www.slideshare.net/ihower/rails-performance
http://www.infoq.com/presentations/Architecture-Scale-ZeroMQ
https://speakerdeck.com/keithpitt/keith-and-marios-guide-to-fast-websites
http://www.slideshare.net/anton_chuvakin/log-mining-beyond-log-analysis
http://marco-pivetta.com/proxy-pattern-in-php/#/9
http://www.slideshare.net/guilhermeblanco/object-calisthenics-applied-to-php
http://talks.php.net/show/dojo/
https://github.com/symfony/symfony/pull/7818
https://speakerdeck.com/roidrage/metrics-monitoring-logging
http://codahale.com/codeconf-2011-04-09-metrics-metrics-everywhere.pdf
http://pivotallabs.com/139-metrics-metrics-everywhere/
http://blog.pkhamre.com/2012/07/24/understanding-statsd-and-graphite/
References

Caching and data analysis will move your Symfony2 application to the next level

Recommended

Recommended

More Related Content

Similar to Caching and data analysis will move your Symfony2 application to the next level

Similar to Caching and data analysis will move your Symfony2 application to the next level (20)

More from Giulio De Donato

More from Giulio De Donato (11)

Recently uploaded

Recently uploaded (20)

Caching and data analysis will move your Symfony2 application to the next level

Editor's Notes