40. PEP 369 / Post import hooks
@imp.when_imported('django.core.handlers.wsgi')
def instrument(module):
method = module.WSGIHandler.__call__
module.WSGIHandler.__call__ =
WSGIApplicationWrapper(method)
41. DIY post import hooks (1)
class ImportHookFinder:
def __init__(self):
self._skip = {}
def find_module(self, fullname, path=None):
if not fullname in _import_hooks: return None
if fullname in self._skip: return None
self._skip[fullname] = True
try:
__import__(fullname)
finally:
del self._skip[fullname]
return ImportHookLoader()
42. DIY post import hooks (2)
class ImportHookLoader:
def load_module(self, fullname):
module = sys.modules[fullname]
_notify_import_hooks(fullname, module)
return module
sys.meta_path.insert(0, ImportHookFinder())
63. Enough already
✴Too much information to cover.
✴Is never enough time for it all.
✴So so come and talk to us later.
64. Enough already
✴Too much information to cover.
✴Is never enough time for it all.
✴So so come and talk to us later.
✴Get a demonstration of New Relic.
65. Enough already
✴Too much information to cover.
✴Is never enough time for it all.
✴So so come and talk to us later.
✴Get a demonstration of New Relic.
✴Drinks and nibbles Wednesday evening.
As most would know, I am the author of mod_wsgi. Some time back I started adding instrumentation to mod_wsgi to track time spent in handling requests and number of concurrent requests, so could use it to tune Apache/mod_wsgi. I gave up though, as no good way to display it. Was thus very excited when introduced to New Relic as allowed me to do what I had wanted to do. Decided I had just had to work for them and now I am.\n
The key missing ingredient here I was needing was the web interface that New Relic provides for their existing agents. It can show you what is going on in your web application. It actually has the ability to show much more data than I was intending to capture. More importantly, they position there product as being for your production web application, not only for development or staging. So what are some of the things the New Relic web interface can display?\n
We have throughput.\n
We have response time as recorded by the application.\n
We can see a breakdown of how much time is spent doing different things as part of the request. Queueing time. Database. Memcache. Externals. As well as a rollup of time spent in the rest of the application.\n
We get a record of what exceptions occurred within the application.\n
And we get a sampling of requests which were regarded as being slow.\n
There is the ability to drill down into web transactions and what parts of the application they were handled by.\n
And for the slow web transactions, you can drill down again and get a targeted breakdown of where time was spent for that specific request.\n
You also have a breakdown of your database queries. Which occur the most, which are the slowest, the type of query and so on. This is only a part of the functionality available through the web interface though. There is actually a great deal more than this, but this talk is meant to be about the techniques we used to capture this information in Python web applications so we will move on to that.\n
Summarising what I pointed out in the web interface, the main things we are therefore interested in tracking for web transactions are (ABOVE).\n
Drilling down within a web transaction we are interested in (ABOVE). I will not be able to cover all the work we had to do to capture all this information, but will try and cover some of the more interesting aspects.\n
The key thing we are interested in and which everything else rests on is how long things take. For that we can create a simple trace class which is triggered on entry to a block of code and thence on exit. The duration along with details about what was being traced would then be collected up and stored for later processing.\n
The code block where we wish to record time taken is thence just wrapped with the calls to enter and exit. Problem with this simplistic approach though is if an exception occurs within the code block. As the code is written, if an exception does occur the exit call will be skipped. We therefore have to scope the code block with try/finally.\n
In some cases though we are interested not only in how long something took, we are also interested in what exceptions may have occurred. This is important for Django because Django swallows exceptions and converts them to a 500 error response. They will therefore not propagate up to the top level and we will want to remember them.\n
What we will want to do therefore is add an except clause and also capture the exception details. Having this information within the exception clause where we are wrapping the code block isn't terribly useful though. We really want to pass that off to our trace object instead.\n
What we can do here is overload the use of the exit function and have it accept any exception details with None being used for case where no exception occurred.\n
Our wrapper code is then modified to call exit in both the except and finally clauses. Because finally will be called for successful case and when exception occurs, we do need to protect ourselves to ensure that the exit method isn't called twice. \n
But wait, doesn't this all look sort of familiar. (NEXT) Haven't we just reinvented the wheel and ended up with the pattern which underlies how context managers work.\n
But wait, doesn't this all look sort of familiar. (NEXT) Haven't we just reinvented the wheel and ended up with the pattern which underlies how context managers work.\n
For those not familiar with context managers it is the name used to refer to objects used in conjunction with the Python 'with' statement. For this language construct, the interpreter itself will ensure that enter and exit methods are called for us automatically, including the variation for when the exception occurs.\n
We will need to just make a slight change to our trace object, and that is to rename the enter and exit methods to match the dunderscore names which a context manager must use.\n
One other thing to note about the 'with' statement and context managers is that it does provide for an additional feature that we aren't actually making use of. That is that the exit function of a context manager can choose to swallow the exception by returning a true value. The implementation of 'with' is therefore more like that shown here. (ABOVE).\n
Although you could trace specific blocks of code, usually though we are more interested in tracing time spent in a function as a whole. For this we can introduce a decorator and apply that to the function to be traced. Unfortunately decorators aren't necessarily that straight forward and there are various gotchas you have to be aware of.\n
The first one is stacking of decorators. Take for example the situation of the 'csrf_exempt' decorator in Django. If we wrap our decorator around it, then it doesn't work out how we want it. (NEXT) The problem in this case is that the 'csrf_exempt' decorator ends up ineffectual and no longer does what it is meant to.\n
The reason for this is that is that 'csrf_exempt' decorator adds an attribute to the wrapped function which Django then checks for to determine if CSRF checks are required or not. By wrapping the 'csrf_exempt' decorator we have hidden that attribute. We need therefore to use 'functools.wraps()' in the implementation of the decorator, which copies the attributes up to the level of our decorator.\n
Now, because our decorator is to be used to track time taken, we want it to work in any situation. And for class methods it does work, but we end up with further problems when we start stacking decorators commonly used with methods, in this case with the 'staticmethod' decorator. (NEXT) Here the problem is 'functools.wraps()' which tries to access the '__module__' attribute which doesn't exist for 'staticmethod' decorator objects.\n
This one is actually a bug or shortcoming in Python which is fixed in Python 3.1 or 3.2, with 'wraps' not actually checking whether the attributes it is trying to copy actually exists. So, we need to workaround that and only copy certain attributes if they exist. (NEXT) Fix that and we still have a problem though in that a 'staticmethod' decorator object doesn't appear to be callable.\n
The issue here is that the presence of a '__call__' method isn't the sole arbiter that an object is callable. The use of nested functions to implement a decorator doesn't take that into consideration. We instead need to deal with the case where the object being wrapped may implement a method descriptor via '__get__()' to return the callable object. To do that we need to use a class object for the wrapper which daisy chains '__get__()' from the wrapped object.\n
So we think we now have a way of wrapping functions that will work, what about when we now apply that to a WSGI application. What is going to happen? For the simple case where a list is returned as iterable, it appears to work, but in reality it doesn't count time taken to write the response back to the client, which we don't want to leave out.\n
The case of where 'yield' is being used is even worse. The timing will stop immediately as the function will return a generator object before any code of the function is actually executed. That is, code in the function only gets executed when there is an attempt to retrieve the first item from the generator. So, we came up with this way of wrapping functions only to find it doesn't work for the most important case we have of a WSGI application itself. What do we do?\n
The only choice in this situation is to create a WSGI middleware which is aware of the special way that an iterable/generator can be returned by the WSGI application. We cannot use the 'with' statement. Our wrapper instead needs to explicitly call the enter method before invoking the application. If an error occurs, then we will call exit. Otherwise we return the result from the application in a special iterable object.\n
This iterable object overrides '__iter__' and proxies that through to the iterable/generator that was originally returned from the wrapped WSGI application.\n
The important bit though is that our iterable wrapper provides a 'close()' method. The 'close()' method is special and is a method which when present on the object returned is required to be called by a WSGI adapter or WSGI middleware when the complete response is consumed or the request otherwise ends. In our case we call any 'close()' method on wrapped object and call the exit method on our trace object as appropriate signalling the end of timing.\n
With this problem solved, how then do things look for actually initialising and wrapping a WSGI application when using our actual Python agent? For that we have our special 'wsgi_application()' decorator which would be applied to the WSGI application in the WSGI script file or module. The actual initialisation is done via an explicit initialisation call and passing in the agent configuration file. This in place we will get basic web transaction metrics.\n
What about Django? For Django we still have the initialisation call, but for the case of the WSGI application entry point, although we could have wrapped it, it actually isn't necessary to do so. Only the initilisation is required. So, although I have gone and explained the mechanics of wrapping stuff, we don't want you to have to do that. We want things to be automatic so that the only thing that needs to be done is the initialisation. But how can we do that?\n
Python has this great PEP, PEP 369 for post import hooks which can help us out here. It defines a mechanism whereby one can trigger code to be executed when a module is imported, allowing us to go in and monkey patch the module before the code doing the import gets it. That way, we can transparently make this change and others and you don't need to. One small problem. PEP 369 has never been adopted into the Python core. You have to implement your own.\n
To implement our own we need to hook into the Python import loader. To do that we need to implement two parts. First is the finder. When these are registered they are called to try and resolve the target of a module import. What we do in ours is to call back into '__import__' again being mindful to protect ourselves second time through to avoid a loop. If that works we return with the second part, which is our loader.\n
The loader simply grabs the module out of 'sys.modules' from the prior '__import__' we did in our finder. It then calls any of the import hooks we have registered against that module to trigger any instrumentation.\n
Our instrumentation function for 'django.core.handlers.wsgi' thus being as shown (ABOVE). You will note though that we have a new helper function called 'wrap_wsgi_application()' which eliminates the need for us to explicitly call '__getattr__' and '__setattr__'. It is just given the module and the object path of what we want to wrap with a WSGI application wrapper and does it thus simplifying it further.\n
So although the intent is that you should never need to use it, we still have used a layer of objects ourselves. This starts with the context manager which can be used with the 'with' statement directly if need be. (NEXT) A wrapper object which in this case has special knowledge of how a WSGI application works.\n
So although the intent is that you should never need to use it, we still have used a layer of objects ourselves. This starts with the context manager which can be used with the 'with' statement directly if need be. (NEXT) A wrapper object which in this case has special knowledge of how a WSGI application works.\n
Plus also have our decorator function (NEXT) and a high level wrapping function which can bring that altogether. As is obvious, you could use all these directly yourself and manually modify your code to instrument it, but we don't want you having to do that. Thus we use our import hook mechanism to trigger instrumentation we supply for Django and it goes in and does everything. All you need is to add the initialisation in an appropriate place.\n
Plus also have our decorator function (NEXT) and a high level wrapping function which can bring that altogether. As is obvious, you could use all these directly yourself and manually modify your code to instrument it, but we don't want you having to do that. Thus we use our import hook mechanism to trigger instrumentation we supply for Django and it goes in and does everything. All you need is to add the initialisation in an appropriate place.\n
Similarly to when wrapping the WSGI application itself to time the request as a whole. We also have these layers of objects for function tracing. The context manager (NEXT) and the wrapper object.\n
Similarly to when wrapping the WSGI application itself to time the request as a whole. We also have these layers of objects for function tracing. The context manager (NEXT) and the wrapper object.\n
Plus the decorator (NEXT) and high level wrap function.\n
Plus the decorator (NEXT) and high level wrap function.\n
Same again for tracing functions when recording database queries, external service requests and memcache operations. As with wrapping WSGI applications, we have lots of built in instrumentation. For Django we will wrap function traces around middleware, template rendering and other operations of interest. We also wrap DBAPI compliant database client modules. Everything is automatic and you don't need to change the code yourself.\n
The next problem is that in most trace object types we need to capture additional detail. For database queries we want to capture what the SQL was. Our high level wrap function for this can accept the SQL as a string, which would be static, or be supplied a function which can use to derive it from the arguments to the called function.\n
The trick here is our wrapper objects are implementing this dual role of being a wrapper, but also a method descriptor in the case of a bound method. We thus have to modify our wrapper to take either an actual function, or a instance and a bound method.\n
When the '__get__()' method is called to create the bound method, it will supply both the instance and the now bound method.\n
Why is this important? Well things get a bit messy at this point. Where a function is supplied to derive what the SQL is from the arguments, for the generalised case we really want to know what the instance was as well and if you take only the arguments being passed to the bound method you don't get that. Thus we have to explicitly insert the instance object so our function deriving the SQL from the arguments can use it if need be.\n
So, we do everything right but it still fails in some cases. Turns out not all database client modules are pure Python code. (NEXT) And you cannot monkey patch methods of Python objects implemented in C. Well, you can't do it from Python code. You can do it from C code by being evil and modify the Python C object '__dict__', but that is shared across sub interpreters which would cause problems for mod_wsgi.\n
So, we do everything right but it still fails in some cases. Turns out not all database client modules are pure Python code. (NEXT) And you cannot monkey patch methods of Python objects implemented in C. Well, you can't do it from Python code. You can do it from C code by being evil and modify the Python C object '__dict__', but that is shared across sub interpreters which would cause problems for mod_wsgi.\n
What we have to do in this case is start from the top and start wrapping objects from the only place we can change things, which is the module instance itself. We therefore wrap the DBAPI 'connect()' function and replace it with a factory object.\n
That factory object intercepts the call and wraps the result with a connection object wrapper. The connection object wrapper in turn intercepts the 'cursor()' call and wraps the result with our own cursor object wrapper. \n
Finally we can then intercept the 'execute()' method in which we can capture the SQL. Now remember how we used 'functools.wraps()' before. We can’t use that here as the ‘description’ attribute of a Cursor object is updated after ‘execute()’ is called. The 'functools.wraps()' function only makes a copy at the time of its call and doesn't reflect live changes. We therefore must use ‘__getattr__()’ instead.\n
Too much stuff. Too little time. (NEXT) Come and talk to us outside where we will be in one of the side rooms. (NEXT) We will be happy to give you a demonstration. (NEXT) Come to drinks and nibbles on Wednesday evening. Meet some others from New Relic coming up from the office. Also look out for Diane from ActiveState, one of our first partners who will be offering up a special deal for Stackato users.\n
Too much stuff. Too little time. (NEXT) Come and talk to us outside where we will be in one of the side rooms. (NEXT) We will be happy to give you a demonstration. (NEXT) Come to drinks and nibbles on Wednesday evening. Meet some others from New Relic coming up from the office. Also look out for Diane from ActiveState, one of our first partners who will be offering up a special deal for Stackato users.\n
Too much stuff. Too little time. (NEXT) Come and talk to us outside where we will be in one of the side rooms. (NEXT) We will be happy to give you a demonstration. (NEXT) Come to drinks and nibbles on Wednesday evening. Meet some others from New Relic coming up from the office. Also look out for Diane from ActiveState, one of our first partners who will be offering up a special deal for Stackato users.\n
Too much stuff. Too little time. (NEXT) Come and talk to us outside where we will be in one of the side rooms. (NEXT) We will be happy to give you a demonstration. (NEXT) Come to drinks and nibbles on Wednesday evening. Meet some others from New Relic coming up from the office. Also look out for Diane from ActiveState, one of our first partners who will be offering up a special deal for Stackato users.\n
Too much stuff. Too little time. (NEXT) Come and talk to us outside where we will be in one of the side rooms. (NEXT) We will be happy to give you a demonstration. (NEXT) Come to drinks and nibbles on Wednesday evening. Meet some others from New Relic coming up from the office. Also look out for Diane from ActiveState, one of our first partners who will be offering up a special deal for Stackato users.\n
One final thing. Specially for DjangoCon, we have moved up our public beta just so you could all get in on the action, and even perhaps go as far as trying it out while we are here to help you to get it going. We will be more than happy to then see what sort of data you are getting and help you to understand it. Also will be interested in any feedback. \n
If we have still have time we can do some questions, but if not then you can get me later. While doing questions here is the link for signing up for the beta and the email address if you want to send us questions that way instead.\n
