Missional Code alert('Him');

8Sep/110

Counting and Timing with Etsy, Flickr

Articles:

http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/

http://codeascraft.etsy.com/2010/12/08/track-every-release/

http://code.flickr.com/blog/2008/10/27/counting-timing/

I stumbled upon that first Etsy article a few months ago and was so confused about the content that I have tried to read it on a number of occasions and only now can begin to understand what it was talking about. I thought a cool blog post might be taking articles off a tech blog and trying to translate them to language that I (or some other starting programmer) may be able to understand more easily.

All three articles are concerned with analytics - being able to track how often something occurs (counting) and how long something takes (timing). Etsy uses a tool called Graphite, while Flickr uses something called RRDTool.

Counting would like to keep track of how often a certain action occurs, like how often cached data needs to be reloaded. Since this action could be occurring relatively often, in conjunction with a bunch of other actions that we might also want to count, the cost of connecting to a database and incrementing a number begins to be undesirable, as this may take more and more time and your site may begin to suffer in terms of performance. In the first and third articles, UDP packets are utilized to increment some global variable, that can then be interpreted by a graphing tool like Graphite or RRDTool.

UDP packets are preferred over TCP packets because UDP doesn't really care if a packet is received successfully or not - it simply sends them out. Because it is not concerned with the packet being received, UDP is faster and takes less bandwidth. Fortunately, we can also easily track how many UDP packets failed.

For Graphite, counting can be accomplished by calling an increment function on a keyword. Timing can be determined by simply logging the time before the action, logging the time after the action, then recording the difference with an associated keyword. After saving these associated values together, Graphite will be able to give useful statistics like quartiles and standard deviations.

The second article speaks about continuous deployment, in which developers are constantly deploying to a live site, and the importance in being able to interpret whether certain analytics results may be the direct result of a previous deployment (that could be crashing the site or causing other errors) or not. Using Graphite, Etsy records whenever a deployment is made, then uses a specific feature to draw lines whenever new code has been pushed to the live site.

Comments (0) Trackbacks (0)

No comments yet.


Leave a comment

(required)

No trackbacks yet.