Chris Jackel Systems Engineer, Fastly at Altitude 2016
A/B testing is valuable, aggravating, and everyone has an opinion on the best way to do it, with a strong belief that all other methods are flawed. At Fastly, we want to provide a set of tools to help you with experiments you can integrate into your own analytics. This workshop covers:
*Randomization with Fastly
*Persistent segments with and without cookies
*The power and limitations of HTTP vary
*Using Edge Dictionaries for updating tests programmatically
*Integration with external tools
If your page traffic is delivered directly by Fastly and you add information to the real-time log stream, there may be no need for third-party trackers. Otherwise, you can easily add the custom variables to existing cookies. These techniques aren’t designed to replace your existing tools, but may give you new options to build something exciting.
23. Segmentation
sub vcl_recv {
if (randombool(5,100)) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = "A";
}
}
24. Segmentation
sub vcl_recv {
if (!req.http.Cookie:FastAB) {
if (randombool(5,100)) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = "A";
}
}
}
25. Segmentation
sub vcl_recv {
if (!req.http.Cookie:FastAB) {
if (randombool(5,100)) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = "A";
}
} else {
set req.http.X-FastAB =
req.http.Cookie:FastAB;
}
}
26. What We Need
• ‘A’ and a ‘B’
• Method of (persistent) segmentation
• Keeping things separate (vary)
35. HTTP Vary
- The chef (or server) tells us what is important information (e.g. flavor)
- The customer (or browser) tells us the flavor
- Max of 200 variations!
- Otherwise change the hash
- Don’t vary on too big a set (e.g. user-agent)
60. Cookie-less
set req.http.X-ClientIDHash = digest.hash_md5(client.ip
req.http.User-Agent)
# e.g. 3aaf91f17e7c63b07f8490ab242f9335
if (randombool_seeded(5,100,seed))) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = “A";
}
61. Cookie-less
set req.http.X-ClientIDHash = digest.hash_md5(client.ip
req.http.User-Agent)
set req.http.X-ClientID = std.strtol(req.http.X-ClientIDHash,16)
# e.g. 9223372036854775807
if (randombool_seeded(5,100,seed))) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = “A";
}
62. Cookie-less
set req.http.X-ClientIDHash = digest.hash_md5(client.ip
req.http.User-Agent)
set req.http.X-ClientID = std.strtol(req.http.X-ClientIDHash,16)
# e.g. 9223372036854775807
if (randombool_seeded(5,100,std.atoi(req.http.X-ClientID))) {
set req.http.X-FastAB = "B";
} else {
set req.http.X-FastAB = “A";
}