Twittex - From Idea To Live in Seven Days

Twittex:
From Idea To Live in 7 Days
Stuart Herbert
Technical Manager
www.gradwell.com

stuart.herbert@gradwell.net
stuart@stuartherbert.com

Welcome to my talk for the PHPNW 08 conference.

Let me introduce myself. My name is Stuart Herbert. You have probably come across my blog,
which is syndicated on Planet PHP.

I’ve been a professional software engineer since 1994, and I’ve been working almost exclusively on
web-based projects since 1993. My career has given me the opportunity to gain a lot of real-world
experience with UK household names over the years, and I’ve also been an active contributor to
open-source software since the early 1990’s.

I am currently the Technical Manager for Gradwell, a role I started in April 2008. We are what is
fashionably called a Uniﬁed Communications company. We do premium quality broadband, email,
hosting, and Voice-over-IP (VoIP). VoIP is the majority of our business in 2008, and we are the UK’s
third largest VoIP provider, with an annual turnover in excess of two million GBP. Celebrating our
10th anniversary this year, the company was founded, and is run by Peter Gradwell, who has been
very active in the UK internet scene for the past decade.

One of the services we offer, and the subject of this talk, is Twittex (http://twittex.com). Twittex
allows you to receive SMS messages from your Twitter feed.

All you have to do is to register your twitter account and your mobile phone with Twittex.

Decide which friends you want to watch, and when they post a status update to Twitter, you will
receive an SMS containing their status update.

Twittex is a very simple, and very straight forward service.

Twittex.com Team
Twittex was built from scratch in a seven day sprint by a team from Gradwell.

From left to right, back row: me (project management, queue servers, queue management,
provisioning), Ben Cole (Twittex.com website), Daniel Leech (Gradwell symfony components), Matt
Park (Twittex logo).

From left to right, front row: Ben Smithurst (SMS integration), Paul Thomas (server provisioning,
server monitoring), Tim Douglas (Twittex.com website).

Objectives

1. How We Built Twittex In Just Seven Days
2. What We Got Wrong
3. What We’d Do Differently Next Time

In this talk, I want to share with you our experience of building Twittex.com. I’ll start off by
explaining the key factors that allowed us to build this service from scratch in just seven days.

Objectives


Although we think it was a great achievement to build Twittex in just seven days, with hindsight we
made a few mistakes along the way. Some of them were obvious, and should have been avoidable,
but not all of them were, and I’d like to share those with you too.

Objectives


Finally, if we could go back in time and do it all over again, what would we change?

Questions Welcome!

Please feel free to ask any questions you have at any point during the talk. There will also be time
at the end of the talk for questions.

Background

1. Introducing Twitter
2. Twitter and SMS
3. The Opportunity
4. The Competition

Not everyone has heard of Twitter - surprising as that seems - so let’s start by introducing Twitter
and explaining what it does.

Twitter is a microblogging service hosted at http://twitter.com. Anyone can signup for free.

Once you’ve logged in, you can post short messages to the Twitter website (known as the public
timeline). It’s a great way to broadcast to the world what you are currently up to.

Each user on twitter gets their own “feed” - a list of messages from people that you choose to
follow. Here’s a screenshot from my Twitter feed. Your feed is also available via the Twitter API.

As well as keeping in touch with your friends without having to keep up with the latest blog posts,
Twitter is also proving to be very useful for creating online communities. One example can be
found on the PHPNW 08 conference website, where all posts tagged with #phpnw automatically
appear in the right sidebar.

Background

2. Twitter and SMS
3. The Opportunity
4. The Competition

So that’s what Twitter is. But where does SMS come into it?

Twitter allows you to register your mobile phone in the Devices tab.

Once you have registered a device, you simply navigate to a Twitter user that you’re following, and
click where it says “Device Updates OFF”. Twitter will then send you an SMS every time that Twitter
user posts a status update. And, best of all, these SMS messages are free.

This is a great service, and at Gradwell we’ve recently integrated it into our customer information
feeds. Our customers have long been keen on getting SMS alerts about service-related problems,
so we decided to achieve this by publishing our status information through Twitter.

Background

2. Twitter and SMS
3. The Opportunity
4. The Competition

A couple of weeks after we’d relaunched our GradwellStatus service, Twitter made some changes to
their service. These changes left us with a bit of a problem, but also with an opportunity to launch
a new service of our own.

On Wed 13th August, Twitter decided that it could no longer afford to provide free SMS alerts for
several countries, including the UK where Gradwell operates. If memory services, I believe these
SMS alerts were costing Twitter 1,000 USD per Twitter user per year. That’s a lot of money for a
ﬁrm with no revenue model!

The story was quickly picked up by TechCrunch (if you work on the web, and you don’t read
TechCrunch, you should), and from there spread like wildﬁre around the ’net.

Background

2. Twitter and SMS
3. The Opportunity
4. The Competition

Twitter’s withdrawal of free SMS alerts suddenly created a new market, one that possibly had real
potential. Gradwell wasn’t the only company to recognise this by any means.

First to decide that they were going to launch a service was tweetSMS. They got the word out very
quickly and very successfully.

tweetSMS was quickly followed by Zygo, which is run by former colleagues of mine from the old
New Media department at Orange.

By 22nd of August - just nine days after Twitter withdrew their service - no less than eight rival
services were competing in this new market. That’s a lot of competition!

We Can Do Better!
(Or Have Fun Trying :)

At Gradwell, we decided that we wanted to compete in this market too. As we were the only telco
launching a service (all our competitors are new media companies), we believed that we could
provide a better service. And even if we failed, it would be a lot of fun to try, and it would also be a
great way for me to learn a lot about my staff.

Timeline #1:
Actual Events

Let’s start by looking at a timeline of what actually happened, and when. This timeline was
compiled using BeeDocs, and at the PHPNW 08 conference, it was presented as an animated
timeline.

I’ve split the timeline up into two streams. Along the top is what was happening out on the web,
and we start with Twitter withdrawing their SMS service on the 13th August.

TechCrunch picked up the story on the same day, and began to spread the word far and wide.

On the very next day, TweetSMS announced their plans on Mashable (another website you should
add to your Google Reader feed if you don’t have it already).

Zygo announced their service on the following day ...

... by the 22nd of August, there were eight competing services analysed by the
OnlineJournalismBlog.

The second stream in the timeline looks at what we did at Gradwell. Our story starts on the 14th of
August, when I came across the TechCrunch story in my Google Reader lists. I immediately thought
that we could plug the gap created by Twitter, and that it was something we should do.

By the middle of the afternoon, Ben Smithurst and I had sketched out the architecture we wanted to
build for the service, and felt conﬁdent with our underlying design.

By 7pm on the 14th, I phoned Peter up, and left him a voice mail asking for his approval to build
what would become Twittex.

As Peter was camping out in the wilds of Scotland, we had to wait until he drove to the nearest town
for some milk before he had a phone signal. This didn’t happen until the following morning.

With approval secured, we settled down to getting on with the work. Ben Cole immediately set to
work on building the website.

We spent a few hours coming up with a name for the service, ﬁnally settling on twittex. As soon as
I approved the name, we registered the domain to make sure no-one else came up with the same
name :)

Shortly after, we had ourselves a logo. Although Gradwell doesn’t have a graphic designer in-
house, we’re very lucky that one of our staff can turn his hand to this sort of work, and that he was
very happy to make the time to do so.

How long would it take you to get a new logo for a service that you wanted to launch? Days?
Weeks? Longer? In a show of hands at the PHPNW 08 conference, many of the audience said that it
would take weeks or longer to get a logo designed.

We carried on working into the evening, and by 7pm we had the ability to send SMS messages out
to mobile phones.

By 9:30pm, we’d ﬁnished provisioning the servers to run the queues on (more on the queues later
on), and had ﬁnished compiling and installing MySQL onto the servers.

So to recap: on the Wednesday, Twittex announced that it was withdrawing the SMS service. By the
end of the Friday, we had our servers, our domain, our logo, and the ability to send out SMS
messages.

The team continued to work throughout the weekend. Whilst work was continuing on the web site,
we spent Saturday and Sunday writing the queue management code.

We also sorted out billing and payments on the Saturday.

On Sunday, we adopted the 960 grid system by Nathan Smith, and began using it to prototype the
user-visible parts of the website.

On the Monday morning, whilst work on the website was continuing, we put the server monitoring
in place.

And, on the Tuesday, we’d finished the website, and opened it up to staff for internal testing. We
were kept very busy finding and fixing corner cases in our code that day!

With all the obvious bugs squashed, on the Wednesday, we were ready to launch. We contacted
TechCrunch to announce our service ...

... we also contacted Twitter’s own PR people ...

... but heard nothing back from them at all. At the same time, we sorted out an introductory offer
price ...

Shortly after, Ben and I had to jump in the car and drive up from Bath to Sheffield for an important
meeting the following day. Unfortunately, there was a serious accident which closed the M5 for
four hours, giving us plenty of time to do even more bug ﬁxing whilst we waited for the road to re-
open. Laptops and mobile broadband are essential tools for any developer who travels!

So, we launched our service just 7 days after Twitter announced they had withdrawn theirs, and
what’s more we were the ﬁrst folks to have a working service live. Were we swamped by demand?
No. One of the reasons was that our initial pricing was badly wrong. We had to reduce our pricing
from 10p per SMS message to 5p per SMS message, and we had to do it quickly.

With our meeting in Sheffield out the way, we headed our separate ways (Ben back up north, me
back to South Wales), doing more bug ﬁxing on the way. We improved the log messages that
Twittex writes, to help us diagnose problems quicker.

We also created a new Twitter account just for testing twittex with. Up til now, we’d all be using our
personal Twitter accounts, which put a bit of a limit on the type of test messages we wanted to
send.

Our friends at PlusNet kindly mentioned us on their blog - our very ﬁrst bit of publicity!

As we still hadn’t heard anything from TechCrunch or Twitter at all, and hadn’t been mentioned by
them either, we decided to widen the net by contacting The Register and Mashable. Sadly, neither
website wrote anything about us :(

On the following Monday, Jemima Kiss wrote a piece for the Guardian’s website about Twitter and
SMS messages. We dropped her a quick email about our service. Jemima was kind enough to write
back - the only one who did out of all the organisations we contacted - letting us know that we’d
basically missed the boat.

By keeping below the radar until we had a service to launch, we’d completely missed the news cycle.

But Twittex Really
Started Months Ago!

So that’s how we built Twittex in just seven days. Except - it’s not the whole story. Work on Twittex
had actually started *years* before.

Huh?

Years before? What do I mean by that?

Twitter.com

Twittex.com

Netsuite Gradwell SecPay

Let’s start by looking at the architecture for Twittex.com. We have the Twittex.com website, built in
just seven days.

Twitter.com

Twittex.com


It pulls feeds down from Twitter ...

Twitter.com

Twittex.com


... and then re-uses Gradwell’s existing telco infrastructure.

Twitter.com

Twittex.com


An important part of this infrastructure is our ERP/CRM system, Netsuite. As a company with a
multi-million pound turnover, Gradwell has to produce audited accounts every year. A company of
our size simply cannot create a new service, hook it up to a PayPal shopping cart, and leave it at
that.

Twitter.com

Twittex.com


For payments, we use SecPay, because it’s the only payment gateway in the UK that’s certiﬁed to
work with Netsuite at this time.

Software
Software
Load-balancers
Load-balancers

PHP Servers MySQL Server
PHP Servers

Queue Servers
PHP Servers

That’s the architecture for Twittex. If we look inside the Twittex.com box, we’ll ﬁnd a pair of PHP
servers.

Software
Software
Load-balancers
Load-balancers

PHP Servers

Queue Servers
PHP Servers

These servers sit behind a pair of load-balancers.

Software
Software
Load-balancers
Load-balancers

PHP Servers

Queue Servers
PHP Servers

Twittex uses MySQL to hold subscriber information.

Software
Software
Load-balancers
Load-balancers

PHP Servers

Queue Servers
PHP Servers

And ﬁnally we have a set of queue servers which manage the job of polling Twitter for new
messages.

At the heart of our infrastructure is VMWare ESX, which we adopted late 2007. ESX gives Gradwell
the ability to provision new servers extremely quickly. We don’t have to wait for new hardware to
arrive, and we don’t have to have someone physically installing kit into racks in a data centre.

We’ve been using symfony for over a year now as our framework of choice. We chose symfony
because we felt it had a more responsive and more supportive community that its competitors.

As I mentioned, we use MySQL for our databases. I can’t imagine there’s anyone reading this who
hasn’t heard of MySQL, but I’ve included it for completeness :)

We adopted the 960 grid system for our prototyping, and our live Twittex.com site also uses 960
gs. If you haven’t come across it before, check it out. We found it extremely useful. Using pure
CSS, you can create pages that are 960 pixels wide, cut up into 12 or 16 columns. These columns
can then be merged to form the ﬁnal layout. The only change we’d love to see Nathan make to his
system would be to add a 20 column version, to allow for even more precise layouts.

For the queues, we used the Q4M storage engine for MySQL 5.1.

Q4M allows you to create ﬁrst-in, ﬁrst-out queues that can be accessed via SQL SELECT statements.
The really nice thing about using Q4M is that our own code doesn’t have to handle any sort of
locking at all. We can have multiple processes emptying the queues, and Q4M ensures that any one
piece of data from the queues isn’t presented to more than one of these processes.

I came across Q4M when it was selected for the architecture of a Japanese rival to Twitter.

Why Queues?

But why use queues at all? Let’s take a look at that.

Twittex User
1 Min Queue

2 Min Queue Twitter Message :)

3 Min Queue

4 Min Queue No Twitter Message :(

5 Min Queue

We start off with an individual Twittex user.

1 Min Queue


3 Min Queue


5 Min Queue

We put the data about this user into a queue. We check all data in this queue every minute.

1 Min Queue


3 Min Queue


5 Min Queue

Eventually, our Twittex user makes it to the front of the queue.

1 Min Queue


3 Min Queue


5 Min Queue

We go and check Twitter, to see if there any new messages for this user. If there are new
messages ...

SMS Sent

1 Min Queue


3 Min Queue


5 Min Queue

... we send out the messages as SMS’s, and put the user’s record back onto the end of the queue.

1 Min Queue


3 Min Queue


5 Min Queue

Eventually, the user once again makes it to the front of the queue.

1 Min Queue


3 Min Queue


5 Min Queue

What happens where there are no new Twitter messages? Well, we won’t want to poll Twitter every
minute for every user. That’s a lot of wasted effort.

1 Min Queue


3 Min Queue


5 Min Queue

Instead, we have additional queues, which are checked less and less frequently.

1 Min Queue


3 Min Queue


5 Min Queue

The user’s record works its way through these queues, until eventually it hits our 5 minute queue.

1 Min Queue


3 Min Queue


5 Min Queue

If there are still no Twitter messages for the user, the user’s record stays in the ﬁve minute queue.
We used to have 10 minute and 15 minute queues too, but during internal testing we decided that
they made for a poor user experience.

1 Min Queue


3 Min Queue


5 Min Queue

If there are any messages for the user on Twitter ...

1 Min Queue


3 Min Queue


5 Min Queue

... we move them straight back into the 1 minute queue. This makes Twittex very responsive if you
strike up a conversation through Twitter. The user’s record will then work its way back down the
queues as before until there’s another message on Twitter for the user.

Timeline #2:
Groundwork

Our architecture and our infrastructure re-uses technology that we adapted long before we had the
idea to create Twittex. Without this re-use, we could not have built Twittex in just 7 days.

Here’s another timeline, this time looking at all the things we did in the past that made Twittex
possible.

We’ve been using AQL for sending and receiving SMS messages since at least October 2003. AQL
provide a very easy to use RESTful API, and we’ve had a lot of practice with it over the years.

Do excuse this rant, but if you’re going to build an API for any sort of service, do make it RESTful.
SOAP APIs should never be the only API you offer. For a start, PHP’s SOAP client is a toy, and
currently does not implement enough of the SOAP standard to make it inter-operate with real
enterprise SOAP APIs. Secondly, too many SOAP API authors need to learn that WSDL is not
documentation - it is speciﬁcation. And ﬁnally, SOAP APIs also suffer from poor design, because
they focus on exposing objects rather than business services. So, please please please, stick with
simple but effective RESTful APIs :)

We adopted Netsuite for ERP and billing in May 2007. As a company, we already have an
established platform to handle new products, and the ﬁnancial accounting that goes with them.

We’ve been using symfony since June 2007, and in that time we’ve built many re-usable
components for it. We’re also very lucky that we have an internal champion for symfony, who not
only advocates for its use but also that it’s used in the proper manner. This keeps our code lean,
clean, and very easy to reuse.

We made ESX our main platform earlier this year. ESX gives us ﬂexibility that we couldn’t afford if
we were still working solely with physical hardware. How would you launch a new service in just 7
days if you had to wait for new hardware to be delivered ﬁrst?

The Twitter APIs are very easy to use, being REST based, but nonetheless we’d already had
experience working with them. We’d used them in July to integrate our GradwellStatus.com website
with Twitter.

Twittex was the ﬁrst time we’d used Q4M at Gradwell, but it wasn’t something we had to go out and
ﬁnd on the day. A big part of my job as Technical Manager (CTO/CIO in US language) is to have a
wide knowledge of what’s available in the open source community and beyond just in case we ever
have a need to introduce something new to our established technology stack. I rely heavily on
Google Reader to help me keep on top of hundreds of RSS feeds from around the net, as well as
attending conferences, trade shows, and talking to my peers.

What Made It Possible

1. We Were Prepared
2. We Were Up For It
3. An Extension Of What We Already Do

So, to recap the elements that made Twittex possible. First and most importantly, we were well
prepared. We have been consistent over the years in adopting technology that is simple to use and
that gives us increased ﬂexibility.


1. We Were Prepared

It had already been an extremely hectic week, but with no prior notice, my team willingly gave up
their evenings and the weekend to work on this project. Do you think we would have been able to
build this with a 9-5 mentality? We would have been too late to market, and the project would
probably have failed, because we would not have managed to maintain the momentum in the ﬁrst
place.

In questions after the talk, I was asked why my guys were up for it. I think it comes down to two
key things: they thought it was an interesting project to do, and to them programming isn’t a job,
it’s a vocation. It’s what they love to do.


1. We Were Prepared

Finally, we believe that a critical factor was the nature of the work. Gradwell is an established and
successful telco, and is already used to handling this sort of messaging. We’re not a New Media
company suddenly trying to get our head around a different paradigm. This is stuff we know how
to do, so we can just get on and do it.

What We Got Right

1. Recognised Opportunity
2. Authority To Proceed
3. Technology
4. Delivery Of Working Solution

What did we get right? First of all, we very quickly recognised that there was an opportunity to be
tackled. We needed to do something, because we had been relying on Twitter for SMS updates to
our customers. We didn’t um and ah about it, we got our heads down straight away on solving the
problem.

What We Got Right

3. Technology

Despite the amusing story of how long it took to reach Peter for approval, I’m actually very pleased
about how quickly we had authority to proceed with the project. We had the green light just 24
hours after coming up with the idea.

In my own experience, most companies are utterly crap at making decisive decisions, especially in
such a short period of time. The feedback I had after the talk was that many companies couldn’t
make decisions this quickly.

What We Got Right

3. Technology

I’m very happy indeed with our choice of technology. From ESX through to Q4M, from symfony
through to the 960 grid system, everything we used made the job much easier - not much harder.
At no point were we having to ﬁght with the technology. We were able to just get on with creating
the service.

What We Got Right

3. Technology

Let’s not forget - we actually launched a working service. It works very well.

But unfortunately, it wasn’t successful in attracting a large user base ...

What We Would Change

1. Marketing
2. Pricing
3. Project Management

The ﬁrst thing we got wrong was our marketing. We got this badly wrong. We deliberately kept off
the radar until we had a working service. We hadn’t done a sprint like this before, so we weren’t
certain how long it would take us to build a launch service. By the time we wanted to tell the world
about our marvellous creation, the world was no longer interested. The news cycle had moved on,
and we had missed the boat. Those behind the news cycle, it turns out, have neither the time nor
the attention span to care about working services :)

In my mind, tweetSMS did exactly the right thing. They put up a ‘sign up now - coming soon’ page,
and they put it up very quickly. This got them coverage, and no doubt it allowed them to hoover up
many of the early adopters that Twittex was also aimed at.


1. Marketing
2. Pricing

Boy, did we get this wrong too :) My initial pricing, at 10p a message, was designed to cover our
costs. That was a mistake. We had to drop our prices after the ﬁrst day, once we realised that our
competitors were planning on launching services that were around the 5p a message mark.

Instead, I should have made the price as low as possible, and used pricing and our existing bulk
discount to gain the largest market share over our competitors.

As it turns out, the market for paid SMS messages from Twitter in the UK appears to be tiny. No-
one has made a killing on this at all. The wholesale price for SMS messages is just too expensive.
I’m extremely interested in making Twittex a free service supported by advertising. If there’s
anyone out there who’d be interested in making this possible, do get in touch :)


1. Marketing
2. Pricing

I made some obvious mistakes with the project management too. The main one was that we only
started prototyping the web pages themselves three days into the work on the website. What I
should have done was had a bit more balls, and committed extra resources on the Thursday (before
we had approval to proceed) to create the prototypes up front.

Timeline #3:
A Better Timeline

If I could go back in time and do this project all over again, knowing what I know now, what would
be different?

Twitter would still have stopped sending SMS updates out on the 13th ...

... and TechCrunch would still have spread the word on the same day.

TweetSMS would still have been ﬁrst to get the word out that they were launching a replacement
service.

But if I could do this all over again, we would have gotten the word out to anyone and everyone as
quickly as possible too. The earliest we could have done this would have been the 15th, after the
approval to proceed.

Would Zygo (and all the other competitors) have announced too, if we had made our intentions
known? That’s impossible to speculate on.

... but I can’t help but wonder whether we’d have ended up with a two horse race instead of an
eight-way ﬁnish.

Inside Gradwell, some things would stay the same. I’m very happy with how early we recognised
the opportunity.

Sketching out the architecture up front is a necessary step, and I don’t think we could have achieved
it any sooner than we did.

But one thing I would change would be to prototype the website up front too - even before having
approval to proceed.

I don’t think we could have made the project request any sooner. We needed to have conﬁdence
that we could build the service ﬁrst.

We got approval to proceed very quickly.

If we had had the prototypes up front, the website would have been built even quicker. We would
probably have taken a day out of the timeline.

We should have created the twitter test account up front, so that we could do a wider range of
testing during development. Whether or not this would have allowed us to launch earlier is hard to
say, but it would have perhaps reduced the amount of bug ﬁxing we did post launch.

I’m very happy with how quickly we came up with a name and purchased the domain.

We should have paid more attention to the commercials up front. We should have had a clearer
idea about them, so that we got our launch pricing right ﬁrst time.

I’m very happy with how quickly we had the logo done for the site.

Our integration with AQL was done very quickly, and I’m very happy with how that went.

I’m also extremely happy with how quickly we had working servers to deploy code onto. In the
past, I worked with organisations who rely on third parties such as Rackspace to provision new
servers. The lead times are often 20 working days or longer, because new hardware must be
provided.

We probably couldn’t have built the queue management code any quicker ...

... and we probably couldn’t have had the billing arrangements in place much quicker.

I’m happy with the time it took to get the server monitoring in place. This couldn’t have been done
earlier, as we needed to provision the servers ﬁrst before hooking them up to our monitoring
system.

If we’d had the prototypes up front, the site would probably have been ready for internal testing a
day earlier.

... and we would have been able to launch the site at least one day earlier. Who knows? Just maybe
we might have been able to launch on the Monday :)

I doubt we would have been able to avoid being stuck in the traffic jam on the M5.

We would have been able to avoid having to reduce our prices, if we’d got the commercials right up
front.

I’m comfortable with having had to improve our log messages after the service had launched. I
think it made sense to focus on getting the customer-visible side of the service working ﬁrst, and
then going back and polishing up things behind the scenes to reduce the support burden.

I’d like to think that our friends at PlusNet would still have mentioned us when they did.

Your Thoughts?

That’s what we think would have happened had we done a few things differently. What do you
think?

Thank you for viewing this presentation. I hope you’ve found it interesting and that you’ve learned
something from it. We certainly learned a lot from doing this project. If you have any questions or
feedback, please get in touch with me stuart.herbert@gradwell.net.

Twittex - From Idea To Live in Seven Days

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (6)

Similar to Twittex - From Idea To Live in Seven Days

Similar to Twittex - From Idea To Live in Seven Days (20)

Recently uploaded

Recently uploaded (20)

Twittex - From Idea To Live in Seven Days