From the Desk of the General

Today, during the Highlanders v Stormers game, fellow HQ partner Slacker and I experienced what’s called the SuperBru 75th Minute Reversal.

We had both picked the men from Otago to beat the men from the Mountain, and as the game wore on I became increasingly despondent that the jet lag I had expected to affect the Stormers in the second half was nowhere to be seen.

When Liebenberg was carded, Slacker and I both thought the tables would start to turn, but that incredibly organised and resilient Stormers defence held strong. As our hopes of a Highlanders comeback faded, we reached the point of the 75th Minute Reversal.

The Stormers were leading by around 15 points, which meant anyone in the 10-20 range would bag a Margin Point. However, if the Highlanders scored, it would shift to a lead of Stormers by 8 or 10 and that would spell disaster for Slacker and me as our pool was packed with picks for Storners by under 10. “It’ll be Margin Point city,” I sighed.

And so came the reversal: after egging on the Highlanders the whole game, we switched to supporting the Stormers for a final try. Our Win Point hopes were shot so it was damage limitation time - all we could hope for was as few Margin Points to get dished out as possible, and more points for the Stormers would be the safest way of pushing the result out of margin reach of the Stormers by under 10 crowd.

Then came the millimetre-close almost-try for the Highlanders. It was a tense moment for us. It was a funny thing: five minutes earlier we’d have been screaming that his boot never touched the line, but post-Reversal we were quite the opposite. No ways that was a try! Boot clearly scrapes the line!

Funny, isn’t it, how SuperBru induces these sudden, brutal changes in support?

I grew up in Durban and, sometime in my late teens, developed the kind of emotional bond with the Sharks that all sports fans know about (it’s a weird sort of thing, isn’t it?)

Last week’s pick was tough for me. The Sharks were taking on the Brumbies in Canberra and hadn’t won there since 1998. I still have memories of the bitter 2001 final where Mark Andrews sarcastically thanked the crowd for being so sporting. Lambie was out, and the somewhat erratic M. Michalak was in his place. And they’d blown it against the Tahs the week before when they should have won.

Add to this the form of the Brumbies, who seem to have turned a corner under the tutelage of Mr White (sure knows a thing or two about rugby, that guy).

So, head said Brumbies, whilst heart of course said Sharks.

It ate away at me all week, but eventually I went for the traitor’s hedge - I picked the Brumbies. Let’s define it.

The Traitor’s Hedge

Picking the opposition, like I did - Brumbies to beat my Sharks. If the Sharks had lost, I’d have felt a bit guilty about being a traitor, but at least the guilt would have been eased by my Win Point. The risk of course is that your team wins, like the Sharks did, in which case you’re the worst of everything: not only are you a traitor, but you have a big 0 in your Win Point column. The guilt takes over, and your pick the next week gets even harder… are you going to be a traitor two weeks in a row? Are you going to pick the Canes? Well, General, are you? (no, is the answer - can’t handle the guilt)

The worst part of the Traitor’s Hedge cropped up in that Brumbies v Sharks game. At the death, the Brumbies nearly won. If they’d hung onto the ball long enough, they probably would have scored. That final set of phases seemed to last forever. And deep inside the General, there was that nasty, evil little voice egging them on… “if they win, the Sharks will still get two Bonus Points… and you, self, will get one too!” The internal conflict was horrific and scarring.

The opposite scenario is…

The All or Nothing Gambit

Picking your favourite team even when you know their chances of winning are slim (Rebels fans must be familiar with this). When they go down in flames, at least you weren’t a traitor and you can sleep well at night with no blemish on your conscience. When they win against the odds, you’re golden - you’re scooping up the points, nobody else in your pool picked your team, the relative gain is huge… 

Ok, good talk. The Sharks better win this week… I couldn’t handle regretting not being a traitor…

If you’re reading this, you have probably been affected by performance issues on SuperBru the last two Fridays. Here at HQ, we refer to Super Rugby Fridays as Frantic Fridays (TM). Nobody is more frustrated by Frantic Fridays (TM) than me - they suck the life, soul and sense of humour out of me as I sit there driving mitigation strategies as we try to get through the day.

I thought it might be helpful to be quite open about what happens, how we prepared this year and what we’re trying to do to improve things.

Frantic Fridays(TM): what happens

Early on Friday mornings we experience a severe traffic spike as brus try to get their picks in for the first game, which typically kicks off at 08:35 SA time. As people are waking up and commuting early on, it really peaks between 07:30 and 08:30 - brus are at work, they realise they’ve not made their picks, and they head to the site. The traffic sustains at heavy levels through the first game as those who haven’t picked for game two make their picks, and is then knocked by the first game result processing, which creates a stack of extra effort. 

In terms of volume we receive upwards of 25x normal traffic, and it’s all the worse for being concentrated and fixated on  deadlines: people need to get their picks in, so they keep trying rather than come back later, and the traffic keeps compounding itself.

Websites and the technology behind them are great at dealing with heavy load that is spread out, but inherently struggle with severe, concentrated spikes. Which leads to…

How we prepared this year

We were expecting growth after RWC, and knew we needed to prepare for it (as it turned out, Super Rugby 2012 is 70% bigger than 2011 was for us, and is our biggest tournament ever - thank you for playing!).

Standard industry practice gives three ways to handle scale better:

1. Scale up - increase the capacity of your servers
2. Scale out - add more servers
3. Optimise - improve your processes so that they do things more efficiently

We did all of these in preparation for this year…

We invested in a new hosting provider and a new scaling technology in December: we moved to Rackspace, probably the world’s leading hosting company, and took advantage of their cloud technology, which lets us spin up new servers on demand. Whereas before we were constrained to a fixed infrastructure, we can now scale out our webservers by adding more, quickly. In the build-up to Super Rugby we had two webservers (connecting to a powerful dedicated database server) and they met our needs just fine. We currently have nine webservers (and are blowing some budget as a consequence!)

We also scaled up by offloading some services to other new servers behind the scenes, leaving more capacity available where it is needed for the core applications.

And lastly we optimised a whole bunch of legacy code and processes, and are continuing to do so. For example, after last Friday we spent the weekend reviewing our result processing procedure, and found some more optimisations we could squeeze out of it that made this Friday’s results process much faster.

So why the problems?

We still have a single bottleneck in the form of our database server, despite some clever stuff which offloads some of that server’s work to other back end servers and the cloud webservers.

I’m going to hit you with a classic tech analogy (sorry): it basically works very similarly to a restaurant. Tables and waiters are webservers, and the kitchen is the core application on the database server. For the Friday breakfast run, Chez SuperBru lays out 5 times more tables and waitstaff, and accommodates a lot more brus ordering their croissants and cappuccinos than normal. However, strain builds on the kitchen. The kitchen is huge, and way more than is required for most days of the week, but as those orders start coming in at a much increase rate, it starts to struggle.

The difference between SuperBru and a content site (like a news site) is that our site is database-heavy because of its nature: you’re logged in, and we need to constantly serve content and features totally bespoke to you. A news site, by contrast, can let its webservers do the vast majority of the work. In the News Restaurant, everyone orders the same croissant, so there’s just a massive pile of croissants and you just add more waiters and tables to distribute them. At Chez SuperBru, every meal is a personalised, hand-crafted creation with your name written lovingly in chocolate sauce, thereby requiring the kitchen’s attention…

Conclusion

Our focus continues to be finding ways to offload work from the kitchen to the waiters, and to find the right balance between the number of tables we lay out and the capacity of the kitchen, all without compromising too much on the application features that you want.

We are investing a lot of time, money and emotion into this - please know it means everything to us to try to get this right. 

Two hours ago I posted an article saying that registration for Super Rugby on SuperBru looked like it was going to close out as the second biggest day ever (BDE) for us today… looks like I was a little conservative as sign-ups are pumping through the night.

Today is now our biggest day ever, beating the Thursday before RWC 2011. 16,213 players have joined today, including 8,324 brand new brus.

From our live joiners report today:

We should hit around 140,000 users a couple of months into Super Rugby. For comparison, a South African TV audience for a really big rugby game is around 600,000. Or, to look at it another way, there are more brus in Super Rugby than the total population - every man, woman and child - of George.

Rookie brus - welcome! Veteran brus - great to have you back.

Registration has been pumping hard on SuperBru today. 14,184 players have joined Super Rugby so far today (over 7,000 of them brand new users) and we’ve got a few hours of antipodean activity still to go… 

We need another 1,200 to beat our previous biggest day ever (BDE), the Thursday before RWC 2011… suspect we’ll fall just short.

A big thank you to everyone who’s joined up!