Today was the big day – at 9AM pacific, tickets went on sale for San Diego Comic-con 2013. Unfortunately, their web developers had a few bugs in the software, one of which bit me bad enough it killed my ability to secure tickets.
As a web developer it hit me about 15 minutes in as to why I wasn’t moving up in line (and the “customer service” line was busy the entire time). Once I opened the Developer Console in Google Chrome and saw the JavaScript errors I knew it was over.
- Store all static assets on a CDN. This is the bug that bit me – there were some JavaScript resources that it was trying to pull from the server that timed out with 503 errors (service unavailable, usually due to demand). If these files were stored on a CDN all over North America & the world, it wouldn’t have been an issue. Even if they were hosted within the same “cloud” instance, if one critical piece of the cloud goes down, it would take all those requests with it. Why is it a big deal if the JavaScript doesn’t load? Because the JavaScript had the code that would refresh your position in line, and when it was your turn, redirect you to the portal to buy the badges. So what ended up happening to me was that I was #22,397 in line, but I never advanced in line because the JS that was responsible for moving me up in line never loaded and never executed.
- If the next page is going to take a while to load, tell people. The other big issue I saw the white page of mystery. Basically, after you clicked the big green “GO” button at 9AM, if you made it into the EPIC (the name of their ticketing system) waiting room, the screen would be blank for a while, for me it was about 90-120 seconds. But then the screen would come up and tell you what place you were in line, what not to do (don’t refresh the page, etc). But people didn’t know what to do with this infinitely loading page. What I would have done was serve a static HTML page, and then used AJAX to retrieve that person’s place in line (via the session key), fill in all the details and then let them wait it out until they’re at the front of the line.
- Sometimes, just putting it in on a cloud-based system isn’t enough. Sometimes you might need more than one cloud-based backend. Partition the servers (e.g. 20,000 tickets on each) and go from there. Likewise, multiple card payment services might need to be used in order to process card swipes quickly (you might need 3 or 4 different providers, and you just switch off between them based on a hash of an order ID #).
So thats what I learned as a web developer to cope with high demand events.
Now as a nerd, I’m still pretty pissed off I wasn’t able to get a ticket, even though I was lucky enough to make it into the waiting room. Some time around May they’ll release some more badges, but likely the scramble for them will be more desperate and furious than this one.