A note about yesterday’s downtime

RSS
Godville, Sep 07 2011

Lots of you may have experienced problems when checking your heroes yesterday, as GV servers were not available or were returning error messages. First of all, we’re sorry that happened, and we want to let you know about the situation and what we plan to do in order to prevent it from happening again.

Our hosting provider informed us that due to some excavation work, a fiber optic cable that connects our datacenter (and 4 others) to the Internet had been cut, which caused everything to work really really slowly or sometimes not work at all. We tried to mitigate the issue by tuning up the servers, but that wasn’t enough to work around the underlying connectivity issue completely. After approximately 8 hours, the underlying connectivity issue was fixed by our hosting provider, which resolved the issues in accessing our servers.

Unfortunately it’s almost impossible to manually restore or rollback all the bad things that happened to your heroes while they were wandering around without supervision from their gods, but here’s what we’re going to do to prevent it in the future:
  • tune up timeouts in our mobile apps to better handle situations like this
  • fix some newly found bugs in our Android app related to error messages when servers are not available (some of them were already fixed in v2.5 available in Android Market, but this version is still going through the Amazon Appstore review process and not yet available to those who downloaded the app from the Amazon Appstore)
  • provision for additional server capacity to handle even more requests
Here are some tips for our players:
  • follow us on twitter @godvillegame to get the latest updates, especially at times when our site is not accessible
  • if you’re using our Android app and for some reason you were logged out of your account, but had never set a password: please try to login with your godname and an empty password
  • be aware that due to Amazon’s review process, app updates to Amazon Appstore are very likely to reach you several weeks later than the same update in Android Market

UPD

During yesterday’s non-stop endless patching & tuning session, we accidentally messed up one of the servers, which eventually backfired causing some of the accessibility issues observed today. As of now, this problem is resolved and everything should work fine. And to prevent this happening again, we promise to try get some sleep and be more careful even when our inbox is flooding and things are on fire.

If you don’t use twitter, here is the good old-fashioned RSS feed for blog posts.

Comments (80)
Helen of Troy Sep 08 2011 03:09
Gravatar

I realize you have no control over severed fiber optic cables. However, the fix that kinda worked last night haven’t done ** for this evening. Hope it all gets fixed soon.

SenseiP Sep 08 2011 03:10
Gravatar

Alright team, enough Red Bull, let’s stop the shakes and focus on the problem. There are heroes out there starving for guidance.

Big5hit Sep 08 2011 03:19
Gravatar

Yea I play lol on the john. Miss the funny reading guys!

Evil MoFugger Sep 08 2011 03:50
Gravatar

I emailed godville support to diversify their servers and move some into my data center. Multiple redundant fiber paths. Global access in south bend indiana. C’mon godville techs!!! Call me and I promise sh!t like this will NEVER happen again.

JtheProducer Sep 08 2011 03:58
Gravatar

Hasn’t worked since 1130 eastern time for me hopefully can be fixed quickly and possibly made preventable so it won’t happn again. Great job to game devs keep up the good work

CeNedra Sep 08 2011 04:00
Gravatar

Down for 36 hours, won’t let me on my computer, can’t check on my girl, plz fix soon

His Royal Dudeness Sep 08 2011 04:30
Gravatar

Its back up!!!!

Zorkon Sep 08 2011 04:35
Gravatar

It is about 10:30 PM CDT 9/7/11 at the time of this post, and I have not been able to get my app to refresh since 11:30 AM today (11 hours), and I’m NOT on Android; I’m on iPhone.

Zorkon Sep 08 2011 04:37
Gravatar

But i AM relieved to know it isn’t just my iPhone; I was afraid my iPhone had somehow screwed itself or its Internet connection up or something like that.

Isis Jane Sep 08 2011 04:47
Gravatar

We’re back! Thank you, Omnipotent Ones!

Bomu Sep 08 2011 05:10
Gravatar

Thanks fo fixing this. Much appreciated.

SPUSMC13 Sep 08 2011 06:05
Gravatar

Yay Hercules Hercules

CodexAngel Sep 08 2011 06:14
Gravatar

Huzzah! Thank you!

KingCrimson 6 Sep 08 2011 07:58
Gravatar

Phew! Thanks for the tip about twitter.

Hisoka Raizen Sep 10 2011 04:01
Gravatar

Glad to see this was acknowledged. As someone who downloaded it from Amazon, I can’t wait for the update.

Myoka 5 Sep 10 2011 23:25
Gravatar

Unfortunately my little guy died during the downtime was tough but thanks for getting things back up!

Joshua Millington Sep 10 2011 23:37
Gravatar

Facebook accumulator recharger isn’t working, i would like to donate to the game and got Facebook credits to do so but cannot purchase accumulator charges. error message is as follows: There Was a Problem Processing Your Payment Sorry, but we’re having trouble processing your payment. You have not been charged for this transaction. Please try again.

ive been trying for most of the day to no avail.

Whitetiger74 Sep 14 2011 16:28
Gravatar

I can’t even make a purchase keeps telling Me can’t do in timely fashion

Natalie w Sep 23 2011 03:56
Gravatar

My game is still stuck.When will android be back up?

KolomboK Oct 12 2011 09:40
Gravatar

Still takes about 30 secs to renew the game, and 2 out of 3 times it just says it is reloaded, but really it is not… The game is running VERY SLOW. It is very annoying that I have to “shout” something or encourage but actually have to wait for a long time for it to happen, renewing the app every 10-20 secs. And I already used 2 of my restore charges and 2 of my encourage charges (50% godpower both times) just trying to hit the button. The game is EXTRA slow, I know this may be just temporary, but quite frustrating. Especially for those who pay or donate for the charges…

Please login to leave comment

back to all posts