Recently in Maintenance Category

Service on www.hi5.com is severely degraded as of 19:00 PM PST.

Update: 19:15 -- our network team is on site and diagnosing a network related issue.

Update 2: 21:00 -- No ETA to total resolution yet.  Our team, network consultants and vendors are diagnosing and fixing as fast as possible.

Update 3: 22:50 -- Problem appears to be isolated.  Preparing to resume full service.  Thanks for everyone's patience.

Update 4: 23:05 -- Service fully restored.

Today we've had intermittent performance degradation problems that may have led to you seeing features disabled.  We have managed to work around the problems and bring back site stability.  Our apologies for the disruption you may have experienced.

The Hi5 site was knocked offline at 1:50 PM Pacific by a power loss in our Data Center in San Francisco.  We lost about 1/4 of our servers which negatively impacted performance.  We have disabled access while we evaluate the impact of the outage.

Update (3:45 PM): Our engineering and operations team just finished checking all systems.  We are starting to restore access.  Thanks for your patience.

Update (4:10 PM): This outage knocked out other popular Internet services like LiveJournal, Craigslist, Yelp, and Technorati.  Here's some news coverage of the outage for the curious:

• At least 20,000 without power in downtown S.F.
• Power Outages?!
• Power outages hit 50,000 PG&E customers

A change in our core networking is causing temporary problems with Scraps and Groups on the site.  ETA for resolution is 18:35 Pacific Standard Time.  We'll update this post with more information as it comes in.

Update 18:40: Fives have returned, groups and scraps coming back online.  New ETA 19:00.
Update 17:10: All systems back to normal.  Bug fix release will be deployed shortly.
Update 23:59: Bug fix release is deployed 100%

You can find out more about this release in the Hi5 group - How can we make Hi5 better.  Here's a short synopsis of the fixes:

• Improved the rendering of profiles that have content from external sites.
  
• Profile backgrounds and custom styles are working again.
• You can now delete multiple messages in the Message Center.
• Removed the ability to send messages to your entire list of friends, as a spam control measure.
• Re-enabled transparency for content modules on the profile page.

We recieved a particularly nasty attack on our networking infrastructure today.  It caused much of the site to be unavailable from 5:30-7:30 Pacific Time.  I hate it when friday the 13th occurs on a wednesday.

We've worked with our networking and load balancing vendors to implement a somewhat permanent fix for this issue.