April 2008 Archives

Starting April 15th we started noticing poor performance overnight.  You've seen it, timeouts, 500 errors, gateway timeouts, etc.  We started our normal site stability processes to target the problem.  Stack traces were generated, memory dumps analyzed, etc.  This technique had served us well in diagnosing performance problems generating FOAF data for users with thousands of users.  At the same time we added 50% more servers to the pool.

In our analysis we quickly focused our attention to concurrency problems in some portions of our code.  We found that Java itself has some serious contention for global Character Set data and Crypto providers.  So we spent a lot of time working around these problems.  However we'd fix one concurrency problem, and quickly hit another one.  This went on for days, we'd think we had the problem solved, then it would come back with a vengeance. 

This past friday, the 25th we were dealing with the same issues.   We noticed that the poor performance coincided with notification database alerts. At 11:00 we turned off notifications and the entire system was suddenly stable.

With this breathing room we've now had time to finish a project to improve notifications scalability.  This code will go out on Sunday, with most applications having notifications functionality by Sunday.


Why Did This Happen?

We underestimated the amount of notifications sent, and the popularity of their use on the site.  At first glance this just meant that posting and browsing notifications were slow.  We didn't expect that other requests would suffer collateral damage.

Changes Going Forward...

Today we made the following Notifications changes.  Our goal is to get Notifications on for all applications while maintaining overall site stability:

• Accepting Notifications Asynchronously
• Applying Notification Retention policy to remove the oldest Notifications from the system (14 day retention)
• Adding Extra Notification Capacity
• Only allowing Notification REST calls with a token generated in the preceding 4 hours.
• Additional privacy controls to insure that the notification is legitimate.
• Conversations with our partners that are using the Notifications Feature the most.

Both Operations and Engineering have been happy to get a few good nights sleep since friday.  We look forward to working on new features instead of fighting fires.

One final note on platform status -- we know that stats were unavailable in the dev console this weekend and expect to have them back online tomorrow.

We have gotten many questions about how we rank applications in the app gallery and how an app developer can increase their exposure to users.  I'd like to offer a brief explanation about our current and long-term goals regarding application ranking.

    First of all, I wanted to state that our ultimate goal is to provide the richest, most engaging experience to our users.  To this end, we plan on offering application developers a variety of incentives/penalties for providing a positive/negative user experience.

    At launch we do not have much data to work with, so we decided to provide a randomized application gallery until we have enough data to provide a meaningful sort order of applications.  We feel that this will provide the most level playing field initially and let the applications speak for themselves.

    As soon as we switch over to using metrics to determine the sort order, we will offer an explanation as to how you can improve your engagement score.  The general guidelines we will follow should be fairly self-explanatory.  Actions such as installs, views and accepted invites will be positive influences, while rejected invites, blocked applications and removes will be negative influences.