This link has been bookmarked by 44 people . It was first bookmarked on 14 Sep 2007, by someone privately.
-
10 Aug 09
-
09 Jun 09
-
04 Apr 09
-
27 Jan 09
Chris LasherA link-fest for Twitter's explanation of how they solved their scalability issues.
-
22 Jul 08
-
21 Jul 08
-
29 Jun 08
-
24 Jun 08
-
10 Jun 08
Harjeet SinghScaling Twitter: Making Twitter 10000 Percent Faster
Todd Hoff's picture
Thu, 01/17/2008 - 16:08 — Todd Hoff
* Scaling Twitter: Making Twitter 10000 Percent Faster (2219)
Update 2: a commenter in Twitter Fails Macworld Keynote Test said this entry needs to be updated. LOL. My uneducated guess is it's not a language or architecture problem, but more a problem of not being able to add hardware fast enough into their data center. The predictability of this problem is debatable, but once you have it, it's hard to fix.
Update: Twitter releases Starling - light-weight persistent queue server that speaks the MemCache protocol. It was built to drive Twitter's backend, and is in production across Twitter's cluster.
Twitter started as a side project and blew up fast, going from 0 to millions of page views within a few terrifying months. Early design decisions that worked well in the small melted under the crush of new users chirping tweets to all their friends. Web darling Ruby on Rails was fingered early for the scaling problems, but Blaine Cook, Twitter's lead architect, held Ruby blameless:
For us, it’s really about scaling horizontally - to that end, Rails and Ruby haven’t been stumbling blocks, compared to any other language or framework. The performance boosts associated with a “faster” language would give us a 10-20% improvement, but thanks to architectural changes that Ruby and Rails happily accommodated, Twitter is 10000% faster than it was in January. -
03 Mar 08
-
13 Feb 08
-
For us, it’s really about scaling horizontally - to that end, Rails and Ruby haven’t been stumbling blocks, compared to any other language or framework. The performance boosts associated with a “faster” language would give us a 10-20% improvement, but thanks to architectural changes that Ruby and Rails happily accommodated, Twitter is 10000% faster than it was in January.
-
Abuse
- A lot of down time because people crawl the site and add everyone as friends. 9000 friends in 24 hours. It would take down the site.
- Build tools to detect these problems so you can pinpoint when and where they are happening. - 4 more annotations...
-
-
Abuse
- A lot of down time because people crawl the site and add everyone as friends. 9000 friends in 24 hours. It would take down the site.
- Build tools to detect these problems so you can pinpoint when and where they are happening.
- Be ruthless. Delete them as users. -
Talk to the community. Don't hide and try to solve all problems yourself. Many brilliant people are willing to help if you ask.
-
Cache the hell out of everything. Individual active records are not cached, yet. The queries are fast enough for now.
-
Turn your website into an open service by creating an API. Their API is a huge reason for Twitter's success. It allows user's to create an ever expanding and ecosystem around Twitter that is difficult to compete with. You can never do all the work your user's can do and you probably won't be as creative. So open you application up and make it easy for others to integrate your application with theirs.
-
-
-
11 Feb 08
-
18 Jan 08
-
05 Oct 07
-
27 Sep 07
-
600 requests per second.
-
Average 200-300 connections per second. Spiking to 800 connections per second.
- 2 more annotations...
-
-
MySQL handled 2,400 requests per second.
-
180
-
-
-
20 Sep 07
-
18 Sep 07
-
17 Sep 07
-
Gary BurgeTwitter started as a side project and blew up fast, going from 0 to millions of page views within a few terrifying months. Early design decisions that worked well in the small melted under the crush of new users chirping tweets to all their friends.
Twitter RubyOnRails scale business systemdesign architecture
-
16 Sep 07
-
15 Sep 07
-
Thomas Vander WalTwitter folks explain how they scaled their system
tech twitter web webdev clustering coding database development howto mysql optimization programming rails ruby rubyonrails server sysadmin
-
14 Sep 07
-
- Use caching with memcached a lot.
- For example, if getting a count is slow, you can memoize the count into memcache in a millisecond.
- Getting your friends status is complicated. There are security and other issues. So rather than doing a query, a friend's status is updated in cache instead. It never touches the database. This gives a predictable response time frame (upper bound 20 msecs).
- ActiveRecord objects are huge so that's why they aren't cached. So they want to store critical attributes in a hash and lazy load the other attributes on access.
- Use caching with memcached a lot.
-
- Send message to invalidate friend's cache in the background instead of doing all individually, synchronously.
- 5 more annotations...
-
-
Moved to Starling, a distributed queue written in Ruby.
- Distributed queues were made to survive system crashes by writing them to disk. Other big websites take this simple approach as well. -
Abuse
- A lot of down time because people crawl the site and add everyone as friends. 9000 friends in 24 hours. It would take down the site. -
- Be ruthless. Delete them as users.
-
Build in user limits. People will try to bust your system. Put in reasonable limits and detection mechanisms to protect your system from being killed.
-
For example, they store all a user IDs friend IDs together, which prevented a lot of costly joins.
-
-
-
-
For us, it’s really about scaling horizontally - to that end, Rails and Ruby haven’t been stumbling blocks, compared to any other language or framework. The performance boosts associated with a “faster” language would give us a 10-20% improvement, but thanks to architectural changes that Ruby and Rails happily accommodated, Twitter is 10000% faster than it was in January.
-
-
-
Update 6: Some interesting changes from Twitter's Evan Weaver: everything in RAM now, database is a backup; peaks at 300 tweets/second; every tweet followed by average 126 people; vector cache of tweet IDs; row cache; fragment cache; page cache; keep separate caches; GC makes Ruby optimization resistant so went with Scala; Thrift and HTTP are used internally; 100s internal requests for every external request; rewrote MQ but kept interface the same; 3 queues are used to load balance requests; extensive A/B testing for backwards capability; switched to C memcached client for speed; optimize critical path; faster to get the cached results from the network memory than recompute them locally.
-
Page Comments
Would you like to comment?
Join Diigo for a free account, or sign in if you are already a member.