Scaling Twitter: Making Twitter 10000 Percent Faster | High Scalability - The Diigo Meta page

• 20 Dec 15
  

  markhuyong

  twitter architecture scalability performance

• 13 Jan 15
  

  fajenkyne

  twitter inspiration scalability architecture performance scaling optimization

• 27 Aug 14
  
• 08 Jul 13
  

  Pap Tom

  twitter scalability architecture

  • Use explain to how your queries are running. Indexes may not be being as you expect.
    - Denormalize a lot. Single handedly saved them. For example, they store all a user IDs friend IDs together, which prevented a lot of costly joins.
  • ong running processes should be abstracted to daemons.
  • Most performance comes not from the language, but from application design.

  1 more annotation...

• 10 Jun 13
  

  Liem Phan

  Twitter infrastructure

• 04 Dec 12
  

  Marc-Alexandre Gagnon

  twitter scalability architecture rails performance scaling optimization ruby

• 27 Mar 12
  

  Y A
  • Saturday, June 27, 2009 at 11:46PM
• 04 Feb 12
  

  devil tod
• 07 Jan 12
  

  Christophe Addinquy

  Architecture and evolution of Twitter

  twitter architecture performance

  • Started with DRb
  • Moved to Rinda
  • Moved to Starling
  • Keeping the service simple

  2 more annotations...

• 27 Sep 11
  

  Jochen Fromm

  Making Twitter 10000 Percent Faster | High Scalability

  scaling scalability ruby_on_rails rails performance tuning twitter architecture

• 26 Aug 11
  

  yasuhiro nishioka

  scalability twitter rails performance scaling architecture optimization ruby zootool_imported

• 29 Apr 11
  

  naim kazi

  twitter scalability architecture rails performance

• 22 Mar 11
  

  atirado0617

  joe

• 06 Mar 11
  

  Eduard

  to-read Development scalability architect

• 02 Mar 11
  

  erik pettersen

  performance twitter scaling optimization scalability architecture rails web tutorial howto

• 17 Jan 11
  

  smostafak
• 15 Dec 10
  

  Tanausú Cerdeña

  twitter scalability scaling architecture

• 11 Dec 10
  

  Michael Oczak

  twitter scalability performance architecture scaling optimization

• 04 Dec 10
  

  Heinz Wittenbrink

  Zur Entwicklung der Twitter-Infrastruktur

  twitter scala cloud computing scalability

• 26 Nov 10
  

  Seunghoon Yeon

  twitter preformance scalability

• 24 Nov 10
  

  Nolboo Kim

  twitter scalability performance optimization architecture

• 13 Sep 10
  

  hardimeybaum

  architecture performance ruby rubyonrails scalability scaling twitter

• 25 Aug 10
  

  rinaldi fonseca

  twitter scala

• 01 Mar 10
  

  Simon Reavely

  twitter scalability performance web optimization rails architecture scaling

• 30 Jan 10
  

  Jochen Bachmann

  twitter performance

• 04 Jan 10
  

  Jörg Ramb

  twitter performance programming scalability delicious

• 24 Nov 09
  

  Paulo Gaspar

  scalability architecture distributed computing interesting-very case-study _wrk _must-read

• 25 Sep 09
  

  paulosilveira

  twitter scalability scaling ruby rails performance

• 10 Aug 09
  

  kaspar_ru

  twitter toread

• 14 Jul 09
  

  alienspaces
• 08 Jul 09
  

  eduballester

  twitter scalability performance rails architecture scaling ruby

• 09 Jun 09
  

  alfred westerveld

  unread

• 02 Jun 09
  

  Kevin Makice

  Twitter API_Development scalability performance rails architecture scaling ruby delicious

• 01 Jun 09
  

  timwee

  scaling twitter architecture startup

• 31 May 09
  

  varkas

  web2.0 databases optimization

• 12 May 09
  

  Emran Hasan

  optimization performance development web database mysql scalability twitter

• 07 May 09
  

  Micah Elliott

  scaling

• 02 May 09
  

  bonkey

  rails performance scala twitter

• 27 Apr 09
  

  Sérgio Carvalho

  development scalability performance optimization web architecture ruby twitter

• 08 Apr 09
  

  Dan Creswell

  architecture scalability twitter

• 04 Apr 09
  

  goodday

  twitter architecture scaling design

• 30 Mar 09
  

  Alan Ho

  scalability architecture scaling tuning sysadmin ruby rails twitter performance rubyonrails

• 19 Mar 09
  

  Kent Yu

  scalability rails twitter Delicious

• 06 Mar 09
  

  okoze7

  web development ruby rails twitter scalability performance architecture

• 05 Feb 09
  

  ruby performance rails optimization scaling architecture twitter

• 27 Jan 09
  

  Chris Lasher

  A link-fest for Twitter's explanation of how they solved their scalability issues.

  web ruby twitter scalability rails

• 12 Oct 08
  

  kamtorus

  webdev web2.0 web twitter toread tips sysadmin server

• 17 Sep 08
  

  JOhn O'Sullivan

  scalability programming web2.0

• 25 Aug 08
  

  Petter Hareim

  Twitter started as a side project and blew up fast, going from 0 to millions of page views within a few terrifying months. Early design decisions that worked well in the small melted under the crush of new users chirping tweets to all their friends.

  webdev web2.0 web twitter toread sysadmin tips server scalability scaling

• 09 Aug 08
  

  thesuffixed

  scalability

• 30 Jul 08
  

  Hardy Heron

  twitter scaling webdev rubyonrails

• 22 Jul 08
  

  Bill Christian

  architecture

• 23 Jun 08
  

  Mark Masterson

  twitter web2.0 scalability ruby architecture enterprise+architecture enterprise2.0 cloudcomputing

• 10 Jun 08
  

  Harjeet Singh

  Scaling Twitter: Making Twitter 10000 Percent Faster
  Todd Hoff's picture
  Thu, 01/17/2008 - 16:08 — Todd Hoff
  
  * Scaling Twitter: Making Twitter 10000 Percent Faster (2219)
  
  Update 2: a commenter in Twitter Fails Macworld Keynote Test said this entry needs to be updated. LOL. My uneducated guess is it's not a language or architecture problem, but more a problem of not being able to add hardware fast enough into their data center. The predictability of this problem is debatable, but once you have it, it's hard to fix.
  Update: Twitter releases Starling - light-weight persistent queue server that speaks the MemCache protocol. It was built to drive Twitter's backend, and is in production across Twitter's cluster.
  
  Twitter started as a side project and blew up fast, going from 0 to millions of page views within a few terrifying months. Early design decisions that worked well in the small melted under the crush of new users chirping tweets to all their friends. Web darling Ruby on Rails was fingered early for the scaling problems, but Blaine Cook, Twitter's lead architect, held Ruby blameless:
  
  
  For us, it’s really about scaling horizontally - to that end, Rails and Ruby haven’t been stumbling blocks, compared to any other language or framework. The performance boosts associated with a “faster” language would give us a 10-20% improvement, but thanks to architectural changes that Ruby and Rails happily accommodated, Twitter is 10000% faster than it was in January.

  scalability architecture ruby rails frameworks grid

• 29 May 08
  

  andrew baldwin

  twitter scalability

• 22 May 08
  

  Jarek Jarek

  devel scaling

• 22 Apr 08
  

  C K

  Twitter Scalability Software_Engineering Programming ***** DIY Tutorials Reference Blogging SQL MySQL memcached Caching

• 31 Mar 08
  

  dschwarz

  twitter scalability

• 03 Mar 08
  

  Arrix Z

  performance scalability scaling twitter

• 13 Feb 08
  

  fulvius longhi
  • For us, it’s really about scaling horizontally - to that end, Rails and Ruby haven’t been stumbling blocks, compared to any other language or framework. The performance boosts associated with a “faster” language would give us a 10-20% improvement, but thanks to architectural changes that Ruby and Rails happily accommodated, Twitter is 10000% faster than it was in January.
  • Abuse
     - A lot of down time because people crawl the site and add everyone as friends. 9000 friends in 24 hours. It would take down the site.
     - Build tools to detect these problems so you can pinpoint when and where they are happening.
  • Abuse
     - A lot of down time because people crawl the site and add everyone as friends. 9000 friends in 24 hours. It would take down the site.
     - Build tools to detect these problems so you can pinpoint when and where they are happening.
     - Be ruthless. Delete them as users.
  • Talk to the community. Don't hide and try to solve all problems yourself. Many brilliant people are willing to help if you ask.
  • Cache the hell out of everything. Individual active records are not cached, yet. The queries are fast enough for now.
  • Turn your website into an open service by creating an API. Their API is a huge reason for Twitter's success. It allows user's to create an ever expanding and ecosystem around Twitter that is difficult to compete with. You can never do all the work your user's can do and you probably won't be as creative. So open you application up and make it easy for others to integrate your application with theirs.

  4 more annotations...

• 11 Feb 08
  

  beaconlight

  2b looked @ twitter

• 17 Jan 08
  

  mumumu

  Plan to partition in the future. Currently they don't. て時点で終わってる気が(´ー｀; )

  api business database column

• 04 Nov 07
  

  alex de carvalho

  Twitter started as a side project and blew up fast, going from 0 to millions of page views within a few terrifying months. Early design decisions that worked well in the small melted under the crush of new users chirping tweets to all their friends.

  howto programming strategy tech tricks twitter webdev stats community advice

• 25 Oct 07
  

  F H

  architecture performance rails ruby programming

• 10 Oct 07
  

  Ivo Frazão

  ruby performance scalability from_delicious

• 

  François Charoy

  architecture scalability scaling twitter performance pcal

• 06 Oct 07
  

  Mathieu Martin

  great scalability rails performance delicious

• 05 Oct 07
  

  losamorales g

  del.icio.us imported performance rails scalability twitter

• 

  architecture performance development rails scalability

• 04 Oct 07
  

  zigorou

  どう考えてもマニアック過ぎるけど、そんなkuたんが好き＞＜

  api architecture article business caching clustering coding coldfusion optimization ruby scaling performance rails scalability twitter

• 27 Sep 07
  

  Renato Albano

  scalability twitter rails scaling performance architecture optimization

• 

  hideaki-t

  scalability performance architecture

• 26 Sep 07
  

  Webdesign tuning performance server scalability

• 21 Sep 07
  

  clarify

  rails scaling tips howto memcached optimization performance architecture twitter

• 20 Sep 07
  

  Brent Sordyl

  twitter scalability rubyonrails

• 

  Pedro Trindade

  scalability twitter rails performance scaling architecture optimization ruby advice api article articles blogging coldfusion business

• 

  Geo Philips

  twitter rubyonrails scalability web

• 

  steve t

  twitter

• 

  Terry Jones

  scaling twitter

• 18 Sep 07
  

  Vishnu Gopal

  twitter scalability architecture delicious_import

• 

  Joel Price

  article ia data scalability twitter architecture webdev

• 17 Sep 07
  

  Marcos Pereira

  architecture articles performance twitter scalability ruby rails

• 

  Scott Converse

  scalability twitter rails performance scaling optimization architecture

• 

  Mike Barone

  performance optimization scalability toread twitter from:del.icio.us

• 

  Gary Burge

  Twitter started as a side project and blew up fast, going from 0 to millions of page views within a few terrifying months. Early design decisions that worked well in the small melted under the crush of new users chirping tweets to all their friends.

  Twitter RubyOnRails scale business systemdesign architecture

• 16 Sep 07
  

  Gabriela Grosseck

  twitter

• 

  bkeepers

  twitter scaling rails performance

• 

  Jason Reis

  scalability rails twitter performance

• 

  Aníbal Rojas

  caching cached memcached messaging mongrel munin mysql optimization performance rails scalability scaling scale tips tuning twitter architecture

• 15 Sep 07
  

  adam smith

  rubyonrails

• 

  aj

  optimization scalability rails architecture

• 

  Thomas Vander Wal

  Twitter folks explain how they scaled their system

  tech twitter web webdev clustering coding database development howto mysql optimization programming rails ruby rubyonrails server sysadmin

• 

  Steve Goguen

  Lessons Twitter learned when they needed to scale

  architecture scalability rails performance

• 

  Thomas Neumann

  scalability rails performance scaling twitter architecture ruby delicious

• 

  Chrys R

  twitter scalability optimization architecture performance

• 

  A. D.

  performance

• 

  paulcowles

  interesting article on scaling Twitter, lots of good ideas

  scalability rails mysql deployment

• 

  Ken Wei

  scaling twitter

• 

  Matt Constantine

  rails scalability twitter performance interesting

• 14 Sep 07
  

  Wade Ren

  performance scalability

  • • Use caching with memcached a lot.
       - For example, if getting a count is slow, you can memoize the count into memcache in a millisecond.
       - Getting your friends status is complicated. There are security and other issues. So rather than doing a query, a friend's status is updated in cache instead. It never touches the database. This gives a predictable response time frame (upper bound 20 msecs).
       - ActiveRecord objects are huge so that's why they aren't cached. So they want to store critical attributes in a hash and lazy load the other attributes on access.
  • - Send message to invalidate friend's cache in the background instead of doing all individually, synchronously.
  • Moved to Starling, a distributed queue written in Ruby.
     - Distributed queues were made to survive system crashes by writing them to disk. Other big websites take this simple approach as well.
  • Abuse
     - A lot of down time because people crawl the site and add everyone as friends. 9000 friends in 24 hours. It would take down the site.
  • - Be ruthless. Delete them as users.
  • Build in user limits. People will try to bust your system. Put in reasonable limits and detection mechanisms to protect your system from being killed.
  • For example, they store all a user IDs friend IDs together, which prevented a lot of costly joins.

  5 more annotations...

Page Comments

• 
  annabay21 on 2007-09-17
  Cool...I love Twitter...and it is a pleasure that I can enhance it!!!

Would you like to comment?

Join Diigo for a free account, or sign in if you are already a member.