Stack Overflow Architecture | High Scalability - The Diigo Meta page

• 26 Sep 09
  

  fullness Time

  comp hosting architecture hardware

• 26 Aug 09
  

  Fumiaki Nagai

  webdeb scalability

• 24 Jul 09
  

  scitlec

  programming

  • http://www.peakinternet.com/
  • 128G is $4378
• 23 Jul 09
  

  Rajan Datta

  scalability architecture microsoft application development

• 20 Jul 09
  

  alfred westerveld
• 18 Jul 09
  

  W. Sarka

  stackoverflow joel spolsky scalability stack overflow java vs. c# .net jeff atwood

• 

  Graham Perrin

  architecture scalability stackoverflow webdev infrastructure unread 2009

• 

  Lachlan Gemmell
• 17 Jul 09
  

  David Burns

  architecture scalability stackoverflow programming infrastructure scaling

• 

  Liudvikas Bukys

  arch

• 

  Joel Liu

  architecture scalability

  • • 16 million page views a month
      
      
    • 3 million unique visitors a month (Facebook reaches 77 million unique visitors a month)
      
      
    • 6 million visits a month
      
      
    • 86% of traffic comes from Google
      
      
    • 9 million active programmers in the world and 30% have used Stack Overflow.
      
  • • If you’re comfortable managing servers then buy them. The two biggest problems with renting costs were: 1) the insane cost of memory and disk upgrades 2) the fact that they [hosting providers] really couldn’t manage anything.
      
      
    • Make larger one time up front investments to avoid recurring monthly costs which are more expensive in the long term.
  • 3 more annotations...
  • • Memory is incredibly cheap. Max it out for almost free performance. At Dell, for example, upgrading from 4G memory to 128G is $4378.
    • Stack Overflow copied a key part of the Wikipedia database design. This turned out to be a mistake which will need massive and painful database refactoring to fix. The refactorings will be to avoid excessive joins in a lot of key queries. This is the key lesson from giant multi-terabyte table schemas (like Google’s BigTable) which are completely join-free. This is significant because Stack Overflow's database is almost completely in RAM and the joins still exact too high a cost.
    • At low server volume, the key cost driver is not rackspace, power, bandwidth, servers, or software; it is NETWORKING EQUIPMENT. You need a gigabit network between your DB and Web tiers. Between the cloud and your web server, you need firewall, routing, and VPN devices. The moment you add a second web server, you also need a load balancing appliance. The upfront cost of these devices can easily be 2x the cost of a handful of servers.
• 

  Markus Vole

  architecture scalability stackoverflow performance scaling optimization infrastructure programming microsoft webdev

• 

  Anglea Dipps

  architecture scalability stackoverflow performance scaling optimization infrastructure programming microsoft webdev

• 

  John Bumpsey

  architecture scalability stackoverflow performance scaling optimization infrastructure programming microsoft webdev

• 

  quintoelefante
• 16 Jul 09
  

  sven duzont

  scalability architecture stackoverflow

• 

  JingleMan Sweep

  stackoverflow arch architecture scalability performance scaling optimization programming infrastructure microsoft c# .net asp.net mvc sqlserver sql server ec2 article windows delicious

• 

  Kenyth Zeng

  stackoverflow scalability architecture business-model

  • The publicity these days is all going scale out using NoSQL databases.
  • • 16 million page views a month
      
      
    • 3 million unique visitors a month (Facebook reaches 77 million unique visitors a month)
      
      
    • 6 million visits a month
      
      
    • 86% of traffic comes from Google
      
      
    • 9 million active programmers in the world and 30% have used Stack Overflow.
      
  • 17 more annotations...
  • • develop StackExchange as a white label and self hosted version of Stack Overflow, and perhaps develop some sort of programmer rating system.
    • 48 GB RAM
    • SQL Server's full text search is used extensively for the site search and detecting if a question has already been asked. Lucene.net is considered an attractive alternative.
    • At Dell, for example, upgrading from 4G memory to 128G is $4378.
    • CPU speed is surprisingly important to the database server.
    • The bottleneck is the database 90% of the time.
    • it is NETWORKING EQUIPMENT.
    • RAID-10 is awesome in a heavy read/write database workload.
    • they use a two-tier architecture where they access the database directly from the web server code.
    • It's hard to separate out the Microsoft stack and the scale up approach because for licensing reasons they tend to go together.
    • Scaling Up vs. Scaling Out and Server Hosting — Rent vs. Buy?.
    • There's no Group By or Order By. There are no cursors for easy paging through result sets.
    • The biggest problem of all is that transactions can not span arbitrary boundaries.
    • no ACID guarantees
    • Every copy of denormalized data must be manually tracked and updated taking into account the possibility of partial failures and externally visible inconsistency.
    • they basically use very little of Oracle and have written their own table structure, indexing and query processor on top of Oracle.
    • All data from all customers is mapped into the same data table, including indexes.
• 

  Florin Machidon
• 

  Marc Gemis

  stackoverflow architecture software website scalability

• 

  Artem Russakovskii

  scalability performance architecture optimization stackoverflow microsoft scaling serverfault superuser beyondcompare asp .net sql server 2008 vs visual studio jquery c# svn spolsky atwood codinghorror joelonsoftware joel

Would you like to comment?

Join Diigo for a free account, or sign in if you are already a member.