Joel Liu's Library tagged Architecture → View Popular, Search in Google

- 16 million page views a month
- 3 million unique visitors a month (Facebook reaches 77 million unique visitors a month)
- 6 million visits a month
- 86% of traffic comes from Google
- 9 million active programmers in the world and 30% have used Stack Overflow.
- If you’re comfortable managing servers then buy them. The two biggest problems with renting costs were: 1) the insane cost of memory and disk upgrades 2) the fact that they [hosting providers] really couldn’t manage anything.
- Make larger one time up front investments to avoid recurring monthly costs which are more expensive in the long term.
3 more annotation(s)...
- Memory is incredibly cheap. Max it out for almost free performance. At Dell, for example, upgrading from 4G memory to 128G is $4378.
- Stack Overflow copied a key part of the Wikipedia database design. This turned out to be a mistake which will need massive and painful database refactoring to fix. The refactorings will be to avoid excessive joins in a lot of key queries. This is the key lesson from giant multi-terabyte table schemas (like Google’s BigTable) which are completely join-free. This is significant because Stack Overflow's database is almost completely in RAM and the joins still exact too high a cost.
- At low server volume, the key cost driver is not rackspace, power, bandwidth, servers, or software; it is NETWORKING EQUIPMENT. You need a gigabit network between your DB and Web tiers. Between the cloud and your web server, you need firewall, routing, and VPN devices. The moment you add a second web server, you also need a load balancing appliance. The upfront cost of these devices can easily be 2x the cost of a handful of servers.

Jul

2009

Gojko Adzic » QCon London 2009: Upgrading Twitter without service disruptions

gojko.net/...er-without-service-disruptions

- Preview

twitter architecture Performance

A very interesting observation during the talk was that Twitter started up with a CMS model and that they gradually moved towards a messaging model. I’ve seen this in a few applications so far, including a casino system, where the messaging model seems to fit best an application intended to power massive community of online users, it seems regardless of what the application actually does business wise. Applications start out completely different, but then more and more functionality gets bolted on top of user messaging capabilities that the whole architecture on the end gets refactored to utilise the messaging channels as the core information transport. With Twitter, I’d expect this to be more obvious from the start as it was intended to help people notify each other.
The interesting thing, however, was that all the upgrades were done live, without shutting down the system. The changes were always introduced to one node, then regression issues were sorted out, and then the software would be rolled out to the whole cluster. They went as far as building a whole messaging system based on memcached APIs in order to be able to slot in such changes.
3 more annotation(s)...
- a write-through vector cache of primary tweet keys with 99% hit ratio, a write-through row cache for tweets and users with 95% hit rate and a read-through fragment cache with rendered versions of different tweets for different clients with 95% hit rate. All these caches are based on memcached.
- this change allowed them to increase the web server performance from 3.32 requests per second without caching to 139.03 requests per second. Weaver said that API services work about four times faster than the web, which means that the API performance is roughly 550 requests/s [my calculation, not given during the talk].
- Web is only 10-20% of the traffic, the rest is through API services
  Web servers are still 50% of the cluster.
  Regular incoming traffic peaks are around 80 tweets per second. I expected this to be a lot more.
  before the upgrades their web servers shipped only 3.32 requests per second!
  For each tweet, message gets inserted for each user which follows a tweet. In average, a user has 120 followers so this comes to about 9600 messages/s at peak times
  Message servers run on three nodes. They decided to write their own messaging software in order to make the protocol memcached-like, and did not evaluate other available solutions.
  During Obama’s inauguration, they peaked at about 350 tweets per second for around five minutes.
  They had a ton of problems with garbage collection but strangelly haven’t looked into JRockit RT or something similar that has predictable GC. Twitter JVM middleware runs on the SUN JVM

InfoQ: Twitter, an Evolving Architecture

www.infoq.com/...Twitter-Architecture

- Preview

twitter architecture Performance

Most of the tools used by Twitter are open source. The stack is made up of Rails for the front side, C, Scala and Java for the middle business layer, and MySQL for storing data. Everything is kept in RAM and the database is just a backup.
1 more annotation(s)...

May

2009

Drop ACID and Think About Data | High Scalability

highscalability.com/drop-acid-and-think-about-data

- Preview

database scalability Architecture

Jul

2008

GoogleTalk Architecture | High Scalability

highscalability.com/googletalk-architecture

- Preview

Scalability Architecture google talk IM

Measure the right thing.
- People ask about how many IMs do you deliver or how many active users. Turns out not to be the right engineering question.
- Hard part of IM is how to show correct present to all connected users because growth is non-linear: ConnectedUsers * BuddyListSize * OnlineStateChanges
- A linear user grown can mean a very non-linear server growth which requires serving many billions of presence packets per day.
- Have a large number friends and presence explodes. The number IMs not that
big of deal.