Skip to main content

Joel Liu's Library tagged Scalability →  View Popular

17 Jul 09

Stack Overflow Architecture | High Scalability

    • 16 million page views a month

    • 3 million unique visitors a month (Facebook reaches 77 million unique visitors a month)

    • 6 million visits a month

    • 86% of traffic comes from Google

    • 9 million active programmers in the world and 30% have used Stack Overflow.
    • If you’re comfortable managing servers then buy them. The two biggest problems with renting costs were: 1) the insane cost of memory and disk upgrades 2) the fact that they [hosting providers] really couldn’t manage anything.

    • Make larger one time up front investments to avoid recurring monthly costs which are more expensive in the long term.
  • 3 more annotations...
03 Jul 09

Scaling Twitter: Making Twitter 10000 Percent Faster | High Scalability

  • Update 6: Some interesting changes from Twitter's Evan Weaver: everything in RAM now, database is a backup; peaks at 300 tweets/second; every tweet followed by average 126 people; vector cache of tweet IDs; row cache; fragment cache; page cache; keep separate caches; GC makes Ruby optimization resistant so went with Scala; Thrift and HTTP are used internally; 100s internal requests for every external request; rewrote MQ but kept interface the same; 3 queues are used to load balance requests; extensive A/B testing for backwards capability; switched to C memcached client for speed; optimize critical path; faster to get the cached results from the network memory than recompute them locally.

GoogleTalk Architecture | High Scalability

  • Measure the right thing.

    - People ask about how many IMs do you deliver or how many active users. Turns out not to be the right engineering question.

    - Hard part of IM is how to show correct present to all connected users because growth is non-linear: ConnectedUsers * BuddyListSize * OnlineStateChanges

    - A linear user grown can mean a very non-linear server growth which requires serving many billions of presence packets per day.

    - Have a large number friends and presence explodes. The number IMs not that

    big of deal.
13 Dec 08

Engineering @ Facebook's Notes | Facebook

  • If you've read anything about scaling large websites, you've probably heard about memcached. memcached is a high-performance, distributed memory object caching system. Here at Facebook, we're likely the world's largest user of memcached. We use memcached to alleviate database load. memcached is already fast, but we need it to be faster and more efficient than most installations. We use more than 800 servers supplying over 28 terabytes of memory to our users. Over the past year as Facebook's popularity has skyrocketed, we've run into a number of scaling issues. This ever increasing demand has required us to make modifications to both our operating system and memcached to achieve the performance that provides the best possible experience for our users.
21 Jan 08

Database War Stories #3: Flickr

  • "started as a big vertically scaled classic replication tree (well, started as a
    single box) and soon hit against performance walls in terms of writes (repl
    gives you more reads, same writes as slowest machine in your tree). when we
    started, we barely knew anything about how mysql works with indexes (the
    documentation is fairly non-beginner), so months of index wrangling to get
    better performance ensued.
29 Jul 07

Kitchen Soap » Blog Archive » Slides from ‘Capacity Planning for LAMP’ talk at MySQL Conf 2007

  • This was a fun talk. I saw a lot of nods in the audience when I mentioned things pertaining to social

O'Reilly Radar > Web 2.0 and Databases Part 1: Second Life

  • From my end, the worst MySQL moment was when, in the midst of a colo move we decided that we could bring the system back up before we had moved our slave database. After all, what are the odds of the primary going down in the 2 hours it would take to schlep the slave over and bring it up? Apparently the odds were 100%.
  • Like everybody else, we started with One Database All Hail The Central Database, and have subsequently been forced into clustering. However, we've eschewed any of the general purpose cluster technologies (mysql cluster, various replication schemes) in favor of explicit data partitioning. So, we still have a central db that keeps track of where to find what data (per-user, for instance), and N additional dbs that do the heavy lifting. Our feeling is that this is ultimately far more scalable than black-box clustering. Right now we're still in the transition process, so we remain vulnerable to overload. As Cory mentioned, we're moving to an HTTP-based internal communication model in order to improve our flexibility.
1 - 20 of 33 Next ›
Showing 20▼ items per page

Selected Tags

Related Tags

Diigo is about better ways to research, share and collaborate on information. Learn more »

Join Diigo