Ken Wei on 2009-02-27
11111111111
Joel Liu's personal annotations on this page
Joel bookmarked
on 2009-02-27
-
As our database has grown, we have tried to iteratively deal with the scaling issues that come with rapid growth. We did the typical things, like using read slaves and memcache to increase read throughput and sharding our database to improve write throughput. However, as we grew, scaling our existing features to accomodate more traffic turned out to be much less of an issue than adding new features.
This link has been bookmarked by 88 people . It was first bookmarked on 27 Feb 2009, by Ted Louie.
-
-
making schema changes or adding indexes to a database with more than 10 - 20 million rows completely locks the database for hours at a time. Removing old indexes takes just as much time, and not removing them hurts performance because the database will continue to read and write to those unused blocks on every
INSERT, pushing important blocks out of memory -
We can change the "schema" simply by storing new properties.
- 3 more annotations...
-
-
-
Our datastore stores schema-less bags of properties (e.g., JSON objects or Python dictionaries).
-
-
Navneet Kumara "schema-less" storage system on top of MySQL
-
making schema changes or adding indexes to a database with more than 10 - 20
million rows completely locks the database for hours at a time. Removing old
indexes takes just as much time, and not removing them hurts performance because
the database will continue to read and write to those unused blocks on everyINSERT, pushing important blocks out of memory -
InnoDB stores data rows physically in primary key order. The
AUTO_INCREMENTprimary key ensures new entities are written
sequentially on disk after old entities, which helps for both read and write
locality (new entities tend to be read more frequently than old entities since
FriendFeed pages are ordered reverse-chronologically
-
-
Wade Rensharding our database
-
A DWe use MySQL for storing all of the data in FriendFeed. Our database has grown a lot as our user base has grown. We now store over 250 million entries and a bunch of other data, from comments and "likes" to friend lists
-
-
JSON objects
-
If we want to index three properties in each entity, we will have three MySQL tables - one for each index
-
-
Chris LasherBrett Taylor on how FriendFeed wrestled with MySQL in adding features, and finally decided to implement a "schema-less" database, complete with high level overview of how they actually did this. It's exciting to see a site that actually is interested in d
-
-
However, as we grew, scaling our existing features to accomodate more traffic turned out to be much less of an issue than adding new features.
-
making schema changes or adding indexes to a database with more than 10 - 20 million rows completely locks the database for hours at a time.
- 8 more annotations...
-
-
Lantis Gaius CapistranoSchema-less Relational DBMS ~ That's one way to speed up LINQ ^^
-
Rajan DattaInteresting article - detailed look on FriendFeed's data management approach. Relies on MySQL
-
Jaroslaw Zabiello"schema-less" storage system on top of MySQL
-
Daniel AndrlikThis is a fascinating read by those of you who, like myself, are particularly interested in approaches to scaling applications at the database level. According to this post, FriendFeed has implemented a very interesting schema-less model for storing data within MySQL.
From the article:
>As our database has grown, we have tried to iteratively deal with the scaling issues that come with rapid growth. We did the typical things, like using read slaves and memcache to increase read throughput and sharding our database to improve write throughput. However, as we grew, scaling our existing features to accomodate more traffic turned out to be much less of an issue than adding new features.
>
> [snip]
>
>Our datastore stores schema-less bags of properties (e.g., JSON objects or Python dictionaries). The only required property of stored entities is id, a 16-byte UUID. The rest of the entity is opaque as far as the datastore is concerned. We can change the "schema" simply by storing new properties.
>
>We index data in these entities by storing indexes in separate MySQL tables. If we want to index three properties in each entity, we will have three MySQL tables - one for each index. If we want to stop using an index, we stop writing to that table from our code and, optionally, drop the table from MySQL. If we want a new index, we make a new MySQL table for that index and run a process to asynchronously populate the index without disrupting our live service.
>
>As a result, we end up having more tables than we had before, but adding and removing indexes is easy. We have heavily optimized the process that populates new indexes (which we call "The Cleaner") so that it fills new indexes rapidly without disrupting the site. We can store new properties and index them in a day's time rather than a week's time, and we don't need to swap MySQL masters and slaves or do any other scary operational work to make it happen.
Make sure to read the article for a lot of design details, it is a very interesting approach to scaling.-
Our datastore stores schema-less bags of properties (e.g., JSON objects or Python dictionaries). The only required property of stored entities is
id, a 16-byte UUID. The rest of the entity is opaque as far as the datastore is concerned. We can change the "schema" simply by storing new properties.We index data in these entities by storing indexes in separate MySQL tables. If we want to index three properties in each entity, we will have three MySQL tables - one for each index. If we want to stop using an index, we stop writing to that table from our code and, optionally, drop the table from MySQL. If we want a new index, we make a new MySQL table for that index and run a process to asynchronously populate the index without disrupting our live service.
As a result, we end up having more tables than we had before, but adding and removing indexes is easy. We have heavily optimized the process that populates new indexes (which we call "The Cleaner") so that it fills new indexes rapidly without disrupting the site. We can store new properties and index them in a day's time rather than a week's time, and we don't need to swap MySQL masters and slaves or do any other scary operational work to make it happen.
-
We have optimized our primary indexes quite a bit in this new system, and we are quite pleased with the results. Here is a graph of FriendFeed page view latency for the past month (we launched the new backend a couple of days ago, as you can tell by the dramatic drop):
-
-
-
As our database has grown, we have tried to iteratively deal with the scaling issues that come with rapid growth. We did the typical things, like using read slaves and memcache to increase read throughput and sharding our database to improve write throughput. However, as we grew, scaling our existing features to accomodate more traffic turned out to be much less of an issue than adding new features.
-
-
-
Tristan RivoallanWe use MySQL for storing all of the data in FriendFeed. Our database has grown a lot as our user base has grown. We now store over 250 million entries and a bunch of other data, from comments and "likes" to friend lists.
Public Stiky Notes
Page Comments
Would you like to comment?
Join Diigo for a free account, or sign in if you are already a member.