Mike Chelen's Library tagged file → View Popular

Filter:
All
Annotated

19 Nov 08

USENIX IMC '05 Technical Paper

Existing studies on BitTorrent systems are single-torrent based, while more than 85% of all peers participate in multiple torrents according to our trace analysis. In addition, these studies are not sufficiently insightful and accurate even for single-torrent models, due to some unrealistic assumptions. Our analysis of representative BitTorrent traffic provides several new findings regarding the limitations of BitTorrent systems: (1) Due to the exponentially decreasing peer arrival rate in reality, service availability in such systems becomes poor quickly, after which it is difficult for the file to be located and downloaded. (2) Client performance in the BitTorrent-like systems is unstable, and fluctuates widely with the peer population. (3) Existing systems could provide unfair services to peers, where peers with high downloading speed tend to download more and upload less. In this paper, we study these limitations on torrent evolution in realistic environments. Motivated by the analysis and modeling results, we further build a graph based multi-torrent model to study inter-torrent collaboration. Our model quantitatively provides strong motivation for inter-torrent collaboration instead of directly stimulating seeds to stay longer. We also discuss a system design to show the feasibility of multi-torrent collaboration.

www.usenix.org/...guo_html - Preview

bit torrent p2p file sharing

05 Nov 08

Index of /~mitch/linux/gitfs

gitfs is a FUSE-based filesystem for working with source trees stored in git repositories. The eventual goal is to provide a convenient way to work with lots of branches and patches. Currently only very basic functionality is implemented -- read-only access to the existing tags and objects. Still, it's already a somewhat handy tool which is why I'm releasing early snapshots.

www.sfgoth.com/gitfs - Preview

git fuse file system

02 Nov 08

31 Oct 08

Bitzi

bitzi.com - Preview

p2p file share

Openomy - The Online File System - Organizing your web based world

www.openomy.com - Preview

online storage file api

Tahoe Storage Illustration

bigasterisk.com/tahoe-playground - Preview

guide interactive distribute file system parity tahoe

30 Oct 08

Slyck.com - File Sharing, Digital Media and Tech News

www.slyck.com - Preview

p2p file sharing news

26 Oct 08

Zamzar - Free online file conversion

www.zamzar.com - Preview

online file convert

18 Oct 08

Amazon's Dynamo - All Things Distributed

www.allthingsdistributed.com/...amazons_dynamo.html - Preview

p2p distribute file system

it was imperative for us to avoid routing
requests through multiple nodes (which is the typical design adopted by several
distributed hash table systems such as Chord and Pastry)
Dynamo can be characterized as a zero-hop DHT, where
each node maintains enough routing information locally to route a request to
the appropriate node directly.
48 more annotations...
- Dynamo stores objects associated with a key through a simple
  interface; it exposes two operations: get() and put().
- The get(key)
  operation locates the object replicas associated with the key in the
  storage system and returns a single object or a list of objects with
  conflicting versions along with a context.
- The put(key, context,
  object) operation determines where the replicas of the object should
  be placed based on the associated key, and writes the replicas to
  disk.
- The context encodes system metadata about the object that is
  opaque to the caller and includes information such as the version of the
  object. The context information is stored along with the object so that the
  system can verify the validity of the context object supplied in the put
  request.
- Dynamo’s
  partitioning scheme relies on consistent hashing to distribute the load across
  multiple storage hosts.
- In consistent hashing [10], the output
  range of a hash function is treated as a fixed circular space or “ring” (i.e.
  the largest hash value wraps around to the smallest hash value).
- The principle advantage
  of consistent hashing is that departure or arrival of a node only affects its
  immediate neighbors and other nodes remain unaffected.
- The basic consistent hashing algorithm presents some
  challenges. First, the random position assignment of each node on
  the ring leads to non-uniform data and load distribution. Second, the
  basic algorithm is oblivious to the heterogeneity in the performance of nodes. To
  address these issues, Dynamo uses a variant of consistent hashing (similar to
  the one used in [10, 20]): instead of mapping a node to a single point in the
  circle, each node gets assigned to multiple points in the ring. To this end,
  Dynamo uses the concept of “virtual nodes”. A virtual node looks like a single
  node in the system, but each node can be responsible for more than one virtual
  node. Effectively, when a new node is added to the system, it is assigned
  multiple positions (henceforth, “tokens”) in the ring.
- Using virtual nodes has the following advantages:
  
  If a node becomes unavailable (due to failures or
  routine maintenance), the load handled by this node is evenly dispersed across
  the remaining available nodes.
  
  When a node becomes available again, or a new node
  is added to the system, the newly available node accepts a roughly equivalent
  amount of load from each of the other available nodes.
  
  The number of virtual nodes that a node is responsible
  can decided based on its capacity, accounting for heterogeneity in the physical
  infrastructure.
- Each data item is replicated at N hosts,
  where N is a parameter configured “per-instance”.
- Each key, k,
  is assigned to a coordinator node (described in the previous section). The
  coordinator is in charge of the replication of the data items that fall within
  its range. In addition to locally storing each key within its range, the
  coordinator replicates these keys at the N-1 clockwise successor nodes in the
  ring.
- In Figure 2, node B replicates
  the key k at nodes C and D in addition to storing it locally. Node D
  will store the keys that fall in the ranges (A, B], (B, C], and (C, D].
- The list of nodes that is responsible for storing a
  particular key is called the preference list.
- every node in the system can
  determine which nodes should be in this list for any particular key
- it is possible that the first
  N successor positions for a particular key may be owned by less than N distinct
  physical nodes
- To address this, the preference list for a key is constructed by skipping
  positions in the ring to ensure that the list contains only distinct physical
  nodes.
- Dynamo provides eventual consistency, which allows for
  updates to be propagated to all replicas asynchronously.
- A put() call may return
  to its caller before the update has been applied at all the replicas, which can
  result in scenarios where a subsequent get() operation may return an object
  that does not have the latest updates.
- If there are no failures then there is
  a bound on the update propagation times.
- However, under certain failure
  scenarios (e.g., server outages or network partitions), updates may not arrive
  at all replicas for an extended period of time.
- For example, the shopping cart application requires
  that an “Add to Cart” operation can never be
  forgotten or rejected.
- If the most recent state of the cart is unavailable, and
  a user makes changes to an older version of the cart, that change is still
  meaningful and should be preserved. But at the same time it shouldn’t supersede
  the currently unavailable state of the cart, which itself may contain changes
  that should be preserved.
- both “add to cart” and “delete
  item from cart” operations are translated into put requests to Dynamo
- When
  a customer wants to add an item to (or remove from) a shopping cart and the
  latest version is not available, the item is added to (or removed from) the older
  version and the divergent versions are reconciled later.
- Dynamo treats
  the result of each modification as a new and immutable version of the data
- Most of the time, new versions subsume the previous version(s), and
  the system itself can determine the authoritative version (syntactic
  reconciliation).
- However, version branching may happen, in the presence of
  failures combined with concurrent updates, resulting in conflicting versions of
  an object
- In these cases, the system cannot reconcile the multiple versions of
  the same object and the client must perform the reconciliation in order to collapse
  multiple branches of data evolution back into one (semantic reconciliation).
- A
  typical example of a collapse operation is “merging” different versions of a
  customer’s shopping cart.
- Using this reconciliation mechanism, an “add to cart”
  operation is never lost. However, deleted items can resurface.
- It is important to understand that certain failure modes can
  potentially result in the system having not just two but several versions of
  the same data.
- Updates in the presence of network partitions and node failures
  can potentially result in an object having distinct version sub-histories,
  which the system will need to reconcile in the future.
- This requires us to
  design applications that explicitly acknowledge the possibility of multiple
  versions of the same data (in order to never lose any updates).
- Dynamo uses vector clocks [12] in order to capture causality
  between different versions of the same object.
- One vector clock is associated with every version
  of every object. One can determine whether two versions of an object are on
  parallel branches or have a causal ordering, by examine their vector clocks.
- A vector clock is effectively a
  list of (node, counter) pairs.
- If the counters on the first object’s clock are less-than-or-equal to all
  of the nodes in the second clock, then the first is an
  ancestor of the second and can be forgotten.
- Otherwise, the two changes are
  considered to be in conflict and require reconciliation.
- In Dynamo, when a client wishes to update an object, it must
  specify which version it is updating.
- This is done by passing the context it
  obtained from an earlier read operation, which contains the vector clock
  information.
- Upon processing a read request, if Dynamo has access to multiple
  branches that cannot be syntactically reconciled, it will return all the
  objects at the leaves, with the corresponding version information in the
  context.
- An update using this context is considered to have reconciled the
  divergent versions and the branches are collapsed into a single new version.
- To illustrate the use of vector clocks, let us consider the
  example shown in Figure 3. A client writes a new object. The node (say Sx)
  that handles the write for this key increases its sequence number and uses it
  to create the data's vector clock. The system now has the object D1 and its
  associated clock [(Sx, 1)]. The client updates the object. Assume the same node
  handles this request as well. The system now also has object D2 and its
  associated clock [(Sx, 2)]. D2 descends from D1 and therefore
  over-writes D1, however there may be replicas of D1 lingering at nodes that
  have not yet seen D2. Let us assume that the same client updates the object
  again and a different server (say Sy) handles the request. The system now has
  data D3 and its associated clock [(Sx, 2), (Sy, 1)].
  
  Next assume a different client reads D2 and then tries to
  update it, and another node (say Sz) does the write. The system now has D4
  (descendant of D2) whose version clock is [(Sx, 2), (Sz, 1)]. A node that is
  aware of D1 or D2 could determine, upon receiving D4 and its clock, that D1 and
  D2 are overwritten by the new data and can be garbage collected. A node that is
  aware of D3 and receives D4 will find that there is no causal relation between
  them. In other words, there are changes in D3 and D4 that are not reflected in each
  other. Both versions of the data must be kept and presented to a client (upon a
  read) for semantic reconciliation.
  
  Now assume some client reads both D3 and D4 (the context
  will reflect that both values were found by the read). The read's context is a
  summary of the clocks of D3 and D4, namely [(Sx, 2), (Sy, 1), (Sz, 1)]. If the
  client performs the reconciliation and node Sx coordinates the write, Sx will
  update its sequence number in the clock. The new data D5 will have the
  following clock: [(Sx, 3), (Sy, 1), (Sz, 1)].
- A possible issue with vector clocks is that the size of
  vector clocks may grow if many servers coordinate the writes to an object. In
  practice, this is not likely because the writes are usually handled by one of the
  top N nodes in the preference list. In case of network partitions or multiple
  server failures, write requests may be handled by nodes that are not in the top
  N nodes in the preference list causing the size of vector clock to grow.
- In
  these scenarios, it is desirable to limit the size of vector clock. To this
  end, Dynamo employs the following clock truncation scheme: Along with each
  (node, counter) pair, Dynamo stores a timestamp that indicates the last time
  the node updated the data item. When the number of (node, counter) pairs in the
  vector clock reaches a threshold (say 10), the oldest pair is removed from the
  clock.
- Clearly, this truncation scheme can lead to inefficiencies in
  reconciliation as the descendant relationships cannot be derived accurately.
  However, this problem has not surfaced in production and therefore this issue has
  not been thoroughly investigated.

13 Oct 08

In Defense of Piracy - WSJ.com

online.wsj.com/...SB122367645363324303.html - Preview

lawrence lessig copyright law pirate file share

Universal manages the copyrights of Prince. It fired off a letter to YouTube demanding that it remove the unauthorized "performance" of Prince's music. YouTube, to avoid liability itself, complied.
YouTube are deluged with demands to remove material from their systems. No doubt a significant portion of those demands are fair and justified. Universal's demand, however, was not.
24 more annotations...
- Electronic Frontier Foundation
- The foundation's lawyers thought this was a straightforward case of fair use.
- EFF and filed a "counter-notice" to YouTube, arguing that no rights of Universal were violated by Holden's dance
- We could craft copyright law to encourage a wide range of both professional and amateur creativity, without threatening Prince's profits.
- Viacom, for example, has effectively promised to exempt practically any amateur remix from its lawyers' concerns.
- During the Republican primary, for example, Fox News ordered John McCain's campaign to stop using a clip of Sen. McCain at a Fox News-moderated debate in an ad. And two weeks ago, Warner Music Group got YouTube to remove a video attacking Barack Obama, which used pieces of songs like the Talking Heads' "Burning Down the House." (Spokesman Will Tanous of Warner Music Group, which represents the Talking Heads, says the request came from the band's management.) Around the same time, NBC asked the Obama campaign to pull an ad that remixed some NBC News footage with Tom Brokaw and Keith Olbermann.
- platforms like YouTube, or MySpace, with others spread across the world, whom they never met, or never even spoke to, but whose creativity has inspired them to create in return
- And it could inspire a deeper, much more meaningful practice of learning for a generation
- our attention is not focused on these creators. It is focused instead upon "the pirates."
- the question we as a society must focus on is whether this is any good
- when we reckon the cost of this corruption, any losses of the content industry pale in comparison
- Copyright law must be changed
- Deregulate amateur remix
- Deregulate "the copy"
- The law should also give up its obsession with "the copy," and focus instead on uses -- like public distributions of copyrighted work -- that connect directly to the economic incentive copyright law was intended to foster.
- Simplify
- tax-code complexity regulating speech is a First Amendment nightmare
- Restore efficiency
- It should be clear who owns what
- Decriminalize Gen-X
- build upon a host of proposals that would assure that artists get paid for their work, without trying to stop "sharing."
- Lawrence Lessig
- Stanford Law School
- Creative Commons

09 Oct 08

Podmailing: Send large files - unlimited file sharing with BitTorrent

Use Podmailing to:

* send your full quality home videos to your family,
* receive the entire high-quality photo collection of your best friend's wedding,
* share powerpoints, PDFs and other heavy pro documents,
* and share torrents, folders and full CD and DVD images easily...

www.podmailing.com - Preview

file share p2p bit torrent email