Mike Chelen's Library tagged distribute → View Popular, Search in Google

Jan

2009

GridRepublic | Grid Republic | BOINC Volunteer Distributed Grid Computing

www.gridrepublic.org

- Preview

boinc distribute compute grid

Dec

2008

PS3Cluster Guide: By The Cluster Workshop

Our community guide allows you to set up your own MPI (Message Passing Interface) based supercomputer cluster with the Playstation 3. This guide was co-written by Gaurav Khanna, based on his previous work on the Gravity Grid and is a current run-time environment for the research of co-author (Chris Poulin), based on his current work in distributed pattern recognition. As such, we currently utilize the Fedora Core for this infrastructure and illustrate a "how-to" below. NOTE: We focus on the Fedora 8 distribution, due to prevalence of Fedora and its Cell SDK (3.0) compatibility. Finally, this content should be considered open source, and here is the license.

www.ps3cluster.org

- Preview

ps3 cluster mpi distribute compute ps3cluster guide

Dec

2008

How to set up Disco on Amazon EC2 — Disco v0.1.2 documentation

With the following three steps, you can set up a Disco cluster in the Amazon’s Elastic Computing Cloud. This will cost you a few dollars (or more, depending on your needs) but requires no resources on your side besides a single machine that you use to setup the cluster.

In this setup Disco master and nodes run on EC2. Your Disco client can run either on the master node on EC2 or on a local machine.

discoproject.org/...ec2setup.html

- Preview

disco amazon ec2 distribute compute mapreduce python

Nov

2008

[GUIDE] Getting started with PPI [Start 2 Finish] - Pay Per Install

In this guide I'll lay out exactly how to start with PPI, from joining a Network, to downloading, binding, and uploading files.

This Guide is VERY in depth, it is not as complicated as it may appear.

I will be demonstrating a method that uses torrents to distribute warez bound with your pay-per-install executable. This is the most common method used at Pay Per Install.org

www.pay-per-install.org/...tarted-ppi-start-2-finish.html

- Preview

bit torrent distribute process

Nov

2008

bitbucket.org

Mercurial hosting - we're here to serve.

www.bitbucket.org

- Preview

mercurial hg distribute host free

Oct

2008

Tahoe Storage Illustration

bigasterisk.com/tahoe-playground

- Preview

guide interactive distribute file system parity tahoe

Oct

2008

FreenetWiki: jSite

jSite is a tool that you can use to insert freesites into the Freenet distributed storage.
Creating a freesite is done in two steps, first you build a website in HTML, then you upload your creation to Freenet. That's when jSite comes into the picture: with a nice user-friendly point-and-click interface and five languages to choose from, uploading freesites couldn't be easier.
jSite is written in Java; it will run on any java-enabled operating system, no need for Windows versions, Mac versions, etc: one file fits all - and that file is jSite.jar

wiki.freenetproject.org/FreenetJsite

- Preview

java free net html publish anonymous distribute p2p

Oct

2008

Apache Thrift

Thrift is a software framework for scalable cross-language services development. It combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml.

incubator.apache.org/thrift

- Preview

open source distribute compute ocaml java erlang python php

Amazon's Dynamo - All Things Distributed

www.allthingsdistributed.com/...amazons_dynamo.html

- Preview

p2p distribute file system

The first generation of P2P systems, such as Freenet and Gnutella, were predominantly used as file sharing systems. These were examples of unstructured P2P networks where the overlay links between peers were established arbitrarily.
next generation into what is widely known as structured P2P networks
75 more annotation(s)...
- efficiently route a search query to some peer that has the desired data
- Pastry [16] and Chord [20] use routing mechanisms to ensure that queries can be answered within a bounded number of hops
- To reduce the additional latency introduced by multi-hop routing, some P2P systems (e.g., [14]) employ O(1) routing where each peer maintains enough routing information locally so that it can route requests (to access a data item) to the appropriate peer within a constant number of hops.
- Oceanstore [9] and PAST [17] were built on top of these routing overlays
- Oceanstore resolves conflicts by processing a series of updates, choosing a total order among them, and then applying them atomically in that order.
- data is replicated on an untrusted infrastructure
- PAST provides a simple abstraction layer on top of Pastry for persistent and immutable objects. It assumes that the application can build the necessary storage semantics (such as mutable files) on top of it
- Ficus [15] and Coda [19] replicate files for high availability at the expense of consistency. Update conflicts are typically managed using specialized conflict resolution procedures
- Farsite system [1] is a distributed file system that does not use any centralized server like NFS. Farsite achieves high availability and scalability using replication
- GFS uses a simple design with a single master server for hosting the entire metadata and where the data is split into chunks and stored in chunkservers
- Bayou is a distributed relational database system that allows disconnected operations and provides eventual data consistency
- Bayou, Coda and Ficus allow disconnected operations and are resilient to issues such as network partitions and outages
- Coda and Ficus perform system level conflict resolution and Bayou allows application level resolution
- Dynamo allows read and write operations to continue even during network partitions and resolves updated conflicts using different conflict resolution mechanisms
- Distributed block storage systems like FAB [18] split large size objects into smaller blocks and stores each block in a highly available manner
- a key-value store is more suitable in this case because: (a) it is intended to store relatively small objects (size < 1M) and (b) key-value stores are easier to configure on a per-application basis
- Antiquity is a wide-area distributed storage system designed to handle multiple server failures [23]. It uses a secure log to preserve data integrity, replicates each log on multiple servers for durability, and uses Byzantine fault tolerance protocols to ensure data consistency.
- Dynamo does not focus on the problem of data integrity and security and is built for a trusted environment
- Bigtable is a distributed storage system for managing structured data. It maintains a sparse, multi-dimensional sorted map and allows applications to access their data using multiple attributes [2].
- Compared to Bigtable, Dynamo targets applications that require only key/value access with primary focus on high availability where updates are not rejected even in the wake of network partitions or server failures.
- Traditional replicated relational database systems focus on the problem of guaranteeing strong consistency to replicated data. Although strong consistency provides the application writer a convenient programming model, these systems are limited in scalability and availability [7]. These systems are not capable of handling network partitions because they typically provide strong consistency guarantees.
- Dynamo is targeted mainly at applications that need an “always writeable” data store where no updates are rejected due to failures or concurrent writes.
- Dynamo is built for an infrastructure within a single administrative domain where all nodes are assumed to be trusted
- applications that use Dynamo do not require support for hierarchical namespaces (a norm in many file systems) or complex relational schema (supported by traditional databases)
- Dynamo is built for latency sensitive applications that require at least 99.9% of read and write operations to be performed within a few hundred milliseconds
- it was imperative for us to avoid routing requests through multiple nodes (which is the typical design adopted by several distributed hash table systems such as Chord and Pastry)
- Dynamo can be characterized as a zero-hop DHT, where each node maintains enough routing information locally to route a request to the appropriate node directly.
- Dynamo stores objects associated with a key through a simple interface; it exposes two operations: get() and put().
- The get(key) operation locates the object replicas associated with the key in the storage system and returns a single object or a list of objects with conflicting versions along with a context.
- The put(key, context, object) operation determines where the replicas of the object should be placed based on the associated key, and writes the replicas to disk.
- The context encodes system metadata about the object that is opaque to the caller and includes information such as the version of the object. The context information is stored along with the object so that the system can verify the validity of the context object supplied in the put request.
- Dynamo’s partitioning scheme relies on consistent hashing to distribute the load across multiple storage hosts.
- In consistent hashing [10], the output range of a hash function is treated as a fixed circular space or “ring” (i.e. the largest hash value wraps around to the smallest hash value).
- The principle advantage of consistent hashing is that departure or arrival of a node only affects its immediate neighbors and other nodes remain unaffected.
- The basic consistent hashing algorithm presents some challenges. First, the random position assignment of each node on the ring leads to non-uniform data and load distribution. Second, the basic algorithm is oblivious to the heterogeneity in the performance of nodes. To address these issues, Dynamo uses a variant of consistent hashing (similar to the one used in [10, 20]): instead of mapping a node to a single point in the circle, each node gets assigned to multiple points in the ring. To this end, Dynamo uses the concept of “virtual nodes”. A virtual node looks like a single node in the system, but each node can be responsible for more than one virtual node. Effectively, when a new node is added to the system, it is assigned multiple positions (henceforth, “tokens”) in the ring.
- Using virtual nodes has the following advantages:
  
  If a node becomes unavailable (due to failures or routine maintenance), the load handled by this node is evenly dispersed across the remaining available nodes.
  When a node becomes available again, or a new node is added to the system, the newly available node accepts a roughly equivalent amount of load from each of the other available nodes.
  The number of virtual nodes that a node is responsible can decided based on its capacity, accounting for heterogeneity in the physical infrastructure.
- Each data item is replicated at N hosts, where N is a parameter configured “per-instance”.
- Each key, k, is assigned to a coordinator node (described in the previous section). The coordinator is in charge of the replication of the data items that fall within its range. In addition to locally storing each key within its range, the coordinator replicates these keys at the N-1 clockwise successor nodes in the ring.
- In Figure 2, node B replicates the key k at nodes C and D in addition to storing it locally. Node D will store the keys that fall in the ranges (A, B], (B, C], and (C, D].
- The list of nodes that is responsible for storing a particular key is called the preference list.
- every node in the system can determine which nodes should be in this list for any particular key
- it is possible that the first N successor positions for a particular key may be owned by less than N distinct physical nodes
- To address this, the preference list for a key is constructed by skipping positions in the ring to ensure that the list contains only distinct physical nodes.
- Dynamo provides eventual consistency, which allows for updates to be propagated to all replicas asynchronously.
- A put() call may return to its caller before the update has been applied at all the replicas, which can result in scenarios where a subsequent get() operation may return an object that does not have the latest updates.
- If there are no failures then there is a bound on the update propagation times.
- However, under certain failure scenarios (e.g., server outages or network partitions), updates may not arrive at all replicas for an extended period of time.
- For example, the shopping cart application requires that an “Add to Cart” operation can never be forgotten or rejected.
- If the most recent state of the cart is unavailable, and a user makes changes to an older version of the cart, that change is still meaningful and should be preserved. But at the same time it shouldn’t supersede the currently unavailable state of the cart, which itself may contain changes that should be preserved.
- both “add to cart” and “delete item from cart” operations are translated into put requests to Dynamo
- When a customer wants to add an item to (or remove from) a shopping cart and the latest version is not available, the item is added to (or removed from) the older version and the divergent versions are reconciled later.
- Dynamo treats the result of each modification as a new and immutable version of the data
- Most of the time, new versions subsume the previous version(s), and the system itself can determine the authoritative version (syntactic reconciliation).
- However, version branching may happen, in the presence of failures combined with concurrent updates, resulting in conflicting versions of an object
- In these cases, the system cannot reconcile the multiple versions of the same object and the client must perform the reconciliation in order to collapse multiple branches of data evolution back into one (semantic reconciliation).
- A typical example of a collapse operation is “merging” different versions of a customer’s shopping cart.
- Using this reconciliation mechanism, an “add to cart” operation is never lost. However, deleted items can resurface.
- It is important to understand that certain failure modes can potentially result in the system having not just two but several versions of the same data.
- Updates in the presence of network partitions and node failures can potentially result in an object having distinct version sub-histories, which the system will need to reconcile in the future.
- This requires us to design applications that explicitly acknowledge the possibility of multiple versions of the same data (in order to never lose any updates).
- Dynamo uses vector clocks [12] in order to capture causality between different versions of the same object.
- One vector clock is associated with every version of every object. One can determine whether two versions of an object are on parallel branches or have a causal ordering, by examine their vector clocks.
- A vector clock is effectively a list of (node, counter) pairs.
- If the counters on the first object’s clock are less-than-or-equal to all of the nodes in the second clock, then the first is an ancestor of the second and can be forgotten.
- Otherwise, the two changes are considered to be in conflict and require reconciliation.
- In Dynamo, when a client wishes to update an object, it must specify which version it is updating.
- This is done by passing the context it obtained from an earlier read operation, which contains the vector clock information.
- Upon processing a read request, if Dynamo has access to multiple branches that cannot be syntactically reconciled, it will return all the objects at the leaves, with the corresponding version information in the context.
- An update using this context is considered to have reconciled the divergent versions and the branches are collapsed into a single new version.
- To illustrate the use of vector clocks, let us consider the example shown in Figure 3. A client writes a new object. The node (say Sx) that handles the write for this key increases its sequence number and uses it to create the data's vector clock. The system now has the object D1 and its associated clock [(Sx, 1)]. The client updates the object. Assume the same node handles this request as well. The system now also has object D2 and its associated clock [(Sx, 2)]. D2 descends from D1 and therefore over-writes D1, however there may be replicas of D1 lingering at nodes that have not yet seen D2. Let us assume that the same client updates the object again and a different server (say Sy) handles the request. The system now has data D3 and its associated clock [(Sx, 2), (Sy, 1)].
  
  Next assume a different client reads D2 and then tries to update it, and another node (say Sz) does the write. The system now has D4 (descendant of D2) whose version clock is [(Sx, 2), (Sz, 1)]. A node that is aware of D1 or D2 could determine, upon receiving D4 and its clock, that D1 and D2 are overwritten by the new data and can be garbage collected. A node that is aware of D3 and receives D4 will find that there is no causal relation between them. In other words, there are changes in D3 and D4 that are not reflected in each other. Both versions of the data must be kept and presented to a client (upon a read) for semantic reconciliation.
  
  Now assume some client reads both D3 and D4 (the context will reflect that both values were found by the read). The read's context is a summary of the clocks of D3 and D4, namely [(Sx, 2), (Sy, 1), (Sz, 1)]. If the client performs the reconciliation and node Sx coordinates the write, Sx will update its sequence number in the clock. The new data D5 will have the following clock: [(Sx, 3), (Sy, 1), (Sz, 1)].
- A possible issue with vector clocks is that the size of vector clocks may grow if many servers coordinate the writes to an object. In practice, this is not likely because the writes are usually handled by one of the top N nodes in the preference list. In case of network partitions or multiple server failures, write requests may be handled by nodes that are not in the top N nodes in the preference list causing the size of vector clock to grow.
- In these scenarios, it is desirable to limit the size of vector clock. To this end, Dynamo employs the following clock truncation scheme: Along with each (node, counter) pair, Dynamo stores a timestamp that indicates the last time the node updated the data item. When the number of (node, counter) pairs in the vector clock reaches a threshold (say 10), the oldest pair is removed from the clock.
- Clearly, this truncation scheme can lead to inefficiencies in reconciliation as the descendant relationships cannot be derived accurately. However, this problem has not surfaced in production and therefore this issue has not been thoroughly investigated.

Oct

2008

nutrun » Blog Archive » Distributed programming with Jabber and EventMachine

Jabber and its underlying protocol XMPP are typically associated with instant messaging applications, although the breadth and flexibility of the technology allows for implementations that can span further from traditional online chatting.

ejabberd is a fault tolerant and clusterable Jabber/XMPP server written in Erlang and presents an interesting option as a simple, lightweight and scalable message transport for distributed applications.

EventMachine is a simple and fast library for lightweight concurrency in Ruby. Its use mainly involves, but is not limited to, spawning lightweight processes whose execution can be programatically scheduled, easy and fast socket abstractions and an implementation of the Deferrable pattern as introduced by the Twisted event-driven Python networking engine.

nutrun.com/...g-with-jabber-and-eventmachine

- Preview

xmpp erlang ruby distribute program