Mike Chelen's Library tagged distribute → View Popular

Filter:
All
Annotated

31 Jan 09

GridRepublic | Grid Republic | BOINC Volunteer Distributed Grid Computing

27 Dec 08

PS3Cluster Guide: By The Cluster Workshop

Our community guide allows you to set up your own MPI (Message Passing Interface) based supercomputer cluster with the Playstation 3. This guide was co-written by Gaurav Khanna, based on his previous work on the Gravity Grid and is a current run-time environment for the research of co-author (Chris Poulin), based on his current work in distributed pattern recognition. As such, we currently utilize the Fedora Core for this infrastructure and illustrate a "how-to" below. NOTE: We focus on the Fedora 8 distribution, due to prevalence of Fedora and its Cell SDK (3.0) compatibility. Finally, this content should be considered open source, and here is the license.

www.ps3cluster.org - Preview

ps3 cluster mpi distribute compute ps3cluster guide

05 Dec 08

How to set up Disco on Amazon EC2 — Disco v0.1.2 documentation

With the following three steps, you can set up a Disco cluster in the Amazon’s Elastic Computing Cloud. This will cost you a few dollars (or more, depending on your needs) but requires no resources on your side besides a single machine that you use to setup the cluster.

In this setup Disco master and nodes run on EC2. Your Disco client can run either on the master node on EC2 or on a local machine.

discoproject.org/...ec2setup.html - Preview

disco amazon ec2 distribute compute mapreduce python

20 Nov 08

[GUIDE] Getting started with PPI [Start 2 Finish] - Pay Per Install

In this guide I'll lay out exactly how to start with PPI, from joining a Network, to downloading, binding, and uploading files.

This Guide is VERY in depth, it is not as complicated as it may appear.

I will be demonstrating a method that uses torrents to distribute warez bound with your pay-per-install executable. This is the most common method used at Pay Per Install.org

www.pay-per-install.org/...tarted-ppi-start-2-finish.html - Preview

bit torrent distribute process

14 Nov 08

bitbucket.org

Mercurial hosting - we're here to serve.

www.bitbucket.org - Preview

mercurial hg distribute host free

31 Oct 08

Tahoe Storage Illustration

bigasterisk.com/tahoe-playground - Preview

guide interactive distribute file system parity tahoe

30 Oct 08

FreenetWiki: jSite

jSite is a tool that you can use to insert freesites into the Freenet distributed storage.
Creating a freesite is done in two steps, first you build a website in HTML, then you upload your creation to Freenet. That's when jSite comes into the picture: with a nice user-friendly point-and-click interface and five languages to choose from, uploading freesites couldn't be easier.
jSite is written in Java; it will run on any java-enabled operating system, no need for Windows versions, Mac versions, etc: one file fits all - and that file is jSite.jar

wiki.freenetproject.org/FreenetJsite - Preview

java free net html publish anonymous distribute p2p

18 Oct 08

Apache Thrift

Thrift is a software framework for scalable cross-language services development. It combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml.

incubator.apache.org/thrift - Preview

open source distribute compute ocaml java erlang python php

Amazon's Dynamo - All Things Distributed

www.allthingsdistributed.com/...amazons_dynamo.html - Preview

p2p distribute file system

The first generation of
P2P systems, such as Freenet and Gnutella, were predominantly
used as file sharing systems. These were examples of unstructured P2P networks
where the overlay links between peers were established arbitrarily.
next generation
into what is widely known as structured P2P networks
75 more annotations...
- efficiently route a
  search query to some peer that has the desired data
- Pastry [16] and
  Chord [20] use routing mechanisms to ensure that queries can be answered within
  a bounded number of hops
- To reduce the additional latency introduced by
  multi-hop routing, some P2P systems (e.g., [14]) employ O(1) routing where each
  peer maintains enough routing information locally so that it can route requests
  (to access a data item) to the appropriate peer within a constant number of
  hops.
- Oceanstore [9] and PAST [17]
  were built on top of these routing overlays
- Oceanstore resolves conflicts by processing a
  series of updates, choosing a total order among them, and then applying them
  atomically in that order.
- data is
  replicated on an untrusted infrastructure
- PAST provides a
  simple abstraction layer on top of Pastry for persistent and immutable objects.
  It assumes that the application can build the necessary storage semantics (such
  as mutable files) on top of it
- Ficus [15] and Coda [19] replicate files for high availability at the
  expense of consistency. Update conflicts are typically managed using
  specialized conflict resolution procedures
- Farsite system [1] is a
  distributed file system that does not use any centralized server like NFS.
  Farsite achieves high availability and scalability using replication
- GFS uses a simple design with a single
  master server for hosting the entire metadata and where the data is split into
  chunks and stored in chunkservers
- Bayou is a distributed relational database
  system that allows disconnected operations and provides eventual data
  consistency
- Bayou,
  Coda and Ficus allow disconnected operations and are resilient to issues such
  as network partitions and outages
- Coda and Ficus perform system level
  conflict resolution and Bayou allows application level resolution
- Dynamo
  allows read and write operations to continue even during network partitions and
  resolves updated conflicts using different conflict resolution mechanisms
- Distributed
  block storage systems like FAB [18] split large size objects into smaller blocks
  and stores each block in a highly available manner
- a key-value store is more suitable in this case because: (a) it is
  intended to store relatively small objects (size < 1M) and (b) key-value
  stores are easier to configure on a per-application basis
- Antiquity is a
  wide-area distributed storage system designed to handle multiple server
  failures [23]. It uses a secure log to preserve data integrity, replicates each
  log on multiple servers for durability, and uses Byzantine fault tolerance
  protocols to ensure data consistency.
- Dynamo does not
  focus on the problem of data integrity and security and is built for a trusted
  environment
- Bigtable is a distributed storage system for managing structured
  data. It maintains a sparse, multi-dimensional sorted map and allows
  applications to access their data using multiple attributes [2].
- Compared to
  Bigtable, Dynamo targets applications that require only key/value access with
  primary focus on high availability where updates are not rejected even in the
  wake of network partitions or server failures.
- Traditional replicated relational database systems focus on
  the problem of guaranteeing strong consistency to replicated data. Although
  strong consistency provides the application writer a convenient programming
  model, these systems are limited in scalability and availability [7]. These
  systems are not capable of handling network partitions because they typically provide
  strong consistency guarantees.
- Dynamo is targeted mainly
  at applications that need an “always writeable” data store where no updates are
  rejected due to failures or concurrent writes.
- Dynamo is built for an
  infrastructure within a single administrative domain where all nodes are
  assumed to be trusted
- applications that use Dynamo do not require
  support for hierarchical namespaces (a norm in many file systems) or complex
  relational schema (supported by traditional databases)
- Dynamo is built
  for latency sensitive applications that require at least 99.9% of read and
  write operations to be performed within a few hundred milliseconds
- it was imperative for us to avoid routing
  requests through multiple nodes (which is the typical design adopted by several
  distributed hash table systems such as Chord and Pastry)
- Dynamo can be characterized as a zero-hop DHT, where
  each node maintains enough routing information locally to route a request to
  the appropriate node directly.
- Dynamo stores objects associated with a key through a simple
  interface; it exposes two operations: get() and put().
- The get(key)
  operation locates the object replicas associated with the key in the
  storage system and returns a single object or a list of objects with
  conflicting versions along with a context.
- The put(key, context,
  object) operation determines where the replicas of the object should
  be placed based on the associated key, and writes the replicas to
  disk.
- The context encodes system metadata about the object that is
  opaque to the caller and includes information such as the version of the
  object. The context information is stored along with the object so that the
  system can verify the validity of the context object supplied in the put
  request.
- Dynamo’s
  partitioning scheme relies on consistent hashing to distribute the load across
  multiple storage hosts.
- In consistent hashing [10], the output
  range of a hash function is treated as a fixed circular space or “ring” (i.e.
  the largest hash value wraps around to the smallest hash value).
- The principle advantage
  of consistent hashing is that departure or arrival of a node only affects its
  immediate neighbors and other nodes remain unaffected.
- The basic consistent hashing algorithm presents some
  challenges. First, the random position assignment of each node on
  the ring leads to non-uniform data and load distribution. Second, the
  basic algorithm is oblivious to the heterogeneity in the performance of nodes. To
  address these issues, Dynamo uses a variant of consistent hashing (similar to
  the one used in [10, 20]): instead of mapping a node to a single point in the
  circle, each node gets assigned to multiple points in the ring. To this end,
  Dynamo uses the concept of “virtual nodes”. A virtual node looks like a single
  node in the system, but each node can be responsible for more than one virtual
  node. Effectively, when a new node is added to the system, it is assigned
  multiple positions (henceforth, “tokens”) in the ring.
- Using virtual nodes has the following advantages:
  
  If a node becomes unavailable (due to failures or
  routine maintenance), the load handled by this node is evenly dispersed across
  the remaining available nodes.
  
  When a node becomes available again, or a new node
  is added to the system, the newly available node accepts a roughly equivalent
  amount of load from each of the other available nodes.
  
  The number of virtual nodes that a node is responsible
  can decided based on its capacity, accounting for heterogeneity in the physical
  infrastructure.
- Each data item is replicated at N hosts,
  where N is a parameter configured “per-instance”.
- Each key, k,
  is assigned to a coordinator node (described in the previous section). The
  coordinator is in charge of the replication of the data items that fall within
  its range. In addition to locally storing each key within its range, the
  coordinator replicates these keys at the N-1 clockwise successor nodes in the
  ring.
- In Figure 2, node B replicates
  the key k at nodes C and D in addition to storing it locally. Node D
  will store the keys that fall in the ranges (A, B], (B, C], and (C, D].
- The list of nodes that is responsible for storing a
  particular key is called the preference list.
- every node in the system can
  determine which nodes should be in this list for any particular key
- it is possible that the first
  N successor positions for a particular key may be owned by less than N distinct
  physical nodes
- To address this, the preference list for a key is constructed by skipping
  positions in the ring to ensure that the list contains only distinct physical
  nodes.
- Dynamo provides eventual consistency, which allows for
  updates to be propagated to all replicas asynchronously.
- A put() call may return
  to its caller before the update has been applied at all the replicas, which can
  result in scenarios where a subsequent get() operation may return an object
  that does not have the latest updates.
- If there are no failures then there is
  a bound on the update propagation times.
- However, under certain failure
  scenarios (e.g., server outages or network partitions), updates may not arrive
  at all replicas for an extended period of time.
- For example, the shopping cart application requires
  that an “Add to Cart” operation can never be
  forgotten or rejected.
- If the most recent state of the cart is unavailable, and
  a user makes changes to an older version of the cart, that change is still
  meaningful and should be preserved. But at the same time it shouldn’t supersede
  the currently unavailable state of the cart, which itself may contain changes
  that should be preserved.
- both “add to cart” and “delete
  item from cart” operations are translated into put requests to Dynamo
- When
  a customer wants to add an item to (or remove from) a shopping cart and the
  latest version is not available, the item is added to (or removed from) the older
  version and the divergent versions are reconciled later.
- Dynamo treats
  the result of each modification as a new and immutable version of the data
- Most of the time, new versions subsume the previous version(s), and
  the system itself can determine the authoritative version (syntactic
  reconciliation).
- However, version branching may happen, in the presence of
  failures combined with concurrent updates, resulting in conflicting versions of
  an object
- In these cases, the system cannot reconcile the multiple versions of
  the same object and the client must perform the reconciliation in order to collapse
  multiple branches of data evolution back into one (semantic reconciliation).
- A
  typical example of a collapse operation is “merging” different versions of a
  customer’s shopping cart.
- Using this reconciliation mechanism, an “add to cart”
  operation is never lost. However, deleted items can resurface.
- It is important to understand that certain failure modes can
  potentially result in the system having not just two but several versions of
  the same data.
- Updates in the presence of network partitions and node failures
  can potentially result in an object having distinct version sub-histories,
  which the system will need to reconcile in the future.
- This requires us to
  design applications that explicitly acknowledge the possibility of multiple
  versions of the same data (in order to never lose any updates).
- Dynamo uses vector clocks [12] in order to capture causality
  between different versions of the same object.
- One vector clock is associated with every version
  of every object. One can determine whether two versions of an object are on
  parallel branches or have a causal ordering, by examine their vector clocks.
- A vector clock is effectively a
  list of (node, counter) pairs.
- If the counters on the first object’s clock are less-than-or-equal to all
  of the nodes in the second clock, then the first is an
  ancestor of the second and can be forgotten.
- Otherwise, the two changes are
  considered to be in conflict and require reconciliation.
- In Dynamo, when a client wishes to update an object, it must
  specify which version it is updating.
- This is done by passing the context it
  obtained from an earlier read operation, which contains the vector clock
  information.
- Upon processing a read request, if Dynamo has access to multiple
  branches that cannot be syntactically reconciled, it will return all the
  objects at the leaves, with the corresponding version information in the
  context.
- An update using this context is considered to have reconciled the
  divergent versions and the branches are collapsed into a single new version.
- To illustrate the use of vector clocks, let us consider the
  example shown in Figure 3. A client writes a new object. The node (say Sx)
  that handles the write for this key increases its sequence number and uses it
  to create the data's vector clock. The system now has the object D1 and its
  associated clock [(Sx, 1)]. The client updates the object. Assume the same node
  handles this request as well. The system now also has object D2 and its
  associated clock [(Sx, 2)]. D2 descends from D1 and therefore
  over-writes D1, however there may be replicas of D1 lingering at nodes that
  have not yet seen D2. Let us assume that the same client updates the object
  again and a different server (say Sy) handles the request. The system now has
  data D3 and its associated clock [(Sx, 2), (Sy, 1)].
  
  Next assume a different client reads D2 and then tries to
  update it, and another node (say Sz) does the write. The system now has D4
  (descendant of D2) whose version clock is [(Sx, 2), (Sz, 1)]. A node that is
  aware of D1 or D2 could determine, upon receiving D4 and its clock, that D1 and
  D2 are overwritten by the new data and can be garbage collected. A node that is
  aware of D3 and receives D4 will find that there is no causal relation between
  them. In other words, there are changes in D3 and D4 that are not reflected in each
  other. Both versions of the data must be kept and presented to a client (upon a
  read) for semantic reconciliation.
  
  Now assume some client reads both D3 and D4 (the context
  will reflect that both values were found by the read). The read's context is a
  summary of the clocks of D3 and D4, namely [(Sx, 2), (Sy, 1), (Sz, 1)]. If the
  client performs the reconciliation and node Sx coordinates the write, Sx will
  update its sequence number in the clock. The new data D5 will have the
  following clock: [(Sx, 3), (Sy, 1), (Sz, 1)].
- A possible issue with vector clocks is that the size of
  vector clocks may grow if many servers coordinate the writes to an object. In
  practice, this is not likely because the writes are usually handled by one of the
  top N nodes in the preference list. In case of network partitions or multiple
  server failures, write requests may be handled by nodes that are not in the top
  N nodes in the preference list causing the size of vector clock to grow.
- In
  these scenarios, it is desirable to limit the size of vector clock. To this
  end, Dynamo employs the following clock truncation scheme: Along with each
  (node, counter) pair, Dynamo stores a timestamp that indicates the last time
  the node updated the data item. When the number of (node, counter) pairs in the
  vector clock reaches a threshold (say 10), the oldest pair is removed from the
  clock.
- Clearly, this truncation scheme can lead to inefficiencies in
  reconciliation as the descendant relationships cannot be derived accurately.
  However, this problem has not surfaced in production and therefore this issue has
  not been thoroughly investigated.

10 Oct 08

nutrun » Blog Archive » Distributed programming with Jabber and EventMachine

Jabber and its underlying protocol XMPP are typically associated with instant messaging applications, although the breadth and flexibility of the technology allows for implementations that can span further from traditional online chatting.

ejabberd is a fault tolerant and clusterable Jabber/XMPP server written in Erlang and presents an interesting option as a simple, lightweight and scalable message transport for distributed applications.

EventMachine is a simple and fast library for lightweight concurrency in Ruby. Its use mainly involves, but is not limited to, spawning lightweight processes whose execution can be programatically scheduled, easy and fast socket abstractions and an implementation of the Deferrable pattern as introduced by the Twisted event-driven Python networking engine.

nutrun.com/...g-with-jabber-and-eventmachine - Preview

xmpp erlang ruby distribute program

05 Oct 08

Wooki Home Site

wooki.sourceforge.net - Preview

wooki distribute p2p wiki

04 Oct 08

Prototype for distributed / decentralised microblogging using semantics at Cloudlands

www.johnbreslin.com/...-microblogging-using-semantics - Preview

twitter foaf distribute sioc semantic web

RTorrentUsingDHT - The libTorrent and rTorrent Project - Trac

libtorrent.rakshasa.no/...RTorrentUsingDHT - Preview

lib r torrent distribute hash table dht trackerless

30 Sep 08

MogileFS

www.danga.com/mogilefs - Preview

open source distribute file system

redhat.com | Red Hat Global File System

www.redhat.com/gfs - Preview

open source distribute file system

26 Sep 08

ConfReference - the-cassandra-project - Google Code - Configurable parameters of Cassandra, mostly found in conf/storage-conf.xml

code.google.com/...ConfReference - Preview

thecassandraproject distribute file system

the-cassandra-project - Google Code

code.google.com/...the-cassandra-project - Preview

p2p distribute storage file system

22 Sep 08

1 - 20 of 76 Next › Last »

Showing 20▼ items per page

Selected Tags

distribute

Related Tags

Top Contributors

Click in to find related links.

Groups interested in distribute

Tech It 2 Me

Members (9)

bookmarks (205)
OpenSciInfo

Members (11)

bookmarks (263)
kikeread

Members (1)

bookmarks (10)

Related Lists on Diigo

Scenario Planning

A list to gather together a...

Items: 18 | Visits: 6

Created by: Will Allen
Neighborhood Communication Tools

A list of resources to show...

Items: 42 | Visits: 4

Created by: Clay Leben

Highlighter, Sticky notes, Tagging, Groups and Network: integrated suite dramatically boosting research productivity. Learn more »

Join Diigo

Mike Chelen's Library tagged distribute → View Popular

GridRepublic | Grid Republic | BOINC Volunteer Distributed Grid Computing

PS3Cluster Guide: By The Cluster Workshop

How to set up Disco on Amazon EC2 — Disco v0.1.2 documentation

[GUIDE] Getting started with PPI [Start 2 Finish] - Pay Per Install

bitbucket.org

Tahoe Storage Illustration

FreenetWiki: jSite

Apache Thrift

Amazon's Dynamo - All Things Distributed

nutrun » Blog Archive » Distributed programming with Jabber and EventMachine

Wooki Home Site

Prototype for distributed / decentralised microblogging using semantics at Cloudlands

RTorrentUsingDHT - The libTorrent and rTorrent Project - Trac

MogileFS

redhat.com | Red Hat Global File System

ConfReference - the-cassandra-project - Google Code - Configurable parameters of Cassandra, mostly found in conf/storage-conf.xml

the-cassandra-project - Google Code

Disco

Erlang

Erlang (programming language) - Wikipedia, the free encyclopedia

Selected Tags

Related Tags

Sponsored Links

Top Contributors

Groups interested in distribute

Tech It 2 Me

OpenSciInfo

kikeread

Scenario Planning

Neighborhood Communication Tools