Sean Champ's List: KM Notes

•        

  Foreign Culture Categories

• Riftsaw is based on Apache ODE

• a whole raft of radically new post-Hadoop architectures are now being developed that are, in most cases, orders of magnitude faster at scale than Hadoop.

• The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware.
• HDFS relaxes a few POSIX requirements to enable streaming access to file system data.
• HDFS was originally built as infrastructure for the Apache Nutch web search engine project
• An HDFS instance may consist of hundreds or thousands of server machines, each storing part of the file system’s data.
• detection of faults and quick, automatic recovery from them is a core architectural goal of HDFS.
• Applications that run on HDFS need streaming access to their data sets.
• The emphasis is on high throughput of data access rather than low latency of data access
• HDFS is designed more for batch processing rather than interactive use by users
• POSIX semantics in a few key areas has been traded to increase data throughput rates.
• A typical file in HDFS is gigabytes to terabytes in size.
• HDFS is tuned to support large files.
• It should support tens of millions of files in a single instance.
• HDFS applications need a write-once-read-many access model for files.
• A file once created, written, and closed need not be changed.
• simplifies data coherency issues and enables high throughput data access.
• “Moving Computation is Cheaper than Moving Data”
• Datanodes
• Namenode
• HDFS has a master/slave architecture. An HDFS cluster consists of a single Namenode, a master server that manages the file system namespace and regulates access to files by clients. In addition, there are a number of Datanodes, usually one per node in the cluster, which manage storage attached to the nodes that they run on.
• HDFS exposes a file system namespace and allows user data to be stored in files.
• Internally, a file is split into one or more blocks and these blocks are stored in a set of Datanodes.
• Namenode executes file system namespace operations like opening, closing, and renaming files and directories
• Datanodes are responsible for serving read and write requests from the file system’s clients.
• also determines the mapping of blocks to Datanodes.
• Datanodes also perform block creation, deletion, and replication
• A typical deployment has a dedicated machine that runs only the Namenode software. Each of the other machines in the cluster runs one instance of the Datanode software
• The Namenode is the arbitrator and repository for all HDFS metadata.
• user data never flows through the Namenode.
• The Namenode maintains the file system namespace.
• An application can specify the number of replicas of a file that should be maintained by HDFS. The number of copies of a file is called the replication factor of that file.
• all blocks in a file except the last block are the same size. The blocks of a file are replicated for fault tolerance. The block size and replication factor are configurable per file. An application can specify the number of replicas of a file. The replication factor can be specified at file creation time and can be changed later.
• stores each file as a sequence of blocks
• periodically receives a Heartbeat and a Blockreport from each of the Datanodes in the cluster.
• A Blockreport contains a list of all blocks on a Datanode.
• Optimizing replica placement distinguishes HDFS from most other distributed file systems.
• The purpose of a rack-aware replica placement policy is to improve data reliability, availability, and network bandwidth utilization. The current implementation for the replica placement policy is a first effort in this direction.
• This is a feature that needs lots of tuning and experience.
• Large HDFS instances run on a cluster of computers that commonly spread across many racks. Communication between two nodes in different racks has to go through switches. In most cases, network bandwidth between machines in the same rack is greater than network bandwidth between machines in different racks.
• The NameNode determines the rack id each DataNode belongs to via the process outlined in Rack Awareness.
• For the common case, when the replication factor is three, HDFS’s placement policy is to put one replica on one node in the local rack, another on a different node in the local rack, and the last on a different node in a different rack.
• The current, default replica placement policy described here is a work in progress.
• Persistence of File System Metadata
• The Namenode keeps an image of the entire file system namespace and file Blockmap in memory.
• All HDFS communication protocols are layered on top of the TCP/IP protocol.
• Data Organization
• DFSShell
• some sample action/command pairs:
• Create a directory
• View the contents of a file
• DFSAdmin
• some sample action/command pairs:
• Put a cluster in SafeMode
• Generate a list of Datanodes
• Decommission Datanode datanodename

52 more annotations...

• A  HDFS cluster primarily consists of a NameNode that manages the  file system metadata and DataNodes that store the actual data.
• HDFS is the primary distributed storage used by Hadoop applications.
• Most of the time, configuration  needs to be tuned only for very large clusters.
• Hadoop supports shell-like commands to interact with HDFS directly
• The NameNode and Datanodes have built in web servers that makes it  easy to check current status of the cluster.

3 more annotations...