Aditya Thakkar's List: Cloud

• Items can be persons, organizations, or groups and will be defined more thoroughly below. 
• “Item”: An item is a specific user in Tencent Weibo, which can be a person, an organization, or a group, that was selected and recommended to other users.
• The size of this is about 6K items in the dataset.
• Items are organized in categories; each category belongs to another category, and all together they form a hierarchy. For example, an item, a vip user Dr. Kaifu LEE,
• categories in different levels are separated by a dot ‘.’
• The dataset represents a sampled snapshot of Tencent Weibo users’ preferences for various items
• the history of users’ ‘following’ history
• the recommendation of items to users
• The users in the dataset, numbered in millions, are provided with rich information (demographics, profile keywords, follow history, etc.) for generating a good prediction model.
• Format of the above 2 files:
• (UserId)\t(ItemId)\t(Result)\t(Unix-timestamp)
• Result: values are 1 or -1, where 1 represents the user UserId accepts the recommendation of item ItemId and follows it (i.e., adds it to his/her social network), and -1 represents the user rejects the recommended item.
•   i.              User profile data: user_profile.txt
• (UserId)\t(Year-of-birth)\t(Gender)\t(Number-of-tweet)\t(Tag-Ids)
• ii.              Item data: item.txt
• (ItemId)\t(Item-Category)\t(Item-Keyword)
• Item-Keyword contains the keywords extracted from the corresponding Weibo profile of the person, organization, or group. The format is a string “id1;id2;…;idN”, where each unique keyword is encoded as an unique integer such that no real term is revealed.
• iii.              User action data: user_action.txt
• (UserId)\t(Action-Destination-UserId)\t(Number-of-at-action)\t(Number-of-retweet )\t(Number-of-comment)
• iv.              User sns data: user_sns.txt
• (Follower-userid)\t(Followee-userid)
• v.              User key word data: user_key_word.txt
• (UserId)\t(Keywords)

21 more annotations...

• HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware.
• provides high throughput access to application data and is suitable for applications that have large data sets.
• detection of faults and quick,   automatic recovery from them is a core architectural goal of HDFS.
• HDFS is designed more for batch processing rather than interactive use by users.
• need a write-once-read-many access model for files
• computation requested by an application is much more efficient if it is executed near the data it operates on.
• HDFS has a master/slave architecture.
• consists of a single NameNode, a master server that manages the file   system namespace and regulates access to files by clients.
• DataNodes, usually one per node   in the cluster, which manage storage attached to the nodes that they run on.
• NameNode executes file system namespace operations like opening, closing, and renaming files and directories. It also   determines the mapping of blocks to DataNodes.
• DataNodes also perform block creation, deletion, and replication upon instruction from the NameNode
• DataNodes are responsible for serving read and write requests from the file   system’s clients.
• The NameNode is the arbitrator   and repository for all HDFS metadata. The system is designed in such a way that user data never flows through the NameNode.
• HDFS supports a traditional hierarchical file organization.
• An application can specify the number of replicas of a file that should be maintained by   HDFS. The number of copies of a file is called the replication factor of that file.
• t stores each file as a sequence   of blocks; all blocks in a file except the last block are the same size.
• NameNode makes all decisions regarding replication of blocks.
• he placement of replicas is critical to HDFS reliability and performance.
• purpose   of a rack-aware replica placement policy is to improve data reliability, availability, and network bandwidth utilization.
• NameNode determines the rack id each DataNode belongs to
• when the replication factor is three, HDFS’s placement policy is to put one replica   on one node in the local rack, another on a node in a different (remote) rack, and the last on a different node in the   same remote rack.
• This policy cuts the inter-rack write traffic which generally improves write performance.
• Replication of data blocks does not occur   when the NameNode is in the Safemode state.
• On startup, the NameNode enters a special state called Safemode.
• NameNode uses a transaction log called the EditLog   to persistently record every change that occurs to file system metadata.
• he entire file system namespace, including the mapping   of blocks to files and file system properties, is stored in a file called the FsImage.
• When the NameNode starts up, it reads the FsImage and EditLog from   disk, applies all the transactions from the EditLog to the in-memory representation of the FsImage, and flushes   out this new version into a new FsImage on disk. It can then truncate the old EditLog because its transactions   have been applied to the persistent FsImage. This process is called a checkpoint.
• DataNode has no knowledge about HDFS files.
• hen a DataNode starts   up, it scans through its local file system, generates a list of all HDFS data blocks that correspond to each of these   local files and sends this report to the NameNode: this is the Blockreport.
• HDFS communication protocols are layered on top of the TCP/IP protocol.
• NameNode never initiates any RPCs. Instead, it only   responds to RPC requests issued by DataNodes or clients.
• three common types   of failures are NameNode failures, DataNode failures and network partitions.
• typical block size used by HDFS is 64 MB.
• When the local file accumulates data worth over one HDFS block size, the   client contacts the NameNode.
• The NameNode inserts the file name into the file system hierarchy   and allocates a data block for it.
• applications need streaming writes to files.
• The first DataNode starts receiving the data in small portions (4 KB),   writes each portion to its local repository and transfers that portion to the second DataNode in the list.

35 more annotations...

• Hadoop MapReduce is a software framework for easily writing   applications which process vast amounts of data (multi-terabyte data-sets)   in-parallel on large clusters (thousands of nodes) of commodity   hardware in a reliable, fault-tolerant manner.
• A MapReduce job usually splits the input data-set into   independent chunks which are processed by the map tasks in a  completely parallel manner. The framework sorts the outputs of the maps,   which are then input to the reduce tasks.
• the MapReduce framework and the Hadoop Distributed File System (see HDFS Architecture Guide)   are running on the same set of nodes.
• MapReduce framework consists of a single master   JobTracker and one slave TaskTracker per   cluster-node.
• Hadoop   job client
• job configuration
• framework views the   input to the job as a set of <key, value> pairs and   produces a set of <key, value> pairs as the output of   the job, conceivably of different types.
• Mapper
• Reducer
• JobConf
• Reporter
• Partitioner
• OutputCollector
• JobClient
• OutputFormat
• OutputCommitter
• InputFormat
• Mapper maps input key/value pairs to a set of intermediate   key/value pairs.
• The Hadoop MapReduce framework spawns one map task for each   InputSplit generated by the InputFormat for   the job.
• The Mapper outputs are sorted and then   partitioned per Reducer.
• Reducer reduces a set of intermediate values which share a key to  a smaller set of values.

19 more annotations...