Sean Champ's List: Apache Accumulo

• users can specify a security label for each value. This is done as the Mutation is created by passing a ColumnVisibility object
• Security labels consist of a set of user-defined tokens that are required to read the value the label is associated with.
• suppose
• terms of user roles
• can be specified alone or combined using logical operators:
• (admin|system)&audit
• any security labels present are examined against the set of authorizations passed by the client code when the Scanner or BatchScanner are created.
• Authorizations are specified as a comma-separated list of tokens the user possesses:
• Authorization auths = new Authorization("admin","system");
• auths
• connector.createScanner
• use the setuaths and getauths commands. These may also be modified using the java security operations API.
• Each accumulo user has a set of associated security labels
• In this case an accumulo user with all authorizations needed by any of the applications users must be created.

  • ...huh?
• These authorizations could be obtained from a trusted 3rd party.

  • This describes a complex security model
• Often production systems will integrate with Public-Key Infrastructure (PKI) and designate client code within the query layer to negotiate with PKI servers in order to authenticate users and retrieve their authorization tokens (credentials)

  • Sounds like a good place to apply ApacheDS

14 more annotations...

• provides a richer data model than simple key-value stores
• but is not a fully relational database
• the key and value are comprised of the following elements
• All elements of the Key and the Value are represented as byte arrays except for Timestamp
• Accumulo is a distributed data storage and retrieval system and as such consists of several architectural components
• maintaining certain properties of the data, such as organization, availability, and integrity, across many commodity-class machines.
• one Garbage Collector process
• TabletServers
• write-ahead Logger servers
• An instance of Accumulo
• one Master server
• many Clients
• Tablet Server
• tablets (partitions of tables).
• sorting new key-value pairs
• write-ahead log
• a merge-sorted view of all keys and values from all the files it has created and the sorted in-memory store.
• in HDFS
• flushing sorted key-value pairs
• recovery of a tablet that was previously on a server that failed
• reapplying any writes found in the write-ahead log to the tablet.
• Loggers
• Loggers accept updates to Tablet servers and write them to local on-disk storage
• o multiple loggers to preserve data in case of hardware failure.
• Garbage Collector
• Accumulo processes will share files stored in HDFS.
• Garbage Collector will identify files that are no longer needed by any process, and delete them.

• Master

   
• ries to balance the load across TabletServer by assigning tablets carefully and instructing TabletServers to migrate tablets when necessary
• Accumulo Master is responsible for detecting and responding to TabletServer failure.
• ensures all tablets are assigned to one TabletServer each,
• handles table creation, alteration, and deletion requests from clients
• also coordinates startup, graceful shutdown and recovery of changes in write-ahead logs
• Client

• contains logic for finding servers managing a particular tablet, and communicating with TabletServers to write and retrieve key-value pairs.

   
• a client library that is linked to every application.

34 more annotations...

• ZooKeeperInstance
• Connector conn = inst.getConnector("user", "passwd")
• Text rowID = new Text("row1");

  • Note that the class, Mutation, also provides a constructor for CharSequence, an interface implemented by the String class

  • Textual Row IDs? (not numeric??)
• new Text("myColFam")

  • ^ quote, "The Apache™ Hadoop™ project develops open-source software for reliable, scalable, distributed computing."

  • The web site for the Apache Hadoop project is http://hadoop.apache.org/
  • 1 more sticky notes...

  • • Note that Text is the class org.apache.hadoop.io.Text
• public

  • Refer to the manual's section about data security, for an explanation of the relevance and usage of the ColumnVisibility value
• ColumnVisibility colVis

  • API Docs for ColumnVisibility http://accumulo.apache.org/1.4/apidocs/org/apache/accumulo/core/security/ColumnVisibility.html
• long timestamp
• Value value

  • Note that Value is a class native to Accumulo - API docs, http://accumulo.apache.org/1.4/apidocs/org/apache/accumulo/core/data/Value.html
• Mutation mutation

  • API docs for Mutation - http://accumulo.apache.org/1.4/apidocs/org/apache/accumulo/core/data/Mutation.html
• mutation.put

  • The class, Scanner, provides some mechanisms for retrieving data.

  • Note that in some divergence with regards to the typical CRUD model on databases, Accumulo's Mutation class does not provide a 'get method, though it does provide a number of 'put' methods.
• Authorizations
• public
• Scanner scan
• auths
• createScanner
• setRange
• Range
• fetchFamily

16 more annotations...