Malcolm McRoberts's List: Large Datasets

• 50-200M pages from the 100K 1 million top (by US-based traffic) domains
• The crawl is done by a custom Bixo workflow created by Bixo Labs, built on top of Cascading/Hadoop
• stored as compressed warcAvro files in S3
• The code used to run the crawl, as well as code to access the crawl data, will  be available at  http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=263

2 more annotations...

• 44 million blog posts
• The total size of the dataset is 142 GB uncompressed, (27 GB compressed)

• What is
  Multivariate Data Analysis(MVA)
   

  MVA is a powerful set of techniques for understanding the relationships between variables in large data sets, which classical statistics may not adequately identify or explain. MVA lets you understand, visualize and make predictions from your data.

• Synopsis

   

  Outputs search results to the specified csv file. 

   

  Syntax

   

  outputcsv [create_empty=<bool>] [dispatch=<bool>] [usexml=<bool>] [singlefile=<bool>] [<filename>]

• SplunkMSE allows Splunk to be used as a data storage back end for the MySQL RDBMS.  What this means is that SQL, Tables, Databases, ODBC, JDBC and buzzwords well-known in the Relational Database community can now be used to access data that lives within Splunk.  No importing or exporting required.
• plunkMSE allows the user to map Splunk searches to MySQL tables.

• Avizo® software is a powerful, multifaceted tool for visualizing, manipulating, and understanding scientific and industrial data.

• We deliver multivariate software and solutions for analyzing large, complex data sets quickly, easily and accurately.