dan mcquillan's List: social computing IS52026B - lecture 12

• Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it.

• As a catch-all term, “big data” can be pretty nebulous, in the same way that the term “cloud” covers diverse technologies. Input data to big data systems could be chatter from social networks, web server logs, traffic flow sensors, satellite imagery, broadcast audio streams, banking transactions, MP3s of rock music, the content of web pages, scans of government documents, GPS trails, telemetry from automobiles, financial market data, the list goes on. Are these all really the same thing?

   
• Assuming that the volumes of data are larger than those conventional relational database infrastructures can cope with, processing options break down broadly into a choice between massively parallel processing architectures — data warehouses or databases such as Greenplum — and Apache Hadoop-based solutions. This choice is often informed by the degree to which the one of the other “Vs” — variety — comes into play. Typically, data warehousing approaches involve predetermined schemas, suiting a regular and slowly evolving dataset. Apache Hadoop, on the other hand, places no conditions on the structure of the data it can process.
• At its core, Hadoop is a platform for distributing computing problems across a number of servers. First developed and released as open source by Yahoo, it implements the MapReduce approach pioneered by Google in compiling its search indexes. Hadoop’s MapReduce involves distributing a dataset among multiple servers and operating on the data: the “map” stage. The partial results are then recombined: the “reduce” stage.
• One of the most well-known Hadoop users is Facebook, whose model follows this pattern. A MySQL database stores the core data. This is then reflected into Hadoop, where computations occur, such as creating recommendations for you based on your friends’ interests. Facebook then transfers the results back into MySQL, for use in pages served to users
• There are two main reasons to consider streaming processing. The first is when the input data are too fast to store in their entirety: in order to keep storage requirements practical some level of analysis must occur as the data streams in. At the extreme end of the scale, the Large Hadron Collider at CERN generates so much data that scientists must discard the overwhelming majority of it — hoping hard they’ve not thrown away anything useful. The second reason to consider streaming is where the application mandates immediate response to the data. Thanks to the rise of mobile applications and online gaming this is an increasingly common situation.
• As mentioned above, it’s not just about input data. The velocity of a system’s outputs can matter too. The tighter the feedback loop, the greater the competitive advantage. The results might go directly into a product, such as Facebook’s recommendations, or into dashboards used to drive decision-making.
• It’s this need for speed, particularly on the web, that has driven the development of key-value stores and columnar databases, optimized for the fast retrieval of precomputed information. These databases form part of an umbrella category known as NoSQL, used when relational models aren’t the right fit.
• Big data is big
• It is a fundamental fact that data that is too big to process conventionally is also too big to transport anywhere. IT is undergoing an inversion of priorities: it’s the program that needs to move, not the data. If you want to analyze data from the U.S. Census, it’s a lot easier to run your code on Amazon’s web services platform, which hosts such data locally, and won’t cost you time or money to transfer it.

• Big data is messy

   

  It’s not all about infrastructure. Big data practitioners consistently report that 80% of the effort involved in dealing with data is cleaning it up in the first place, as Pete Warden observes in his Big Data Glossary: “I probably spend more time turning messy source data into something usable than I do on the rest of the data analysis process combined.”
• The phenomenon of big data is closely tied to the emergence of data science, a discipline that combines math, programming and scientific instinct. Benefiting from big data means investing in teams with this skillset, and surrounding them with an organizational willingness to understand and use data for advantage.
• • Technical expertise: the best data scientists typically have deep expertise in some scientific discipline.
   
  • Curiosity: a desire to go beneath the surface and discover and distill a problem down into a very clear set of hypotheses that can be tested.
   
  • Storytelling: the ability to use data to tell a story and to be able to communicate it effectively.
   
  • Cleverness: the ability to look at a problem in different, creative ways.

11 more annotations...

• All that focus being put on Big Data is all good and well, but ITDMs would be better off focusing on Open Data if they want to increase organisational value. 
• According to research firm Gartner, an Open Data strategy should be a top priority for any organisation that uses the Web as a channel for delivering services.
• "Big data is a topic of growing interest for many business and IT leaders, and there is little doubt that it creates value by enabling organizations to uncover previously unseen patterns and develop sharper insights about their environments," said David Newman, research vice president at Gartner. "However, for clients seeking competitive advantage through direct interactions, Open Data is the solution. For example, more government agencies are now opening their data to the public Web to improve transparency."
• But there's precious little agreement on what constitutes 'open' .  According to Gartner analysts, an informal definition of openness is a level playing field where everyone can play a game that can evolve. There is a positive relationship between the openness of information goods (for example, code, data, content and standards) and information services (for example, services that offer information goods, such as the Internet, Wikipedia, OpenStreetMap and GPS) and the size and diversity of the community sharing them. 
• Open data APIs are a lightweight approach to data exchange. Their use is now considered a best practice for opening data and functionality to developers and other businesses. Organizations use APIs to generate new sources of revenue, spur innovation, increase transparency and improve brand equity.
• "The challenge for organizations is to determine how best to use APIs and how an open data strategy should align with priorities," Newman said.
• Gartner says maturing technologies for data quality and data anonymization help mitigate regulatory restraints and risk factors while Open Data APIs provide simple, Web-oriented means for data exchange, and linked data techniques are effective for generating big datasets. 
• Emerging data marketplaces are also places for organizations to open their data." The challenge is to keep the barriers to entry low to enable participation and streamlined processes for adding and vetting data sources.  

6 more annotations...

• In an earlier paper dealing with the digital divide discussion I suggested the use of the concept of “effective use” to distinguish between the opportunity for digitally-enabled activity presented by ICT access, from the actual realization of those opportunities in the form of “effective use”.
• Efforts to extend access to “data” will perhaps inevitably create a “data divide” parallel to the oft-discussed “digital divide” between those who have access to data which could have significance in their daily lives and those who don’t.
• The suggestion implicit in most of the discussions on “open data” (and explicit in Berners-Lee’s above quoted talk) is that “everyone” has the potential to make use of the data
• A very interesting and well-documented example of this empowering of the empowered can be found in the work of Solly Benjamin and his colleagues looking at the impact of the digitization of land records in Bangalore. Their findings were that newly available access to land ownership and title information in Bangalore was primarily being put to use by middle and upper income people and by corporations to gain ownership of land from the marginalized and the poor. The newly digitized and openly accessible data allowed the well to do to take the information provided and use that as the basis for instructions to land surveyors and lawyers and others to challenge titles, exploit gaps in title, take advantage of mistakes in documentation, identify opportunities and targets for bribery, among others. They were able to directly translate their enhanced access to the information along with their already available access to capital and professional skills into unequal contests around land titles, court actions, offers of purchase and so on for self-benefit and to further marginalize those already marginalized.

• Certainly the newly digitized information was “accessible” to all on an equal basis but the availability of resources to translate that “access” into a beneficial “effective use” was directly proportional to the already existing resources available to those to whom the access was being provided.  The old story about the pauper and the millionaire having equal opportunity to purchase a printing press as a means to promote their interests can be seen as holding equally here as in the 19^th century.

   
• Benjamin’s meticulously documented paper shows how the digitization and related digital access to land title records in Bangalore had the direct effect of shifting power and wealth to those with the financial resources and skills to use this information in self-interested ways.  This is not to suggest that processes of computerization inevitably lead to such outcomes but rather to say that in the absence of efforts to equalize the playing field with respect to enabling opportunities for the use of newly available data, the end result may be increased social divides rather than reduced ones particularly with respect to the already poor and marginalized.
• As well, this is not to argue against “open data” which in fact is a very significant advance and support to broad-based democratic action and empowerment but rather to argue that in the absence of specific efforts to ensure the widest possible availability of the pre-requisites for “effective use” the outcome of “open data” may be quite the opposite to that which is anticipated (and presumably desired) by its strongest proponents

• An interesting example of how open data, with appropriate attention being given to some of these pre-conditions, in fact can provide a basis for effective use can be seen in how the UCLA Centre for Health Policy Research’s California Health Interview Survey (CHIS) has been put to use by Community Advocates in Solano County.  The CHPR conducts a bi-annual California Health Interview Survey  in conjunction with the California Department of Health “to provide a snapshot of the health and healthcare of Californians”.  The survey is used by a range of political authorities but most interestingly they provide free and widely accessible training on how to use the information “to develop appropriate and targeted policy responses” and overall “to learn how to use and apply the data to improve health and health care”. That is, the information is not only made accessible but attention is paid and resources are provided to ensure that the data is usable by those who might make effective use of it.

   

  In this instance, the Solana County Community Advocates were trained so as to be able to take the data provided by the CHIS, and plot incidences of asthma by local electoral district. They were then able to create a map showing an extremely high frequency of asthma among residents in a particular local area. The Community Advocates successfully argued against developing another truck stop along I-80 in the county based on CHIS 2001 data estimates that showed Solano County to have the state’s highest rate of asthma symptom  prevalence overall and one of the highest rates for children.

   

  While in many respects this example parallels the earlier one from North Carolina the difference here is that the skills required for doing the analysis of the online data were provided through training to the local community who were then able to mount a local campaign to achieve the desired end. The key difference here was the attention that was paid by the provider of the information, the CHPR to ensuring that the data could be effectively used without the need for highly skilled (and expensive) professional intermediaries. This involved the development of end user oriented training programs.

   

6 more annotations...