Rhm2ktmi's List: BI & Analytics

• Druid, an open source database designed for real-time analysis, is moving to the Apache 2 software license in order to hopefully spur more use of and innovation around the project. It was open sourced in late 2012 under the GPL license, which is generally considered more restrictive than the Apache license in terms of how software can be reused.

• There is an emerging type of service provider that leverages the business intelligence (BI) platform-as-a-service (PaaS) model to give clients rapid access to their own transaction data. It does so by acting as a trusted broker in both its own operational business model and in the accumulation, management, and analysis of data.
• While some service-provider organizations host their own proprietary tools, a growing number of these companies manage their services on cloud-based BI PaaS. This model is appealing because the service provider can easily develop portals that give its client communities visibility and access to their own reporting information.
• Because the service provider manages all of the data in a single location, it can act as a trusted third party for clients willing to franchise their data. It can analyze the combined, crowdsourced data and generate benchmark analyses and comparative performance reports.

1 more annotation...

• There is an emerging type of service provider that leverages the business intelligence (BI) platform-as-a-service (PaaS) model to give clients rapid access to their own transaction data.
• It does so by acting as a trusted broker in both its own operational business model and in the accumulation, management, and analysis of data.
• The cloud-based model has changed BI economics. Reduced expenses and freedom from acquisition, management, and maintenance lower the barriers to entry, so small- and medium-sized businesses (SMBs) can now benefit from analytics.

1 more annotation...

• The company also uses the tool, which is an R package called BreakoutDetection, to track large upticks in engagement during big events such as the Super Bowl

• Predixion Software has released the latest version of its predictive analytics platform
• Predixion, San Juan Capistrano, Calif., said Oct. 15 version 4.0 of its “Insight” platform aims to expand access to predictive analytics to a range of production environments in vertical markets like health care and financial services.

• Predixion Software CEO Simon Arkell stressed the increasing difficulty of gaining a return on investments in big data technologies. In a statement, Arkell noted that the company’s latest release eases deployment of predictive analytics by “embed[ding] them directly into workflows and endpoints where it will generate the most value.”

   

1 more annotation...

• Keen IO is a custom analytics back end for software developers. The company offers a set of APIs that allow customers to build their own data analytic tools, rather than working with off-the-shelf services to build a new engine from scratch.
• Keen IO's approach of developing a back-end offering rather than an off-the shelf analytics platform has resulted in a product that meets the requirements of scalability and flexibility that emerging technologies for wearable devices and IoT require.
• Keen IO's platform helps software developers set up, operate and evolve custom analytics for mobile, tablet, Web and IoT applications
• Keen IO's business model is cloud-based SaaS, with recurring monthly revenue. The company also provides consulting services to help get customers up to speed on analytics training, data modeling, integration and setup, building dashboards and reports, and experimentation and exploration.
• Contenders include large companies with comprehensive analytic and 'big data' services for developers, such as Google BigQuery, which helps businesses crunch large data sets; Amazon Kinesis, an AWS service for real-time apps that allows developers to pull any amount of data from any number of sources; and Intel Cloud IoT Analytics, a cloud-based analytics system for IoT.
• Leading off-the-shelf vendors with analytic tools include Microsoft, which recently acquired analytics firm Capptain; Adobe, which has a suite of mobile services that includes Adobe Analytics; and Yahoo, which recently bought analytics firm Flurry.
• Other players with analytic services that focus on IoT include
• Arrayent, a private company founded in 2002 that provides back-end services and software to enable devices to connect to the Internet; Glassbeam, a provider of a SaaS-based data analytics tool for connected devices; and Infobright, a purveyor of an open source analytic database for analyzing large volumes of machine-generated data.

6 more annotations...

• The Snowflake Elastic Data Warehouse is a homegrown SQL relational database that Snowflake claims can analyze both transactional data and machine data, making it a sort of hybrid of the application performance gathering capabilities of New Relic and the machine data analytics of Splunk, explained Jon Bock, Snowflake’s vice president of products and marketing.
• With Snowflake, users should be able to analyze together all of the data pertaining to their applications and infrastructure and correlate that information to help make business decisions.

• Key findings of this report include
• Self-service tools using visualization may have shortened the delivery cycle for data analysis and in certain cases allowed business users to take matters into their own hands. However, without proper backing, self-service analytics is a double-edged sword, as it provides a false sense of confidence that can lead to statistically unsupported conclusions.
• Some may see data scientists as the new rock stars with organizations putting a burden on their shoulders to deliver everything they hope their data analysis can be. But the all-encompassing notion of a data scientist is inaccurate. Real data scientists are in short supply, and placing such strong dependency on one person or a small team is a risky proposition for any business.
• Machine learning is a process that has been hard and expensive to configure and execute. Even experts have a hard time interpreting generated results. Business users and experts need more-comprehensible self-service machine learning.

2 more annotations...

• Thomas Davenport, in his keynote at the Hadoop Summit San Jose 2014, said that the big data analytics has entered a new phase: From Analytics 2.0 to 3.0. One aspect of the new phase is data discovery, curation, and relationships, creating new products and services in the data economy.
• However, while they’ve accelerated an organization’s ability to quickly explore and analyze data, they struggle to help link disparate data sets. These data sets lack a foreign key such as an account number to link them. Without a key, users are forced to overlay the data models without the ability to use algorithms to aid in the deconfliction process, leading to collisions and duplication.
• These data sets lack a foreign key such as an account number to link them. Without a key, users are forced to overlay the data models without the ability to use algorithms to aid in the deconfliction process, leading to collisions and duplication.
• Other users are applying ETL processes written for existing sources to these new data sets, but cannot support semi-structured and unstructured sources.
• entity resolution and relationship dimension-building application
• By building foundational entity indices of people, organizations, locations, product, and events, organizations have the ability to understand how a person or product is represented in different systems across the enterprise.

• The questions that analysts can now ask:

   
   
  • Who is an individual and to whom or what are they connected?
   
  • What is an organization, both formal and informal?
   
  • What is a product?
   
  • What events are interconnected between the people, organizations and products that we offer?
   
  • What types of relationships could I uncover that drive operational efficiency, improve customer satisfaction, and better measure risk or detect fraud?
   
  • How can these new data sets and data points transform our business by incorporating the use of leading indicators instead of lagging indicators?

5 more annotations...

• Much of this discussion could apply to machine-generated data in general. But right now I think more players are doing product management with an explicit conception either of log management or event-series analytics, so for this post I’ll share that focus too.
• A short answer might be “Splunk, but with more analytic functionality and more scalable performance, at lower cost, plus numerous coupons for free pizza.”

• Glassbeam is building out is multi-tenant analytic service in the cloud with an eye on broadening its appeal and functionality
• First up is a search and log management application, which will be generally available by the end of March. It will be followed by a number of other applications, including a BI front end for predictive analytics as well as industry-specific apps.
• The startup is also revving an engine for high-velocity, real-time streaming requirements, and opening up the back end of the Glassbeam platform to enable third-party applications to be plugged into it, and applications to be developed on it.
• The aim behind the latter move is to enable partners to utilize the semiotic parsing language (SPL), which is the lynchpin of its hosted analysis service, to write their own SPL-based applications.
• SPL is, essentially, a domain-specific language designed to describe the structure, meaning and relationships of unstructured data so that it can be structured and analyzed.

3 more annotations...

• Glassbeam has introduced a back-end platform in the cloud for its hosted machine-data analysis applications, which are underpinned by a homegrown data-mapping and metadata definition language to ascertain structure, meaning and metadata in complex multi-structured log files.
• The company has also served up a development tool to automate the generation of its Semiotic Parsing Language (SPL), which is a fundamental part of its core intellectual property.
• SaaSy analytics play around The Internet of Things. Glassbeam has also expanded beyond direct sales to indirect deals, signing up Hitachi Data Systems as its first OEM customer.
• We have yet to come across a vendor that provides exactly the same capabilities as Glassbeam. We think Splunk is the closest. Why? Machine-data analytics is a core focus for Splunk, which recently made a cloud play and, furthermore, is seeking to become a full-fledged analytics provider.
• TIBCO could also shape up as a more direct rival now that it has acquired Splunk's longtime rival LogLogic. TIBCO currently positions LogLogic within its complex-event-processing portfolio for log management and log management intelligence.
• It is also worth noting that Glassbeam can correlate structured and unstructured data, even though it has focused more on the latter in its positioning and go-to-market strategy
• A similar high-level analytical focus – albeit with different technical underpinnings from each other and from Glassbeam – is also shared by Actuate, Pentaho, Jaspersoft, Datawatch and ClearStory Data.

5 more annotations...

• Kx Systems is no startup – it was founded in 1993 by Arthur Whitney and Janet Lustgarten. However, the company remains privately held to this day. Lustgarten is CEO, while Whitney is chairman. Kx claims to have 100+ customers and about 20 employees, with offices in North America, Europe and Asia.
• The architecture that they ended up with is a massively parallel, columnar database that uses a partially in-memory approach to speed queries. Commonly accessed data is pushed into memory, which is an order of magnitude faster to access than data stored on disk.

• Note that kdb+ is not a pure in-memory database like VoltDB or MemSQL – the company positioned its technology for the analysis of very large datasets, and when its architecture was being designed, memory was still relatively expensive. But the most commonly accessed data is stored in-memory to speed query times.

   
• While there are other columnar analytic databases out there, including HP Vertica, Actian Matrix (previously ParAccel) and Exasol, not all are geared up for real-time data streaming in quite the same way, and there is also the fact that they don't have their own built-in languages.
• We'd also note that kdb+ was built with time-series data in mind, whereas the others were aimed at general-purpose data-warehousing tasks.
• The relational database giants, of course, would surely argue that their databases are perfectly acceptable for most use cases – we're talking here about the likes of Oracle, Microsoft and IBM.

4 more annotations...

• It has been awhile since we last caught up with Predixion Software, which has expanded beyond an initial vertical sector in healthcare to other industry segments. The startup is drawing on the assistance of investor and partner Accenture for its horizontal go-to-market strategy, which is all about enabling folks without any statistical knowledge to craft predictive analytic applications using its Machine Learning Semantic Model (MLSM) – a key technical underpinning of its Predixion Insight stack.
• The startup is gearing up to release Predixion Insight 4.0 in September. The latest version aims to address the needs of more sophisticated users with complex environments, and it has a roadmap in place to further support its horizontal play.
• Five-year-old Predixion is seeking to diversify its business, which has so far largely been concentrated in the healthcare industry.
• Predixion is now going horizontal by building out use cases for other industries, including insurance and retail, so folks can use its predictive analytic stack for fraud detection, customer churn and customer 'next best offer' scenarios, for example.
• Accenture – a key investor in Predixion's $20m series C round last July and also an alliance partner and global distributor – is aiding Predixion's horizontal go-to-market strategy. Management reports that Accenture is working on delivering specific software capabilities for Predixion Insight while continuing to play a key role in systems integration where accounts need change management services.
• Predixion has a long-standing relationship with Microsoft, which became evident when Predixion emerged from stealth mode with an advanced-analysis offering for Excel and PowerPivot in September 2010.
• Predixion continues to report SAS Institute and IBM SPSS as its main competitors in head-to-head bake-offs.
• We continue to think that Alpine Data Labs is a potential alternative to Predixion's wares given its collaborative nature and, moreover, its shared focused on providing an advanced analytic workflow that can be embedded in a third-party environment, such as a business application.
• SAP and Oracle are the other leviathans in predictive analytics.
• We think RapidMiner and Alteryx are competitive to one degree or another given that both are courting a similar audience and support integration with R.
• 0xdata is a new market entrant looking to provide scalable, fast, multi-structured analytics by drawing on R and Hadoop as foundations for its homegrown in-memory columnar compressed engine dubbed H20.
• the new breed of machine-learning startups employing machine-learning algorithms to ease advanced analysis as it takes a more horizontal approach.
• Machine-learning startups include Skytree, Yottamine Analytics, Ayasdi, Context Relevant, Nutonian, Wise.io and BigML.

11 more annotations...

• Splunk, widely known for its search and analysis tools for aggregations of machine-generated data (device, system, and application event and exception logs, typically), recently launched the Cloudmeter Stream offering for network traffic integration as part of Splunk Enterprise and Splunk Cloud.
• This simplifies adding network traffic events to existing repositories and provides a valuable tool for fleshing out system and application visibility without having to add log file writing events explicitly. Given Splunk's remarkable success to date helping customers mine value from log files, the addition of wire data will raise the value of network visibility and monitoring and analysis, and potentially open up exciting new uses of network data in application and business process visibility and analysis.
• Splunk is a dominant player in the overall enterprise machine-data-analysis space, and more often partners rather than competes with adjacent companies
• Splunk's core log management space is replete with large established companies including TIBCO and HP, as well as smaller companies such as ArcSight and LogLogic.

2 more annotations...

• GE’s Industrial Data Lake, enabled by cloud computing, provides a fast path to Predix-powered GE Predictivity solutions, driving outcomes based on industrial big data. GE’s Industrial Data Lake, built on software elements of Pivotal’s big data suite, is optimized for the Industrial Internet to meet stringent requirements for availability, security, elasticity, and performance for mission-critical workloads
• • Fast ingestion, storage and compute for all data types. 
   
  •   High-performance analysis. 
   
  •   Optimized for mission-critical workloads. 
   
  •  Data governance and federation. 
   
   
   
      
   
   
    
   

         

   

   

• An easy way to understand state in stream processing is to think about the kinds of operations you might do in SQL. Imagine running SQL queries against a real-time stream of data. If your SQL query contains only filtering and single-row transformations (a simple select and where clause, say), then it is stateless.
• However, if your query involves aggregating many rows (a group by) or joining together data from multiple streams, then it must maintain some state in between rows.

• Remote state

   

  Let’s start by understanding how you can use an external database to handle remote state in stream processing, then I’ll contrast this with local state.

   

• Local state

   

  Local state is just data that is kept in memory or on disk on the machines doing the processing. The job can query or modify this data in response to its input. The simplest way of maintaining state might be a simple local key-value store.

• Fault-tolerant local state

   

  At Samza, we make local state changes fault tolerant by modeling them as just another output log — a commitlog or changelog.

• Databases are streams

   

  One aspect of the usage patterns that may not be immediately obvious is that you can treat databases as being a source of a stream: the stream of row updates. MySQL, HBase, Oracle, Postgres, and MongoDB all give access to the stream of changes in this way. Kafka has good support for this kind of keyed data and can act as a data subscription layer for data coming out of these databases, making it easy to have a lot of processing jobs subscribing to these changes.

• Okay, so now I’ve explained how you can have fault-tolerant local state, but why would you want this? Why not just use a remote database?

   

• There are three primary reasons:

   
   
  1. Local state can be indexed and accessed in a variety of rich ways that are much harder with remote databases
   
  2. Local, in-process data access can be much, much faster than remote access
   
  3. Local data access is easier to isolate
   
   

6 more annotations...