Big Data in 2015—Power to the People!

December 16, 2014

Last year I speculated that the big data ‘power curve’ in 2014 would be shaped by business demands for data blending. Customers presenting at our debut PentahoWorld conference last October, from Paytronix, to RichRelevance, to NASDAQ, certainly proved my speculations to be true. Businesses like these are examples of how increasingly large and varied data sets can be used to deliver high and sustainable ROI. In fact, Ventana Research recently confirmed that 22 percent of organizations now use upwards of 20 data sources, and 19 percent use between 11 – 20 data sources.[1]

Moving into 2015, and fired up by their initial big data bounties, businesses will seek even more power to explore data freely, structure their own data blends, and gain profitable insights faster. They know “there’s gold in them hills” and they want to mine for even more!

With that said, here are my big data predictions for 2015:

Big Data Meets the Big Blender!

The digital universe is exploding at a rate that even Carl Sagan might struggle to articulate. Analysts believe it’s doubling every year, but with the unstructured component doubling every three months. By 2025, IDC estimates that 40 percent of the digital universe will be generated by machine data and devices, while unstructured data is getting all the headlines.[2] The ROI business use cases we’ve seen require the blending of unstructured data with more traditional, relational, data. For example, one of the most common use cases we are helping companies create is a 360 view of their customers. The de facto reference architecture involves the blending of relational/transactional data detailing what the customer has bought, with unstructured weblog and clickstream data highlighting customer behavior patterns around what they might buy in the future. This blended data set is further mashed up with social media data describing sentiment around the company’s products and customer demographics. This “Big Blend” is fed into recommendation platforms to drive higher conversion rates, increase sales, and improve customer engagement. This “blended data” approach is fundamental to other popular big data use cases like Internet of Things, security and intelligence applications, supply chain management and regulatory and compliance demands in Financial Services, Healthcare and Telco industries.

Internet of Things Will Fuel the New ‘Industrial Internet’

Early big data adoption drove the birth of new business models at companies like our customers Beachmint and Paytronix. In 2015, I’m convinced that we’ll see big data starting to transform traditional industrial businesses by delivering operational, strategic and competitive advantage. Germany is running an ambitious Industry 4.0 project to create “Smart Factories” that are flexible, resource efficient, ergonomic and integrated with customers and business partners. The machine data generated from sensors and devices, are fueling key opportunities like Smart Homes, Smart Cities, and Smart Medicine, which all require big data analytics. Much like the ‘Industrial Internet’ movement in the U.S., Industry 4.0 is is being defined by the Internet of Things. According to Wikibon, the value of efficiency from machine data could reach close to $1.3 trillion dollars and will drive $514B in IT spend by 2020.[3]The bottlenecks are challenges related to data security and governance, data silos, and systems integration.

Big Data Gets Cloudy!

As companies with huge data volumes seek to operate in more elastic environments, we’re starting to see some running all, or part of, their big data infrastructures in the cloud. This says to me that the cloud is now “IT approved” as a safe, secure, and flexible data host. At PentahoWorld, I told a story about a “big datathrow down” that occurred during our Strategic Advisory Board meeting. At one point in the meeting, two enterprise customers in highly regulated industries started one-upping each other about how much data they stored in Amazon Redshift Cloud. One shared that they processed and analysed 5-7 billion records daily.. The next shared that they stored a half petabyte of new data every day and on top of that, they had to hold the data for seven years while still making it available for quick analysis. Both of these customers are held to the highest standards for data governance and compliance – regardless of who won, the forecast for their big data environments is the cloud!

Embedded Analytics is the New BI

Although “classic BI,” which involves a business analyst looking at data with a separate tool outside the flow of the business application, will be around for a while, a new wave is rising in which business users increasingly consume analytics embedded within applications to drive faster, smarter decisions. Gartner’s latest research estimates that more than half the enterprises that use BI now use embedded analytics.[4] Whether it’s a RichRelevance data scientist building a predictive algorithm for a recommendation engine, or a marketing director accessing Marketo to consume analytics related to lead scoring or campaign effectiveness, the way our customers are deploying Pentaho leave me with no doubt that this prediction will bear out.

As classic BI matured, we witnessed a final “tsunami” in which data visualization and self-service inspired business people to imagine the potential for advanced analytics. Users could finally see all their data – warts and all – and also start to experiment with rudimentary blending techniques. Self-service and data visualization prepared the market for what I firmly expect to be the most significant analytics trend in 2015….

Data Refineries Give Real Power to the People!

The big data stakes are higher than ever before. No longer just about quantifying ‘virtual’ assets like sentiment and preference, analytics are starting to inform how we manage physical assets like inventory, machines and energy. This means companies must turn their focus to the traditional ETL processes that result in safe, clean and trustworthy data. However, for the types of ROI use cases we’re talking about today, this traditional IT process needs to be made fast, easy, highly scalable, cloud-friendly and accessible to business. And this has been a stumbling block – until now. Enter Pentaho’s Streamlined Data Refinery, a market-disrupting innovation that effectively brings the power of governed data delivery to “the people,” unlocking big data’s full operational potential. I’m tremendously excited about 2015 and the journey we’re on with both customers and partners. You’re going to hear a lot more about the Streamlined Data Refinery in 2015 – and that’s a prediction I can guarantee will come true!

Finally, as I promised at PentahoWorld in October, we’re only going to succeed when you tell us you’ve delivered an ROI for your business. Let’s go forth and prosper together in 2015!

Quentin Gallivan, CEO, Pentaho

120914-Quentin-Big-Data-2015-Prediction-Graphic

[1] Ventana Research, Big Data Integration Report, Tony Cosentino and Mark Smith, May 2014.

[2] IDC, The Digital Universe of Opportunities: Rich Data and the Increasing Value of the Internet of Things, Gantz, Minton, Turner, Reinsel, Turner, April 2014.

[3] Wikibon, Defining and Sizing the Industrial Internet, David Floyer, June 2013.

[4] Gartner, Use Embedded Analytics to Extend the Reach and Benefits of Business Analytics, Daniel Yuen, October 3, 2014.


Pentaho 5 has arrived with something for everyone!

September 18, 2013

I am tremendously excited to announce that Pentaho Business Analytics 5 is available for download!  This release is represents the culmination of over 30 man years of engineering effort and contains over 250 new features and improvements.  There truly is something for everyone in Pentaho 5.  If you are an end user, administrator, executive or developer I wanted to share with you what I think are the 3 top areas of improvement for you:

  1. Improving productivity for end users and administrators
  2. Empowering organizations to easily and accurately answer questions using blended big data sets
  3. Simplifying the experience for developers integrating with or embedding Pentaho Business Analytics

Improving Productivity for End Users and Administrators

multi-charts

homepage

18 months ago, we challenged ourselves to think deeply about the different profiles of users working with the Pentaho suite and identify the top areas where we could significantly improve our ease-of-use.  Based on the feedback from countless customer interviews and usability studies, the first thing you will notice about Pentaho 5 is a dramatically overhauled User Console.  Beyond the fresh, new, modern look and feel, we’ve introduced a new concept called “perspectives” making it easier than ever for end users to:

  • navigate between open documents
  • browse the repository
  • manage scheduled activities

Throughout the User Console, end users will enjoy numerous improvements and better feedback for common workflows such as designing dashboards or scheduling the execution of a parameterized report. Administrators will appreciate that we have consolidated all Administration capabilities directly into the User Console, enhanced security with the ability to create more specific role types with control the types of actions they can perform, and bundled a comprehensive audit mart providing out-of-the-box answers to common questions about usage patterns, performance and errors.

Analytics-ready Big Data Blending

MattCasters_Blog_graphic

partner logos

In the dawn of the Big Data era, a wide range of new storage and processing technologies have flooded the market, each bringing specialized characteristics to help solve the next wave of data challenges.   Pentaho has long been a leader and innovator in delivering an end-to-end platform for designing scalable and easily maintainable Big Data solutions.  Powered by the Pentaho Adaptive Big Data Layer, we’ve dramatically expanded our support for Hadoop with all new certifications for the latest distributions from Cloudera, Hortonworks, MapR and Intel.  Furthermore, we’ve integrated our complete analytics platform for use with Cloudera Impala.  Other Big Data highlights in Pentaho 5 include new integration with Splunk and dramatic ease-of-use improvements when working with NoSQL platforms such as MongoDB and Cassandra.

blendingAs organizations large and small map out their next generation data architectures, we see best practice design patterns emerging that help organizations target the appropriate data technology for each use case.  Evident in all of these design patterns is the fact that Big Data technologies are rarely information silos.  Solving common use cases such as optimizing your data warehousing architecture or performing 360 degree analysis on a customer require that all data be accessible and blended in an accurate way.  Pentaho Data Integration provides the connectivity and design ease-of-use to implement all of these emerging patterns, and with Pentaho 5 I’m excited to announce the world’s first SQL (JDBC) driver for runtime transformation.  This integration empowers data integration designers to accurately design blended data sets from across the enterprise, and put them directly in the hands of end users using tools they are already familiar with – reporting, dashboards and visual discovery – as well as predictive analytics.

Simplified Platform for OEMs and Embedders

marketplace perspective

integration samples

Finally, I’d like to highlight how this release further solidifies the Pentaho suite as the best platform for enterprises and OEMs who want to enrich their applications with better data processing or business analytics.  Pentaho 5 delivers a more customizable User Console providing developers with complete control over the menu bar and toolbar, improvements to the underlying theming engine and an all new plugin layer for adding custom perspectives.  Furthermore, we’ve dramatically simplified our service architecture by introducing a brand new REST-based API along with a rich library of integration samples and documentation to get you started.

These enhancements are just a few of the many great improvements in Pentaho 5. If you want a more in-depth overview and demonstration, register for the Pentaho 5.0 webinar on September 24th – 2 times to choose:  North America/LATAM & EMEA. You can also access great resources from videos to solutions briefs at Pentaho.com/5.0.

Jake Cornelius

SVP Products


Follow

Get every new post delivered to your Inbox.

Join 105 other followers