Make Your Voice Heard! – 2013 Wisdom of Crowds Business Intelligence Market Study

March 12, 2013

Make your voice heard!

Participate in the 2013 Wisdom of Crowds ® Business Intelligence Market Study and get a complimentary copy of the study findings. 

Dresner Advisory Services is inviting all Business Intelligence (BI) users to participate in its annual examination of the state of the BI marketplace focusing on BI usage, deployment trends, and products.

The 2013 report will build on previous years’ research and will expand to include questions on the latest and emerging trends such as Collaborative BI, BI in the Cloud, and Embedded BI. It will also rank vendors and products, providing an important tool for organizations seeking to invest in BI solutions.

BI users in all roles and throughout all industries are invited to contribute their insight, which should take approximately 15 minutes.  The final report is scheduled to be out in late Spring, and qualified survey participants will receive a complimentary copy.

Click here to start the survey today!


Improving Customer Support using Hadoop and Device Data Analytics

March 6, 2013
20130228_145940.jpg

L – Dave Henry, Pentaho | R – Ben Llyod, NetApp

At Strata 2013 last week, Pentaho had the privilege to host a speaking session with Ben Lloyd, Sr. Program Manager, AutoSupport (ASUP) at NetApp. Ben leads a project called ASUP.Next, which has the goal of implementing a mission-critical data infrastructure for a worldwide customer support program for NetApp’s storage appliances. With design and development assistance from Think Big Analytics and Accenture, NetApp has reached the “go-live” milestone for ASUP.Next and will go into production this month.

A Big Data Problem

More than 250,000 NetApp devices are deployed worldwide; they “phone home” with device statistics and diagnostic information and represent a continuously growing collection of structured data that must be reliably captured, parsed, interpreted and aggregated to support a large collection of use cases. Ben’s presentation highlighted the business and IT challenges of the legacy AutoSupport environment:

  • The total cost of processing, storing and managing data represents a major ongoing expense ($15M / year). The storage required for ASUP-related data doubles every 16 months — by the end of 2013 NetApp will have more than 1PB of ASUP-related data available for analysis
  • The legacy ETL (PL/SQL) and data warehouse-based approach has resulted in increased latency and missed SLAs. Integrated data for reporting and analysis is typically only available 72-hours after the receipt of device messages
  • For NetApp Customer Support, the information required to resolve support cases is not easily available in the time required
  • For NetApp Professional Services, it’s difficult or impossible to aggregate the volume of performance data needed to provide valuable recommendations
  • For Product Engineering, failure analysis and defect signatures over long time periods are impossible to identify

Cloudera Hadoop: at the Core of NetApp’s Solution

The ASUP.Next project aims to address these issues by eliminating data volume constraints and building a Hadoop-centered infrastructure that will scale to support projected volumes. Ben discussed the new architecture in detail during his presentation. It enables a complete end-to-end workflow including:

  • Receipt of ASUP device messages via HTTP and e-mail
  • Message parsing and ingestion into HDFS and HBase
  • Distribution of messages to case-generation processes and downstream ASUP consumers
  • Long –term storage of messages
  • Reporting and analytic access to structured and unstructured data
  • RESTful services that provide access to AutoSupport data and processes

Pentaho’s Data Integration platform (PDI) is used in ASUP.Next for overall orchestration of this workflow as well as implementation of transformation logic using Pentaho’s visual development solution for MapReduce. Pentaho’s main value to NetApp comes from shortening the development cycle and providing ETL and job control capabilities that span the entire data infrastructure, from HDFS, HBase and MapReduce to Oracle and SAP. Pentaho also worked closely with Cloudera to ensure compatibility with the latest CDH client libraries.

NetApp’s use of Hadoop as a scalable infrastructure for ETL is increasingly common. Pentaho is seeing this use case across a variety of industries including capital markets, government, telecommunications, energy and digital publishing. In general, the reasons these customers use PDI with Hadoop include:

  • Leveraging existing team members for rapid development and ongoing maintenance of the solution. Most organizations have a core ETL team that can bring a decade or more of subject matter expertise to the table. By removing the requirement to use Java, a scripting language or raw XML, team members are able to actively help with the build-out of jobs and transformations. This also lessens the need to recruit, hire and orient outside developers
  • Increasing the “logic density” of transformations. As you can see in the demo example below, it’s possible to express a lot of transformation logic in a single mapper or reducer task. This makes it possible to reduce the number of unique jobs that must be run to achieve a complete workflow. In addition to improving performance, this can result in designs that are easier to document and explain

PDI

  • Focusing on the “what”, not the “how” of MapReduce development. I was surprised (actually shocked) to see how many of the speakers at Strata were still walking through code examples to illustrate a development technique. The typical organization has no desire and little ability to turn itself into a software development shop. The language-based approach may work for the Big Data “Titans”, but not for businesses that need to implement Big Data solutions quickly and with minimal risk

Key Takeaways

Since this was a Pentaho-sponsored session, Ben summarized his experience working with the Pentaho Services and Engineering teams. His main points are illustrated in the photo above. Most of his points revolve around how Pentaho provided support during early development and testing. A large number of Pentaho employees contributed their time, energy and brain-power to ensure the project’s success. Many enhancements in PDI 4.4 are a direct result of improvements needed to support ASUP.Next use cases.

What has Pentaho learned from this project? Pentaho gained a number of valuable insights:

  • Big Data architectures to support low-latency use cases can be complex. Not only are multiple functional components needed, but they must integrate with existing systems such as enterprise data warehouses. These architectures demand a high degree of flexibility
  • Big Data projects require customers, system integrators and technology providers to “plumb the last 5%” as the solution is being developed. Inevitably, new capabilities are used for the first time and need to be fine-tuned to support real-world use cases, data volumes and encoding formats. A good example is PDI’s support for AVRO. Although we anticipated needing to adapt the existing AVRO Input Step to work with NetApp’s schemas, we only understood the full set of requirements after seeing their actual data during an early system test
  • Pentaho’s plugin-based architecture isolates the core “engines” from the layer where point-functionality is implemented. Pentaho is able to implement all of the required enhancements without a single architectural change. The AVRO enhancements and other improvements (such as HTableInput format support for MapReduce jobs) were all coded and field-deployed via updates to plug-ins, completely eliminating the possibility of introducing defects into PDI’s data flow engine.
  • Open source is a significant “enabler” making it easy for everyone to understand how integration works. It’s hard to overestimate the importance of code transparency. It allows the customer, the system integrators and the technology partners to get right to the point and experiment quickly with different designs.

It’s been a pleasure working with NetApp and its partners on the ASUP.Next solution. We look forward to continuing our work with NetApp as their use of device data evolves to exploit new opportunities not previously possible with their legacy application.

-Dave

Dave Henry, SVP Enterprise Solutions
Pentaho


Looking for the perfect match

February 28, 2013

image

I’m at the O’Reilly Strata Big Data Conference in Santa Clara, CA this week where there’s lots of buzz about the value and reality of big data. It’s a fun time to be part of a hot new market in technology. But, of course, a hot new market brings a new set of challenges.

After talking to several attendees, I would not be surprised if someone took out an advertisement in the San Francisco Guardian that reads:

SEEKING BDT (Big Data Talent)

“Middle-aged attractive company seeks hot-to-trot data geek for mutually enjoyable discrete relationship, mostly involving analytics. Must enjoy long discussions about wild statistical models, short walks to the break room and large quantities of caffeine.”

The feedback from the presentations and attendees at Strata mimics the results from a Big Data survey that Pentaho released last week showing there is a lack of current skills to address new big data technologies such as Hadoop among existing staff and more generally on the market. This is good news for folks looking for jobs in Big Data and a good indication for others who want to learn new skills.

The market has created the perfect storm – the combination of hot new technology mixed with a myriad of very complex systems plus highly complicated statistical models and calculations. This storm is preventing the typical IT generalist or BI expert from applying.  More experienced data scientists who can spin models on their head with a twist of a mouse are in high demand. The need to garner value quickly from Big Data means there is little time to look for the “perfect match.”

It seems like new companies and technologies pop up almost every week, each with the promise of business benefits, but with the added cost of high complexity.  Shouldn’t things get easier with new technologies?

Pentaho’s Visual MapReduce is a prime example of things getting easier.  Getting data out of Hadoop quickly can be a challenge.  However, with Visual MapReduce any IT professional could pull the right information from a Hadoop cluster, improve the performance of a MapReduce job and make results available in the optimal format for business users.

New technologies might need new talent, but in the case of Pentaho Visual MapReduce, new technologies might only need new tools to help address them.

Looks like Pentaho is the perfect match.

Chuck Yarbrough
Technical Solutions Marketing


The Tesla vs. NY Times – How Analytics Helped Tesla Win

February 21, 2013

Tesla’s-Pricing-Strategy-for-the-Model-S-Luxury-Sedan-25

In the last couple of weeks the feud between The NY Times Editor, John Broder – and Tesla Motors’ CEO, Elon Musk has played out in the media.

It all started when Broder took a highway trip between Washington D.C. and Boston, cruising in Tesla’s Model S luxury sedan. The purpose of the trip was to range test the car between two new supercharging stations. This 200 miles trip was well under the Model S’s 265-mile estimated range. But nonetheless the trip was filled with anxiety for Broder. Fearful of not reaching his charging destination, he had to turn off the battery-draining amenities such as radio and heater (in a 30 degree weather) to finally reach his destination – feet and knuckles “frozen”.

In rebutting Broder’s claims, Tesla’s chief executive, Elon Musk, has charged that the story was faked, that Mr. Broder intentionally caused his car to fail. On his Tesla blog, he released graphs and charts, based on driving logs that contest many of the details of Mr. Broder’s article.

With the logs now published, one thing is clear — Tesla’s use of predictive analytics helped them warn Broder on what is ahead. By calculating the range based on the energy consumption, Tesla signaled Broder to charge the vehicle in time. Had Tesla not been able to call its log files as witness, this futuristic motor tech company could have experienced serious brand damage.

What’s interesting is that Tesla’s story is not unique. Today, virtually anything that we use, an appliance, a mobile phone, an application, generates some sort of data – machine-generated data. And the truth exists behind that data. Such data, when analyzed and mined properly, provides indicators that solve problems, ahead of time.

Having real-time access to machine-generated data to foresee problems and improve performance is exactly why NetApp is using Pentaho. Using Hadoop and Pentaho Business Analytics to process and drive insights from 2-5 TBs of incoming data per week, NetApp has built a solution that sends alerts and notifications ahead of the actual hardware failure. The solution has helped NetApp predict its appliance interruptions for the E-Series storage units, offering new ways to exceed customer SLAs and protect the brand’s image.

Tesla, NetApp or other, if you run a data-driven business, the more your company can act on that data to improve your application, service or product performance, the better off your customers and the better your brand will be.

Pentaho Business Analytics gives companies fast and easy ways for collecting, analyzing and predicting data patterns. Pentaho’s customers see the value of analytics in many different facets and use cases. NetApp’s use case will be featured in Strata’s upcoming conference on Thursday, February 28, 2012.

Join us to find out more.

- Farnaz Erfan, Product and Solution Marketing, Pentaho


Xyratex and Pentaho – Making Big Data, Fast Data.

February 11, 2013

Pentaho and Xyratex today announced our strategic partnership to deliver the world’s first integrated Big Data analytics and scalable storage solution.  We have been working on this joint initiative for some time with the ClusterStor team at Xyratex. ClusterStor is the worlds fastest and most performant storage sub-system.  This will be significantly enhanced by the addition of Hortonworks Hadoop and Pentaho Business Analytics.

Xyratex and Pentaho will make Big Data, Fast Data.  This solves a key pain point for Xyratex’s customers. With all of the compute, storage, database and analytics in one true integrated platform, this appliance will eliminate the large data silos as well as put all of that Big Data, into the hands of the business users.  And it will do that fast!  The ClusterStor, Hadoop and Pentaho Big Data Appliance will deliver business analytics on huge data sets, at the lowest TCO and allow the ClusterStor customers to realize rapid business value from their data with a very short time to value.

Xyratex has taken the complexity of deploying Hadoop away from the customer with this integrated appliance. Critically, ClusterStor also meets all the key criteria in the deployment of an enterprise class Big Data solution; scalable, best in class performance, reliability and rapid time.


Big Data Speeds Across the Chasm

December 14, 2012

Last week I visited our European team and met with customers, prospects, press and analysts to learn and talk about big data. My week in Europe confirmed my belief that we are definitely in the right business at the most exciting possible time. In a region that is rife with economic challenges, my conversations were optimistic and inspiring.

Seasoned industry people will be familiar with Geoffrey Moore’s famous curve showing the phases of technology adoption, in which the toughest challenge is ‘crossing the chasm between the early adopters and the early majority. With some technologies, this journey can take years. Many never make it across.

Technology Adoption Lifecycle

After speaking to me and executives from MapR, Cloudera and ParAccel, Brian McKenna of Computer Weekly proposes in his article “Big data analytics set to confound conventional adoption curve in UK” that big data adoption is moving relatively fast in the UK and Europe. The UK industry analyst Clive Longbottom, who I met with, reinforced this saying that big data adoption in the UK was only three months behind the US.

Of course the real proof is in what customers are doing. During my visit, our customer Carsten Bomsdorf of Travian Games presented at the Big Data Analytics conference in London about how his company uses Pentaho to analyze the behavior of its 140 million gamers to continuously innovate its award-winning products. And in marked contrast to last year, every single European customer and prospect I met with was either executing or actively planning for big data analytics.

Why is the adoption curve for big data moving faster than other technologies, even in Europe’s more traditionally risk and hype-averse markets? The answer is economic urgency. Big data analytics has demonstrated that it can help companies identify new revenue streams – even needles in haystacks – regardless of the economic climate. Quite simply big data is the ultimate tool for matching supply with demand.

If Europe’s enthusiasm for big data is anything to go on, I have to conclude that 2013 really will be the year that it starts to enter mainstream production. Fasten your seat belts – it’s going to be a wild ride!

Quentin Gallivan, CEO, Pentaho


Going mobile this year? What’s your biggest big data challenge?

November 16, 2012

We received insightful responses to the polls from our “Mobile and Big Data go Instant and Interactive” webinars about the challenges users of all types face with business analytics. The complexity of data integration, lack of skills and resources, and the need to analyze unstructured data are the most significant big data challenges identified for over 80% of attendees. 50% of our attendees either have a current mobile BI solution in place or plan to in the future.

What does this mean for the future of analytics? Whether mobilizing your sales force or empowering data analysts to discover meaning from data in Hadoop, a complete business analytics solution must address the business pressures of a continual inundation of data and the need to access and interact with that data instantly in simple, familiar ways.

Not surprising that the response to Pentaho’s Business Analytics 4.8 has been overwhelmingly positive — the best of analytics offered up in a mobile optimized experience for business users and Instaview broadening big data access to data analysts for data discovery.

If you missed out on our webinar, access the on demand recording at:

Watch the Pentaho 4.8 On-Demand Webinar

Data Integration and business analytics in a single, unified, modern platform — Pentaho is the future of analytics

Let me know what you think about Pentaho 4.8.

Donna Prlich

Director, Product Marketing

Pentaho


Looking to the Future of Business Analytics with Pentaho 4.8

November 12, 2012

Last week Pentaho announced Pentaho 4.8, another milestone in delivering the future of analytics. It has been an exciting ride. Our partners’ and our customers’ feedback have kept us ecstatic and ready to excel further into the future.

Pentaho 4.8 is a true testament on what the future of analytics needs. The future of analytics is driven by the data problems that businesses face every day – and is dependent on the information users and their expectations for solving those problems.

Let me give you a good example. I recently had the pleasure to meet with one of our customers – BeachMint. BeachMint is a fashion and style ecommerce company who uses celebrities / celebrity stylists to promote its retail business.

This rapidly growing online retailer needed to keep tabs on its large twitter and facebook communities to track customer sentiment and social influence. It then uses the social data to define customer cohorts and design marketing campaigns that best target each cohort.

For BeachMint insight to data is extremely important. But on one hand, the volumes and variety of data – in this case unstructured social data and click-through ad feeds – has increased its complexity. And on the other hand, the speed in which it gets created has accelerated rapidly. For example, in addition to analyzing the impact of customer sentiments on their purchasing behavior, BeachMint also needed to gain up-to-the-minute information on the activity of key promotional codes – to immediately identify those that leak out.

Pentaho understands these data challenges and user expectations. In this release Pentaho takes full advantage of its tightly coupled Data Integration and Business Analytics platform – to simplify data exploration, discovery and visualization for all users and all data types – and to deliver this information to users immediately – sometimes even at a micro-second level. In this release Pentaho delivers:

- Pentaho Mobile – the only Mobile BI application with the power to instantly create new analysis on the go.

- Pentaho Instaview – the industry’s first instant and interactive big data visualization application.

Want to find out more? Register for Pentaho 4.8 webinar and see for yourself.

- Farnaz Erfan, Product Marketing, Pentaho


Follow

Get every new post delivered to your Inbox.

Join 96 other followers