Good news, your data scientist just got a personal assistant

June 3, 2014

personal asstIf you are or have a data scientist in house you’re in for good news.

Today at Hadoop Summit in San Jose, Pentaho unveiled a toolkit built specifically for data scientists to simplify the messy, time-consuming data preparation, cleansing and orchestration of analytic data sets. Don’t just take it from us…

The Ventana Research Big Data Analytics Benchmark Research estimates the top two time-consuming big data tasks are solving data quality and consistency issues (46%) and preparing data for integration (52%). That’s a whopping amount of time just spent getting data prepped and cleansed, not to mention the time spent in post processing results.  Imagine if time spent preparing, managing and orchestrating these processes could be handed off to a personal assistant leaving more time to focus on analyzing and applying advanced and predictive algorithms to data (i.e. doing what a data scientist is paid to do).

Enter the Pentaho Data Science Pack, the personal assistant to the data scientist.  Built to help operationalize advanced analytic models as part of a big data flow, the data science pack leverages familiar tools like R, the most-used tool for data scientists and Weka, a widely used and popular open source collection of machine learning algorithms. No new tools to learn. In the words of our own customer, Ken Krooner, President at ESRG “There was a gap in the market until now and people like myself were piecing together solutions to help with the data preparation, cleansing and orchestration of analytic data sets. The Pentaho Data Science Pack fills that gap to operationalize the data integration process for advanced and predictive analytics.”

Pentaho is at the forefront of solving big data integration challenges, and we know advanced and predictive analytics are core ingredients for success. Find out how close at hand your data science personal assistant is and take a closer look at the Data Science Pack.

Chuck Yarbrough
Director, Big Data Product Marketing


Weka goes BIG

December 4, 2013

funny_science_nerd_cartoon_character_custom_flyer-rb4a8aff0894a4e25932056b8852f8b18_vgvyf_8byvr_512.jpgThe beakers are bubbling more violently than usual at Pentaho Labs and this time predictive analytics is the focus.  The lab coat, pocket-protector and taped glasses clad scientists have turned their attention to the Weka machine learning software.

Weka, a collection of machine learning algorithms for predictive analytics and data mining, has a number of useful applications. Examples include, scoring credit risk, predicting downtime of machines and analyzing sentiment in social feeds.  The technology can be used to facilitate automatic knowledge discovery by uncovering hidden patterns in complex datasets, or to develop accurate predictive models for forecasting.

Organizations have been building predictive models to aid decision making for a number of years, but the recent explosion in the volume of data being recorded (aka “Big Data”) provides unique challenges for data mining practitioners. Weka is efficient and fast when running against datasets that fit in main memory, but larger datasets often require sampling before processing. Sampling can be an effective mechanism when samples are representative of the underlying problem, but in some cases the loss of information can negatively impact predictive performance.

To combat information loss, and scale Weka’s wide selection of predictive algorithms to large data sets, the folks at Pentaho Labs developed a framework to run Weka in Hadoop. Now the sort of tasks commonly performed during the development of a predictive solution – such as model construction, tuning, evaluation and scoring – can be carried out on large datasets without resorting to down-sampling the data. Hadoop was targeted as the initial distributed platform for the system, but the Weka framework contains generic map-reduce building blocks that can be used to develop similar functionality in other distributed environments.

If you’re a predictive solution developer or a data scientist, the new Weka framework is a much faster path to solution development and deployment.  Just think of the questions you can ask at scale!

To learn more technical details about the Weka Hadoop framework I suggest to read the blog, Weka and Hadoop Part 1, by Mark Hall, Weka core developer at Pentaho.

Also, check out Pentaho Labs to learn more about Predictive Analytics from Pentaho, and to see some of the other cool things the team has brewing.

Chuck Yarbrough
Technical Solutions Marketing


Follow

Get every new post delivered to your Inbox.

Join 101 other followers