#Rstats for Business Intelligence

This is a short list of several known as well as lesser known R ( #rstats) language codes, packages and tricks to build a business intelligence application. It will be slightly Messy (and not Messi) but I hope to refine it someday when the cows come home.

It assumes that BI is basically-

a Database, a Document Database, a Report creation/Dashboard pulling software as well unique R packages for business intelligence.

What is business intelligence?

Seamless dissemination of data in the organization. In short let it flow- from raw transactional data to aggregate dashboards, to control and test experiments, to new and legacy data mining models- a business intelligence enabled organization allows information to flow easily AND capture insights and feedback for further action.

BI software has lately meant to be just reporting software- and Business Analytics has meant to be primarily predictive analytics. the terms are interchangeable in my opinion -as BI reports can also be called descriptive aggregated statistics or descriptive analytics, and predictive analytics is useless and incomplete unless you measure the effect in dashboards and summary reports.

Data Mining- is a bit more than predictive analytics- it includes pattern recognizability as well as black box machine learning algorithms. To further aggravate these divides, students mostly learn data mining in computer science, predictive analytics (if at all) in business departments and statistics, and no one teaches metrics , dashboards, reporting  in mainstream academia even though a large number of graduates will end up fiddling with spreadsheets or dashboards in real careers.

Using R with

1) Databases-

I created a short list of database connectivity with R here at https://rforanalytics.wordpress.com/odbc-databases-for-r/ but R has released 3 new versions since then.

The RODBC package remains the package of choice for connecting to SQL Databases.

http://cran.r-project.org/web/packages/RODBC/RODBC.pdf

Details on creating DSN and connecting to Databases are given at  https://rforanalytics.wordpress.com/odbc-databases-for-r/

For document databases like MongoDB and CouchDB

( what is the difference between traditional RDBMS and NoSQL if you ever need to explain it in a cocktail conversation http://dba.stackexchange.com/questions/5/what-are-the-differences-between-nosql-and-a-traditional-rdbms

Basically dispensing with the relational setup, with primary and foreign keys, and with the additional overhead involved in keeping transactional safety, often gives you extreme increases in performance

NoSQL is a kind of database that doesn’t have a fixed schema like a traditional RDBMS does. With the NoSQL databases the schema is defined by the developer at run time. They don’t write normal SQL statements against the database, but instead use an API to get the data that they need.

instead relating data in one table to another you store things as key value pairs and there is no database schema, it is handled instead in code.)

I believe any corporation with data driven decision making would need to both have atleast one RDBMS and one NoSQL for unstructured data-Ajay. This is a sweeping generic statement 😉 , and is an opinion on future technologies.

  • Use RMongo

From- http://tommy.chheng.com/2010/11/03/rmongo-accessing-mongodb-in-r/

http://plindenbaum.blogspot.com/2010/09/connecting-to-mongodb-database-from-r.html

Connecting to a MongoDB database from R using Java

http://nsaunders.wordpress.com/2010/09/24/connecting-to-a-mongodb-database-from-r-using-java/

Also see a nice basic analysis using R Mongo from

http://pseudofish.com/blog/2011/05/25/analysis-of-data-with-mongodb-and-r/

For CouchDB

please see https://github.com/wactbprot/R4CouchDB and

http://digitheadslabnotebook.blogspot.com/2010/10/couchdb-and-r.html

  • First install RCurl and RJSONIO. You’ll have to download the tar.gz’s if you’re on a Mac. For the second part, we’ll need to installR4CouchDB,

2) External Report Creating Software-

Jaspersoft- It has good integration with R and is a certified Revolution Analytics partner (who seem to be the only ones with a coherent #Rstats go to market strategy- which begs the question – why is the freest and finest stats software having only ONE vendor- if it was so great lots of companies would make exclusive products for it – (and some do -see https://rforanalytics.wordpress.com/r-business-solutions/ and https://rforanalytics.wordpress.com/using-r-from-other-software/)

From

http://www.jaspersoft.com/sites/default/files/downloads/events/Analytics%20-Jaspersoft-SEP2010.pdf

we see

http://jasperforge.org/projects/rrevodeployrbyrevolutionanalytics

RevoConnectR for JasperReports Server

RevoConnectR for JasperReports Server RevoConnectR for JasperReports Server is a Java library interface between JasperReports Server and Revolution R Enterprise’s RevoDeployR, a standardized collection of web services that integrates security, APIs, scripts and libraries for R into a single server. JasperReports Server dashboards can retrieve R charts and result sets from RevoDeployR.

http://jasperforge.org/plugins/esp_frs/optional_download.php?group_id=409

 

Using R and Pentaho
Extending Pentaho with R analytics”R” is a popular open source statistical and analytical language that academics and commercial organizations alike have used for years to get maximum insight out of information using advanced analytic techniques. In this twelve-minute video, David Reinke from Pentaho Certified Partner OpenBI provides an overview of R, as well as a demonstration of integration between R and Pentaho.
and from
R and BI – Integrating R with Open Source Business
Intelligence Platforms Pentaho and Jaspersoft
David Reinke, Steve Miller
Keywords: business intelligence
Increasingly, R is becoming the tool of choice for statistical analysis, optimization, machine learning and
visualization in the business world. This trend will only escalate as more R analysts transition to business
from academia. But whereas in academia R is often the central tool for analytics, in business R must coexist
with and enhance mainstream business intelligence (BI) technologies. A modern BI portfolio already includes
relational databeses, data integration (extract, transform, load – ETL), query and reporting, online analytical
processing (OLAP), dashboards, and advanced visualization. The opportunity to extend traditional BI with
R analytics revolves on the introduction of advanced statistical modeling and visualizations native to R. The
challenge is to seamlessly integrate R capabilities within the existing BI space. This presentation will explain
and demo an initial approach to integrating R with two comprehensive open source BI (OSBI) platforms –
Pentaho and Jaspersoft. Our efforts will be successful if we stimulate additional progress, transparency and
innovation by combining the R and BI worlds.
The demonstration will show how we integrated the OSBI platforms with R through use of RServe and
its Java API. The BI platforms provide an end user web application which include application security,
data provisioning and BI functionality. Our integration will demonstrate a process by which BI components
can be created that prompt the user for parameters, acquire data from a relational database and pass into
RServer, invoke R commands for processing, and display the resulting R generated statistics and/or graphs
within the BI platform. Discussion will include concepts related to creating a reusable java class library of
commonly used processes to speed additional development.

If you know Java- try http://ramanareddyg.blog.com/2010/07/03/integrating-r-and-pentaho-data-integration/

 

and I like this list by two venerable powerhouses of the BI Open Source Movement

http://www.openbi.com/demosarticles.html

Open Source BI as disruptive technology

http://www.openbi.biz/articles/osbi_disruption_openbi.pdf

Open Source Punditry

TITLE AUTHOR COMMENTS
Commercial Open Source BI Redux Dave Reinke & Steve Miller An review and update on the predictions made in our 2007 article focused on the current state of the commercial open source BI market. Also included is a brief analysis of potential options for commercial open source business models and our take on their applicability.
Open Source BI as Disruptive Technology Dave Reinke & Steve Miller Reprint of May 2007 DM Review article explaining how and why Commercial Open Source BI (COSBI) will disrupt the traditional proprietary market.

Spotlight on R

TITLE AUTHOR COMMENTS
R You Ready for Open Source Statistics? Steve Miller R has become the “lingua franca” for academic statistical analysis and modeling, and is now rapidly gaining exposure in the commercial world. Steve examines the R technology and community and its relevancy to mainstream BI.
R and BI (Part 1): Data Analysis with R Steve Miller An introduction to R and its myriad statistical graphing techniques.
R and BI (Part 2): A Statistical Look at Detail Data Steve Miller The usage of R’s graphical building blocks – dotplots, stripplots and xyplots – to create dashboards which require little ink yet tell a big story.
R and BI (Part 3): The Grooming of Box and Whiskers Steve Miller Boxplots and variants (e.g. Violin Plot) are explored as an essential graphical technique to summarize data distributions by categories and dimensions of other attributes.
R and BI (Part 4): Embellishing Graphs Steve Miller Lattices and logarithmic data transformations are used to illuminate data density and distribution and find patterns otherwise missed using classic charting techniques.
R and BI (Part 5): Predictive Modelling Steve Miller An introduction to basic predictive modelling terminology and techniques with graphical examples created using R.
R and BI (Part 6) :
Re-expressing Data		 Steve Miller How do you deal with highly skewed data distributions? Standard charting techniques on this “deviant” data often fail to illuminate relationships. This article explains techniques to re-express skewed data so that it is more understandable.
The Stock Market, 2007 Steve Miller R-based dashboards are presented to demonstrate the return performance of various asset classes during 2007.
Bootstrapping for Portfolio Returns: The Practice of Statistical Analysis Steve Miller Steve uses the R open source stats package and Monte Carlo simulations to examine alternative investment portfolio returns…a good example of applied statistics using R.
Statistical Graphs for Portfolio Returns Steve Miller Steve uses the R open source stats package to analyze market returns by asset class with some very provocative embedded trellis charts.
Frank Harrell, Iowa State and useR!2007 Steve Miller In August, Steve attended the 2007 Internation R User conference (useR!2007). This article details his experiences, including his meeting with long-time R community expert, Frank Harrell.
An Open Source Statistical “Dashboard” for Investment Performance Steve Miller The newly launched Dashboard Insight web site is focused on the most useful of BI tools: dashboards. With this article discussing the use of R and trellis graphics, OpenBI brings the realm of open source to this forum.
Unsexy Graphics for Business Intelligence Steve Miller Utilizing Tufte’s philosophy of maximizing the data to ink ratio of graphics, Steve demonstrates the value in dot plot diagramming. The R open source statistical/analytics software is showcased.
I think that the report generation package Brew would also qualify as a BI package, but large scale implementation remains to be seen in
a commercial business environment
  • brew: Creating Repetitive Reports
 brew: Templating Framework for Report Generation

brew implements a templating framework for mixing text and R code for report generation. brew template syntax is similar to PHP, Ruby's erb module, Java Server Pages, and Python's psp module. http://bit.ly/jINmaI
  • Yarr- creating reports in R
to be continued ( when I have more time and the temperature goes down from 110F in Delhi, India)
Related articles

Newer Doctrines for Newer Wars

On the Memorial Day, some thoughts on the convergence of revolutions in technology and war fare-

 

War – 

War is an openly declared state of organized conflict, typified by extreme aggression, societal disruption, and high mortality

1) Disrupting command and control objects is the primary stage of attack. Evading detection of your own command and control objects while retaining secure channels of communication with redundant lines of control is the primary stage of defense.

2) Pre emptive strikes are in. Reactive all out wars are out. Countries will no longer “declare war” before going to war. They already dont.

3) Commando /Special Forces/Terror strikes /Guerrilla warfare weapons, tactics and technology will have a big demand. So will be specialist trainers.

4) Improving the predictability of your own detect and destroy mechanisms, and disrupting the predictability of enemy detect and react mechanisms will be hugely in- even more than commissioning one more submarine and one more aircraft type.

5) Countries will revert to ancient tribal paradigms in fast shifting alliances for economics as well as geo politics. Very stupidly religion can be  factor in warfare even in the 21 st century.

 

6) Number of Kills per Weapons fired will converge to a constant .  Risks of secondary collateral damage will need to have a higher weight-age because they spur more retal attacks. Fewer prisoner of wars, higher KIA/ MIA ratio.

7) Fewer civilian casualties than all previous wars. This includes fewer civilian casualties even in nuclear war than previous nuclear scenarios.

8) War is a business. It will not be allowed to disrupt global supply chains for more than 2-3 weeks (or inventory replenishment of critical goods and /or services). commodities will lead to wars explicitly, especially since nuclear energy is discredited and carbon energy is diminishing. Expect synchronization with financial derivatives activity. War futures anyone.

9) The Geneva Convention is overdue for an update. Call it Geneva Convention 3.0 United Nations will remain critical to preventing or hastening global conflicts (remember the league of extra ordinary nations .)

10) Economic weapons, climate changing weapons, and sky weapons will emerge. Expect newer kinds of gun powder to be invented. Cyber weapons and hackers will be in demand . Thats the only bright spot.

Happy Memorial Day.

 

Enjoy that freedom to eat an barbecue- it was paid for in more blood than you will ever care to know.

 

I need more data to make a decision

Paper money, extreme macro
Image by kevindooley via Flickr

I need more data

To take a decision

Keep your panties on

We need more precision

It is the owners money

That pays for your bills

You can go elsewhere

If you want primal egoistic thrills

 

People are precious

Money comes and goes

The Older you get

The lesser greed shows

 

Is this too much information

To overload your comprehension

Analysis led to paralysis

But time wont wait for your permission

 

We need better models

We need them now

The cost of delayed decisions

can hurt us and how

 

We will pay thousands

of dollars in annual fees

To earn or save millions of dollars

Now, If you please

 

Still here, but slightly offended

By coming straight to the truth

Every body swings and misses

From Barry Bonds to Babe Ruth

 

Data is all around you

and so is all the money

You keep ignoring decision management

and you will lose your shirt, honey.

 

Related articles

Viva Americana- Old Eyes Version

Here is a newer version of Viva Americana, based on the feedback loop.You can click on the picture (in this as well as the older to read) version to get a higher zoom level.

The Monk who Danced

P dance
Image via Wikipedia

Some monks pray

Some monks dance

Some monks drink

All monks hope for a second chance

 

I have played all

kinds of funny games

Been to all kinds of places

Called all unkind names

 

The thrill of the chase

The prospect of romance

is rarely expressed in thudding crashes

More often expressed in dance

 

Here comes tango

There goes hip hop

Ballroom dancing for the propah

Disco dancing for the lower crop

 

This monk got grace

It helps he has style

For very soon he will start sweating

Because he has not practiced in a while

 

While the forbidden and elusive

Pleasures of the flesh are often the downfall

Of talented men who began to crawl

Dancing never killed anyone

and no one got killed dancing.

 

The wars the fights and the peacock struts

Have continued since mankind begun

Choose your arena, choose your weapon

Dancing shoes or the barrel of a gun.

 

Music heals what people hurt

Poetry steals what movies covet

Paintings end up hugging stony walls

But dancing above all is in the moment

 

Transitory like a flutter of a piano sheet

Dance now or forever hold your feet.

 

Related-

http://www.themonkwholovestodance.com/

http://www.gakkaionline.net/Myths/Dancing.html

 

 

 

 

 

Interview with Rob La Gesse Chief Disruption Officer Rackspace

Here is an interview with Rob La Gesse ,Chief Disruption Officer ,Rackspace Hosting.
Ajay- Describe your career  journey from not finishing college to writing software to your present projects?
Rob- I joined the Navy right out of High School. I had neither the money for college, or a real desire for it. I had several roles in the Navy, to include a Combat Medic station with the US Marine Corps and eventually becoming a Neonatal Respiratory Therapist.

After the Navy I worked as a Respiratory Therapist, a roofer, and I repaired print shop equipment. Basically whatever it took to make a buck or two.  Eventually I started selling computers.  That led me to running a multi-line dial-up BBS and I taught myself how to program.  Eventually that led to a job with a small engineering company where we developed WiFi.

After the WiFi project I started consulting on my own.  I used Rackspace to host my clients, and eventually they hired me.  I’ve been here almost three years and have held several roles. I currently manage Social Media, building 43 and am involved in several other projects such as the Rackspace Startup Program.

Ajay-  What is building43 all about ?

Rob- Building43 is a web site devoted to telling the stories behind technology startups. Basically, after we hired Robert Scoble and Rocky Barbanica we were figuring out how best we could work with them to both highlight Rackspace and customers.  That idea expanded beyond customers to highlighting anyone doing something incredible in the technology industry – mostly software startups.  We’ve had interviews with people like Mark Zuckerberg, CEO and Founder of FaceBook.  We’ve broken some news on the site, but it isn’t really a news site. It is a story telling site.

Rackspace has met some amazing new customers through the relationships that started with an interview.

Ajay-  How is life as Robert Scoble’s boss. Is he an easy guy to work with? Does he have super powers while he types?

Rob- Robert isn’t much different to manage than the rest of my employees. He is a person – no super powers.  But he does establish a unique perspective on things because he gets to see so much new technology early.  Often earlier than almost anyone else. It helps him to spot trends that others might not be seeing yet.
Ajay – Hosting companies are so so many. What makes Rackspace special for different kinds of customers?
Rob- I think what we do better than anyone is add that human touch – the people really care about your business.  We are a company that is focused on building one of the greatest service companies on the planet.  We sell support.  Hosting is secondary to service. Our motto is Fanatic Support®

and we actually look for people focused on delivering amazing customer experiences during our interviewing and hiring practices. People that find a personal sense of pride and reward by helping others should apply at
Rackspace.  We are hiring like crazy!

Ajay – Where do you see technology and the internet 5 years down the line? (we will visit the answers in 5 years 🙂 )?
Rob- I think the shift to Cloud computing is going to be dramatic.  I think in five years we will be much further down that path.  The scaling, cost-effectiveness, and on-demand nature of the Cloud are just too compelling for companies not to embrace. This changes business in fundamental ways – lower capital expenses, no need for in house IT staff, etc will save companies a lot of money and let them focus more on their core businesses. Computing will become another utility.  I also think mobile use of computing will be much more common than it is today.  And it is VERY common today.  Phones will replace car keys and credit cards (they already are). This too will drive use of Cloud computing  because we all want our data wherever we are – on whatever computing device we happen tobe using.
Ajay- GoDaddy CEO shoots elephants. What do you do in your  spare time, if any.
Rob- Well, I don’t hunt.  We do shoot a lot of video though! I enjoy playing poker, specifically Texas Hold ’em.  It is a very people oriented game, and people are my passion.

Brief Biography- (in his own words from http://www.lagesse.org/about/)

My technical background includes working on the development of WiFi, writing wireless applications for the Apple Newton, mentoring/managing several software-based start-ups, running software quality assurance teams and more. In 2008 I joined Rackspace as an employee – a “Racker”.  I was previously a 7 year customer and the company impressed me. My initial role was as Director of Software Development for the Rackspace Cloud.  It was soon evident that I was better suited to a customer facing role since I LOVE talking to customers. I am currently the Director of Customer Development Chief Disruption Officer.  I manage building43 and enjoy working with Robert Scoble and Rocky Barbanica to make that happen.  The org chart says they work for me.  Reality tells me the opposite :)

Go take a look – I’m proud of what we are building there (pardon the pun!).

I do a lot of other stuff at Rackspace – mostly because they let me!  I love a company that lets me try. Rackspace does that.Going further back, I have been a Mayor (in Hawaii). I have written successful shareware software. I have managed employees all over the world. I have been all over the world. I have also done roofing, repaired high end print-shop equipment, been a Neonatal Respiratory Therapist, done CPR on a boat, in a plane, and in a hardware store (and of course in hospitals).

I have treated jumpers from the Golden Gate Bridge – and helped save a few. I have lived in Illinois (Kankakee), California (San Diego, San Francisco and Novato), Texas (Corpus Christi and San Antonio), Florida (Pensacola and Palm Bay), Hawaii (Honolulu/Fort Shafter) and several other places for shorter durations.

For the last 8+ years I have been a single parent – and have done an amazing job (yes, I am a proud papa) thanks to having great kids.  They are both in College now – something I did NOT manage to accomplish. I love doing anything someone thinks I am not qualified to do.

I can be contacted at rob (at) lagesse (dot) org

you can follow Rob at http://twitter.com/kr8tr

Related articles