Book Review – Big Data Analytics with R and Hadoop

I have written about Vignesh ‘s impressive work in R before including helping update the RGoogleAnalytics package for the API changes while at Tatvic* He is quite young and very eager to contribute to open source and knowledge.

This is a fairly timely impressive book given that both R and Hadoop are hot topics, have a lot of noise and hoopla around them, and need a straight forward explanation on how to do things using R and Hadoop. It demystifies both R and Hadoop sufficiently for you to actually not be intimidated at the thought  of learning multiple languages (R / Java/ Map Reduce), multiple paradigms (distributed computing and analysis) and multiple installations ( R/ Hadoop/RHadoop). Sufficient to say if the future belongs to Big Data/ Hadoop. Linux users will have it easier than Windows people.

One main criticism I found is to the lay reader everything is written in bullet points which can affect the readability if you are trying to get the big picture. However for the technical user or reader this is really a brilliant way, as everything is neatly written as do this and then do that etc.

The book thus aims to be more of a tutorial and has many nice examples too. I wish however a few more examples from Industry would have added more juice in this. I therefore hope for a companion site which has all the R code and datasets for testing and trying out the business analytics examples .

One wishes the author had written more about the biglm, ff  packages or even RevoScaleR packages . Chapter 5 with Data Analytics should have been more elaborate.  This can be done with more references – the section on visualizing data is just  2 pages and ignores some packages like GoogleVis or even bigvis package. The section about MongoDB and other data types is very useful but again is much more technical and much less analytical. For eg. when does one typically encounter MongoDB versus other data types- what are the drawbacks etc

This is thus a very practical handbook for the tech minded and it is quite affordable for the ebook ( Indian version is just 3.5 $)

I recommend this book highly for people who are aiming to practically implement Big Data Analytics . It is not for statisticians or business users but for people who actually want to set up the whole thing.

Please take a look at and try it out for a price of less than a (Starbucks!) latte or  a movie DVD .


R in the cloud – Revolution takes to AWS

Finally the people at Revolution Analytics have made their software available on AWS .Interesting development and it remains how it will be followed by other providers in stats software.

Users now have the opportunity to perform statistical analysis and advanced analytics on data sets they might have stored in Amazon’s cloud-based object store Simple Storage Service (S3) or access data from Amazon’s Relational Data Service (RDS).

The cloud offers many benefits to the user, and the AWS Marketplace is no exception. The ability to spin up pre-installed versions of RRE 7 takes all the guesswork out of deployment and provides for a consistent and reliable experience with the software.  Within minutes a user can gain access to R-based analysis from anywhere he or she has an Internet connection.

The Windows version is accessed via Windows Remote Desktop and leverages RRE DevelopR IDE. The Linux version is browser-based and leverages RStudio Server Pro to provide a multi-user IDE.  Both versions are available on instances from 2 – 32 vCPUs and can handle data sets of up to 1 TB for RRE ScaleR analysis. The solution is single-instance only and does not currently offer support for grids or clusters

Technical Details

  • General, Compute, Memory and Storage instances available, 2-32 vCPUs
  • Instances with attached storage recommended. Long-term storage requires EBS or backup to S3
  • Single-server instances only (no cluster or grid support).
  • Revolution R Enterprise DeployR not included.
  • Tech support forums monitored from Sunday, 5:00 PM PDT to Friday, 5:00 PM PDT. Tech support provided in English to registered users only.

Windows Instances

Platform: Windows Server 2008 R2
Revolution R Enterprise version: 7.0.0 (includes R 3.0.0)
Client Requirements: Windows Remote Desktop to access Revolution R Enterprise DeployR IDE

Linux Instances

Platform: Redhat Enterprise Linux 6.4
Revolution R Enterprise version: 7.0.0 (includes R 3.0.0)
Client Requirements: Compatible browse

Try one instance of this product for 14 days. There will be no software charges but AWS infrastructure charges still apply. Free Trials will automatically convert to a paid subscription upon expiration.
Hourly Fees (includes Windows 2008 R2 2008R2 X64)
Total hourly fees will vary by instance type and EC2 region.
EC2 Instance Type Software EC2 Total
Standard Large (m1.large) $2.50/hr $0.364/hr $2.864/hr
Standard XL (m1.xlarge) $5.00/hr $0.728/hr $5.728/hr
High-Memory 2XL (m2.2xlarge) $5.00/hr $1.02/hr $6.02/hr
High-Memory 4XL (m2.4xlarge) $10.00/hr $2.04/hr $12.04/hr
High-CPU XL (c1.xlarge) $5.00/hr $0.90/hr $5.90/hr
High I/O 4XL (hi1.4xlarge) $20.00/hr $3.58/hr $23.58/hr
Cluster Compute 8XL (cc2.8xlarge) $20.00/hr $2.97/hr $22.97/hr
EBS Storage Fees
$0.05 / GB / Month for Standard EBS Storage


Big Data Evil Empire

  1. Much more progress has been made in data storage , data querying and data analysis of huge amounts of personally identifiable information , than in encrypting such information
  2. Big Data has as much dual use usage for governments and corporations as uranium has for building bombs or power plants.
  3. There is as much lucre and potential revenue for encrypted data streams in the cloud era – as there for anti virus software in the PC era
  4. Tracking citizens totally is evil- the total costs of such programs is unjustified given the thwarted terrorism plots by Big Data ‘s Cyber Spying. At best I can understand governments spying on citizen’s of other countries to gain advantages in trade
  5. The American dominance of cyber spying and big data threaten to unravel and undermine it’s credibility as de facto leader of the Internet. It proves China’s vision of a walled off internet makes sense and that is a dangerous precedent which could lead to the break up of the internet along national boundaries of electronic fire walls.

Writing on APIs for Programmable Web

I have been writing free lance on APIs for Programmable Web. Here is an updated list of the articles, many of these would be of interest to analytics users. Note- some of these are interviews and they are in bold. Note to regular readers: I keep updating this list , and at each updation bring it to the front page, then allowing the blog postings to slide it down!

Scoreoid Aims to Gamify the World Using APIs January 27th, 2014’s Plot to Visualize More Data January 22nd, 2014

LumenData’s Acquisition of is a Win-Win January 8th, 2014

Yactraq API Sees Huge Growth in 2013  January 6th, 2014 Describes a Better Way to Extract Data December 20th, 2013

Exclusive Interview: App Store Analytics API December 4th, 2013

APIs Enter 3d Printing Industry November 29th, 2013

PW Interview: José Luis Martinez of Textalytics November 6th, 2013

PW Interview Simon Chan PredictionIO November 5th, 2013

PW Interview: Scott Gimpel Founder and CEO October 23rd, 2013

PW Interview Brandon Levy, cofounder and CEO of Stitch Labs October 8th, 2013

PW Interview: Jolo Balbin Co-Founder Text Teaser  September 18th, 2013

PW Interview:Bob Bickel CoFounder Redline13 July 29th, 2013

PW Interview : Brandon Wirtz CTO   July 4th, 2013

PW Interview: Andy Bartley, CEO  June 4th, 2013

PW Interview: Francisco J Martin, CEO 2013/05/30

PW Interview: Tal Rotbart Founder- CTO, SpringSense 2013/05/28

PW Interview: Jeh Daruwala CEO Yactraq API, Behavorial Targeting for videos 2013/05/13

PW Interview: Michael Schonfeld of Dwolla API on Innovation Meeting the Payment Web  2013/05/02

PW Interview: Stephen Balaban of Lamda Labs on the Face Recognition API  2013/04/29

PW Interview: Amber Feng, Stripe API, The Payment Web 2013/04/24

PW Interview: Greg Lamp and Austin Ogilvie of Yhat on Shipping Predictive Models via API   2013/04/22

Google Mirror API documentation is open for developers   2013/04/18

PW Interview: Ricky Robinett, API, Ordering Food meets API    2013/04/16

PW Interview: Jacob Perkins, Text Processing API, NLP meets API   2013/04/10

Amazon EC2 On Demand Windows Instances -Prices reduced by 20%  2013/04/08

Amazon S3 API Requests prices slashed by half  2013/04/02

PW Interview: Stuart Battersby, Chatterbox API, Machine Learning meets Social 2013/04/02

PW Interview: Karthik Ram, rOpenSci, Wrapping all science API2013/03/20

Viralheat Human Intent API- To buy or not to buy 2013/03/13

Interview Tammer Kamel CEO and Founder Quandl 2013/03/07

YHatHQ API: Calling Hosted Statistical Models 2013/03/04

Quandl API: A Wikipedia for Numerical Data 2013/02/25

Amazon Redshift API is out of limited preview and available! 2013/02/18

Windows Azure Media Services REST API 2013/02/14

Data Science Toolkit Wraps Many Data Services in One API 2013/02/11

Diving into Codeacademy’s API Lessons 2013/01/31

Google APIs finetuning Cloud Storage JSON API 2013/01/29

Ergast API Puts Car Racing Fans in the Driver’s Seat 2012/12/05
Springer APIs- Fostering Innovation via API Contests 2012/11/20
Statistically programming the web – Shiny,HttR and RevoDeploy API 2012/11/19
Google Cloud SQL API- Bigger ,Faster and now Free 2012/11/12
A Look at the Web’s Most Popular API -Google Maps API 2012/10/09
Cloud Storage APIs for the next generation Enterprise 2012/09/26 API: Sultan of Musical APIs 2012/09/12
Socrata Data API: Keeping Government Open 2012/08/29
BigML API Gets Bigger 2012/08/22
Bing APIs: the Empire Strikes Back 2012/08/15
Google Cloud SQL: Relational Database on the Cloud 2012/08/13
Google BigQuery API Makes Big Data Analytics Easy 2012/08/05
Your Store in The Cloud -Google Cloud Storage API 2012/08/01
Predict the future with Google Prediction API 2012/07/30
The Romney vs Obama API 2012/07/27