Interview Alvaro Tejada Galindo, SAP Labs Montreal, Using SAP Hana with #Rstats

Here is a brief interview with Alvaro Tejada Galindo aka Blag who is a developer working with SAP Hana and R at SAP Labs, Montreal. SAP Hana is SAP’s latest offering in BI , it’s also a database and a computing environment , and using R and HANA together on the cloud can give major productivity gains in terms of both speed and analytical ability, as per preliminary use cases.

Ajay- Describe how you got involved with databases and R language.

Blag- I used to work as an ABAP Consultant for 11 years, but also been involved with programming since the last 13 years, so I was in touch with SQLServer, Oracle, MySQL and SQLite. When I joined SAP, I heard that SAP HANA was going to use an statistical programming language called “R”. The next day I started my “R” learning.

Ajay- What made the R language a fit for SAP HANA. Did you consider other languages? What is your view on Julia/Python/SPSS/SAS/Matlab languages

Blag- I think “R” is a must for SAP HANA. As the fastest database in the market, we needed a language that could help us shape the data in the best possible way. “R” filled that purpose very well. Right now, “R” is not the only language as “L” can be used as well (http://wiki.tcl.tk/17068) …not forgetting “SQLScript” which is our own version of SQL (http://goo.gl/x3bwh) . I have to admit that I tried Julia, but couldn’t manage to make it work. Regarding Python, it’s an interesting question as I’m going to blog about Python and SAP HANA soon. About Matlab, SPSS and SAS I haven’t used them, so I got nothing to say there.

Ajay- What is your view on some of the limitations of R that can be overcome with using it with SAP HANA.

Blag- I think mostly the ability of SAP HANA to work with big data. Again, SAP HANA and “R” can work very nicely together and achieve things that weren’t possible before.

Ajay- Have you considered other vendors of R including working with RStudio, Revolution Analytics, and even Oracle R Enterprise.

Blag- I’m not really part of the SAP HANA or the R groups inside SAP, so I can’t really comment on that. I can only say that I use RStudio every time I need to do something with R. Regarding Oracle…I don’t think so…but they can use any of our products whenever they want.

Ajay- Do you have a case study on an actual usage of R with SAP HANA that led to great results.

Blag- Right now the use of “R” and SAP HANA is very preliminary, I don’t think many people has start working on it…but as an example that it works, you can check this awesome blog entry from my friend Jitender Aswani “Big Data, R and HANA: Analyze 200 Million Data Points and Later Visualize Using Google Maps “ (http://allthingsr.blogspot.com/#!/2012/04/big-data-r-and-hana-analyze-200-million.html)

Ajay- Does your group in SAP plan to give to the R ecosystem by attending conferences like UseR 2012, sponsoring meets, or package development etc

Blag- My group is in charge of everything developers, so sure, we’re planning to get more in touch with R developers and their ecosystem. Not sure how we’re going to deal with it, but at least I’m going to get myself involved in the Montreal R Group.

About-

http://scn.sap.com/people/alvaro.tejadagalindo3


Name:	Alvaro Tejada Galindo

Email:	a.tejada.galindo@sap.com

Profession:	Development
Company:	SAP Canada Labs-Montreal
Town/City:	Montreal
Country:	Canada
Instant Messaging Type:	Twitter
Instant Messaging ID:	Blag
Personal URL:	http://blagrants.blogspot.com
Professional Blog URL:	http://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/u/252210910
My Relation to SAP:	employee
Short Bio:	Development Expert for the Technology Innovation and Developer Experience team.Used to be an ABAP Consultant for the last 11 years. Addicted to programming since 1997.

http://www.sap.com/solutions/technology/in-memory-computing-platform/hana/overview/index.epx

and from

http://en.wikipedia.org/wiki/SAP_HANA

SAP HANA is SAP AG’s implementation of in-memory database technology. There are four components within the software group:^[1]

SAP HANA DB (or HANA DB) refers to the database technology itself,
SAP HANA Studio refers to the suite of tools provided by SAP for modeling,
SAP HANA Appliance refers to HANA DB as delivered on partner certified hardware (see below) as anappliance. It also includes the modeling tools from HANA Studio as well replication and data transformation tools to move data into HANA DB,^[2]
SAP HANA Application Cloud refers to the cloud based infrastructure for delivery of applications (typically existing SAP applications rewritten to run on HANA).

R is integrated in HANA DB via TCP/IP. HANA uses SQL-SHM, a shared memory-based data exchange to incorporate R’s vertical data structure. HANA also introduces R scripts equivalent to native database operations like join or aggregation.^[20] HANA developers can write R scripts in SQL and the types are automatically converted in HANA. R scripts can be invoked with HANA tables as both input and output in the SQLScript. R environments need to be deployed to use R within SQLScript

More blog posts on using SAP and R together

Dealing with R and HANA

http://scn.sap.com/community/in-memory-business-data-management/blog/2011/11/28/dealing-with-r-and-hana
R meets HANA

http://scn.sap.com/community/in-memory-business-data-management/blog/2012/01/29/r-meets-hana

HANA meets R

http://scn.sap.com/community/in-memory-business-data-management/blog/2012/01/26/hana-meets-r
When SAP HANA met R – First kiss

http://scn.sap.com/community/developer-center/hana/blog/2012/05/21/when-sap-hana-met-r–first-kiss

Using RODBC with SAP HANA DB-

SAP HANA: My experiences on using SAP HANA with R

http://scn.sap.com/community/in-memory-business-data-management/blog/2012/02/21/sap-hana-my-experiences-on-using-sap-hana-with-r

and of course the blog that started it all-

Jitender Aswani’s http://allthingsr.blogspot.in/

PMML Augustus

Here is a new-old system in open source for

for building and scoring statistical models designed to work with data sets that are too large to fit into memory.

http://code.google.com/p/augustus/

Augustus is an open source software toolkit for building and scoring statistical models. It is written in Python and its
most distinctive features are:
• Ability to be used on sets of big data; these are data sets that exceed either memory capacity or disk capacity, so
that existing solutions like R or SAS cannot be used. Augustus is also perfectly capable of handling problems
that can fit on one computer.
• PMML compliance and the ability to both:
– produce models with PMML-compliant formats (saved with extension .pmml).
– consume models from files with the PMML format.
Augustus has been tested and deployed on serveral operating systems. It is intended for developers who work in the
financial or insurance industry, information technology, or in the science and research communities.
Usage
Augustus produces and consumes Baseline, Cluster, Tree, and Ruleset models. Currently, it uses an event-based
approach to building Tree, Cluster and Ruleset models that is non-standard.

New to PMML ?

Read on http://code.google.com/p/augustus/wiki/PMML

The Predictive Model Markup Language or PMML is a vendor driven XML markup language for specifying statistical and data mining models. In other words, it is an XML language so that Continue reading “PMML Augustus”

Using Google Fusion Tables from #rstats

But after all that- I was quite happy to see Google Fusion Tables within Google Docs. Databases as a service ? Not quite but still quite good, and lets see how it goes.

https://www.google.com/fusiontables/DataSource?dsrcid=implicit&hl=en_US&pli=1

http://googlesystem.blogspot.com/2011/09/fusion-tables-new-google-docs-app.html

But what interests me more is

http://code.google.com/apis/fusiontables/docs/developers_guide.html

The Google Fusion Tables API is a set of statements that you can use to search for and retrieve Google Fusion Tables data, insert new data, update existing data, and delete data. The API statements are sent to the Google Fusion Tables server using HTTP GET requests (for queries) and POST requests (for inserts, updates, and deletes) from a Web client application. The API is language agnostic: you can write your program in any language you prefer, as long as it provides some way to embed the API calls in HTTP requests.

The Google Fusion Tables API does not provide the mechanism for submitting the GET and POST requests. Typically, you will use an existing code library that provides such functionality; for example, the code libraries that have been developed for the Google GData API. You can also write your own code to implement GET and POST requests.

Also see http://code.google.com/apis/fusiontables/docs/sample_code.html

Google Fusion Tables API Sample Code

Libraries

SQL API

Language	Library	Public repository	Samples
Python	Fusion Tables Python Client Library	fusion-tables-client-python/	Samples
PHP	Fusion Tables PHP Client Library	fusion-tables-client-php/	Samples

Featured Samples

An easy way to learn how to use an API can be to look at sample code. The table above provides links to some basic samples for each of the languages shown. This section highlights particularly interesting samples for the Fusion Tables API.

SQL API

Language	Featured samples	API version
cURL	Hello, cURLA simple example showing how to use curl to access Fusion Tables.	SQL API
Google Apps Script	Google Apps Script ExampleA simple example showing how to run the SHOW TABLES query in Google Spreadsheets.	SQL API
Java	Hello, WorldA simple walkthrough that shows how the Google Fusion Tables API statements work. OAuth example on fusion-tables-apiThe Google Fusion Tables team shows how OAuth authorization enables you to use the Google Fusion Tables API from a foreign web server with delegated authorization.	SQL API
Python	Docs List ExampleDemonstrates how to: List tables Set permissions on tables Move a table to a folder	Docs List API
Android (Java)	Basic Sample ApplicationDemo application shows how to create a crowd-sourcing application that allows users to report potholes and save the data to a Fusion Table.	SQL API
JavaScript – FusionTablesLayer	Using the FusionTablesLayer, you can display data on a Google Map The basics Basic FusionTablesLayer example Multiple Layers per map Heatmap Changing the layer query Using a select menu Using a slider On zoom Using checkboxes Autocomplete Textbox Spatial Queries (see also “Working with Geographic Data”) Find features within a radius Find features within a bounding box Find the n nearest features Dynamic Styling Basic Styling Demo Multiple Columns (with Legend!) Advanced Custom Info Windows Info Window with Driving Directions Custom Marker Icons Legend Pan and Zoom to User-Entered Address Also check out FusionTablesLayer Builder, which generates all the code necessary to include a Google Map with a Fusion Table Layer on your own website.	FusionTablesLayer, Google Maps API
JavaScript – Google Chart Tools	Using the Google Chart Tools, you can request data from Fusion Tables to use in visualizations or to display directly in an HTML page. Note: responses are limited to 500 rows of data. Basic Request Data Table Visualization Bar Chart Visualization Word Cloud Visualization	Google Chart Tools

External Resources

Google Fusion Tables is dedicated to providing code examples that illustrate typical uses, best practices, and really cool tricks. If you do something with the Google Fusion Tables API that you think would be interesting to others, please contact us at googletables-feedback@google.com about adding your code to our Examples page.

Shape EscapeA tool for uploading shape files to Fusion Tables.
GDALOGR Simple Feature Library has incorporated Fusion Tables as a supported format.
Arc2CloudArc2Earth has included support for upload to Fusion Tables via Arc2Cloud.
Java and Google App EngineODK Aggregate is an AppEngine application by the Open Data Kit team, uses Google Fusion Tables to store survey data that is collected through input forms on Android mobile phones. Notable code:
- Create a Fusion Table – FusionTableServlet.java
- Insert data into a Fusion Table – SubmissionFusionTable.java
R packageAndrei Lopatenko has written an R interface to Fusion Tables so Fusion Tables can be used as the data store for R.
RubySimon Tokumine has written a Ruby gem for access to Fusion Tables from Ruby.

Updated-You can use Google Fusion Tables from within R from http://andrei.lopatenko.com/rstat/fusion-tables.R

ft.connect <- function(username, password) {
  url = "https://www.google.com/accounts/ClientLogin";
  params = list(Email = username, Passwd = password, accountType="GOOGLE", service= "fusiontables", source = "R_client_API")
 connection = postForm(uri = url, .params = params)
 if (length(grep("error", connection, ignore.case = TRUE))) {
 	stop("The wrong username or password")
 	return ("")
 }
 authn = strsplit(connection, "\nAuth=")[[c(1,2)]]
 auth = strsplit(authn, "\n")[[c(1,1)]]
 return (auth)
}

ft.disconnect <- function(connection) {
}

ft.executestatement <- function(auth, statement) {
      url = "http://tables.googlelabs.com/api/query"
      params = list( sql = statement)
      connection.string = paste("GoogleLogin auth=", auth, sep="")
      opts = list( httpheader = c("Authorization" = connection.string))
      result = postForm(uri = url, .params = params, .opts = opts)
      if (length(grep("<HTML>\n<HEAD>\n<TITLE>Parse error", result, ignore.case = TRUE))) {
      	stop(paste("incorrect sql statement:", statement))
      }
      return (result)
}

ft.showtables <- function(auth) {
   url = "http://tables.googlelabs.com/api/query"
   params = list( sql = "SHOW TABLES")
   connection.string = paste("GoogleLogin auth=", auth, sep="")
   opts = list( httpheader = c("Authorization" = connection.string))
   result = getForm(uri = url, .params = params, .opts = opts)
   tables = strsplit(result, "\n")
   tableid = c()
   tablename = c()
   for (i in 2:length(tables[[1]])) {
     	str = tables[[c(1,i)]]
   	    tnames = strsplit(str,",")
   	    tableid[i-1] = tnames[[c(1,1)]]
   	    tablename[i-1] = tnames[[c(1,2)]]
   	}
   	tables = data.frame( ids = tableid, names = tablename)
    return (tables)
}

ft.describetablebyid <- function(auth, tid) {
   url = "http://tables.googlelabs.com/api/query"
   params = list( sql = paste("DESCRIBE", tid))
   connection.string = paste("GoogleLogin auth=", auth, sep="")
   opts = list( httpheader = c("Authorization" = connection.string))
   result = getForm(uri = url, .params = params, .opts = opts)
   columns = strsplit(result,"\n")
   colid = c()
   colname = c()
   coltype = c()
   for (i in 2:length(columns[[1]])) {
     	str = columns[[c(1,i)]]
   	    cnames = strsplit(str,",")
   	    colid[i-1] = cnames[[c(1,1)]]
   	    colname[i-1] = cnames[[c(1,2)]]
   	    coltype[i-1] = cnames[[c(1,3)]]
   	}
   	cols = data.frame(ids = colid, names = colname, types = coltype)
    return (cols)
}

ft.describetable <- function (auth, table_name) {
   table_id = ft.idfromtablename(auth, table_name)
   result = ft.describetablebyid(auth, table_id)
   return (result)
}

ft.idfromtablename <- function(auth, table_name) {
    tables = ft.showtables(auth)
	tableid = tables$ids[tables$names == table_name]
	return (tableid)
}

ft.importdata <- function(auth, table_name) {
	tableid = ft.idfromtablename(auth, table_name)
	columns = ft.describetablebyid(auth, tableid)
	column_spec = ""
	for (i in 1:length(columns)) {
		column_spec = paste(column_spec, columns[i, 2])
		if (i < length(columns)) {
			column_spec = paste(column_spec, ",", sep="")
		}
	}
	mdata = matrix(columns$names,
	              nrow = 1, ncol = length(columns),
	              dimnames(list(c("dummy"), columns$names)), byrow=TRUE)
	select = paste("SELECT", column_spec)
	select = paste(select, "FROM")
	select = paste(select, tableid)
	result = ft.executestatement(auth, select)
    numcols = length(columns)
    rows = strsplit(result, "\n")
    for (i in 3:length(rows[[1]])) {
    	row = strsplit(rows[[c(1,i)]], ",")
    	mdata = rbind(mdata, row[[1]])
   	}
   	output.frame = data.frame(mdata[2:length(mdata[,1]), 1])
   	for (i in 2:ncol(mdata)) {
   		output.frame = cbind(output.frame, mdata[2:length(mdata[,i]),i])
   	}
   	colnames(output.frame) = columns$names
    return (output.frame)
}

quote_value <- function(value, to_quote = FALSE, quote = "'") {
	 ret_value = ""
     if (to_quote) {
     	ret_value = paste(quote, paste(value, quote, sep=""), sep="")
     } else {
     	ret_value = value
     }
     return (ret_value)
}

converttostring <- function(arr, separator = ", ", column_types) {
	con_string = ""
	for (i in 1:(length(arr) - 1)) {
		value = quote_value(arr[i], column_types[i] != "number")
		con_string = paste(con_string, value)
	    con_string = paste(con_string, separator, sep="")
	}

    if (length(arr) >= 1) {
    	value = quote_value(arr[length(arr)], column_types[length(arr)] != "NUMBER")
    	con_string = paste(con_string, value)
    }
}

ft.exportdata <- function(auth, input_frame, table_name, create_table) {
	if (create_table) {
       create.table = "CREATE TABLE "
       create.table = paste(create.table, table_name)
       create.table = paste(create.table, "(")
       cnames = colnames(input_frame)
       for (columnname in cnames) {
         create.table = paste(create.table, columnname)
    	 create.table = paste(create.table, ":string", sep="")
    	   if (columnname != cnames[length(cnames)]){
    		  create.table = paste(create.table, ",", sep="")
           }
       }
      create.table = paste(create.table, ")")
      result = ft.executestatement(auth, create.table)
    }
    if (length(input_frame[,1]) > 0) {
    	tableid = ft.idfromtablename(auth, table_name)
	    columns = ft.describetablebyid(auth, tableid)
	    column_spec = ""
	    for (i in 1:length(columns$names)) {
		   column_spec = paste(column_spec, columns[i, 2])
		   if (i < length(columns$names)) {
			  column_spec = paste(column_spec, ",", sep="")
		   }
	    }
    	insert_prefix = "INSERT INTO "
    	insert_prefix = paste(insert_prefix, tableid)
    	insert_prefix = paste(insert_prefix, "(")
    	insert_prefix = paste(insert_prefix, column_spec)
    	insert_prefix = paste(insert_prefix, ") values (")
    	insert_suffix = ");"
    	insert_sql_big = ""
    	for (i in 1:length(input_frame[,1])) {
    		data = unlist(input_frame[i,])
    		values = converttostring(data, column_types  = columns$types)
    		insert_sql = paste(insert_prefix, values)
    		insert_sql = paste(insert_sql, insert_suffix) ;
    		insert_sql_big = paste(insert_sql_big, insert_sql)
    		if (i %% 500 == 0) {
    			ft.executestatement(auth, insert_sql_big)
    			insert_sql_big = ""
    		}
    	}
        ft.executestatement(auth, insert_sql_big)
    }
}

How to surf anonymously on the mobile- Use Orbot

This is an interesting use case of anonymous surfing through mobile by using Tor Project on the Android Mobile OS.

Source- https://guardianproject.info/apps/orbot/

Orbot requires different configuration depending on the Android operating system version it is used on.

For standard Android 1.x devices (G1, MyTouch3G, Hero, Droid Eris, Cliq, Moment)

WEB BROWSING: You can use the Orweb Privacy Browser which we offer, which only works via Orbot and Tor.
For Instant Messsaging, please try Gibberbot which provides integrated, optional support for Orbot and Tor.

For Android 2.x devices: Droid, Nexus, Evo, Galaxy

WEB BROWSING: Non-rooted devices should use Firefox for Android with our ProxyMob Add-On to browse via the Tor network. Rooted devices can take advantage of transparent proxying (see below) and do not need an additional app installed.
Transparent Proxying: You must root your device in order for Orbot to work transparently for all web and DNS traffic. If you root your device, whether it is 1.x or 2.x based, Orbot will automatically, transparently proxy all web traffic on port 80 and 443 and all DNS requests. This includes the built-in Browser, Gmail, YouTube, Maps and any other application that uses standard web traffic.
For Instant Messsaging, please try Gibberbot which provides integrated, optional support for Orbot and Tor.

Developers

Source code is available via SVN: svn co https://svn.torproject.org/svn/projects/android/ and Git: git clone https://gitweb.torproject.org/orbot.git

The Amazing Google MapMaker

Literally helping make the world a better place by making it- see real time edits of people

http://www.google.com/mapmaker/pulse

Map Maker’s review process allows users to easily edit content, while ensuring that data quality remains intact.

The first time a Map Maker user makes edits to a map, the edits may require review and approval before the edits will be published. Once a Map Maker user has made a few approved edits, most of the subsequent edits will go live automatically. However, some types of edits or edits in specific regions will always require review, regardless of how experienced the mapper is. In addition, some edits may require multiple reviews before the edits appear on Google Maps.

Why do edits need to be reviewed?

There are several reasons why edits are sent for review:

To provide feedback to new editors, and to help them understand the mapping process.
To provide feedback on more complex edits.
To ensure that sensitive edits meet the Reviewing Guidelines.

Learn how you can review edits in Map Maker.

Review some edits and get your edits reviewed faster!

When you review the edits made by your co-mappers, your edits are prioritized in the reviewing queue so that they appear above other edits. This enables other mappers to review them and your edits get published on Maps faster! from-
http://www.google.com/support/mapmaker/bin/static.py?page=guide.cs&guide=1094316

Best of Google Plus – Week 1 Top1/0

Stuff I like from Google Plus meme- animated GIFS are just one of them-

LIST OF GOOGLERS ON GOOGLE+

OK, this was fun to put together — love how active the Googlers are on this platform! Please feel free to add anyone I missed and share this. Circle the ones that most interest you based on what they do, or circle them all as I did 🙂

Co-Founders
+Sergey Brin
+Larry Page

VPs/Senior VPs
+Vic Gundotra (Engineering)
+Bradley Horowitz (Product Management)
+Jeff Huber (Commerce & Local)
+Marissa Mayer (Local, Maps & Location Services)

Community Managers
+Brian Rose
+Toby Stein
+Natalie Villalobos

Product Managers
+Anish Acharya (Google+ Mobile)
+Shimrit Ben-Yair
+Frances Haugen (Google+ Profiles)
+Caroline McCarthy (Marketing)
+Jonathan McPhie
+Joe Rideout
+Punit Soni (Google+ Mobile)

Engineering Directors/Managers
+Chee Chew
+Dave Besbris
+Chris Millikin

Software Engineers
+Eric W. Barndollar (Google+)
+Andrew Bunner (Google+)
+David Byttow
+Eric Cattell (Social Graph Tech Lead)
+John Costigan (Google Profiles)
+Matt Cutts (Webspam)
+Pavan Desikan (Google+/Gmail)
+Kelly Ellis
+Trey Harris (Site Reliability)
+Griff Hazen
+Andy Hertzfeld
+Matt Keoshkerian
+Todd Knight
+Jean-Christophe Lilot
+Lan Liu
+Vincent Mo (Google+ Photos)
+Dobromir Montauk (Google+ Infrastructure)
+Stephen Ng (Gmail)
+Owen Prater
+Joseph Smarr (Technical)
+Martin Strauss
+Na Tang
+Yonatan Zunger (Social)

Consumer Operations Manager
+Michael Hermeston (Google+ Support)

Developer Advocates
+Chris Chabot (Developer Relations)
+Timothy Jordan

Designers
+Brett Lider (Product/User Experience)
+Jonathan Terleski (Google+)
+Charles Warren (User Experience Lead, Google Social)

Program Managers
+Julian Harris (Technical)
+Adam Lasnik (Google Map Maker)

Tech Lead Manager
+Natalie Glance (Google Shopping)
Test Engineer
+Erick Fejta (Tester for Google Storage)

Account Executive
+Dave Miller (Local & Education)

President, Enterprise
+Dave Girouard (Cloud Apps)
3. Jokes-
I have a friend on Facebook that seemed suicidal, said he was standing on a ledge….so I poked him

4.Google Squash

5.Social Media Explained

6. Google Plus slaps Facebook AND Troopers Googling for Droids

7. Why did Google Wave Fail

8. When Google+ is available to the public..

9. Evolution

10. Safe Tweeting

Special Mentions-

QGIS and R

Image via Wikipedia

Qgis is Quantum GIS http://www.qgis.org/

Quantum GIS (QGIS) is a user friendly Open Source Geographic Information System (GIS) licensed under the GNU General Public License. QGIS is an official project of the Open Source Geospatial Foundation (OSGeo). It runs on Linux, Unix, MacOSX, and Windows and supportsnumerous vector, raster, and database formats and functionalities.

Learn more about QGIS

Quantum GIS provides a continously growing number of capabilities provided by core functions and plugins. You can visualize, manage, edit, analyse data, and compose printable maps

Also you can use both Qgis and R through Python (!!!)

http://www.qgis.org/wiki/HomeRange_plugin#Home-range_analyses_in_QGIS_using_R_through_Python

Interesting app for webs (sometimes better suited than some R map packages)

https://plugins.qgis.org/plugins/HomeRange_plugin/

Based on a Google Summer of Code _

Also

https://sites.google.com/site/eospansite/introqgis_r

and

HomeRange_plugin

http://hub.qgis.org/projects/quantum-gis/wiki/HomeRange_plugin