Tag: data
Software Review- BigML.com – Machine Learning meets the Cloud
I had a chance to dekko the new startup BigML https://bigml.com/ and was suitably impressed by the briefing and my own puttering around the site. Here is my review-
1) The website is very intutively designed- You can create a dataset from an uploaded file in one click and you can create a Decision Tree model in one click as well. I wish other cloud computing websites like Google Prediction API make design so intutive and easy to understand. Also unlike Google Prediction API, the models are not black box models, but have a description which can be understood.
2) It includes some well known data sources for people trying it out. They were kind enough to offer 5 invite codes for readers of Decisionstats ( if you want to check it yourself, use the codes below the post, note they are one time only , so the first five get the invites.
BigML is still invite only but plan to get into open release soon.
3) Data Sources can only be by uploading files (csv) but they plan to change this hopefully to get data from buckets (s3? or Google?) and from URLs.
4) The one click operation to convert data source into a dataset shows a histogram (distribution) of individual variables.The back end is clojure , because the team explained it made the easiest sense and fit with Java. The good news (?) is you would never see the clojure code at the back end. You can read about it from http://clojure.org/
As cloud computing takes off (someday) I expect clojure popularity to take off as well.
Clojure is a dynamic programming language that targets the Java Virtual Machine (and the CLR, and JavaScript). It is designed to be a general-purpose language, combining the approachability and interactive development of a scripting language with an efficient and robust infrastructure for multithreaded programming. Clojure is a compiled language – it compiles directly to JVM bytecode, yet remains completely dynamic. Every feature supported by Clojure is supported at runtime. Clojure provides easy access to the Java frameworks, with optional type hints and type inference, to ensure that calls to Java can avoid reflection.
Clojure is a dialect of Lisp
5) As of now decision trees is the only distributed algol, but they expect to roll out other machine learning stuff soon. Hopefully this includes regression (as logit and linear) and k means clustering. The trees are created and pruned in real time which gives a slightly animated (and impressive effect). and yes model building is an one click operation.
The real time -live pruning is really impressive and I wonder why /how it can ever be replicated in other software based on desktop, because of the sheer interactive nature.
Making the model is just half the work. Creating predictions and scoring the model is what is really the money-earner. It is one click and customization is quite intuitive. It is not quite PMML compliant yet so I hope some Zemanta like functionality can be added so huge amounts of models can be applied to predictions or score data in real time.
If you are a developer/data hacker, you should check out this section too- it is quite impressive that the designers of BigML have planned for API access so early.
BigML.io gives you:
- Secure programmatic access to all your BigML resources.
- Fully white-box access to your datasets and models.
- Asynchronous creation of datasets and models.
- Near real-time predictions.
Note: For your convenience, some of the snippets below include your real username and API key.
Please keep them secret.
REST API
BigML.io conforms to the design principles of Representational State Transfer (REST). BigML.io is enterely HTTP-based.
BigML.io gives you access to four basic resources: Source, Dataset, Model and Prediction. You cancreate, read, update, and delete resources using the respective standard HTTP methods: POST, GET,PUT and DELETE.
All communication with BigML.io is JSON formatted except for source creation. Source creation is handled with a HTTP PUT using the “multipart/form-data” content-type
HTTPS
All access to BigML.io must be performed over HTTPS
and https://bigml.com/developers/quick_start ( In think an R package which uses JSON ,RCurl would further help in enhancing ease of usage).
Summary-
Overall a welcome addition to make software in the real of cloud computing and statistical computation/business analytics both easy to use and easy to deploy with fail safe mechanisms built in.
Check out https://bigml.com/ for yourself to see.
The invite codes are here -one time use only- first five get the invites- so click and try your luck, machine learning on the cloud.
If you dont get an invite (or it is already used, just leave your email there and wait a couple of days to get approval)
Oracle R Updated!
Interesting message from https://blogs.oracle.com/R/ the latest R blog
_——–_
Oracle just released the latest update to Oracle R Enterprise, version 1.1. This release includes the Oracle R Distribution (based on open source R, version 2.13.2), an improved server installation, and much more. The key new features include:
- Extended Server Support: New support for Windows 32 and 64-bit server components, as well as continuing support for Linux 64-bit server components
- Improved Installation: Linux 64-bit server installation now provides robust status updates and prerequisite checks
- Performance Improvements: Improved performance for embedded R script execution calculations
In addition, the updated ROracle package, which is used with Oracle R Enterprise, now reads date data by conversion to character strings.
We encourage you download Oracle software for evaluation from the Oracle Technology Network. See these links for R-related software: Oracle R Distribution, Oracle R Enterprise, ROracle, Oracle R Connector for Hadoop. As always, we welcome comments and questions on the Oracle R Forum.
Oracle R Distribution 2-13.2 Update Available
Oracle has released an update to the Oracle R Distribution, an Oracle-supported distribution of open source R. Oracle R Distribution 2-13.2 now contains the ability to dynamically link the following libraries on both Windows and Linux:
- The Intel Math Kernel Library (MKL) on Intel chips
- The AMD Core Math Library (ACML) on AMD chips
To take advantage of the performance enhancements provided by Intel MKL or AMD ACML in Oracle R Distribution, simply add the MKL or ACML shared library directory to the LD_LIBRARY_PATH system environment variable. This automatically enables MKL or ACML to make use of all available processors, vastly speeding up linear algebra computations and eliminating the need to recompile R. Even on a single core, the optimized algorithms in the Intel MKL libraries are faster than using R’s standard BLAS library.
Open-source R is linked to NetLib’s BLAS libraries, but they are not multi-threaded and only use one core. While R’s internal BLAS are efficient for most computations, it’s possible to recompile R to link to a different, multi-threaded BLAS library to improve performance on eligible calculations. Compiling and linking to R yourself can be involved, but for many, the significantly improved calculation speed justifies the effort. Oracle R Distribution notably simplifies the process of using external math libraries by enabling R to auto-load MKL orACML. For R commands that don’t link to BLAS code, taking advantage of database parallelism usingembedded R execution in Oracle R Enterprise is the route to improved performance.
For more information about rebuilding R with different BLAS libraries, see the linear algebra section in the R Installation and Administration manual. As always, the Oracle R Distribution is available as a free download to anyone. Questions and comments are welcome on the Oracle R Forum.
Easter Eggs in #Rstats
Yes.
Cite-http://en.wikipedia.org/wiki/Easter_egg_(media)
A virtual Easter egg is an intentional hidden message, in-joke, or feature in a work such as a computer program, web page, video game, movie, book, or crossword. The term was coined — according to Warren Robinett — by Atari after they were pointed to the secret message left by Robinett in the game Adventure.[1] It draws a parallel with the custom of the Easter egg hunt observed in many Western nations as well as the last Russian imperial family’s tradition of giving elaborately jeweled egg-shaped creations by Carl Fabergé which contained hidden surprises
In R.
Cite-http://stackoverflow.com/questions/7910270/are-there-any-easter-eggs-in-base-r-or-in-major-packages
I like this
just type
example(readLine)
and these two
on 32 bit R type
memory.limit(4096)
and on any version try four question marks
Perhaps the prettiest eggs are the demos in animation package.
But there is magic in asking for help on internal functions in R
Just type-
and you get the sobering thought that you probably are a R Muggle
Call an Internal Function
Description
.Internal performs a call to an internal code which is built in to the R interpreter.
Only true R wizards should even consider using this function, and only R developers can add to the list of internal functions.
Usage
.Internal(call)
Arguments
call |
a call expression |
See Also
.Primitive, .External (the nearest equivalent available to users).
I liked that I could see the actual internal functions in svn at http://svn.r-project.org/R/trunk/src/main/names.c
The opening of the internals document floored me.
It must have been a curious year in 2003-4 when the copyright of R was held (briefly it seems) by the R Foundation and also by the R Development Core Team. (which sounds better?)
* R : A Computer Language for Statistical Data Analysis * Copyright (C) 1995, 1996 Robert Gentleman and Ross Ihaka * Copyright (C) 1997--2012 The R Development Core Team * Copyright (C) 2003, 2004 The R Foundation
My contribution
R help discourages for loop
Try ??for or ?for
you go into a loop till you hit escape
Facebook Search- The fall of the machines
Increasingly I am beginning to search more and more on Facebook. This is for the following reasons-
1) Facebook is walled off to Google (mostly). While within Facebook , I get both people results and content results (from Bing).
Bing is an okay alternative , though not as fast as Google Instant.
2) Cleaner Web Results When Facebook increases the number of results from 3 top links to say 10 top links, there should be more outbound traffic from FB search to websites.For some reason Google continues to show 14 pages of results… Why? Why not limit to just one page.
3) Better People Search than Pipl and Google. But not much (or any) image search. This is curious and I am hoping the Instagram results would be added to search results.
4) I am hoping for any company Facebook or Microsoft to challenge Adsense . Adwords already has rivals. Adsense is a de facto monopoly and my experiences in advertising show that content creators can make much more money from a better Adsense (especially ) if Adsense and Adwords do not have a conflict of interest from same advertisers.
Adwords should have been a special case of Adsense for Google.com but it is not.
5) Machine learning can only get you from tau to delta tau. When ad click behavior is inherently dependent on humans who behave mostly on chaotic , or genetic models than linear CPC models. I find FB has an inherent advantage in the quantity and quality of data collected on people behavior rather than click behavior. They are also more aggressive and less apologetic about behavorially targeted ads.
Additional point- Analytics for Google Analytics is not as rich as analytics from Facebook pages in terms of demographic variables. This can be tested by anyone.
Cricinfo StatsGuru Database for Statistical and Graphical Analysis
Data from the ESPN Cricinfo website is available from the STATSGURU website.
The url is of the form-
http://stats.espncricinfo.com/ci/engine/stats/index.html?
class=1;team=6;template=results;type=batting
If you break down this URL to get more statistics on cricket, you can choose the following parameters.
class
1=Test
2=ODI
3=T20I
11=Test+ODI+T20I
team
1=England
2=Australia
3=South America
4-West Indies
5=New Zealand
6=India ,7=Pakistan and 8=Sri Lanka
type
batting
bowling
fielding
allround
fow
official
team
aggregate
ESPN Terms of Use are here-you may need to check this before trying any web scraping.
http://www.espncricinfo.com/ci/content/site/company/terms_use.html
However ESPN has unleashed the API (including both free and premium)for Developers at http://developer.espn.com/docs.
and especially these sports http://developer.espn.com/docs/headlines#parameters
| /sports | News across all sports/sections | ||||
| /sports/baseball/mlb | Major League Baseball (MLB) | ||||
| /sports/basketball/mens-college-basketball | NCAA Men’s College Basketball | ||||
| /sports/basketball/nba | National Basketball Association (NBA) | ||||
| /sports/basketball/wnba | Women’s National Basketball Association (WNBA) | ||||
| /sports/basketball/womens-college-basketball | NCAA Women’s College Basketball | ||||
| /sports/boxing | Boxing | ||||
| /sports/football/college-football | NCAA College Football | ||||
| /sports/football/nfl | National Football League (NFL) | ||||
| /sports/golf | Golf | ||||
| /sports/hockey/nhl | National Hockey League (NHL) | ||||
| /sports/horse-racing | Horse Racing | ||||
| /sports/mma | Mixed Martial Arts | ||||
| /sports/racing | Auto Racing | ||||
| /sports/racing/nascar | NASCAR Racing | ||||
| /sports/soccer | Professional soccer (US focus) | ||||
| /sports/tennis | Tennis |
I wonder when this can be enabled for Cricket as well (including APIs free,academic,premium,partner ).
(Note you can use R packages XML , RCurl , rjson, to get data from the web among others).
Plotting is best done using ggplot2 http://had.co.nz/ggplot2/ or d3.js at http://mbostock.github.com/d3/, and the current status of cricket graphics can surely look a change- they are mostly a single radial plot of shots played /runs scored or a combined barplot/line graph.
Doing RFM Analysis in R
RFM is a method used for analyzing customer behavior and defining market segments. It is commonly used in database marketing and direct marketing and has received particular attention in retail.
RFM stands for
- Recency – How recently did the customer purchase?
- Frequency – How often do they purchase?
- Monetary Value – How much do they spend?
To create an RFM analysis, one creates categories for each attribute. For instance, the Recency attribute might be broken into three categories: customers with purchases within the last 90 days; between 91 and 365 days; and longer than 365 days. Such categories may be arrived at by applying business rules, or using a data mining technique, such as CHAID, to find meaningful breaks.
from-http://en.wikipedia.org/wiki/RFM
If you are new to RFM or need more step by step help, please read here
https://decisionstats.com/2010/10/03/ibm-spss-19-marketing-analytics-and-rfm/
and here is R code- note for direct marketing you need to compute Monetization based on response rates (based on offer date) as well
##Creating Random Sales Data of the format CustomerId (unique to each customer), Sales.Date,Purchase.Value sales=data.frame(sample(1000:1999,replace=T,size=10000),abs(round(rnorm(10000,28,13)))) names(sales)=c("CustomerId","Sales Value") sales.dates <- as.Date("2010/1/1") + 700*sort(stats::runif(10000)) #generating random dates sales=cbind(sales,sales.dates) str(sales) sales$recency=round(as.numeric(difftime(Sys.Date(),sales[,3],units="days")) ) library(gregmisc) ##if you have existing sales data you need to just shape it in this format rename.vars(sales, from="Sales Value", to="Purchase.Value")#Renaming Variable Names ## Creating Total Sales(Monetization),Frequency, Last Purchase date for each customer salesM=aggregate(sales[,2],list(sales$CustomerId),sum) names(salesM)=c("CustomerId","Monetization") salesF=aggregate(sales[,2],list(sales$CustomerId),length) names(salesF)=c("CustomerId","Frequency") salesR=aggregate(sales[,4],list(sales$CustomerId),min) names(salesR)=c("CustomerId","Recency") ##Merging R,F,M test1=merge(salesF,salesR,"CustomerId") salesRFM=merge(salesM,test1,"CustomerId") ##Creating R,F,M levels salesRFM$rankR=cut(salesRFM$Recency, 5,labels=F) #rankR 1 is very recent while rankR 5 is least recent salesRFM$rankF=cut(salesRFM$Frequency, 5,labels=F)#rankF 1 is least frequent while rankF 5 is most frequent salesRFM$rankM=cut(salesRFM$Monetization, 5,labels=F)#rankM 1 is lowest sales while rankM 5 is highest sales ##Looking at RFM tables
table(salesRFM[,5:6]) table(salesRFM[,6:7]) table(salesRFM[,5:7])
Code Highlighted by Pretty R at inside-R.org
Note-you can also use quantile function instead of cut function. This changes cut to equal length instead of equal interval. or see other methods for finding breaks for categories.
DECISION STATS 
