Month: August 2015

Using ggplot in Python #python #dataviz

Based on the open source project at http://ggplot.yhathq.com/ here is small training ppt created by one of our wonderful summer interns Sarah

Hat Tip  to http://www.amazon.com/Grammar-Graphics-Statistics-Computing/dp/0387245448

Leland Wilkinson the inventor of Grammar of Graphics now works for Tableau Software

 

 

Get you app on Droid

1 Download and Install   IntelliJ IDEA 

https://www.jetbrains.com/idea/help/basics-and-installation.html#d1847332e131

2 To update which version of Java you want

$ sudo update-alternatives --config java

3 Download and Install Android Studio

http://developer.android.com/sdk/installing/index.html?pkg=studio

4 Learn about basic app building from MIT App Builder ( its a GUI so relax)

http://ai2.appinventor.mit.edu/ Screenshot from 2015-08-19 11:44:13

5 Give up building yourself and post for a developer for your Android app at http://www.appfutura.com/

 

Sources-

http://askubuntu.com/questions/64329/how-to-replace-openjdk-6-with-openjdk-7

 

Installing and Using Spark easily with Python or R on Ubuntu #python #rstats

  1. Download spark from https://spark.apache.org/downloads.html (say to home/ajay/Desktop/BDA )
  2. Change to the directory from terminal cd /home/ajay/Desktop/BDA
  3. Unzip the file 
    tar -xvf spark-1.4.1-bin-hadoop2.6.tgz
  4. Change to the directory created ( say you unizpped spark file above and renamed it spark) ajay@Inspiron-3542:~/Desktop/BDA$ cd spark
  5. Run the command ./bin/pyspark Screenshot from 2015-08-18 11:58:34
  6. To look at local jobs see http://192.168.1.18:4040/jobs/ (or based on what you get from your terminal after running command in step 4)

Screenshot from 2015-08-18 11:57:56

 

 

 

 

 

 

 

to do this with R just use .bin/sparkR

Screenshot from 2015-08-18 12:02:19

 

Sources- http://stackoverflow.com/questions/30483409/installing-apache-spark-on-ubuntu-14-04

 

Traps to avoid if you are a grey hat hacker

  1. Bait and Switch – used to plant embedded malware or logging systems https://en.wikipedia.org/wiki/Bait-and-switch the action (generally illegal) of advertising goods which are an apparent bargain, with the intention of substituting inferior or more expensive goods. It can be avoided by periodically changing your hardware and software with a reliance for open source and open market and of course by avoiding things that are too good to be true.
  2. The Honey Trap –https://en.wikipedia.org/wiki/Honey_trapping
    a stratagem in which an attractive person entices another person into revealing information or doing something unwise. This one got Julian Assange
  3. The Honey Pot  –https://en.wikipedia.org/wiki/Honeypot_(computing) a honeypot is a trap set to detect, deflect, or, in some manner, counteract attempts at unauthorized use of information systems. Generally, a honeypot consists of a computer, data, or a network site that appears to be part of a network, but is actually isolated and monitored, and which seems to contain information or a resource of value to attackers. This is similar to the police baiting a criminal and then conducting undercover surveillance. This one got Sabu.
  4. The Tax Trap- This one got Al Capone. Since there is no evidence against your cyber activities they put you in tax court based on the mismatch of your income and expenditure. It can be avoided by creating appropriate legal mechanisms including corporations.
  5. The Informer Trap– God can protect you against your enemies but not your friends. This can be avoided by delineating the personal private and professional life of your activities in different compartments, hardware and virtual machines including your own personality and brain. Reveal you true identity to boast and you will end up a Reservoir Dog

Screenshot from 2015-08-18 10:55:41

Is R going to be better than Python for Big Data Analytics and Data Science? #rstats #python

My last articles seems to have touched a nerve or two judging by the 2000 views I got in a single day on a Sunday ( and India’s national Independence Day / and  V-J Day). Here I am simply reproducing the unedited and very interesting comments I got with an interesting R package.

 

On Google Plus, there is a vibrant community for R and Statistics. Yes Google plus exists still 😉 The following excellent comment makes you think.

David Reinke

This is pretty much a ho-hum topic with me. I don’t find this article very convincing. If you like Python, fine! Use Python. The problem I have with Python is that it is an interpreted language. Anything written in pure Python is going to take a long time to run on a big data set. Sure, there are Python packages for data analysis that run quickly, but you either have to depend on what someone else provides or develop your own package in compiled code.

I’ve found most software apps written specifically for “big data” to be very limited: a lot of them begin and end at N/N (pretty old hat now and inferior to a number of other methods for many analyses). If you can’t look under the hood and see what goes on in an analysis package, well, then good luck to you if you to use it, but don’t expect me to.

So far I’ve found that R works well for the large data sets I work with. (I’ll leave aside the issue of graphics for now; I have yet to see anything else that can hold a candle to R in that regard.) If the base packages that come with R can’t do a particular task I’ll first search among the over 5,000 packages currently available on CRAN. If that doesn’t work I’ll send a request to the R help list server. If that doesn’t work I’ll write my own routine in C or C# (I prefer the latter). BTW, if you are in the data analysis game you need to know enough to be able to do your own numerical analysis programming, say at the level of Numerical Recipes. Otherwise you are going to be overly dependent on someone else to provide software for you.

I’m not writing this to persuade anyone to pick one over the other. It’s just that there are a lot of possible choices out there — it’s not just R vs Python. And I’m just tired of these endless debates that go nowhere. As we say in the software engineering world: don’t try to convince the other person that your text editor/IDE/programming language is better than theirs.

—-
and
Anthony the creator or RHadoop was kind enough to not only write a comment here but also provide a tech solution AND throw a challenge at all pythonistas.

The lack of activity on rmr2 reflects maturity of the package and a shift away from Hadoop mapreduce toward spark. Please check the dplyr.spark package on github. It’s the easiest way to run spark bar none, including python, in its author very biased opinion. Example: find the best and worst flight by arrival delay on each day:

group_by(flights, year, month, day) %>%
select(flight, arr_delay) %>%
filter(arr_delay == min(arr_delay) || arr_delay == max(arr_delay))

Runs on spark, scales to whatever your cluster can store. Please show me the equivalent in any other language, python included. I am waiting.

and finally after all that violence and doubletalk ( as Dire Straits sang in the Walk of Life)  the R package that will beat all packages on Big Data —-(apparently)
https://github.com/RevolutionAnalytics/dplyr-spark

download spark and build it as follows

cd <spark root>
build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests -Phive -Phive-thriftserver clean package

Then start the thift service.

sbin/start-thriftserver.sh  
install.packages(c("RJDBC", "dplyr", "DBI", "devtools"))
devtools::install_github("hadley/purrr")

Indirectly RJDBC needs rJava. Make sure that you have rJava working with:

library(rJava)
.jinit()
install.packages("devtools")
library(devtools)
install_url(
  "https://github.com/RevolutionAnalytics/dplyr-spark/releases/download/0.2.2/dplyr.spark_0.2.2.tar.gz")

library(dplyr)

library(dplyr.spark)

spark.src = src_SparkSQL(“localhost“, “10000“)