Running R on Windows Azure #rstats #cloud

Here is a brief tutorial for people to run R on Windows Azure Cloud (OS=Windows in this case , but there are 4 kinds of Linux also available)

There is a free 90 day trial so you can run R for free on the cloud for free (since Google Cloud Compute is still in closed hush hush beta)

Go to https://www.windowsazure.com/en-us/pricing/free-trial/

WordPress.com Analytics

The Analytics (or stats) dashboard at WordPress.com continues to disappoint, and is a major reason for people to move out of WordPress.com hosting (since they need better analytics like that by Google Analytics which cant be enabled on the default mode)

Its not really beautiful unlike the rest of WordPress Universe!

It can be made better if people try harder! Analytics matters

Here are some points

1) Bar charts and Histograms are not really the best way to visualize trends across time

2) Location Analytics is limited to just country level analysis and the heatmap (?) is aweful in terms of distinguishing gradients 

3) Referrers Tab needs to do a better job on distinguishing between mobile and non mobile traffic, social and non social traffic (and there are better ways to visualize than just a simple list)!

4)  I cant even export my traffic stats (and forget an api !) so I am stuck with the bad data viz here

US Congress cedes cyber-war to Executive Branch

From–

http://www.nytimes.com/2012/06/01/world/middleeast/obama-ordered-wave-of-cyberattacks-against-iran.html?_r=2

Obama Order Sped Up Wave of Cyberattacks Against Iran

By
Published: June 1, 2012

WASHINGTON — From his first months in office, President Obama secretly ordered increasingly sophisticated attacks on the computer systems that run Iran’s main nuclear enrichment facilities, significantly expanding America’s first sustained use of cyberweapons,

From–

http://www.politico.com/news/stories/0612/76973.html

Can the White House declare a cyberwar?

By JENNIFER MARTINEZ and JONATHAN ALLEN | 6/1/12
“When we see the results it’s pretty clear they’re doing it without anybody except a very few people knowing about it, much less having any impact on whether it’s happening or not,” said Rep. Jim McDermott (D-Wash.).

McDermott is troubled because “we have given more and more power to the president, through the CIA, to carry out operations, and, frankly, if you go back in history, the reason we have problems with Iran is because the CIA brought about a coup.”

 

From–

http://www.house.gov/house/Constitution/Constitution.html

Article. I.

Section 1.

All legislative Powers herein granted shall be vested in a Congress of the United States, which shall consist of a Senate and House of Representatives.

Section. 8.

The Congress shall have Power

Clause 11: To declare War, grant Letters of Marque and Reprisal, and make Rules concerning Captures on Land and Water;

 

Related-

http://www.huffingtonpost.com/2009/10/09/obama-wins-nobel-peace-pr_n_314907.html

Obama Wins Nobel Peace Prize

KARL RITTER and MATT MOORE   10/ 9/09 11:02 PM ET

http://www.law.uchicago.edu/media

Statement Regarding Barack Obama 

The Law School has received many media requests about Barack Obama, especially about his status as “Senior Lecturer.”

From 1992 until his election to the U.S. Senate in 2004, Barack Obama served as a professor in the Law School. He was a Lecturer from 1992 to 1996. He was a Senior Lecturer from 1996 to 2004, during which time he taught three courses per year.

 

Data Frame in Python

Exploring some Python Packages and R packages to move /work with both Python and R without melting your brain or exceeding your project deadline

—————————————

If you liked the data.frame structure in R, you have some way to work with them at a faster processing speed in Python.

Here are three packages that enable you to do so-

(1) pydataframe http://code.google.com/p/pydataframe/

An implemention of an almost R like DataFrame object. (install via Pypi/Pip: “pip install pydataframe”)

Usage:

        u = DataFrame( { "Field1": [1, 2, 3],
                        "Field2": ['abc', 'def', 'hgi']},
                        optional:
                         ['Field1', 'Field2']
                         ["rowOne", "rowTwo", "thirdRow"])

A DataFrame is basically a table with rows and columns.

Columns are named, rows are numbered (but can be named) and can be easily selected and calculated upon. Internally, columns are stored as 1d numpy arrays. If you set row names, they’re converted into a dictionary for fast access. There is a rich subselection/slicing API, see help(DataFrame.get_item) (it also works for setting values). Please note that any slice get’s you another DataFrame, to access individual entries use get_row(), get_column(), get_value().

DataFrames also understand basic arithmetic and you can either add (multiply,…) a constant value, or another DataFrame of the same size / with the same column names, like this:

#multiply every value in ColumnA that is smaller than 5 by 6.
my_df[my_df[:,'ColumnA'] < 5, 'ColumnA'] *= 6

#you always need to specify both row and column selectors, use : to mean everything
my_df[:, 'ColumnB'] = my_df[:,'ColumnA'] + my_df[:, 'ColumnC']

#let's take every row that starts with Shu in ColumnA and replace it with a new list (comprehension)
select = my_df.where(lambda row: row['ColumnA'].startswith('Shu'))
my_df[select, 'ColumnA'] = [row['ColumnA'].replace('Shu', 'Sha') for row in my_df[select,:].iter_rows()]

Dataframes talk directly to R via rpy2 (rpy2 is not a prerequiste for the library!)

 

(2) pandas http://pandas.pydata.org/

Library Highlights

  • A fast and efficient DataFrame object for data manipulation with integrated indexing;
  • Tools for reading and writing data between in-memory data structures and different formats: CSV and text files, Microsoft Excel, SQL databases, and the fast HDF5 format;
  • Intelligent data alignment and integrated handling of missing data: gain automatic label-based alignment in computations and easily manipulate messy data into an orderly form;
  • Flexible reshaping and pivoting of data sets;
  • Intelligent label-based slicing, fancy indexing, and subsetting of large data sets;
  • Columns can be inserted and deleted from data structures for size mutability;
  • Aggregating or transforming data with a powerful group by engine allowing split-apply-combine operations on data sets;
  • High performance merging and joining of data sets;
  • Hierarchical axis indexing provides an intuitive way of working with high-dimensional data in a lower-dimensional data structure;
  • Time series-functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging. Even create domain-specific time offsets and join time series without losing data;
  • The library has been ruthlessly optimized for performance, with critical code paths compiled to C;
  • Python with pandas is in use in a wide variety of academic and commercial domains, including Finance, Neuroscience, Economics, Statistics, Advertising, Web Analytics, and more.

Why not R?

First of all, we love open source R! It is the most widely-used open source environment for statistical modeling and graphics, and it provided some early inspiration for pandas features. R users will be pleased to find this library adopts some of the best concepts of R, like the foundational DataFrame (one user familiar with R has described pandas as “R data.frame on steroids”). But pandas also seeks to solve some frustrations common to R users:

  • R has barebones data alignment and indexing functionality, leaving much work to the user. pandas makes it easy and intuitive to work with messy, irregularly indexed data, like time series data. pandas also provides rich tools, like hierarchical indexing, not found in R;
  • R is not well-suited to general purpose programming and system development. pandas enables you to do large-scale data processing seamlessly when developing your production applications;
  • Hybrid systems connecting R to a low-productivity systems language like Java, C++, or C# suffer from significantly reduced agility and maintainability, and you’re still stuck developing the system components in a low-productivity language;
  • The “copyleft” GPL license of R can create concerns for commercial software vendors who want to distribute R with their software under another license. Python and pandas use more permissive licenses.

(3) datamatrix http://pypi.python.org/pypi/datamatrix/0.8

datamatrix 0.8

A Pythonic implementation of R’s data.frame structure.

Latest Version: 0.9

This module allows access to comma- or other delimiter separated files as if they were tables, using a dictionary-like syntax. DataMatrix objects can be manipulated, rows and columns added and removed, or even transposed

—————————————————————–

Modeling in Python

Continue reading “Data Frame in Python”

Scribd Analytics

I really liked Scribd Analytics feature (and that we have racked up 3200 reads in 11 months for Poets and Hackers). I think Google Docs /Drive should really incorporate more Scribd like document sharing features including social (now that Slideshare is off the market for a relatively cheap $119m) and turn on analytics by default

I really liked the heatmap on the document feature in the second screenshot.

Anyways nice to see someone out there cares for Poets &….

 

 

R and Hadoop #rstats

Lovely ppt from the formidable Jeffrey Bean, whose lucid style in explaining R has made me a big fan of his awesome work!

Take at look at his extensive collection of Big Data with R slides  at http://jeffreybreen.wordpress.com/2012/03/10/big-data-step-by-step-slides/ – they are both very comprehensive and a delightful addition to anyone wishing to go the cloud, hadoop, R  route
His blog at http://jeffreybreen.wordpress.com/ talks of lots of very relevant topics.

Different kinds of Clouds

Some slides I liked on cloud computing infrastructure as offered by Amazon, IBM, Google , Windows and Oracle