Author: Ajay Ohri

https://linkedin.com/in/ajayohri

Making Government Transparent Using R

Here is a terrific interview on O’Reilley Radar at http://radar.oreilly.com/2009/07/making-government-transparent.html

It actually talks of using open source statistics like R to make Government more transparent- like analyzing waste.

Some interesting extracts- like I didnt know S is being maintained by SAS.( I thought Tibco had S Plus)

Citation-http://radar.oreilly.com/2009/07/making-government-transparent.html

James Turner: So switching gears, the other thing you’re talking about and a big part of your professional life is the R language. Now I will confess that like Erlang, R is something that is on my radar and I see and I look at it and I say, “Okay. When am I ever going to use it?” I mean Erlang is used some places, but R I guess has a very nichey type of audience, doesn’t it?

Danese Cooper: You know, interestingly enough that’s changing. I think that’s been true. R has been in production or in development, let’s say, for the last 20 years. It is patterned after the S language, which was developed in the ’60s at Bell Labs around the same time that UNIX and C were being developed. And it was S for statistics, right? R is sort of a, “If we had known then what we know now” version of S. They’ve been working on it for 20 years in an academic setting. So it has been very slow to grow. But just in the last couple of years, it’s really gotten to a place where it’s ready for enterprise use. And just this year, the people that maintain S, a company called SAS, S-a-s, in South America, south of this country, have announced that they’re going to have to support R, like it’s that widely used now, particularly in schools.

Danese Cooper works for Revolution COmputing that creates a wonderful and professional version of R called Revolution R – some of the work on parallelization and enabling 64 bit Windows R is great. Danese is also a solid open source credentials person having worked with the Board and also with Apache. O Reilley Media’s work in open source conferences is terrific as well.

That apart, the great stuff is in the rest of this must read interview which is available athttp://radar.oreilly.com/2009/07/making-government-transparent.html

Experimental Ad AuDio-Video

As an experiment I will be putting Random Images /U Tube songs in the next 7 posts/ post this week. This would be viewable only by reaching my site and not the RSS ( now restored to full rather than summary).

Let me know if the server hangs ( sigh!!) or if you find them distracting.

or if you know a better song.

So what happened to S Plus

Splus – The corporate version of S ( the predecessor of R) is still being marketed by Tibco corporation- again rumoured to be an acquisition target of  (???)

  • SAS ( who have desired R like capabilties especially in their IML  product to be released soon
  • SAP who lost out to IBM in the SPSS acquisition
  • Oracle
  • Microsoft
  • Rogue Wave (acquirer of Visual Numerics)
  • etc etc.

Anyways S Plus is still alive and kicking-

“The S language and the S+ application have been critical to our ability to manage big data objects intrinsic to wind analytics and wind energy development,” said Brad Horn, Director of Wind Analytics at NextEra Energy.  “We credit our long-term interface and Spotfire consulting with unlocking new ideas and sources of value.  Joint dialogue on configuration alternatives and our recent efforts to restructure legacy code is allowing us to transition from simple interactive use of S+ to a customized S+ configuration with integrated batch processing, server load balancing, and parallel processing.  S+ has a central role in supporting internal decisions and our group emphasis on scale, speed, and quality.”

http://spotfire.tibco.com/news/press-releases/2009/2_17_2009.aspx

  • Wavelets, Spatial Stats, EnvironmentalStats: Apply statistics for advanced analysis of signal and image data, spatially correlated data, and environmental data.
  • Resampling: Apply resampling techniques, such as bootstrap and permutation tests, to enable the use of standard statistics on smaller data sets.
  • Association Rules: Uncover relationships between variables in large data sets, most commonly to detect purchase patterns (Market Basket Analysis), or in many other areas like web site usage analysis.
  • Recode Values: Easily handle and prepare data from multiple sources by changing the values in a column to a new value.

    • Deployment and Integration:

    • Spotfire Integration: Read and write Spotfire Text Data files, and leverage examples of using Spotfire Professional to visualize, explore and share model results.
    • Custom Java & C++ nodes: Extend Spotfire Miner by writing custom nodes in Java and C++.
    • Remote Script Execution: Execute S+ scripts remotely on S+ Server to offload and distribute intensive jobs.
    • Global Worksheet Parameters: Make workflows more flexible and reusable to interactive and batch applications.
    • FlexBayes: Create more realistic models, provide a natural way to address missing data, and take advantage of prior analysis.

    Data Access and Preparation:

    • New Data File Types: Unlock more data sources by reading new formats including Spotfire Text Data, Microsoft Excel 2007, Microsoft Access 2007, and Matlab 7.
    • JDBC Access: Access new data sources for analysis with data import and export via the sjdbc library in Spotfire S+ 8.1.

    Citation:

    http://spotfire.tibco.com/Products/S-Plus-Overview.aspx

    http://spotfire.tibco.com/Products/Whatsnew-Splus.aspx


    Webfocus RStat: Pervasive BI using R

    Here is a great reporting and BI tool from Information Builders  and uses the Rattle R GUI ( covered earlier here http://www.decisionstats.com/2009/01/13/interview-dr-graham-williams/).

    So if you are looking for generation next reporting solution here is one called WebFocus RStat.

    Citation:

    http://www.informationbuilders.com/products/webfocus/predictivemodeling.html

    Predict the Future and Make Effective Decisions Today

    Traditional reporting solutions provide a clear picture of past occurrences, but have little power to shed light on the future. The ability to anticipate and prepare for upcoming events can greatly impact the decisions that need to be made today.

    WebFOCUS RStat is the market’s first fully-integrated business intelligence and data mining environment, seamlessly bridging the gap between backward and forward-facing views of business operations. With WebFOCUS RStat, companies can easily and cost-effectively deploy predictive models as intuitive scoring applications. So business users at all levels can make decisions based on accurate, validated future predictions, instead of relying on gut instinct alone.

    WebFOCUS RStat provides a single platform for BI, data modeling, and scoring. This eliminates the need to purchase and maintain multiple tools, and frees analysts and other statisticians from spending countless hours extracting and querying data. At the same time, it reduces costs, simplifies maintenance, and optimizes IT resources.

    But, the greatest benefit WebFOCUS RStat offers is significantly increased accuracy. With the R engine – a powerful and flexible open source statistical programming language – as its underlying analysis tool, WebFOCUS RStat can deliver results that are consistent, complete, and correct – every time.

    WebFOCUS RStat provides:

    • A single tool, fully integrated with Developer Studio and WebFOCUS Reporting Servers with access to over 300 data sources, for both BI developers and data miners
    • Comprehensive data exploration, descriptive statistics, and interactive graphs
    • In-depth data visualization and transformation
    • Hypothesis testing, clustering, and correlation analysis

    Other key WebFOCUS RStat features include:

    • The ability to build and export models for prediction and classification
    • Comprehensive model evaluation

    Incidently the parent company which is based in Tennessee has some interesting numbers-

    http://www.informationbuilders.com/about_us/index.html

    Company At A Glance
    • $300 million in revenue
    • Over 30 years of experience
    • More than 1,400 employees
    • Over 12,000 customers
    • Over 350 business partners
    • 47 offices and 26 worldwide distributors
    • Rapid application creation through easy incorporation of scoring routines into WebFOCUS reports

    See Also-

    http://www.informationbuilders.com/cgi-shell/press/intpr/f_intpr.pl?intpr_code=06_03_08_rstat

    http://rattle.togaware.com/

    Poem: The Fine Print

    did you read the fine print
    when you signed your life away
    or did you believe them badly
    when they said your life was good to give today

    did all the drums, the ribbons and the music
    tilt your head to emotion away from fact
    and did the inherent absurdity of it all
    was swallowed by you intact

    for as the world spins tilted
    around the bright unforgiving sun
    words in a language built to deceive
    mask the coming pain below the frosting of fun

    deception is the game here
    and an unwilling player you have to be
    fool them or be fooled in turn
    reality is spotless for you to see

    what old promises where tokens of love
    it is all cash and carry now
    as willed in your destiny from above

    and even though eyes grow misty
    by potential of what could be
    you keep one eye on the rolling ball
    lest more surprises it brings to see

    Social Network Analysis: Using R

    Here is a great video and slides on doing statistical network analysis using R. It is by Drew Conway from NYU.

    Social Network Analysis in R from Drew Conway on Vimeo.