SPSS and R

I rarely use SPSS now, but in college ( www.iiml.ac.in) my marketing professors kind of ensured I was buried in it for weeks. Much later I did to some ARIMA forecasting in SPSS for macro economic indicators prediction ( details coming up)–

 

However the SPSS help list is a great one ( SPSSX-L@LISTSERV.UGA.EDU) , not just for staying in touch with SPSS but also with the latest statistical modeling techniques. Here is an extract from the list ( www.listserv.uga.edu/archives/spssx-l.html ) on using SPSS and R together

 

Assuming version 16 or later, you need to install the R plug-in from Developer Central.  Then your R syntax can be run in the syntax window between

BEGIN PROGRAM R.

and

END PROGRAM R.

The output automatically appears in the SPSS Viewer with two cautions.  1) In version 16, R graphics are written to files and don’t appear in the Viewer.  Version 17 integrates the graphics directly.  2) When using R interactively, expression output appears in your console windows, e.g.,

summary(dta)

displays the summary statistics for a data frame, dta.  In non-interactive mode, which is what you are in when running BEGIN PROGRAM, you need to enclose the expression in a print function for it to display, e.g.,

print(summary(dta))

The documentation for the apis to communicate between SPSS and R is installed along with the plug-in, and there are examples in the Data Management book linked on Developer Central (www.spss.com/devcentral).

You might also go through the PowerPoint article on Developer Central, "Programmability in SPSS Statistics 17", which you will find on the front page of the site.  It includes a detailed example of using the R Quantreg package in SPSS as an extension command.  There is also a download in the R section on creating an SPSS dialog box that generates an R program directly.  Look for Rboxplot – Creating an R Program from a Dialog.  This has a simple dialog box that generates code for an R boxplot along with an article that explains what is happening.

 

Ajay ‘s 2 cents– SPSS treats R as an opportunity rather than a threat, partly because SPSS is a much lower priced software , and has been working to displace SAS in vain for some time now.

SAS ( the company and not the language) as the market leader has the most to lose due to

  • its high market share ( which it has maintained by aggressively seeking both legal action as well as by pumping in or investing or generously giving — huge amounts of money in hosting conferences,papers and research and keeping alumni and current employees happy and loyal),

and

  • premium pricing ( which comes under greater pricing pressure amid a general economic downturn amongst its preferred customers -especially banks and companies like Amazon , GE Money etc)

and

  • multi pronged competition with tacit support from bigger players waiting on sidelines
  • ( like IBM has an alliance with WPS which is almost a de facto Base SAS clone as it can take in SAS datasets, SAS code, and output SAS code, SAS datasets besides having it’s own Eclipse based design for the Workbench
  • Microsoft expanding data mining capabilities in SQL Server and initiatives like Microsoft Azure ( OS for Cloud Computers ) and Microsoft Mesh .
  • open source players like R, KNIME, Rapid Miner getting commercial momentum due to better value for cost ( 0 ).

and

  • data and code portability between SAS,SPSS,R due to PMML standards means switching barriers are getting lowered. There are almost no switching barriers between Base SAS and WPS in my testing experience.

The coming market share battles between SAS, and WPS and R will be interesting to watch for the analyst/customers — that is if the current economic crisis doesn’t claim any of the companies or the clients first. Alliances as well community networking among users and developers could be critical.

Still innovation flows from creative destruction of old ideas, mindsets, attitudes and yes even software.

Vote for the SAS-L Rookie of the Year

If you are on the SAS-L list, you can vote for the following

 

SAS-L Rookie of the Year (SASLROY)


Scott Bucher
Joe Matise
Akshaya Nathilvar
Ajay Ohri                        (This is me…..by the way)
Karma Tarap

You can vote (one vote per person please) at:
http://ires.ku.edu/~ipsr/SGF2009/saslbof.htm
Voting will end February 12th.

And, as usual, the winners will be announced at the annual SAS-L BOF, at SAS Global Forum:
When: Monday, March 23
Where: TBA
Time: 7-8 pm

Ps-I wonder if the R –Help list has something  like this.

Revolution Computing Releases Commercial R –The Analytics Market just grew better

I just downloaded R Comp’s latest release of REvolution R. The individual Win 32 version is free, while Enterprise version with Win 64 versions. Tech support is included in services contract for the software which should help with any corporate willing to take R on a trial basis.

 

From the press release ,

REvolution Computing Makes High Performance ‘REvolution R’

Available For Download

New Haven, CT – January 28, 2009 – REvolution Computing, a leading provider of open source predictive analytics solutions, today announced that it has made a public version of its commercial grade REvolution R program available for download from its website. REvolution R is REvolution Computing’s distribution of the popular R statistical software, optimized for use in commercial environments.

With the latest release of REvolution R, REvolution Computing has added significant performance enhancements to the base system, which can prove to be of great value in both commercial and research settings. A key feature includes the use of powerful optimized libraries capable of boosting performance by a factor of 5 or 10 for commonly used operations. In addition, REvolution R has been put through a quality process designed to meet regulatory agency audit standards, making the subscription version reliable for use in mission critical research and production.

“In making our latest release of REvolution R available for download, REvolution Computing is providing all R users the ability to take advantage of optimized and validated software previously available only to commercial users,” said REvolution Computing CEO, Richard Schultz. “In a true commercial open source way, we have reached the point in our development that we are able to offer significant value to both sets of our community users – REvolution R for all users, and REvolution R Enterprise, with additional commercial-grade capabilities and support, available by annual subscription.”

REvolution’s commercial distribution, REvolution R Enterprise, features advanced functionality, including ParallelR, which speeds deployment across both multiprocessor workstations and clusters to enable the same codes to be used for prototyping and production. REvolution R Enterprise is functional with 64-bit platforms and Linux enterprise platforms and provides for telephone support and response guarantees.

Some background on the company itself ………..from the company itself-

 

About REvolution Computing

New Haven, Connecticut-based REvolution Computing is the leading commercial provider of software and support for the statistical computing language known as “R.” 

Our products, including REvolution R and REvolution R Enterprise, enable statisticians, scientists and others to create superior predictive models and derive meaning from large sets of mission-critical data in record time. REvolution Computing

 

works closely with the R community to incorporate the latest developments in open source R, and with our clients to support their efforts to produce groundbreaking innovations in life sciences, financial services, defense technology and other industries where high-level analytics are crucial to success. At REvolution Computing, “We do the math.”

The product names “RPro,” “ParallelR,” “REvolution R,” and “REvolution R Enterprise,” are trademarks of REvolution Computing.

 

This basically gives the company first mover

advantage in commercial R. The timing is also fortunate as companies across the world look to cut costs (unfortunately labor costs are being cut faster than software costs) as well as move beyond traditional analytics softwares that performed ah so well in the sub prime prediction market.

REvolution R is available for download on Windows and Intel MacOS X, both in 32-bit mode at http://www.revolution-computing.com/downloads/revolution-r.php

I just downloaded R Comps latest release of REvolution R. The individual Win 32 version is free, while Enterprise version with Win 64 versions. Tech support is included in services contract for the software which should help with any corporate willing to take R on a trial basis.

 

From the press release ,

REvolution Computing Makes High Performance REvolution R

Available For Download

New Haven, CT January 28, 2009 REvolution Computing, a leading provider of open source predictive analytics solutions, today announced that it has made a public version of its commercial grade REvolution R program available for download from its website. REvolution R is REvolution Computings distribution of the popular R statistical software, optimized for use in commercial environments.

With the latest release of REvolution R, REvolution Computing has added significant performance enhancements to the base system, which can prove to be of great value in both commercial and research settings. A key feature includes the use of powerful optimized libraries capable of boosting performance by a factor of 5 or 10 for commonly used operations. In addition, REvolution R has been put through a quality process designed to meet regulatory agency audit standards, making the subscription version reliable for use in mission critical research and production.

In making our latest release of REvolution R available for download, REvolution Computing is providing all R users the ability to take advantage of optimized and validated software previously available only to commercial users, said REvolution Computing CEO, Richard Schultz. In a true commercial open source way, we have reached the point in our development that we are able to offer significant value to both sets of our community users REvolution R for all users, and REvolution R Enterprise, with additional commercial-grade capabilities and support, available by annual subscription.

REvolutions commercial distribution, REvolution R Enterprise, features advanced functionality, including ParallelR, which speeds deployment across both multiprocessor workstations and clusters to enable the same codes to be used for prototyping and production. REvolution R Enterprise is functional with 64-bit platforms and Linux enterprise platforms and provides for telephone support and response guarantees.

Some background on the company itself ..from the company itself-

 

About REvolution Computing

New Haven, Connecticut-based REvolution Computing is the leading commercial provider of software and support for the statistical computing language known as R. 

Our products, including REvolution R and REvolution R Enterprise, enable statisticians, scientists and others to create superior predictive models and derive meaning from large sets of mission-critical data in record time. REvolution Computing

 

works closely with the R community to incorporate the latest developments in open source R, and with our clients to support their efforts to produce groundbreaking innovations in life sciences, financial services, defense technology and other industries where high-level analytics are crucial to success. At REvolution Computing, We do the math.

The product names RPro, ParallelR, REvolution R, and REvolution R Enterprise, are trademarks of REvolution Computing.

 

This basically gives the company first mover

advantage in commercial R. The timing is also fortunate as companies across the world look to cut costs (unfortunately labor costs are being cut faster than software costs) as well as move beyond traditional analytics softwares that performed ah so well in the sub prime prediction market.

REvolution R is available for download on Windows and Intel MacOS X, both in 32-bit mode at http://www.revolution-computing.com/downloads/revolution-r.php

Using Google Docs for Web Scraping

While trying to scrape some data from a Website , I chanced upon the getXML function which is pretty neat, as it basically allows you to import the XML feed of a webpage and then parse the data appropriately.

 

Here is an example-

 

Using the getXML function I parsed all links for “analytics consultant in India” search results in Google.

The GetXML function works as follows (from the support page here )

Functions:

=importXML("URL","query")

  • URL – the URL of the XML or HTML file
  • query – the XPath query to run on the data given at the URL. For example, "//a/@href" returns a list of the href attributes of all <a> tags in the document (i.e. all of the URLs the document links to). For more information about XPath, please visithttp://www.w3schools.com/xpath/
  • Example: =importXml("www.google.com", "//a/@href"). This returns all of the href attributes (the link URLs) in all the <a> tags on www.google.com home page

 

You can see it here-

http://spreadsheets.google.com/pub?key=pS9vSxWuwOllXHdueY0TDdg

or Using the Embed Function

 

R in a CorpoRate Environment

Any concerns of using R in a corporate environment especially for compliance reasons can be mitigated from reading the following documents.

R: Regulatory Compliance and Validation Issues A Guidance Document for the Use of R in Regulated Clinical Trial Environments

and

Keeling & Parvur’s "A comparative study of the reliability to nine statistical software
packages, May 1, 2007 Computational Statistics & Data Analysis, Vol.51,pp 3811-3831.

 

Thanks to Bob for pointing this out on the R-Help list.

Technorati Tags:

Updated-R for SAS and SPSS Users

Updated  –I  finally got my hardback copy of the R for SAS and SPSS users . Digital copies are one thing, but a paper book is really beautiful .I had written an article on R ( with some mild sarcasm on some other softwares that are mildly more expensive) at Smart Data Collective. That created around 711 views of that article, ( my website got X00 hits that day, which is a personal best ,ehmm 🙂

It also inspired Sandro, a terrific data miner from Switzerland and a PhD to write an article called 5 reasons R is good for you, which can be accessed here http://smartdatacollective.com/Home/15756 and http://dataminingresearch.blogspot.com/2009/01/top-5-reasons-r-is-good-for-you.html

The story of how I wrote that Top Ten R article is also amusing – mentioned here by Jerry who creates terrific communities for content , all extremely digital and informative , readable here –http://www.socialmediatoday.com/SMC/67268

Now the reason I originally became involved with R, was because I couldn’t afford SAS and SPSS on my own computer after years of getting companies to pick up the tab. A question on the R help list led me to Bob Muenchen , who had written a short guidebook on R for SAS and SPSS users, and was then finishing his book. The following article is interesting given that it was done almost 3-4 months back yet some themes and events seemed to recur exactly as Bob mentioned them. I still bounce between Bob’s book and the Rattle guide for R programming but I am getting there !!!

Note-Robert Muenchen (pronounced Min’-chen) is the author of the famous R for SAS and SPSS users, and his book is an extensive tutorial on anyone wanting to learn either SAS,SPSS,or R or even to migrate from one platform to another. In an exclusive interview Bob agreed to answer some questions on the book , and on students planning to enter science careers.

What made you write the R For SAS and SPSS users?

The book-

A few years ago, all my colleagues seemed to be suddenly talking about R. Had I tried it? What did I think? Wasn’t it amazing? I searched around for a review and found an article by Patrick Burns, "R Relative to Statistics Packages" which is posted on the UCLA site (http://www.ats.ucla.edu/stat/technicalreports/). That article pointed out the many advantages of R and in it Burns claimed that knowing a standard statistics package interfered with learning R. That article really got my interest up. Pat’s article was a rejoinder to "Strategically using General Purpose Statistics Packages: A Look at Stata, SAS and SPSS" by Michael Mitchell, then the manager of statistical consulting at UCLA (it’s at that same site). In it he said little about R, other than he had "enormous difficulties" learning it that he had especially found the documentation lacking.

I dove in and started learning R. It was incredibly hard work, most of which was caused by my expectations of how I thought it ought to work. I did have a lot to "unlearn" but once I figured a certain step out, I could see that explaining it to another SAS or SPSS user would be relatively easy. I started keeping notes on these differences for myself initially. I finally posted them on the Internet as the first version of R for SAS and SPSS Users. It was only 80 pages and much of its explanation was in the form of extensive R program comments. I provided 27 example programs, each done in SAS, SPSS and R. A person could see how they differed, topic by topic. When a person ran the sections of the R programs and read all the comments, he or she would learn how R worked.

A web page counter on that document showed it was getting about 10,000 hits a month. That translates into about 300 users, paging back and forth through the document. An editor from Springer emailed me to ask if I could make it a book. I said it might be 150 pages when I wrote out the prose to replace all the comments. It turned out to be 480 pages!

What are the salient points in this book ?

The main point is that having R taught to you using terms you already know will make R much easier to learn. SAS and SPSS concepts are used in the body of the book as well as the table of contents, the index and even the glossary. For example, the table of contents has an entry for "Value Labels or Formats" even though R uses neither of those terms as SPSS and SAS do, respectively. The index alone took over 80 hours to compile because it is important for people to be able to look up things like "length" as both a SAS statement and as an R function. The glossary defines R terms using SAS/SPSS jargon and then again using proper R definitions.

SAS and SPSS each have five main parts: 1) commands to read and manage data, 2) procedures for statistics & graphics, 3) output management systems that allow you to use output as input to other analyses, 4) a macro language to automate the above steps and finally 5) a matrix language to help you extend the packages. All five of these parts use different statements and rules that do not apply to the others. Due to the complexity of all this, many SAS and SPSS users never get past the first two parts.

R instead has all these functions unified into a common single structure. That makes it much more flexible and powerful. This claim may seem to be a matter of opinion, but the evidence to back it up comes from the companies themselves. The developers at SAS Institute and SPSS Inc. don’t write their procedures in their own languages, R developers do.

How do you think R will impact the statistical software vendors?

With more statistical procedures than any other package, and its free price, some people think R will put many of the proprietary vendors out of business. R is a tsunami coming at the vendors and how they respond will determine their future. Take SPSS Inc. for example. They have written an excellent interface to R that lets you transfer your data back and forth, letting you run R functions in the middle of your SPSS programs. I show how to use it in my book. Starting with SPSS 17, you can also add R functions to the SPSS menus. This is particularly important because most SPSS users prefer to use menus. The company itself is adding menus to R functions, letting them rapidly expand SPSS’ capabilities at very little expense. They saw the R tsunami coming and they hopped on a surfboard to make the most of it. I think this attitude will help them thrive in the future.

SAS Institute so far as been ignoring R. That means if you need to use an analytic method that is only available in R, you must learn much more R than an SPSS user would. Once you have done that, you might be much more likely to switch over completely to R. Colleagues inside SAS Institute tell me they are debating whether they should follow SPSS’ lead and write a link to R. T
his has already been done by MineQuest, LLC (see http://www.minequest.com/Products.html ) with their amusingly named, "A bridge to R" product (playing off "A Bridge Too Far.")

Statistica is officially supporting R. You can read about the details at (http://www.statsoft.com/industries/Rlanguage.htm) . Statacorp has not supported R in Stata yet, although a user, Roger Newson, has written an R interface to it (http://ideas.repec.org/c/boc/bocode/s456847.html).

The company with the most to lose are the makers of S-PLUS. That was Insightful Corp. until they were recently bought out by Tibco. Since R is an implementation of the S language, S-PLUS could be hit pretty hard. On the other hand, they do have functions that handle "big data" so there is a chance that people will develop programs in R, run out of memory and then end up porting them to S-PLUS. S-PLUS also has a more comprehensive graphical user interface than R does, giving them an advantage. However, XL-Solutions Corp. has their new R-PLUS version that adds a slick GUI to R (http://www.experience-rplus.com/). There could be a rocky road ahead for S-PLUS. IBM faced a similar dilemma when computing hardware started becoming commodities. They prospered by making up the difference with service income. Perhaps Tibco can too.

Do you have special discounts for students?

My original version of R for SAS and SPSS Users is still online at http://RforSASandSPSSusers.com so students can get it there for free. The book version has a small market that is mostly students so pricing was set with that in mind.

What made you choose a career in Science and what have been the reasons for your success in it.

I started out as an accounting major. I was lucky enough to have had two years of bookkeeping in high school, and I worked part-time in the accounting department of ServiceMaster Industries for several years. I got to fill in for whoever was on vacation, so I got a broad range of accounting experience. I also got my first experience with statistics by helping the auditors. We took a stratified sample of transactions. With transactions divided into segments by their value, and sample a greater proportion as the value increased. For the most expensive transactions, we examined them all. My job was to be the "gofer" who collected all the invoices, checks, etc. to prove that the transactions were real. For a kid in high school, that was great fun!

By the time I was a freshman at Bradley University, I became excited by three new areas: mathematics, computing and psychology. I got to work in a lab at the Peoria Addictions Research Institute, studying addiction in rats and the parts of the brain that were involved. I wrote a simple stat package in FORTRAN to analyze data. After getting my B.A. in psychology, I worked on a PhD in Educational Psychology at Arizona State University. I loved that field and did well, but the job market for professors in that field was horrible at the time. So I transferred to a PhD program in Industrial/Organizational Psychology at The University of Tennessee. It turned out that I did not really care for that area at all, and I spent much of my time studying computing and calculus. My assistantship was with the Department of Statistics. By the time my first year was up, I transferred to statistics. At the time the department lacked a PhD program, so after four years of grad school I stopped with an M.S. in Statistics and got a job as a computing consultant helping people with their SAS, SPSS and STATGRAPHICS programs. Later I was able to expand that role, creating a full-fledged statistical consulting center in partnership with the Department of Statistics. Ongoing funding cuts have been chipping away at that concept though.

What made me a success? I love my job! I get to work with a lot of smart scientists and their grad students, expanding scientific knowledge. What could be better?

Science is boring, and not well paying career compared to being a lawyer or a sales job. People think you are a nerd. Please comment based on your experiences.

Science is constantly making new discoveries. That’s not boring! An area that most people can relate to is medicine. When we finish a study that shows a new treatment is better than an old one, our efforts will help thousands of people. In one study we compared a new, very expensive anti-nausea drug to an old one that was quite cheap. The pharmaceutical company claimed the new drug was better of course, but our study showed that it was not. That ended up helping to control health care costs that we all see escalating rapidly.

Another study found for the first time, a measure that could predict how well a hearing aid would help a person. Now, it’s easy to measure a hearing aid and see that it is doing what it is supposed to do, but a huge proportion of people who buy them don’t like them and stop wearing them after a brief period. Scientists tried for decades to predict which people would not be good candidates for hearing aids. A very sharp scientist at UT, Anna Nabelek, came up with the concept of Acceptable Noise Level. We measured how much background noise people were willing to tolerate before trying a hearing aid. That allowed us to develop a model that could predict well for the first time if someone should bother spending up to $5,000 for hearing aids. For retired people on a fixed income, that was an important finding. An audiology journal devoted an entire issue to the work.

It’s true that you can make more money in many other fields. But the excitement of discovery and the feeling that I’m helping to extend science very satisfying and well worth the lower salary. Plus, having a job in science means you will never have a chance to get bored!

What is your view on Rice University’s initiatives to create open source textbooks at http://cnx.org/ .

I think this is a really good idea. One of my favorite statistics books is Statnotes: Topics in Multivariate Analysis, by G David Garson. You can read it for free at http://www2.chass.ncsu.edu/garson/pa765/statnote.htm .

Universities pay professors to spend their time doing research, which must be published to get credit. So why not pay professors to write text books too? There have been probably hundreds of introductory books in every imaginable field. They cannot all make it in the marketplace so when they drop out of publication, why not make them available for free? I still have my old Introductory Statistics textbook from 30 years ago and the material is still good. It may be missing a few modern things like boxplots, but it would not take much effort to bring it up to date.

I’m also a huge fan of Project Gutenburg (http://www.archive.org/details/gutenberg). That is a collection of over 20,000 books, articles, etc. available there for free download. My wife does volunteer project management and post-processing with Distributed Proofreaders (http://www.pgdp.net/) which supplies books for Gutenburg.

What are your vie
ws on students uploading scanned copies of books to torrent sharing web sites because of expensive books.

The cost of textbooks has gotten out of hand. I think students should pressure universities and professors to consider cheaper alternatives. However scanning books putting them up on web sites isn’t sharing, it’s stealing. I put in most of my weekends and nights for 2 ½ years on my book that will be lucky to sell a few thousand copies. That works out to pennies per hour. Seeing it scanned in would be quite depressing.

When is the book coming out ? What is taking so long ?

We ran into problems when the book was translated from Microsoft Word to LaTeX. The translator program did not anticipate that an index would already be in place. That resulted in 2-3 errors per page. We’re working through that and should finally get it printed in early October.

Biography

Robert A. Muenchen is a consulting statistician with 28 years of experience. He is currently the manager of the Statistical Consulting Center at the University of Tennessee. He holds a B.A. in Psychology and an M.S. in Statistics. Bob has conducted research for a variety of public and private organizations and has assisted on more than 1,000 graduate theses and dissertations. He has coauthored over 40 articles published in scientific journals and conference proceedings. Bob has served on the advisory boards of SPSS Inc., the Statistical Graphics Corporation and PC Week Magazine. His suggested improvements have been incorporated into SAS, SPSS, JMP, STATGRAPHICS and several R packages. His research interests include statistical computing, data graphics and visualization,text analysis, data mining, psychometrics and resampling.

Ajay-He is also a very modest and great human being.

http://www.amazon.com/SAS-SPSS-Users-Statistics-Computing/dp/0387094172/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1217456813&sr=8-1