Check out the Hearst Data Mining Challenge- a new competition-sponsored by DMA, Hearst Magazine, and EXL
THE HEARST CHALLENGE STARTS ON OCTOBER 14TH
CHALLENGE
DESCRIPTION
Over the years, the magazine publishing industry has made significant strides in improving subscription based circulation by developing analytic frameworks that better predict customer response to acquisition and renewal offers. The objective of this contest is to apply the same analytic discipline and effectively predict newsstand locations “response”. Specifically the objective is to predict the number of copies to be placed in each newsstand location to optimize the overall contribution of the newsstand location typically referred to as draw.
Data for the competition is provided by CMG and Experian.
and
RULES
HOW TO ENTER: Beginning October 14th, 2010 at 12:01 AM (ET) throughDecember 3rd, 2010 at 11:59 PM (ET) go to the Hearst Challenge website located at http://www.HearstChallenge.com (the “Site”) and complete and submit the entry form pursuant to the onscreen instructions. Entrants will be provided a historical sample of newsstand location draw, sales and associated location level data to help develop their predictive algorithm. Hearst will in turn hold back two distinct sets of draw/sales data, one to be used as a validation set by the contestant and one to be used as a final contest evaluation set. Entrants may not include any other external variables for the challenge. Additional details will be provided with the data. Entrants will be able to track their performance against the validation set throughout the course of the challenge via a leader tracking board to be made available on the Site. Entries must include the following documentation:
Data file with id variables and expected sales values by store and publication
The final model/ algorithm code used to score the final data set
Any supporting documentation that pertains to the development of the submitted model/algorithm including variable creation. Variables that were used in the model need to be traced through from input to coefficient / node (if using a tree based methodology).
From the good folks at AsterData, a webcast on a slightly interesting analytics topic
Enterprises and government agencies can become overwhelmed with information. The value of all that data lies in the insights it can reveal. To get the maximum value, you need an analytic platform that lets you analyze terabytes of information rapidly for immediate actionable insights.
Aster Data’s massively parallel database with an integrated analytics engine can quickly reveal hard-to-recognize trends on huge datasets which other systems miss. The secret? A patent-pending SQL-MapReduce framework that enables business analysts and business intelligence (BI) tools to iteratively analyze big data more quickly. This allows you to find anomalies more quickly and stop disasters before they happen.
Discover how you can improve:
Network intelligence via graph analysis to understand connectivity among suspects, information propagation, and the flow of goods
Security analysis to prevent fraud, bot attacks, and other breaches
Geospatial analytics to quickly uncover details about regions and subsets within those communities
Visual analytics to derive deeper insights more quickly
Revolution News
Every month, we’ll bring you the latest news about Revolution’s products and events in this section. Follow us on Twitter at @RevolutionR for up-to-the-minute news and updates from Revolution Analytics!
Revolution R Enterprise 4.0 for Windows now available. Based on the latest R 2.11.1 and including the RevoScaleR package for big-data analysis in R, Revolution R Enterprise is now available for download for Windows 32-bit and 64-bit systems. Click here to subscribe, or available free to academia.
New! Integrate R with web applications, BI dashboards and more with web services. RevoDeployR is a new Web Services framework that integrates dynamic R-based computations into applications for business users. It will be available September 30 with Revolution R Enterprise Server on RHEL 5. Click here to learn more.
Inside-R: A new site for the R Community. At www.inside-R.org you’ll find the latest information about R from around the Web, searchable R documentation and packages, hints and tips about R, and more. You can even add a “Download R” badge to your own web-page to help spread the word about R.
R News, Tips and Tricks from the Revolutions blog
The Revolutions blog brings you daily news and tips about R, statistics and open source. Here are some highlights from Revolutions from the past month.
R’s key role in the oil spill response: Read how NIST’s Division Chief of Statistical Engineering used R to provide critical analysis in real time to the Secretaries of Energy and the Interior, and helped coordinate the government’s response.
Animating data with R and Google Earth: Learn how to use R to create animated visualizations of geographical data with Google Earth, such as this video showing how tuna migrations intersect with the location of the Gulf oil spill.
Are baseball games getting longer? Or is it just Red Sox games? Ryan Elmore uses nonparametric regression in R to find out.
Keynote presentations from useR! 2010: the worldwide R user’s conference was a great success, and there’s a wealth of useful tips and information in the presentations. Video of the keynote presentations are available too: check out in particular Frank Harrell’s talk Information Allergy, and Friedrich Leisch’s talk on reproducible statistical research.
Looking for more R tips and tricks? Check out the monthly round-ups at the Revolutions blog.
Upcoming Events Every month, we’ll highlight some upcoming events from R Community Calendar.
September 23: The San Diego R User Group has a meetup on BioConductor and microarray data analysis.
September 28: The Sydney Users of R Forum has a meetup on building world-class predictive models in R (with dinner to follow).
September 28: The Los Angeles R User Group presents an introduction to statistical finance with R.
September 28: The Seattle R User Group meets to discuss, “What are you doing with R?”
The deal values Netezza at $27 a share, a 9.8 percent premium to its closing price on Friday.
Since Netezza was an existing SAS partner, probably it would impact it more if at all, since IBM-SPSS acquisition. Also Netezza was one of the foremost BI companies for both using and expounding R-
JMP 9 releases on Oct 12- it is a very good reliable data visualization and analytical tool ( AND available on Mac as well)
AND IT is advertising R Graphics as well (lol- I can visualize the look on some ahem SAS fans in the R Project)
Updated Pricing- note I am not sure why they are charging US academics 495$ when SAS On Demand is free for academics. Shouldnt JMP be free to students- maybe John Sall and his people can do a tradeoff analysis for this given JMP’s graphics are better than Base SAS (which is under some pressure from WPS and R)
Be First in Line for JMP® 9
Save up to $300 when you pre-order a
single-user license by Oct. 11
Make JMP your analytic hub for visual data discovery with this special offer, good through Oct. 11, 2010. Pre-order a single-user license of JMP 9 – for a discount of up to $300 – and get ready for a leap in data interactivity.
Order now and enjoy the compelling new features of JMP 9 when the software is released Oct. 12. New capabilities in JMP 9 let you:
Optimize and simulate using your Microsoft Excel spreadsheets.
Use maps to find patterns in your geographic data.
Enjoy the updated look and flexibility of JMP 9 on Microsoft Windows.
Create and share custom add-ins that extend JMP.
Leverage an expanded array of advanced statistical methodologies.
Display analytic results from R using interactive graphics.
What if I already have a JMP 8 single-user license?
Great news! You can upgrade to JMP 9 for less than half the regular price.
What if I’m an annual license customer?
Don’t worry, we’ve got you covered. Annual license customers enjoy priority access to all the latest JMP releases as soon as they become available. JMP 9 will be shipped to you automatically.
What if I work or study in the academic world?
Call 1-877-594-6567 to learn about significant discounts for students and professors through the JMP Academic Program.
rattle (2.5.42) unstable; urgency=low
* Update rattle.info() to recursively identify all dependencies,
report
their version number and any updates available from CRAN and generate
command to update packages that have updates available. See
?rattle.info for the options.
* Fix bug causing R Dataset option of the Evaluate window to always
revert to the first named dataset.
* Fix bug in transforms where weights were not being handled in
refreshing of the Data tab.
* Fix a bug in box plots when trying to label outliers when there aren't
any.
-- Graham Williams <Graham.Williams@togaware.com> Sun,
19 Sep 2010 05:01:51 +1000
rattle (2.5.41) unstable; urgency=low
* Use GtkBuilder for Export dialog.
* Test use of glade vs GtkBuilder on multiple platforms.
* Rename rattle.info to rattle.version.
* Add weight column to data tab.
* Support weights for nnet, multinom, survival.
* Add weights information to PMML as a PMML Extension.
* Ensure GtkFrame is available as a data type whilst waiting for
updated
RGtk2.
* Bug fix to packageIsAvailable not reruning any result.
* Replace destroy with withdraw for plot window as the former has
started crashing R.
* Improve Log formatting for various model build commands.
* Be sure to include the car package for Anova for multinom models.
* Release pmml 1.2.24: Bug fix glm binomial regression - note as
classification model.
-- Graham Williams <Graham.Williams@togaware.com> Wed, 15 Sep 2010
14:56:09 +1000
And a video I did of exploring various Rattle options using Camtasia,
a very useful software for screen capture and video tutorials
from http://www.techsmith.com/download/camtasiatrial.aspUpdated- my video skils being quite bad- I replaced it with another video. However Camtasia is the best screen capture video tool
His argument of love is not very original though it was first made by these four guys
I am going to argue that “some” R developers should be paid, while the main focus should be volunteers code. These R developers should be paid as per usage of their packages.
Let me expand.
Imagine the following conversation between Ross Ihaka, Norman Nie and Peter Dalgaard.
Norman- Hey Guys, Can you give me some code- I got this new startup.
Ross Ihaka and Peter Dalgaard- Sure dude. Here is 100,000 lines of code, 2000 packages and 2 decades of effort.
Norman- Thanks guys.
Ross Ihaka- Hey, What you gonna do with this code.
Norman- I will better it. Sell it. Finally beat Jim Goodnight and his **** Proc GLM and **** Proc Reg.
Ross- Okay, but what will you give us? Will you give us some code back of what you improve?
Norman – Uh, let me explain this open core …
Peter D- Well how about some royalty?
Norman- Sure, we will throw parties at all conferences, snacks you know at user groups.
Ross – Hmm. That does not sound fair. (walks away in a huff muttering)-He takes our code, sells it and wont share the code
Peter D- Doesnt sound fair. I am back to reading Hamlet, the great Dane, and writing the next edition of my book. I am glad I wrote a book- Ross didnt even write that.
Norman-Uh Oh. (picks his phone)- Hey David Smith, We need to write some blog articles pronto – these open source guys ,man…
———–I think that sums what has been going on in the dynamics of R recently. If Ross Ihaka and R Gentleman had adopted an open core strategy- meaning you can create packages to R but not share the original where would we all be?
At this point if he is reading this, David Smith , long suffering veteran of open source flameouts is rolling his eyes while Tal G is wondering if he will publish this on R Bloggers and if so when or something.
Lets bring in another R veteran- Hadley Wickham who wrote a book on R and also created ggplot. Thats the best quality, most often used graphics package.
In terms of economic utilty to end user- the ggplot package may be as useful if not more as the foreach package developed by Revolution Computing/Analytics.
However lets come to open core licensing ( read it here http://alampitt.typepad.com/lampitt_or_leave_it/2008/08/open-core-licen.html ) which is where the debate is- Revolution takes code- enhances it (in my opinion) substantially with new formats XDF for better efficieny, web services API, and soon coming next year a GUI (thanks in advance , Dr Nie and guys)
and sells this advanced R code to businesses happy to pay ( they are currently paying much more to DR Goodnight and HIS guys)
Why would any sane customer buy it from Revolution- if he could download exactly the same thing from http://r-project.org
Hence the business need for Revolution Analytics to have an enhanced R- as they are using a product based software model not software as a service model.
If Revolution gives away source code of these new enhanced codes to R core team- how will R core team protect the above mentioned intelectual property- given they have 2 decades experience of giving away free code , and back and forth on just code.
Now Revolution also has a marketing budget- and thats how they sponsor some R Core events, conferences, after conference snacks.
How would people decide if they are being too generous or too stingy in their contribution (compared to the formidable generosity of SAS Institute to its employees, stakeholders and even third party analysts).
Would it not be better- IF Revolution can shift that aspect of relationship to its Research and Development budget than it’s marketing budget- come with some sort of incentive for “SOME” developers – even researchers need grants and assistantships, scholarships, make a transparent royalty formula say 17.5 % of the NEW R sales goes to R PACKAGE Developers pool, which in turn examines usage rate of packages and need/merit before allocation- that would require Revolution to evolve from a startup to a more sophisticated corporate and R Core can use this the same way as John M Chambers software award/scholarship
Dont pay all developers- it would be an insult to many of them – say Prof Harrell creator of HMisc to accept – but can Revolution expand its dev base (and prospect for future employees) by even sponsoring some R Scholarships.
And I am sure that if Revolution opens up some more code to the community- they would the rest of the world and it’s help useful. If it cant trust people like R Gentleman with some source code – well he is a board member.
——————————————————————————————–
Now to sum up some technical discussions on NeW R
1) An accepted way of benchmarking efficiencies.
2) Code review and incorporation of efficiencies.
3) Multi threading- Multi core usage are trends to be incorporated.
4) GUIs like R Commander E Plugins for other packages, and Rattle for Data Mining to have focussed (or Deducer). This may involve hiring User Interface Designers (like from Apple 😉 who will work for love AND money ( Even the Beatles charge royalty for that song)
5) More support to cloud computing initiatives like Biocep and Elastic R – or Amazon AMI for using cloud computers- note efficiency arguements dont matter if you just use a Chrome Browser and pay 2 cents a hour for an Amazon Instance. Probably R core needs more direct involvement of Google (Cloud OS makers) and Amazon as well as even Salesforce.com (for creating Force.com Apps). Note even more corporates here need to be involved as cloud computing doesnot have any free and open source infrastructure (YET)
“If something goes wrong with Microsoft, I can phone Microsoft up and have it fixed. With Open Source, I have to rely on the community.”
And the community, as much as we may love it, is unpredictable. It might care about your problem and want to fix it, then again, it may not. Anyone who has ever witnessed something online go “viral”, good or bad, will know what I’m talking about.
DECISION STATS 