Analytics – Page 76 – DECISION STATS

Analytics Conferences for 2012

NOTE: Early Bird registration for PAW and TAW San Francisco is January 20th – $400 lower than Onsite Price.

CONFERENCE: Predictive Analytics World – San Francisco
March 4-10, 2012 in San Francisco, CA
http://predictiveanalyticsworld.com/sanfrancisco/2012
Discount Code : AJBP12

CONFERENCE: Text Analytics World – San Francisco
March 6-7, 2012 in San Francisco, CA
http://textanalyticsworld.com/sanfrancisco/2012
Discount Code :AJBP12

VARIOUS ANALYTICS WORKSHOPS:
A plethora of 1-day workshops are held alongside PAW and TAW
For details see: http://pawcon.com/sanfrancisco/2012/analytics_workshops.php

SEMINAR: Predictive Analytics for Business, Marketing & Web
March 22-23, 2012 in New York City, NY
July 26-27, 2012 in São Paulo, Brazil
A concentrated training program lead by Eric Siegel.
http://businessprediction.com

CONFERENCE: Predictive Analytics World – Toronto
April 26-27, 2012 in Toronto, Ontario
http://predictiveanalyticsworld.com/toronto/2012
Discount Code :AJBP12

CONFERENCE: Predictive Analytics World – Chicago
June 25-26, 2012 in Chicago, IL
http://www.predictiveanalyticsworld.com/chicago/2012/
Discount Code :AJBP12

MORE ANALYTICS EVENTS:
PAW Düsseldorf: November 6-7, 2012 – http://www.predictiveanalyticsworld.de
PAW London: November 27-28, 2012 – http://www.pawcon.com
PAW Videos: Available on-demand – http://www.pawcon.com/video

Topic Models

Some stuff on Topic Models-

http://en.wikipedia.org/wiki/Topic_model

In machine learning and natural language processing, a topic model is a type of statistical model for discovering the abstract “topics” that occur in a collection of documents. An early topic model was probabilistic latent semantic indexing (PLSI), created by Thomas Hofmann in 1999.[1] Latent Dirichlet allocation (LDA), perhaps the most common topic model currently in use, is a generalization of PLSI developed by David Blei, Andrew Ng, and Michael Jordan in 2002, allowing documents to have a mixture of topics.[2] Other topic models are generally extensions on LDA, such as Pachinko allocation, which improves on LDA by modeling correlations between topics in addition to the word correlations which constitute topics. Although topic models were first described and implemented in the context of natural language processing, they have applications in other fields such as bioinformatics.

http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation

In statistics, latent Dirichlet allocation (LDA) is a generative model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar. For example, if observations are words collected into documents, it posits that each document is a mixture of a small number of topics and that each word’s creation is attributable to one of the document’s topics. LDA is an example of a topic model

David M Blei’s page on Topic Models-

http://www.cs.princeton.edu/~blei/topicmodeling.html

a general introduction to topic modeling .
At KDD-2011 a long tutorial about topic modeling. The slides are here .
slides from a talk on dynamic and correlated topic models applied to the journal Science . (Here is a video of the talk.)
a more technical review paper about this field.
David Mimno maintains a bibliography of topic modeling papers and software.

The topic models mailing list is a good forum for discussing topic modeling.

In R,

topicmodels and lda are two R packages for LDA analysis.

Some resources I compiled on Slideshare based on the above- Continue reading “Topic Models”

Information Ladder for Analytics

One very commonly used diagram in marketing and sales by analytics providers, which is hardly ever credited to its author is the Information Ladder

http://en.wikipedia.org/wiki/Information_ladder

The information ladder is a diagram created by education professor Norman Longworth to describe the stages in human learning. According to the ladder, a learner moves through the following progression to construct “wisdom” at the highest level from “data” at the lowest level:

Data →

Information →

Knowledge →

Understanding →

Insight →

Wisdom

Whereas the first two steps can be scientifically exactly defined, the upper parts belong to the domain of psychology and philosophy.

I sometimes think the information ladder and especially the latter two parts are underutilized, under-quantified as metrics and rarely understood completely by the wise men in analytics and information display.

Some visual versions are below

Funny enough, it is one of the rare concepts first inspired by poetry-

http://en.wikipedia.org/wiki/DIKW

The earliest formalized distinction between wisdom, knowledge, and information may have been made by poet and playwright T.S. Eliot

Where is the Life we have lost in living?
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?

Business Analytics Projects

As per me, Analytics Projects get into these four broad phases-

Business Problem Phase– What needs to be done?

Increase Revenues
Cut Costs
Investigate Unusual Events
Project Timelines

Technical Problem Phase – Technical Problems in Project Execution

Data Availability /Data Quality/Data Augmentation Costs
Statistical -(Technique based approach) , Hypothesis Formulation,Sampling, Iterations
Programming-(Tool based approach) Analytics Platform Coding (Input, Formats,Processing)

Technical Solution Phase – Problem Solving using the Tools and Skills Available

Data Cleaning /Outlier Treatment/Missing Value Imputation
Statistical -(Technique based approach) Error Minimization, Model Validation, Confidence Levels
Programming-(Tool based approach) Analytics Platform Coding (Output, Display,Graphs)

Business Solution Phase– Put it all together in a word document, presentation and/or spreadsheet

Finalized- Forecasts , Models and Data Strategies
Improvements in existing processes
Control and Monitoring of Analytical Results post Implementation
Legal and Compliance guidelines to execution
(Internal or External) Client Satisfaction and Expectation Management
Audience Feedback based on presenting final deliverable to broader audience

December Snowflakes R 2.14.1

Almost missed this one due to Christmas-

R 2.14.1 is out, and so are binaries

so download them here (winduh users!)

http://cran.r-project.org/bin/windows/base/

David S sums it all up here

http://blog.revolutionanalytics.com/2011/12/r-2141-is-released.html

This update makes a few small improvements (such as the ability to accurately count the number of available cores for parallel processing on Solaris and Windows, and improved support of grayscale Postscript and PDF graphics export) and fixes a few minor bugs (such as a correction to BIC calculations in the presence of zero-weight observations).

Binaries are here-

http://cran.r-project.org/bin/windows/base/R-2.14.1-win.exe

Prof Peter D speeaks here-

https://stat.ethz.ch/pipermail/r-announce/2011/000548.html

Changes in recent versions are here-

http://cran.r-project.org/bin/windows/base/CHANGES.R-2.14.1.html

Major Changes-

Direct support in R is starting with release 2.14.0 for High Performance Computing

2011 Analytics Recap

Events in the field of data that impacted us in 2011

1) Oracle unveiled plans for R Enterprise. This is one of the strongest statements of its focus on in-database analytics. Oracle also unveiled plans for a Public Cloud

2) SAS Institute released version 9.3 , a major analytics software in industry use.

3) IBM acquired many companies in analytics and high tech. Again.However the expected benefits from Cognos-SPSS integration are yet to show a spectacular change in market share.

2011 Selected acquisitions

Emptoris Inc. December 2011

Cúram Software Ltd. December 2011

DemandTec December 2011

Platform Computing October 2011

Q1 Labs October 2011

Algorithmics September 2011

i2 August 2011

Tririga March 2011

4) SAP promised a lot with SAP HANA- again no major oohs and ahs in terms of market share fluctuations within analytics.

http://www.sap.com/india/news-reader/index.epx?articleID=17619

5) Amazon continued to lower prices of cloud computing and offer more options.

http://aws.amazon.com/about-aws/whats-new/2011/12/21/amazon-elastic-mapreduce-announces-support-for-cc2-8xlarge-instances/

6) Google continues to dilly -dally with its analytics and cloud based APIs. I do not expect all the APIs in the Google APIs suit to survive and be viable in the enterprise software space. This includes Google Cloud Storage, Cloud SQL, Prediction API at https://code.google.com/apis/console/b/0/ Some of the location based , translation based APIs may have interesting spin offs that may be very very commercially lucrative.

7) Microsoft -did- hmm- I forgot. Except for its investment in Revolution Analytics round 1 many seasons ago- very little excitement has come from MS plans in data mining- The plugins for cloud based data mining from Excel remain promising yet , while Azure remains a stealth mode starter.

8) Revolution Analytics promised us a GUI and didnt deliver (till yet 🙂 ) . But it did reveal a much better Enterprise software Revolution R 5.0 is one of the strongest enterprise software in the R /Stat Computing space and R’s memory handling problem is now an issue of perception than actual stuff thanks to newer advances in how it is used.

9) More conferences, more books and more news on analytics startups in 2011. Big Data analytics remained a strong buzzword. Expect more from this space including creative uses of Hadoop based infrastructure.

10) Data privacy issues continue to hamper and impede effective analytics usage. So does rational and balanced regulation in some of the most advanced economies. We expect more regulation and better guidelines in 2012.