Rexer Analytics Data Mining Survey Results

Latest Results from Rexer Analytics

HIGHLIGHTS from the 2015 Data Science Survey:
 
    SURVEY & PARTICIPANTS:  59-item online survey conducted in 2015.  Participants: 1,220 analytic professionals from 72 countries.  This is the 7th survey in this ongoing research program.
 
    CORE ALGORITHM TRIAD:  Regression, Decision Trees, and Cluster analysis remain the most commonly used algorithms in the field.
 
    THE ASCENDANCE OF R:  76% of respondents report using R.  This is up dramatically from just 23% in 2007.  More than a third of respondents (36%) identify R as their primary tool.
 
    JOB SATISFACTION:  Job satisfaction in the field remains high, but has slipped since the 2013 survey. A number of factors predict Data Scientist job satisfaction levels.
 
    DEPLOYMENT:  Deployment continues to be a challenge for organizations, with less than two thirds of respondents indicating that their models are deployed most or all of the time.  Getting organizational buy-in is the largest barrier to deployment, with real-time scoring and other technology issues also causing significant deployment problems.
 
    TERMINOLOGY:  The term “Data Scientist” has surged in popularity with over 30% of us describing ourselves as data scientists now compared to only 17% in 2013.
 
Please contact us if you have any questions about the attached report or this research program.  Summary reports for all 7 surveys (2007-2015) are free and available by contacting DataMinerSurvey@RexerAnalytics.com.  Additional information about the surveys is available at www.RexerAnalytics.com, including verbatim best practices and insights shared by respondents in previous surveys — see the Data Miner Survey links in the top right corner of our home page.
 
Information about Rexer Analytics and our consulting services is also available at www.RexerAnalytics.com.

Internships for Data Scientists at DecisionStats in New Delhi India

img_20160930_152513http://www.letsintern.com/internship/IT-internships/Decisionstats/Data-Science-Interns/68307

It’s your chance to work in the field of Data Science Interns with Decisionstats for IT Internship.

About the Internship: They are unpaid.
The data scientists will create , edit and make data science research and assist in writing.

The intern will be given on the job training for data science and analytics in Python SAS and R .

The  data science intern will create , edit and make schedules and assist in coordination.

The data science intern will be given on the job training for managing in a start up environment, web analytics and search engine marketing as well as an understanding of digital business.

The data science intern will also proof read, edit and write content including blog posts and social media.

The data science intern will be given on the job training for social media, web analytics and search engine optimization as well as an understanding of digital business.
Number of Internships available: 5
Perks:
Certificate, Letter of recommendation, Flexible work hours, Informal dress code, 5 days a week.
Who can apply:
Only those candidates can apply who are available for full time (in-office) internship. They can start the internship between 30th Sep’16 and 30th Oct’16.
1.  are available for minimum 2 months duration.
2.  are living or staying in Delhi.
3.  are pursuing any degree but have relevant skills and interest.
4.  are currently in any year of study or are recent graduates.

International Students can also apply

screenshot-from-2016-09-30-17-49-58

 

Interview Kiran Rama India’s Number One Data Scientist

Here is an interview with Kiran Rama. He is currently Director, Data Sciences & Advanced Analytics at VMWare. I have chosen Kiran as India’s number one data scientist for the following reasons

  1. He has both an impeccable academic record as well as steady work experience across multiple companies
  2. He has demonstrated his expertise in competitions like Kaggle and KDD cup (which is tougher)
  3. He spends more time doing and expanding data science in India

12976970_10154173725606579_4175448674223012627_o

Here is the interview with Kiran Rama, India’s Number One Data Scientist as per 2016 as per Decisionstats.com

Ajay- Describe your career as a data scientist from corporate job, entrepreneurial work experience, winning competitions, patents and finally back to industry
Kiran- I always had a flair for programming being a computer science engineer and winning inter-college on-the-spot programming and debugging fests. Post computer science engineering graduation, my interesting work was at Motorola using C/Linux for developing features for protocol of Multimedia Messaging Service on several mobile phones. I also owned protocol analytics that involved debugging log files looking for null pointer errors!
Post a couple of years of work, I pursued my post-graduation in management from Indian Institute of Management Kozhikode (IIMK) where I finished in the top 5 of the batch majoring in Information Technology & Systems. I was confused what to do after engineering getting 99 percentile+ in both CAT and GATE. I did not want to stop programming or lose my technical orientation and therefore a role in analytics/data sciences seemed like a natural fit as it involved both technical and business stuff.
I was one of the first hires of the e-business analytics team in Dell in 2006.  I got certified in Base SAS and SAS Enterprise Miner and used SAS primarily for data sciences while I used Omniture tools, Excel, SQL for analytics. At Dell Global Analytics, I took on diverse responsibilities and grew from the equivalent of a senior analyst to a Senior Analytics Manager. I touched all parts of e-commerce and e-business.
Some of my achievements in Dell included:
  • “2012 India Innovator of the Year” Award from Michael Dell
  • 3 patents filed at US PTO on various aspects of e-commerce and marketing analytics
  • World Quality Day Finalist in 2010
  • Won the Best Project Award in Global Consumer & Small Business Analytics for 4 consecutive quarters
While at Dell, in 2010 or so, knowing SAS well I was frustrated that I could not freelance using SAS owing to the high cost. At that time I picked up R and it is the best decision I made in my career as I took to R like a fish to water owing to it’s many similarities with C.
At Dell, I started participating in data mining competitions on Kaggle.com and had several top finishes.  I was a “Master Data Miner” on kaggle. I had great results in the Amazon Employee Access Challenge, Merck Competition to predict Molecular activity, GEFcom competition on load forecasting, on wind forecasting….etc. My Kaggle pursuits was one of the reason I was recruited by Amazon.
I worked as a Senior Business Analytics Lead at Amazon in their Bangalore office as a Level-6 individual contributor. Level-6 in Amazon in those days was one of the senior-most individual contributor position that they had in Bangalore in the engineering teams and very difficult to get laterally on the technical side. However the role was not to my liking and I decided to leave to head marketing & Customer analytics at Flipkart.
I freelanced for several US startups as part of part-time proprietorship “Chaotic Experiments”. Some projects included:
  • Software Errors: Predict which line in software code is likely to be an error for a US based startup
  • Accident evaluation analysis for a US semi sized startup
  • Predict which music label to recommend to a startup
  • Trying to predict futures prices in the stock market for a US Startup
  • HLA Imputation of Genomic data
At Flipkart, I had the good opportunity of leading several data sciences & analytics projects for Flipkart of which the below ones I am proud of:
  • Leveraging Data Sciences to come up with customer segments for Flipkart’s digital properties
  • Coming up with an email rules engine to determine the best customers to target per category
  • Setting up mobile app analytics at Flipkart
I worked closely at Flipkart with the CTO on data scientist hiring and helped in hiring data scientists being the decision maker for the “data sciences depth” round for data scientists at Flipkart
I continued my hand at Kaggle while at Flipkart and at one time for over a year, I was ranked amongst the top 10 data miners in the world on Kaggle. My top rank was 7 out of some 300K data miners competing in data sciences competitions for sport and the icing on the cake came when I finished in the top 3 in KDD Cup 2014 winning the competition to predict which essay was likely to  get a donation on donorschoose.org
Post Flipkart, I was hired into VMW, where I play the role of “Director, Data Sciences & Advanced Analytics”. I play dual roles of functional (where I lead the data sciences innovations team for VMW globally working closely with digital analytics, Digital store/e-commerce, Professional Services, Sales, Marketing, Partner, Pricing, Support,… verticals) and dotted-line (where I represent the equivalent of the Enterprise Information Management in India comprising of Master Data Management, Business Intelligence & Advanced Analytics).
At VMW, have had a unique experience driving B2B data sciences with industry leading projects like:
  • First ever digital buyer journey data sciences project at VMW
  • “Propensity to Buy” models for several products of VMW, for the Technical Account Manager organization,..
  • “Propensity to Sell” models for the partner organization of VMW
  • “Propensity to Respond” models
  • Deployment models
I currently am at VMW and have been here since almost 2.5 years and loving every moment of it.
Ajay- What are the key things you want to say to someone with no work experience and who wants to work as a data scientist ? What would you say to someone who has a few years work experience and wants to switch to data science
Kiran- For both (freshers and experienced), I would say the following are key in the order of priority:
  • Debugging Skills: You cannot give up as a data scientist and should be a person who can sit at one place and continuously debug for hours. Data Science techniques usage will involve installations, OS issues, nitty-gritty aspects of the code,… etc
  • Programming Skills: You cannot be a data scientist if you cannot program. You need to be good at programming. Comments like code is available on the net and I will copy-paste do not work. I judge a data scientist by different parameters and one of the most important ones is the quality of the code!
  • Knowledge of a Programming Language that has a machine learning library (R or Python are an example. R has access to many of the libraries on the CRAN repository while Python has the world beating scikit-learn package)
  • Strong understanding of the mathematical and computer science and statistical background of the data sciences techniques behind the techniques
  •  Ability to translate a business problem into a data sciences problem. This involves key decisions like which is the target, is this a prediction or classification problem, what is the right cross-validation technique, what algorithms to use for data mining, what should be the right evaluation criteria, how the model will likely be deployed,…
  • Strong business/domain understanding can lead to great feature engineering and great success while deployment.
  • Ability to present the results to stakeholders and get buy-in for implementation is very important as well
There is a lot of misconception that everyone should do data sciences. Not everyone is suited for this. If you cannot sit in a place for a stretch and code for 5-6 hours in R or Python and SQL, this is not the right job for you. If you do, this is the best thing to do.
Ajay- you have used many tools like SAS Python and R. How would advise a new data scientist on which tool to learn and how to structure their tools training
Kiran-I would suggest a new data scientist to use Python > R > SAS mainly because:
  • Python and R have better and wider machine learning libraries than SAS
  • Most of the academic work and latest advancements are in Python & R
  • Python is better than R because there are more things you can do in Python including software development. Trust me – there is no money in machine learning libraries. There is money only in applications and closer you are to software development + machine learning, the better
  • Most of the high paying startups and young firms use Python/R and not SAS
  • It is easier to learn Python/R and then if you happen to work for an old behemoth that is a SAS shop, pick up SAS as well
  • Python/R are actual programming languages and better than SAS. SAS uses macros and not functions. SAS uses proprietary dataset format that is largely inefficient. SAS requires you to know different syntax for different methods and also different types of plots. On the other hand, the interface to call any function in R or Python is the same. Example: predict function in R. Since everything is returned as an object in R & Python it is easier to examine them (contrast looking at the object sub-objects to running multiple commands in SAS to find the output datasets – the infamous “ods trace on” in SAS,………etc)
A good data scientist should know both R & Python but better to start out in one and master for a year.
Great books to learn R and Python are:
  • “The Art of R Programming” by Norman Matloff
  • “Python for Data Analysis” by Wes Mc Kinney
For Machine learning fundamentals would recommend:
  • Learning from Data by Mostafa
  • Applied Data Mining by Paolo Giudici
  • Machine Learning by Tom Mitchell

13725095_10154440244451579_3409709718217659176_o

Ajay- What are some key best practises you want to tell to people preparing for data science competitions
Kiran- Here is the link to my kaggle interview on winning the KDD 2014 cup: http://blog.kaggle.com/2014/08/05/3rd-place-interview-from-the-kdd-cup-2014/
Here is the link to my code repository for the winning solution in KDD 2014 cup:https://github.com/rkirana/kdd2014
Best practices I suggest are:
  • Build your own repository of functions and methods that you can re-use
  • Understand what the winners of prior competitions did. For example: my code above
  • Keep yourself current with the latest techniques. For example: xgboost
  • Choose the right cross-validation technique. Else, you will overfit
  • Be paranoid about leakage and look for ways to fix leakage in everything including data preparation, feature engineering and modeling
  • Feature Engineering is the key. Even with lesser data, better features will do better than big data
  • Try different methods that are varied. Example: one learner can be tree-based, one bagging, one boosting based, one neural network….etc
  • Always ensemble. It can give 2-5% lift
Last mile optimization is difficult. While you can get a 0.85  AUC easily, taking it to 0.88 AUC can be an uphill task
Ajay- What are some of the key algorithms that a data scientist should know?
Kiran- Some of the algorithms that one should know are:
  • Regularized Logistic Regression (glmnet in R)
  • Bagging technique: Random Forest
  • Boosting Technique: Gradient Boosting Machine, Extreme Gradient Boosting
  • Collaborative Filtering Techniques: LIBFM
  • Non linear learners like Neural Networks
  • Bayesian Methods like BayesTree, bartMachine
  • Support Vector Machines – LIBSVM library
  • Fast learners like Vowpal Wabbit
One should always ensemble multiple techniques in order to get better results
Ajay- Describe your favourite online learning resources for learning data science languages, algorithms etc
Kiran- kaggle.com – nothing beats it
cran.r-project.org – all the vignettes there
Ajay- How do you keep yourself updated on data science knowledge 
Kiran- I have not participated in data science competitions for last 2 years – participating was a way in which I kept myself and pushed myself to be updated
I am very keen on making some original contributions to data sciences research and teaching. I am pursuing a part time doctoral program (PhD) at IIM Lucknow while I do my full-time job. Spend a lot of time on scholar.google.com these days to understand the existing contributions to data science theory and how I can make original contributions to the same
I also drive industry-academia interaction with the Data Center and Analytics Lab at IIM Bangalore where I represent VMW on the DCAL board. I am at the forefront of organizing industry events on data sciences to share knowledge and learn about the latest in the industry.
I am thankful to my leaders and my direct team at VMW for giving me so many interesting business problems to solve using data sciences and that pushes me and drives me to keep myself updated

About-

Kiran is a Data Sciences Leader with more than 12 years of experience across marketing, digital (web/mobile), retail, pricing, partner, sales. Experience across B2C, e-commerce & B2B data sciences. One of the Top 10 Player on Kaggle – data mining competition platform – in 2013 and half of 2014 world-wide, Kiran is also KDD 2014 Prize Winner and Holder of 3 US patents. 2012 Innovator of the Year award from Michael Dell.

You can read about him here https://www.linkedin.com/in/rkirana

 

 

Saving Private Snowden

No apologies no pardons

Full disclosure on what was stolen and what was not

Helping cooperate to make system more secure

Unconditional surrender

Obama is a lawyer and so is Clinton.  They don’t get privacy but they do get popular opinion.  So a popular campaign for Snowmen pardon needs to be organized.

Personally I doubt if Edward can come come home before 2020

A Tutorial at Statisticviews.com

I just wrote a tutorial on SAS language here at Statisticviews.com

SAS was the first statistical language I learnt, followed by R, Python, Julia..

Of course the first language I learnt was in school BASIC

Here is a tutorial on SAS language again for learners, it uses the SAS University Edition

http://www.statisticsviews.com/details/feature/9692841/A-Tutorial-on-SAS-language.html

SAS (pronounced “sass”) once stood for “statistical analysis system” but now is known simply as SAS. It is a computer language for statistical computing. Much before the term data science, business analytics and business intelligence was coined, SAS was created at North Carolina State University to do the important and useful task of turning raw data into analysis using code and statistics.

SAS System is a suite of products that SAS Institute has been selling since 1976.

Jim Goodnight has been the CEO and vision behind the growth of SAS language since almost 40 years now. John Sall has made additional contributions to statistics by creating JMP (interviewed here at StatisticsViews )

SAS Institute has consistently ranked as one of the best employers within USA. However recently R and Python languages have challenged traditional share of SAS language in statistical computing, while SPSS has been acquired by IBM.

Tutorial Overview

This tutorial is here to help a reader with learning the simple  SAS language, and perhaps to inspire other languages to be both simple and responsive to a wide diversity of users from beginners to advanced, from academics to enterprises of various sizes and in geographies. The tutorial is based on SAS Studio interface given for free in the SAS University Edition.

Read more at

http://www.statisticsviews.com/details/feature/9692841/A-Tutorial-on-SAS-language.html

additional

Python Tutorial at

http://www.statisticsviews.com/details/feature/8868901/A-Tutorial-on-Python.html

Crime Forecasting Challenge : Data Science Contest

The Real-Time Crime Forecasting Challenge seeks to harness the advances in data science to address the challenges of crime and justice. It encourages data scientists across all scientific disciplines to foster innovation in forecasting methods. The goal is to develop algorithms that advance place-based crime forecasting through the use of data from one police jurisdiction.

aims to:

  1. Encourage “nontraditional” crime forecasting researchers to compete against more “traditional” crime forecasting researchers.
  2. Compare available crime forecasting methods.
  3. Improve place-based crime forecasting.

Accordingly, the Challenge will have three categories of contestants: students; individuals/small businesses; and large businesses

This Challenge will be based on the locations listed in calls-for-service (CFS) records provided by the Portland Police Bureau (PPB) for the period of March 1, 2012 through February 28, 2017

find:

  1. Overview
  2. How to Enter
  3. Important Dates
  4. Judges
  5. Judging Criteria
  6. Prizes
  7. Other Rules and Conditions
  8. Prize Disbursement and Challenge Winners
  9. Contact Information
  10. Data for Download

 

Source

http://nij.gov/funding/Pages/fy16-crime-forecasting-challenge.aspx?utm_source=KDNuggets&utm_medium=Ad&utm_content=Digital&utm_campaign=ForecastingChallenge