Ten ways to build a wrong scoring model

 

Some ways to build a wrong scoring model are below- The author doesn’t take any guarantee if your modeling team is using one of these and still getting a correct model.

1) Over fit the model to the sample. This over fitting can be checked by taking a random sample again and fitting the scoring equation and compared predicted conversion rates versus actual conversion rates. The over fit model does not rank order – deciles with lower average probability may show equal or more conversions than deciles with higher probability scores.

2) Choose non random samples for building and validating the scoring equation. Read over fitting above.

3) Use Multicollinearity (http://en.wikipedia.org/wiki/Multicollinearity ) without business judgment to remove variables which may make business sense.Usually happens a few years after you studied and forgot Multicollinearity.

If you don’t know the difference between Multicollinearity , Heteroskedasticity http://en.wikipedia.org/wiki/Heteroskedasticity this could be the real deal breaker for you

4) Using legacy codes for running scoring usually with step wise forward and backward  regression .Happens usually on Fridays and when in a hurry to make models.

5) Ignoring signs or magnitude of parameter estimates ( that’s the output or the weightage of the variable in the equation).

6) Not knowing the difference between Type 1 and Type 2 error especially when rejecting variables based on P value. ( Not knowing P value means you may kindly stop reading and click the You Tube video in the right margin )

7) Excessive zeal in removing variables. Why ? Ask yourself this question every time you are removing a variable.

8) Using the wrong causal event (like mailings for loans) for predicting the future with scoring model (for mailings of deposit accounts) . or using the right causal event in the wrong environment ( rapid decline/rise of sales due to factors not present in model like competitor entry/going out of business ,oil prices, credit shocks sob sob sigh)

9) Over fitting

10) Learning about creating models from blogs and not  reading and refreshing your old statistics textbooks

Guns and No Glory

Beginning henceforth here is the policy on comments and posts.It is in response of the comments on my “A Farewell to Guns “ post in which I projected my Gandhian non violence too far in suggesting the remote possibility of  a ban on guns based on Alabama and Germany events.

  1. No more political posts (including on India) or poems (including funny) will be imposed on this blog or unsuspecting readers.*
  2. I use an offline blogging system called Windows Live Writer and do not go to approve the comments online (requires me to login to word press and manually do it). All comments are read via email settings and feedback incorporated.Comments sometimes get deleted within 7 days because of auto settings (like me not going and logging in to WordPress for 7 days) not because I am ignoring anything
  3. You didn’t like the “Guns” article – you have the right to say so on the comments page. You didn’t like the R article or the package–comments page please.
  4. Read Page “Fine Print” . I use a professional analytics tracking system which I pay for every month- it tracks Ip address ,Ip provider, organization, location, time, country, with an integrated Google maps that allows me to see which block the material entered the net.Writing offensive comments from your work computer is not a great idea- not with the angry one.Not if you are……
  5. I never use the analytics system for individuals unless the comments are ghastly. Then I delete the comments and don’t use the analytics system for individuals.
  6. Akismet for spam will catch and has caught multiple attempts at malware linking and spamming.It will do so.
  7. Anonymous comments are not anonymous as explained above.

A blog on Decision Stats should quote political statements only when accompanied with statistics. or better still nothing but the statistics.So it will be.

*They will be posted on www.iwannacrib.com

Interview- BI Dashboards dMINE Sanjay Patel

If you have ever felt frustrated in  knowing business metrics in your or your client organization, negotiated with a host of either legacy applications that don’t talk to each other or good solutions that cost more than the benefit they bring – a young man from India has a solution for you. With a total implementation time range of 1-6 weeks and costs to as low as 10,000 USD for Enterprise WIDE implementation , Dmine promises to shake things up. Here is an interview with the co founder of this startup.

 

Ajay- Describe your career journey. What advice would you give to  new entrepreneurs in this recession.

Sanjay- Geared with an M. Tech from BITS Pilani, and MBA from Jamnalal Bajaj Institute of Management, Mumbai, in 2000 I teamed up with Praveen Wicliff and  ventured to start our own  Product company with a focus to deliver critical enterprise solutions  and the company has now grown to a strong team of 100 innovative minds. My current mission is to make Icicle a strong leader in business driven Software Products across all segments with the best of the delivery capabilities.
Recession is a trying time for most people and this is one phase which brings out the best in everyone as most of the innovative solutions are floated during this phase. My only suggestion would be to move from emotional connect with customers to direct tangible benefits, stay focussed on cash flow, aim high, set your goals & targets and never give up on any of these. Even in the darkest moments, find the faith to keep going.

Ajay- One more Dashboard Solution. How is dMine different from it’s competitors. What are the principal competitors.

Sanjay- If you just plainly look at dMine you would see that dMine is just another Dashboard solution but what makes dMine different from all other competitive products is its intuitiveness and User friendly features which help even the Business users to use the product most effectively.

dMine_logo_72dpi_11032008

dMine is positioned as a product for Business Users and not for IT team. Unlike other Dashboard or BI products dMine can create Dashboards in just 3 easy steps:

1- IT team connects to Data-sources & Creates Business Views

2- Users can create Dashboards & Charts with dMine’s Intuitive interface

3- Users can share Dashboards & schedule Emails in PDF or PPT

The potent combination of best-of-class looking Comprehensive Graphical and Analytical reports, Easier representation and Interpretation of Key Business Data, Integrates data from multiple systems on a single chart and / or Dashboard for real-time Analysis, all these, with minimized IT overheads is a unique proposition from us. See dMine-in-action on  www.dminebi.com and you will know the difference.

Ajay- What is the area where dMine would not be suited for dashboards.Suppose I have data for 200,000 rows x 40 columns – would dMine work for me .

Sanjay-
dMine is positioned as a pure Dashboard product that does not implement a complete BI stack which requires to work on Transactional Data to create cubes and universe.

We look beyond Data warehouses and Datamarts and emphasize on summarized data to deliver key business performance metrics with high focus on Data Visualization.

The idea here is to target the Business Executives who would see these Dashboards and they are not interested in Transactional Data but the overall performance hence the summarized Data.

Here the summarized Data could be in the form of any RDBMS Database, Flat Files, Spreadsheets, Analytics output, Cubes or Universe. We Support almost 16 Database vendors in the market starting from as small as MS Access to as big as DB2.

Ajay- What is the pricing strategy of dMine . Any other products or complements that you are thing about. Name some customer case studies or big wins.

Sanjay- dMine pricing strategy is very simple and is based keeping in mind that the product can be used by customers in the SME/SMB segment or even at the Enterprise level.

Currently we are offering dMine in two forms

  • firstly On-Premise and
  • second one as a Hosted service.

In an on-premise version, dMine has a Product License fee and per user license fees issued separately.

At additional cost you can have loads of Add-on goodies catering to various needs of the customers. All the cost mentioned above are just one time.

The Hosted version, is on a monthly subscription model where the cost is decided based on the various parameters like the usage of Bandwidth, Server configuration, Disk space etc. Very soon we will be enabled to the Amazon cloud service.

Typically for an on-premise version a smaller implementation just at a corporate level with Dashboard access to only few top executives would cost anywhere between 10,000 USD to 16,000 USD plus the implementation cost which is on actuals plus Applicable taxes.

For larger implementation like for e.g in BFSI segment where you need to roll-out user licenses to all the branch managers in addition to the top executives the total User licenses goes to a few 100 licenses. Usually the implementation period is as low as a week and not more than 6 weeks.

Few of the Customers using dMine are some of the Marketing Analytics companies that use the product for submitting the final report of their Marketing / Customer Analysis to their end customers.

image

Case Study – Summary

We recently implemented dMine at a leading FM Radio channel to monitor the performance across its radio stations spread all over India.

The client being a major player in the media and communication space has to constantly monitor all their stations for their entire  operations like Revenues from sponsors, Peak and Non-Peak time Inventory & Sales, Market Share & Channel Ranking, P&L, Forecasts and other critical informations.

Currently the client uses multiple Applications for supporting these business functions which capture data in different databases and sources. The MIS-reports were manually created by extracting data from multiple sources in spreadsheets. These spreadsheets are distributed among the management, with a turn around time of about 15 days.

The dMine solution implemented, collates information from multiple data sources like ERP and Sales systems including lots of Spreadsheets. More than 70 Metrics (KPI’s) and Analytics are defined for the Client and now the management has access to these information whenever they require.

All these metrics are identified as critical and are categorized under 5 Dashboards  – Organizational KPI, Financial Dashboard, Sales Dashboard, Market Share
& Metrics Dashboard and Operational Dashboard. The Metrics are parameterized and drill-downs allows the management to get the source of issue/problem rapidly.

The implementation of dMine Business Dashboard product helped the client in effectively monitoring business operations, KPIs, and organizational performance.Making actionable and Real-time information available on-demand for the decision Makers and Operational Managers, has also helped in taking any timely critical business decisions, all these while minimizing IT overhead cost. The short implementation time lines also allows the users to see the benefits quite quickly and achieve the ROI within 1 to 3 months time. The detailed case study is readily available for your reference at our website www.dminebi.com .

Ajay-  Do you read or write blogs. What do you think about the Web 2.0 paradigm for social and community marketing.

Sanjay –I do read and write lots of blogs and am myself a member with quite a few groups that share interest in the virtual community. Web 2.0 provides a platform of many-to-many communications and in its social sense is based on the principles of collaboration & sharing, information & content putting social interaction at heart of it all.

A recent study by Fox Interactive Media reveals that 40% of social network users rely on social media outlets to learn more about brands and products. Whether you’re a freelancer promoting your own brand or part of a company, social media marketing is an essential component of an integrated campaign. If you are looking to startup new business, launch a new product & services or even expand your presence you cannot miss-out on the eMarketing process focusing on three prong strategy i.e. social networking sites, your own website, and the blogosphere these will help empower your brand and positively convey your message.

Ajay – Sanjay Patel is an experienced entrepreneur with Icicle Technologies and the Dmine dashboard is currently winning rave reviews ( see http://www.dminebi.com/ibm-nominates-icicle-as-isv-on-ibm-smart-business-platform/ ) . Here’s wishing luck to Mr Patel for the summarized data dashboard that can be a game changer at www.dminebi.com.

IcicleTech-logo_72

Interview with Anne Milley, SAS II

Anne Milley is director of product marketing, SAS Institute . In part 2 of the interview Anne talks of immigration in technology areas, open source networks ,how she misses coding and software as a service especially SAS Institute’s offering . She also reveals some preview on SAS ‘s involvement with R and mentions cloud computing.

Anne_Milley

Ajay – Labor arbitrage outsourcing versus virtual teams located globally. What is the SAS Inst position and your opinion on this. What do you feel about the recent debate on HB1 visas and job cuts. How many jobs if at all is SAS planning to cut in 2009-2010.

Anne – SAS is a global company, with customers in more than 100 countries around the world.  We hire employees in these countries to help us better serve our global customers.  Our workforce decisions are based on our business needs.  We also employ virtual teams–the feedback and insights from our global workforce help us improve and develop new products to meet the evolving needs of our customers.  (As someone who works from her home office in Connecticut, I am a fan of virtual teaming!)  We see these approaches as complementary.

The issue of the H-1B visa is a different discussion entirely.  H-1B visas, although capped, permit US employers to bring foreign employees in “specialty occupations” into this country.   The better question, though, is what is necessitating the need for H-1B visas.  We would submit that the reason the U.S. has to look outside its borders for highly qualified technical workers is because we are not producing a sufficient number of workers with the right skill sets to meet U.S. demand.  In turn, that means that our educational system is not producing students interested or qualified to pursue the STEM (science, technology, engineering or mathematics) professions (either at a K-12 or post-secondary level), or developing the workforce improvement programs that may allow workers to pursue these “specialty occupations.”  Further, any discussion about H-1B visas (or any other type of visa) should include a more comprehensive review of our nation’s immigration policies—are they working, are they not working, how or why are they, are we able to limit illegal immigration and if not, why not, etc.

I am not aware of any planned job cuts at SAS.  In fact, I am aware of a few groups which are actively hiring.

Ajay- What open source softwares have SAS Institute worked in the past and it continues to support financially as well as technologically.  Any exciting product releases in 2009-2010 that you can tell us about.

Anne- Open source software provides many options and benefits.  We see many (SAS included) embracing open source for different things.  Our software runs on Linux and we use some open-source tools in development. There are different aspects of open source software in developing SAS software:

-Development with open source tools such as Eclipse, Ant, NAnt, JUnit, etc. to build, test, and package our software

-Using open source software in our products; examples include Apache/Jakarta products such as the Apache Web Server.

-Developing open source software, making changes to an open source codebase, and optionally contributing that source back to the open source project, to adapt an open source project for use in a SAS product or for internal use. Example: Eclipse.

And we plan to do more with open source in the future.  The first step of SAS integrating with R will be shown at SAS Global Forum coming up in DC later this month.  Other announcements for new offerings are also planned at this event. 

Ajay- What do you feel about adopting Software as a service for any of  SAS Institute’s products. Any new initiatives from SAS on the cloud computing front especially in terms of helping customers cut down on hardware costs.

Anne- SAS Solutions OnDemand, the division which oversees the infrastructure and support of all our hosted offerings, is expanding in this rapidly growing market.  SAS Solutions OnDemand Drug Development was our first SaaS offering announced in January.  Additional news on new hosted offerings will be announced at SAS Global Forum later this month.  SAS doesn’t currently offer any external cloud computing options, but we’re actively looking at this area.

AjayWhich software do you personally find best to write code into and why. Do you miss writing code, if so why ?

Anne- In my current role, I have limited opportunity to write code.  At times, I do miss the logical thought process coding forces you to adopt (to do the job as elegantly as possible).  I had the opportunity to do a long-term assignment at a major financial services company in the UK last year and did get to use some SAS and JMP, including a little JSL (JMP scripting language).  There’s nothing like real-world, noisy, messy data to make you thankful for the power of writing code!  Even though I don’t write code on a regular basis, I am happy to see continued investment in the languages SAS provides—among the most recent, the addition of an algebraic optimization modeling language in our SAS/OR module contained within the SAS language as “PROC OPTMODEL.”

I have great respect for people who invest in learning (or even getting exposure to) more than one language and who appreciate the strengths of different languages for certain tasks and applications.

Ajay- It is great to see passionate people at work on both sides of the open source as well as packaged software teams- and even better for them to collaborate once in a while.Most of our work is based on scientists who came before us (especially in math theory).

Ultimately we are all just students of science anyway.

SAS Global Forum –http://support.sas.com/events/sasglobalforum/2009/

Annual event of SAS language practitioners.SAS language consists of data step and proc steps for input and output thus simplifying syntax for users.

SAS Institute – The leader of analytics software since 1970’s , it grew out of the North Carolina University, and provides jobs to thousands of people. The world’s largest privately held company, admired for it’s huge investments in Research and Development and criticized for its premium price  on packaged software solutions.A recent entrant in corporate users who are willing to support R language.

Weathering the Stormy Economy

Here is a conference you may want to visit. At first glance it may look like one of those self-help “free” webinars but it is a very relevant topic with a great speaker. Plus it is on the web.

 

Free Seminar Hosted by SAP Business Objects
Thursday, March 12, 2009
11 a.m. PST / 2 p.m. EST
Robin Fray Carey, CEO of Social Media Today will discuss the best ideas gathered from MyVenturePad.com, SMT’s online community for growth companies. Plus, two fast-growing companies, Fresh Direct, and The Life is good Company, will share their practical recommendations on how to manage business and IT priorities in these challenging times. Register today.
http://events.businessobjects.com/forms/Q109/ideas/?source=SMtoday1

Social Media Today builds Wordframe based communities like Smart Data Collective ( for data ,BI,Analytics people) Best of the Blogs ( for progressive bloggers), Energy Collective ( for Green energy enthusiasts,thinkers and researchers) ,Social Media Today ( for understanding and leveraging Social Media and Networks) and My VenturePad (for Entrepreneurs).

These communities basically work as online newspapers by aggregating and moderating the RSS feeds of thousands of bloggers (for some sites) and their sites. I have written on Wordframe’s concept of content driven communities and Ning’s concept of community driven content earlier.

Disclaimer- I have worked as an evangelist to SMT , have been awarded the Blogger of the Week once (for my article on R).

For other conferences you may also want to see AnalyticBridge ‘s page on conferences.

http://www.analyticbridge.com/group/conferences/forum/topics/best-ideas-for-weathering-the

Disclaimer -I have been awarded the Member of the Month twice by them.

I like the third party apps of Ning better than the old outdated format and themes. One Ning application can actually serve as a competitor to Wordframe – that is the RSS application ( see feed on my page ).Wordframe has capabilities for even category level filters so Analytics category  feed goes to Smart Data ,Internet category feed gets published on Social Media (when i am lucky) and my attempts at poetry go to Best of The Blogs.

The Decision Stats group (on Linkedin) also has a group on AnalyticBridge.

But why join so many communities and go to webinars ? Because knowledge is useful and productive and fun – and I have a personal motto of learning one new thing a day .

Where do you get the time ?Just sleep one hour less and devote that one hour purely to your self learning for yourself.8 hours to the boss, 4-5 hours to the family.

1 hour to yourself ??

Sounds reasonable, eh  🙂

So try this one –

http://events.businessobjects.com/forms/Q109/ideas/?source=SMtoday1

Interview – Anne Milley, SAS Part 1

Anne Milley has been a part of SAS Institute’s core strategy team.

She was in the news recently with an article by the legendary Ashlee Vance in the Bits Blog of  New York Times http://bits.blogs.nytimes.com/2009/02/16/sas-warms-to-open-source-one-letter-at-a-time/

In the article,  Ms. Milley said, “I think it addresses a niche market for high-end data analysts that want free, readily available code. We have customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet.”

To her credit, Ms. Milley addressed some of the critical comments head-on in a subsequent blog post.

This sparked my curiosity in knowing Anne ,and her perspective more than just a single line quote and here is an interview. This is part 1 of the interview . Anne_Milley

Ajay -Describe your career journey , both out of and in SAS Institute. What advice would you give to young high school students to pursue careers in science. Do you think careers in science are as rewarding as other careers.

Anne-

Originally, I wanted to major in international business to leverage my German (which is now waning from lack of use!).  I found the marketing and management classes at the time provided little practical value and happily ended up switching to the college of social science in the economics department, where I was challenged with several quantitative courses and encouraged to always have an analytical perspective.  In school, I was exposed to BASIC, SPSS, SHAZAM, and SAS.  Once I began my thesis (bank failure prediction models and the term structure of interest rates) and started working, it was SAS that served as the best software investment, both in banking (Federal Home Loan Bank of Dallas) and in retail (7-Eleven Corp.).  After 5+ years in Dallas, my husband wanted to move back to New England and SAS happened to be opening an office at the time.  From there, I enjoyed a few years as a pre-sales technical consultant, many years in analytical product management, and most recently in product marketing.  All the while, it has been a great motivating factor to work with so many talented people focused on solving problems, revealing opportunities and doing things better—both within and outside of SAS.

For high school and college students, I urge them to invest in studying some math and science, no matter the career they’re pursuing.  Whether they are interested in banking/finance, medicine and the life sciences, engineering or other fields, courses that will help them explore and analyze data, and come up with new approaches, new solutions, new advances based on a more scientific approach will pay off.

Course work in statistics, operations research, computer science and others will help hone skills for today’s data- and analytics-driven world.  One example of this idea in action:  North Carolina State University’s (NCSU) Institute for Advanced Analytics is seeing a huge increase in interest.  Its first graduating class last year saw higher average salaries than other graduate programs and multiple job offers per graduate.  Why?  Because there is still a huge demand for graduates with the ability to manipulate and analyze data in order to make better, more informed decisions.  I personally think careers in math and science are especially rewarding, but we need many diverse skills to make the world go round :o)

Ajay- Big corporations versus Startups. Where do you think is the balance between being big in terms of stability and size and being swift and nimble in terms of speed of product roll outs. What are the advantages and disadvantages of being a big corporation in a fast changing technology field.

Anne-

Ever a balancing act, with continuous learning along the way.  The advantage of being big (and privately held) is that you can be more long-term-oriented.  The challenge with fast-changing technology is to know where to best invest.  While others may go to market faster with new capabilities, we seek to provide superior implementations (we invest in ‘R’ (Research) AND ‘D’ (Development), making capabilities available on a number of platforms. 

In today’s economy, I think the big vs. small comparison is becoming less and less relevant.  Big corporations need to be agile and innovative, like their smaller rivals.  And small- to medium-sized businesses (SMBs) need to use the same techniques and technologies as the “big boys.”

First, on the big side, I’ll use an example of which I’m very familiar:  At SAS, a company founded more than 30 years ago as an entrepreneurial venture, we’ve certainly changed over the decades.  SAS started out in a small office with a handful of people.  It’s now a global company with hundreds of offices and thousands of employees around the world.  Yet one thing that has not changed for SAS in all this time:  a laser-like focus on the customer.  This has been the key to SAS’ success and uninterrupted growth .Not really a “secret sauce.” Just a simple yet profound approach: listen carefully to your customers and their changing needs, and innovate, develop and adapt based on these needs.

Of course, being large has its advantages:  we have more ideas from more people, and creativity and innovation knows no borders.  From Sydney to Warsaw, São Paulo to Singapore, Shanghai to Heidelberg, SAS employees work closely with customers to meet their business needs today and in the future.

SAS provides the stability and proven success that businesses look for, particularly in troubled economic times.  Being large and privately held enables SAS to grow when others are cutting back, and continue to invest in R&D at a high rate – 22% of revenues in 2008.

Yet with our annual subscription licensing model, SAS cannot rest on its laurels.  Each year, customers vote with their checkbooks:  if SAS provided them with business benefits, results and a positive ROI, they renew; if not, they can walk away.  Happily for SAS, the overwhelming majority of customers keep coming back.  But the licensing model keeps SAS on its toes, customer-focused, and always listening and innovating based on customer feedback.

As for SMBs, they are rapidly adopting the technologies used by large companies – such as business analytics – to compete in the global economy.  Two examples of this:

BGF Industries is a manufacturer of high-tech fabrics used in jet fighters, bullet-proof vests, movie-theater screens and surfboards, based in Greensboro, NC. BGF turned to SAS business analytics to help it deal with foreign competition.  BGF created a cost-effective, easy-to-use early-warning system that helps it track quality and productivity.  Per BGF, data is now available in minutes instead of hours.  And in the business world, this speed can be the difference between success and failure.  Per Bobby Hull, a BGF systems analyst: “The early-warning system we built with SAS allowed us to go from nothing to everything.  SAS allows us to focus away from clerical tasks to focus on the quality and process side of the job. Because of SAS, we’re never more than three clicks away from finding an answer.”

For Los Angeles-based The Wine House, installing a SAS-powered
inventory-management system helped it discover nearly $400,000 in “lost” inventory sitting on warehouse shelves.  For an SMB with annual sales of $20 million, that was a major find.  Business analytics helps it to compete with major retail and grocery chains.  Per Bill Knight, owner of The Wine House: “The first day the SAS application was live, we identified approximately 1,000 cases of wine that had not moved in over a year. That’s significant cash tied up in inventory.  We had a huge sale to blow it out, and just in time, because in today’s economy, we would be choking on that inventory.”

So regardless of size, businesses must remain agile, listen to their customers, and use technologies like business analytics to make sense of and derive value from their data – whether on the quality of surfboard covers or the number of cases of Oregon Pinot Noir in stock.

3) SAS Institute has been the de-facto leader in both market volume share as well as market value share in the field of data analytics. What are some of the factors do you think have contributed to this enduring success. What have been the principal challengers over the years.(Any comments on the challenge from SAS language software WPS please ??)

At SAS, we seek to provide a complete environment for analytics—from data collection, data manipulation, data exploration, data analysis, deployment of results – and the means to manage that whole process.  Competition comes in many forms and it pushes us to keep delivering value.  For me, one thing that sets SAS apart from other vendors is that we care so deeply about the quality of results.  Our Technical Support, Education and consulting services organizations really do partner with customers to help them achieve the best results.  That kind of commitment is deep in the DNA of SAS’ culture.

The good thing about competition is that it forces you to re-examine your value proposition and rethink your business strategy.  Customers value attributes of their analytics infrastructure in varying degrees— speed, quality, support, flexibility, ease of migration, backward and forward compatibility, etc.  Often there are options to trump any one or a subset of these and when that aligns with the customers’ priorities of what they value, they will vote with their pocketbooks.  For some customers with tight batch-processing windows, speed trumps everything.  In tests conducted by Merrill Consultants, an MXG program running on WPS runs significantly longer, consumes more CPU time and requires more memory than the same MXG program hosted on its native SAS platform.

While it’s easy to get caught up in fast-changing technology, one has to also consider history.  Some programming languages come and go; others have stood the test of time.  Even the use of different flavors of analysis ebbs and flows.  For instance, when data mining was all the rage almost a decade ago, many asked the very good question, “Why so much excitement about analyzing so much opportunistic data when design of experiments offers so much more?”  Finally, experimental design is being more readily adopted in areas like marketing.

At the end of the day, innovation is the only sustainable competitive advantage.  As noted above in question 2, SAS has remained firmly committed to customer-driven innovation.  And SAS has “stuck to its knitting” with respect to analytics.  A while back, SAS used to stand for “Statistical Analysis System.” If not literally, then philosophically, Analytics remains our middle name.

(Ajay- to be continued)