Category: Analytics

Images to Data :OCR Softwares

Some softwares I have used to convert images into text and even tables are –

 http://code.google.com/p/ocropus/

and http://code.google.com/p/tesseract-ocr/

Note- Both are open source , funded by Google   , who uses OCR to enhance search as well as for its book scanning project and these softwares are greatly helpful for say email marketing or converting images into rows and columns of text.

An additional open source imaging software is from http://www.leptonica.com/

You may need to tweak the resolution a bit and the highlighted scan area in order to get good results and can thus convert images into text and numeric data with a simple desktop scanner.

For higher end needs like production environments for questionnaires and responses-

The following come from the SPSS X List – ( it’s a nice list with many business problems that are also familiar in the SAS and R lists)

The software that SPSS recommends is ReadSoft (http://www.readsoft.com/).  Additionally, SPSS have a couple of complimentary products mrPaper (http://www.spss.com/mrpaper/) and mrScan (http://www.spss.com/mrScan/).

and from Dr Steven Lars in the same list

The high 90%+ accuracy of OCR technology of modern scanning, Remarks OMR (optical mark recognition) algorithms produce 99.9%+ accuracy for detected full or empty closed circles and squares.  It worked very well .

Scanning has been accomplished using upper end, hobby grade scanners with automatic form-feed options driven by Windows software.  Though these machines were purchased through university purchasing, all could have been purchased at Best Buy ( discount technology stores), a comparable store, or at internet discount sources.

You can store data for research purposes, access the data using SPSS or Excel, and respond to counselor questions within a week of receiving the raw, paper surveys.

The current preference is to use web-based information gathering developed through university information technology resources or developed on www.Zoomerang.com ( or even www.surveymonkey.com ).  

The biggest challenges were:
You have to be careful in the production of our to-be-scanned forms.  Our forms had to be printed on the same copy machine from a set master or printed on the same laser printer to ensure accuracy.  Poor quality copies and printing yields inaccurate scanning.  Survey color also has to be managed carefully, as some colors are opaque to some optical scanners.
 

A link to the publishers of Remark OMR:
http://www.gravic.com/remark/officeomr/index.html?gclid=CJj01MTSiZgCFRxNagodd0_IDQ

The SAS-L Rookie of the Year

Well I have been told, I am on the SAS-L rookie of the year list at http://www.listserv.uga.edu/cgi-bin/wa?A1=ind0901b&L=sas-l#29.

With 351 posts in 2008 and 0 in 2007, you can certainly say I have been an active rocky, I mean rookie on the list.

Some the things I did were-

  • Share experiences with SAS language code including Automation
  • Share and ask on non –SAS software areas like Google Docs, Cloud Computing ,WPS comparisons.
  • Provoke by design and mostly by accident discussion on R , SAS Software Pricing, relationship and dependence between SAS Community .Org and the SAS Institute and diversity and international issues on the list.

I believe SAS is a good software and the SAS institute has been a pioneer, and it needs to listen to feedback from its retail customers just as much it needs to make money.

  1. A more transparent way of announcing strategic intent on where they are going to concentrate research and
  2. maybe a more nuanced public  relationship stance on rival softwares , with
  3. a readiness to once again experiment if not embrace open source contributions (particularly by using some interface to R code, as well as R datasets) could lead to great stuff from SAS Institute again.

p.s. I Don’t expect to win though. I am bad at elections.

Top ten RRReasons R is bad for you ?

 

This is the original symbol of the Perl progra...
Image via Wikipedia

 

R stands for programming language based out of www.r-project.org

R is bad for you because –

1) It is slower with bigger datasets than SPSS language and SAS language .If you use bigger datasets, then you should either consider more hardware , or try and wait for some of the ODBC connect packages.

2) It needs more time to learn than SAS language .Much more time to learn how to do much more.

3) R programmers are lesser paid than SAS programmers.They prefer it that way.It equates the satisfaction of creating a package in development with a world wide community with the satisfaction of using a package and earning much more money per hour.

4) It forces you to learn the exact details of what you are doing due to its object oriented structure. Thus you either get no answer or get an exact answer. Your customer pays you by the hour not by the correct answers.

5) You can not push a couple of buttons or refer to a list of top ten most commonly used commands to finish the project.

6) It is free. And open for all. It is socialism expressed in code. Some of the packages are built by university professors. It is free.Free is bad. Who pays for the mortgage of the software programmers if all softwares were free ? Who pays for the Friday picnics. Who pays for the Good Night cruises?

7) It is free. Your organization will not commend you for saving them money- they will question why you did not recommend this before. And why did you approve all those packages that expire in 2011.R is fReeeeee. Customers feel good while spending money.The more software budgets you approve the more your salary is. R thReatens all that.

8) It is impossible to install a package you do not need or want. There is no one calling you on the phone to consider one more package or solution. R can make you lonely.

9) R uses mostly Command line. Command line is from the Seventies. Or the Eighties. The GUI’s RCmdr and Rattle are there but still…..

10) R forces you to learn new stuff by the month. You prefer to only earn by the month. Till the day your job got offshored…

Written by a R user in English language

( which fortunately was not copyrighted otherwise we would be paying Britain for each word)

Related Articles

the above post was reprinted by request.

Using R and Excel Together

I put up a question to the R list on using VBA macros from within excel. It seems you can use R from within Excel and can customize it so that the end user doesnot know R. It is called RExcel (what else !)

Quoting Erich from R archives ”
There is RExcel (available by downloading the CRAN package RExcelInstaller. It allows to transfer data between R and Excel, and run R code from within Excel. So you can start with your data in Excel, let R do an analysis, and transfer the results back to Excel. You can write VBA macros which do this, but “hidden from exposure”,
so the Excel user does not even notice that R is doing the hard work.

It also has an Excel worksheet function RApply which allows to call an R function from an Excel cell formula. =RApply(”rfun”,A1)
would apply the R function rfun to the value in cell A1.
If the value in A1 changes, Excel will force R to recalculate the formula.

There is a (half hour long) video demo about RExcel
at http://rcom.univie.ac.at/RExcelDemo/

http://rcom.univie.ac.at/ has more information about the project

 

 

This can help save a huge number of costs as Excel is the least expensive analytical software and is present on all analytics companies.

 

More news on R here http://bits.blogs.nytimes.com/2009/01/08/r-you-ready-for-r/

Yahoo Finance for Analysis-Indian ITES

A great graphical tool is Yahoo finance- As a test case I took the Indian Outsourcing and IT industry and compared their stock performance for the whole of 2008.

Note I am using the website http://www.tiny.cc/ which has a great 10 sec method of shortening urls, just like www.tinyurl.com

You can view the graph here http://tiny.cc/ITES and add in last year’s value including SIFY (Satyam’s arm here http://tiny.cc/INDIA

Business case- As you can see, the Indian IT sector lost tremendous value in 2008 , perhaps too much so you can see the sharp pullbacks in the last one week ( WNS pulled almost 18 % up). One view I belive this industry will do well to adopt is stop being an Information Technology Enabled Services (ITES) and start moving towards Information Technology Enabled Products ( ITEP)

Updated- I didnt factor in Satyam’s acounting scandal in the analysis.If you click the link above now – you would note the impact such events have on domains ( and curiously happen again and again.It also leads to better data mining solutions than simplistic history repeats itself models).

Yahoo finance is extremely user friendly and the design of the interface is sweet,short and simple. But the biggest weakness is ( and it seems like a Yahoo weakness) – there are almost no ads compared to Google Finance which focuses on the ad placement and relevancy over anything else. I hope this site endures ..as it is a great boon for analysts and even stock market researchers.

image

Interview: Roger Haddad, Founder of KXEN Automated Modeling Software

KXEN_logo_300dpi I first talked about KXEN,the automated modeling software  in this post http://www.decisionstats.com/2008/12/automating-regression-models-kxen/

So I asked Roger Haddad ,its founder and CEO if he could give an email interview and Roger being the great guy he is , both remembered me from user analyst days as well worked in the holidays to give this interview. Before founding KXEN, Mr. Roger Haddad was president of Azlan France, the first network distributor in France. Under his management, sales revenues increased 850 percent in four years.In 1977, Mr. Haddad founded Metrologie International, a leading software and hardware provider, which went public on the Paris Stock Exchange in 1985. When Mr. Haddad left in 1991, Metrologie had 4,500 employees, $900M in revenue and subsidiaries in 13 European countries.Mr. Haddad holds a master’s degree in electrical metrology from George Washington University and a bachelor’s degree in electrical engineering from Ecole Supérieure d’Electricité, Paris, France.

 

Ajay : What would your advice be to young professionals entering the job world today ?

Roger : If you are talking about Statisticians, I would tell them to concentrate on the data and the process rather than on the statistical orthodoxy

Ajay : What interested you most in being the head of KXEN. What is the best feature you like in KXEN. – both as a company and as a product.


Roger :To Make it happen !! Data mining is at its infancy, because SAS and others made it difficult to work with !! they made for an elite of people !!

KXEN role is to open this bottleneck and give power to the users – Analysts will help to train business users and get them confident with their findings.

As a product, I am always suprised by the quality of KXEN results in a fraction of the time compared to first generation workbench and automatically !! 🙂

Ajay : What areas has KXEN been most suitable for ? Biggest sucess story so far.

Roger : Classification, regression with thousands of variables and tricky data sets !! We have hundreds of success stories 

Ajay :Could you also comment on how the slowdown and recession would affect the analytics world in terms of newer solutions , Software as a service , more acceptance of trying out the unfamiliar etc ?

Roger : I believe the recession and the slowdown will push analytics further and particularly KXEN approach , since we allow corporations to do much more with less or with the same Team. We are seeing many Analytics Group being reduced and people calling on us to deliver what need to be delivered!!

Ajay : What areas would you rather not recommend KXEN? What other softwares would you recommend in those cases ?

Roger : I would not recommend KXEN in genetics – SVM would be more apropriate

Ajay : Asia has a nascent but high potential market. What are you Asian plans and any clients /case studies here ?

Roger:  We have a presence in every countries but in India – Japan is by far our best country and we have there a fantastic Distributor -WE also have Customers in China , in Asean too – We are looking for a good Distributor in India , but this seems quite difficult.(Note from Ajay – I decided to apply straight away)

Ajay : What is the biggest challenge you have faced while introducing KXEN to a wider audience.
Roger: THe resistance to Change and the fear of classical statisticians that they will loose their job !! in fact this never happened and on  the contrary they become hero in their Corporation after adopting KXEN

  Roger Haddad 1Founder and Chief Executive Officer

Mr. Haddad is responsible for overseeing the KXEN sales team, the distribution channel management, as well as the direction of the company and the strategic growth of the organization. With more than 30 years experience as an industry expert, Mr. Haddad is a forward-looking entrepreneur with an expertise in successfully running companies with multiple channels
of distribution. Mr. Haddad has a long and successful track record in developing new companies into profitable enterprises.

Smart Data Collective

Here is a great online community for decision scientists especially the ones reading these posts. It is called Smart Data Collective and it is hosted at www.smartdatacollective.com

It works like a blog aggregator for specific topics and the quality of posts is quite nice. Just like your blogroll does for you and it can also help your blog, if you write to get some added views. Basically it works as a newpaper for data topics from featured posts by data bloggers. I joined it just over the year end , and  was able to both write and read on nice topics.

 

Some Self Promotion by humbly yours

And also, if you didnt notice , we had a revamped website at www.decisionstats.com

Improvements include a better SEO url structure, new WordPress theme (disliked by many due to blue grey color) ,enhanced blogroll with labels, RSS feeds from NYT and DM Review besides my own poetry blog, and revamping the old pages. Let me know what you think about the change.