Tag: why

Web Analytics using R , Google Analytics and TS Forecasting

This is a continuation of the previous post on using Google Analytics .

Now that we have downloaded and plotted the data- we try and fit time series to the website data to forecast future traffic.

Some observations-

1) Google Analytics has 0 predictive analytics, it is just descriptive analytics and data visualization models (including the recent social analytics). However you can very well add in basic TS function using R to the GA API.

Why do people look at Website Analytics? To know today’s traffic and derive insights for the Future

2) Web Data clearly follows a 7 day peak and trough for weekly effects (weekdays and weekends), this is also true for hourly data …and this can be used for smoothing historic web data for future forecast.

3) On an advanced level, any hugely popular viral posts can be called a level shift (not drift) and accoringly dampened.

Test and Control!

Similarly using ARIMAX, we can factor in quantity and tag of posts as X regressor variables.

and now the code-( dont laugh at the simplicity please, I am just tinkering and playing with data here!)

You need to copy and paste the code at the bottom of   this post  http://www.decisionstats.com/using-google-analytics-with-r/ if you want to download your GA data down first.

Note I am using lubridate ,forecast and timeSeries packages in this section.

#Plotting the Traffic  plot(ga.data$data[,2],type="l") 

library(timeSeries)
library(forecast)

#Using package lubridate to convert character dates into time
library(lubridate)
ga.data$data[,1]=ymd(ga.data$data[,1])
ls()
dataset1=ga.data$data
names(dataset1) <- make.names(names(dataset1))
str(dataset1)
head(dataset1)
dataset2 <- ts(dataset1$ga.visitors,start=0,frequency = frequency(dataset1$ga.visitors), names=dataset1$ga.date)
str(dataset2)
head(dataset2)
ts.test=dataset2[1:200]
ts.control=dataset2[201:275]

 #Note I am splitting the data into test and control here

fitets=ets(ts.test)
plot(fitets)
testets=ets(ts.control,model=fitets)
accuracy(testets)
plot(testets)
spectrum(ts.test,method='ar')
decompose(ts.test)

library("TTR")
bb=SMA(dataset2,n=7)#We are doing a simple moving average for every 7 days. Note this can be 24 hrs for hourly data, or 30 days for daily data for month # 

to month comparison or 12 months for annual
#We notice that Web Analytics needs sommethening for every 7 thday as there is some relation to traffic on weekedays /weekends /same time last week
head(dataset2,40)
head(bb,40)

par(mfrow=c(2,1))
plot(bb,type="l",main="Using Seven Day Moving Average for Web Visitors")
plot(dataset2,main="Original Data")

Created by Pretty R at inside-R.org

Though I still wonder why the R query, gA R code /package could not be on the cloud (why it  needs to be downloaded)– cloud computing Gs?

Also how about adding some MORE predictive analytics to Google Analytics, chaps!

To be continued-

auto.arima() and forecasts!!!

cross validations!!!

and adapting the idiosyncratic periods and cycles  of web analytics to time series !!

iPhones Siri to replace call centers . Is it doable

I was wondering why the planet  spends  so much money in the $150-billion business process outsourcing industry, especially in voice calls to call centers.

If your iPhone Siri phone can be configured to answer any query, Why can’t it be configured to be  a virtual assistant, customer support, marketing outbound or even a super charged call center interactive voice response .

Can we  do and run some tests on this?

Using Google Adwords to target Vic Gundotra and Matt Cutts stochastically

Over the Christmas break, I created a Google Adwords campaign using the $100 credit generously given by Google. I did it using my alumni id, even though I have a perfectly normal gmail id. I guess if Google allows me to use the credit on any account- well I will take it. and so a free experiment was borne.

But whom to target -with Google- but Google itself. It seemed logical

So I created a campaign for the names of prominent Googlers  (from a list of Google + at https://plus.google.com/103399926392582289066/posts/LX4g7577DqD ) and limited the ad location to Mountain View, California.

NULL HYPOTHESIS- People who are googled a lot from within the office are either popular or just checking themselves.

Since Google’s privacy policy is great, has been now shown billions of times, well I guess what’s a little ad targetting between brother geeks. Right?

My ad was-

Hire Ajay Ohri
He is
Awesome
linkedin.com/in/ajayohri 

or see screenshot below.

Here are the results-88 clicks and 43000 impressions (and 83$ of Google’s own money)

clearly Vic Gundotra is googled a lot within Mountain View, California. Does He Google himself.

so is Matt Cutts. Does HE Google himself or does he get elves to help him.

to my disappointment not many people clicked my LI offer, I am still blogging

and there were few clicks on Marissa Myers. Why Google her when she is right down the corridor.

The null hypothesis is thus rejected. Also most clicks were from display and not from search.

I need to do something better to do with Christmas break this year. I still got a credit of 16$ left.

 

Why the West needs China to start moving towards cyber conflict

Hypothesis-

Western countries are running out of people to fight their wars. This is even more acute given the traditional and current demographic trends in both armed forces and general populations.

A shift to cyber conflict can help the West maintain parity over Eastern methods of assymetrical warfare (by human attrition /cyber conflict).

Declining resources will lead to converging conflicts of interest and dynamics in balance of power in the 21 st century.

Assumed Facts–

The launch of Sputnik by USSR led to the moon shot rush by the US.1960s

The proposed announcement of StarWars by USA led to unsustainable defence expenditure by USSR.1980s

The threat of cyber conflict and espionage by China (and Russian cyber actions in war with Georgia) has led to increasing budgets for cyber conflict research and defense in USA. -2010s

Assumptions–

If we do not learn from history, we are condemned to repeat it.

Declining Populations in the West and Rising Populations in the East in the 21 st century. The difference in military age personnel would be even more severe, due to more rapid aging in the west.

Economic output will be proportional to number of people employed as economies reach similar stages of maturity (Factor-Manufacturing-Services-Innovation)

Data-

http://esa.un.org/unpd/wpp/unpp/panel_population.htm

http://www.census.gov/population/international/

GDP projections to 2050 graphicView larger picture

GDP projections to 2050:
Summary-
Western defence forces would not be able to afford a human attrition intensive war by 2030 given current demographic trends  (both growth and aging). Existing balance of power could be maintained if resources are either shared or warfare is moved to cyber space. Technological advances can help augment resources reducing case for conflict scenarios.
Will the Internet be used by US against China in the 21 st century as Opium was used by GB in the 19th? Time will tell 🙂

Interview Michal Kosinski , Concerto Web Based App using #Rstats

Here is an interview with Michal Kosinski , leader of the team that has created Concerto – a web based application using R. What is Concerto? As per http://www.psychometrics.cam.ac.uk/page/300/concerto-testing-platform.htm

Concerto is a web based, adaptive testing platform for creating and running rich, dynamic tests. It combines the flexibility of HTML presentation with the computing power of the R language, and the safety and performance of the MySQL database. It’s totally free for commercial and academic use, and it’s open source

Ajay-  Describe your career in science from high school to this point. What are the various stats platforms you have trained on- and what do you think about their comparative advantages and disadvantages?  

Michal- I started with maths, but quickly realized that I prefer social sciences – thus after one year, I switched to a psychology major and obtained my MSc in Social Psychology with a specialization in Consumer Behaviour. At that time I was mostly using SPSS – as it was the only statistical package that was taught to students in my department. Also, it was not too bad for small samples and the rather basic analyses I was performing at that time.

 

My more recent research performed during my Mphil course in Psychometrics at Cambridge University followed by my current PhD project in social networks and research work at Microsoft Research, requires significantly more powerful tools. Initially, I tried to squeeze as much as possible from SPSS/PASW by mastering the syntax language. SPSS was all I knew, though I reached its limits pretty quickly and was forced to switch to R. It was a pretty dreary experience at the start, switching from an unwieldy but familiar environment into an unwelcoming command line interface, but I’ve quickly realized how empowering and convenient this tool was.

 

I believe that a course in R should be obligatory for all students that are likely to come close to any data analysis in their careers. It is really empowering – once you got the basics you have the potential to use virtually any method there is, and automate most tasks related to analysing and processing data. It is also free and open-source – so you can use it wherever you work. Finally, it enables you to quickly and seamlessly migrate to other powerful environments such as Matlab, C, or Python.

Ajay- What was the motivation behind building Concerto?

Michal- We deal with a lot of online projects at the Psychometrics Centre – one of them attracted more than 7 million unique participants. We needed a powerful tool that would allow researchers and practitioners to conveniently build and deliver online tests.

Also, our relationships with the website designers and software engineers that worked on developing our tests were rather difficult. We had trouble successfully explaining our needs, each little change was implemented with a delay and at significant cost. Not to mention the difficulties with embedding some more advanced methods (such as adaptive testing) in our tests.

So we created a tool allowing us, psychometricians, to easily develop psychometric tests from scratch an publish them online. And all this without having to hire software developers.

Ajay -Why did you choose R as the background for Concerto? What other languages and platforms did you consider. Apart from Concerto, how else do you utilize R in your center, department and University?

Michal- R was a natural choice as it is open-source, free, and nicely integrates with a server environment. Also, we believe that it is becoming a universal statistical and data processing language in science. We put increasing emphasis on teaching R to our students and we hope that it will replace SPSS/PASW as a default statistical tool for social scientists.

Ajay -What all can Concerto do besides a computer adaptive test?

Michal- We did not plan it initially, but Concerto turned out to be extremely flexible. In a nutshell, it is a web interface to R engine with a built-in MySQL database and easy-to-use developer panel. It can be installed on both Windows and Unix systems and used over the network or locally.

Effectively, it can be used to build any kind of web application that requires a powerful and quickly deployable statistical engine. For instance, I envision an easy to use website (that could look a bit like SPSS) allowing students to analyse their data using a web browser alone (learning the underlying R code simultaneously). Also, the authors of R libraries (or anyone else) could use Concerto to build user-friendly web interfaces to their methods.

Finally, Concerto can be conveniently used to build simple non-adaptive tests and questionnaires. It might seem to be slightly less intuitive at first than popular questionnaire services (such us my favourite Survey Monkey), but has virtually unlimited flexibility when it comes to item format, test flow, feedback options, etc. Also, it’s free.

Ajay- How do you see the cloud computing paradigm growing? Do you think browser based computation is here to stay?

Michal – I believe that cloud infrastructure is the future. Dynamically sharing computational and network resources between online service providers has a great competitive advantage over traditional strategies to deal with network infrastructure. I am sure the security concerns will be resolved soon, finishing the transformation of the network infrastructure as we know it. On the other hand, however, I do not see a reason why client-side (or browser) processing of the information should cease to exist – I rather think that the border between the cloud and personal or local computer will continually dissolve.

About

Michal Kosinski is Director of Operations for The Psychometrics Centre and Leader of the e-Psychometrics Unit. He is also a research advisor to the Online Services and Advertising group at the Microsoft Research Cambridge, and a visiting lecturer at the Department of Mathematics in the University of Namur, Belgium. You can read more about him at http://www.michalkosinski.com/

You can read more about Concerto at http://code.google.com/p/concerto-platform/ and http://www.psychometrics.cam.ac.uk/page/300/concerto-testing-platform.htm

Why LinkedIn and Twitter are up for grabs in 2012-14?

Given Facebook’s valuation at $60-$100 billion , Apple’s $100 billion cash pile, Microsoft’s cash of $ 52 billion, Google’s cash of 43 billion $ , there is a lot of money floating. I am not counting Amazon as it deals with its own Fire issues.

But what is left to buy. In terms of richness of data available for data mining for better advertising- it is Twitter and LinkedIn that have the best sources of data.

and LinkedIn is worth only 9 billion dollars and Twitter is only $8.5 billion dollars. Throw in a competitive dynamic  premium, and you can get 50 % of both these companies at 13 billion dollars. if owners dont want to sell 100%, well buy a big big stake.

Makes a good case- buy the company- buy the data- sell them ads- sell them better products.

What do you think?

Stanford Courses Delayed Again

Message from the guys at Palo Alto— Why dont they just make videos using Sal Academy’s help?

We’re sorry to have to tell you that our Machine Learning course will be delayed further. There have naturally been legal and administrative issues to be sorted out in offering Stanford classes freely to the outside world, and it’s just been taking time. We have, however, been able to take advantage of the extra time to debug and improve our course content!

We now expect that the course will start either late in February or early in March. We will let you know as soon as we hear a definite date. We apologize for the lack of communication in recent weeks; we kept hoping we would have a concrete launch date to give you, but that date has kept slipping.

Thanks so much for your patience! We are really sorry for repeatedly making you wait, and for any interference this causes in your schedules. We’re as excited and anxious as you are to get started, and we both look forward to your joining us soon in Machine Learning!

Andrew Ng and the ML Course Staff