Tag: usa

Using Rapid Miner and R for Sports Analytics #rstats

Rapid Miner has been one of the oldest open source analytics software, long long before open source or even analytics was considered a fashion buzzword. The Rapid Miner software has been a pioneer in many areas (like establishing a marketplace for Rapid Miner Extensions) and the Rapid Miner -R extension was one of the most promising enablers of using R in an enterprise setting.
The following interview was taken with a manager of analytics for a sports organization. The sports organization considers analytics as a strategic differentiator , hence the name is confidential. No part of the interview has been edited or manipulated.

Ajay- Why did you choose Rapid Miner and R? What were the other software alternatives you considered and discarded?

Analyst- We considered most of the other major players in statistics/data mining or enterprise BI.  However, we found that the value proposition for an open source solution was too compelling to justify the premium pricing that the commercial solutions would have required.  The widespread adoption of R and the variety of packages and algorithms available for it, made it an easy choice.  We liked RapidMiner as a way to design structured, repeatable processes, and the ability to optimize learner parameters in a systematic way.  It also handled large data sets better than R on 32-bit Windows did.  The GUI, particularly when 5.0 was released, made it more usable than R for analysts who weren’t experienced programmers.

Ajay- What analytics do you do think Rapid Miner and R are best suited for?

 Analyst- We use RM+R mainly for sports analysis so far, rather than for more traditional business applications.  It has been quite suitable for that, and I can easily see how it would be used for other types of applications.

 Ajay- Any experiences as an enterprise customer? How was the installation process? How good is the enterprise level support?

Analyst- Rapid-I has been one of the most responsive tech companies I’ve dealt with, either in my current role or with previous employers.  They are small enough to be able to respond quickly to requests, and in more than one case, have fixed a problem, or added a small feature we needed within a matter of days.  In other cases, we have contracted with them to add larger pieces of specific functionality we needed at reasonable consulting rates.  Those features are added to the mainline product, and become fully supported through regular channels.  The longer consulting projects have typically had a turnaround of just a few weeks.

 Ajay- What challenges if any did you face in executing a pure open source analytics bundle ?

Analyst- As Rapid-I is a smaller company based in Europe, the availability of training and consulting in the USA isn’t as extensive as for the major enterprise software players, and the time zone differences sometimes slow down the communications cycle.  There were times where we were the first customer to attempt a specific integration point in our technical environment, and with no prior experiences to fall back on, we had to work with Rapid-I to figure out how to do it.  Compared to the what traditional software vendors provide, both R and RM tend to have sparse, terse, occasionally incomplete documentation.  The situation is getting better, but still lags behind what the traditional enterprise software vendors provide.

 Ajay- What are the things you can do in R ,and what are the things you prefer to do in Rapid Miner (comparison for technical synergies)

Analyst- Our experience has been that RM is superior to R at writing and maintaining structured processes, better at handling larger amounts of data, and more flexible at fine-tuning model parameters automatically.  The biggest limitation we’ve had with RM compared to R is that R has a larger library of user-contributed packages for additional data mining algorithms.  Sometimes we opted to use R because RM hadn’t yet implemented a specific algorithm.  The introduction the R extension has allowed us to combine the strengths of both tools in a very logical and productive way.

In particular, extending RapidMiner with R helped address RM’s weakness in the breadth of algorithms, because it brings the entire R ecosystem into RM (similar to how Rapid-I implemented much of the Weka library early on in RM’s development).  Further, because the R user community releases packages that implement new techniques faster than the enterprise vendors can, this helps turn a potential weakness into a potential strength.  However, R packages tend to be of varying quality, and are more prone to go stale due to lack of support/bug fixes.  This depends heavily on the package’s maintainer and its prevalence of use in the R community.  So when RapidMiner has a learner with a native implementation, it’s usually better to use it than the R equivalent.

SAS and Hadoop

Awesomely informative post on sascom magazine (whose editor I have I interviewed before here at http://www.decisionstats.com/interview-alison-bolen-sas-com/ – )

Great piece by Michael Ames ,SAS Data Integration Product Manager.

http://www.sas.com/news/sascom/hadoop-tips.html

 

Also see SAS’s big data thingys here at

http://www.sas.com/software/high-performance-analytics/in-memory-analytics/index.html

Solutions and Capabilities Using SAS® In-Memory Analytics

  • High-Performance Analytics – Get near-real-time insights with appliance-ready analytics software designed to tackle big data and complex problems.
  • High-Performance Risk – Faster, better risk management decisions based on the most up-to-date views of your overall risk exposure.
  • High-Performance Liquidity Risk Management – Take quick, decisive actions to secure adequate funding, especially in times of volatility.
  • High-Performance Stress Testing – Make faster, more precise decisions to protect the health of the firm.
  • Visual Analytics – Explore big data using in-memory capabilities to better understand all of your data, discover new patterns and publish reports to the Web and iPad®.

(Ajay- I liked the Visual Analytics piece especially for Big Data )

Note-

 

Possible Digital Disruptions by Cyber Actors in USA Electoral Cycle

Some possible electronic disruptions  that threaten to disrupt the electoral cycle in United States of America currently underway is-

1) Limited Denial of Service Attacks (like for 5-8 minutes) on fund raising websites, trying to fly under the radar of network administrators to deny the targeted  fundraising website for a small percentage of funds . Money remains critical to the world’s most expensive political market. Even a 5% dropdown in online fund-raising capacity can cripple a candidate.

2)  Limited Man of the Middle  Attacks on ground volunteers to disrupt ,intercept and manipulate communication flows. Basically cyber attacks at vulnerable ground volunteers in critical counties /battleground /swing states (like Florida)

3) Electro-Magnetic Disruptions of Electronic Voting Machines in critical counties /swing states (like Florida) to either disrupt, manipulate or create an impression that some manipulation has been done.

4) Use search engine flooding (for search engine de-optimization of rival candidates keywords), and social media flooding for disrupting the listening capabilities of sentiment analysis.

5) Selected leaks (including using digital means to create authetntic, fake or edited collateral) timed to embarrass rivals or influence voters , this can be geo-coded and mass deployed.

6) using Internet communications to selectively spam or influence independent or opinionated voters through emails, short messaging service , chat channels, social media.

7) Disrupt the Hillary for President 2016 campaign by Anonymous-Wikileak sympathetic hacktivists.

 

 

Why the West needs China to start moving towards cyber conflict

Hypothesis-

Western countries are running out of people to fight their wars. This is even more acute given the traditional and current demographic trends in both armed forces and general populations.

A shift to cyber conflict can help the West maintain parity over Eastern methods of assymetrical warfare (by human attrition /cyber conflict).

Declining resources will lead to converging conflicts of interest and dynamics in balance of power in the 21 st century.

Assumed Facts–

The launch of Sputnik by USSR led to the moon shot rush by the US.1960s

The proposed announcement of StarWars by USA led to unsustainable defence expenditure by USSR.1980s

The threat of cyber conflict and espionage by China (and Russian cyber actions in war with Georgia) has led to increasing budgets for cyber conflict research and defense in USA. -2010s

Assumptions–

If we do not learn from history, we are condemned to repeat it.

Declining Populations in the West and Rising Populations in the East in the 21 st century. The difference in military age personnel would be even more severe, due to more rapid aging in the west.

Economic output will be proportional to number of people employed as economies reach similar stages of maturity (Factor-Manufacturing-Services-Innovation)

Data-

http://esa.un.org/unpd/wpp/unpp/panel_population.htm

http://www.census.gov/population/international/

GDP projections to 2050 graphicView larger picture

GDP projections to 2050:
Summary-
Western defence forces would not be able to afford a human attrition intensive war by 2030 given current demographic trends  (both growth and aging). Existing balance of power could be maintained if resources are either shared or warfare is moved to cyber space. Technological advances can help augment resources reducing case for conflict scenarios.
Will the Internet be used by US against China in the 21 st century as Opium was used by GB in the 19th? Time will tell 🙂

Funny Economics

Some wry observations from me  on the world on economics-

1) 150 years after humiliating their country in the Opium Wars, Chinese mandarins have somehow convinced their leaders and military to park 2 trillion assets in Anglo Saxon debt. If Greece geting a 50% discount on its loan is the new precedent, when will the USA force its lendors to the negotiation table.

2) Income inequality and protests are something the Arabs and Israelis have in common. Besides being the sons of Abraham of course. Note the Persians are not considered the same as Arabs.

3) Advance knowledge of geo-political events can and ensures Western financial dealers have an edge on the sovereign funds in the other hemisphere.  What used to be the playgrounds of Eton has now shifted to the pubs of Boston and So Cal.

4) After spending 1 trillion USD on arms in the past one decade (funded by guys in item 1), the United States military forces is in a much better more advanced position to wage simultaneous war.

5) Can a war in Korean peninsula affect war in the Persian sphere of influence. Just follow the money , baby.

6) Saudi Wahabis continue to fund terror despite losing a lot of money in the economic meltdown in past few years. For every 1 $ increase in Saudi oil revenue, western oil companies ,traders, financiers make more, much more.

7) Demographics is an important key to economics. An aging Japan, and stagnant West is one cause to shift from manpower intensive warfare to cyber warfare. Plus Cyber warfare is good business . Underpopulated Russia and Arabs continue to lack true economic potential.

8) There are new economic incentives to develop tools to disseminate as well as distort information flow in real time in a hyper connected digital world.

 

Credit Downgrade of USA and Triple A Whining

As a person trained , deployed and often asked to comment on macroeconomic shenanigans- I have the following observations to make on the downgrade of US Debt by S&P

1) Credit rating is both a mathematical exercise of debt versus net worth as well as intention to repay. Given the recent deadlock in United States legislature on debt ceiling, it is natural and correct to assume that holding US debt is slightly more risky in 2011 as compared to 2001. That means if the US debt was AAA in 2001 it sure is slightly more risky in 2011.

2) Politicians are criticized the world over in democracies including India, UK and US. This is natural , healthy and enforced by checks and balances by constitution of each country. At the time of writing this, there are protests in India on corruption, in UK on economic disparities, in US on debt vs tax vs spending, Israel on inflation. It is the maturity of the media as well as average educational level of citizenry that amplifies and inflames or dampens sentiment regarding policy and business.

3) Conspicuous consumption has failed both at an environmental and economic level. Cheap debt to buy things you do not need may have made good macro economic sense as long as the things were made by people locally but that is no longer the case. Outsourcing is not all evil, but it sure is not a perfect solution to economics and competitiveness. Outsourcing is good or outsourcing is bad- well it depends.

4) In 1944 , the US took debt to fight Nazism, build atomic power and generally wage a lot of war and lots of dual use inventions. In 2004-2010 the US took debt to fight wars in Iraq, Afghanistan and bail out banks and automobile companies. Some erosion in the values represented by a free democracy has taken place, much to the delight of authoritarian regimes (who have managed to survive Google and Facebook).

5) A Double A rating is still quite a good rating. Noone is moving out of the US Treasuries- I mean seriously what are your alternative financial resources to park your government or central bank assets, euro, gold, oil, rare earth futures, metals or yen??

6) Income disparity as a trigger for social unrest in UK, France and other parts is an ominous looming threat that may lead to more action than the poor maths of S &P. It has been some time since riots occured in the United States and I believe in time series and cycles especially given the rising Gini coefficients .

Gini indices for the United States at various times, according to the US Census Bureau:[8][9][10]

  • 1929: 45.0 (estimated)
  • 1947: 37.6 (estimated)
  • 1967: 39.7 (first year reported)
  • 1968: 38.6 (lowest index reported)
  • 1970: 39.4
  • 1980: 40.3
  • 1990: 42.8
    • (Recalculations made in 1992 added a significant upward shift for later values)
  • 2000: 46.2
  • 2005: 46.9
  • 2006: 47.0 (highest index reported)
  • 2007: 46.3
  • 2008: 46.69
  • 2009: 46.8

7) Again I am slightly suspicious of an American Corporation downgrading the American Governmental debt when it failed to reconcile numbers by 2 trillion and famously managed to avoid downgrading Lehman Brothers.  What are the political affiliations of the S &P board. What are their backgrounds. Check the facts, Watson.

The Chinese government should be concerned if it is holding >1000 tonnes of Gold and >1 trillion plus of US treasuries lest we have a third opium war (as either Gold or US Treasuries will burst)

. Opium in 1850 like the US Treasuries in 2010 have no inherent value except for those addicted to them.

8   ) Ron Paul and Paul Krugman are the two extremes of economic ideology in the US.

Reminds me of the old saying- Robbing Peter to pay Paul. Both the Pauls seem equally unhappy and biased.

I have to read both WSJ and NYT to make sense of what actually is happening in the US as opinionated journalism has managed to elbow out fact based journalism. Do we need analytics in journalism education/ reporting?

9) Panic buying and selling would lead to short term arbitrage positions. People like W Buffet made more money in the crash of 2008 than people did in the boom years of 2006-7

If stocks are cheap- buy. on the dips. Acquire companies before they go for IPOs. Go buy your own stock if you are sitting on  a pile of cash. Buy some technology patents in cloud , mobile, tablet and statistical computing if you have a lot of cash and need to buy some long term assets.

10) Follow all advice above at own risk and no liability to this author 😉

 

Hacking Hackers

This is a ten step program to fight hacking attacks. You may or may not choose to ignore it, laugh at it, or ponder on it.

1) Internet security is a billion dollar business which will only grow in size as cloud computing approaches. Pioneers in providing security will earn considerable revenue like McAffee  , Norton did in the PC era. Incidentally it also means the consulting/partner group that is willing to work with virtual workers and virtual payments to offshore consultants.

2) Industrial espionage has existed from the days the West stole Gunpowder and Silk formula from China (and China is now doing the same to its software). The company and country will the best hackers will win. Keep your team motivated mate, or it is very easy for them to defect to the other side of the (cyber) wall.

3) When 2 billion people have access to internet the number of hackers will grow in number and quality much more rapidly than when only 100 million people across the world had access. Thanks to Google Translate, Paypal, Skype video Call, Tor Project, and Google Voice i can and have collaborative with hackers almost in all geographies. You can only imagine what the black hats are doing.

4) Analyzing hackers is like reading Chinese Tea Leaves. If you have experienced analysts, you will slip up. recruit the hackers in the dormitory before China recruits them using Lulz Security as a bogus cover. or USA recruits them as cover for spreading democracy in the Arab countries.

5) get your website audited for security breaches. sponsor a hack my website contest. before someone else does it for you.

6) Fighting hackers was always tough. But now we have part time hackers , people with perfectly respectable jobs who look like Mr Andersen and hack like Neo from the Matrix. Every kid once wanted to be a firefighter. Every geek dreams  of the one ultimate hack.

7) if you cant beat hackers, join them.

8) the more machine data is generated, the more you need external experts and newer software interfaces. Investing in open data, datasets is good. Keeping Bradley manning naked in his cell is bad. ignore the bad PR at your own cost.

9) Stop blaming China for every hack attack. You are a techie not a politician

10) Hack hard. Hack well. If someone hacks you, you will need to hack them off offensively unless you just want to be an easy mark for the rest of your lives. Counter -hacking expertise needs to be strengthened and groomed. hacking is an offense not just a defense game.

 

 

%d bloggers like this: