R for Recession

Here is how you can use R ( the software) to beat recession.

A new plugin just came out by Li for WordPress pages for R Web Interfaces.

http://lixiaoxu.lxxm.com/rwebfriend/

You can also do R analysis straight from the Web here –

 

http://pbil.univ-lyon1.fr/Rweb/

 

Using R for The Web Interfaces ( http://cran.r-project.org/doc/FAQ/R-FAQ.html#R-Web-Interfaces) ,hosting them on a cloud environment ,using encryption and compression through the Ohri Framework ( see http://www.decisionstats.com/?p=217)

you can extend the use of R for not only scoring models as Zementis (www.zementis.com ) currently does , but also building them online .

All you would need is Ubuntu Linux (www.ubuntu.com) , and a fast web connection.

Thats it- No fast servers, No Windows Licenses, No Analytical Licenses at all needed.

And someone to build all the above- any takers ?

 

Note- I retain authorship of the idea- Ohri framework .

 

IDEA_

1) Use encryption and compression software on data

2) Upload compressed data to R Web Interface Server hosted on cloud computer

3) R Web Interface decompresses and decrypts the data before executing the statistical analysis using appropriate package.

4) View Analysis online or download graphics

5) Encrypt and compress scored data ( in case of regression scoring for propensity scoring or risk modeling)

6) Download scored data and decrypt and decompress.

You can cut the download and use the web for straight online scoring , but that would require a private cloud.

Life's Little Instruction Book……………………

 


 Have a firm handshake. 
 Look people in the eye. 
 Sing in the shower. 
 Own a great stereo system. 
 If in a fight, hit first and hit hard. 
 Keep secrets. 
 Never give up on anybody. Miracles happen everyday. 
 Always accept an outstretched hand. 
Be brave. Even if you’re not, pretend to be. No one can tell the 
difference. 
 Whistle. 
 Avoid sarcastic remarks. 
 Choose your life’s mate carefully. From this one decision will come 90 
per cent of all your happiness or misery. 
 Make it a habit to do nice things for people who will never find out. 
 Lend only those books you never care to see again. 
 Never deprive someone of hope; it might be all that they have. 
 When playing games with children, let them win. 
 Give people a second chance, but not a third. 
 Be romantic. 
 Become the most positive and enthusiastic person you know. 
 Loosen up. Relax. Except for rare life-and-death matters, nothing is 
as important as it first seems. 
 Don’t allow the phone to interrupt important moments. It’s there for 
your convenience, not the caller’s. 
 Be a good loser. 
 Be a good winner. 
 Think twice before burdening a friend with a secret. 
 When someone hugs you, let them be the first to let go. 
 Be modest. A lot was accomplished before you were born. 
 Keep it simple. 
Beware of the person who has nothing to lose. 
 Don’t burn bridges. You’ll be surprised how many times you have to 
cross the same river. 
 Live your life so that your epitaph could read, No Regrets 
 Be bold and courageous. When you look back on life, you’ll regret the 
things you didn’t do more than the one’s you did. 
 Never waste an opportunity to tell someone you love them. 
 Remember no one makes it alone. Have a grateful heart and be quick to 
acknowledge those who helped you. 
Take charge of your attitude. Don’t let someone else choose it for 
you. 
Visit friends and relatives when they are in hospital; you need only 
stay a few minutes. 
 Begin each day with some of your favorite music. 
 Once in a while, take the scenic route. 
 Send a lot of Valentine cards. Sign them, ‘Someone who thinks you’re 
terrific.’ 
Answer the phone with enthusiasm and energy in your voice. 
Keep a note pad and pencil on your bed-side table. Million-dollar 
ideas sometimes strike at 3 a.m. 
 Show respect for everyone who works for a living, regardless of how 
trivial their job. 
 Send your loved ones flowers. Think of a reason later. 
 Make someone’s day by paying the toll for the person in the car behind 
you. 

 Become someone’s hero. 
 Marry only for love. 
 Count your blessings. 
 Compliment the meal when you’re a guest in someone’s home. 
 Wave at the children on a school bus. 
 Remember that 80 per cent of the success in any job is based on your 
ability to deal with people. 
 Don’t expect life to be fair.
Source-(From a email sent )

Life’s Little Instruction Book……………………


Have a firm handshake.
Look people in the eye.
Sing in the shower.
Own a great stereo system.
If in a fight, hit first and hit hard.
Keep secrets.
Never give up on anybody. Miracles happen everyday.
Always accept an outstretched hand.
Be brave. Even if you’re not, pretend to be. No one can tell the
difference.
Whistle.
Avoid sarcastic remarks.
Choose your life’s mate carefully. From this one decision will come 90
per cent of all your happiness or misery.
Make it a habit to do nice things for people who will never find out.
Lend only those books you never care to see again.
Never deprive someone of hope; it might be all that they have.
When playing games with children, let them win.
Give people a second chance, but not a third.
Be romantic.
Become the most positive and enthusiastic person you know.
Loosen up. Relax. Except for rare life-and-death matters, nothing is
as important as it first seems.
Don’t allow the phone to interrupt important moments. It’s there for
your convenience, not the caller’s.
Be a good loser.
Be a good winner.
Think twice before burdening a friend with a secret.
When someone hugs you, let them be the first to let go.
Be modest. A lot was accomplished before you were born.
Keep it simple.
Beware of the person who has nothing to lose.
Don’t burn bridges. You’ll be surprised how many times you have to
cross the same river.
Live your life so that your epitaph could read, No Regrets
Be bold and courageous. When you look back on life, you’ll regret the
things you didn’t do more than the one’s you did.
Never waste an opportunity to tell someone you love them.
Remember no one makes it alone. Have a grateful heart and be quick to
acknowledge those who helped you.
Take charge of your attitude. Don’t let someone else choose it for
you.
Visit friends and relatives when they are in hospital; you need only
stay a few minutes.
Begin each day with some of your favorite music.
Once in a while, take the scenic route.
Send a lot of Valentine cards. Sign them, ‘Someone who thinks you’re
terrific.’
Answer the phone with enthusiasm and energy in your voice.
Keep a note pad and pencil on your bed-side table. Million-dollar
ideas sometimes strike at 3 a.m.
Show respect for everyone who works for a living, regardless of how
trivial their job.
Send your loved ones flowers. Think of a reason later.
Make someone’s day by paying the toll for the person in the car behind
you.

Become someone’s hero.
Marry only for love.
Count your blessings.
Compliment the meal when you’re a guest in someone’s home.
Wave at the children on a school bus.
Remember that 80 per cent of the success in any job is based on your
ability to deal with people.
Don’t expect life to be fair.
Source-(From a email sent )

Indian Marriage Economics

On a much deserved break from analyzing numbers and economies , the family and I headed out to a neighbour’s wedding. This is November , and peak time in the Indian Marriage season. However I could not help but noticing the folowing things while at the wedding-

  1. People in Indian weddings mostly wear clothes of all colors , unlike the West where Black is worn by the Groom and his men , while the Bride wears white. Here almost all ladies are festooned in rainbow colored silks and flash golden (real and fake) jewellry. This has a noticeable effect on Gold prices ( almost all imported in India) and clothes ( almost totally made in India and exported too)
  2. Thanks to municipal regulations – weddings in the capital ,Delhi have to me only in community halls and licensed wedding halls. Weddings in farmhouses are banned. This has led to huge shortage of wedding spaces in the capital area, leading to almost all open air weddings being conducted in the sub urbs. The shortage of wedding places is due to lack of planning by urban planners from the pre liberalized India, whoi could not figure out the annual growth rate of population in Delhi  would also lead to an annual growth rate of weddings in Delhi.
  3. There are 365 days in an year but Indian weddings can only be made in certain auspicious days as per the collective Hindu priestdom , which though much impoverished today , earns principally during the wedding times . This also leads to shortage of space in Wedding season and hence an artificial price hike in the booking of marriage spaces. During lean season ( apparently Planet Shani or Saturn has something to do with it in Hindu astrology), the wedding spaces and caterers do lookout for Muslim ,Sikh and Christian weddings ( together 15 % of India’s population- numbers for Delhi not updated). No link has been found in either Hindu priests or the wedding planners but some cartelization seems a possibilty as they gain from the artifical shortage of days on which you can get married. Co relation however is not causation.
  4. If you disbelive this – my own wedding got postponed by 2 months while I waited for the planets to get back on track.
  5. The various vendors and sub contractors in an Indian wedding are
    1. Horse Minder and Horse ( Hindu Grooms come riding on a Horse. Very romantic but quite painful I assure you for the groom. The pain seems delibrate as India has the lowest rate of divorces and re marriages.
    2. Caterer of Food ( Seperate Charges for Vegetarian and Non Vegetarian with Non Vegetarian food quite expensive.)Food is generally charged per plate with a minimum committment. Charges can range on a median of 400 Rs per plate. Thats 8 dollars a plate ( Rupee got depreciated again)
    3. Tent Chap – One who puts up the big tent
    4. Flower People – Literally Flower people
    5. Wedding Space rent
    6. Clothes ( Specialized Saree sellers loved by women and universally hated by the men who have to pay for the sarees and cant make out the difference between dark blue and ..mauve.
    7. Jewellers- India remians the largest importer of Gold. It has the highest per capita gold consumption. Part of the culture. Funnily it makes sense to keep money in Gold and hand it down from generation to generation.
    8. Music Dj – who plays the Indian Bollywood songs and in Northern Urban India a lot of Punjabi music
    9. Photographers and Video graphers
    10. Very few professional wedding planners for most middle class weddings. This is thanks to the Indian penchant for saving money and also because of economic reason ( see below)
  6. Enormous money is pumped in Indian weddings compared to the Indian per capita income ( 800 USD PER YEAR ). It makes the weddings more memorable. Adds to the pain so people stay married and avoid divorces.

Thats all for this week. The Dow has crashed again, the rupee has crossed 50 per dollar, Oil has fallen below 50 USD, and US congress is now re thinking on the auto bailout.Gold prices internationally remain confused and Indian inflation fell to 8.9 %. Indian bankers continue to ask for a rate cut while the central bankers continue to smile like the Buddha.

Recessions come and go but as long as people get married again and again life continues.

Happy Thanksgiving !!

SQL and Hadoop: What is this cloud thing

Here is a very good ,in fact brilliant post from Joe Hellerstein, a Professor of Computer Science at UC Berkeley at http://radar.oreilly.com/2008/11/the-commoditization-of-massive.html

It explains the difference between the two databases type.

 

Enterprise IT camp tends to favor relational databases and the SQL language, while the web upstarts have rallied around the MapReduce programming model popularized at Google, and cloned in open source as Apache Hadoop. Hadoop is in wide use at companies like Yahoo! and Facebook, and gets a lot of attention in tech blogs as the next big open source project. But if you mention Hadoop in a corporate IT shop you are often met with blank stares — SQL is ubiquitous in those environments

 

Setting aside the trash talk, the usual cases made for the two technologies can be summarized as follows:

Relational Databases

  • multipurpose: useful for analysis and data update, batch and interactive tasks
  • high data integrity via ACID transactions
  • lots of compatible tools, e.g. for loading, management, reporting, data visualization and mining
  • support for SQL, the most widely-used language for data analysis
  • automatic SQL query optimization, which can radically improve performance
  • integration of SQL with familiar programming languages via connectivity protocols, mapping layers and user-defined functions

MapReduce (Hadoop)

  • designed for large clusters: 1000+ computers
  • very high availability, keeping long jobs running efficiently even when individual computers break or slow down
  • data is accessed in "native format" from a filesystem — no need to transform data into tables at load time
  • no special query language; programmers use familiar languages like Java, Python, and Perl
  • programmers retain control over performance, rather than counting on a query optimizer
  • the open-source Hadoop implementation is funded by corporate donors, and will mature over time as Linux and Apache did

Hadoop is still relatively young, and by all reports much slower and more resource intensive than Google’s MapReduce implementation.

What I liked about the article was explaining Hadoop in simple terms to corporate SQL types like me.

It’s interesting how Hadoop would be configured on the NVidia Tesla supercomputer ( at 10000 USD)

 

– Update – Mathematica is already being modified for the GPU versus CPU system, and there was an interesting discussion in R _help list today on this.

Mathematica is launching a version working with Nvidia GPUs. It is claimed that it’d make it
~10-100x faster!
http://www.physorg.com/news146247669.html

Laptops and SuperComputers

The One Laptop per Child project is having its annual give get one promotion. Basically you pay 400 $ , you get one XO laptop free and another XO laptop is donated in your name in a devleoping country. For the technically minded ,here is a great review of the XO laptop at http://www.bunniestudios.com/blog/?p=218

 

On a slightly different scale is the NVidia GPU ( as opposed to CPU powered computers). They are available here at http://www.nvidia.com/object/wtb_notebooks.html

These and the forthcoming series of NVidia powered GPus are going to give extremely high speeds within a price range of 10,000 USD. How high is the speed ?

Well here is a case study from the NYT "

Techniscan Medical Systems of Salt Lake City has turned to Nvidia’s graphics processors to speed up a three-dimensional breast scanning device that could be used for cancer detection if the machine received regulatory approval. Techniscan must turn tens of gigabytes of raw data generated by transmitting pulses of energy through a breast submerged in water into medical image files that consume just 100 megabytes. This whole process used to take a couple of hours using Intel’s processors and now takes just 15 minutes with Nvidia’s hardware."

 

And here is finally your desktop supercomputer, the Tesla from Nvidia.

http://www.nvidia.com/object/personal_computing.html

"Get your own supercomputer. Experience cluster level computing performance—up to 250 times faster than standard PCs and workstations—right at your desk. The NVIDIA® Tesla™ Personal Supercomputer is based on the revolutionary NVIDIA® CUDA™ parallel computing architecture and powered by up to 960 parallel processing cores."

Now data mining and analytics people love processing power. With this much processing power  it can be quite a lot of fun !

So if you deal with data more than 1 Gb at a time or have more than 10 Pcs, or 2 servers, try the Tesla.

Tesla Architecture

  • Massively-parallel many-core architecture
  • 240 scalar processor cores per GPU
  • Integer, single-precision and double-precision floating point operations
  • Hardware Thread Execution Manager enables thousands of concurrent threads per GPU
  • Parallel shared memory enables processor cores to collaborate on shared information at local cache performance
  • Ultra-fast GPU memory access with 102 GB/s peak bandwidth per GPU
  • IEEE 754 single-precision and double-precision floating point
  • Each Tesla C1060 GPU delivers 933 GFlops Single Precision and 78 GFlops Double Precision performance

Software Development Tools

  • C language compiler, debugger, profiler, and emulation mode for debugging
  • Standard numerical libraries for FFT (Fast Fourier Transform), BLAS (Basic Linear Algebra Subroutines), and CuDPP (CUDA Data Parallel Primitives)

Product Details

  • 3 or 4 Tesla C1060 Computing Processors with 4GB of dedicated memory per GPU
  • 2.33 GHz+ Quad-core AMD Phenom or Opteron, — OR — Quad-core Intel Core 2 or Xeon
  • Minimum system memory: 12 GB for 3 Tesla C1060s and 16 GB for 4 Tesla C1060s (at least 4GB per Tesla C1060)
  • 12GB+ system memory (at least 4GB per Tesla C1060)
  • 1200-1350 Watt Power supply
  • Acoustics < 45dbA

Supported Platforms

  • Microsoft® Windows® XP 64-bit and 32-bit (64-bit recommended)
  • Linux® 64-bit and 32-bit (64-bit recommended)
    • Red Hat Enterprise Linux 4 and 5
    • SUSE 10.1, 10.2 and 10.3

Source-http://www.nvidia.com/object/tesla_supercomputer_tech_specs.html

Cougar Airlines – Extract from Book

The first thing that you notice , especially if you are Indian, is the striking difference between US Airlines air hostess and Indian air hostess ( especially from the private sector). While US Air Hostess are more often in their late thirties and early forties, Indian air hostess are in the early twenties to late twenties. This could be explained by the fact that India of course has a median age in the twenties while the US and the west have much higher median ages due to lowered birth rate.

These were some of the amusing lines I heard while flying in the US.

Hostess- Sir, can I get you a drink.

Old Indian Man- I like whisky.

Hostess- Sir, we cant serve you in Indian air space . We can serve you once we are in international borders ( thanks to ancient indian laws)

Old Indian Man ( The chap is 70 plus) – Kicks up huge row . I am preparing to pay ,blah blah.

Hostess finally gets his whisky.

2

Hostess- Sir, Can I get you a drink.

Me- I would like a Red Bull

Hostess- Red Bull ?

Me- Yeah, its an energy drink.

Hostess- Confused- Well I am sorry we dont have any Red Bull

Me- Guess I will just take water then.

3

Hostess- Sir, Can I get you anything.

Me- Yes I would like a cougar

Hostess bends down and says- Oh no dear, we all know what that means.

Me- Guess I will take water then.

Another annoying thing are the dumb airlines rules. I might have mentioned if before but I am still mad about it so here it is.

You are allowed to take two bags. Each bag can take 20 pounds. But if one bag has 23 pounds and the other ten pounds you still have to pay the overweight charge. How much is that ? That will be fifty dollars sir. Any attempt to reason and point the sheer mathematical madness was met with a smile. So I end up taking out my bag opening both and transferring stuff from one to another. The rest of the passengers, especially the well clad ladies give me pitying, pathetic look.

Security rergulations is another nightmare. You can be brown, but if a brown male, slightly tall and thin ( 5 ft 10 ‘ and 150 pounds) walks , he gets to do all kinds of strip tease.

I empty my pockets, my wallet ,my belt, and my shoes. Take the laptop off the cover. Put the mobile and the spare change in the trays.

When I walk in , the metal detector goes ” Beep , Beep , Beep”.

And this tiny customs lady in a Hispanic voice shouts loud enoguh for half the airport to hear.

“Sir, Please get back immediately. Please take off your sweater”

I take off the sweater and attempt the metal detecting walk again.

Again, ” Beep , Beep”.

” SIR, PLEASE TAKE OFF YOUR T SHIRT”

I am now in bare vest and trousers.

Still , ” Beep ,Beep ,Beep’.

The custom lady eyes me. I eyeball her back. She finally notices my thin bracelet ( its copper and less than o.4 cm thick , its called a Kadha , and a religious item’.

” Sir , I think its your bracelet”.

I try and take it off, which I havent done since my wedding three years ago. I am not as thin as I think I am. Much squirming, twisting and bending later I get it off.

Sucess. No beep ,beep. No loud Sir take this off,take that off.

As i gather up my stuff , I think I can see the glimmer of a smile on the Customs lady. Its okay.. she is good looking enough for me not to resent that smile.

“This is an extract from :Flying with the Cougars” an upcoming book on a brown boys trip to Austin, Texas. its semi -fictional so some coincidences would be just a resemblance to a reality.

All rights reserved for publication and reprint.-Ajay