How Jupyter Ipython threaten the dominance of RStudio for data science developers

RStudio is the clear market leader in IDE used by developers for R data science.

R is the clear market leader for data science.

Python can do with more wrappers for R like packages.

But Jupyter is awesome (once you get it working!)

Screenshot from 2015-10-20 14:29:53

Hopefully, multi core stuff and cloud hosted stuff should be easy too. Google Cloud Data Labs with hosted Jupyter is just the first step. see https://cloud.google.com/datalab/

One of the best things I like about Jupyter OVER RStudio’s interface is the ability to divide code blocks in cells. In addition the ability to install new packages from with RSTUDIO really helps me over the Jupyter. The syntax prompt in latest version of RSTUDIO is something I wish JUPYTER really worrks on.

Can we have a RSTUDIO like interface to working with Python. Yes Yhat made one and called it RODEO. This is because the interface is based on the ACE editor ( yes esseentially RStdudio the company married ACE Editor to Hadley Wickham to get RSTUDIO the product 😉  . Shiny was wonderful but for scalable data science Python and Java help me just as much as R does for BIg DATA ANALYSIS) Scalability is the key here! Rpubs isnt as popular as NBviewer is and now we can wrap markdown within a Jupyter notebook

Screenshot from 2015-10-20 14:28:03

Screenshot from 2015-10-20 14:27:36

Screenshot from 2015-10-15 18:37:41

can Jupyter help in my data science work more than RStudio? These are early days but I prefer a cross platform cross language ( Julia, Python and R) solution anyday. Provided it works just as seamlessly than the established market leader RStudio.

BIG DATA ANALYTICS is where I clearly see JUPYTER help data scientists more than RStdudio as you can use the IRKERNEL. I am especially hoping to see the Spark Kernel , JS Kernel  https://www.npmjs.com/package/ijavascript  and others be  more production ready for business enterprises.

https://github.com/ibm-et/spark-kernel

A version of the Spark Kernel is deployed as part of the Try Jupyter! site. Select Scala 2.10.4 (Spark 1.4.1) under the New dropdown. Note that this version only supports Scala.

https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages

Python/Jupyter kernels:

The Kernel Zero, is of course IPython, which you can get though ipykernel, and still comes (for now) as a dependency of jupyter. The IPython kernel can be thought as a reference implementation, here are other available kernels:

Name Link Jupyter/IPython Version Language(s) Version 3rd party dependencies
ICSharp https://github.com/zabirauf/icsharp Jupyter 4.0 C# 4.0+ scriptcs
IRKernel http://irkernel.github.io/ IPython 3.0 R 3.2 rzmq
SageMath http://www.sagemath.org/ IPython 3.2 Any

Screenshot 2015-10-20 14.07.08 (1)

Screenshot 2015-10-17 18.10.10

 

Screenshot 2015-10-20 11.17.52

The God of Big Data

An Omniscient God should be someone who can analyze all data all the time in any form. Think of a twitter created by the alpha waves of the thoughts of your brain. God is following you. Like Tom of MySpace , HE is friends to all of us. Or thats what the definition of GOD is supposed to be

super small sensitive sensors generate data that are analyzed by algorithms that present results for decision making by angels. sensors embedded in your brain your mobile your watch your laptop.

What do humans do with this data? Do they give you a personalized learning program on what to read today. Do they give you a personalized exericse and food program on how to stay healthy today? or do they just ask you to click click click, so you can maybe convert a bt, and then buy buy buy.

Buying generates debt and economic activity and puts pressure on the buyer to keep working harder so he can come to play the buying game in conspicuous consumption. Does the product you buy list your carbon impact. No that will lower sales. Does it give you a rush of chemicals approximating happiness. Of course retail therapy is the best psycho therapy.

A merciful loving God that has access to all data would intervene not only yo keep us buying and working, but to keep us informed of choices we make and choices we disdain. On how to be better educated, better informed, better health and better impact on government and enironment.

Google failed us as the God of all Data. Alphabet is just a primal soup right now.

The God of Big Data? Now accepting applications.

a little knowledge is a unicorn startup

I see them all day and I see them from morning to evening. Bright-eyed young men on one side, cynical middle-aged men smiling on the other side of the table. Playing a game that is half-way between strip poker and Russian roulette. The one where one side claims they are trying to change the world by giving the other side 10X on their investment. If you could only sign the cheque Mr Middle Aged Man, and if you could only could rub your nose with powder Mr Young Man.Being a data scientist can be an advantage in these oxygen sucking bone marrow draining meetings as you play umpire to Sharapova grunting to beat Hingis in this tennis match. You can do a tech jargon dump, avoid questions by claiming just to be the tech guy or you can give them crazy brilliant scientist with the bad joke routine

What grates my teeth , gets my goat Billy, irritates my allergic cough and what really annoys me is when the potential investor tries to show off their technology chutzpah and when the-the potential investee starts sucking icecream just to stay afloat in the game. Life is unfair and Karma is a bitch cliche, but when I feel like shouting at the inane conversations in the meetings I have to attend, I amuse myself by calculating how much money I am making every second ( it’s not much , but we have a long term plan).

A little knowledge is a dangerous thing. Young men  often shoot more of this not knowing enough, but old men who just won’t do a google search before they speak inane is what drives me  temp insane. Do you really want to hand over 25% of your company to this guy for 300K kid.

That’s right, the valuations in Indian startups are so mind blowing fucking low, that all the failed VC’s or the homesick desis return home and over night go from upper middle-class suburb desperate househusbands to indian angel investors returned from cowboy land to the glorious motherland to do what is right by her. They used to be proud first generation Americans till they returned to become Wily Oriental Gentlemen.

Technology startups in India are hot, but I think it’s not a bubble because valuations are low in dollars, but its a correlated and cointegrated bubble with the hash they smoke in San Carlos streets.

But

  • the tech talent is shoddy (crumbs after the outsourcing lions have been fed )
  • and its all wild wild east laisse faire (in one of the most corrupt countries still)
  • and all the  sex  (500 million people less than 30 in India , cough)
  • and all the Opium Triangle drugs make this more of a Mad Man on TV than a MAD magazine on live street.

Google your jargon, kid and google your bullshit unicorn breeders.

In India, cow dung is holy, but bullshit is not.

That’s not even a metaphor. Its reality and reality bites the slumdog millionaires faster than it bites the husband of a Kardashian.

Cheques and Balances for Startups

The concept of Western democracy rests in the idea of checks and balances. Judiciary (Law Courts)acts as check to Executive (Governments) and in turn balance is provided by Legislature ( Senate /Parliament). To counter the friction between plebians and patricians in a society, Legislature is further divided into Lower House (House of Commons, Congress, Lok Sabha) and Upper House ( House of Lords, Senate, Rajya Sabha). This divide and then rule policy serves as a risk mitigation force in governance for countries. The risks of course are unstable, violent and unpredictable changes in operation (like revolutions, coups, revolts, dictatorships)

Organizations are famously known to be non-democratic. Startups even less so. Yet every team member in a technology startup is valuable and that value increases after they have got on the job training in cutting edge technologies and exposed to solving interesting problems. You balance their tendency to churn by giving good food, great culture, flexible timing. An important part is stock options granted timely and in a fair manner, and salaries paid. Cheques check attrition in startups. A healthy bank balance in the organization is the only thing that balances the uncertainty and insecurity of working in unstructured dynamic environments like technology startups.

Learning must be balanced by earning for young members of each startup. Older experienced members of startup need checks and balances of  a different sort as they have multiple opportunities in employability and opportunity costs in getting a lower startup salary. Older members of a startup , okay, lets call them seasoned members of a startup have more obligations like student debt, house debt, child education etc.

I wonder if the level of entrepreneurship in a society would increase if we had lesser student debt, by giving cheaper education, by using mobile, internet and video more. Trainings in various kinds of coding languages should also be increased in accessibility and duration. I see many technology stacks being coded in bad choices of platforms solely because the techies were not prompted to enroll in the right paid program for online self-learning. So for example if there is shortage of Ruby on Rails or Python developers in data science in India, rather than invest a small 50$ per employee per month in online based training, companies continue to work with legacies of LAMP, and other legacy software.

You just can’t google your way to being a data scientist or a big data engineer. Trainings are just as important as good food for members in your startup.

What balances the benevolent dictatorship of the passion of the founder. A seasoned mentor in the team. What checks people from hire and fire and reneging on stock option promises. Defined processes established by a compliance person.

It is upto you the founder, or the investor on what kinds of checks and balances you want to create in your startup. Do you want the startup to look like Canada or do you want it to be like Syria? Choose the cheques, checks, and balances accordingly.

 

 

 

 

Comparing the User Interface in Indian Websites in ECommerce

I did a  search for ” samsung 4g mobile” across the following – Google Search, Askmebazaar.com, Amazon.in, Snapdeal.com, Flipkart.com and got following results.

I wanted to compare user interface, the search results, the navigation experience, and lastly any price arbitrage opportunities. Part One of this series just looks at initial search results for a single keyword and compares it across websites.

Findings-

  1. Apparently FlipKart does not believe in Adwords, or SEO

https://www.google.co.in/search?q=buy+a+samsung+4g+mobile+in+india&oq=buy+a+samsung+4g+mobile+in+india&aqs=chrome..69i57j0.4451j1j7&sourceid=chrome&es_sm=122&ie=UTF-8

Screenshot from 2015-10-17 09:36:262) Amazon has a clean interface and good graphic icons for the product

http://www.amazon.in/s/ref=nb_sb_noss?url=search-alias%3Daps&field-keywords=samsung+4g+mobile

Screenshot from 2015-10-17 09:34:45

Surprisingly it suggested in associated products – a beard trimmer for me. Amazon also prompts saying how the item is limited and only X are left in stock.

3) Flipkart sells FlipKart itself (of apps etc) in half the page and has no associated results ( you may also buy this- or add to cart)

http://www.flipkart.com/search?q=samsung+4g+mobile&as=on&as-show=on&otracker=start&as-pos=1_q_samsung+4

However I liked the ordering of the filters in left margin, with price on top, then brand etc. Quite clearly price is the top filter in such sites.

Screenshot from 2015-10-17 09:33:29

4) Askmebazaar has a menu layout in horizontal rows and columns than a list layout. Interesting to see the Indian origin interfaces had menu layout while the US derived websites had a list (top to bottom layout)

It was interesting to see that every price in it had a discount with a strikethrough in the initial prices. Striking and interesting as its different.

http://www.askmebazaar.com/index.php?

Filters in right margin were Category first, then brand ( both irrelevant since I am askign for Samsung Mobile) and price at bottom. Price was a dropdown filter than a radio button or checkbox

filterapp_data=c2VhcmNoX3F1ZXJ5PXNhbXN1bmcgNGcgbW9iaWxlJmRlZlNlYXJjaD0w

5) Ebay has really awesomely long URLS and is a mix of Amazon and Askmebazaar interface

http://www.ebay.in/sch/i.html?_from=R40&_trksid=p2050601.m570.l1313.TR10.TRC0.A0.H0.Xsamsung+4+g+mobile.TRS0&_nkw=samsung+4+g+mobile&_sacat=0

Screenshot from 2015-10-17 09:52:14

6) Lastly Snapdeal had a nice clean interface and it was less cluttered than some of the other sites. I was also interested that they showed a prompt to buy iPhone so an interesting association analysis result.

Which e commerce company in India has the best user interface? You decide. Beauty and ease lie in the eye of the beholder.

Screenshot from 2015-10-17 09:33:59

AI versus BI -Machines can’t learn but Nerds hate Humans

Beer is an alcoholic beverage produced by the saccharification of starch and fermentation of the resulting sugar. The starch and saccharification enzymes are often derived from malted cereal grains, most commonly malted barley and malted wheat.

Organizations especially in technology consist of a mixture of business managers and technology managers. Bizops and Techops. Jocks and Geeks.

Business managers like to think of using strategic as a prefix to anything. Strategic thinking, strategic planning, strategic campaign , strategic project. Strategic is a word to make things look more important than they are. Anything that has the word strategic should have more than a 12 month gestation period.

I come across machine learning often. It is a cute term and it ranks in terms of abused terms often. Algorithm is another abused term. You say algorithm but what you mean is you need a function.

Basically machine learning means your machine can learn. That by itself is an oxymoron two words that are self-contradictory juxtaposed together. It should be program learning or code learning. Your code or program is trying to approximate learning by taking in outputs  in a previous step as inputs in a future step. Learning is presumed to you can learn only from past experience.

My Dell Laptop has not learned anything. My Dell Laptop is the machine. Sometimes I use functions from packages in languages. These functions are based on well-established algorithms like apriori, k means , and even math theorems like Bayes. I am not learning anything. My machine is not learning anything. the function converges to a solution based on math written by a human, coded by a human, implemented by a human and finally called by a human. Where did the machine in the machine learning come from?

It is derived from Artifical Intelligence. That is another oxymoron. If something is intelligent what is artificial about it. Isn’t a work created by a human as natural as a work created by Nature.

Most nerds and geeks  I have met are not sorted emotionally. They can tell you half a dozen algorithms to sort data, but they don’t know when to shut up, when to listen, when to say yes and when to keep quite than just say no. These are humans that are full of knowledge but won’t learn about human behaviour as fast as they learn about algorithms. Geeks dont make good business managers. Atleast most geeks. I personally cant make out the difference between a geek and a nerd and who gets paid more and who has more fun.

But geeks are incredibly focused people doing tasks that are incredibly boring to normal humans. The reason they dont gel with other humans is because hey they didnt spend time with other humans – they retreated to the safety of the machines and computer screens a long time ago in their formative years, and it paid them enough to keep staying and spending more time with machines than humans

Humans are just as important as machines, maybe more. The geek writing the algorithm can learn more much more about psychology, philosophy and creative liberal arts that affect human behaviour.  That human behaviour generates the data that the algorithm runs on eventually. Cross training your geeks to get insights about humans and cross training your business managers on the difference between AI and BI, machine learning and statistical modeling, what is an algorithm and what is a package.

Artificial Intelligence AI versus  Business Intelligence BI is the new phenomenon. Tech teams brought up on dashboards and reports have to adopt to machine learning algorithms incorporated as decision-making assistance tools. More AI in your BI maybe.

Cliches and buzz words and hype jargon are pick up lines in business world. They can get you some action for sure but they cant get you into Heaven. Sort your geeks emotionally and your business managers algorithmically. Beer is a great way for lonely introverts to bond with flashy extroverts.Now that is just my personal beer algorithm. You may discover your own beer game for your organization.

 

 

 

 

Introduction to Sufi Music

Why Sufi Music is the best example of two different religions and people can coexist and have a big party.

Sufi Music

Sufi music is the devotional music of the Sufis, inspired by the works of Sufi poets, like Rumi, Hafiz, Bulleh Shah, Amir Khusrow and Khwaja Ghulam Farid. Qawwali is the most well known form of Sufi music, and is most commonly found in the Sufi culture of the Indian Subcontinent.

yeh jo halka halka suroor hai                        – This slight intoxication that I have
ye tere nazron ka kasoor hai                         – Is all the fault of your looking at me
ke sharab peena sikha diya hai                     -That has taught me to take up drinking
tere pyar ne tere chahne                                                                            —– Your love and your wanting
teri bhehki behki neegah ne mujhe ek sharaabi banadiya                 —–Your intoxicating  looks has made me an alcoholic
pilaadi hai apni kiss nazar se ke mujhe hosh nhi hai                          —– You have served the drink with a single look and now I am not in my senses