Critical thinking for data scientists and hackers

I emphasize decision-making biases in my lectures on data science and hacking. Here are a few moot points.

A hacker knows the hacker attitude.

http://www.catb.org/esr/faqs/hacker-howto.html

The Hacker Attitude

1. The world is full of fascinating problems waiting to be solved.
2. No problem should ever have to be solved twice.
3. Boredom and drudgery are evil.
4. Freedom is good.
5. Attitude is no substitute for competence.

Basic Hacking Skills

1. Learn how to program.
2. Get one of the open-source Unixes and learn to use and run it.
3. Learn how to use the World Wide Web and write HTML.
4. If you don’t have functional English, learn it.

Status in the Hacker Culture

1. Write open-source software
2. Help test and debug open-source software
3. Publish useful information
4. Help keep the infrastructure working
5. Serve the hacker culture itself

A data scientist knows the definition of data science and he knows some cheat sheets to write code

http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram

Cheat Sheets

http://www.kdnuggets.com/2014/05/guide-to-data-science-cheat-sheets.html

a data scientist is simply a person who can write code in a few languages (primarily R, Python and SQL) for data querying, manipulation , aggregation, and visualization using enough statistical knowledge to give back actionable insights to the business for making decisions.

Since this rather practical definition of a data scientist is reinforced by the accompanying words on a job website for “data scientists” , ergo, here are some tools for learning the primary languages in data science- Python, R and SQL.

Coding in Python

http://www.statisticsviews.com/details/feature/8868901/A-Tutorial-on-Python.html

Cognitive Biases

A hacker knows the cognitive biases that impair his decision making

https://en.wikipedia.org/wiki/Cognitive_bias

A cognitive bias refers to a systematic pattern of deviation from norm or rationality in judgment, whereby inferences about other people and situations may be drawn in an illogical fashion.[1] Individuals create their own “subjective social reality” from their perception of the input.[2] An individual’s construction of social reality, not theobjective input, may dictate their behaviour in the social world.[3] Thus, cognitive biases may sometimes lead to perceptual distortion, inaccurate judgment, illogical interpretation, or what is broadly called irrationality.

 

Among the “cold” biases,

  • some are due to ignoring relevant information (e.g. neglect of probability)
  • some involve a decision or judgement being affected by irrelevant information (for example the framing effect where the same problem receives different responses depending on how it is described; or the distinction bias where choices presented together have different outcomes than those presented separately)
  • others give excessive weight to an unimportant but salient feature of the problem (e.g., anchoring)

 

Logical Fallacies

A hacker knows how to recognize  the logical fallacies  that are used to counter his scientific and systematic research

A fallacy is an incorrect argument in logic and rhetoric which undermines an argument’s logical validity or more generally an argument’s logical soundness.

A logical fallacy is a flaw in reasoning. Logical fallacies are like tricks or illusions of thought, and they’re often very sneakily used by politicians and the media

https://yourlogicalfallacyis.com/poster

ad hominem

You attacked your opponent’s character or personal traits in an attempt to undermine their argument. Ad hominem attacks can take the form of overtly attacking somebody

https://yourlogicalfallacyis.com/system/App/Settings/size24x36posters/000/000/001/original/FallaciesPoster24x36.pdf

 

Basic Math

Some basic math always helps you be  a better hacker. It is critical to be  data scientist.

This can make it easy

Basic Statistics

http://web.mit.edu/~csvoss/Public/usabo/stats_handout.pdf

Exploratory Data Analysis

https://en.wikibooks.org/wiki/Data_Science:_An_Introduction/Exploratory_Analysis

This can be intimidating

http://math.mit.edu/~bandeira/TenLecturesFortyTwoProblems.pdf

Machine Learning

Machine learning helps give you a day job. Hackers with higher salaries generally have more choices to direct their energies.

http://scikit-learn.org/stable/

 

Now- lets get more interesting

A hacker is educated and well rounded to understand Politics, Economics, Society, Technology, Legal and Environmental terms as well as their intersection for a country or an organization. He also understands the term regulatory arbitrage as well understand how law enforcement and financial regulators work.

The PESTLE Analysis is a framework used to scan the organization’s external macro environment. The letters stand for Political, Economic Socio-cultural, Technological, Legal and Environmental. Some approaches will add in extra factors, such as International, or remove some to reduce it to PEST

 

I repeat from the hacker’s attitude.

 

1. The world is full of fascinating problems waiting to be solved. andNo problem should ever have to be solved twice.

Lastly I believe hackers need to take care of their health and have a moral ethos that is unwavering.

Yoga for Geeks https://decisionstats.com/2014/06/21/yoga-for-geeks/

http://www.niu.edu/wac/archives/files/ethos.html

Of the three proofs–logos, pathos, and ethos–ethos is associated with the character of the speaker or writer. People are persuaded by people they trust, even if the argument is not terribly strong. Conversely, if the argument is strong, but the reader does not trust the writer or does not like the writer, the appeal to logic alone will seldom persuade. Therefore, it is essential to consider ethos.

Aristotle said that ethos consists of three sub parts: (1) good moral character, (2) good sense, (3) good will. If the writer or speaker can project an image of good moral character, then the audience will think that he or she can be trusted because the person has a conscience that will keep him or her on the up and up.

 

Lastly hackers have a great sense of humour and humility and a mock insult culture that thrives on intellectual freedom

Top Cartoonists:Updated

 

On Social Media

There are 3 kinds of people on social media. People who make comments, People who share content, and people who stay silent.

Out of the people who share ,they are 2 types of people, people who want to promote their professional interests, and people who want to share personal stories.

People who share get better responses than people who promote.

There are 2 types of people who make comments. One who say they like content because they do, and the other who say they dislike.

I just wish the negative commenters learn to speak less, and the people who are silent on social media speak more

Screenshot from 2016-06-29 09:14:36

 

from

http://www.linkedin.com/hp/update/6153903470568099841

and

https://infograph.venngage.com/p/123961/social-media_new

Will Rio Olympics be like Munich

The continued inability of American led IC to decrypt , intercept and predict extreme events associated with the assholes known as ISIS- leads one to wonder if Big Data,  Data Science and Complex Event Processing can ever come in time to easy to use interfaces. Analysts are currently on overworked scrambled mode, and so are law enforcement while sleeper agents of ISIS sleep.

The targeted events of California (Pakistani Shooter), Paris (ISIS bombers-shooters), Belgium (continued) and Orlando (Afghan origin shooter) suggest terrorism communication has evolved or are using freelance agents from multiple sources. While Communist Russia used to provide freelance advice to terrorism in the past, and Pakistanis have been master in low intensity terror led assymetric warfare, one can only shudder if Intelligence can evolve in time as Terror. or Does Eisenhower’s famous remark on the military industrial complex suggest- USA needs another Nobel PEace Prize winner and a war to justify it

Will the 5 eyes ( America , Canada, Australia, UK, et al) ever take help from the rest of the free world. Or is Presidential ego in building a bigger library bigger than the need for a clean Olympics

Complex event processing, or CEP, is event processing that combines data from multiple sources to infer events or patterns that suggest more complicated circumstances. The goal of complex event processing is to identify meaningful events (such as opportunities or threats) and respond to them as quickly as possible.

Related- Time Series Analysis of Complex Events and Signal/Noise Ratio Evolution

The Battle of Badr On 17 Ramadan

https://www.facebook.com/notes/maktab-channel/the-battle-of-badr-on-17-ramadan/239795319392896/

A New York Hackthon

I was asked to be a replacement judge for The FinTech City that Never Sleeps hackathon! This two day hackathon took place on Friday to Saturday in New York . It was brought by a startup accelator Startupbootcamp FinTech New York (@sbcFinTech) and sponsored by Byte Academy

 (Blogger conflict of interest disclaimer- I briefly advised one of the startups at Startupbootcamp for two weeks in data science and I gave a couple of free talks at meetups organized by Byte Academy)

Overall the theme of this hackathon was how to use innovation in technology for the biggest industry in New York (finance). Ergo the curious but rather catchy term fintech. Technology in Finance! Wall Street for the good of main street! One curious and common refrain was how millennials trust their mobile and Uber more than Banks or financial advisors.

My observations were- engineering talent in New York continues to lag behind California. Access to both money and design  makes New York a serious contender for usurping Silicon Valley’s mountain glory.

A serious advantage New York has over California is water. A serious advantage California has over New York is the sunlight.

The hacking kids in this hackathon were bright and innovative. Hacking is both a social experiment as well as practical need to think and innovate in stagnant industries like Finance, Fashion , Theatre, Social Empowerment and Entertainment. A healthy dose of reality shall ensure hackathons become the new more productive passtime of the children of America.  Star Trek Next Generation!

This one man team /guy won the best technical award and took all his prizes.

IMG_7998

The judges pose

IMG_8003

These guys (ideal karma !) won the overall hackathon

Screenshot from 2016-06-16 12:47:53

 

For details see http://www.eventbrite.com/e/the-fintech-city-that-never-sleeps-hackathon-tickets-25550744966 or more tweets me at https://twitter.com/holydatascience

 

I really enjoyed judging the hackthon in New York, which ironically is still my favourite city because of the libraries, trees and well pretty people besides the lovely Hudson in the summer.  It was nice to be back in New York since 2009 when I came over for the first Big Data meeting ( by AsterData later acquired by Teradata).

Afterthought- What is built by a human can be and should be hacked by another human. That is the only way we are going to be continuously improving as a species and as one world. Hacking is important to simplify a complex world where resources are precious and the costs of error huge 

 

 

Hack for Change #hackforchange

As part of National Day of Hacking, I took part in the hackathon in New York. Here were some insights

A few teams had come prepared reading the challenge questions. These teams had an advantage on time

Creating something in time was a big challenge ( how do you make a product in a single day)

Hackathon consists of 1) organizer giving challenge questions 2) people coming to venue 3) making teams 4) working together as team 5) presenting results (usually one person per team)

The idea is the most important in how relevant and closely aligned to the questions in hackthon you were. Creativity rules

The next important thing was making a balanced team in which everyone gels well, and have skill sets that are complementary ( one front end, one back end, one data scientist in Python, one person who is good at presentation etc)

The next important thing was not getting intimidated by other teams and working on your team idea till last moment

The presentation should be given to a person who is best at expressing 1) what you did 2) how the solution is innovative 3) how it is relevant and useful to challenge

Lastly have fun hacking. People who have fun hacking generally tend to be better hackers.

Screenshot from 2016-06-10 06:40:04