Category: Internet

using R for Cricket Analysis #rstats

New Zealand just made it to their first ever world cup final ( yes it is cricket) and they made it with a thrilling six ( like a home run) for the last ball. Congrats to New Zealand .Of course R was created in New Zealand too and Hadley Wickham is from New Zealand

I recently installed the rvest package from https://github.com/hadley/rvest and its now on CRAN as well

 

rvest helps you scrape information from web pages. It is designed to work with magrittr to make it easy to express common web scraping tasks, inspired by libraries like beautiful soup.

library(rvest)
lego_movie <- html("http://www.imdb.com/title/tt1490017/")

rating <- lego_movie %>% 
  html_nodes("strong span") %>%
  html_text() %>%
  as.numeric()
rating
#> [1] 7.9

cast <- lego_movie %>%
  html_nodes("#titleCast .itemprop span") %>%
  html_text()
cast
#>  [1] "Will Arnett"     "Elizabeth Banks" "Craig Berry"    
#>  [4] "Alison Brie"     "David Burrows"   "Anthony Daniels"
#>  [7] "Charlie Day"     "Amanda Farinos"  "Keith Ferguson" 
#> [10] "Will Ferrell"    "Will Forte"      "Dave Franco"    
#> [13] "Morgan Freeman"  "Todd Hansen"     "Jonah Hill"

poster <- lego_movie %>%
  html_nodes("#img_primary img") %>%
  html_attr("src")
poster
#> [1] "http://ia.media-imdb.com/images/M/MV5BMTg4MDk1ODExN15BMl5BanBnXkFtZTgwNzIyNjg3MDE@._V1_SX214_AL_.jpg"

The most important functions in rvest are:

  • Create an html document from a url, a file on disk or a string containing html with html().
  • Select parts of a document using css selectors: html_nodes(doc, "table td") (or if you’ve a glutton for punishment, use xpath selectors with html_nodes(doc, xpath = "//table//td")). If you haven’t heard of selectorgadget, make sure to read vignette("selectorgadget") to learn about it.
  • Extract components with html_tag() (the name of the tag), html_text() (all text inside the tag), html_attr() (contents of a single attribute) and html_attrs() (all attributes).
  • (You can also use rvest with XML files: parse with xml(), then extract components using xml_node(), xml_attr(), xml_attrs(), xml_text() and xml_tag().)
  • Parse tables into data frames with html_table().
  • Extract, modify and submit forms with html_form(), set_values() and submit_form().
  • Detect and repair encoding problems with guess_encoding() and repair_encoding().
  • Navigate around a website as if you’re in a browser with html_session(), jump_to(), follow_link(), back(), forward(), submit_form() and so on. (This is still a work in progress, so I’d love your feedback.)

 

While Hadley Wickham seems busy with reading excel files ( see https://github.com/hadley/readxl) maybe using rvest can help in more sports analysis now!

https://decisionstats.com/2013/04/25/using-r-for-cricket-analysis-rstats-ipl/

Meanwhile I am searching for equivalent of readHTMLtable

Turing Connection

  • I was born 23 years to the day Alan Turing died (gasps).
  • 23 is a number that is considered magical in (older) hacker circles. http://en.wikipedia.org/wiki/23_enigma
  • I donot believe in magic though I loved Harry P. I was disappointed at the lack of diversity in Hogwarts. No brown wizards?
  • I write on cryptography sometimes when I am bored https://decisionstats.com/2013/12/14/play-color-cipher-and-visual-cryptography/ However I dont have the time , money or Christopher(ine) in my life to pursue it more.
  • I am hopeless in calculus though. I almost always need to visualize something ( sometimes with my eyes closed). Hence my focus on Graphical User (Interfaces).
  • Sometimes I slur my speech
  • I did long distance running during my army cadet days in NDA. I hated it though

AND

  • I think I really like this guy to win an Oscar and will forgive the insults to Gary Oldman over the years

Benedict-Cumberbatch-Alan-Turing

 

Movie Review – Imitation Game

This is a brilliant movie by the talented Benedict Cumberbatch playing the genius Alan Turing and his war time adventures. It is also a tribute to the  injustice served by gay people  by then notions of British Justice and is a dramatized version of the actual true story of the pioneers of cryptology. Alas, their legacy is now in the hands of the GCHQ and NSA to establish digital tyranny and an empire in cyber-space.

Knowing Mr Turing ‘s standards , the world suffered when he was cut down in his prime by pressures brought by official governmental machinery after they had used him through for their own ends.

Watch it and admire the father of Computer Science.

Denial of Freedom to Laugh

What is common between somebody hacking a movie studio to stop the release of a critical movie and somebody killing some cartoonists in Paris. Both are examples of terrorism (one digital, the other physical)

Both deny your freedom to laugh at silly stuff.

This is the same reason I deplore denial of service attacks on websites that are critical to your ideas ( with the possible exception of paedophiles). Terror recruiting websites should be monitored by analytics not shut down by attacks.

The difference between the good guys and the bad guys is not just the reasons they do nasty stuff, it is also in their methods.

The internet needs more freedom not more regulation. Attempts to do so shall just cause activists to create technology that bypass future web anaytics in a more damaging way that current Tor projects do.

Creation of browsers that have both bit torrents and tor embedded by default could be one such example.Screenshot from 2015-01-16 20:08:49

Everybody is an hacker ?

Because I keep getting trolled-

The Hacker Attitude

1. The world is full of fascinating problems waiting to be solved.
2. No problem should ever have to be solved twice.
3. Boredom and drudgery are evil.
4. Freedom is good.
5. Attitude is no substitute for competence.

and

The basic difference is this: hackers build things, crackers break them.

Hacker-logo.sh-600x600

 

Open Source Intelligence and Data Scientists

In view of recent events-a note on OS Intelligence

Open–source intelligence (OSINT) is intelligence collected from publicly available sources. In the intelligence community (IC), the term “open” refers to overt, publicly available sources (as opposed to covert or clandestine sources); it is not related to open–source software or public intelligence

from http://en.wikipedia.org/wiki/Open-source_intelligence

My  Will’s views on data scientists working with people in intelligence..

Will: Why shouldn’t I work for the N.S.A.? That’s a tough one, but I’ll take a shot.

Say I’m working at the N.S.A. Somebody puts a code on my desk, something nobody else can break. Maybe I take a shot at it and maybe I break it. And I’m real happy with myself, ’cause I did my job well.

But maybe that code was the location of some rebel army in North Africa or the Middle East. Once they have that location, they bomb the village where the rebels were hiding and fifteen hundred people that I never met and that I never had no problem with get killed.

Now the politicians are sayin’, “Send in the marines to secure the area” ’cause they don’t give a shit. It won’t be their kid over there, gettin’ shot. Just like it wasn’t them when their number was called, ’cause they were pullin’ a tour in the National Guard. It’ll be some kid from Southie takin’ shrapnel in the ass.

And he comes home to find that the plant he used to work at got exported to the country he just got back from. And the guy who put the shrapnel in his ass got his old job, ’cause he’ll work for fifteen cents a day and no bathroom breaks.

Meanwhile he realizes the only reason he was over there in the first place was so we could install a government that would sell us oil at a good price. And of course the oil companies used the skirmish over there to scare up domestic oil prices. A cute little ancillary benefit for them but it ain’t helping my buddy at two-fifty a gallon.

They’re takin’ their sweet time bringin’ the oil back, and maybe even took the liberty of hiring an alcoholic skipper who likes to drink martinis and fuckin’ play slalom with the icebergs, and it ain’t too long ’til he hits one, spills the oil and kills all the sea life in the North Atlantic.

So now my buddy’s out of work and he can’t afford to drive, so he’s walking to the fuckin’ job interviews, which sucks ’cause the schrapnel in his ass is givin’ him chronic hemorroids. And meanwhile he’s starvin’ ’cause every time he tries to get a bite to eat the only blue plate special they’re servin’ is North Atlantic scrod with Quaker State.

So what did I think? I’m holdin’ out for somethin’ better. I figure, fuck it, while I’m at it, why not just shoot my buddy, take his job and give it to his sworn enemy, hike up gas prices, bomb a village, club a baby seal, hit the hash pipe and join the National Guard? I could be elected president.

from –

https://decisionstats.com/2014/11/26/why-data-scientists-should-refuse-to-work-with-the-nsa/