Lost in New York : A R Writer uses code to analyze

I was in New York for past two days. New York is very pretty, very cold and the trains are very confusing. So I ended up walking and going back and forth.

Later on, when I reached home, heat, food, bed and jet lag, I decided to analyze where and what did I see in the visual ephiphany tour.

This was my route

Screenshot from 2016-02-10 06:13:03

This was my R code

a=fromJSON("/home/ajay/Desktop/Takeout/Location History/LocationHistory.json")
mygoog$time=as.POSIXct(as.numeric(b$locations.timestampMs)/1000 , origin="1970-01-01")
#Clearly that is over the API limit for free usage
library(magrittr) #to make code easier to read
fivenum(mygoog$latitude) #tukey
#or using Dr H over Tukey
#deleting Non NY data
#Starting Point
#Starting Time
  #Median Location
#Lingering Location
Mode <- function(x) {
  ux <- unique(x)
  ux[which.max(tabulate(match(x, ux)))]
#Creating New Fields to minimize API calls to Google Maps
 mygoog2=mygoog[!duplicated(mygoog[c("longitude", "latitude")]),]
result <- do.call(rbind,
mygoog2 <- cbind(mygoog2,result)
mygoog2$zipcode <- substr(str_extract(mygoog2$result," [0-9]{5}, .+"),2,6)
#merge(x, y, by=c("k1","k2")) # NA's match
#Cleaning up workspace
Map <- get_googlemap(center = c(lon = median(mygoog$longitude), lat = median(mygoog$latitude)),
                     zoom = 13, 
                     size = c(640, 640), 
                     scale = 2, maptype = c("terrain"), 
                     color = "color")
plot1 <- ggmap(Map) + 
  geom_path(data = mygoog, aes(x = longitude, y = latitude
  alpha = I(0.9), 
  size = 1.8)

Code contains Easter Eggs created by Pretty R at inside-R.org

How I became a social media expert without spamming people


  1. Polite hello gets me more work contracts than aggressive pitches. I am not afraid to say hello, but I always write a crisp two line on why I am saying hello. Short and sweet messaging rules today’s era. Attention span is short. Brevity is the soul of tweet.
  2. There is no such thing as a free lunch or a free connection. People are getting too many emails asking them for stuff. I dont ask for free advice, jobs, or anything in first three exchanges on a digital medium
  3. I try to curb my impatience and listen to people. Everyone has an interesting story.
  4. Internet and Social media change everyday. This will lead to you making mistakes if you are passionate on expanding your network. I make mistakes while learning new things on internet. I learn from mistakes. I make new mistakes next time not repeat them
  5. everyone dislikes spammers. A spam is an unsolicited email a digital cold call that asks the receiver for his time. yes we all have to build our brand, project our knowledge and sell our services. minimize digital spam.
  6. Sense of humor helps build connections. If you can make them smile they are on the hook. 1902893_10152395459325658_5599190800305402465_n
  7. I avoid search engine optimization because I assume people at Google are smarter than me. I use honesty and common sense in my blog writing titles and categories.
  8. Loving my job and my field of expertise is more important than showing my greed for new contracts or clients. As Warren Buffet said , balance your greed and your fear.
  9. When someone is unpleasant to me on Internet, I use block button very fast. That is why I try and not lose my temper at all on internet. It is working better as I am growing older
  10. I am curious to learn new things from new people. 80% of my business has come from my 11000 LinkedIn  connections in past five years.

Dr Eric Siegel updates popular book on Predictive Analytics

Dr Eric Siegel has just released the updated version of his very popular book on Predictive Analytics, Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die

at http://www.predictiveanalyticsworld.com/book/overview.php

The book which is a bestseller in many categories on Amazon has met with overwhelming praise from industry. One of the reasons is that it is chockablock with real case studies that make it much more easy to learn and execute predictive analytics. I frequently recommend it as an additional book when I am teaching data science online. Here is the link to my 2013 review of the earlier edition




Blogging Conflict of Interest Disclaimer – The book author is also founder of Predictive Analytics Conference, a sponsor of this site since many years.

Analyze Wireshark Data in R

Wireshark is the world’s foremost network protocol analyzer. It lets you see what’s happening on your network at a microscopic level. It is the de facto (and often de jure) standard across many industries and educational institutions.


First, we install Wireshark from the terminal.






Type wireshark from terminal.Screenshot from 2016-01-08 16:43:46

Start capture by looking at Capture Tab and interfaces

Screenshot from 2016-01-08 16:44:34


Export data as a csvScreenshot from 2016-01-08 16:45:41


Import file in R to analyze

(from http://www.statmethods.net/input/importingdata.html )

Slideshare for DataScience

I increasingly use Slideshare since the past few years for dumping my Presentations or material I read and want to  share. While Google Docs remains my tool of choice for making Presentations, Slideshare.net is just a one click upload and gets a wide audience for my presentations. I also like just browsing through for stuff as in http://www.slideshare.net/featured/category/data-analytics or searchingScreenshot from 2016-01-02 20:35:53

Plus I can embed it a much easier to read format for a ready to go blog post. Even my latest slideshare on a Py data science tutorial got 8000+ views in a single week ( of Christmas … hmm)


These are my stats (all time and last year). You can get yours at http://www.slideshare.net/insight

Screenshot from 2016-01-02 20:33:33

Screenshot from 2016-01-02 20:33:56



We got the maximum number of views in Year 8 of DecisionStats. Created in 2007, with 191,000 views DecisionStats continues to be one of the largest single author blogs in open source data science.

This year we began our dalliance with Pythonic power.

With the seventh year itch firmly behind us, lets have a rocking 2016. Lets be more honest in 2016!!!

I would like to thank the readers, all 131,258 of you ;)

and the sponsors, Predictive Analytics Conference.



Screenshot from 2016-01-01 13:16:01


Interview Mike Bayer SQLAlchemy #pydata #python

Here is an interview with Mike Bayer, the creator of popular Python package SQLAlchemy.

Ajay (A)-How and why did you create SQLAlchemy?

Mike (M) – SQLAlchemy was at the end of a string of various database abstraction layers I’d written over the course of my career in various languages, including Java, Perl and (badly) in C. Working for web agencies in the 90’s when there were no tools, or only very bad tools, available for these platforms, we always had to invent things.  So the parts of repetition in writing a CRUD application, e.g. those aspects of querying databases and moving their data in and out of object models which we always end up automating, became apparent.

Additionally I had a very SQL-intense position in the early 2000’s at Major League Baseball where we spent lots of time writing “eager” queries and loaders, that is trying to load as much of a particular dataset in as few database round trips as possible, so the need for “eager loading” was also a core use case I learned to value.  Other use cases, such as the need to deal with the database in terms of DDL, the need to deal with SQL in terms of intricate SELECT queries with deep use of database-specific features, and the need to relate database rows to in-memory objects in a way that’s agnostic of the SQL which generated those rows, were all things I learned that we have to do all the time.

These were all problems I had spent a lot of time trying and re-trying to solve over and over again so when I approached doing it in Python for SQLAlchemy, I had a lot of direction in mind already.  I then read Fowler’s “Patterns of Enterprise Architecture” which gave me a lot more ideas for things I thought the ultimate SQL tool should have.

I wrote the Core first and then the ORM on top.   While the first releases were within a year, it took years and years of rewriting, refactoring, learning correct Python idioms and refactoring again for each one,
collecting thousands of end-user emails and issues each of which in some small way led to incremental improvements, as well as totally breaking things for my very early users quite often in the beginning, in order to slowly build up SQLAlchemy as a deeply functional and reliable system without large gaps in capability, code or design quality.

A- What is SQl Alchemy useful for? Name some usage stats on it’s popularity.

M- It’s useful anytime you want to work with relational databases to the degree that the commands you are sending to your database can benefit from being programmatically automated.  SQLAlchemy is scripting and automation for databases.

The site gets about 2K unique visitors a day and according to Pypi we have 25K downloads a day, though that is a very inaccurate number; Pypi’s stats themselves record more downloads than actually occur, and a single user might be downloading SQLAlchemy a hundred times a day for a mutli-server continuous integration environment, for example.   So I really don’t have any number of users, but it’s a lot at this point for sure.

A- Describe your career journey. What other Python packages have you created?

M- The career journey was way longer and more drawn out than it is for most people I meet today, meaning I had years and years of programming time under my belt but it still took an inordinately long time for me to be “good” at it from a formal point of view, and I still have gaps in my abilities that most people I work with don’t.

I only did a few years of computer programming in college and I didn’t graduate.

 Eventually I got into programming in the 90’s because it was a thing I could do better than anything else and due to the rising dot-com bubble in places like NYC it was a totally charged job scene that made it easy to build up a career and income.

But in the 90’s it was much harder to get guidance from better coders, at least for me, so while I was always very good at getting a problem solved and writing things that were more elaborate and complex than what a lot of other people did, I suffered from a lack of good mentors and my code was still very much that awful stuff that only remains inside of a corporate server and gets thrown away every few years anyway.   I was obsessed with improving, though.

After I left MLB I decided to get into Python and the first thing I did was port a Perl package I liked called HTML::Mason to Python, and I called it Myghty.

It was an absolutely horrible library from a code quality point of view, because I was an undisciplined Perl programmer who had never written a real unit test.

Then I started SQLAlchemy, early versions of it were equally awful, then as I slowly learned Python while rewriting SQLA over and over I wrote an all-new Myghty-like template system called Mako, so that nobody would ever have to see Myghty again, then I published Alembic migrations and dogpile.cache.

Along with all kinds of dinky things those are the major Python libraries I’ve put out.

A- Is it better or faster to store data within a RDBMS like MySQL and then run queries to it from Python, or is it better to import data say  to a Pandas like object. What is the magnitude of the difference in speed and computation?

M- That’s a really open-ended question that depends a ton on what kind of data one is working with and what kind of use cases.   I only have a small amount of experience with numpy/pandas but it seems like if one is dealing with chunks of scientifically oriented numerical data that is fairly homogeneous in format, where different datasets are related to each other in a mathematical sense,  the fluency you get from a tool like Pandas is probably much easier to work with than an RDBMS.

An RDBMS is going to be better if you are instead dealing with data that is more heterogeneous in format, with a larger number of datasets (e.g. tables) which are related to each other in a relational sense (e.g. row identity).

RDBMS is also the appropriate choice if you need to write or update portions of the data in a transactional way.

As far as speed and computation, that’s kind of an apples to oranges comparison.   Pandas starts with the advantage that the data is all in memory, but then what does that imply for datasets that are bigger than typical memory sizes or in cases where the datasize is otherwise prohibitive to move in and out of memory quickly, not to mention relational databases can often get their whole dataset in memory too. But then Pandas can optimize for things like joins in a different way than SQL does which may or may not provide better performance for some use cases.

I don’t have much experience with Pandas performance, though I did write a tool some years ago that expresses SQLAlchemy relational operations in terms of Pandas (google for CALCHIPAN); most relational operations except for extremely simple SELECTs and a specific subset of joins did not translate very well at all.

So Pandas might be super fast for the certain set of things you need to do, but for the more general case, particularly where the data spans across a relational structure, you might have fewer bottlenecks overall with regular SQL (or maybe not).

A- What makes Python a convenient language to work with data?

M- To start with, it’s a scripting language; there’s no compile step. That’s what first brought me to it – a language with strong OO that was still scripting.

The next is that it’s an incredibly consistent and transparent / non-mysterious system with a terrific syntax; from day one I loved that imported modules were just another Python object like everything else, rather than some weird ephemeral construct hoisted in by the interpreter in some mysterious way (I’m thinking of Perl’s “use” here).

It is strongly typed; none of those “conveniences” we get from something like Perl where it decided that hey, that blank string meant zero, right?
That Python is totally open source too is something we take for granted now.  I’ve worked with Matlab, which has an awful syntax, but we also had to fight all the time with license keys and license managers and being able to embed it or not and basically copy-protected commercial software implementing a programming language is not a thing that has any place in the world anymore.

I’ve not seen any language besides Python that is scripting, has very good OO as well as a little bit (but not too much) of functional paradigms mixed in, has strong typing, and a huge emphasis on readability and importantly learnability. I’ve never been that interested in learning to write genius-level cleverness in something like Haskell that nobody understands.

If you’re writing code that nobody understands, be very wary – it might be because you’re just so brilliant, or because your code totally sucks, noting that these two things often overlap heavily.

A- What are the key things that a Python package developer should keep in mind ?


Please try to follow as many common conventions as possible.

Use the distutils/setuptools system, have a setup.py file.

Write your docs using Sphinx and publish them on readthedocs.

Make sure you’ve read pep8 and are following most or all of it (and if you’re not, rewrite your code ASAP to do so, don’t wait).

Make sure your code runs on Python 2.7 and Python 3.3+ without any translation steps.

Make sure you have a test suite, make sure it runs simply and quickly and is documented for other people to use, and try to get it on continuous integration somewhere.

Make sure you’re writing small tests that each test just one thing; and verify that a test actually tests the thing it targets by ensuring it fails when that feature is intentionally broken.

Maintain your project’s homepage, bugtracker, mailing list, etc. so that people know how to get to you, and try as hard as possible to be responsive and polite.

Always reply to people, even if it’s to say that you’re sorry you really can’t help them.   There is a significant issue with project maintainers that simply don’t reply to emails or bug reports, or just go missing entirely and leave the whole world wondering for months / years if their critical library is something we need to start forking or not.

A- What is your opinion on in-database analytics ? How can we extend the  principles and philosophy of SQLAlchemy for Big Data Databases and tools

M- I only had a vague notion what this term meant, but reading the Wikipedia page confirmed my notion was the right idea.   The stored procedure vs. app-side debate is a really old one that I’ve been exposed to for a long time.

Traditionally, I’m on the app-side of this.  By “traditional” I mean you’re using something like a SQL Server or Oracle with an app server. For this decision, life is much easier if you don’t put your business logic on the database side.  With the tools that have been around for the last several decades, the stored procedure route is difficult to travel in, because it is resistant to now-essential techniques like that of using source control, organizing code into modules, libraries and dependencies, and using modern development paradigms such as object-oriented or functional programming.

Critically, it forces us to write much more code than when we place the business logic in the app side and emit straight SQL, because the stored procedure’s data, both incoming and outgoing, still has to be marshaled to and from our application layer, yet this is difficult to automate when dealing with a procedure that has a custom, coarse-grained form of calling signature.

Additionally, SQL abstraction tools that are used to automate the production of SQL strings don’t generally exist in the traditional stored procedure world.  Without tools to automate anything, we get the worst of both worlds; we have to write all our SQL by hand on the database side using a typically arcane language like Transact-SQL or PL/SQL, *and* we have to write all the data-marshaling code totally custom to our stored procedures on the app side.

Instead, using modern tools on the app side like a SQLAlchemy we can express data moving between an object model and relational database tables in a very succinct and declarative way without losing any of our SQL fluency for those parts where it’s needed.

Non-traditionally, I think the concept of software embedded in the database could be amazing – note i don’t even want to call it “stored procedures” because already, that implies “procedural development”, which is a dev model that reached its pinnacle with Fortran.

A database like Postgresql allows Python to run within the database process itself, which means that I could probably get SQLAlchemy itself to run within Postgresql.   While I don’t have any time to work on it, I do have a notion of a system where a tool like SQLAlchemy could actually run on both the database side and the app side simultaneously, to produce a Python ORM that actually invokes some portion of its logic on the server.

I would imagine this is already the kind of thing a system like Datomic or Vertica is doing, but I’ve not seen this kind of thing outside of the commercial / JVM-oriented space.


Mike Bayer is the creator of many open source programming libraries for the Python Programming Language, including SQLAlchemy, Alembic MigrationsMako Templates for Python, and Dogpile Caching.

He blogs at http://techspot.zzzeek.org/

SQLAlchemy is an open source SQL toolkit and object-relational mapper (ORM) for the Python programming language released under the MIT License. It gives application developers the full power and flexibility of SQL.


Get every new post delivered to your Inbox.

Join 1,204 other followers