Month: January 2020

Is R becoming Irrelevant compared to Python in Data Science

Karma is a bitch. The language R which once claimed to bury SAS language in data science is increasingly losing share of market and share of mind in data science.

TensorFlow. Deep Learning. Advanced Text mining. Computer Vision. Cloud data science pipelines.

All embrace Python faster than R.

Is R still Relevant

See http://r4stats.com/articles/popularity/

and https://www.kdnuggets.com/2018/05/poll-tools-analytics-data-science-machine-learning-results.html

and https://www.kdnuggets.com/2019/05/which-deep-learning-framework-growing-fastest.html

 

A project is more than just a Kaggle dataset

a few criteria that define a good data science project

  • Learnability- What did you learn in the Project
  • Capability – What capabilities were showcased in the project
  • Difficulty- How difficult or easy was the project
  • Potential Hireability- How likely are you going to be hired based on that project
  • Ability- What creative approaches did you bring to the solution

 

A few datasets I liked only from a teaching purpose- iris, Boston, mtcars, Titanic, German Credit and mnist handwriting

A project is more than just a Kaggle dataset. datascience datasets kaggle machinelearning

Saving Dataframe as a table

  • ModelData2=ModelData.toPandas()  #CONVERTS SPARK DF TO PANDAS DF
  • table_model = spark.createDataFrame(ModelData2) # CREATES SPARK DF
  • table_model.write.saveAsTable(‘LIBRARYPATH.model_data’) #SAVES AS TABLE

AND

new_df = transformed_chrn2[[‘Var1’, ‘Var2’, ‘Var3’, ‘Var4′,’Var5’]]

table_df = spark.createDataFrame(new_df)

table_df.write.saveAsTable(‘directory_name.table_name’)

SOURCE

https://stackoverflow.com/questions/30664008/how-to-save-dataframe-directly-to-hive

https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-connect-to-sql-database

https://docs.microsoft.com/en-us/azure/databricks/getting-started/spark/dataframes

https://stackoverflow.com/questions/53212396/pyspark-saveastable-how-to-insert-new-data-to-existing-table