Category: Analytics

Analytic Professionals — Share your views:  Participate in the 2020 Data Science Survey

 

Analytic Professionals — Please participate in the 2020 Data Science Survey.
Please use and share these links:
 
— Access Code:  ZVWX3
           It is OK to share this Access Code with others:  It can be used by multiple people
— Our main Data Science Survey page also has more survey information & FREE downloads of the 2007-2017 Survey Summary Reports
Rexer Analytics has been conducting the Data Science Survey since 2007.  Each survey explores the analytic behaviors, views and preferences of data scientists and analytic professionals.  Over 1100 people from around the globe participated in the 2015 and 2017 surveys.  A report summarizing the 2020 Survey findings will be available free to everyone in 4Q-2020.
The survey should take approximately 20 minutes to complete.  Your responses are completely confidential.

A project is more than just a Kaggle dataset

a few criteria that define a good data science project

  • Learnability- What did you learn in the Project
  • Capability – What capabilities were showcased in the project
  • Difficulty- How difficult or easy was the project
  • Potential Hireability- How likely are you going to be hired based on that project
  • Ability- What creative approaches did you bring to the solution

 

A few datasets I liked only from a teaching purpose- iris, Boston, mtcars, Titanic, German Credit and mnist handwriting

A project is more than just a Kaggle dataset. datascience datasets kaggle machinelearning

Saving Dataframe as a table

  • ModelData2=ModelData.toPandas()  #CONVERTS SPARK DF TO PANDAS DF
  • table_model = spark.createDataFrame(ModelData2) # CREATES SPARK DF
  • table_model.write.saveAsTable(‘LIBRARYPATH.model_data’) #SAVES AS TABLE

AND

new_df = transformed_chrn2[[‘Var1’, ‘Var2’, ‘Var3’, ‘Var4′,’Var5’]]

table_df = spark.createDataFrame(new_df)

table_df.write.saveAsTable(‘directory_name.table_name’)

SOURCE

https://stackoverflow.com/questions/30664008/how-to-save-dataframe-directly-to-hive

https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-connect-to-sql-database

https://docs.microsoft.com/en-us/azure/databricks/getting-started/spark/dataframes

https://stackoverflow.com/questions/53212396/pyspark-saveastable-how-to-insert-new-data-to-existing-table