Use df.printSchema()
Author: Ajay Ohri
http://about.me/ajayohri
Elt on
Is R becoming Irrelevant compared to Python in Data Science
Karma is a bitch. The language R which once claimed to bury SAS language in data science is increasingly losing share of market and share of mind in data science. 
TensorFlow. Deep Learning. Advanced Text mining. Computer Vision. Cloud data science pipelines.
All embrace Python faster than R.
Is R still Relevant
See http://r4stats.com/articles/popularity/
and https://www.kdnuggets.com/2018/05/poll-tools-analytics-data-science-machine-learning-results.html
and https://www.kdnuggets.com/2019/05/which-deep-learning-framework-growing-fastest.html
Machine Learning Week with 15% Discount Code
A project is more than just a Kaggle dataset
a few criteria that define a good data science project
- Learnability- What did you learn in the Project
- Capability – What capabilities were showcased in the project
- Difficulty- How difficult or easy was the project
- Potential Hireability- How likely are you going to be hired based on that project
- Ability- What creative approaches did you bring to the solution
A few datasets I liked only from a teaching purpose- iris, Boston, mtcars, Titanic, German Credit and mnist handwriting
A project is more than just a Kaggle dataset. hashtagdatascience hashtagdatasets hashtagkaggle hashtagmachinelearning
Saving Dataframe as a table
- ModelData2=ModelData.toPandas() #CONVERTS SPARK DF TO PANDAS DF
- table_model = spark.createDataFrame(ModelData2) # CREATES SPARK DF
- table_model.write.saveAsTable(‘LIBRARYPATH.model_data’) #SAVES AS TABLE
AND
new_df = transformed_chrn2[[‘Var1’, ‘Var2’, ‘Var3’, ‘Var4′,’Var5’]]
table_df = spark.createDataFrame(new_df)
table_df.write.saveAsTable(‘directory_name.table_name’)
SOURCE
https://stackoverflow.com/questions/30664008/how-to-save-dataframe-directly-to-hive
https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-connect-to-sql-database
https://docs.microsoft.com/en-us/azure/databricks/getting-started/spark/dataframes
