How to set up a communication reading and manipulating files in Azure data lake using Databricks.
Before starting
I assume that you already have an Azure Data Lake. Otherwise, it’s easy to set up using Azure portal or using ARM template described here.
You also need a comma delimited file (csv) in my example I use a tweet file downloaded from Kaggle. I also have a python code where you could convert this file into parquet that you could need in the mount point example.
​
The most simple way reading from Data Lake using notebook and Python.
Cmd 1
Using spark.conf.set("spark.sql.name-of-property", value) to set up or configure configuration to Access keys in your data lake,
​
The result should look something like:
spark.conf.set("fs.azure.account.key.labstoac.dfs.core.windows.net", "QswwB4jNa/TbPweLzzUuBkwA2EAQoxCthLnIHn67RitYLd00F5kjffAtebtGFjDMVMEI2bNIcWPZE+AStPeQ9zA==")
In Databricks notebook add command
spark.conf.set("spark.sql.name-of-property", value)
​
Replace name-of-property with fs.azure.account.key.ReplaceMe.dfs.core.windows.net with the name of your Storage Acount (Lake).
Replace value with the key (key1 or key2) found under Access Keys.
​
​
​
​
Cmd 2
File location is the container were you store your file.
​
file_location = "abfss://raw@labstoac.dfs.core.windows.net/"
​