Master Azure Databricks

About Master Azure Databricks

Module 1 :

What is Data Pipeline

What is Azure databricks

Azure Databricks Architecture

Azure Account Setup

WorkSpace Setup

Module 2:

Navigate the Workspace

Runtimes

Clusters

Notebooks

Libraries

Repos

Databricks File System (DBFS)

DBUTILS

Widgets

Workflows

Metastore - Setup external Metastore

Module 3 :

What is RDD

Creating RDD

RDD transformations

RDD Actions

RDD Joins

Pair RDD

Broadcast Variables

Accumulators

Convert RDD to DataFrame

Import & Read data

Create a table using the UI

Create a table in a Notebook

Module 4 :

Create DataFrames

Define Schema

Functions

Casting Operations

Filter Transformation

Update, Update ALL & UpdateByName

OrderBy & SortBY

GroupBy

Remove Duplicates

Window Functions

Date and Timestamp Functions

UDF (User Defined Function)

JOIN

Handle corrupt records using the badRecordsPath

File metadata column

Module 5 :

Read Parquet File

Read CSV Files

Read JSON Files

Read XML Files

Read Excel file

SQL databases using JDBC

Azure blob storage

Module 6 :

What is Spark Structure Streaming

Data Source & Sink

Rate & File Source

Kafka Source

Sink : Console, Memory, File & Custom

Build Streaming ETL

Stream ETL 1 : Setup Event Hub

Streaming ETL 2 : Event Hub Producer

Streaming ETL 3 : Integrate Event Hubs with Data Bricks

Streaming ETL 4 : Transformation

Streaming ETL 5 : Ingest into Azure Data storage

Twitter Sentiment Analysis - Introduction

Setup Twitter Developer Account

Twitter Sentiment Analysis - II

Twitter Sentiment Analysis - III

Module 7 :

Components in Databricks SQL

Configuring a SQL Endpoint

Creating a Table from a CSV File

Create Queries

Parameterized Query

Query Profile

Building Visualization (Table, BAR & PIE )

Building Line Chart & Counter Chart

Adding Charts to Dashboards

Defining a Query Alert

Access Control on Databricks SQL Objects

Lab: Data Object Access Control

Transfer Ownership

Access SQL Endpoint from Python

Databricks SQL CLI

Databricks SQL CLI