Get in Touch
 Duration 35 hours

Course Outline

Databricks Platform and Lakehouse Essentials

  • Databricks Lakehouse architecture and constituent elements
  • Managing workspaces and catalogues

Databricks Workspace and Notebooks

  • Workspace navigation and development via notebooks
  • Organising code into reusable notebook structures

Apache Spark Architecture and Execution

  • Spark runtime architecture and execution mechanics
  • Lazy evaluation and the job DAG

PySpark DataFrames and the DataFrame API

  • DataFrame abstractions and schema definitions
  • Core DataFrame operations and column expressions

Translating SQL to PySpark DataFrames

  • Mapping core SQL clauses to DataFrame operations
  • Utilising window functions and aggregations in PySpark

Reading and Writing Data in Databricks

  • Ingesting data from common file and database sources
  • Writing and partitioning data within the Lakehouse

Delta Lake and Table Management

  • Delta tables and ACID transaction support
  • Time travel features and schema evolution

Data Cleaning and Transformation Patterns

  • Data cleansing and type conversion techniques
  • Developing reusable transformation logic

User-Defined Functions and Modular Code

  • Implementing Python UDFs and pandas UDFs
  • Modularising procedural logic into discrete functions

Performance Tuning and Optimisation

  • Partitioning and caching strategies
  • Identifying bottlenecks using the Spark UI

Structured Streaming Fundamentals

  • Comparing batch versus streaming processing models
  • Working with streaming DataFrames and basic aggregations

Databricks Jobs and Workflow Orchestration

  • Scheduling notebooks as jobs and tasks
  • Constructing multi-step workflows with defined dependencies

Unity Catalogue and Data Governance

  • Unity Catalogue architecture and namespaces
  • Access control mechanisms and data lineage

Testing, Debugging, and Production Practices

  • Unit testing PySpark logic
  • Debugging techniques and maintaining code quality standards

End-to-End Financial Services Use Cases

  • Constructing an end-to-end banking ETL pipeline
  • Translating legacy SQL processes into PySpark

Migrating SQL Workloads to PySpark

  • Migration strategies and planning patterns
  • Incrementally converting SQL workflows to PySpark

Requirements

  • Familiarity with Python programming, encompassing functions and data types
  • A solid understanding of SQL, including joins, aggregations, and subqueries
  • No prior experience with Databricks or PySpark is necessary

Audience

  • Data engineers, analysts, and data professionals
  • Teams in the process of migrating SQL-based workflows to Databricks and PySpark

Custom Corporate Training

Training solutions designed exclusively for businesses.

  • Customised Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
  • Flexible Schedule: Dates and times adapted to your team's agenda.
  • Format: Online (live), In-company (at your offices), or Hybrid.
Investment

Price per private group, online live training, starting from £8000 + VAT*

Contact us for an exact quote and to hear our latest promotions

Testimonials (2)

Provisional Upcoming Courses (Contact Us For More Information)

Related Categories