Course Outline
Databricks Platform and Lakehouse Essentials
- Databricks Lakehouse architecture and constituent elements
- Managing workspaces and catalogues
Databricks Workspace and Notebooks
- Workspace navigation and development via notebooks
- Organising code into reusable notebook structures
Apache Spark Architecture and Execution
- Spark runtime architecture and execution mechanics
- Lazy evaluation and the job DAG
PySpark DataFrames and the DataFrame API
- DataFrame abstractions and schema definitions
- Core DataFrame operations and column expressions
Translating SQL to PySpark DataFrames
- Mapping core SQL clauses to DataFrame operations
- Utilising window functions and aggregations in PySpark
Reading and Writing Data in Databricks
- Ingesting data from common file and database sources
- Writing and partitioning data within the Lakehouse
Delta Lake and Table Management
- Delta tables and ACID transaction support
- Time travel features and schema evolution
Data Cleaning and Transformation Patterns
- Data cleansing and type conversion techniques
- Developing reusable transformation logic
User-Defined Functions and Modular Code
- Implementing Python UDFs and pandas UDFs
- Modularising procedural logic into discrete functions
Performance Tuning and Optimisation
- Partitioning and caching strategies
- Identifying bottlenecks using the Spark UI
Structured Streaming Fundamentals
- Comparing batch versus streaming processing models
- Working with streaming DataFrames and basic aggregations
Databricks Jobs and Workflow Orchestration
- Scheduling notebooks as jobs and tasks
- Constructing multi-step workflows with defined dependencies
Unity Catalogue and Data Governance
- Unity Catalogue architecture and namespaces
- Access control mechanisms and data lineage
Testing, Debugging, and Production Practices
- Unit testing PySpark logic
- Debugging techniques and maintaining code quality standards
End-to-End Financial Services Use Cases
- Constructing an end-to-end banking ETL pipeline
- Translating legacy SQL processes into PySpark
Migrating SQL Workloads to PySpark
- Migration strategies and planning patterns
- Incrementally converting SQL workflows to PySpark
Requirements
- Familiarity with Python programming, encompassing functions and data types
- A solid understanding of SQL, including joins, aggregations, and subqueries
- No prior experience with Databricks or PySpark is necessary
Audience
- Data engineers, analysts, and data professionals
- Teams in the process of migrating SQL-based workflows to Databricks and PySpark
Custom Corporate Training
Training solutions designed exclusively for businesses.
- Customised Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
- Flexible Schedule: Dates and times adapted to your team's agenda.
- Format: Online (live), In-company (at your offices), or Hybrid.
Price per private group, online live training, starting from £8000 + VAT*
Contact us for an exact quote and to hear our latest promotions
Testimonials (2)
Showing different ways to optimise our PySpark queries.
Ivan Abrashev - UBB + DZI (KBC GROUP)
Course - Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars
I liked that it was practical. Loved to apply the theoretical knowledge with practical examples.