.webp)
.svg)
Program overview
Online
On-site
Hybrid
Databricks Data Engineering and Pipelines
Build Databricks pipelines with notebooks, PySpark, and Delta Lake - from ADLS Gen2 ingest through medallion layers to Auto Loader, MERGE patterns, and production Jobs.
Duration:
2 Days
Level:
Advanced
.avif)
1500+ users onboarded
Who will Benefit from this Training?
Training Objectives
Build a high-performing, job-ready tech team.
Personalise your team’s upskilling roadmap and design a befitting, hands-on training program with Uptut
Key training modules
Comprehensive, hands-on modules designed to take you from basics to advanced concepts
Download Curriculum
- Databricks Notebooks and PySpark for Pipeline Work
- Notebook cell types, magic commands (%sql, %python, %md), and widgets for parameterisation
- PySpark DataFrames for analysts: transformations versus actions
- Common DataFrame operations used in pipeline notebooks
- Organising notebook code for reusable Bronze-to-Gold steps
- Hands-on: Build and parameterise a pipeline notebook with SQL and Python cells
- Ingesting Data from ADLS Gen2 into Delta
- Reading from ADLS Gen2 with the abfss:// protocol
- Loading CSV, JSON, and Parquet into DataFrames
- Writing to Delta tables with append, overwrite, and merge modes
- Choosing write patterns for raw landing versus curated layers
- Validating row counts and table schemas after ingest
- Medallion Architecture: Bronze, Silver, and Gold
- Medallion design principles for lakehouse data engineering
- Bronze: land raw ADLS Gen2 data into Delta with minimal transformation
- Silver: cleanse, deduplicate, standardise, and cast types
- Gold: business aggregations and wide tables ready for reporting and Power BI
- Hands-on: Implement a Bronze-to-Silver-to-Gold pipeline on a sample ADLS Gen2 dataset
- Schema Evolution and Bad-Record Handling
- Schema enforcement versus schema evolution: when to allow change
- Configuring schema behaviour for evolving source files
- Corrupt-record modes: PERMISSIVE, DROPMALFORMED, and FAILFAST
- Quarantining bad records without failing the entire pipeline
- Practical rules for production schema change management
- Auto Loader for Incremental Ingestion
- Streaming ingestion from ADLS Gen2 into Delta with Auto Loader
- Auto Loader versus batch COPY INTO: when to use each
- Schema inference and evolution with cloudFiles.schemaLocation
- Using the rescue column for unexpected fields
- Operational considerations for continuous and triggered streaming jobs
- Incremental Loads and MERGE INTO for CDC
- Full load versus incremental versus Change Data Capture patterns
- MERGE INTO for CDC-style upserts in a single statement
- Handling inserts, updates, and deletes against Delta targets
- Idempotent pipeline design to avoid duplicate Gold metrics
- Validating MERGE outcomes with before/after row checks
- Orchestrating Pipelines with Databricks Jobs
- Creating Jobs, adding tasks, and applying cluster policies
- Scheduling pipelines for recurring production runs
- Multi-task workflows: dependencies, parallel tasks, and conditional branching
- Failure handling with retries, on-failure tasks, and email alerts
- Passing parameters and task values across notebook jobs
- Secrets, Identity, and Secure Storage Access
- Creating secret scopes and storing credentials safely
- Referencing secrets from notebooks and Jobs without hardcoding
- Managed identities and service principals for ADLS Gen2 access
- Least-privilege patterns for pipeline identities
- Checklist for removing secrets from source-controlled notebooks
- Monitoring, Cost Control, and Production Ops
- Using job run history, cluster logs, and event logs for troubleshooting
- Job clusters versus all-purpose clusters for pipeline cost efficiency
- Sizing clusters and enabling auto-termination for idle workloads
- Detecting common failure modes: schema drift, bad files, and permission errors
- Operational checklist for handing pipelines to production support
- Capstone: Scheduled Incremental Pipeline to Gold
- Ingest new files from ADLS Gen2 with Auto Loader into Bronze
- Apply Silver cleansing and Gold aggregations with MERGE where needed
- Orchestrate the flow as a multi-task Databricks Job on a schedule
- Secure credentials with secrets or managed identity patterns
- Hands-on: Deliver a scheduled incremental pipeline from ADLS Gen2 to a Gold Delta table
Hands-on Experience with Tools
Training Delivery Format
Flexible, comprehensive training designed to fit your schedule and learning preferences
Opt-in Certifications
AWS, Scrum.org, DASA & more
.avif)
100% Live
on-site/online training
.avif)
Hands-on
Labs and capstone projects
.avif)
Lifetime Access
to training material and sessions
Skill-Gap Assessment
Analysing skill gap and assessing business requirements to craft a unique program
1
Personalisation
Customising curriculum and projects to prepare your team for challenges within your industry
2
Implementation
Supplementing training with consulting support to ensure implementation in real projects
3
Why this course
- Reliable lakehouse pipelines: Design Bronze-to-Gold flows on Delta that stay consistent from ingest to reporting.
- Incremental by default: Use Auto Loader and MERGE INTO patterns instead of brittle full reloads.
- Production orchestration: Schedule multi-task Databricks Jobs with dependencies, retries, and alerts.
- Secure operations: Manage secrets, identities, monitoring, and cost controls for pipeline workloads.
Training objectives
- Build data transformation pipelines using Databricks notebooks and PySpark
- Implement incremental data loading patterns using Delta Lake MERGE INTO and Auto Loader
- Design and schedule multi-task workflows using Databricks Jobs
- Apply the medallion architecture (Bronze, Silver, Gold) to structure data engineering pipelines
- Ingest data from Azure Data Lake Storage Gen2 into Delta tables across pipeline layers
- Handle schema evolution, bad records, and data quality in pipelines
- Manage secrets and credentials securely in Databricks pipeline code
- Monitor, troubleshoot, and maintain production pipelines
Who will benefit
- Data Engineers
- Data Analysts
- Analytics Engineers
- BI Professionals
- Cloud Engineers
- Data Platform Teams
.avif)
Lead the Digital Landscape with Cutting-Edge Tech and In-House " Techsperts "
Discover the power of digital transformation with train-to-deliver programs from Uptut's experts. Backed by 70,000+ professionals across the world's leading tech innovators.
.svg)
.svg)
.svg)
.svg)
.svg)
Recommended Training
Continue your learning journey with these hand-picked courses and bundles
Frequently Asked Questions
2. Will my team get any practical experience with this training?
.svg)
With our focus on experiential learning, we have made the training as hands-on as possible with assignments, quizzes and capstone projects, and a lab where trainees will learn by doing tasks live.
3. What is your mode of delivery - online or on-site?
We conduct both online and on-site training sessions. You can choose any according to the convenience of your team.
4. Will trainees get certified?
Yes, all trainees will get certificates issued by Uptut under the guidance of industry experts.
5. What do we do if we need further support after the training?
We have an incredible team of mentors that are available for consultations in case your team needs further assistance. Our experienced team of mentors is ready to guide your team and resolve their queries to utilize the training in the best possible way. Just book a consultation to get support.