.webp)
.svg)
Program overview
Online
On-site
Hybrid
ETL/ELT with Apache Spark + Databricks Bootcamp
Build a strong foundation in building scalable ETL/ELT pipelines with Apache Spark and Databricks, from core transformations to Delta Lake lakehouse patterns. Learn how to implement incremental processing, optimize performance, enforce data quality, operationalize pipelines, and troubleshoot real production failures confidently.
Duration:
4 days
Level:
Intermediate
.avif)
1500+ users onboarded
Who will Benefit from this Training?
- Data Engineers
- Cloud Data Engineers
- Analytics Engineers
- Data Platform Engineers
- Backend engineers moving into big data engineering
- BI engineers working with transformation pipelines at scale
Training Objectives
- Understand Spark architecture and why it is a standard for large-scale ETL.
- Build ETL/ELT pipelines using Databricks notebooks and Spark DataFrames.
- Read and write data in CSV, JSON, Parquet, and Delta Lake formats.
- Apply production transformation patterns including cleansing, deduplication, joins, aggregations, and incremental logic.
- Implement scalable patterns using partitioning, file sizing, and performance tuning fundamentals.
- Build reliable pipelines using structured logging, validation checks, and error handling strategies.
- Understand Delta Lake and lakehouse patterns including ACID tables, time travel, and merge/upsert.
- Orchestrate pipelines using Databricks Jobs and understand Airflow integration approaches.
- Deliver an end-to-end ETL/ELT pipeline capstone project with production readiness practices.
Build a high-performing, job-ready tech team.
Personalise your team’s upskilling roadmap and design a befitting, hands-on training program with Uptut
Key training modules
Comprehensive, hands-on modules designed to take you from basics to advanced concepts
- Module 1: Spark Architecture and Execution Model (Why Spark for Large-Scale ETL)
- Why Spark is used (distributed processing, scale-out workloads, fault tolerance)
- Spark components (driver, executors, cluster manager concepts)
- Lazy evaluation and the DAG execution model
- Transformations vs actions and what triggers compute
- Hands-on: Lab: Inspect a Spark job plan and identify stages/shuffles for a sample workload
- Module 2: Databricks Fundamentals (Workspace, Notebooks, Clusters)
- Databricks workspace components and notebook workflow
- Cluster types (interactive vs job clusters) and sizing basics
- Databricks filesystem concepts and data access patterns
- Development workflow (notebook, repos, job runs concept)
- Hands-on: Lab: Create a notebook, start a cluster, and validate Spark session and basic DataFrame operations
- Module 3: Reading and Writing Data (CSV, JSON, Parquet, Delta)
- Reading CSV (schema inference vs explicit schema) and common pitfalls
- Reading nested JSON and flattening selected fields
- Parquet fundamentals (columnar format, schema, performance benefits)
- Delta Lake basics (tables, transactions, metadata)
- Hands-on: Lab: Read CSV + JSON, transform to Parquet, and write curated output as a Delta table
- Module 4: Production Transformations with Spark DataFrames
- Cleansing patterns (null handling, type casting, standardizing dates)
- Deduplication patterns (keys, window functions concepts)
- Joins and join pitfalls (row explosion, skew awareness)
- Aggregations for analytics datasets (daily revenue, KPIs)
- Hands-on: Lab: Build a cleansing + dedup + join + aggregate pipeline for an orders dataset
- Module 5: Incremental Processing Logic (ELT at Scale)
- Full refresh vs incremental processing trade-offs
- Partition-based incremental loads (by date/region/source)
- Watermark strategy concepts (updated_at, ingestion timestamp)
- Handling late-arriving data concept and reprocessing windows
- Hands-on: Lab: Implement an incremental pipeline and validate correct row counts across multiple runs
- Module 6: Delta Lake and Lakehouse Patterns (ACID, Time Travel, Merge)
- Why Delta (ACID transactions, consistent writes, schema enforcement)
- Time travel and table history for auditability
- Merge/upsert patterns for CDC-like workflows
- Building curated dimensions and facts using MERGE
- Hands-on: Lab: Write Delta tables, query versions, and implement MERGE into a dimension table
- Module 7: Scalability and Performance Fundamentals (Partitioning, File Sizing, Tuning)
- Shuffles and why they slow down Spark
- Partitioning controls (repartition vs coalesce) and when to use each
- Small files problem and file sizing strategy
- Intro tuning patterns (caching, broadcast joins concept)
- Hands-on: Lab: Optimize a slow job by reducing shuffle and improving join strategy
- Module 8: Reliability and Quality in Data Pipelines (Logs, Validation, Errors)
- Structured logging patterns (run id, dataset, rows processed)
- Validation checks (schema, nulls, duplicates, ranges)
- Fail-fast vs warn-only strategies and quarantine patterns
- Debugging broken pipelines (schema mismatch, corrupt records, join explosion)
- Hands-on: Lab: Add validation stages, generate a quality report, and fix a broken pipeline scenario
- Module 9: Orchestration with Databricks Jobs (Airflow Integration Overview)
- Databricks Jobs fundamentals (scheduling, parameters, retries, notifications)
- Job cluster vs interactive cluster for production
- Parameterizing pipelines (run_date, env, paths)
- Airflow integration approaches (trigger job runs, dependency chaining concepts)
- Hands-on: Lab: Convert a notebook pipeline into a scheduled Databricks Job with retries and parameters
- Module 10: Capstone Project (End-to-End ETL/ELT Pipeline with Production Readiness)
- Capstone goal: Deliver an end-to-end lakehouse pipeline (raw → cleansed → curated)
- Ingest and transform using Spark DataFrames (CSV/JSON → Parquet/Delta)
- Implement incremental processing and Delta MERGE for curated tables
- Add validation checks, structured logging, and error handling
- Operationalize using Databricks Jobs and provide runbook notes
- Hands-on: Capstone Lab: Deliver working notebooks/jobs, Delta tables with history, and KPI query outputs
Hands-on Experience with Tools
No items found.
No items found.
No items found.
Training Delivery Format
Flexible, comprehensive training designed to fit your schedule and learning preferences
Opt-in Certifications
AWS, Scrum.org, DASA & more
.avif)
100% Live
on-site/online training
.avif)
Hands-on
Labs and capstone projects
.avif)
Lifetime Access
to training material and sessions
Skill-Gap Assessment
Analysing skill gap and assessing business requirements to craft a unique program
1
Personalisation
Customising curriculum and projects to prepare your team for challenges within your industry
2
Implementation
Supplementing training with consulting support to ensure implementation in real projects
3
Why ETL/ELT with Spark + Databricks for your business?
- Faster big data processing: Spark handles large-scale transformations efficiently across massive datasets.
- Unified lakehouse approach: Combine data engineering, analytics, and ML on one platform with Databricks.
- Improved pipeline reliability: Delta Lake patterns support ACID transactions and safer incremental loads.
- Better performance optimization: Use partitioning, caching, and tuning to reduce runtime and cost.
- Future-proof data platform: Enable advanced analytics and AI workloads with scalable architectures.
.avif)
Lead the Digital Landscape with Cutting-Edge Tech and In-House " Techsperts "
Discover the power of digital transformation with train-to-deliver programs from Uptut's experts. Backed by 50,000+ professionals across the world's leading tech innovators.
.svg)
.svg)
.svg)
.svg)
.svg)
Recommended Training
Continue your learning journey with these hand-picked courses and bundles
Frequently Asked Questions
2. Will my team get any practical experience with this training?
.svg)
With our focus on experiential learning, we have made the training as hands-on as possible with assignments, quizzes and capstone projects, and a lab where trainees will learn by doing tasks live.
3. What is your mode of delivery - online or on-site?
We conduct both online and on-site training sessions. You can choose any according to the convenience of your team.
4. Will trainees get certified?
Yes, all trainees will get certificates issued by Uptut under the guidance of industry experts.
5. What do we do if we need further support after the training?
We have an incredible team of mentors that are available for consultations in case your team needs further assistance. Our experienced team of mentors is ready to guide your team and resolve their queries to utilize the training in the best possible way. Just book a consultation to get support.