Challenge
The reinsurer was moving its data workflows off Azure Data Factory and onto Azure Databricks. The actuarial analytics team was expected to build and maintain pipelines on the new platform itself, not hand requirements to someone else. The target way of working was specific: pipelines defined as configuration through the Lakeflow Framework, deployment as Declarative Automation Bundles, version control in Git, and data quality expectations and data classification built in from the start. The team's starting point was a long way from that. Delegates knew SQL and the Data Factory approach to orchestration, but Databricks was new to the organisation and the team was new to version control. The team also wanted MLflow covered in a way that fits actuarial work. That meant tracking machine learning experiments, and also bringing version history and audit trails to assumption setting and catastrophe model outputs. Neither of those is a machine learning model.
Why custom
A standard Databricks data engineering course teaches hand-written PySpark pipelines. This team was adopting a framework where pipelines are YAML and JSON Data Flow Specs. A generic course would have taught a way of working they were not going to use. The order of topics mattered as much as the content. The framework deploys through bundles, and bundles live in Git, so the programme gave Git two half-days starting from first principles, taught bundles before the framework, and had delegates ingest data by hand before automating the same flow through configuration. Each layer was understood before the next one hid it. Labs used a purpose-built synthetic catastrophe-exposure dataset so every exercise felt like actuarial work without touching client data. See our Data Engineering training courses for related public programmes.
What we built
| Module | Focus | Labs |
|---|
| Day 1 — Foundations, environment and Git (part 1) | Data engineering concepts, ETL vs ELT, lakehouse, Azure Data Factory migration path, Delta Lake time travel, medallion architecture, Git working directory through remotes | 6 |
| Day 2 — Git (part 2), Unity Catalog, bundles, framework introduction | Branches, pull requests, merge conflicts, Databricks Repos, Unity Catalog grants and lineage, bundle validate/deploy/run, Lakeflow Framework repository structure | 11 |
| Day 3 — Framework configuration and Databricks SQL | Reading and changing pipeline configs, adding an ingestion source, event logs, window and higher-order functions, JSON handling, SQL Editor vs notebooks | 8 |
| Day 4 — Advanced framework patterns | Auto Loader streaming, SCD Type 2, schema evolution, OPTIMIZE/Z-ORDER/VACUUM, event-log troubleshooting through fix, pull request and redeploy | 5 |
| Day 5 — Data quality, governance and MLflow | Expectations with warn/drop/fail, classification tags through medallion layers, MLflow tracking and registration, MLflow for assumption sets and catastrophe model outputs | 8 |
Labs ran on Azure Databricks provisioned with Terraform. One shared actuarial catalog in Unity Catalog; each delegate had a personal schema. Every Lab used the same synthetic catastrophe-exposure dataset. No client data was used.
| File | Size | What it contains |
|---|
| Premium bordereau | 5,000 policies | One row per policy |
| Claims bordereau | 2,829 claim snapshots | Most claims appear more than once, showing how each claim developed over time |
| Catastrophe events | 6 events | Illustrative events with their date windows |
| Postcode risk lookup | 20 rows | Postcode-to-region and risk band reference data |
Delivery
The programme ran live online from 3 to 7 August 2026, 9am to 5pm AEST, for a cohort of five actuarial analytics delegates, as part of our corporate IT training in Australia. Online delivery shaped the design in three ways. Environment access was confirmed in the first Lab on Day 1, so connection and permission problems surfaced in the first hour rather than halfway through a framework deployment. Git concepts were shown visually before any command line work, which matters more when the trainer cannot see every screen. Grant and masking Labs were done in pairs, with each delegate testing access from a partner's identity. The framework's sample pipelines take 20 to 30 minutes to run. Those Labs used serverless compute, and delegates read a real Data Flow Spec while the runs completed.
Results
- 4.8 / 5 average delegate feedback rating at close
- 5 of 5 delegates completed the Day 4 event-log fix, pull request and bundle redeploy Lab
- First pipeline change shipped through the framework and a pull request within 3 weeks of the programme
Plan a similar programme
Talk through your stack, cohort, and outcomes with an Uptut expert – then map a custom training roadmap for your team.
Book a free consultationClient impact
- The actuarial team works in the same Git, bundle and framework workflow the new platform is built on
- Pipelines that ran in Azure Data Factory have a defined path to config-driven Spark Declarative Pipelines, with quality expectations and classification written alongside them
- Assumption sets and catastrophe model outputs have a documented pattern for version history and audit through MLflow
- Within two weeks of the programme, the team landed its first production Silver table through a Declarative Automation Bundle rather than a notebook-only change
Proof
The Labs forced us to use Git, bundles and the Lakeflow Framework the way our platform team expects — by Day 4 we were diagnosing a broken pipeline, fixing it on a branch and redeploying ourselves.
Actuarial Analyst, Australian reinsurer
Trainer
Trainer details bind in Designer to the trainer reference.
Related programmes
Related courses and consulting bind in Designer to multi-reference fields.
Frequently Asked Questions
Can this programme be adapted to our stack?
Yes. We adapt Labs, tools and sequence to how your Databricks platform actually runs. This programme was built around Azure Databricks, pipelines defined through the Lakeflow Framework, deployment through Declarative Automation Bundles, and version control in Git. That includes teams moving off Azure Data Factory, and teams replacing hand-written notebook pipelines with config-driven Spark Declarative Pipelines.
What is the minimum and maximum cohort size?
For hands-on Databricks training we recommend up to 15 delegates with one trainer, and treat 20 as the ceiling. Live online cohorts do better at the lower end, because diagnosing a broken bundle or pipeline over screen share takes longer than in a room. Larger groups run with a second facilitator or as parallel cohorts.
Live online or in-person, and how do you decide?
We recommend live online when the team is distributed across Australia and the Labs run in a shared cloud environment. This reinsurance team ran five consecutive live online days in AEST. We deliver onsite when network or security constraints make a room easier. The environment was confirmed in the first Lab, so access problems surfaced on Day 1 rather than mid-deployment.
What lead time do you need from brief to delivery?
Typically three to four weeks from first scoping to delivery. That covers curriculum review, lab environment provisioning and cloud access. Framework-based programmes can take longer, because the curriculum has to match the exact tooling and conventions your team will use. This one went through several revisions as the brief sharpened.
Do you build custom Labs, or use standard ones?
Custom. This programme used a synthetic catastrophe-exposure dataset built for the course, with policies, claim snapshots, catastrophe events and a postcode risk lookup. Every Lab felt like actuarial work without touching client data. Labs also include planted faults, such as a merge conflict and a broken pipeline configuration, so delegates practise diagnosis as well as building.
How is training effectiveness measured?
We agree the measures before delivery. Typically that means Lab completion during the programme, a delegate feedback form at close, and a follow-up with the sponsor a few weeks later on what the team now does differently. For framework programmes, the most useful signal is whether the team has shipped its first pipeline change through a pull request and bundle deployment.
.webp)
.svg)
