Senior Data Engineer - Microsoft Fabric at WA Consultants, City of London, 3 Months, £400-£450 per day

Role overview

We are seeking Senior Data Engineers to join a delivery team supporting a major financial-services client's Data Strategy implementation. You will be onsite 4 days a week 1 day remote. Outside IR35

You will design, build, test and deploy data-engineering features, including onboarding new data sources and migrating existing ETL processes from Azure Data Factory and Azure Data Lake Storage to Microsoft Fabric.

Responsibilities

• Enhance an existing metadata-driven ADF ingestion framework to onboard new data sources.
• Develop incremental load processes, watermarks, merge logic and ingestion mappings.
• Migrate ETL processes from ADF and ADLS to Microsoft Fabric Pipelines.
• Integrate Fabric Pipelines through On-Premises Data Gateways.
• Build parameterised templates and reusable Fabric deployment patterns.
• Produce and maintain source-to-target data mappings and technical specifications.
• Promote build artefacts through DEV, TEST, UAT and PROD using established CI/CD and DevOps processes.
• Test new and migrated data pipelines and resolve material defects.
• Support system testing, UAT, defect investigation and retesting.
• Work collaboratively within fortnightly Agile sprint cycles.
• Engage with architects, analysts, testers and business stakeholders to clarify requirements and deliver agreed outcomes.

Essential experience

• Strong hands-on commercial experience with Microsoft Fabric, particularly Fabric Pipelines.
• Strong Azure Data Factory and Azure Data Lake Storage experience.
• Proven experience designing and implementing ETL and data-ingestion pipelines.
• Experience with metadata-driven ingestion frameworks.
• Strong understanding of incremental loads, watermarks and merge patterns.
• Experience integrating cloud data platforms with On-Premises Data Gateways.
• Experience implementing CI/CD and promoting data solutions through multiple environments.
• Strong SQL and data-mapping skills.
• Experience testing data pipelines and resolving data-quality and ingestion defects.
• Ability to produce clear technical documentation and reusable engineering patterns.
• Experience delivering within Agile sprint teams.
• Banking or wider financial-services experience.
• Experience in Python

Desirable experience

• Azure DevOps.
• Experience migrating enterprise data workloads from ADF/ADLS to Fabric.
• Experience working within regulated environments and established security controls.

Advise on rates and availability on application

WA Consultants is an Employment Business and an Employment Agency as described within The Conduct of Employment Agencies and Employment Businesses Regulations 2003.