Data engineer- £850 p/d OUTSIDE IR35/Market Risk/12 months/remote at CipherTek Recruitment, ec2a, £600-£850 per day

Senior Data Engineer - Databricks/Spark (High-Performance Platform)

£850 p/d OUTSIDE IR35 (higher for Lead)

12-month rolling (multi-year programme)

1 day/week Max - St Paul's, London

We're hiring a Senior Data Engineer to build a Databricks lakehouse platform in a high-performance, business-critical Front office Risk trading environment.

This is a hands-on engineering role focused on building and optimising large-scale distributed data systems.

This is a highly technical team operating at scale, we're looking for engineers with deep data engineering expertise, strong low-level Spark knowledge, and experience building high-performance systems using modern Databricks and AI-driven platforms.

This is working for a Front office Risk Data team, so you MUST have experience working closely with Front/middle office users and have good domain knowledge covering Market and credit Risk and derivatives

What you'll do

1. Defining technical approach for migrating historic data from existing SQL server instances to Databricks then keeping that data flow on a daily/intraday basis.
2. Building pipelines from bronze to silver/gold/platinum
3. Curation of data products in the market risk, credit risk and historic market data domains.
4. Build and optimise Spark pipelines on Databricks
5. Develop a lakehouse platform (Medallion architecture)
6. Own data modelling, architecture, and pipeline design
7. Work with large-scale data (TB-PB)
8. Drive performance, scalability, and reliability in production

What we're looking for

• Strong experience running Spark workloads in production
• Proven ability to optimise Spark at scale (Tb/PB datasets)
• Solid Python (Scala beneficial, not essential)
• Experience with data modelling and lakehouse architecture
• Ability to debug and improve performance in distributed systems

Important

• Must have recent, hands-on Spark experience
• Databricks strongly preferred (not essential if Spark depth is very strong)
• Experience supporting AI/ML or advanced analytics platforms is a big plus
• Financial services/trading exposure
• Experience in performance-critical environments

Not a fit if

• Primarily BI/reporting focused
• Spark used only at small scale or outside production
• No experience with performance optimisation in distributed systems

Stack

Databricks (Azure), Spark, Delta Lake, Python (+ Scala optional)

Bottom line

We're looking for engineers who can design, build, and optimise Spark-based systems at scale and operate effectively in a performance-critical environment from day one.