Senior & Principal Data Engineers | Databricks | Outside IR35 at Primus Connect Ltd, London, 3 to 6 Months, £550-£700 per day

Senior/Principal Data Engineer - Databricks - Outside IR35

£550-£700 per day | Outside IR35 | Fully Remote UK | ASAP Start

We're supporting a fast-growing Data & AI Consultancy that is expanding its Databricks engineering capability across several client projects.

We're looking for experienced Senior and Principal Data Engineers with strong hands-on Databricks expertise. Multiple contract positions are available, ranging from senior hands-on delivery through to principal-level technical leadership and architecture.

The Role

Depending on your level, you'll take ownership of designing, building and/or leading modern Databricks data platforms, working across large-scale data engineering, platform architecture, governance and deployment.

You'll work with engineering, architecture and business teams to deliver scalable production-grade solutions while helping establish strong engineering standards across the wider team.

Key Responsibilities

• Design and build production-grade data pipelines using Databricks, PySpark and Spark SQL

• Design and implement Delta Lake/Medallion Architecture solutions

• Lead technical design across ingestion, transformation and data consumption

• Implement and optimise Unity Catalog for governance, lineage and access control

• Build and maintain CI/CD and Infrastructure-as-Code using Terraform/Databricks Asset Bundles

• Optimise Spark workloads for performance, scalability and cost

• Establish engineering standards around testing, deployment and code quality

• Work closely with architects, engineers and client stakeholders

• Mentor and technically support other engineers

• At Principal level, own platform-level architecture and set technical direction across Databricks workstreams

Essential Experience

We're particularly interested in contractors with:

• Strong commercial Databricks experience

• Advanced PySpark and Spark SQL

• Strong understanding of Spark internals and performance optimisation

• Hands-on Delta Lake and Medallion Architecture experience

• Strong Python engineering skills

• Hands-on Unity Catalog experience

• CI/CD and Infrastructure-as-Code experience

• Terraform and/or Databricks Asset Bundles

• Cloud experience across Azure, AWS or GCP

• Experience delivering large-scale production data platforms

• Strong stakeholder communication skills

For the Principal-level positions, you should also have experience setting technical direction, owning platform architecture and operating as the technical authority across large Databricks engagements.

Desirable

• Databricks Professional-level certifications

• Delta Sharing/Lakehouse Federation

• Structured Streaming/Kafka

• MLflow or AI/ML-adjacent data engineering

• Agentic or AI-driven data engineering patterns

• Previous consultancy or contract delivery experience

Contract Details

Senior Data Engineer: £550-£625 per day
Principal Data Engineer: Up to £700 per day
IR35: Outside IR35
Location: Fully Remote - UK
Start: ASAP
Contract: 3-6 Initial with potential for extension

If you're an experienced Databricks Data Engineer available for a new contract, apply with your latest CV or contact Tom Fielding at Primus Connect for more information.