Data Engineer - Databricks - Outside IR35 - £450 - £525 at Primus Connect Ltd, London, 3 to 6 Months, £450-£525 per day

Databricks Data Engineer

Contract Opportunity | Data & AI Consultancy | Databricks

Contract: Outside IR35
Rate: £450-£525 per day
Location: Fully Remote - UK
Start: ASAP
Focus: Databricks/Data Engineering

The Opportunity

Primus Connect is working with a growing Data & AI Consultancy that is delivering modern data platforms and Databricks solutions for a range of enterprise clients.

They are expanding their engineering capability and are looking for a Mid-Level Data Engineer with strong commercial Databricks and Spark experience to join an active delivery team.

This is a hands-on engineering position where you will take ownership of data pipeline development, work with modern Databricks technologies and collaborate closely with senior engineers and wider delivery teams.

Key Responsibilities

• Design, build and maintain ETL/ELT data pipelines using Databricks, PySpark and Spark SQL
• Develop and optimise Delta Lake tables using Bronze, Silver and Gold medallion architecture
• Ingest data from a variety of source systems into Databricks
• Write clean, maintainable and well-tested Python/data engineering code
• Implement data quality checks, validation and monitoring
• Participate in code reviews and collaborative development
• Contribute to CI/CD pipelines for data engineering workloads
• Work with senior engineers to deliver scalable data platform solutions
• Collaborate with technical, analytics and business stakeholders
• Participate in Agile ceremonies and delivery activities

Essential Experience

• Commercial experience building data pipelines using Databricks
• Strong PySpark and Spark SQL skills
• Solid Python experience within data engineering
• Practical experience with Delta Lake
• Understanding of medallion architecture and modern data platform principles
• Experience working with Azure, AWS or GCP
• Good understanding of Git and collaborative development practices
• Experience working within Agile delivery environments
• Strong communication skills and the ability to work effectively in a remote team

Desirable Experience

• Databricks certification
• Unity Catalog
• Databricks Asset Bundles
• Terraform/Infrastructure as Code
• Databricks Workflows
• Airflow
• Spark Structured Streaming
• Kafka or Azure Event Hubs