Databricks SC Cleared Data Engineer at IO Associates, Multiple Locations, £550-£650 per day

Contract: Outside IR35
Location: UK-based, Hybrid / Remote
Clearance: Active SC clearance required (minimum)
Start: June
Day Rate: Competitive market rates

We are recruiting a contract Databricks Data Engineer for a large-scale UK Government programme delivering critical, cloud-native data platforms. This is a long-term engagement within a high-profile public sector transformation, working with sensitive, mission-critical data at national scale.

You must already hold active SC clearance to be considered.

• Design, build and maintain Databricks data pipelines using Apache Spark (PySpark & Spark SQL)
• Implement Delta Lake / Lakehouse architectures (Bronze, Silver, Gold)
• Develop scalable ELT/ETL workflows for batch and streaming data
• Optimise performance, reliability and cost across cloud data platforms
• Apply data quality, governance and security controls aligned to government standards
• Collaborate with architects, analysts and platform teams in an agile delivery environment
• Contribute to CI/CD pipelines and automated testing practices

• Proven commercial experience as a Databricks Data Engineer
• Strong PySpark and SQL capability
• Hands-on delivery within AWS, Azure or GCP
• Experience with Delta Lake, data lakes and data warehouses
• Strong understanding of data modelling and scalable pipeline design
• Experience with Git, CI/CD and agile ways of working
• Active SC clearance (mandatory)

• Previous UK government or public sector contracts
• Experience with Airflow, Azure Data Factory, AWS Glue or Kafka
• Exposure to Snowflake or hybrid data architectures
• Knowledge of secure, regulated data environments