Outside Spy
Remote (England, United Kingdom)
Data Engineer / Data Engineering / Azure / Databricks / Ingestion / Pipelines / Insurance / Financial Services / 2-3 month contract / Outside IR35 / Fully remote One of our leading clients is looking to recruit a Data Engineer. Location – fully remote / home based Duration – 2-3 months plus possible extension Outside IR35 Role / Responsibilities: Define and build a data lake architecture on Azure (ADLS + Databricks) Implement layered data model (bronze / silver / gold) for ingestion, processing, and reporting Develop ingestion patterns from API sources (batch and streaming) Establish data governance, metadata, and access controls Define storage structures, naming conventions, and schema design Develop API connectors and ingestion pipelines using Python Implement integration services feeding into the Databricks environment Build data transformation logic using PySpark / Python (data cleansing, standardisation, validation) Configure monitoring, logging, and...