Outside Spy
London, UK
A Data Engineer is required for our client to support the migration of data cohorts from the TRIO and Olympus systems to the Kraken platform or ADLS Gen2 storage. The role is to ensure the efficient, secure, and accurate movement of data while maintaining the integrity of existing operations. The role will also include collaborating with architects to build and optimize data pipelines, ensuring scalability and reliability. Key Responsibilities Migration Activities Collaborate with architects on migration requirements. Perform ETL from TRIO and Olympus to Kraken/ADLS Gen2 via Databricks. Validate data accuracy and resolve migration issues. Optimize scripts for efficiency. Data Pipeline Development Design and maintain scalable data pipelines. Integrate with Databricks, Kraken, and other systems. Monitor, troubleshoot, and ensure resilience. Governance and Security Implement data masking, logging, and compliance measures. Support...