Role: Lead Data Engineer (Hands-on)

Rate: £425.00 - £470.00

Inside Ir35

Clearance: SC Cleared (actice, used within last 12 months)

Location: Hybrid/London (once every month)

Start date: ASAP

Company Overview: Recruiting for a leading cloud solutions provider specialising in digital transformation and data engineering services. The mission is to empower organisations to unlock their data's full potential through innovative cloud solutions. Seeking a Lead Data Engineer to support the data engineering team in designing and implementing solutions for complex migration projects.

Job Overview: The Lead Data Engineer will oversee the design and implementation of data engineering solutions, leading a team in migrating Legacy Oracle databases to AWS managed databases. Expertise in AWS services like AWS Glue, AWS Managed Flink, S3, AWS RDS, and Lambda is essential for mentoring, troubleshooting, and optimizing data solutions.

Key Responsibilities:

Lead ETL processes using AWS Glue to migrate data from Legacy Oracle databases to AWS managed databases, document databases, and data lakes.
Oversee development and maintenance of scalable data pipelines.
Provide technical leadership and mentorship to Data Engineers.
Collaborate with solution architects, data scientists, and stakeholders.
Optimize data workflows using AWS services.
Ensure adherence to data quality, security, and governance standards.
Automate and monitor data pipelines with proactive alerting mechanisms.
Drive continuous improvement and stay updated with emerging AWS services.
Lead technical discussions and ensure alignment on design and implementation.
Provide regular updates on project status and challenges to senior leadership.

Qualifications:

Proven expertise in designing and developing ETL processes using AWS Glue.
Experience migrating Legacy Oracle databases to AWS managed databases.
Strong knowledge of AWS services (S3, AWS Managed Flink, Lambda, CloudWatch).
Expertise in building and maintaining data lakes, document databases, and relational databases.
Hands-on experience with data pipeline architecture and development.
Proficiency in SQL and Python.
Strong understanding of data governance, quality, and security.
Desirable - AWS certifications (AWS Certified Data Analytics, AWS Certified Solutions Architect).

Preferred Skills:

Experience with big data processing technologies (Apache Spark, PySpark).
Knowledge of DevOps practices and tools.
Familiarity with data warehousing, dimensional modelling, and data architecture best practices.