Job Title: Senior Data Engineer
Location: London, UK - 100% REMOTE
- Analyse source systems, define underlying data sources and transformation requirements, design suitable data models and document the design/specifications
- Build and maintain robust, scalable, automated cloud-based data pipelines using industry best practises
- Ensure quality of technical solutions as data moves across environments
- Recommend ways to improve data reliability, efficiency, and quality
- Partake in cross team code reviews to ensure all solutions developed meet required standards
- Contribute to the ongoing formation of the Data Engineering team's processes and skill sets
- Excellent coding skills in Python, PySpark and SQL
- Passionate about Test Driven Development
- Create and maintain architecture and systems documentation
- Experience of developing AWS data pipelines using Glue, Step Functions, Lambda to different data stores such as S3, Redshift and RDS.
- Big Data ecosystem data design and processing - including file systems, data structures, automation, security, messaging, movement etc.
- In production cloud data infrastructure - including design, implementation, and maintenance (preferable AWS)
- With SQL development across multiple language variations - PostgreSQL, Amazon Athena
- Good communications skills across all levels of stakeholder (business & technical)
- Continually keep up with advancements in data engineering practises
- 5+ years of experience
- Mentor other members of the team in the principles of data engineering.
- Expertise with version control e.g. Git
- Experience of encrypting and obfuscating sensitive data
- Knowledge of CI\CD best practices
- Experience of deploying Infrastructure as Code preferably using Terraform
- Experience working with both structured and unstructured data in various formats - such as JSON, AVRO, CSV, TSV, PDF etc.