Lead Data Engineer - Artificial Intelligence
How you will contributeYou will oversee data engineering within the company. Your role is to take the algorithms they have, and make their system scale to huge datasets, as well as build it into a full platform suite.The successful candidate will be responsible for:
- You will be spending the majority of your time writing code, but will also provide leadership on all matters data engineering in the team.
- Making the codebase production-ready and able to handle terabytes of data per second
- Making the system scale to work on real time news, streaming at multiple objects per second, on huge databases e.g. Twitter firehose
The ideal candidate should have experience with most of the following:·You possess a good understanding of both machine learning and production-ready software engineering practices. ·Solid Data Engineering experience·Excellent knowledge of Python and engineering practices involved in scaling systems, including server management etc.·You are familiar with a combination of the following technologies: Spark, Hadoop, ElasticSearch, Go, Docker, Kubernetes, Python, AWS·DevOps and the infrastructure aspects of building a scalable system, including experience deploying cloud services using container infrastructure, auto-scaling, monitoring·You have worked on architecture handling 10TB a day
It would be beneficial if you had the following:·Experience working in a fast-paced startup environment·The ability to adapt quickly to new requirements and communicate them·A track record of leading in an Agile way·Experience working with NLP and an understanding of the concepts behind it·Any previous experience in applying machine learning to novel problems
What’s on offer?
Our client is offering a very competitive salary depending on your level of experience, as well as a company share scheme.
This job was originally posted as www.cwjobs.co.uk/job/79971863