Data Engineer - SQL/Spark/Hadoop/Python
My client is a new breed of consultancy blending proprietary technology and machine learning with a deep expertise and pedigree in data-based insights.
They are creating a SaaS platform that interprets complex data and tells stories about customer passions and behaviour and they are looking for a Data Engineer to push boundaries with what is possible with their data; to design, develop and automate a data pipeline that will collect data from numerous transactional, social media and geographical data sets as they build the next generation of their product.
- Applying systems architecture and software design skills in the development of data pipelines
- Design and develop new algorithms for extracting insight from social data
- Develop crawlers to extract data from the web or APIs
- Develop infrastructure around existing internal tools to enhance capabilities and improve data flow.
- Implement statistical models and algorithms including clustering on large scale graph data
Some of the skills and technologies you'll need experience of:
Hadoop, PostgreSQL, Spark, SQL, Pig, Python, Pyspark
This job was originally posted as www.jobsite.co.uk/job/960128594