Data Engineer

Recruiter
The Bridge Ltd
Location
Leeds
Posted
10 Feb 2018
Closes
15 Feb 2018
Contract Type
Permanent
Hours
Full Time
Data Team - This is the team that thinks outside the box as they put their minds to collecting, storing and analysing mountains of real-time data - all to help them design, build and support their next generation data platform. Keeping their business lean and agile, our client provides the insight, data science and CRM expertise their product teams need to make the right data-driven decisions. What you'll do as a Data Engineer:- Our client have an existing set of data pipelines into and out of the warehouse which are currently created in HiveQL and a company specific DSL. A significant number of these will be transitioned over an ETL tool-based approach with a lot of additional new data sources to be collected and integrated as well. During the transition, the existing pipelines will require support and maintenance, a number of them will require refactoring into their commitidity ETL tooling, and a series of new pipelines based on the new pattern of work will be needed as well. They already have a number of real-time event stream data, but most are focused on per-market flows rather than on customer events. They are starting to offer real-time personalised promotions to our customers and introducing stream based processing using Kafka, Spark Streaming and Kafka streams. How you'll do it:- Good development experience is essential as it's a base skill in this area, along with being a quick learner as the Hadoop ecosystem is constantly evolving. The following would be a requirement: ETL tooling (Talend/Pentaho/Informatica), Spark, good knowledge and experience of Hadoop MapReduce, Hive, HBase and general data processing; & being previously part of an agile delivery team. A key requirement is to show a desire to continuously learn and adapt to new technologies and requirements. Why should you work for them? Our client utilises small, agile and autonomous teams who work collaboratively with a shared purpose. So if you join, you will work with a group of people with raw energy, natural talent, and the kind of spirit that helps them think big, act bold, and change the game. Our client's overall ambition is to attract and develop diverse and talented people to meet their current and future growth needs. Their ultimate aim is to create the best digital business to work at.

Similar jobs

Similar jobs