Data Engineer

Hydrogen Group
13 Feb 2018
16 Feb 2018
Contract Type
Full Time
Data Engineers
6 month contact


After investment over the last 3 years in Hadoop we have most of our core transactional and behavioural data collated in a Cloudera Hadoop cluster. We are now about the significantly widen the scope and usage of the Data Warehouse, and bring in a lot of secondary data sources. We are now in the process of re-building our Batch data-flows into standard ETL tooling and re-building the core ingestion and Feature Store/Mart generation systems.

Key responsibilities:

Data pipeline creation and maintainace

Our client have an existing set of data pipelines into and out of the warehouse which are currently created in HiveQL and a company specific DSL. A significant number of these will be transitioned over an ETL tool-based approach with a lot of additional new data sources to be collected and integrated as well.

During the transition, the existing pipelines will require support and maintenance, a number of them will require refactoring into our commitidity ETL tooling, and a series of new pipelines based on the new pattern of work will be needed as well.

Real-time Data Streaming & Processing

We already have a number of real-time event stream data flows, but most are focused on per-market flows (e.g. changing bet prices in trading) rather than on customer events. We are starting to offer real-time personalised promotions to our customers and introducing stream based processing using Kafka, Spark Streaming and Kafka streams.

Key Experience:

Good development experience is essential as it's a base skill in this area, along with being a quick learner as the Hadoop ecosystem is constantly evolving and the following would be a requirement: ETL tooling (Talend/Pentaho/Informatica), Spark, experience; Good knowledge and experience of Hadoop MapReduce, Hive, HBase and general data processing; previously part of an agile delivery team.

A key requirement is to show a desire to continuously learn and adapt to new technologies and requirements.

Hydrogen UK Limited is acting as an Employment Business in relation to this vacancy.
This job was originally posted as

Similar jobs

Similar jobs