Big Data Engineer, Hadoop, Spark Streaming OR Flink, Scala

ResourceFlow Recruitment Ltd
London (Greater)
£75k - £85k pa + Bonus + Benefits
09 Nov 2017
16 Nov 2017
Contract Type
Full Time

Big Data Engineer, Hadoop, Spark Streaming OR Flink, Scala

We have an excellent opportunity for an experienced Big Data Engineer, with hands on Scala and Spark Streaming OR Flink programming experience to join our client's in-house data science team.

As a Big Data Engineer, you will join our clients vibrant, skilled and creative product development team to engineer their flagship IT event management product which uses advanced algorithmic methods to cluster events for IT failure, to detect the root cause, and automatically select the best escalation, notification and resolution paths.


  • Design and build data and machine learning pipelines for stream based event processing
  • Refactor algorithms into a distributed processing production environment using apache Spark and MapR
  • Transform all the code and software artefacts from a pre-prototype stage to be production-ready


  • Expert in distributed computing technologies such as Spark, MapR (or Cloudera/Hortonworks/), Hadoop
  • Experience building real time pipeline processing using Spark Streaming (or Apache Flink) and Kafka (or Kinesis)
  • Experience with Scala and Python - you will need to write, test and maintain production-quality Scala and Python code
  • Strong software engineering skills - you will be expected to work on the full software lifecycle: Software design and architecture, unit tests, coding, integration tests, help in building the CI/CD infrastructure, code deployment, application monitoring
  • Understanding of machine learning and advanced statistical techniques
  • Exceptional attention to detail
  • Strong presentation and communication skills. Ability to work in teams with people with different skills and background.
  • Experience with NoSQL Columnar Databases (Hbase, Cassandra, )
  • Keen to explore the growing world of AIOPS

Nice to have/expected to learn

  • Experience/Knowledge of C++
  • Experience/Knowledge of Docker
  • Experience with Unix/Linux systems
  • Experience with AWS (EC2, S3, SQS, )
  • Experience building analytics platforms
  • Experience working with data scientists and software engineers
  • Experience working on data engineering and data analysis
  • BSc, MSc in Computer Science, Engineering, Mathematics, etc

This job was originally posted as