DevOps Engineer, Big Data, Hadoop, Automation

ResourceFlow Recruitment Ltd
London (Greater)
£70k - £80k pa + Benefits + 5% Bonus
12 Nov 2017
16 Nov 2017
Contract Type
Full Time

DevOps Engineer, Big Data, Hadoop, Automation

We have an urgent requirement for a Big Data DevOps Engineer to provide expert guidance and deliver the following as part of a team:

Define and Develop the necessary instruments to enable Dev/Ops within the big data organisation;Define, build, automate, the CI/CD pipeline in terms of Deployment, Code build/Promotion, support to test automation, configuration management.Comply with the framework and Dev/Ops approach defined by the Big Data Platform central team which defines the Dev/Ops and Automation strategy, tools, ways of working to be applied in every Local Market.Apply and promote Dev/Ops practices in the organisation helping other teams (eg. QA/Data Engineering) to adopt them and contribute to their evolution.Contribute to the Quality Assurance of the Big Data practice focussing on processes and automation.

Key accountabilities and decision ownership:

  • Implement / configure specific solution as part of the Dev/Ops and Automation Strategy
  • Maintain the Dev/Ops process up to date and aligned with the Big Data delivery portfolio and Operating Model
  • Proactively work with the Data Engineering/QA and the other delivery teams as one squad to make sure Dev/Ops practices are in place at the various stages of the Delivery lifecycle
  • Support and contribute, with the central team of Data Engineering/QA/Data Science, to define best practices for the agile development of applications to run on the Big Data Platform in an efficient CI/CD pipeline

Core competencies, knowledge and experience:

  • Proven experience in designing, building and managing applications to process large amounts of data in a Hadoop ecosystem;
  • Proven experience with performance tuning applications on Hadoop and configuring Hadoop systems to maximise performance;
  • Proven Experience with setting up and running the development life-cycle for agile software development projects in CI/CD pipeline
  • Experience in deployment automation (eg. Ansible), test automation, Cloud implementations and/or containers (eg. Docker/Kubernetes/Openshift)
  • Consistent experience in team working, particularly in a Scrum/Agile way, working in a "multi skill" squad, and in a multi-cultural environment

Must have technical / professional qualifications:

  • Strong ability in configuring/programming automation (yaml, Ansible playbook, and scripting)
  • Deep knowledge of CI/CD tools like Jenkins, Go, Mavel, Git Hub
  • Some experience with Cloud implementation and / or use of Virtual Machines/Containers (eg. Docker Kubernetes)
  • Experience with using Spark, Yarn, Hive and Oozie;
  • Full understanding and consistent experience with the entire software delivery lifecycle and agile methodology.

This job was originally posted as