Big Data Engineer (Pyspark)

London (Greater)
£60000 - £90000 per annum + Stock options.
13 Nov 2017
15 Nov 2017
Contract Type
Full Time
My client is looking for a highly experienced Big Data Engineer (PySpark Developer) to join a diligent team of engineers and lead on provisioning their customer facing PySpark environment.

You will be involved in the design and build of the environment as-well-as keeping the product up to date with the latest AWS versions. You will also be presented with the opportunity to make informed decisions about the future of the product.

One of the core challenges relates to scale. They are currently running Python code in a local environment; they need to scale up from, let's say, 10 TB 100TB which involves re-writing a lot of the original code in PySpark.

You will be expected to be able to write highly efficient PySpark code, as well as having experience with Hadoop, Spark and MapReduce. You should be the sort of person that enjoys product ownership and interacting with multiple teams across engineering, data science and devops.

An ideal candidate would have the following blend of technical skills:
  • PySpark
  • Python including Numpy, Scipy etc
  • A numerical background with knowledge of Mathematics and Statistics running alongside your Engineers/CS education.

As the first hire in this space for my client you will have complete ownership end to end and with plans to scale the team again in about 6 months, it presents an almost immediate leadership opportunity as well.

Allegis Group Limited and Aston Carter Limited operate Employment Businesses and Agencies and are companies within the Allegis Group Inc. group of companies, the fourth largest staffing company in the world, (collectively referred to as the "Allegis Group"). TEKsystems and Aerotek are divisions of Allegis Group Limited. Applicant data will be treated in accordance with the Allegis Group's Privacy Notice By submitting personal data to any company or division within the Allegis Group, the applicant is providing explicit consent to the use of such data by the Allegis Group and to the transfer of such data to and from the Allegis Group companies within the UK, Europe and outside the European Economic Area in connection with the fulfilment of the applicant's voluntary requests, and the fulfilment of other job opportunities that match the applicant's profile, and confirms that they may be contacted about such job opportunities.
This job was originally posted as