Data Engineer (PySpark&Airflow)

The project aims at constructing, scaling and maintaining data pipelines for a simulation platform. You will be working on a solution to grant connectivity between AWS s3 and Cloudian s3. A previously completed Proof of Concept used Airflow to spin Spark job for some data extraction and to then expose the collected data via Airflow built-in XComs feature. Further work required productionization of the PoC solution, testing it on scale, or proposing an alternate solution.

As a Data Engineer in Store Ops, you will dive into projects that streamlining retail operations through the use of analytics and ML, by applying your Python, Spark, Airflow, Kubernetes skills.

• Developing heavy data pipelines with cooperation with data scientists and other engineers.
• Working with distributed data processing tools such as Spark, to parallelise computation for Machine Learning and data pipelines.
• Diagnosing and resolving technical issues, ensuring availability of high-quality solutions that can be adapted and reused.
• Collaborating closely with different engineering and data science teams, providing advice and technical guidance to streamline daily work.
• Championing best practices in code quality, security, and scalability by leading by example.
• Taking your own, informed decisions moving a business forward.

Python, PySpark, Airflow, Docker, Kubernetes, Dask, xgboost, pandas, scikit-learn, numpy, GitHub Actions, Azure DevOps, Terraform, Git @ GitHub

• Enhancing the monitoring, reliability, and stability of deployed solutions, including the development of automated testing suites.
• Productionization of new data pipeline responsible for exposing data on demand and improve the performance on production.
• Collaborating with cross-functional teams enhancing customer experiences through innovative technologies.

5 engineers