Executives Placements Data Engineer - Developer - Python - Jobs in South Africa
Executives Placements Data Engineer - Developer - Python - Jobs in South Africa
Job PurposeBuild and support data pipelines and datamarts built off those pipelines. Both must be scalable, repeatable and secure.
Help to facilitate gathering data from a variety of different sources, in the correct format, assuring that it conforms to data quality standards and assuring that downstream users can get to that data timeously.
This role functions as a core member of an agile team.
Responsible for the infrastructure that provides insights from raw data, handling and integrating diverse sources of data seamlessly.
Enable solutions, by handling large volumes of data in batch and real-time by leveraging emerging technologies from both the big data and cloud spaces.
Additional responsibilities include developing proof of concepts and implements complex big data solutions with a focus on collecting, parsing, managing, analysing and visualising large datasets.
Responsibilities
Design and develop data feeds from an on-premise environment into a datalake environment in an AWS cloud environment
Design and develop programmatic transformations of the solution, by correctly partitioning, formatting and validating the data quality
Design and develop programmatic transformation, combinations and calculations to
populate complex datamarts based on feed from the datalake
Provide operational support to datamart datafeeds and datamarts
Design infrastructure required to develop and operate datalake data feeds
Design infrastructure required to develop and operate datamarts, their user interfaces and the feeds required to populate the datalake.
Requirements
Qualifications
IT Degree/Diploma
AWS Certification at least to associate level
Experience
Experience in the Retail environment is advantageous
5+ years experience in Business Intelligence & ETL processes
2+ years experience in Agile (eKanban or Scrum)
2+ years Cloud AWS experience
Essential Knowledge (at least 2 years)
Creating data feeds from on-premise to AWS Cloud.
Support data feeds in production on break fix basis.
Creating data marts using Talend or similar ETL development tool.
Manipulating data using python and pyspark.
Processing data using the Hadoop paradigm particularly using EMR, AWS’s distribution of Hadoop.
Devop for Big Data and Business Intelligence including automated testing and deployment.
Skills
Talend
AWS: EMR, EC2, S3
Python
Business Intelligence Data modelling
SQL
PySpark or Spark (desirable)
How to Apply
For more information and job application details, see; Executives Placements Data Engineer - Developer - Python - Jobs in South Africa
Click here to post comments
Join in and write your own page! It's easy to do. How? Simply click here to return to NGO Jobs in Africa.
