Managing Consultant Data Engineering – System Limited – Multiple Cities – Pakistan
System Limited invites applications for Managing Consultant Data Engineering in Multiple Cities of Pakistan
Job Title:
Managing Consultant Data Engineering
Responsibilities:
· Design and implement scalable data processing pipelines using Apache NiFi, Apache Spark, and Databricks.
· Develop and manage workflows with Apache Airflow to ensure efficient data transformation and loading processes.
· Utilize AWS services including Lambda, EMR, EventBridge, and Redshift for comprehensive cloud-based data solutions.
· Design and maintain robust event buses, integrating Apache Kafka for real-time data streaming.
· Manage and optimize data storage and analytics using AWS Redshift and Delta Lake for improved data reliability and performance.
· Develop complex SQL queries for data extraction, transformation, and loading (ETL) processes.
· Write clean, efficient, and well-documented Python code.
· Build and maintain APIs for data integration and accessibility.
· Collaborate with cross-functional teams to understand data needs and deliver scalable and efficient data solutions.
· Lead and mentor a team of data engineers, promoting best practices in data engineering and handling technical challenges.
Qualifications:
· Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
· Minimum of 5 years of experience in data engineering with a strong track record in a leadership role.
· Extensive experience with NiFi, Spark, Airflow, AWS Lambda, EMR, EventBridge, Kafka, Redshift, Databricks, and Delta Lake.
· Strong proficiency in SQL and Python.
· Experience in building and optimizing APIs for data operations.
· Solid understanding of data warehousing concepts, ETL processes, and big data technologies.
· Strong problem-solving and analytical skills.
· Excellent leadership and communication skills.
Preferred Skills:
· Additional experience with other AWS services.
· Knowledge of machine learning and data science methodologies.
· Familiarity with containerization and orchestration technologies like Docker and Kubernetes.
