Managing Consultant Data Engineering – Systems Limited – Multiple Cities – Pakistan
System Limited invites applications for Managing Consultant Data Engineering in multiple Cities of Pakistan
Job Title:
Managing Consultant Data Engineering
We are seeking an experienced Data Engineer to join our team in mainland China. The ideal candidate will be proficient in data engineering practices, comfortable with data services, and have extensive experience with AWS. This role requires a hands-on approach to manage and optimize data workflows, ensuring efficient data processing, storage, and integration. Additionally, the candidate must possess strong communication skills in both English and Mandarin/Chinese.
Key Responsibilities:
· At least 5 years of experience as a Data Engineer, with a strong background in data processing and ETL pipeline development.
· Proficiency in SQL, Python, and PySpark.
· Write and manage SQL scripts and stored procedures for data manipulation and transformation.
· Create and manage DAGs in Apache Airflow for orchestrating data workflows.
· Utilize shell scripting for data processing tasks involving CSVs, ZIPs, and other formats.
· Manage data storage and retrieval in AWS S3, Redshift, and Snowflake.
· Implement data migration and transformation tasks within Redshift.
· Export data through stored procedures and Lambda functions to various formats for external consumption.
· Automate notifications and data alerts using AWS SNS.
· Work with AWS services including Glue, Athena, S3, Lambda, and SNS to ensure seamless data operations.
· Utilize Git/Bitbucket for version control and collaborative development.
Required Qualifications:
· Proven experience as a Data Engineer, with a strong background in data processing and ETL pipeline development.
· Proficiency in SQL, Python, and PySpark.
· Hands-on experience with AWS Glue, Glue Catalog, Redshift, and Snowflake.
· Experience in creating and managing DAGs in Apache Airflow.
· Competency in shell scripting.
· Familiarity with AWS services such as S3, Athena, Lambda, and SNS.
· Strong problem-solving skills and the ability to troubleshoot complex data issues.
· Excellent communication skills in both English and Mandarin/Chinese.
· Strong collaboration skills and the ability to work effectively in a team environment.
Preferred Qualifications:
· Experience with data warehousing concepts and best practices.
· Familiarity with data governance and data quality frameworks.
· Experience with CI/CD pipelines and automation tools.
