Python Pypsark/Pandas Engineer

Job description

Position Description:

We are looking for a skilled Python PySpark ETL Engineer with 3-6 years and 8–10 years of experience in building scalable data pipelines and processing large datasets. The ideal candidate should have hands-on expertise in PySpark, strong Python programming skills, and experience working with distributed data processing frameworks.

Your future duties and responsibilities:

Design, develop, and maintain ETL pipelines using Python and PySpark
Process large-scale structured and unstructured datasets efficiently
Work with big data technologies like Hadoop, Spark, and Hive
Optimize data workflows for performance and scalability
Collaborate with data analysts, data scientists, and stakeholders
Implement data quality checks and validation processes
Troubleshoot and resolve data pipeline issues
Ensure best practices in coding, version control, and deployment

Required qualifications to be successful in this role:

Strong proficiency in Python
Hands-on experience with PySpark and Apache Spark
Experience in ETL development and data pipeline design
Good knowledge of SQL and data warehousing concepts
Familiarity with Hadoop ecosystem (HDFS, Hive, etc.)
Experience with cloud platforms (AWS / Azure / GCP) is a plus
Understanding of data formats like Parquet, ORC, JSON
Knowledge of Airflow or other workflow orchestration tools

Skills:

·  Pandas
·  Python
·  Spark SQL
·  Testing
·  Banking

What you can expect from us:

Together, as owners, let’s turn meaningful insights into action.

Life at CGI is rooted in ownership, teamwork, respect and belonging. Here, you’ll reach your full potential because…

You are invited to be an owner from day 1 as we work together to bring our Dream to life. That’s why we call ourselves CGI Partners rather than employees. We benefit from our collective success and actively shape our company’s strategy and direction.

Your work creates value. You’ll develop innovative solutions and build relationships with teammates and clients while accessing global capabilities to scale your ideas, embrace new opportunities, and benefit from expansive industry and technology expertise.

You’ll shape your career by joining a company built to grow and last. You’ll be supported by leaders who care about your health and well-being and provide you with opportunities to deepen your skills and broaden your horizons.

Come join our team—one of the largest IT and business consulting services firms in the world.