Role: Python & PySpark Data Engineer
Overview
We are looking for a Data Engineer with strong expertise in Python and PySpark to design, build, and optimize scalable data pipelines. You will work with large datasets, distributed systems, and cloud platforms to enable data-driven decision-making.
Key Responsibilities
1. Data Pipeline Development
- Design and build ETL/ELT pipelines using Python and PySpark
- Process large-scale structured and unstructured data
- Ensure high performance and reliability of data workflows
2. Big Data Processing
- Use Apache Spark (especially PySpark) for distributed data processing
- Optimize Spark jobs (partitioning, caching, joins, etc.)
- Handle batch and near real-time data processing
3. Data Integration