Suggest and Implement Improvements :
- Identify areas for enhancement in the existing data pipeline
- Propose and execute modifications to optimize performance and reliability
Understand Stakeholder Requirements:
- Collaborate with stakeholders to gather requirements for data integration
- Assist in integrating datasets into the data warehouse to support dashboard building
Build Data Processing Pipelines:
- Design and construct pipelines for data extraction, transformation, and loading (ETL) from diverse data sources
- Ensure pipelines are efficient and scalable
Communicate with Functions:
- Understand the data needs of various functions within the company
- Ensure the data warehouse is prepared to support these needs
Collaborate on Data Quality:
- Work with data analysts to address data quality and consistency issues
Documentation:
- Develop and maintain documentation for all data engineering processes
- Ensure documentation promotes data discoverability
What You Will Need
Degree in Computer Science, Engineering, or a related field
1-2 years of experience in data engineering, software engineering, or a similar role
Fluency in SQL, Python, or equivalent
Solid understanding of database concepts, including handling and storing structured and unstructured data effectively
Solid understanding of web-scraping and automation
Experience working with big-data technologies
Experience working with data pipeline and ETL frameworks
Excellent communication and teamwork abilities
Strong problem-solving and troubleshooting skills
Knowledge of data modeling and database design principles
Experience with cloud platforms such as AWS or GCP