Senior Data Engineer
1 openings in Mississauga, Canada
Responsibilities:
- Build, deploy and manage cloud based big data solutions that can adequately support the analytics and data science needs
- Build out scalable and reliable ETL pipelines and processes to ingest data from a large number and variety of data sources
- Spearhead the development of systems, architectures, and platforms that can scale to the 3 Vs of Big data (Volume, Velocity, Variety)
- Maintain and optimize the performance of all the data pipelines and ETL processes to ensure accurate, reliable and timely delivery of key insights for decision making
- Work with the data scientists to improve the quality of data by cleaning and preparing subsets of data as preparatory step before deeper analysis
Requirements:
- Bachelors or Masters degree in Computer Science, Software Engineering, Information Technology or other related technical disciplines.
- Experience working with Stream data processing involving technologies such as Kafka, PubSub, Spark, Flink or other equivalent technologies
- Proficient in one of the programming languages - Scala, Python or Java Experience in handling large data sets (multiple TBs) and working with structured and unstructured datasets
- An insatiable passion for working with data, learning new technologies in the Big data landscape
- Excellent communication skills to interact with the Business team to understand the requirements and to solve data related challenges
Bonus Points
- Experience working on Cloud Infrastructure (AWS, Google Cloud)
- A good level of understanding and experience with distributed processing and best engineering practices - monitoring, handling and logging errors, building efficient and reliable streaming data pipelines, continuous integration, performance management
- Familiarity with the inner workings of News / Media Publishing platforms