We are seeking a passionate and experienced Data Engineering Lead to join our dynamic team, responsible for managing and optimizing our data lake platform.The ideal candidate will have extensive experience in data engineering, particularly in managing complex data pipelines and ensuring data quality, performance, and availability.This role requires a proactive leader who owns end-to-end operations of platform and can drive solutions and improvements while effectively communicating with stakeholders.
Responsibilities:
- Diagnose and resolve pipeline issues while implementing effective solutions to minimise outages, enhance performance, and increase platform resilience.
- Implement and maintain monitoring solutions to assess platform performance, health, and data quality.
- Establish and track KPIs to ensure timely data availability.
- Collaborate with an engineering team to ensure the best practices for data ingestion, storage, and processing in Delta Lake.
- Optimize resource usage and platform costs, ensuring a balance between performance and cost-effectiveness as the platform scales.
- Drive initiatives for data quality assurance, proactively identifying and resolving issues before they impact stakeholders.
- Manage deployments of pipeline changes using Azure DevOps, ensuring consistency and reliability through CI/CD best practices.
- Identify and recommend enhancements to processes, tools, and technologies to improve operational efficiency.
- Drive initiatives for data quality assurance, addressing issues proactively before they affect stakeholders.
- Maintain regular updates with stakeholders, ensuring transparent communication of progress and challenges.
Requirements:
- Bachelor's degree in Computer Science or a related field
- 8+ years of experience in data engineering, with hands-on experience in:
- Should have experience in Apache Spark, Apache NiFi, Databricks, Delta Lake, Azure Synapse Analytics, Azure Data Lake Storage, Azure Data Factory, and Power BI.
- Proven track record in managing and troubleshooting complex data pipelines, with a focus on performance improvement and operational efficiency.
- Experience in setting up and integrating open-source tools for ETL and data visualization/analytics.
- Experience in setting up and maintaining Kubernetes clusters for deploying and managing data applications.
- Experience working in Agile/Scrum (SAFe) environments, participating in sprints, stand-ups, and cross-functional team collaboration.
- Excellent problem-solving skills and the ability to think critically under pressure.
- Strong communication skills, with a knack for presenting information to both technical and non-technical stakeholders.
- Self-driven and accountable, with a proactive mindset and strong sense of ownership.
- Microsoft Azure Data Engineer Associate (DP-203) or Databricks certification is preferred.
- Prior experience working in the telecom industry or supporting large-scale, data-driven telecom platforms is preferred.
We have an amazing team of 700+ individuals working on highly innovative enterprise projects & products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.
What makes Confiz stand out is our focus on processes and culture. Confiz is ISO 9001:2015 (QMS), ISO 27001:2022 (ISMS), ISO 20000-1:2018 (ITSM) and ISO 14001:2015 (EMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.
People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.
To know more about Confiz Limited, visit: https://www.linkedin.com/company/confiz-pakistan/
