We are undergoing a significant data transformation to ensure that accurate and consistent data is available precisely when and where it's needed.
Responsibilities:
- Write PySpark and SQL scripts to validate data pipelines, transformations, and integrations.
- Design and run tests for data validation, storage, and retrieval using Azure services like Data Lake, Synapse, and Data Factory, adhering to industry standards.
- Continuously enhance automated tests as new features are developed, ensuring Participate in data reconciliation and verify Data Quality frameworks to maintain data accuracy, completeness, and consistency across the platform.
- Formulate and maintain test strategies—including smoke, performance, functional, and regression testing—to ensure data processing and ETL jobs meet requirements.
- Collaborate with development teams to assess changes in data workflows and update test cases to preserve data integrity.
- Share knowledge and best practices by collaborating with business analysts and technology teams to document testing processes and findings.
- Communicate testing progress effectively with stakeholders, highlighting issues or blockers, and ensuring alignment with business objectives.
- Maintain a comprehensive understanding of the Azure Data Lake platform's data landscape to ensure thorough testing coverage.
Requirements:
- Bachelor’s degree in computer science or related discipline
- 3-6 years of hands-on experience in Data Engineering particularly in Data Lake environments on Azure's cloud platform.
- Proficient in Azure Data Factory, Azure Synapse Analytics and Databricks for big data processing and scaled data quality checks.
- Proficiency in SQL, capable of writing and optimizing both simple and complex queries for data validation and testing purposes.
- Proficient in PySpark, with experience in data manipulation and transformation, and a demonstrated ability to write and execute test scripts for data processing and validation.
- Hands-on experience with Functional & system integration testing in big data environments, ensuring seamless data flow and accuracy across multiple systems.
- Knowledge and ability to design and execute test cases in a behaviour-driven development environment.
- Fluency in Agile methodologies, with active participation in Scrum ceremonies and a strong understanding of Agile principles.
- Familiarity with tools like Jira, including experience with X-Ray or Jira Zephyr for defect management and test case management.
- Proven experience working on high-traffic and large-scale software products, ensuring data quality, reliability, and performance under demanding conditions.
- Identify and resolve issues in complex Big Data environments, ensuring data accuracy and system efficiency.
- Initiate and manage tasks, delivering high-quality results within deadlines in a dynamic setting.
- Confidently challenge existing assumptions and processes to enhance data engineering and QA practices.
- Maintain organized testing schedules and documentation, ensuring thorough coverage of all scenarios.
- Exhibit strong interpersonal skills, working effectively in cross-functional, multicultural teams with minimal supervision.
- Comfortable in Agile and Scrum environments, proficient with tools like Jira, and adaptable to changing priorities.
- Manage multiple projects simultaneously, demonstrating strong planning and organizational skills to ensure timely task completion.
We have an amazing team of 700+ individuals working on highly innovative enterprise projects & products. Our customer base includes Fortune 5 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.
What makes Confiz stand out is our focus on processes and culture. Confiz is ISO 9001:2015 (QMS), ISO 27001:2022 (ISMS), ISO 20000-1:2018 (ITSM) and ISO 14001:2015 (EMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.
People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.
To know more about Confiz Limited, visit: https://web.facebook.com/lifeatconfiz
