Data & Databricks Test Automation Engineer

Data & Databricks Test Automation Engineer

Company Overview
Our client is a global leader in financial and technology-driven services, delivering innovative data solutions to some of the world's largest institutional clients. They are seeking a Test Automation Engineer with a strong focus on Databricks and modern data platforms to ensure the quality, performance, and reliability of their data solutions.

Role Overview
As a Data & Databricks Test Automation Engineer, you will be responsible for developing and implementing automated testing frameworks across Databricks-based data platforms. You'll collaborate closely with data engineering teams to validate pipelines, ensure data quality, and strengthen the integrity of a modern Lakehouse architecture.

Key Responsibilities

Databricks Testing

• Design and implement automated testing for Databricks notebooks and workflows.

• Develop test frameworks for Delta Lake tables and ACID transactions.

• Build automated validation processes for structured streaming pipelines.

• Validate Delta Live Tables implementations.

Data Pipeline Testing

• Automate testing for ETL/ELT processes within Databricks.

• Implement Spark job testing and performance validation.

• Create and manage test cases for data ingestion and transformation workflows.

• Test Unity Catalog configurations, access controls, and governance models.

Quality Assurance

• Design and execute data quality test strategies and reconciliation processes.

• Implement performance testing for large-scale Spark jobs.

• Ensure compliance with internal data governance and quality standards.

Monitoring & Reporting

• Develop test monitoring frameworks and dashboards.

• Automate quality reporting and produce actionable test metrics.

• Maintain clear test documentation and version control across projects.

Requirements & Qualifications

Education

• Bachelor's degree in Computer Science, Data Science, Engineering, or related field.

• Certifications in Databricks or data testing tools are advantageous.

Technical Skills

• 2+ years' hands-on experience with Databricks (Spark).

• Strong programming experience with Python (PySpark) and SQL.

• Exposure to data testing frameworks and tools.

• Familiarity with AWS services (S3, Glue, Lambda) or similar cloud platforms.

• Knowledge of Delta Lake and Lakehouse architectures.

• Proficiency with version control systems such as Git.

Additional Skills

• Strong analytical and problem-solving mindset.

• Experience in large-scale data processing environments.

• Understanding of data governance, compliance, and data quality best practices.

• Previous experience working within Agile or DevOps teams.

Platform Knowledge

• Databricks workspace and notebook development.

• Delta Lake and Delta Live Tables.

• Unity Catalog testing for governance and permissions.

• Spark optimization and performance analysis.