Test Automation Engineer​/Lead; Databricks, ETL​/Data AI​/ML Exp

Position: Test Automation Engineer/Lead (Databricks, ETL/Data) with AI/ML Exp.
Test Automation Engineer/Lead (Databricks, ETL/Data) with AI/ML Exp.
8+Years
Louisville, KY / Dallas, TX/ Chicago, IL
Contract

Required Skills:

6+ years of experience in ETL/Data Testing.
3+ years of experience testing Databricks or Spark-based platforms.
Hands-on experience in AI/ML Model Testing or MLOps validation.
Strong coding experience in Python and SQL.

Databricks & Data Engineering Testing
Test Bronze, Silver, and Gold layer data transformations within Databricks environments.
Verify workflow orchestration, scheduling, and dependency management across Databricks jobs and pipelines.
Perform performance, scalability, and reliability validation of Spark and PySpark transformations.
Validate data quality frameworks, rules, and monitoring mechanisms implemented in Databricks.
Ensure data accuracy, completeness, consistency, and lineage across data pipelines.

AI/ML Testing
Validate AI/ML model integrations within enterprise applications and data ecosystems.
Verify model scoring results, prediction outputs, and business rule compliance.
Perform data drift, schema drift, feature drift, and model validation testing.
Validate feature engineering pipelines, training datasets, and inference datasets.
Execute model input/output verification against expected business outcomes and acceptance criteria.
Validate explain ability, traceability, auditability, and regulatory compliance requirements.
Test model deployment workflows using MLflow, Databricks ML, Azure ML, or similar MLOps platforms.
Validate Generative AI and RAG (Retrieval-Augmented Generation) workflows, including prompt accuracy, response quality, grounding, and hallucination detection.

Integration Testing
Perform end-to-end integration testing across:
Databricks
Data warehouses
APIs and microservices
AI/ML platforms
Cloud services
Reporting and analytics systems
Validate data movement and transformations between source systems, data lakes, ML models, and downstream applications.
Perform API testing, contract testing, and interface validation.
Ensure seamless interoperability across enterprise data and AI ecosystems.
Test Automation
Design and develop automation frameworks for:
ETL/Data Pipeline Testing
API Testing
Data Validation
AI/ML Model Validation
Build reusable test utilities, libraries, and accelerators using Python.
Integrate automated test suites into CI/CD pipelines.
Develop automated quality gates and release validation mechanisms.
Generate automated execution reports, dashboards, test metrics, and quality insights.
Support continuous testing and shift-left quality practices across the software delivery lifecycle.

Key Skills
Databricks, Spark, Py Spark
Delta Lake, Medallion Architecture (Bronze/Silver/Gold)
Python
SQL
ETL/Data Warehouse Testing
API Testing
AI/ML Testing
MLflow, Azure ML
GenAI/RAG Validation
CI/CD Integration
Test Automation Frameworks
Data Quality and Data Governance

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.