More jobs:
Pyspark Engineer
Job in
Dallas, Dallas County, Texas, 75201, USA
Listed on 2026-03-04
Listing for:
TATA Consulting Services
Full Time
position Listed on 2026-03-04
Job specializations:
-
IT/Tech
Data Engineer, Big Data
Job Description & How to Apply Below
Associates with 7-10 years' experience in data/ETL testing with hands-on PySpark and exposure to big data tools.
Ability to perform below responsibilities:
1. Design scalable PySpark-based test architectures for ETL/data pipelines, including modular frameworks for batch processing.
2. Architect end-to-end data validation systems in Hadoop environment for lineage, schema evolution
3. Lead system design for Hadoop/Hive test environments, including YARN resource management, dynamic partitioning.
4. Exposure to Zephyr-Jira-Service Now integrated test management systems with experience on API-driven automation.
5. Design CI/CD test pipelines for PySpark/Hadoop jobs, incorporating artifact management, parallel execution, and blue-green deployments.
6. Create data quality system designs using PySpark integrated with Hive metadata services.
7. Design testing platforms, test data generators
8. Mentor juniors on PySpark testing basics, contribute to testing strategy discussions
Roles & Responsibilities
Ability to perform below responsibilities:
1. Design scalable PySpark-based test architectures for ETL/data pipelines, including modular frameworks for batch processing.
2. Architect end-to-end data validation systems in Hadoop environment for lineage, schema evolution
3. Lead system design for Hadoop/Hive test environments, including YARN resource management, dynamic partitioning.
4. Exposure to Zephyr-Jira-Service Now integrated test management systems with experience on API-driven automation.
5. Design CI/CD test pipelines for PySpark/Hadoop jobs, incorporating artifact management, parallel execution, and blue-green deployments.
6. Create data quality system designs using PySpark integrated with Hive metadata services.
7. Design testing platforms, test data generators
8. Mentor juniors on PySpark testing basics, contribute to testing strategy discussions
Salary Range: $100,000 to $120,000 per year
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
Search for further Jobs Here:
×