More jobs:
Job Description & How to Apply Below
Dear Professionals
Greetings from Tata consultancy Services,
Job Title Pyspark Data Engineer
Experiernce: 6-10 Years
Location:
Chennai / Kolkata / Hyderabad / Pune
Mode of Work :
Work from Office
Job description
Design scalable PySpark-based test architectures for ETL/data pipelines, including modular frameworks for batch processing.
Architect end-to-end data validation systems in Hadoop environment for lineage, schema evolution
Lead system design for Hadoop/Hive test environments, including YARN resource management, dynamic partitioning.
Exposure to Zephyr-Jira-Service Now integrated test management systems with experience on API-driven automation.
Design CI/CD test pipelines for PySpark/Hadoop jobs, incorporating artifact management, parallel execution, and blue-green deployments.
Create data quality system designs using PySpark integrated with Hive metadata services.
Design testing platforms, test data generators
Mentor juniors on PySpark testing basics, contribute to testing strategy discussions
Spark session configurations for memory and core allocations for both local and cluster manager settings
Data handling with distributed file systems like HDFS and writing back to hive tables
Implementation of Partitioning, caching techniques in organizing code for transformation pipelines
Performance tuning implementation like salting, minimizing shuffling
Note that applications are not being accepted from your jurisdiction for this job currently via this jobsite. Candidate preferences are the decision of the Employer or Recruiting Agent, and are controlled by them alone.
To Search, View & Apply for jobs on this site that accept applications from your location or country, tap here to make a Search:
To Search, View & Apply for jobs on this site that accept applications from your location or country, tap here to make a Search:
Search for further Jobs Here:
×