Senior Python Developer & Lead SDET

Senior Data Engineer & Technical Lead (SDET Lead)

Transcend IT Solutions

The Senior Data Engineer & Technical Lead (SDET Lead) will play a pivotal role in delivering major data engineering initiatives within the Data & Advanced Analytics space. This position requires hands‑on expertise in building, deploying, and maintaining robust data pipelines using Python, PySpark, and Airflow, as well as designing and implementing CI/CD processes for data engineering projects.

• Design, develop, and optimize scalable data pipelines using Python and PySpark for batch and streaming workloads.
• Build, schedule, and monitor complex workflows using Airflow, ensuring reliability and maintainability.
• Architect and implement CI/CD pipelines for data engineering projects using Git Hub, Docker, and cloud‑native solutions.
• Apply test‑driven development (TDD) practices and automate unit/integration tests for data pipelines.
• Implement secure coding best practices and design patterns throughout the development lifecycle.
• Work closely with Data Architects, QA teams, and business stakeholders to translate requirements into technical solutions.
• Create and maintain technical documentation, including process/data flow diagrams and system design artifacts.
• Lead and mentor junior engineers, providing guidance on coding, testing, and deployment best practices.
• Analyze and resolve technical issues across the data stack, including pipeline failures and performance bottlenecks.

• Minimum 5+ years of practical experience building production‑grade data pipelines using Python and PySpark.
• Airflow expertise:
  Proven track record designing, deploying, and managing Airflow DAGs in enterprise environments (5+ years).
• CI/CD for data projects:
  Ability to build and maintain CI/CD pipelines for data engineering workflows, including automated testing and deployment (5+ years).
• Cloud & containers:
  Experience with containerization (Docker) and cloud platforms (Google Cloud Platform) for data engineering workloads; understanding of twelve‑factor design principles (5+ years).
• Python fluency:
  Ability to write object‑oriented Python code, manage dependencies, and follow industry best practices (5+ years).
• Version control:
  Proficiency with Git for source code management and collaboration (5+ years).
• Unix/Linux:
  Strong command‑line skills in Unix‑like environments (3+ years).
• SQL:
  Solid understanding of SQL for data ingestion and analysis (3+ years).
• Collaborative development:
  Comfortable with code reviews, pair programming, and using remote collaboration tools effectively (3+ years).
• Engineering mindset:
  Writes code with an eye for maintainability and testability; excited to build production‑grade software (3+ years).
• Education:
  
  Bachelor’s or graduate degree in Computer Science, Data Analytics or related field, or equivalent work experience.

• Minimum 7+ years overall IT experience.
• Experienced in waterfall, iterative, and agile methodologies.

Travel:
None

Degree:
Graduate degree in a related field, such as Computer Science or Data Analytics.

Familiarity with Test‑Driven Development (TDD).

High tolerance for Open Shift, Cloudera, Tableau, Confluence, Jira, and other enterprise tools.