Senior Data Engineer

This job is brought to you by Jobs/Redefined, the UK's leading over-50s age inclusive jobs board.

We are seeking a Senior Data Engineer to lead the charge on ensuring the health, reliability, and security of our critical data pipelines. This is a senior, hands-on technical role for an expert who is comfortable with mission-critical batch data pipelines in a cloud environment, integrating with numerous real-time data sources. You will be responsible for managing highly sensitive and critical data streams and driving strategic initiatives to minimize incidents, optimize performance, and build a resilient hybrid data environment.

Your focus will be on proactive problem-solving, automation, and continuous improvement, transforming our operational processes from reactive to resilient.

• Production Support & Reliability:
  Act as the subject matter expert and technical lead for resolving the most complex, high-impact incidents affecting data pipelines. Manage multiple stakeholders for critical events. Perform in-depth root cause analysis to prevent recurrence, focusing on data pipelines, scheduling platforms such as Control-M and AWS-related services.
• Data Security & Governance:
  Ensure the integrity and security of highly sensitive and critical data throughout the entire pipeline. Implement and enforce security best practices, including managing encryption at rest and in transit, access controls, and compliance.
• Automation & Tooling:
  Develop and implement automation for common operational tasks to reduce manual toil. Focus on building tools and monitoring solutions that provide visibility into the end-to-end health of pipelines.
• Performance Optimization:
  Proactively analyse and tune the performance of batch schedules and AWS resource utilization. Identify and implement optimizations to improve efficiency and reduce operational costs.
• Collaboration & Leadership:
  Act as a technical leader and mentor for both onsite and offshore team members. Ensure seamless collaboration, clear communication, and consistent operational standards across a distributed team. Contribute to the long-term technical strategy for data operations including modernization efforts.

• Extensive, hands-on experience in a production support, site reliability, or data operations role within a large-scale data environment.
• Experience with data distribution platforms (e.g. Ab Initio & Spark centric solutions like AWS Glue & EMR), including deep understanding of ETL/ELT workflows & integration into data platforms like Snowflake.
• Extensive experience with scheduling platforms such as Control-M, including complex scheduling, dependencies, and managing a large batch environment.
• Working knowledge of IBM Sterling File Gateway or similar file transfer (MFT) solutions would be beneficial (e.g. AWS Transfer Family).
• Deep knowledge of AWS and its data-related services, including knowledge of open-source, cloud-first data-pipeline orchestration capabilities like Apache Airflow.
• Proficiency in Shell scripting & Python for automation and system administration.
• Proven ability to manage highly sensitive and critical data pipelines, with a strong understanding of security and compliance requirements.
• Demonstrated experience working effectively with both onsite and offshore teams, ensuring seamless operational handoffs and knowledge sharing.
• Excellent communication skills, with the ability to articulate complex technical issues to both technical teams and business stakeholders.
• Experience with Dev Ops or Data Ops principles and practices is essential.

• Persuasive Communication:
  Skillfully translate complex technical concepts into compelling narratives, using persuasive language and visual aids to appeal to a wide range of audiences. Continuously solicit and integrate feedback to refine and enhance communication strategies for maximum impact.
• Ownership Mentality:
  Take responsibility for the quality and success of the work the team is leading, demonstrating a sense of pride and accountability.
• Sound Judgement:
  Make sound technical decisions based on available information and trade-offs, considering both short-term and long-term implications.
• Proactive Risk Identification:
  Anticipate and identify potential risks and challenges in projects, proposing mitigation strategies and proactively address them to ensure project success.
• Cross-Domain Learning:
  Actively seek opportunities to expand technical knowledge and expertise beyond your primary domain, demonstrating a strong understanding of how different technologies and systems interact and integrate.
• Analytical Thinking:
  Approach problems with a structured and analytical mindset, breaking them down into manageable components.
• Technical Problem Solving:
  Lead troubleshooting efforts and resolve complex technical issues, mentoring others in problem-solving approaches.
• Project/Feature Leadership:
  Take ownership of focused projects or complex features, leading them from design…