Data Ops Engineer

Shamrock Trading Corporation is seeking a Data Ops Engineer who is passionate about data automation and lifecycle management. As a pivotal member of our Data Services team, you will help implement cutting‑edge Data Ops methodologies to optimize our data engineering and analytics frameworks. Your work will ensure high‑quality data solutions and contribute to the overall robustness and reliability of our data systems.

• Develop and enhance Data Ops platforms and lifecycle processes, independently delivering complex automation and reliability improvements
• Develop, optimize, and maintain CI/CD pipelines for data workflows using Git Hub Actions, AWS Code Build/Code Deploy, or similar tools.
• Build and scale automated testing frameworks for data validation, schema checks, and regression testing
• Use a code-first approach to architect and implement data quality checks and monitoring solutions to ensure data accuracy and reliability
• Contribute to scalable data lake and data warehouse architectures using Databricks, Delta Lake, DABs, and modern data platform patterns
• Implement and improve monitoring and alerting using Prometheus, Cloud Watch, and Grafana.
• Enhance metadata management and data cataloging using Databricks Unity Catalog.
• Design and implement robust access controls and governance using AWS IAM, Unity Catalog, and Terraform, ensuring compliance with SOC 2 standards.
• Deploy, manage, and enhance scalability of infrastructure-as-code using Terraform and AWS services (S3, EKS, Lambda, ECS, IAM).
• Collaborate closely with engineering and analytics teams to ensure performance, scalability, cost‑efficiency, and identify operational risk in projects
• Provide tier 1 and 2 operational support for internal reporting and API requests, ensuring timely and secure data delivery
• Identify data anomalies and perform forensic analysis for conclusive RCA of data issues
• Communicate clearly during incidents and coordinate cross-team problem solving.
• Participate in on‑call rotations while improving runbooks and reducing alert noise
• Independently perform proofs of concept (POCs) to validate solutions for operational issues and translate successful approaches into clear Jira stories for implementation
• Contribute to the evolution of Data Ops best practices, platform maturity, and team standards
• Create technical documentation that can be understood by the least technical team members

• Bachelor’s degree in Computer Science, Engineering, Analytics or a related field; or equivalent practical experience
• 3+ years experience with Databricks, including Unity Catalog, Delta Lake, and Spark Streaming or equivalent technologies
• 3+ years experience in build and deployment technologies such as Docker, AWS Code Build, Code Deploy, ECR, ECS
• 3+ years using Terraform or equivalent infrastructure as code
• Proficient in automated testing frameworks and tools
• 3+ years experience monitoring cloud technology stacks and APIs
• 5+ years experience with cloud data technologies and modern programming languages including Python, SQL, Java, and Spark
• 5+ years experience using terminal, CLI tools, and linux
• Preferred:
  Experience mentoring junior engineers or leading small technical initiatives
• Demonstrated ability to work collaboratively in a dynamic team environment with a proactive, positive attitude

• Training and Development: Ongoing training and professional development opportunities
• Medical: Fully paid healthcare, dental and vision premiums for employees and eligible dependents, and gym benefits
• Financial: Generous company 401(k) contributions and employee stock ownership after one year
• Work-Life Balance: Competitive PTO and work from home opportunities after an introductory period