Data Engineer

Overview

Guardian is seeking a Data Engineer to join the team responsible for establishing and maintaining an optimized, scalable data pipelines to provide access to high quality, consistent data in an easy and convenient manner to all authorized end users and applications of Guardian and its subsidiaries.

As a Data Engineer, you will play a vital role in our data infrastructure and analytics ecosystem. You will work closely with data architects and data scientists to build and maintain data pipelines, transform raw data into actionable insights, and contribute to the development of cutting‑edge data solutions. This is an excellent hands‑on opportunity to experience the latest technologies and make a significant impact on our data‑driven initiatives.

The position will integrate new data sources from enterprise sources and external vendors using a variety of ingestion patterns including Structured Query Language (SQL) Ingestion, File, and Application Programming Interfaces (APIs).

• Collaborate with cross‑functional teams to understand data requirements and translate them into effective data pipelines.
• Develop and maintain ETL processes using Python, PySpark, and SQL to extract, transform, and load data from various sources.
• Utilize Databricks for data processing, analysis, and optimization tasks.
• Work with Syncsort to enhance data integration and data quality.
• Assist in data modeling, schema design, and database optimization.
• Monitor and troubleshoot data pipelines to ensure data accuracy, reliability, and performance.

Participate in code reviews, testing, and documentation efforts to maintain high‑quality code and data documentation.

• Stay updated on industry trends and best practices in data engineering.
• Establish and maintain the quality, supportability, and performance of all processes.
• Continue to look for innovative and next generation solutions for solving data challenges.
• Establish real time data pipelines using AWS services like SQS / SNS and Open Search.

• A self‑starter, self‑motivated team player and have excellent communication skills, enabling you to engage with all levels of stakeholders who is fueled by collaboration, able to transform conceptual designs into reliable, scalable processes that meet or exceed customer needs.
• An individual who works closely with IT and business leaders across the enterprise to drive and embed the Enterprise Data Strategy into transformational roadmaps ensuring key principles of Data Governance, Data Management, Data Analytics and Architecture are implemented.

• 10+ Years of IT experience, with 5+ years of experience as a Data Architect, data engineer or other data professional
• Expert Relational database concepts and experience across multiple database technologies (e.g., SQL, No
  
  SQL, Oracle, Hadoop, Postgres) environments.
• Expert Proficiency in data modelling tools — preferably Erwin and strong Logical and Physical Data Modelling experience
• Hands on experience in data engineering desired with extensive knowledge of SQL and PySpark, on Databricks Platform.
• Hands on experience designing, building, and optimizing data pipelines, architecture for large scale data warehousing projects.
• Experience with cloud data services such as Redshift, Relational database services (RDS), Snowflake and Databricks.
• Knowledge of AWS Data services is a big plus
• Demonstrated expertise with data sourcing, profiling, data mapping and data analysis.
• Strong SQL, PL/SQL experience and skilled in creating and optimizing queries and data structures.
• Knowledge of ETL concepts, preferably Informatica or Syncsort
• Experience with one or more SQL-on-Hadoop technology (HIVE, Presto, Spark SQL) and Hadoop ecosystem.
• Ability to design cloud and on Prem data solutions and provide guidance across storage, big data platform services, Hadoop ecosystem, vendor SaaS integrations, RDBMS and No
  
  SQL databases is a plus.
• Excellent written and verbal communication skills and creative problem‑solving skills.
• Ability to adapt to changing business priorities and experience with effectively communicating change across different levels in the organization.
• Communication
  
  Skills:
  
  Articulate the vision and goals,…