Data Engineer

Data Engineer (Mid-level)

Location:

Remote (Quebec/Canada)

About Irth Solutions
Irth Solutions is a leading provider of cloud-based SaaS software for damage prevention, asset integrity, stakeholder engagement and land management, helping energy, utility, telecom, and infrastructure companies protect their critical network infrastructure. With nearly three decades of industry experience, Irth serves customers across North America and continues to expand its platform with new data-driven and AI-powered capabilities.

About the Role
We are looking for a Data Engineer to design, build, and maintain data ingestion and processing pipelines in Databricks, transforming high-volume external data sources into clean, structured, reliable inputs for downstream analysis and intelligence. This role will primarily support our Stakeholder Engagement offering, working closely with our Data Scientist and application teams.

Key responsibilities
1. Data Pipeline Development (Primary Responsibility)

Design, build, and maintain ingestion pipelines from high-volume external APIs, capable of running continuously and reliably at scale.

Implement ingestion and transformation workflows using Databricks (Spark/PySpark, SQL, Delta Live Tables), applying medallion architecture patterns (Bronze → Silver → Gold) to move from raw ingested content to clean, structured, analysis-ready data.

Build the infrastructure for deduplication and relevance filtering of incoming content, implementing filtering logic and quality criteria defined in collaboration with the Data Scientist.

Implement schema evolution handling and data validation rules as data sources and formats change over time.

2. Platform & Storage Implementation

Configure and manage Delta Lake storage structures, tables, partitions, and optimization routines (OPTIMIZE, Z-ORDER, VACUUM).

Design and evolve data schemas that balance query performance, cost, and maintainability as data volume grows.

Maintain clear metadata and documentation of table structures to support easy consumption by the Data Science and application teams.

3. Reliability, Monitoring & Operational Support

Ensure pipeline reliability and observability: error handling, retries, monitoring, and alerting for a continuously running system.

Adapt pipelines to evolving external API contracts, rate limits, authentication changes, and new data sources.

Troubleshoot pipeline failures, perform recovery, and tune performance as needed.

4. Orchestration & Automation

Build, schedule, and monitor workflows using Databricks Workflows, Delta Live Tables, or similar orchestration tools.

Contribute to CI/CD pipelines for code deployment, versioning, and environment management.

5. Collaboration & Documentation

Work closely with the Data Scientist to expose clean, well-structured data feeding LLM/NLP pipelines and downstream models.

Participate in technical decisions around data architecture and propose structuring solutions as the team's needs evolve.

Document pipelines, data dictionaries, job schedules, and transformation logic.

Support the onboarding of new data sources and pipelines as the product expands to additional solution areas.

Ingénieur(e) de données (Intermédiaire)
Lieu :
Télétravail (Québec/Canada)

À propos d'Irth Solutions
Irth Solutions est un fournisseur de premier plan de logiciels SaaS infonuagiques pour la prévention des dommages, l'intégrité des actifs, la mobilisation des parties prenantes et la gestion foncière, aidant les entreprises des secteurs de l'énergie, des services publics, des télécommunications et des infrastructures à protéger leurs réseaux d'infrastructure critiques. Avec près de trois décennies d'expérience dans l'industrie, Irth dessert des clients à travers l'Amérique du Nord et continue d'élargir sa…