Data Engineer; Woodlands, TX

Overview

Paladin Consulting is currently hiring a Data Engineer to join our team working onsite at our client's office located in The Woodlands, TX. We work with companies that offer environments for our employees to contribute, learn, and advance their career. We treat you like you are part of the family.

Job Title: Data Engineer
Work Location: The Woodlands, TX
Education/Experience

Required:

5+ years of experience in software/data engineering with at least 2 years working with Databricks and Apache Spark.

• Proficient in Databricks SQL for executing complex ad-hoc queries on large-scale lake house datasets, enabling rapid data exploration, trend identification, and actionable insights for business decision-making.
• Skilled in leveraging Databricks notebooks to design and implement scalable data transformation workflows, integrating PySpark and SQL to cleanse, enrich, and prepare large datasets for downstream analytics and reporting.
• Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks.
• Architect and implement ETL/ELT workflows leveraging AWS services such as S3, Glue, Lambda, Redshift, and EMR.
• Optimize Spark jobs for performance and cost-efficiency in a cloud environment.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver robust solutions.
• Implement CI/CD pipelines for Databricks notebooks and jobs using tools like Git Hub Actions, Azure Dev Ops, or Jenkins.
• Ensure data quality, security, and governance using tools like Unity Catalog, Delta Lake, and AWS Lake Formation.
• Monitor and troubleshoot production data pipelines and jobs.
• Mentor junior engineers and contribute to best practices and standards.

Required Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in software/data engineering with at least 2 years working with Databricks and Apache Spark.
• Strong proficiency in Python, SQL, and PySpark.
• Deep understanding of AWS cloud architecture and services (especially S3, Glue, Lambda, IAM, Redshift, and Cloud Watch).
• Experience with Delta Lake, Databricks Workflows, and Databricks SQL.
• Familiarity with data modeling, data warehousing, and data lakehouse architectures.
• Experience with infrastructure-as-code tools like Terraform or Cloud Formation.
• Strong problem-solving skills and ability to work in a fast-paced, agile environment.

Preferred Qualifications:

• Databricks Certified Data Engineer or AWS Certified Solutions Architect.
• Experience with streaming data using Kafka, Kinesis, or Structured Streaming.
• Knowledge of MLflow, feature stores, or MLOps practices.
• Familiarity with data governance and compliance frameworks (e.g., GDPR, HIPAA).

• Excellent communication and collaboration skills.
• Strong analytical thinking and attention to detail.
• Ability to mentor and lead technical discussions.

For more information or to view other opportunities, visit us at

Paladin Consulting is an EEOC employer.