Data Engineer Python & Databricks

This range is provided by Phaxis. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.

$/yr - $/yr

• Design, develop, and maintain end-to-end data pipelines and ETL/ELT workflows using Python, SQL, and modern orchestration tools (e.g., Airflow, dbt).
• Architect and optimize data storage solutions, including data lakes and data warehouses, using Databricks, Delta Lake, and cloud-native services.
• Build scalable data processing solutions leveraging Databricks notebooks, jobs, and clusters for both batch and streaming data workloads.
• Develop and manage Databricks workflows using Spark (PySpark, SQL, or Scala) to transform, cleanse, and aggregate large datasets.
• Implement data quality checks, schema validation, and monitoring to ensure data accuracy and reliability.
• Optimize Databricks cluster configurations and job performance to minimize cost and maximize throughput.
• Collaborate with Dev Ops teams to automate deployments, CI/CD pipelines, and infrastructure-as-code (IaC) for data systems.

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related technical field.
• Advanced proficiency in Python and SQL for data manipulation, transformation, and automation.
• Deep experience with Databricks, including Spark optimization, Delta Lake management, job orchestration, and workspace administration.
• Strong understanding of distributed data processing, partitioning strategies, and performance tuning in Databricks and Spark.
• Hands‑on experience with cloud platforms (AWS, Azure, or GCP) and their data ecosystems (e.g., S3, ADLS, Big Query, Snowflake).

Director

Full-time

Information Technology