Data Engineer

Overview

Job Title:

Data Engineer – Databricks. Company: V4C.ai. Type:
Full-time. Experience Level: Mid to Senior.

• Design, build, and maintain scalable data pipelines using Databricks and Apache Spark.
• Integrate data from various sources into data lakes or data warehouses.
• Implement and manage Delta Lake architecture for reliable, versioned data storage.
• Ensure data quality, performance, and reliability through testing and monitoring.
• Collaborate with data analysts, scientists, and stakeholders to meet data needs.
• Automate workflows and manage job scheduling within Databricks.
• Maintain clear and thorough documentation of data workflows and architecture.

• Experience:
  
  3+ years in data engineering with strong exposure to Databricks and big data tools.
• Technical
  
  Skills:
  
  Proficient in Python or Scala for ETL development; strong understanding of Spark, Delta Lake, and Databricks SQL; familiar with REST APIs, including Databricks REST API usage.
• Cloud Platform:
  Experience with AWS, Azure, or GCP.
• Data Modeling:
  Familiarity with data lakehouse concepts and dimensional modeling.
• Version Control & CI/CD:
  Comfortable using Git and pipeline automation tools.
• Soft Skills:
  
  Strong problem-solving abilities, attention to detail, and teamwork.

• Certifications:
  
  Databricks Certified Data Engineer Associate/Professional.
• Workflow Tools:
  Experience with Airflow or Databricks Workflows.
• Monitoring:
  Familiarity with Datadog, Prometheus, or similar tools.
• ML Pipelines:
  Exposure to MLflow or model integration in pipelines.

• Seniority level:
  Mid-Senior level
• Employment type:
  
  Full-time
• Job function:
  Information Technology
• Industries:
  Business Consulting and Services