Software Data Engineer

Job Description

Expected start date:
Dec 1st

Duration of engagement: 3 months, with potential extension

Work location:

Alpharetta, GA or Plano TX.

Work location model:
Hybrid, 3 days in office

The ideal candidate will be responsible for designing and maintaining modern, scalable data solutions on Azure using Databricks. This includes building data pipelines, ETL / ELT workflows, and architectures such as Data Lakes, Warehouses, and Lake houses for both real-time and batch processing. The role involves integrating large datasets from diverse sources, implementing Delta Lake, and preparing data for machine learning through feature stores.

• Design, develop, and optimize scalable data pipelines and ETL / ELT workflows using Databricks on Azure
• Build and maintain modern data architectures (Data Lake, Data Warehouse, Lakehouse) for real-time streaming and batch processing on Azure
• Implement data integration solutions for large-scale datasets across diverse data sources using Delta Lake and other data formats
• Create feature stores and data preparation workflows for machine learning applications on Azure
• Develop and maintain data quality frameworks and implement data validation checks
• Collaborate with data scientists, ML engineers, analysts, and business stakeholders to deliver high-quality, production‑ready data solutions
• Monitor, troubleshoot, and optimize data workflows for performance, cost‑efficiency, and reliability
• Implement data governance, security, and compliance standards across all data processes
• Create and maintain comprehensive technical documentation for data pipelines and architectures

• Data Architecture:
  Deep understanding of Data Lake, Data Warehouse, and Lakehouse concepts with hands‑on implementation experience
• Databricks & Spark: 3+ years of hands‑on experience with Databricks on Azure, Apache Spark (PySpark / Spark SQL), Delta Lake optimization
• Azure Platform: 3+ years working with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Synapse Analytics, Azure ML Studio, Azure Databricks
• Programming:
  Strong proficiency in Python (including pandas, Num Py), SQL, and Unix / Linux shell scripting; experience with Java or Scala is a plus
• Streaming: 3+ years’ experience with Apache Kafka or Azure Event Hubs, Azure Stream Analytics
• Dev Ops:
  Hands‑on experience with Git, CI / CD pipelines (Azure Dev Ops, Git Hub Actions), and build tools (Maven, Gradle)
• Orchestration:
  Working knowledge of workflow schedulers (Apache Airflow, Azure Data Factory, Databricks Workflows, TWS)
• Problem‑solving:
  Strong analytical and debugging skills with ability to work in agile / scrum environments

• Experience with ML frameworks and libraries (scikit‑learn, Tensor Flow, PyTorch) for data preparation and feature engineering on Azure
• Experience with vector databases (Azure AI Search, Pinecone, Weaviate, Milvus) and RAG (Retrieval Augmented Generation) architectures
• Experience with modern data transformation tools (DBT, Spark Structured Streaming on Databricks)
• Understanding of LLM applications, prompt engineering, and AI agent frameworks (Azure OpenAI Service, Semantic Kernel)
• Familiarity with containerization (Docker, Azure Kubernetes Service)
• Experience with monitoring and observability tools (Azure Monitor, Application Insights, Datadog, Grafana)
• Certifications in Databricks, Azure Data Engineer Associate, Azure AI Engineer, or Azure Solutions Architect

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.