Data Engineer

Azure data bricks with fact & dimension table, Spark SQL, Pyspark, autoloader, unity catalogue & file format JSon - Mandatory

Mandate Skills :
Hands on experience in Azure Databricks & Medallion Architecture

7 years - 9 years. in  Azure Databricks technologies.

Experience in  Medallion Architecture  is mandatory.

• Good understanding of ETL , BI and DW technologies.

• Hands-on experience on  Jason or XML files  as sources.

• Experience in  autoloader for incremental loading and explode options for flattening data

• Experience in loading  Delta live tables

• Experience in  PySpark  or  Spark  using data frames.

• Design, implement, and maintain data pipelines for data ingestion, processing, and transformation in Azure.

• Work together with data scientists and analysts to understand the needs for data and create effective data workflows.

• Create and maintain data storage solutions including Azure SQL Database, Azure Data Lake, and Azure Blob Storage.

• Utilizing Azure Data Bricks to create and maintain  ETL  (Extract, Transform, Load) operations.

• Implementing data validation and cleansing procedures will ensure the quality, integrity, and dependability of the data.

• Improve the scalability, efficiency, and cost-effectiveness of data pipelines.

• Monitoring and resolving data pipeline problems to guarantee consistency and availability of the data.

• Good communication skills – Oral & Written