Data Engineer I or II

JOB SUMMARY

The Data Engineer I or II (TO) is responsible for translating data into readily consumable forms to deliver integrated data to consumers by building, operationalizing, and maintaining data pipelines for Data, Analytics & Artificial Intelligence use cases across heterogeneous environments. The Data Engineer I or II (TO) also plays a role in working with various data integration tools which support a combination of data delivery styles such as virtualization, data replication, messaging and streaming in hybrid and multi-cloud integration scenarios.

Education/

Experience:

• Bachelor’s degree in computer science (CS), MIS, CIS, Mathematics, Statistics (Theoretical/Computational), Machine Learning or a related field.
• Proven knowledge of data engineering, data integration and data science principles are required
• 6+ years of related work experience in a fast-paced, competitive organization driven by data and enabled by technology

Knowledge/

Skills:

• Working experience with batch and real-time data processing frameworks.
• Working experience with data modelling, data access, schemas, and data storage techniques.
• Working experience with data quality tools.
• Experience increatingfunctional and technical designs for data engineering and analytics solutions.
• Experience implementing data models of different schemas and working with diverse data source types.
• Hands-on experience developing solutions withbig data technologies such as Hadoop, HIVE and Spark.
• Hands-on experience developing and supporting Python based AI/ML solutions.
• 6+ years hands on experience designing,developing, testing, deploying, and supporting data engineering and analytics solutions usingon-premise stools such as,Microsoft’s BIStack (SSIS/SSAS/SSRS),Informatica, Oracle Golden Gate,SQL, Oracle, and SQL Server.
• 4+ years hands on experience designing, developing, testing, deploying, and supporting data engineering and analytics solutions using Microsoft cloud-based tools such as Azure Data Lake, Azure Data Factory, Azure Databricks, Python, Azure Synapse, Azure Key Vault, and Power BI.
• Experience with Containerization methodologies – Docker, Open Shift etc.,
• Experience with Agile as well as Dev Ops, CI/CD methodologies.
• Hands-on experience designing and developing solutions involving data sourcing, enrichment and delivery using APIs & Web Services.
• Experience working with Jira or similar tools.
• Experience working with Kafka or similar tools.

• Designing and Developing methods to process structured, semi-structured, and unstructured data using batch and real-time data processing techniques.
• Delivering fast, reliable, and scalable data by incrementally and efficiently processing as it arrives from files or streaming sources like Kafka, DBMS and No
  
  SQL.
• Developing Release pipelines to automate recurring manual tasks like creating a build package, checking that build package into a version control repository and deploying it to a DV/UA environment.
• Building and Maintaining templates such as code libraries, pipeline patterns and semantic models to promote reuse and agility.
• Establishing gatekeeping processes that monitor and controls the promotion of successful data processes into production by understanding the business criticality.
• Collaborating with cross-functional teams with a combination of data, business, and technical personas, as well as a product owner/manager as necessary.
• Advocating data reusability by breaking down monolithic data delivery processes into modular data product delivery.
• Ensuring data reliability by defining data quality and integrity controls within the pipeline with defined data expectations and addressing data quality errors with predefined policies.
• Actively working with less experienced data engineers providing technical guidance and oversight.
• Understanding the usage of performance optimization clusters that parallelize jobs and minimize the data movement in Batch and stream data processing.
• Recommending improvements to the processes, technology, and interfaces that reduce the development time and effort and enhance the effectiveness of the team.
• Promptly participating in the Enterprise Social Networking…