Big Data Engineer

Responsibilities

• Data Architecture:
  Design and implement scalable, robust, and efficient data architectures, including data lakes, data warehouses, and data pipelines, to support the organization's data requirements.
• Data Pipeline Development:
  Develop and maintain ETL (Extract, Transform, Load) processes and data pipelines to ingest, process, and transform large volumes of data from various sources, ensuring data quality and reliability.
• Data Modeling:
  Design and implement data models and schemas to organize and structure data for efficient storage, retrieval, and analysis, considering business requirements and data governance principles.
• Data Integration:
  Integrate data from diverse sources, such as databases, APIs, files, and streaming platforms, ensuring seamless data flow and interoperability across systems.
• Data Transformation and Enrichment:
  Implement data transformation and enrichment processes to derive insights, perform aggregations, and create derived datasets for analytics, reporting, and machine learning.
• Data Governance and Compliance:
  Establish and enforce data governance policies, standards, and best practices to ensure data integrity, security, privacy, and compliance with regulatory requirements.
• Performance Optimization:
  Optimize data pipelines and queries for performance, scalability, and efficiency, leveraging techniques such as partitioning, indexing, and query optimization.
• Collaboration:
  
  Collaborate with cross-functional teams, including data scientists, analysts, software engineers, and business stakeholders, to understand data requirements and deliver data solutions that meet business needs.

• 3+ years of Data Management experience in an enterprise environment.
• 2+ years of experience with SQL.
• 1+ years of experience with AWS S3, Spark and Glue.
• Knowledge of metadata management, data governance, and data lineage.

• Good understanding of Agile Software Development practices and how to leverage these to streamline project execution.
• The ability to decompose business problems into smaller parts and solve them correctly, considering the impact on other subsystems.
• AWS certification.
• Team player with excellent problem-solving and troubleshooting skills.
• Familiar with software engineering practices, including CI/CD pipelines, Automated Testing, and Performance Engineering.
• Working knowledge of SDLC, especially regarding improving agile delivery metrics.
• Strong communication skills.
• Bachelor’s degree in Computer Science, MIS, or a related field.
• Financial services experience.
• Consulting experience.