Senior Data Engineer

Senior Data Engineer

Bright Vision Technologies builds innovative software solutions that automate and optimize business operations.

We are seeking an experienced Senior Data Engineer to architect, design, develop, and maintain enterprise‑grade data platforms, scalable pipelines, and distributed processing systems that support analytics, business intelligence, and machine learning initiatives across multiple domains. The role covers the end‑to‑end data engineering lifecycle, from translating business requirements into robust data architectures to deploying cloud‑native solutions and ensuring their operational longevity.

• Design, build, and continuously improve scalable batch and real‑time data pipelines using Python, SQL, Spark, Scala, or equivalent technologies.
• Author secure, reusable, production‑quality ETL/ELT workflows that adhere to enterprise coding standards, governance policies, data quality principles, and security best practices.
• Develop scalable data integration solutions on modern cloud platforms such as AWS, Azure, or Google Cloud using services like Databricks, Snowflake, Big Query, Redshift, Synapse, Data Factory, Glue, or equivalents.
• Design and implement robust data architectures, dimensional models, data lakes, data warehouses, and streaming solutions that integrate diverse data sources with consistency, scalability, and high availability.
• Participate in enterprise data architecture discussions, cloud migration initiatives, technical design reviews, and solution planning, evaluating trade‑offs involving scalability, performance, governance, and cost.
• Continuously monitor, profile, and optimize ETL processes, Spark jobs, SQL queries, database performance, storage utilization, and pipeline throughput.
• Maintain metadata management, data cataloging, lineage tracking, schema evolution, and governance frameworks to ensure trusted, discoverable, and compliant data assets.
• Develop automated testing frameworks for pipelines, ETL workflows, data validation, reconciliation, integration, and performance, ensuring reliable production deployments.
• Contribute to CI/CD pipeline design, infrastructure automation, and deployment processes using Jenkins, Git Hub Actions, Azure Dev Ops, Terraform, Docker, Kubernetes, or equivalents.
• Identify and resolve data pipeline bottlenecks, operational risks, technical debt, and architectural weaknesses, driving continuous improvement.
• Collaborate within Agile/Scrum teams, participating in sprint planning, backlog refinement, daily stand‑ups, architecture discussions, reviews, and retrospectives.
• Maintain comprehensive technical documentation, including diagrams, pipeline specs, metadata, deployment guides, operational runbooks, and disaster recovery procedures.

• Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Software Engineering, Mathematics, or a closely related discipline.
• Five or more years of professional experience designing, developing, and supporting production‑grade enterprise data engineering solutions.
• Strong understanding of data structures, database design, distributed computing, data modeling, ETL/ELT methodologies, and data warehousing concepts.
• Advanced working knowledge of Python, SQL, Spark, Scala, Java, and related frameworks for scalable data processing.
• Hands‑on experience designing and operating batch and streaming pipelines, data lakes, and cloud‑native platforms using solutions such as Databricks, Snowflake, Apache Spark, Kafka, Airflow, or equivalents.
• Proven experience with relational and No
  
  SQL databases, including PostgreSQL, SQL Server, Oracle, MySQL, MongoDB, Cassandra, or equivalents, covering schema design, query optimization, and performance tuning.
• Strong SQL skills and experience designing dimensional models, star and snowflake schemas, data marts, and enterprise‑scale data warehouse solutions.
• Solid experience with Git‑based version control, CI/CD pipelines, Dev Ops practices, release management, and Agile software development methodologies.
• Hands‑on experience deploying data platforms on AWS, Azure, or Google Cloud, covering storage, compute,…