Data Engineer

FAANG & Product & Top tier 1 Companies Mandatory : yes JOB SUMMARY

This role is part of a broader data engineering team that develops data pipelines, integrations, and data products to meet internal operational and analytic use cases. The Sr. Software Engineer – Data will partner with other engineers, architects, analysts and product managers to implement scalable, sustainable solutions. They work on technical assignments that are moderately difficult and require considerable judgment and initiative.

They must understand implications of their work and make recommendations for solutions, receiving little instruction on daily work and only general instructions on newly introduced assignments.

• Data orchestration with tools such as Azure Data Factory
• Data storage in cloud storage such as Azure Data Lake
• SDLC + CI/CD through systems such as Azure Dev Ops
• SQL proficiency (TSQL, PostgreSQL, …) and familiarity with Spark, including the ability to develop ETL processes using PySpark and SparkSQL
• Experience with parquet and other data exchange/storage formats
• Experience with agile development and standard Dev Ops practices
• Understanding principles of indexes and partitioning
• Experience optimizing data engineering code for performance (ability to read a DAG and determine where CBO is using most resources)
• Proficiency in common distributed cloud computing concepts such as fault tolerance, idempotency, SOA, etc.

• Approximate percent of time dedicated to Leadership & Relationship Skills – collaborate with technical teams and utilize system expertise to deliver technical solutions.
• Continuously learn and teach others existing and new technologies.
• Contribute to the development of others through mentoring, in‑house workshops, and learning sessions.
• Drive engineering projects by developing data solutions, conducting tests and inspections, preparing reports and calculations, and supervising base and associate level engineers as needed.
• Understand system protocols, how systems operate and data flows; be aware of current technology benefits; independently develop a full software stack; interact with system engineers to define requirements and automation needs.

• Write basic documentation on how technology works and create clear documentation for new code and systems used.
• Document system designs, presentations, and business requirements for consumption at manager and director level.
• Identify and design solutions to potential data processes that can be improved with automation or optimized with current technical best practices.
• Identify use cases and design technical solutions to include Agentic AI applications.

• Deal with the ambiguity involved in developing new systems/platforms.
• Design and implement new ideas utilizing novel frameworks to improve an existing or new system, process, or service.
• Review existing designs and processes to highlight more efficient ways to complete existing workload through industry perspectives.
• Contribute to new and existing technology options that support business goals and understand how current technology supports existing and future business goals.