Senior Data Engineer & Architect, Data Pipelines

The Data Environments and Engineering Department (L176) in the Data and Human Centered Solutions Innovation Center is seeking highly motivated, adaptive, and enthusiastic candidates with a wide variety of interests and a foundation of skills in systems engineering, software engineering, information systems, or data engineering to join our department as a Lead Data Engineer. Candidates should have a strong interest in addressing and solving challenges related to capturing, managing, and sharing data supporting decision making and enabling mission processes.

Roles and Responsibilities

• Working with experienced project teams and data management experts, project tasks may include:
• Leading the development of high quality, high velocity contributions to sponsor challenges and deliverables
• Designing and building prototypes for ingesting, curating, and analyzing data at scale.
• Collecting and preparing data for use by data scientists and analysts
• Leading stakeholder interviews to determine data requirements and define data flows.
• Documenting data transformation rules to meet the requirements for system interfaces and data integration projects
• Performing data profiling to identify quality gaps and determine if data is fit for use for its intended purpose
• Researching and evaluating data engineering tools and technologies

Minimum Qualifications:

• 8 years of related experience with a bachelor’s degree; or 6 years and a Master’s degree; or a PhD with 3 years’ experience; or equivalent combination of related education and work
• Must be able obtaining and maintaining a DoD clearance.
• Strong developer skills using SQL and Python
• Hands-on experience with one or more database technologies (e.g., PostgreSQL, Oracle, MySQL, SQL Server, DB2, MongoDB, Neo4j, Couchbase, Cassandra)
• Excellent written and verbal communications skills with ability to describe and present complex technical concepts in relatable terms
• Strong knowledge of ETL/ELT processes, tools, and frameworks
• Experience with large scale data architectures. Activities may include analysis of alternatives, technical product evaluations, requirements analysis, design, data systems prototyping, and process automation.
• Experience building conceptual, logical, and physical data models
• Knowledge of cloud platforms and technologies such as Amazon Web Services, Microsoft Azure, and Google Cloud Platform.
• Proficiency in one or more modern programming languages such as Python, R, Scala, C++, Java, C#, and Java Script

Data Engineering skills to include:

• Expertise in preparing data for analysis
• Strong understanding of data science concepts
• Experience with BI tools
• Understanding ML frameworks, algorithms, and libraries
• Applied mathematics: probability and statistics
• Strong innovation, problem-solving, systems analysis, planning, organizational, and interpersonal skills
• Knowledge of data management practices that align with modern/emerging analytics techniques
• Desire to continuously learn new techniques and evolve approaches by regularly engaging in structured educational pursuits, personal research, and outreach to subject matter experts across the corporation

Preferred Qualifications:

• Master’s degree or PhD in computer science, data science, systems engineering, software engineering, or information systems
• Experience migrating legacy data systems to new/modernized platforms
• Hands-on experience with data flow and pipeline frameworks such as Kafka, NiFi, Spark, Airflow, and Hadoop.
• Familiarity with microservice architectures, data warehousing, data fabric, data lakes, and data mesh
• Understand the use of data in AI/ML and LLM
• Knowledge and experience across data management domains, including data science, data governance, metadata management, data quality, business intelligence, data security &…