Big Data Engineer Job Tysons area,Virginia USA,Software Development

Description:
Hybrid 3 days onsite / 2 days remote in Tysons Corner, VA

Our client is seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale data processing systems. You will work with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of big data platforms. The ideal candidate will have expertise in distributed systems, cloud platforms, and modern big data technologies such as Hadoop and Spark.

We can facilitate w2 and corp-to-corp consultants. For our w2 consultants, we offer a great benefits package that includes Medical, Dental, and Vision benefits, 401k with company matching, and life insurance.

Rate: $65.00 to $70.00/hr. w2

Responsibilities:

Design, develop, and maintain large-scale data processing pipelines using Big Data technologies such as Hadoop, Spark, Python, and Scala.
Implement data ingestion, storage, transformation, and analysis solutions that are scalable, efficient, and reliable.
Stay current with industry trends and emerging Big Data technologies to improve the data architecture.
Collaborate with cross-functional teams to translate business requirements into technical solutions.
Optimize and enhance existing data pipelines for performance, scalability, and reliability.
Develop automated testing frameworks and implement continuous testing for data quality assurance.
Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines.
Partner with data scientists and analysts to support data-driven decision-making.
Write and maintain automated unit, integration, and end-to-end tests.
Monitor and troubleshoot data pipelines in production to identify and resolve issues.
Apply SQL skills including window functions, joins, aggregations, and handling of NULLs, duplicates, and ordering.
Design and tune Apache Spark jobs, including partitioning, caching, broadcast joins, and troubleshooting DAG, stages, tasks, and executor performance.
Leverage AWS services such as S3, EMR, Glue, Lambda, and Athena, including Spark with S3 file formats and consistency considerations.
Write clean, modular, and performant Python or Scala code using functional programming concepts, and manage collections, concurrency, and memory.
Employ AI tools and prompt engineering to enhance development workflows, analysis, and continuous improvement.

Experience Requirements:

Bachelor's degree in Computer Science, Information Systems, or related discipline with at least five years of related experience, or equivalent training and work experience. Master's degree and Financial Services experience preferred.
Demonstrated expertise in object-oriented and database technologies leading to enterprise-quality solutions.
Experience delivering enterprise solutions in iterative or Agile environments.
Knowledge of software engineering approaches including test automation, build automation, and configuration management frameworks.
Strong written and verbal technical communication skills.
Ability to build effective working relationships and improve work product quality.
Strong organization, thoroughness, and ability to handle competing priorities.
Ability to maintain focus and develop proficiency in new skills rapidly.
Experience working in a fast paced environment.
Experience with Java, Scala, or Python.
Essential technical skills include AI tool proficiency (e.g., Git Hub Copilot, Q Developer, ChatGPT, Claude), strong software development background, and extensive experience with Scrum, Kanban, and continuous improvement.
Big Data technologies:
Hadoop, Spark, Hive, and Trino, with understanding of data skew, PB-scale data, and resource-related job failures.
Good to have: managing production data pipelines and ETL systems, CI/CD, writing test cases, and AWS certifications.

Education Requirements:

Bachelor's degree in Computer Science, Information Systems, or related discipline required. Master's degree preferred.

Recruitment Transparency Notice

Eliassen Group values transparency in our recruitment practices. Please be advised that Eliassen Group utilizes artificial intelligence (AI) tools as part of its initial application screening and hiring process. You may receive email and SMS notifications from the Eliassen Virtual Recruiting Team (, ) inviting you to complete a brief voice screening as part of your application process. These tools assist our hiring teams in different ways, including but not limited to, assistance in reviewing application materials to help identify candidates whose qualifications most closely match the requirements of the position.

All AI-assisted evaluations and responses are reviewed by human recruiters before any hiring decisions are made. The use of AI in our process is intended to support fairness, efficiency, and consistency, and Eliassen Group takes measures to prevent bias or discrimination in connection with its hiring practices. By proceeding, you acknowledge, agree,…