Sr. Data Engineer

Sr. Data Engineer
- Long Term Project
- Remote (US)

Title:

Sr. Data Engineer

Location:

Remote (US)

Duration:
Long Term Project

Compensation: $90-100/hr W2

Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S.

Our Client is looking for an experienced, motivated, and self-driven candidate who can contribute towards various aspects of taking the Live Analytics offering to the next level and build new data products for our clients. This involves design, development, and production support of the systems as well as working with product teams and engineering teams in shaping the future of the application.

• Write new ETLs to support both real time and new features data and reporting products.
• Work with product teams to gather and understand requirements, discuss approaches, design solutions based on the requirements and create design documents for implementation keeping in mind costs, capacity, shelf-life and choosing the right tools and technologies.
• Design and develop peripheral systems and processes - monitoring and alerting services, ETL processes, automated offline workflows, automated build and deployment pipelines, systems for gathering application performance metrics, ensuring efficiency in cost.
• Ensure best practices in system development ranging from test coverage, use of enterprise standards, adherence to target architecture, high-availability and scalability, low latency, and fault-tolerance, thus ensuring delivery of high-quality systems.
• Create detailed documentation on systems as well as processes ranging from comprehensive overview of systems, runbooks for developing, monitoring and troubleshooting system issues and recommendations on future improvements.
• Managing project delivery from a day-to-day standpoint - participating in daily agile stand-ups, escalating blockers to stakeholders, ensuring timely completion of tasks & overall project & spring goals. Communicating updates to stakeholders and working with them on timelines.

• Bachelor's degree in Computer Science or related technical field. (NOT REQUIRED TO HAVE A DEGREE)
• Experience building Data Lakes, Data Warehouse using big data and cloud stack.
• 5 years of experience on software development using Agile, dev ops that include designing, developing, and implementing large-scale data engineering solutions.
• Expert Level Proficiency in Spark streaming and spark concepts.
• Expert level proficiency in Python Programming language and PySpark with at least 4 years of experience.
• Experience working with Kafka streams and creating ingestion ETL's.
• Expert level proficiency in ANSI SQL /Spark SQL with at least 5 year's experience.
• Experience working with diverse data sources and data formats ( json, parquet, csv, psv, delta)
• Experience working with cloud services such as AWS along with cloud-based data sources(S3)
• Experience building ETL with batch and real time stream processing.
• Experience using version control system such as git and CI/CD

• Experience with building workflows and ETL's in the databricks.
• Experience ingesting data produced by real time systems like Kafka
• Guiding and mentoring skills - guiding and mentoring junior members - helping them understand technologies, the project, best practices and in general setting them up for success.
• Experience building and deploying ML models using ML Flow.

• Strong verbal communication, ability to clearly communicate ideas, work status, advocate for (or against) a particular approach or design.
• Documentation skills - ability to make the path easier for new team members / new hires / interns by organizing knowledge in Confluence, wiki, or similar.
• Ability to manage self and own time, deliver work on time, evaluate and estimate work.
• Ability to work with various teams, including tech and product, incorporate their inputs into the work being done, and manage expectations on issues & enhancements.

• Comprehensive medical benefits
• Competitive pay
• 401(k) retirement plan
• ...and much more!

Technology is our focus and quality is our commitment. As a national expert in delivering flexible technology and talent solutions, we strategically align industry and technical expertise with our clients' business objectives and cultural needs. Our solutions are tailored to each client and include a wide variety of professional services,…