Data Engineer US : Atlanta, Austin, Boston Remote

We foster a dynamic culture rooted in strong engineering, a sense of ownership, and transparency, empowering our team. As part of the expanding Virtus Lab Group, we offer a compelling environment for those seeking to make a substantial impact in the software industry within a forward‑thinking organization.

Building AI-Powered Web Data Acquisition as a Staff Data Engineer and Tech Lead. Deep technical ownership is the main focus of this crucial position. To solve complex scraping problems and set the highest operational standards, you will develop the architecture for our next‑generation, distributed web‑crawling and data‑extraction solution. As you mentor engineers and establish company‑wide best practices across international teams by sharing your knowledge with the community and developing strong networking, this role is also a unique opportunity to develop your personal brand and create a strong community around your technical expertise by attending conferences and planning meetups with Virtus Lab’s full support.

Python Advanced

Kafka Regular

Public cloud (AWS, Azure, GCP) Advanced

Snowflake Regular

Trino/Starburst with Iceberg Regular

IaC Regular

Terraform Regular

Distributed systems/computing Advanced

SQL/No

SQL Regular

Airflow or any cloud‑based orchestrator Regular

ETL processes Expert

CI/CD Regular

LLM productivity tools like Cursor/Claude Code Basic

Testing Regular

Our client is a NASDAQ‑listed company that provides a range of solutions to support Go‑To‑Market (GTM) strategies. They offer a comprehensive B2B database platform that enables sales and marketing professionals to identify, connect with, and engage qualified prospects effectively. The core mission of our client is to equip every company with a complete, 360‑degree view of their ideal customer, enhancing every phase of their GTM strategy and boosting their success in achieving business targets.

Technical Leadership & System Design:

• Proven experience building web crawling or large‑scale data systems from scratch.
• Strong architectural skills in designing scalable, fault‑tolerant distributed systems.
• Track record leading complex technical initiatives and driving architecture direction for teams.
• Demonstrated ability to evolve production systems incrementally while maintaining reliability.
• Experience mentoring engineers at all levels and promoting a collaborative culture.
• Deep background in large‑scale data engineering (terabytes daily).
• Hands‑on experience with cloud data warehouses (Big Query, Snowflake).
• Experience with Apache Kafka, Kubernetes (GKE/EKS), and orchestration tools (Airflow).
• Familiarity with multi‑cloud environments (GCP + AWS).
• Expertise in designing and operating ETL/ELT pipelines.
• Deep expertise in web crawling technologies and advanced scraping (Scrapy or similar).
• Experience in extracting structured/unstructured web data and SERP extraction.
• Knowledge of proxy infrastructure management, anti‑bot detection, and ethical crawling.
• Familiarity with crawling vendors and AI/LLM‑based extraction approaches.

Client Engagement & Advisory:

• Support the Virtus Lab U.S. and international teams by lending senior technical expertise to client‑facing activities, including technical discovery sessions, workshops, and solution architecture.
• Conduct requirements analysis and solution discovery, identifying business and technical needs.
• Provide technical consulting and advisory services, recommending appropriate data architectures aligned with customer goals.
• Prepare and review technical sections of commercial offers, including solution descriptions, statements of work (SoWs), project estimates, timelines, and delivery models.

Seems like lots of expectations, huh? Don’t worry! You don’t have to meet all the requirements.
What matters most is your passion and willingness to develop. Apply and find out!

* The compensation range for this role varies based on location, experience, and level. For candidates based in New York, the expected salary range is higher compared to those located within a broader commuting radius (up to approximately two hours from the city).

• Building tech…