Director of Software Engineering; Node.js & Web Scraping Expert

We are seeking a Director of Software Engineering with deep expertise in Node.js development and large‑scale web scraping
. This role will lead the engineering team
, designing and optimizing high‑performance, distributed web scraping systems
. The ideal candidate has extensive experience in handling anti‑bot measures, data pipeline optimization, and scalable cloud‑based architectures
.

• Architect, develop, and maintain scalable and distributed web scraping systems using Node.js
  .
• Design and implement data extraction pipelines to process large volumes of structured and unstructured data.
• Develop solutions to bypass anti‑bot mechanisms
  , including CAPTCHA handling, session management, fingerprinting, and IP rotation
  .
• Optimize scraping processes for performance, reliability, and efficiency while managing proxy services (residential, datacenter, rotating). Oversee data storage and processing strategies
  , ensuring high availability and consistency.
• Collaborate with Product, Dev Ops, and Data Science teams to integrate extracted data into analytics and business applications.
• Implement best practices for microservices, API integrations, and real‑time data streaming
  .

• Lead the transition to cloud‑native, containerized, and serverless architectures for web scraping.
• Ensure compliance with legal and ethical standards (robots.txt, GDPR, CCPA, etc.). Optimize cloud resources (AWS, GCP, or Azure) to support high‑throughput scraping.
• Manage real‑time monitoring and alerting systems to detect scraping failures, IP bans, or performance bottlenecks
  .
• Work closely with Dev Ops teams to optimize CI/CD pipelines, automated deployments, and system scalability
  .

• Lead, mentor, and grow a high‑performance engineering team
  .
• Define and execute the technology roadmap
  , aligning with business objectives.
• Foster a culture of continuous learning, collaboration, and innovation
  .
• Implement agile development methodologies (Scrum, Kanban) to optimize project execution.
• Ensure code quality, security, and best practices across all engineering efforts.

• 10+ years of experience in software engineering
  , with at least 5+ years in web scraping and large‑scale data extraction
  .
• Strong hands‑on expertise in Node.js, Puppeteer, Playwright, Cheerio, Selenium, and headless browser automation
  .
• Extensive experience in handling CAPTCHAs, IP rotation, session management, and anti‑bot evasion techniques
  .
• Deep knowledge of proxy management (residential, datacenter, rotating, and VPNs).
  
  Experience with
  No
  
  SQL/SQL databases (Mongo
  
  DB, Postgre
  
  SQL, Redis, Elasticsearch, etc.).
• Familiarity with data processing frameworks (Kafka, Rabbit
  
  MQ, Spark, Airflow, etc.) Strong experience with CI/CD, containerization (Docker, Kubernetes), and cloud deployment (AWS/GCP/Azure).

• Proven track record of scaling engineering teams and leading complex projects
  .
• Strong problem‑solving and debugging skills, especially for scraping challenges and performance bottlenecks
  .
• Excellent communication and stakeholder management skills.
• Passion for mentorship, team development, and continuous learning
  .

• Experience with machine learning for data extraction and NLP
  .
• Knowledge of browser fingerprinting and bot detection mechanisms
  .
• Familiarity with enterprise‑scale web crawling frameworks (Scrapy, Colly, Apify, etc.).
• Prior leadership experience in data‑driven businesses or web scraping startups
  .