Senior Data Engineer

Overview

Liminal is a global market intelligence and strategic advisory firm specializing in digital identity, financial crime and compliance, and IT security technology solutions across industries while also catering to the private equity and venture capital community. Founded in 2016,
Liminal offers strategic and analytical services supporting executive decision‑making at all product and business lifecycle stages. We advise some of the world’s most prominent business leaders, investors, and policymakers on building, acquiring, and investing in the next generation of solutions and technologies. We provide access to proprietary data and analysis, strategic frameworks, and integrated insights on the industry’s only market intelligence platform.

Every major company in the world has started focusing on the next generation of digital identity technologies as a necessity for continued growth and security. Our team works with a myriad of organizations, from Fortune 100s to startups, across industries including financial services, technology, telecommunications, and the P2P economy. At Liminal
, we help businesses build solutions, execute strategies, invest intelligently, and connect with key decision‑makers. We know that it’s in the sharing of discovery and insights that groundwork is laid, problems are solved, and entire sectors advance at the speed of light. Keeping information to ourselves delays progress for all. At Liminal
, we don’t just respond to the market; we define it.

This role focuses on building and maintaining strong data architectures
, pipelines
, and systems that support the effective collection, storage, and processing of data across multiple departments. The Data Engineer will play a pivotal role in ensuring the scalability
, reliability
, and performance of our data systems. With a strong background in data engineering
, cloud infrastructure
, and data pipeline automation
, the Data Engineer will work on projects from initial design to deployment
, supporting the seamless integration of data workflows into product and operational teams.

• Cross-Department Data Solutions: Collaborate with various departments to understand data needs
  , assess technical feasibility
  , and design efficient data engineering solutions to support organizational initiatives. Implement scalable data workflows that optimize data availability
  , quality
  , and accessibility for AI, business analytics, and other internal teams. Support product teams in transitioning mature data pipelines and systems to ensure alignment with product goals and technical requirements
  .
• Data Pipeline Development & Optimization: Design, implement, and maintain data pipelines that ingest, process, and transform large-scale datasets for internal applications, including AI and machine learning models. Build efficient ETL (Extract, Transform, Load) processes that streamline the movement of data between systems, databases, and analytics platforms. Optimize data flows to ensure high performance
  , low latency
  , and scalability
  , adapting pipelines to handle both batch and real-time processing
  .
• Cloud Infrastructure & System Integration: Develop and maintain cloud-based data infrastructure on a major cloud platform (e.g.,
  AWS
  , Azure
  , GCP
  , or similar), ensuring data systems are robust
  , cost-effective
  , and performant
  . Implement data storage solutions and distributed databases that ensure seamless integration with other internal systems. Leverage cloud services for scalable data processing and storage, ensuring that infrastructure can support growing datasets and organizational demands.
• Data Quality & Governance: Establish data validation processes to ensure data quality
  , consistency
  , and integrity across all pipelines and systems. Collaborate with data scientists and analysts to ensure data is structured and formatted for optimal use in analytics and AI applications
  . Ensure compliance with data governance policies and best practices for data privacy
  , security
  , and auditability
  .
• Automation & Monitoring: Implement automation for data processing workflows, reducing manual intervention and ensuring consistent delivery of high-quality data
  .…