Senior Data Engineer | Databricks, AWS, Spark, Python | GCP

Description

Colate is looking for Senior Data Engineer. The Senior Data Engineer will lead the design and implementation of Databricks‑based data solutions that enable enterprise‑wide data retention, archival, and deletion capabilities. This role plays a critical part in ensuring regulatory compliance, cost‑optimised storage, and automated data lifecycle management across large‑scale analytics platforms.

The role requires a strong subject‑matter understanding of Databricks and lakehouse architectures, combined with the ability to clearly communicate technical approaches, trade‑offs, and recommendations to business and analytics stakeholder

Requirements

Required Experience

• Strong background in Databricks‑centric data engineering

• Experience working in regulated or governance‑heavy environments is highly desirable

• Demonstrated ownership of end‑to‑end solution design, not just implementation

Responsibilities

• Design and implement data retention, archival, and deletion frameworks using Databricks (Delta Lake, notebooks, jobs, workflows)

• Build scalable, metadata‑driven lifecycle management solutions aligned with regulatory and governance requirements

• Work closely with Data Architects, Product Owners, Legal/Compliance, and Platform teams to translate policy requirements into technical implementations

• Define and explain multiple implementation approaches, clearly articulating pros, cons, risks, and cost implications to business stakeholders

• Guide business and analytics teams toward optimal data lifecycle strategies using clear, non‑jargon communication

• Enable automation of data lifecycle workflows across ingestion, storage, and consumption layers

• Ensure solutions align with enterprise governance, auditability, and compliance standards

• Act as a Databricks SME, supporting design decisions and best practices across analytics initiatives

Benefits

Please visit https://colate.io/