Data lead
Reference: 4567_1788768887
Databricks Data Lead
Role Overview
We are seeking an experienced Databricks Data Lead to provide technical leadership in the design, development, and delivery of modern cloud-based data platforms using Databricks. The role will be responsible for establishing data engineering standards, leading technical delivery, and ensuring that data solutions are scalable, secure, reliable, and aligned with business objectives.
The Databricks Data Lead will work closely with data architects, data engineers, analysts, data scientists, business stakeholders, and cloud engineering teams to deliver enterprise data and analytics capabilities.
Key Responsibilities
- Lead the design and implementation of scalable enterprise data platforms using Databricks.
- Provide technical leadership across data engineering, data integration, data transformation, and analytics initiatives.
- Define and implement modern data architecture patterns using the Databricks Lakehouse architecture.
- Design and implement data pipelines using Apache Spark, PySpark, SQL, and Databricks.
- Establish standards and best practices for data ingestion, transformation, processing, storage, and consumption.
- Lead the development of batch and real-time/streaming data pipelines.
- Design and implement data lake and lakehouse solutions using technologies such as Delta Lake.
- Define approaches for data modelling, data partitioning, optimisation, and storage management.
- Lead the implementation of data governance, security, access control, data quality, lineage, and metadata management.
- Work with architects and engineering teams to establish appropriate integration patterns between Databricks and enterprise data sources and platforms.
- Drive performance optimisation and cost management across Databricks workloads.
- Establish development standards covering coding, testing, deployment, monitoring, and operational support.
- Implement and promote CI/CD and DevOps practices for data engineering solutions.
- Support Infrastructure as Code and automated deployment approaches where appropriate.
- Lead technical design reviews and provide guidance on complex data engineering challenges.
- Mentor and support data engineers, helping establish consistent engineering practices and technical standards.
- Work with delivery managers and product owners to translate business requirements into technical solutions.
- Estimate technical effort, identify dependencies and risks, and contribute to delivery planning.
- Troubleshoot and resolve complex data pipeline, performance, reliability, and integration issues.
- Produce and maintain technical documentation, architecture diagrams, design specifications, and engineering standards.
- Evaluate new Databricks capabilities and emerging data technologies and recommend opportunities for adoption.
Required Skills and Experience
- Significant experience in data engineering, data architecture, or cloud data platforms.
- Strong hands-on experience with Databricks and the Lakehouse architecture.
- Strong knowledge of Apache Spark, PySpark, and SQL.
- Experience designing and implementing enterprise-scale data pipelines.
- Strong understanding of Delta Lake and modern data lake/lakehouse architectures.
- Experience working with cloud platforms such as Azure, AWS, or Google Cloud Platform.
- Strong understanding of data warehousing, data lakes, data modelling, and data integration.
- Experience developing both batch and streaming data solutions.
- Strong understanding of data quality, governance, security, and access-control principles.
- Experience with source control, CI/CD, automated testing, and DevOps practices.
- Experience with data orchestration technologies such as Databricks Workflows, Apache Airflow, Azure Data Factory, or equivalent.
- Experience integrating data from relational databases, APIs, files, event streams, and enterprise applications.
- Strong programming experience in Python and/or Scala.
- Experience leading technical teams or providing technical leadership to data engineering teams.
- Strong problem-solving, communication, and stakeholder-management skills.
Desirable Skills
- Experience with Unity Catalog and Databricks governance capabilities.
- Experience with Delta Live Tables / Lakeflow Declarative Pipelines.
- Experience with Databricks SQL and SQL Warehouses.
- Experience with Structured Streaming and technologies such as Kafka.
- Experience with dbt or other modern data transformation frameworks.
- Experience with Terraform or other Infrastructure as Code technologies.
- Experience implementing data mesh or domain-oriented data architectures.
- Experience with data cataloguing, lineage, and data governance platforms.
- Experience with cloud-native services across Azure, AWS, or GCP.
- Experience with machine learning, AI, or advanced analytics platforms.
- Databricks or relevant cloud certifications.
Leadership Responsibilities
- Set technical direction and engineering standards for the data platform.
- Lead and mentor data engineers and provide technical guidance across delivery teams.
- Review technical designs and ensure alignment with enterprise architecture principles.
- Identify opportunities to improve platform scalability, reliability, performance, and cost efficiency.
- Promote engineering best practices, automation, reusable components, and standardised patterns.
- Act as a technical point of contact between engineering teams, architecture teams, and business stakeholders.
- Support recruitment, technical assessments, capability development, and knowledge sharing where required.
Key Competencies
- Databricks & Lakehouse Architecture
- Apache Spark & PySpark
- Data Engineering
- Data Architecture & Modelling
- Delta Lake
- Cloud Data Platforms
- Data Governance & Security
- Data Pipeline Design
- Performance & Cost Optimisation
- CI/CD & DevOps
- Technical Leadership
- Stakeholder Management
- Mentoring & Team Development
Typical Experience Level
7-12+ years of experience in data engineering, data architecture, analytics engineering, or related disciplines, including significant hands-on experience with Databricks and cloud-based data platforms.
A strong candidate should be capable of operating at both strategic and hands-on technical levels, providing architectural direction while remaining close enough to the engineering detail to guide implementation and resolve complex technical challenges.
GCS is acting as an Employment Business in relation to this vacancy.
Other similar jobs
Popular job searches
Your next job
starts here.
JOB SPECIALISMS
LATEST JOBS
TOP SEARCHES
LOCATIONS
- Engineering
- Controls & Automation
- IT Support & Infrastructure
- Data
- BI & Data Analytics
- Project Management
- DevOps
- Hardware & Electronics
- Network security consultant
- IT Audit & Risk
- Software Development
- Testing & QA
LATEST JOBS
- Product Designer
- UX Designer
- Sr Software Architect
- ISP Technician
- Manual QA Analyst
- QA Test Analyst
- Integration/Assembly Technicia...
- Azure Cloud Architect
- Azure Cloud Security Architect...
- Product Owner
- Datacenter Technician
- Cloud Architect - Hybrid
TOP SEARCHES
LOCATIONS
- Engineer
- Data Scientist
- Senior Data Scientist
- Head of Data Science
- Trainee Data Scientist
- Data Science Graduate
- Senior Financial Accountant
- Management Accountant
- Cost Accountant
- Civil Engineer
- Senior Civil Engineer
- Civil Design Engineer