Job Search

Senior Data Engineer
Location: Abu Dhabi, UAE - Remote Global
Type: Contract

Role Summary

We are seeking a Senior Data Engineer to architect and build the foundational data infrastructure for
a large-scale AI transformation programme. You will move beyond standard ETL tasks to design and
deploy enterprise-grade Data Lakehouses that serve as the "Single Source of Truth" for AI agents and
advanced analytics platforms.
You will lead the technical execution of data and platform foundation initiatives. Your mandate is to
map rigid enterprise systems (e.g., SAP S/4HANA, Ariba, etc.) and collaborate with teams to
understand complex unstructured data (technical drawings, regulatory text) while building high
performance pipelines and systems that power Intelligent Supply Chain forecasting and generative AI
tools.
You will operate within a structured "Sprint Zero" environment, ensuring data lineage, governance,
and security meet strict enterprise standards.

Key Responsibilities

1. Data Lakehouse Architecture (Supply Chain)
* ERP Integration: Architect and deploy ingestion pipelines to extract high-volume
transactional data from SAP S/4HANA, Ariba, and PLM systems, ensuring near real-time
availability for forecasting models.
* External Feed Integration: Build connectors for external market intelligence feeds (e.g., S&P
Global, Orbis, EcoVadis) to enrich internal procurement data with macroeconomic and
geopolitical signals.
* Unified Data Model: Design and implement a standardized procurement data model and
taxonomy across multiple entities, harmonizing fragmented datasets into a cohesive layer for
analytics.

2. Unstructured Data Pipelines
* Complex Ingestion: Engineer pipelines to ingest and process unstructured technical data,
including PDF tender documents, CAD metadata, and historical CONOPS, transforming them
into vector-ready formats for RAG (Retrieval-Augmented Generation) applications.
* Vector Database Management: Manage and optimize Vector Databases (e.g., Weaviate) to
store embeddings of archival proposals and engineering snippets, ensuring high-speed
retrieval for AI drafting assistants.
* Digital Thread Implementation: Establish data lineage and traceability protocols that link
requirements to physical components, supporting Model-Based Systems Engineering (MBSE)
Digital Thread initiatives.

3. Governance & Security
* Enterprise-Grade Security: Implement Role-Based Access Control (RBAC), audit logging,
and data redaction policies to ensure compliance with export controls and strict on-premise
security requirements.
* Quality Control: Deploy automated data quality frameworks to validate BOM (Bill of
Materials) completeness and cost data accuracy before it reaches AI models.
* Infrastructure Optimization: Optimize pipelines for on-premise GPU clusters and air-gapped
environments, ensuring efficiency within existing infrastructure limits.

Technical Requirements

* Core Stack: Expert proficiency in Python, SQL, and modern data engineering frameworks
(Apache Spark, Kafka, Airflow).
* Enterprise ERP: Strong experience extracting data from complex ERP environments,
specifically SAP S/4HANA and SAP Ariba. Familiarity with SAP BTP is a plus.
* Database Technologies: Deep understanding of Data Lakehouse architectures
(Databricks/Delta Lake), Relational Databases (PostgreSQL), and Vector Databases
(Weaviate/Milvus).
* Data Pipeline Development: Experience building pipelines for RAG solutions, conversational
agents, and classical ML models using tools such as dbt, Dagster, or Prefect.
* DevOps/DataOps: Proficiency with containerization (Docker, Kubernetes) and CI/CD
pipelines for deploying data workflows in secure environments.

Professional Qualifications

* Experience: 5+ years of experience in Data Engineering, with at least 2 years focused on
building pipelines for Machine Learning or Generative AI applications in an enterprise setting.
* Domain Knowledge: Experience in Supply Chain, Manufacturing, or Defence sectors is
highly desirable. Ability to understand Bill of Materials (BOM) structures and procurement
lifecycles.
* Problem Solving: Ability to navigate the "Governance Collision" between agile data work and
rigid systems engineering requirements, ensuring data deliverables meet formal Stage Gate
reviews.
* Collaboration: Proven ability to work alongside Data Scientists and Backend Engineers to
define data schemas that support predictive modelling and AI agents.

Why This Role?

You will be the architect of the data foundation behind advanced AI capabilities. Your work will directly
enable AI agents to support procurement workflows, forecasting, analytics, and engineering decision
making. If you are ready to build the robust infrastructure that turns raw data into strategic advantage,
this role is for you.

GCS is acting as an Employment Business in relation to this vacancy.

Read less

Senior Data Engineer Role (Python / Spark / Databricks) - Hybrid - Contract - Banking

Role - Senior Data Engineer

Rate - £850 p/d (Inside IR35)

Duration - 6 months with very likely extension

Location - Hybrid / Liverpool Street (London) - 3 days per week in a Liverpool Street office

About the Role

Global financial institution seeking a highly skilled Senior Data Engineer to join a growing data engineering function responsible for delivering critical data platforms and analytics capabilities across the business.

This is a genuinely hands-on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise-scale data platforms, working alongside engineers, architects and business stakeholders to deliver robust and scalable solutions.

The role offers the opportunity to work on complex data challenges within a modern cloud-based environment, leveraging the latest data engineering technologies and best practices.

Key Responsibilities

Data Engineering & Development

Design, develop and maintain scalable data pipelines using Python, Databricks and PySparkBuild and optimise ETL/ELT solutions processing large-scale datasetsDevelop high-quality, reusable code and engineering frameworksDeliver robust, performant and maintainable data solutionsTroubleshoot and resolve complex production and performance issuesSupport the ongoing enhancement and modernisation of enterprise data platforms

Databricks & Data Platform Engineering

Develop and support Databricks-based data solutions and Lakehouse architecturesUtilise Delta Lake and Spark technologies to deliver scalable data processing capabilitiesOptimise distributed data processing workloads for performance and reliabilityImplement data quality controls and monitoring solutionsContribute to platform stability, scalability and operational excellence

Solution Delivery

Work closely with business and technology stakeholders to understand requirements and deliver effective solutionsParticipate in Agile delivery processes including sprint planning, refinement and estimationSupport the delivery of strategic data initiatives across multiple business areasCollaborate with architects, engineers and product teams to deliver high-quality outcomesIdentify and resolve technical risks and implementation challenges

Engineering Best Practice

Promote software engineering best practices including testing, code reviews and version controlSupport CI/CD adoption and deployment automationContribute to coding standards and engineering governanceDrive continuous improvement across development and support processesMentor junior engineers and contribute to knowledge sharing within the team

Required Skills & Experience

Strong commercial experience as a Data Engineer or Senior Data EngineerExpert-level Python development skillsExtensive hands-on experience with Databricks in enterprise environmentsStrong Apache Spark and PySpark expertiseAdvanced SQL development and optimisation skillsExperience designing and building large-scale ETL and data integration solutionsStrong understanding of distributed computing concepts and data processing frameworksExperience working with modern cloud platforms including Azure, AWS or GCPKnowledge of software engineering principles, testing and code quality practicesExperience working within Agile development environmentsExcellent problem-solving and troubleshooting capabilities

Desirable Experience

Experience within investment banking, financial services or other highly regulated environmentsExposure to risk, trading, treasury or regulatory data domainsExperience with Kafka, streaming technologies or real-time data processingDatabricks certificationsExperience with Airflow, Databricks Workflows or similar orchestration toolsCI/CD pipeline development and DevOps practicesDocker and KubernetesExperience supporting data science or machine learning workloads

This role will suit a Senior Data Engineer who enjoys remaining close to the technology and has strong recent experience building production-grade solutions using Python, Databricks, Spark/PySpark and SQL in enterprise environments.

GCS is acting as an Employment Business in relation to this vacancy.

Read less

Senior Data Engineer Role (Python / Spark / Databricks) - Hybrid - Contract - Banking

Role - Senior Data Engineer

Rate - £850 p/d (Inside IR35)

Duration - 6 months with very likely extension

Location - Hybrid / Liverpool Street (London) - 3 days per week in a Liverpool Street office

About the Role

Global financial institution seeking a highly skilled Senior Data Engineer to join a growing data engineering function responsible for delivering critical data platforms and analytics capabilities across the business.

This is a genuinely hands-on engineering role, requiring strong expertise in Python, Databricks, Spark/PySpark and SQL. The successful candidate will be actively involved in the design, development, optimisation and support of enterprise-scale data platforms, working alongside engineers, architects and business stakeholders to deliver robust and scalable solutions.

The role offers the opportunity to work on complex data challenges within a modern cloud-based environment, leveraging the latest data engineering technologies and best practices.

Key Responsibilities

Data Engineering & Development

Design, develop and maintain scalable data pipelines using Python, Databricks and PySparkBuild and optimise ETL/ELT solutions processing large-scale datasetsDevelop high-quality, reusable code and engineering frameworksDeliver robust, performant and maintainable data solutionsTroubleshoot and resolve complex production and performance issuesSupport the ongoing enhancement and modernisation of enterprise data platforms

Databricks & Data Platform Engineering

Develop and support Databricks-based data solutions and Lakehouse architecturesUtilise Delta Lake and Spark technologies to deliver scalable data processing capabilitiesOptimise distributed data processing workloads for performance and reliabilityImplement data quality controls and monitoring solutionsContribute to platform stability, scalability and operational excellence

Solution Delivery

Work closely with business and technology stakeholders to understand requirements and deliver effective solutionsParticipate in Agile delivery processes including sprint planning, refinement and estimationSupport the delivery of strategic data initiatives across multiple business areasCollaborate with architects, engineers and product teams to deliver high-quality outcomesIdentify and resolve technical risks and implementation challenges

Engineering Best Practice

Promote software engineering best practices including testing, code reviews and version controlSupport CI/CD adoption and deployment automationContribute to coding standards and engineering governanceDrive continuous improvement across development and support processesMentor junior engineers and contribute to knowledge sharing within the team

Required Skills & Experience

Strong commercial experience as a Data Engineer or Senior Data EngineerExpert-level Python development skillsExtensive hands-on experience with Databricks in enterprise environmentsStrong Apache Spark and PySpark expertiseAdvanced SQL development and optimisation skillsExperience designing and building large-scale ETL and data integration solutionsStrong understanding of distributed computing concepts and data processing frameworksExperience working with modern cloud platforms including Azure, AWS or GCPKnowledge of software engineering principles, testing and code quality practicesExperience working within Agile development environmentsExcellent problem-solving and troubleshooting capabilities

Desirable Experience

Experience within investment banking, financial services or other highly regulated environmentsExposure to risk, trading, treasury or regulatory data domainsExperience with Kafka, streaming technologies or real-time data processingDatabricks certificationsExperience with Airflow, Databricks Workflows or similar orchestration toolsCI/CD pipeline development and DevOps practicesDocker and KubernetesExperience supporting data science or machine learning workloads

This role will suit a Senior Data Engineer who enjoys remaining close to the technology and has strong recent experience building production-grade solutions using Python, Databricks, Spark/PySpark and SQL in enterprise environments.

GCS is acting as an Employment Business in relation to this vacancy.

Read less

GCS is acting as an Employment Business in relation to this vacancy.

Read less

Senior Software Engineer - Data, Machine Learning & Applications

Job Summary

We are seeking a versatile Senior Software Engineer with experience in application development, databases, data processing, and Machine Learning systems. This role will focus on designing and building robust, scalable technical solutions while ensuring high standards for software quality and performance.

The ideal candidate combines strong software engineering fundamentals with experience working with relational databases, SQL, data solutions, and modern application development technologies.

ResponsibilitiesDesign, develop, test, and maintain scalable software and Machine Learning applications.Write clean, efficient, and well-documented code that meets performance and business requirements.Participate in application architecture, system design, and technical solution development.Design and build relational databases for application and data processing needs.Write and maintain SQL statements for data retrieval, insertion, and processing.Support data acquisition, data warehousing, archival, and recovery strategies.Evaluate data sources and ensure data quality and integration standards are met.Collaborate with customers, business analysts, and technical teams to understand requirements.Develop and execute testing strategies to ensure reliable software delivery.Track defects, test results, and quality metrics.Identify opportunities to improve software processes, application performance, and technical solutions.Participate in code reviews and recommend improvements to existing systems.Support applications throughout the full software development lifecycle.Required QualificationsBachelor's degree in Computer Science, Information Systems, or a related field, or equivalent professional experience.8+ years of experience in software development, systems engineering, testing, data engineering, or related areas.Strong programming experience with technologies such as:Java.NETNode.jsRubyAngularJSStrong SQL and relational database experience.Experience designing, developing, or supporting scalable software applications.Strong object-oriented design skills.Experience with software testing and quality assurance practices.Ability to work across application development, data, and business teams.Excellent analytical and problem-solving skills.Preferred QualificationsExperience building or supporting Machine Learning systems.Experience with data warehousing or large-scale data processing.Experience evaluating and integrating new data sources.Experience with system architecture and application performance optimization.

GCS is acting as an Employment Business in relation to this vacancy.

Read less

Senior Software Engineer - Quality, Applications & Data

We are seeking an experienced Senior Software Engineer to design, develop, test, and support high-quality software applications and data solutions. This individual will play a key role throughout the full software development lifecycle, with responsibilities spanning software development, quality assurance, test automation, database development, and Machine Learning applications.

The ideal candidate will bring a strong technical foundation in modern programming languages and frameworks, software testing methodologies, relational databases, and system design.

ResponsibilitiesDevelop and execute formal test plans to ensure the delivery of high-quality software applications.Participate in test planning, test case development, test automation, and test execution.Define, track, and analyze quality assurance metrics, including defects, test results, and overall test status.Identify software quality issues, document findings, and assist with problem resolution.Collect and analyze data to evaluate software processes and identify opportunities for improvement.Perform quality audits to ensure IT standards, processes, procedures, and methodologies are consistently followed.Provide guidance and quality process training to technical teams and serve as a subject matter resource for quality best practices.Design, develop, test, and document new or enhanced software applications and Machine Learning systems.Write clean, scalable, well-documented, and high-quality code that meets performance and business requirements.Participate in the full software development lifecycle, including requirements analysis, system design, coding, testing, implementation, maintenance, and production support.Design and build relational database solutions for application data storage and processing.Develop and maintain SQL queries and database integrations.Evaluate and maintain database structures, including adding fields and supporting data quality initiatives.Support data acquisition, warehousing, archival, and recovery strategies.Work closely with business partners, customers, analysts, and technical teams to understand requirements and deliver effective technical solutions.Ensure solutions align with business objectives, IT strategies, and architectural standards.Participate in system and component design, data architecture, performance monitoring, and product evaluations.Make recommendations regarding new development, code reuse, and build-versus-buy decisions.Stay current with emerging technologies, programming languages, development methodologies, database tools, and software engineering best practices.Required QualificationsBachelor's degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent professional experience.8+ years of experience in software development, systems development, testing, quality assurance, or related technical roles.Experience developing and testing enterprise-level software applications.Experience with one or more of the following technologies:Java.NETNode.jsRubyAngularJSHTML/CSSSQLStrong understanding of software development, quality assurance, and systems integration methodologies.Experience developing and maintaining automated test cases and test strategies.Strong object-oriented design and programming skills.Strong analytical and problem-solving abilities.Ability to write SQL statements to retrieve, insert, update, and maintain data.Strong understanding of relational database design, tables, schemas, and data structures.Experience with design documentation, code reviews, in-line documentation, and testing best practices.Knowledge of the full software development lifecycle and application architecture.Preferred QualificationsExperience developing or supporting Machine Learning applications or systems.Experience with data warehousing, data acquisition, archival, and recovery processes.Experience working in Agile software development environments.Experience with performance monitoring and application optimization.Strong understanding of software architecture and component-level design.Ability to work effectively across business and technical teams.What We're Looking For

The ideal candidate is a highly experienced and versatile technology professional who can bridge the gap between software development, quality assurance, data management, and application architecture. This person should be comfortable working hands-on with code while also contributing to testing strategies, technical design decisions, process improvements, and overall software quality.

GCS is acting as an Employment Business in relation to this vacancy.

Read less

GCS is acting as an Employment Business in relation to this vacancy.

Read less

Senior Data Engineer
Location: Dublin, Ireland (Hybrid - 3 days in office)
Employment Type: Full-Time

About Us

We're an innovative smart grid technology company building systems that connect and optimise energy assets as part of a Virtual Power Plant, helping accelerate the decarbonisation of the energy system. Our in-house software and hardware platform spans embedded controllers with rich sensor data collection deployed on customer sites, through to backend applications for control, automation, and data analytics - all the way to customer-facing visualisation and reporting tools.

We're a leader in the evolving smart grid space, offering Load Shifting, Demand Response, and Frequency Response services. As the electricity network evolves to intelligently balance producers and consumers of all sizes, we're building the technology to meet the needs of a fast-changing energy landscape - and growing our IT team to support the next stage of our growth.

The Role

We're looking for a Senior Data Engineer to join our Data and Analytics IT team. You'll work in an agile environment to build a Data Lake and Data Warehouse solution supporting our Data Science and Business Management teams - integrating data from external sources, IoT sensors across our install base, and business systems including CRM and billing.

This integrated data will power solutions for energy optimisation, energy trading, and predictive fault detection, as well as Power BI reporting for internal and external stakeholders. You'll play a key role in our Smart Grid Platform's evolution - from analysis and design through to implementation and delivery of industry-leading big data and data warehousing solutions, built on Azure's cloud services.

Ideally, you've taken projects through the full development lifecycle on Azure - from design through to production.

What You'll Do

Lead the design and delivery of innovative data solutions using the latest Azure ecosystem servicesApply industry best practices and your own experience to build a data platform that enables business growthLiaise with internal and external stakeholders to gather and document technical requirementsContribute ideas to strengthen standards and processes, and proactively look for ways to improve solutions and process efficiency

What You'll Bring

3+ years' professional experience, including 3+ years with Azure or AWS (Azure Data certification a plus)Experience with Data Warehouse design and the Kimball methodologyETL experience using SSIS and Azure Data FactoryHands-on experience with SQL Server, SQL Data Warehouse, and Azure Data LakeSkill in SQL Server schema design and performance optimisationPython programming for data engineering and cleansing (Spark/Databricks experience a plus)Experience with Power BI and DAXSolid understanding of data security on AzureA proven ability to translate complex business requirements into scalable, resilient, end-to-end data system designs (HA/DR)Familiarity with: SQL Server, ETL design, Azure Data Lake, Azure Data Factory, Azure SQL Server, Azure SynapseCore languages: T-SQL, PowerShell, Azure API, DAX

Nice to Have

Experience in the renewable energy sectorA third-level degree in Maths, Physics, Engineering, Computer Science, or a related fieldCommercial experience in data warehousing and business intelligence projectsExperience with large, complex datasets across multiple languages (T-SQL, Python, R, etc.)Experience translating diverse business domains into dimensional designs for analytics and reportingBasic exposure to machine learning, data visualisation, or AI techniques, and experience collaborating with Data Science/ML teamsData governance experience

GCS is acting as an Employment Agency in relation to this vacancy.

Read less

GCS is acting as an Employment Business in relation to this vacancy.

Read less

GCS is acting as an Employment Business in relation to this vacancy.

Read less