Senior Azure Data Engineering Expert at Andishi

Project Overview

• We are seeking an experienced Senior Data Engineer with strong Azure cloud, data engineering, and AI/ML platform development expertise to independently design, develop, and deliver a scalable enterprise-grade data platform.
• The project involves building and improving data processing solutions, developing reliable data pipelines, integrating cloud-based services, and preparing data infrastructure to support AI, machine learning, and advanced analytics applications.
• This is an independent contract engagement. The selected developer will take end-to-end responsibility for technical execution, from architecture and implementation through testing, documentation, and final delivery.
• We are looking for someone who can work autonomously, make sound technical decisions, troubleshoot complex problems, and deliver production-ready solutions.

Key Responsibilities

Data Platform Architecture and Development

• Design and implement scalable Azure-based data engineering architectures.
• Develop enterprise-grade data platforms supporting structured and unstructured datasets.
• Build batch and real-time data processing solutions.
• Design efficient data storage, processing, and integration architectures.
• Identify opportunities to improve platform performance, reliability, scalability, and cost efficiency.

Data Engineering and ETL Development

• Build, maintain, and optimize end-to-end ETL/ELT pipelines.
• Develop data ingestion and transformation workflows.
• Implement data integration between internal systems, external APIs, and cloud services.
• Develop advanced SQL queries, stored procedures, database functions, and data models.
• Implement data validation, error handling, and data quality controls.

Azure Cloud and Distributed Processing

• Develop solutions using Azure Data Factory, Azure Functions, Event Hubs, and related cloud-native services.
• Implement event-driven architectures using Azure Event Grid and Service Bus.
• Work with Azure Databricks and distributed processing technologies.
• Optimize data workflows for high-volume and real-time processing.
• Support cloud migration, integration, and infrastructure improvements.

AI and Machine Learning Data Infrastructure

• Develop data processing frameworks supporting AI/ML applications.
• Prepare and transform datasets for machine learning and advanced analytics.
• Implement reliable data workflows throughout the AI/ML data lifecycle.
• Develop data algorithms and processing logic for analytical applications.
• Ensure data pipelines are scalable, consistent, and suitable for downstream AI/ML workloads.

Software Development and Integration

• Develop backend services and RESTful APIs.
• Integrate databases, cloud applications, and third-party systems.
• Implement secure and maintainable application components.
• Follow software engineering best practices, including version control, code reviews, and automated testing.

Deployment, Testing, and Documentation

• Configure and maintain CI/CD pipelines using Azure DevOps and GitHub.
• Perform unit, integration, and end-to-end testing.
• Identify and resolve technical issues independently.
• Produce technical documentation, architecture diagrams, and implementation guides.
• Provide a complete handover of the developed solution.

Required Technical Stack

• Cloud and Infrastructure
• Microsoft Azure
• Azure Data Factory
• Azure Databricks
• Azure Functions
• Azure Event Hubs
• Azure Event Grid
• Azure Service Bus
• Azure Cosmos DB
• Azure SQL Data Warehouse / Azure SQL
• Programming and Development
• Python
• SQL / T-SQL
• C# / .NET
• Java
• Node.js
• Unix Shell Scripting
• Data Engineering and Processing
• ETL / ELT
• Apache Spark / PySpark
• Spark SQL
• Hive SQL
• Data Modeling
• RDBMS
• Structured and Unstructured Data Processing
• Batch and Stream Processing
• Event-Driven Architecture
• DevOps and Integration
• Azure DevOps
• Git / GitHub
• CI/CD Pipelines
• RESTful APIs
• JSON / JSON-LD / XML
• Version Control and Automated Deployment