Senior Data Engineer at Deimos

Role Overview

We are seeking a seasoned Senior Data Engineer to join our Engineering division and work with an established team dedicated to data-centric solutions. In this role, you will establish and enforce best practices to facilitate the effective delivery of high-quality data products. You will play a crucial role in a high-performing, interdisciplinary team and will report directly to an Engineering Manager.

The ideal candidate will possess a strong foundation in data processing, data warehousing, and data integration technologies.

What you will be doing:

• Design, build, and maintain the data lake of one of Deimos’ prestige clients, an African e-commerce giant.
• Create robust ETL processes and frameworks to support data transformation and integration from diverse data sources.
• Ensure high standards of data quality and integrity, implementing data governance and compliance measures as per client and regulatory requirements.
• Collaborate closely with the team on data requirements, their implementation, process enhancements, and domain-specific analysis.
• Conduct critical evaluations of the data systems and architectures, aiming to identify opportunities for improved data usability, efficiency, and customer satisfaction.
• Designing and developing robust, maintainable, and efficient code that aligns with best practices in data engineering.
• Identifying and addressing performance bottlenecks, resolving data-related issues, and optimising the performance of data processing and storage systems.
• Collaborating with senior stakeholders on the establishment, and growing of a world-class data practice in Deimos.

What you must have:

• At least five(5) years of experience in data engineering, with specific expertise in big data technologies and cloud-based data warehousing.
• Proven experience in data engineering with a solid understanding of key data warehousing, processing, and integration concepts.
• Experience managing data storage and optimisation in AWS S3 for high-performance data warehousing of large volumes of raw and processed data.
• Extensive knowledge of implementing scalable and efficient data processing workflows using Apache Spark (preferably on AWS EMR).
• A strong understanding of data integration and orchestration using AWS services such as Managed Workflows for Apache Airflow (MWAA) and AWS Lambda.
• Well-versed in data modelling and data architecture principles suitable for large-scale data environments.
• A strong understanding of programming principles with proficiency in Python or similar languages used in data processing.
• Excellent understanding of SQL
• Good working knowledge of version control using Git in a collaborative project setting.
• Experience working within an agile development environment, particularly with a focus on iterative and incremental data projects.
• A solid understanding of automated release management for data applications, utilising CI/CD pipelines for efficient deployment and operations.
• Experience with data modeling, data access, and data storage techniques.
• Experience creating and maintaining data catalogs using AWS Glue Catalog to organise metadata
• Familiarity with data management, governance practices, and cloud security protocols.

Nice to have

• Experience with resource provisioning and deployments on AWS with Terraform

About you

For us to achieve our ambitious vision together as a team, it is important for our Martians to lead at all levels, be self starters who take initiative and put their hands up for challenging tasks. A Growth Mindset is important to us and we encourage all our Martians to openly share knowledge, support and help each other, ask questions, get creative with new technologies and learn from setbacks.

Becoming a Martian means:

• Comfortably working and learning from a fully remote, culturally diverse team based predominantly in South Africa, Kenya, Nigeria and Ghana.
• Being an open, honest and respectful communicator.
• You enjoy asking questions, identifying areas of improvement and proposing solutions, no matter your job title or whether you have been with us for a day, a month or years!
• You are comfortable taking initiative and operating independently.
• You thrive in a fast paced environment, where change is constant.
• You find it exciting to work with various clients, from different industries, each with a different problem for you and your team to solve.
• Intentionally sharing tech and industry trends that excite you with your peers.
• Seeking continuous feedback and actively taking steps to continuously grow personally and professionally.

Want to know what you get by joining us?

• Become a member of a team where we value each individual's contribution from day 1 and empower you to make suggestions, get involved and do what you love most!
• Flexibility and the freedom to work remotely.
• Work-life balance where you are not expected to work over weekends or after hours.
• A forward thinking remote company that knows how important it is to stay connected as one team, by providing virtual social platforms for employee engagement.
• A monthly work from home allowance which you can use to set yourself up to work comfortably from home. Whether that is pens, notebooks, new headphones or work snacks!
• A MacBook or Windows laptop for you to do your best work on.
• Become part of a team of exceptionally clever and talented people who like to share their knowledge and learnings.
• We support your career growth and love to celebrate your successes and advancement!