Intermediate Data Engineer at Deimos

Role Overview

We are looking for a skilled Intermediate Data Engineer to join our Engineering division, contributing to a team focused on data-driven solutions. In this role, you will apply and adhere to established best practices to ensure the successful delivery of high-quality data products. You will be an essential member of a high-performing, interdisciplinary team and will report to an Engineering Manager.

What you will be doing

• Assisting in the maintenance & improvement of the data lake for a leading African e-commerce group.
• Collaborating with the team on defining data requirements, implementing solutions, improving processes, and performing domain-specific data analysis.
• Performing critical assessments of data systems and architectures to pinpoint areas for enhanced data usability, operational efficiency, and customer satisfaction.
• Developing robust, maintainable, and efficient data engineering code in line with industry best practices.
• Identifying and addressing performance bottlenecks, troubleshooting data-related issues, and enhancing the efficiency of data processing and storage systems.

What you must have

• Minimum of three (3) years of experience in data engineering, focusing on big data technologies and cloud-based data warehousing.
• Proficiency in managing data storage and optimization in AWS S3 for data warehousing, handling both large volumes of raw and processed data.
• Solid experience in developing scalable and efficient data processing workflows using Apache Spark on AWS EMR.
• Good understanding of data integration and orchestration using AWS services such as Managed Workflows for Apache Airflow (MWAA) and AWS Lambda.
• Knowledgeable in data modelling and architecture principles applicable to large-scale data environments.
• Proficient in programming with Python or similar languages commonly used in data processing.
• Familiarity with using Git for version control in a team-oriented project environment.
• Experience in an agile development setting, with an emphasis on the iterative and incremental development of data projects.
• Understanding of automated release management for data applications, employing CI/CD pipelines for streamlined deployment and operations.
• Experience in setting up and managing data catalogs with AWS Glue Catalog to effectively organize metadata.
• Desirable: Some experience with using Terraform for resource provisioning and deployments on AWS.

About you

For us to achieve our ambitious vision together as a team, It is important for our Martians to lead at all levels, be self starters who take initiative and put their hands up for challenging tasks. A growth mindset is important to us and we encourage all our Martians to openly share knowledge, support and help each other, ask questions, get creative with new technologies and learn from setbacks.

Becoming a Martian means:

• Comfortably working and learning from a fully remote, culturally diverse team based predominantly in South Africa, Kenya, Nigeria and Ghana.
• Being an open, honest and respectful communicator.
• You enjoy asking questions, identifying areas of improvement and proposing solutions, no matter your job title or whether you have been with us for a day, a month or years!
• You are comfortable taking initiative and operating independently.
• You thrive in a fast paced environment, where change is constant.
• You find it exciting to work with various clients, from different industries, each with a different problem for you and your team to solve.
• Intentionally sharing tech and industry trends that excite you with your peers.
• Seeking continuous feedback and actively taking steps to continuously grow personally and professionally.

Want to know what you get by joining us?

• Become a member of a team where we value each individual's contribution from day 1 and empower you to make suggestions, get involved and do what you love most!
• Flexibility and the freedom to work remotely.
• Work-life balance where you are not expected to work over weekends or after hours.
• A forward thinking remote company that knows how important it is to stay connected as one team, by providing virtual social platforms for employee engagement.
• A monthly work from home allowance which you can use to set yourself up to work comfortably from home. Whether that is pens, notebooks, new headphones or work snacks!
• A MacBook or Windows laptop for you to do your best work on.
• Become part of a team of exceptionally clever and talented people who like to share their knowledge and learnings.
• We support your career growth and love to celebrate your successes and advancement!