Big Data Engineer at Techsavanna Technology Limited

Role Purpose

• Perform technical aspects of big data technologies development including design, developing prototypes, and coding assignments.
• Build software applications that will be used to deliver data to end users for exploration, advanced analytics and visualizations for day to day business reporting.
• Plan and deliver highly scalable distributed big data systems, using different open source technologies including but not limited to Hadoop, Kafka, Nifi, Hbase, Cassandra, Hive, MongoDB, Postgres, Redis DB etc.
• Code, test, and document scripts for managing large big data clusters.
• Architect and build data fabric to collect and transport data to the Hadoop ecosystem.

Core competencies, knowledge and experience:

Business Competencies:

Working with Others

• Work closely with infrastructure, network, APIs, database, business intelligence and application teams to ensure business applications are highly available and performing within agreed service levels.
• Build tools for yourself and others to increase efficiency and to make hard or repetitive tasks easy and quick.

Operational Excellence

• Receive escalated, technically complex mission critical issues, and maintain ownership of the issue until it is resolved completely.
• Hands on to troubleshoot incidents, formulate theories and test hypotheses, and narrow down possibilities to find the root cause.
• Support Big Data and analytical solutions leveraging transformational technologies.
• Develop tools, and scripts to automate troubleshooting activities.
• Drive further improvements in the platform, tooling and processes.
• Upgrading products/services and applying patches as necessary.
• Maintaining backup and restoring the ETL and Reports repositories and other Systems binaries and source codes.

Creativity and Innovation

• Apply machine learning algorithms and libraries for problem solving skills with an emphasis on product development.
• Research and provide input on design approach, performance and base functionality improvements for various procedures and applications.
• Generation of ideas to improve efficiency in software and application services offered to the business, and/or generate revenue for business.

Business Know-how

• Conduct business user requirements analysis and specification with relevant stakeholders.
• Perform the necessary technical design and development functionality to ensure that business application systems can be effectively developed and implemented.
• Understand and use computer science fundamentals, including data structures, algorithms, and computer architecture, for systems development.

Working With Change

• Responds flexibly to changing situations
• Manages the business and people aspects of change to drive performance.

Project and Programme Management

• Defines scope and deliverables in terms of time, cost, quality and business benefit.
• Schedules activity and identifies resource needs, dependencies and synergies.
• Evaluates progress, mitigates risks and addresses issues.

Functional Competencies:

• Result-oriented, with great ideas and solid quantitative skills to turn ideas into reality.
• You should have solid data management and analytical skills.
• Data science and machine learning programming.

Requirements

• Experience with Java
• Experience with relational data stores as well as one or more NoSQL data stores (e.g., Mongo, Cassandra)
• Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming
• Demonstrated proficiency with data structures, algorithms, distributed computing, and ETL systems
• Experience with various messaging systems, such as Kafka or RabbitMQ
• Good knowledge of and experience with big data frameworks such as Apache Hive, Spark, Hadoop
• A working knowledge and experience of SQL scripting

Good to have

• Experience in deploying and managing Spark Machine Learning models at scale
• Hands on implementation and delivery of Spark workloads in an Agile DevOps working environment
• Highly proficient in more than one modern language, e.g., Java/C#/NodeJS/Python/Scala