Data Engineer at Viscar Industrial Capacity

We are pleased to announce the above named vacancy in the Digital I.T Department with our client, a technology driven company.

Job Purpose

The position holder will provide leadership for planning, development, testing and delivery of big data solutions based on Hadoop and other open source technologies within the organization

Responsibilities

• Responsible for planning and delivery of highly scalable distributed big data systems, using different open source technologies including but not limited to Hadoop.
• Responsible for loading the necessary data from sources onto the shared Hadoop platform that are necessary for analysis and for commercial action
• Responsible for developing reliable data ingestion and preparation processes.
• Develop and implement information lifecycle management strategies for data platforms.

Requirements

• BSc Degree in computer science or engineering;
• 5 years BI or related experience

Key skills and knowledge  

• Experience with using Spark, Yarn, Hive and Oozie;
• Working knowledge of HBase, Solr, Kafka and Flume
• Some experience with other distributed technologies such as Cassandra, ElasticSearch and Flink.
• Experience in designing, building and managing applications to process large amounts of data in a Hadoop ecosystem;
• Experience with performance tuning applications on Hadoop and configuring Hadoop systems to maximise performance;
• Experience building systems to perform real-time data processing using Spark Streaming or Storm;
• Experience with managing the development life-cycle for agile software development projects;
• Experience working in a multi tenancy Hadoop environment.