Data Engineer at Wikimedia Foundation

Summary

The Wikimedia Foundation is looking for a Data Engineer to join our team, reporting to the Director of Data Engineering. As a Data Engineer, you will be responsible for building, maintaining and expanding the shared data infrastructure that powers a big part of decision making in the Foundation as well as the Wiki Movement. This includes everything from building scalable pipelines using big data technology to defining and creating a suite of datasets which adhere to our privacy principles.

You are responsible for:

• Integrating data from multiple sources to gain insights in areas such as content, traffic, editors, readership and fundraising
• Building scalable data pipelines in collaboration with other data engineers as well as teams across the foundation including product analytics, platform engineering, survey, research and machine learning teams
• Designing the shared data platform that supports use cases for critical aspects of the Wikimedia mission: harassment prevention, image classification, bot detection, DDoS attacks flagging and many more
• Building and maintaining public metrics and datasets
• Implementing data quality monitoring that alerts the team of possible data issues
• Implementing a data governance and lineage solution for all Wikimedia data

Skills and Experience:

• 2+ years of relevant industry experience
• Advanced working knowledge of SQL, relational databases, query authoring, ideally in a variety of flavors (in our team alone we deal with MariaDB, HiveQL, CassandraQL, Spark SQL and Presto)
• Experience with one or more programming languages such as Python, Scala, and Java
• Experience building data pipelines using tools such as Airflow, Spark, Gobblin, Oozie, Yarn
• Familiarity with stream processing systems using Kafka, Spark streaming and/or Flink
• Excellent written and verbal communication skills
• Strong interpersonal and collaboration skills
• BS or MS degree, preferably in Computer Science, or equivalent work experience

Qualities that are important to us:

• Commitment to the mission of the organization and our values
• Commitment to our guiding principles
• Commitment to diversity, equity, and inclusion
• Cross-cultural sensitivity and awareness
• Collaborative working experience

Additionally, we'd love it if you have:

• Experience with Hadoop
• Understanding of related disciplines including Machine Learning, Statistics, Privacy and Algorithms
• Experience working with site reliability engineers