Software Development Manager, Core Data Engineering

Job description

DESCRIPTION

The Amazon Search team builds the largest shopping search engine in the world. Whenever a customer searches or browses using an Amazon website or application, we connect them to the products and services they are looking for. The Search Data and Machine Learning Engineering team designs, builds, and operates distributed infrastructure and applications to process and analyze the petabytes of data that flow through Amazon Search. Our systems power research, train machine learnt models, deliver ranking data to our production systems, and provide business insight into Amazon's retail business. Our data powers live-site features, including search suggestions, query understanding, spelling, search result ranking, and personalization.

This role leads our Core Data Engineering (CoDE) team. They are responsible for the design, implementation and operation of our big data processing framework and the data processing pipelines within it. These pipelines process billions of user events and product updates per day, producing key live ranking features and powering our open data analytics platform. The analytics platform powers our search data science and ML efforts and enables the data driven business decisions of our thousands of internal users. Joining this team, you'll experience the benefits of working in a dynamic, entrepreneurial environment, while leveraging the resources of Amazon, one of the world's leading internet companies. We provide a highly customer-centric, team-oriented environment.

We are looking for a strong technologist with a track record leading big data projects and experience with modern Apache Hadoop Ecosystem tools such as Spark, Flink, Presto and Airflow. The CoDE team's primary responsibility is to provide high quality data feeds. This requires a focus on data quality and system reliability. The team is also responsible for the design and operation of data-marts and full-stack reporting and analytics portals used by more than ten thousand monthly amazon employees. Finally, the CoDE team works with ranking and other search science teams to develop new data pipelines. The team both develops framework to support such data work and consults on best practices and performance with applied science teams.

Responsibilities:
· Lead a team of engineers to design, develop and support a world-class distributed AWS based big data platform, handling billions of records each day and serving thousands of Amazon employees.
· Be a technical mentor and evangelist of modern open source big data tools including Spark, Flink, Presto, Hive metastores and Airflow.
· Architecting peta-byte scale data systems for performance and cost effectiveness.
· Own the technical strategy, roadmap and execution in the areas of distributed systems design, web service APIs and self-service analysis tools.
· Bring a strong perspective that drives change and motivates engineers to develop simple solutions to complex problems
· Mentor the professional development of each direct report through personal and performance management.
· Seek diverse perspectives to drive bottom-up innovation and create buy-in from all technical stakeholders inside and outside the team.
· Be an aggressive recruiter of engineering talent and be comfortable closing candidates.
Joining this team, you'll experience the benefits of working in a dynamic, entrepreneurial environment, while leveraging the resources of Amazon.com (AMZN), one of the world's leading internet companies. We provide a highly customer-centric, team-oriented environment in our offices located in Palo Alto, California

Desired profile

BASIC QUALIFICATIONS

· Bachelors in Computer Science, or related discipline, with at least 10 years of industry experience.
· At least 10 years of experience with Java, C++ and/or other commercial OO languages.
· At least 10 years of experience with scripting languages (eg. Python, Ruby, etc).
· At least 3 years of recent experience managing software development engineers.
· At least 6 years of experience building highly distributed systems / applications.
· At least 6 years of experience with large scale data processing and querying tools.
· At least 3 years of experience leading the development of Hadoop based data pipelines.