Oozie at Yahoo

•Download as PPTX, PDF•

0 likes•703 views

Mona Chitnis

Oozie at Yahoo - new features and learnings

Engineering

Oozie at Yahoo
Purshotam Shah, Ryota Egashira
Oozie Meetup 06/03 2014

Table of Contents
Yahoo Confidential & Proprietary
▪Scale at Yahoo!
▪Scale and Performance
▪Features - Usability
▪High availability and Load Balancing
▪Customer Asks

Scale at Yahoo!
▪Busiest cluster
› 1 million+ workflows per month
› 45 - 55K workflows per day
› 40 - 50K coord actions per day
› 800 - 900 coordinators (5m, 15m, 30m, hourly, daily and weekly)
› 30 - 40 bundles
▪Most complex bundle - 230 coordinators
▪Most complex workflow - 85 forks
▪Video Transcoding - 100-300 workflows per min

Scale and Performance
▪ Database
› CLOB to BLOB to compress and store inline
› Remove unnecessary hadoop config stored in protoActionConf
› Select only needed columns instead of loading whole row
› Partition tables by created time (in Oracle)
▪ Other
› Huge improvements to materialization of coordinator actions
› Reduce Launcher overhead
• Merge the number of small files created per action to one sequence file
• Launcher libraries shipped only once to HDFS
• Uber mode launcher with Hadoop 2.x
› Synchronously execute commands without queueing to speed up action transition
› Automatically killing abandoned coordinator job
› gzip compression for Rest API

Features - Usability
▪UI improvements
› Active Jobs, Custom Global Filters, Child Jobs for Pig/Hive actions
▪Faster log streaming with more filters
▪Updating coordinator definition on the fly
▪Rerun workflows without having to specify all properties again
▪Mark coordinator and actions as ignored
▪Sharelib Enhancements
› Update on the fly without failing jobs
› Command to list different sharelib available
› Specify directories using metafile instead of single share lib directory

High Availability and Load Balancing
▪HCat integration
▪SLA
▪Sharelib
▪Server-server authentication
▪Distributed sequence

Customer Asks
▪Coordinator dependency management
› Ability to view dependencies and rerun part of a pipeline
▪Better error handling and automatic retries
▪Ability to Suspend/Turn off SLA alerting
▪One-click launcher log viewing
▪Zero downtime

Rohini Palaniswamy
Mona Chitnis
Michelle Chiang
Purshotam Shah
Ryota Egashira
Olga L. Natkovich
Yahoo Oozie Team

What's hot

October 2014 HUG : Oozie HAYahoo Developer Network

Oozie @ Riot GamesMatt Goeke

Everything you wanted to know, but were afraid to ask about OozieChicago Hadoop Users Group

May 2012 HUG: Oozie: Towards a scalable Workflow Management System for HadoopYahoo Developer Network

Oozie HUG May12mislam77

July 2012 HUG: Overview of Oozie Qualification ProcessYahoo Developer Network

Data Pipeline Management Framework on OozieShareThis

Hadoop OozieMadhur Nawandar

SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud InfrastructureMatt Ray

TXLF: Chef- Software Defined Infrastructure Today & TomorrowMatt Ray

Ansible for large scale deploymentKarthik .P.R

Oozie or Easy: Managing Hadoop Workloads the EASY WayDataWorks Summit

Clogeny Hadoop ecosystem - an overviewMadhur Nawandar

Empowering developers to deploy their own data storesTomas Doran

May 2013 HUG: Apache Sqoop 2 - A next generation of data transfer toolsYahoo Developer Network

RESTFul development with Apache slingSergii Fesenko

Cool MariaDB Plugins Colin Charles

Hadoop HDFS Madhur Nawandar

JavaCro'14 - Using WildFly core to build high performance web server – Tomaž ...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association

Liquibase & Flyway @ Baltic DevOpsAndrei Solntsev

What's hot (20)

October 2014 HUG : Oozie HA

Oozie @ Riot Games

Everything you wanted to know, but were afraid to ask about Oozie

May 2012 HUG: Oozie: Towards a scalable Workflow Management System for Hadoop

Oozie HUG May12

July 2012 HUG: Overview of Oozie Qualification Process

Data Pipeline Management Framework on Oozie

Hadoop Oozie

SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure

TXLF: Chef- Software Defined Infrastructure Today & Tomorrow

Ansible for large scale deployment

Oozie or Easy: Managing Hadoop Workloads the EASY Way

Clogeny Hadoop ecosystem - an overview

Empowering developers to deploy their own data stores

May 2013 HUG: Apache Sqoop 2 - A next generation of data transfer tools

RESTFul development with Apache sling

Cool MariaDB Plugins

Hadoop HDFS

JavaCro'14 - Using WildFly core to build high performance web server – Tomaž ...

Liquibase & Flyway @ Baltic DevOps

Viewers also liked

Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas NYahoo Developer Network

Building and managing complex dependencies pipeline using Apache OozieDataWorks Summit/Hadoop Summit

A Basic Hive InspectionLinda Tillman

Hive tuningMichael Zhang

HIVE: Data Warehousing & Analytics on HadoopZheng Shao

August 2016 HUG: Recent development in Apache OozieYahoo Developer Network

Viewers also liked (6)

Apache Hadoop India Summit 2011 talk "Oozie - Workflow for Hadoop" by Andreas N

Building and managing complex dependencies pipeline using Apache Oozie

A Basic Hive Inspection

Hive tuning

HIVE: Data Warehousing & Analytics on Hadoop

August 2016 HUG: Recent development in Apache Oozie

Similar to Oozie at Yahoo

Stack Exchange Infrastructure - LISA 14GABeech

Managing multi tenant resource toward Hive 2.0Kai Sasaki

Workflow Engines for HadoopJoe Crobak

SharePoint Saturday The Conference 2011 - SP2010 PerformanceBrian Culver

SharePoint Saturday San Antonio: SharePoint 2010 PerformanceBrian Culver

Service-Oriented Design and Implement with Rails3Wen-Tien Chang

Facebook - Jonthan Gray - Hadoop World 2010Cloudera, Inc.

DrupalCampLA 2014 - Drupal backend performance and scalabilitycherryhillco

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & AlluxioAlluxio, Inc.

Stinger.Next by Alan Gates of HortonworksData Con LA

MariaDB 10.1 what's new and what's coming in 10.2 - Tokyo MariaDB MeetupColin Charles

Storage Infrastructure Behind Facebook Messagesfeng1212

WildFly v9 - State of the Union Session at Voxxed, Istanbul, May/9th 2015.Dimitris Andreadis

Sql plsql online trainingenrollmy training

Best practices for MySQL High Availability TutorialColin Charles

ReactPHP + SymfonyDavid Bergunder

Large-scale Web Apps @ PinterestHBaseCon

Drupal performancePiyuesh Kumar

A Second Look at Oracle RAC 12cLeighton Nelson

Caching strategies with luceeGert Franz

Similar to Oozie at Yahoo (20)

Stack Exchange Infrastructure - LISA 14

Managing multi tenant resource toward Hive 2.0

Workflow Engines for Hadoop

SharePoint Saturday The Conference 2011 - SP2010 Performance

SharePoint Saturday San Antonio: SharePoint 2010 Performance

Service-Oriented Design and Implement with Rails3

Facebook - Jonthan Gray - Hadoop World 2010

DrupalCampLA 2014 - Drupal backend performance and scalability

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio

Stinger.Next by Alan Gates of Hortonworks

MariaDB 10.1 what's new and what's coming in 10.2 - Tokyo MariaDB Meetup

Storage Infrastructure Behind Facebook Messages

WildFly v9 - State of the Union Session at Voxxed, Istanbul, May/9th 2015.

Sql plsql online training

Best practices for MySQL High Availability Tutorial

ReactPHP + Symfony

Large-scale Web Apps @ Pinterest

Drupal performance

A Second Look at Oracle RAC 12c

Caching strategies with lucee

Recently uploaded

Steel Structures - Building technology.pptxNikhil Raut

NO1 Certified Black Magic Specialist Expert Amil baba in Uae Dubai Abu Dhabi ...Amil Baba Dawood bangali

Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsyncssuser2ae721

Introduction-To-Agricultural-Surveillance-Rover.pptxk795866

Internet of things -Arshdeep Bahga .pptxVelmuruganTECE

welding defects observed during the weldingMuhammadUzairLiaqat

Arduino_CSE ece ppt for working and principal of arduino.pptSAURABHKUMAR892774

Software and Systems Engineering Standards: Verification and Validation of Sy...VICTOR MAESTRE RAMIREZ

Solving The Right Triangles PowerPoint 2.pptJasonTagapanGulla

Risk Assessment For Installation of Drainage Pipes.pdfROCENODodongVILLACER

Unit7-DC_Motors nkkjnsdkfnfcdfknfdgfggfgsaravananr517913

🔝9953056974🔝!!-YOUNG call girls in Rajendra Nagar Escort rvice Shot 2000 nigh...9953056974 Low Rate Call Girls In Saket, Delhi NCR

Virtual memory management in Operating SystemRashmi Bhat

Research Methodology for Engineering pdfCaalaaAbdulkerim

Energy Awareness training ppt for manufacturing process.pptxsiddharthjain2303

Class 1 | NFPA 72 | Overview Fire Alarm Systemirfanmechengr

Call Girls Narol 7397865700 Independent Call Girlsssuser7cb4ff

Vishratwadi & Ghorpadi Bridge Tender documentsSachinPawar510423

Introduction to Machine Learning Unit-3 for II MECHC Sai Kiran

Sachpazis Costas: Geotechnical Engineering: A student's Perspective IntroductionDr.Costas Sachpazis

Recently uploaded (20)

Steel Structures - Building technology.pptx

NO1 Certified Black Magic Specialist Expert Amil baba in Uae Dubai Abu Dhabi ...

Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsync

Introduction-To-Agricultural-Surveillance-Rover.pptx

Internet of things -Arshdeep Bahga .pptx

welding defects observed during the welding

Arduino_CSE ece ppt for working and principal of arduino.ppt

Software and Systems Engineering Standards: Verification and Validation of Sy...

Solving The Right Triangles PowerPoint 2.ppt

Risk Assessment For Installation of Drainage Pipes.pdf

Unit7-DC_Motors nkkjnsdkfnfcdfknfdgfggfg

🔝9953056974🔝!!-YOUNG call girls in Rajendra Nagar Escort rvice Shot 2000 nigh...

Virtual memory management in Operating System

Research Methodology for Engineering pdf

Energy Awareness training ppt for manufacturing process.pptx

Class 1 | NFPA 72 | Overview Fire Alarm System

Call Girls Narol 7397865700 Independent Call Girls

Vishratwadi & Ghorpadi Bridge Tender documents

Introduction to Machine Learning Unit-3 for II MECH

Sachpazis Costas: Geotechnical Engineering: A student's Perspective Introduction

Oozie at Yahoo

1. Oozie at Yahoo Purshotam Shah, Ryota Egashira Oozie Meetup 06/03 2014

2. Table of Contents Yahoo Confidential & Proprietary ▪Scale at Yahoo! ▪Scale and Performance ▪Features - Usability ▪High availability and Load Balancing ▪Customer Asks

3. Scale at Yahoo! ▪Busiest cluster › 1 million+ workflows per month › 45 - 55K workflows per day › 40 - 50K coord actions per day › 800 - 900 coordinators (5m, 15m, 30m, hourly, daily and weekly) › 30 - 40 bundles ▪Most complex bundle - 230 coordinators ▪Most complex workflow - 85 forks ▪Video Transcoding - 100-300 workflows per min

4. Scale and Performance ▪ Database › CLOB to BLOB to compress and store inline › Remove unnecessary hadoop config stored in protoActionConf › Select only needed columns instead of loading whole row › Partition tables by created time (in Oracle) ▪ Other › Huge improvements to materialization of coordinator actions › Reduce Launcher overhead • Merge the number of small files created per action to one sequence file • Launcher libraries shipped only once to HDFS • Uber mode launcher with Hadoop 2.x › Synchronously execute commands without queueing to speed up action transition › Automatically killing abandoned coordinator job › gzip compression for Rest API

5. Features - Usability ▪UI improvements › Active Jobs, Custom Global Filters, Child Jobs for Pig/Hive actions ▪Faster log streaming with more filters ▪Updating coordinator definition on the fly ▪Rerun workflows without having to specify all properties again ▪Mark coordinator and actions as ignored ▪Sharelib Enhancements › Update on the fly without failing jobs › Command to list different sharelib available › Specify directories using metafile instead of single share lib directory

6. High Availability and Load Balancing ▪HCat integration ▪SLA ▪Sharelib ▪Server-server authentication ▪Distributed sequence

7. Customer Asks ▪Coordinator dependency management › Ability to view dependencies and rerun part of a pipeline ▪Better error handling and automatic retries ▪Ability to Suspend/Turn off SLA alerting ▪One-click launcher log viewing ▪Zero downtime

8. Rohini Palaniswamy Mona Chitnis Michelle Chiang Purshotam Shah Ryota Egashira Olga L. Natkovich Yahoo Oozie Team

Oozie at Yahoo

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Oozie at Yahoo

Similar to Oozie at Yahoo (20)

Recently uploaded

Recently uploaded (20)

Oozie at Yahoo