Scaling HBase (nosql store) to handle massive loads at Pinterest by Jeremy Carol

•

2 likes•2,168 views

How Pinterest uses HBase at massive scale. This presentation will also focus on how Pinterest runs HBase on Amazon EC2 cloud and also how they monitor, trouble shoot and scale the system.

Technology

HBase

Operations on EC2

Jeremy Carroll
Big Data Gurus

Pinterest Engineering

Overview

• Deployment Strategies for EC2
• Validating Design
• Production Support

Pinterest Engineering

Lets Deploy

• First Question Asked
• Rack Locality?
• Cloud Concepts

Pinterest Engineering

High Availability

Pinterest Engineering

Logical Separation

Pinterest Engineering

Does This Work?

• Schema Design
• Hot Spots
• Load Testing
• Tools

Pinterest Engineering

Production

• Monitoring
• Alerting
• Health

Pinterest Engineering

Snapshots & DNS
HBASE-8473
17:10 <jeremy_carroll> jmhsieh: I think I found the root cuase. All my region servers reach the barrier, but it
does not continue.
17:11 <jeremy_carroll> jmhsieh: All RS have this in their logs:
DEBUG org.apache.hadoop.hbase.procedure.Subprocedure: Subprocedure 'backup1' coordinator notified of 'acquire',
waiting on 'reached' or 'abort' from coordinator.
17:11 <jeremy_carroll> jmhsieh: Then the coordinator (Master) never sends anything. They just sit until the
timeout.
17:12 <jeremy_carroll> jmhsieh: So basically 'reached' is never obtained. Then abort it set, and it fails.
...
17:24 <jeremy_carroll> jmhsieh: Found the bug. The hostnames dont match the master due to DNS resolution
17:25 <jeremy_carroll> jmhsieh: The barrier aquired is putting in the local hostname
from the regionservers. In EC2 (Where reverse DNS does not work well), the master hands the internal name to the
client.
17:26 <jeremy_carroll> jmhsieh: So it's waiting for something like 'ip-10-155-208-202.ec2.internal,
60020,1367366580066'
zNode to show up, but instead 'hbasemetaclustera-d1b0a484,60020,1367366580066,' is being inserted. Barrier is not
reached
17:27 <jeremy_carroll> jmhsieh: Reason being in our environment the master does not
have a reverse DNS entry. So we get stuff like this on RegionServer startup in our logs.
17:27 <jeremy_carroll> jmhsieh: 2013-05-01 00:03:00,614 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
Master passed us hostname to use. Was=hbasemetaclustera-d1b0a484, Now=ip-10-155-208-202.ec2.internal
17:54 <jeremy_carroll> jmhsieh: That was it. Verified. Now that Reverse DNS is working,
snapshots are working. Now how to figure out how to get Reverse DNS working on Route53. I
wished there was something like 'slave.host.name' inside of Hadoop for this. Looking at source code.

Pinterest Engineering

What's hot

This presentation was prepared for a Webcast where John Yerhot, Engine Yard US Support Lead, and Chris Kelly, Technical Evangelist at New Relic discussed how you can scale and improve the performance of your Ruby web apps. They shared detailed guidance on issues like: Caching strategies Slow database queries Background processing Profiling Ruby applications Picking the right Ruby web server Sharding data Attendees will learn how to: Gain visibility on site performance Improve scalability and uptime Find and fix key bottlenecks See the on-demand replay: http://pages.engineyard.com/6TipsforImprovingRubyApplicationPerformance.html

6 tips for improving ruby performance

Engine Yard

When it comes to caching, there are two types of web developers - those with phat stacks of cache money and those suffering from cache anxiety. Caching is particularly handy when scaling Rails apps, however we often avoid putting in effort because it can quickly get complicated without effective strategies. Rails provides a host of built-in caching interfaces that are easy to leverage and extend. I’ll talk about how to do this and combine rails with technologies like CDNs and HTTP accelerators like Varnish so that you can more effectively cache everything, everywhere without fear of serving stale content. Michael May is an API Engineer at Fastly and a former Austinite, now hailing from San Francisco. While in Texas he studied at UT Austin and co-founded CDN Sumo, which was acquired by Fastly. He’s waiting for the day when FaaS (Franklin BBQ as a Service) becomes a thing and dreams about fast websites.

Rails Caching Secrets from the Edge

Michael May

WordPress + NGINX Best Practices with EasyEngine

NGINX, Inc.

Google is making the power of its datacenter, network, and technology innovations available to the world through its Cloud services. This presentation will provide an overview of the Google Cloud Platform and a deeper dive on Google Compute Engine. Google recently made an open source contribution to SaltStack and now you can now use Salt Cloud to manage your Compute Engine resources (IaaS virtual machine services). Come find out more about Google's Cloud Platform and how you can leverage Google scale with SaltStack.

SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...

SaltStack

Boost your website by running PHP on Nginx

Harald Zeitlhofer

Running PHP on Nginx

Harald Zeitlhofer

Ansible with AWS

Allan Denot

Keep Them out of the Database

Martin Berger

Cold fusion is racecar fast

ColdFusionConference

Docstoc.com (founded in 2007, acquired by Intuit in 2013) is one of the largest online repositories of documents. A critical component of our product is our text file service, which delivers text documents to both humans and crawlers. In early 2013 this service, which was file system based, became a prohibitive bottleneck. To meet our scaling needs, we replaced it with one backed by a sharded MongoDB cluster. This talk will cover: Our traffic load (5:1 bots:humans ratio) How we implemented the system in our SOA environment How MongoDB fit our use case out of the box How we load tested peak time traffic before hardware purchase How we loaded the system and how we rolled it out live Performance metrics and gains in stability and reliability

Scalable Text File Service with MongoDB (Intuit)

MongoDB

Caching the Uncacheable: Leveraging Your CDN to Cache Dynamic Content

Fastly

Deploy Rails Application by Capistrano

Tasawr Interactive

Improve ColdFusion Performance by tuning the Connector and using ColdFusion-T...

ColdFusionConference

cPanel - Apache Global Configuration

skuver

Introduction to Chef

Suresh Paulraj

Oh no! My backups aren't progressing! If something happens in production now, and I don't have current backups, I'll be out of a job for sure! If these words resonate with you, don’t worry; you’re not the only one! Backup issues are one of the most common topics we deal with in Technical Services. In this talk, we will go through the backup flow, talk about where things might go wrong, and the symptoms you will see in the logs and the UI. We will also talk about other commands you can run to confirm the diagnosis, and how support can assist if you’re still stuck. Finally, we will talk about the new backup architecture in 4.2 and how it simplifies some of these concerns. This session is suitable for those with all levels of Ops Manager experience, but attendees should have a basic understanding of MongoDB’s replication process before attending this session. After this talk, you will have leveled up your backup superpowers, and can swoop in to save your job (and the day)!

MongoDB .local Bengaluru 2019: Becoming an Ops Manager Backup Superhero!

MongoDB

Stupid Boot Tricks: using ipxe and chef to get to boot management bliss

macslide

Magento is a leading open source, eCommerce platform used by many global brands. However, architecting your Magento platform to grow with your business can sometimes be a challenge. This session walks through the steps needed to take an out-of-the-box, single-node Magento implementation and turn it into a highly available, elastic, and robust deployment. This includes an end-to-end caching strategy that provides an efficient front-end cache (including populated shopping carts) using Varnish on Amazon EC2 as well as offloading the Magento caches to separate infrastructure such as Amazon ElastiCache. We also look at strategies to manage the Magento Media library outside of the application instances, including EC2-based shared storage solutions and Amazon S3. At the data layer we look at Magento-specific Amazon RDSandndash;tuning strategies including configuring Magento to use read replicas for horizontal scalability. Finally, we look at proven techniques to manage your Magento implementation at scale, including tips on cache draining, appropriate cache separation, and utilizing AWS CloudFormation to manage your infrastructure and orchestrate predictable deployments.

(WEB304) Running and Scaling Magento on AWS | AWS re:Invent 2014

Amazon Web Services

Asynchronous Processing with Ruby on Rails (RailsConf 2008)

Jonathan Dahl

DATABASE AUTOMATION with Thousands of database, monitoring and backup

Saewoong Lee

What's hot (20)

6 tips for improving ruby performance

Rails Caching Secrets from the Edge

WordPress + NGINX Best Practices with EasyEngine

SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...

Boost your website by running PHP on Nginx

Running PHP on Nginx

Ansible with AWS

Keep Them out of the Database

Cold fusion is racecar fast

Scalable Text File Service with MongoDB (Intuit)

Caching the Uncacheable: Leveraging Your CDN to Cache Dynamic Content

Deploy Rails Application by Capistrano

Improve ColdFusion Performance by tuning the Connector and using ColdFusion-T...

cPanel - Apache Global Configuration

Introduction to Chef

MongoDB .local Bengaluru 2019: Becoming an Ops Manager Backup Superhero!

Stupid Boot Tricks: using ipxe and chef to get to boot management bliss

(WEB304) Running and Scaling Magento on AWS | AWS re:Invent 2014

Asynchronous Processing with Ruby on Rails (RailsConf 2008)

DATABASE AUTOMATION with Thousands of database, monitoring and backup

Viewers also liked

Implementing a Population Health Model (Hon Pak)

U.S. News Healthcare of Tomorrow

Big data: current technology scope.

Roman Nikitchenko

HBase Low Latency

DataWorks Summit

Core concepts and Key technologies - Big Data Analytics

Kaniska Mandal

Hadoop hbase mapreduce

FARUK BERKSÖZ

Pulsar: Real-time Analytics at Scale with Kafka, Kylin and Druid

Tony Ng

Viewers also liked (6)

Implementing a Population Health Model (Hon Pak)

Big data: current technology scope.

HBase Low Latency

Core concepts and Key technologies - Big Data Analytics

Hadoop hbase mapreduce

Pulsar: Real-time Analytics at Scale with Kafka, Kylin and Druid

Similar to Scaling HBase (nosql store) to handle massive loads at Pinterest by Jeremy Carol

Scaling HBase at Pinterest

bigdatagurus_meetup

HBase has been in production in hundreds of clusters across the HDP customer base. In this talk, we will go over the best practices and lessons learned in supporting these clusters over the years. We will cover top 10 recurring issues like ZooKeeper, GC, number of regions, HBCK, operating system and coprocessor related issues and more across 1000+ support tickets. We will also cover common solutions and lessons learned and go into details of how to tune, monitor and operate your clusters. We will then cover some of the improvements that we have been adding to HBase and Ambari for easing up some of the pain.

Operating and supporting HBase Clusters

enissoz

Operating and Supporting Apache HBase Best Practices and Improvements

DataWorks Summit/Hadoop Summit

DNS_Tutorial 2.pptx

viditsir

2017 DNSSEC KSK Rollover

APNIC

dns-sec-4-slides

kj teoh

Pl2017 High Availability in GCE

Allan Mason

High Availability in GCE

Carmen Mason

Supporting Apache HBase : Troubleshooting and Supportability Improvements

DataWorks Summit

HBase has been in production in hundreds of clusters across the CDH/HDP customer base and Cloudera/Hortonworks support it for many years. In this talk, based on our support experience, we aim to introduce useful information to troubleshoot HBase clusters efficiently. First off, we (Daisuke at Cloudera support) are going to talk about typical log messages and web UI info which we can use for troubleshooting (especially for struggling with performance issues). Since their meanings have been changing over the past versions, we would like to show the difference and improvements as well (e.g. HBASE-20232 for memstore flush, HBASE-16972 for slow scanner, HBASE-18469 for request counter, and also HBASE-21207 for sorting in web UI). We (Toshihiro at Cloudera, a former Hortonworks employee) will also cover some new tools (e.g. HBASE-21926 Profiler Servlet, HBASE-11062 htop, etc.), which should also be useful for performance troubleshooting.

Supporting Apache HBase : Troubleshooting and Supportability Improvements

DataWorks Summit

Your customers expect you to continuously deliver delightful experiences. This means that you’ll need to continuously deliver application and infrastructure updates. Hand-crafted servers lovingly built and maintained by a system administrator are a thing of the past. Golden images are fine for initial provisioning but will quickly fail as your configuration requirements change over time. It’s time for you to fully automate the provisioning and management of your infrastructure components. Welcome to the world of infrastructure as code! In this new world, you’ll be able to programmatically provision and configure the components of your infrastructure. Disposable infrastructure whose provisioning, configuration, and on-going maintenance is fully automated allow you to change the way you build and deliver applications. Move your applications and infrastructure towards continuous delivery. In this talk, we’ll explore the ideas behind “infrastructure as code” and, specifically, look at how Chef allows you to fully automate your infrastructure. If you’re brave enough, we’ll even let you get your hands on some Chef and experience the delight of using Chef to build and deploy some infrastructure components.

Introduction to Infrastructure as Code & Automation / Introduction to Chef

Nathen Harvey

React Server Side Rendering with Next.js

Jamie Barton 👨🏻‍💻

MCITP

Naqib Khan

HBaseConAsia2018 Track2-3: Bringing MySQL Compatibility to HBase using Databa...

Michael Stack

Introduction to Infrastructure as Code & Automation / Introduction to Chef

All Things Open

Whether you’re a longtime container user, or entirely new to Docker, it’s never too late to pick up more tips and tricks for running containers in production. In the February 2017 online Rancher meetup, we covered four topics useful for anyone managing containers and infrastructure at scale. - Autoscaling Docker containers and infrastructure with webhooks in Rancher - Secrets Management with Rancher and Vault - Getting the most out of containers and the Traefik load balancer - Launching applications with Kubernetes Dashboard and Helm You can find a recording of the meetup at: http://rancher.com/event/february-2017-online-meetup/

More tips and tricks for running containers like a pro - Rancher Online MEetu...

Shannon Williams

HBase tales from the trenches

wchevreuil

Intro ProxySQL

I Goo Lee

Pmw2 k3ni 1-3a

hariclant1

The New Root Zone DNSSEC KSK

APNIC

Similar to Scaling HBase (nosql store) to handle massive loads at Pinterest by Jeremy Carol (20)

Scaling HBase at Pinterest

Operating and supporting HBase Clusters

Operating and Supporting Apache HBase Best Practices and Improvements

DNS_Tutorial 2.pptx

2017 DNSSEC KSK Rollover

dns-sec-4-slides

Pl2017 High Availability in GCE

High Availability in GCE

Supporting Apache HBase : Troubleshooting and Supportability Improvements

Introduction to Infrastructure as Code & Automation / Introduction to Chef

React Server Side Rendering with Next.js

MCITP

HBaseConAsia2018 Track2-3: Bringing MySQL Compatibility to HBase using Databa...

Introduction to Infrastructure as Code & Automation / Introduction to Chef

More tips and tricks for running containers like a pro - Rancher Online MEetu...

HBase tales from the trenches

Intro ProxySQL

Pmw2 k3ni 1-3a

The New Root Zone DNSSEC KSK

Recently uploaded

Advantages of Hiring UIUX Design Service Providers for Your Business

Pixlogix Infotech

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Scaling API-first – The story of a global engineering organization

Radu Cotescu

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Histor y of HAM Radio presentation slide

vu2urc

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Real Time Object Detection Using Open CV

Khem

Recently uploaded (20)

Advantages of Hiring UIUX Design Service Providers for Your Business

Apidays New York 2024 - The value of a flexible API Management solution for O...

A Domino Admins Adventures (Engage 2024)

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Scaling API-first – The story of a global engineering organization

Boost PC performance: How more available memory can improve productivity

AWS Community Day CPH - Three problems of Terraform

Strategies for Landing an Oracle DBA Job as a Fresher

Finology Group – Insurtech Innovation Award 2024

How to Troubleshoot Apps for the Modern Connected Worker

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Histor y of HAM Radio presentation slide

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Partners Life - Insurer Innovation Award 2024

Driving Behavioral Change for Information Management through Data-Driven Gree...

How to Troubleshoot Apps for the Modern Connected Worker

Real Time Object Detection Using Open CV

Scaling HBase (nosql store) to handle massive loads at Pinterest by Jeremy Carol

1. HBase Operations on EC2 Jeremy Carroll Big Data Gurus Pinterest Engineering

2. Overview • Deployment Strategies for EC2 • Validating Design • Production Support Pinterest Engineering

3. Powered by HBase Pinterest Engineering

4. Lets Deploy • First Question Asked • Rack Locality? • Cloud Concepts Pinterest Engineering

5. High Availability Pinterest Engineering

6. Logical Separation Pinterest Engineering

7. Cell Based Pinterest Engineering

8. Logical Separation Pinterest Engineering

9. Does This Work? • Schema Design • Hot Spots • Load Testing • Tools Pinterest Engineering

10. Does This Work? Pinterest Engineering

11. Compaction Pinterest Engineering

12. OpenTSDB Pinterest Engineering

13. Production • Monitoring • Alerting • Health Pinterest Engineering

14. Monitoring Pinterest Engineering

15. Baselines Pinterest Engineering

16. Visualization Pinterest Engineering

17. Problems Pinterest Engineering

18. Alerting Pinterest Engineering

19. Baselines Pinterest Engineering

20. Snapshots & DNS HBASE-8473 17:10 <jeremy_carroll> jmhsieh: I think I found the root cuase. All my region servers reach the barrier, but it does not continue. 17:11 <jeremy_carroll> jmhsieh: All RS have this in their logs: DEBUG org.apache.hadoop.hbase.procedure.Subprocedure: Subprocedure 'backup1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator. 17:11 <jeremy_carroll> jmhsieh: Then the coordinator (Master) never sends anything. They just sit until the timeout. 17:12 <jeremy_carroll> jmhsieh: So basically 'reached' is never obtained. Then abort it set, and it fails. ... 17:24 <jeremy_carroll> jmhsieh: Found the bug. The hostnames dont match the master due to DNS resolution 17:25 <jeremy_carroll> jmhsieh: The barrier aquired is putting in the local hostname from the regionservers. In EC2 (Where reverse DNS does not work well), the master hands the internal name to the client. 17:26 <jeremy_carroll> jmhsieh: So it's waiting for something like 'ip-10-155-208-202.ec2.internal, 60020,1367366580066' zNode to show up, but instead 'hbasemetaclustera-d1b0a484,60020,1367366580066,' is being inserted. Barrier is not reached 17:27 <jeremy_carroll> jmhsieh: Reason being in our environment the master does not have a reverse DNS entry. So we get stuff like this on RegionServer startup in our logs. 17:27 <jeremy_carroll> jmhsieh: 2013-05-01 00:03:00,614 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Master passed us hostname to use. Was=hbasemetaclustera-d1b0a484, Now=ip-10-155-208-202.ec2.internal 17:54 <jeremy_carroll> jmhsieh: That was it. Verified. Now that Reverse DNS is working, snapshots are working. Now how to figure out how to get Reverse DNS working on Route53. I wished there was something like 'slave.host.name' inside of Hadoop for this. Looking at source code. Pinterest Engineering

21. Thanks! Pinterest Engineering

Scaling HBase (nosql store) to handle massive loads at Pinterest by Jeremy Carol

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Scaling HBase (nosql store) to handle massive loads at Pinterest by Jeremy Carol

Similar to Scaling HBase (nosql store) to handle massive loads at Pinterest by Jeremy Carol (20)

More from Hakka Labs

More from Hakka Labs (20)

Recently uploaded

Recently uploaded (20)

Scaling HBase (nosql store) to handle massive loads at Pinterest by Jeremy Carol