More Related Content
MongoUK - Approaching 1 billion documents with MongoDB1 Billion Documents
Zero to 1 Billion+ Records: A True Story of Learning & Scaling GameChanger
Lessons Learned Migrating 2+ Billion Documents at Craigslist
Tarantool: как сэкономить миллион долларов на базе данных на высоконагруженно...
Fusion-io and MySQL at Craigslist
Understanding and tuning WiredTiger, the new high performance database engine...
Scylla Summit 2018: Rebuilding the Ceph Distributed Storage Solution with Sea...
What's hot
GlusterFS As an Object Storage
Ceph BlueStore - новый тип хранилища в Ceph / Максим Воронцов, (Redsys)
MySQL And Search At Craigslist
Setting up mongodb sharded cluster in 30 minutes
Linux Kernel Extension for Databases / Александр Крижановский (Tempesta Techn...
MongoDB Memory Management Demystified
Making the case for write-optimized database algorithms / Mark Callaghan (Fac...
[db tech showcase Tokyo 2017] A11: SQLite - The most used yet least appreciat...
Avoiding Data Hotspots at Scale
Update on Crimson - the Seastarized Ceph - Seastar Summit
Realtime Search Infrastructure at Craigslist (OpenWest 2014)
Performance tuning in BlueStore & RocksDB - Li Xiaoyan
Evaluation of RBD replication options @CERN
Sphinx at Craigslist in 2012
RADOS improvements and roadmap - Greg Farnum, Josh Durgin, Kefu Chai
Redis persistence in practice
Viewers also liked
You know, for search. Querying 24 Billion Documents in 900ms
Living with SQL and NoSQL at craigslist, a Pragmatic Approach
MongoDB 3.0 and WiredTiger (Event: An Evening with MongoDB Dallas 3/10/15)
Midas - on-the-fly schema migration tool for MongoDB.
Probabilistic algorithms for fun and pseudorandom profit
MongoDB for Time Series Data
MongoDB for Time Series Data Part 1: Setting the Stage for Sensor Management
Scaling massive elastic search clusters - Rafał Kuć - Sematext
Similar to Webinar - Approaching 1 billion documents with MongoDB
MongoDB Best Practices in AWS
Deployment Strategies (Mongo Austin)
Evaluating NoSQL Performance: Time for Benchmarking
MongoDB and AWS Best Practices
Keeping MongoDB Data Safe
Optimizing MongoDB: Lessons Learned at Localytics
MongoDB: Advantages of an Open Source NoSQL Database
KVSの性能、RDBMSのインデックス、更にMapReduceを併せ持つAll-in-One NoSQL: MongoDB
MongoDB and server performance
Andy Parsons Pivotal June 2011
More from Boxed Ice
MongoDB Tokyo - Monitoring and Queueing
MongoUK 2011 - Rplacing RabbitMQ with MongoDB
MongoDB - Monitoring and queueing
MongoDB - Monitoring & queueing
Monitoring MongoDB (MongoUK)
Monitoring MongoDB (MongoSV)
MongoUK - PHP Development
MongoUK - PHP Development
Recently uploaded
Core Components of Internet of Things (IoT)
From 48% to 99.9% Accuracy: Compliance Turnaround Case Study.pdg
Zuri Ozeomachukwu Njoku Explains 6 Consensus Algorithms That Power Blockchain...
The Next10 Exponential Innovation 2026 TH
AI Infrastructure and the Compute Gap - Matt Dratch
Corporate AI Training to AI Enable a Company Workforce
1. Understanding Security Goals .pdf
Advent of Cyber 2025 TryHackMe Certificate
Cyber_Pitch_Presentationggggggg (2).pptx
RaaS™ — Research as a Service | Sovereign-Grade Energy & Infrastructure
Zac Brown - A Cybersecurity Professional
Unit 1.2 Components of a Computer System.pdf
AI Ethics & Cybersecurity: Building Enterprise Trust
Wavetel IOT Product catalog _ v1.3. pdf
Compare and contrast types of attacks.pptx
RaaS™ — Research as a Service | Sovereign-Grade Energy & Infrastructure
Oxford Nanopore Sequening(ONT) and its applications.pdf
The Road to Superintelligence - Mayur Pathak
Artificial Intelligence and Barbarism - Conceptual Map
Top Websites To ⭐Buy ⭐Old ⭐Gmail ⭐Accounts (PVA & Bulk) (5).docx
Webinar - Approaching 1 billion documents with MongoDB
- 1.
Approaching 1 Billion Documents in MongoDB David Mytton 1/25 david@boxedice.com / www.mytton.net
- 2.
- 3.
db.stats() Documents 981,289,332 Collections 47,962 Indexes 39,684 Data size 369GB Index size 241GB 3/25 As of 25th Apr 2010.
- 4.
- 5.
Initial Setup Replication Master Slave DC1 DC2 8GB RAM 8GB RAM 5/25
- 6.
Vertical Scaling Replication Master Slave DC1 DC2 72GB RAM 8GB RAM 6/25
- 7.
Tip #1 Keep your indexes in memory at all times. db.stats() 7/25
- 8.
Manual Partitioning Replication Master A Slave A DC1 DC2 16GB RAM 16GB RAM Replication Master B Slave B DC1 DC2 8/25 16GB RAM 16GB RAM
- 9.
Database vs collections • Many databases = many data files (small but quickly get large). • Many collections = watch namespace limit. 9/25
- 10.
- 11.
Tip #2 Monitor the 24,000 namespace limit. 11/25
- 12.
- 13.
Console db.system.namespaces.count() 13/25
- 14.
Replica Pairs =Failover Replica Pair Master A Slave A DC1 DC2 16GB RAM 16GB RAM Replica Pair Master B Slave B DC1 DC2 14/25 16GB RAM 16GB RAM
- 15.
Tip #3 Pre-provision your oplog files. 15/25
- 16.
A shell scriptto generate 75GB oplog files for i in {0..40} do echo $i head -c 2146435072 /dev/zero > local.$i done 16/25
- 17.
Tip #4 Expect slower performance during initial replica sync. 17/25
- 18.
Tip #5 You can rotate your log files from the console. 18/25
- 19.
- 20.
Tip #6 Index creation blocks by default. Use background indexing if necessary. 20/25 MongoDB Manual: http://bit.ly/mongobgindex
- 21.
Tip #7 Increase your OS file descriptor limit + use persistent connections. 21/25
- 22.
Too many openfiles! /etc/security/limits.conf mongo hard nofile 10000 mongo soft nofile 10000 user type limit /etc/ssh/sshd_config UsePAM yes 22/25
- 23.
- 24.
Tip #8 10gen commercial support is worth paying for. 24/25
- 25.
Summary 1. Keep indexes in memory. 2. Monitor the 24k namespace limit. 3. Pre-provision oplog files. 4. Expect slower performance on replica sync. 5. Rotate logs from the console. 6. Index creation blocks by default. 7. OS file descriptor limit + persistent connections. 25/25 8. Commercial support is worth it.