Skip to content
Back
Theme
Software Engineering at Google
9 min read
Original article ↗
Foreword
Preface
Programming Over Time
Google’s Perspective
What This Book Isn’t
Parting Remarks
Conventions Used in This Book
O'Reilly Online Learning
How to Contact Us
Acknowledgments
Thesis
What Is Software Engineering?
Time and Change
Hyrum’s Law
Example: Hash Ordering
Why Not Just Aim for “Nothing Changes”?
Scale and Efficiency
Policies That Don’t Scale
Policies That Scale Well
Example: Compiler Upgrade
Shifting Left
Trade-offs and Costs
Example: Markers
Inputs to Decision Making
Example: Distributed Builds
Example: Deciding Between Time and Scale
Revisiting Decisions, Making Mistakes
Software Engineering Versus Programming
Conclusion
TL;DRs
Culture
How to Work Well on Teams
Help Me Hide My Code
The Genius Myth
Hiding Considered Harmful
Early Detection
The Bus Factor
Pace of Progress
In Short, Don’t Hide
It’s All About the Team
The Three Pillars of Social Interaction
Why Do These Pillars Matter?
Humility, Respect, and Trust in Practice
Blameless Post-Mortem Culture
Being Googley
Conclusion
TL;DRs
Knowledge Sharing
Challenges to Learning
Philosophy
Setting the Stage: Psychological Safety
Mentorship
Psychological Safety in Large Groups
Growing Your Knowledge
Ask Questions
Understand Context
Scaling Your Questions: Ask the Community
Group Chats
Mailing Lists
YAQS: Question-and-Answer Platform
Scaling Your Knowledge: You Always Have Something to Teach
Office Hours
Tech Talks and Classes
Documentation
Code
Scaling Your Organization’s Knowledge
Cultivating a Knowledge-Sharing Culture
Establishing Canonical Sources of Information
Staying in the Loop
Readability: Standardized Mentorship Through Code Review
What Is the Readability Process?
Why Have This Process?
Conclusion
TL;DRs
Engineering for Equity
Bias Is the Default
Understanding the Need for Diversity
Building Multicultural Capacity
Making Diversity Actionable
Reject Singular Approaches
Challenge Established Processes
Values Versus Outcomes
Stay Curious, Push Forward
Conclusion
TL;DRs
How to Lead a Team
Managers and Tech Leads (and Both)
The Engineering Manager
The Tech Lead
The Tech Lead Manager
Moving from an Individual Contributor Role to a Leadership Role
The Only Thing to Fear Is…Well, Everything
Servant Leadership
The Engineering Manager
Manager Is a Four-Letter Word
Today’s Engineering Manager
Antipatterns
Antipattern: Hire Pushovers
Antipattern: Ignore Low Performers
Antipattern: Ignore Human Issues
Antipattern: Be Everyone’s Friend
Antipattern: Compromise the Hiring Bar
Antipattern: Treat Your Team Like Children
Positive Patterns
Lose the Ego
Be a Zen Master
Be a Catalyst
Remove Roadblocks
Be a Teacher and a Mentor
Set Clear Goals
Be Honest
Track Happiness
The Unexpected Question
Other Tips and Tricks
People Are Like Plants
Intrinsic Versus Extrinsic Motivation
Conclusion
TL;DRs
Leading at Scale
Always Be Deciding
The Parable of the Airplane
Identify the Blinders
Identify the Key Trade-Offs
Decide, Then Iterate
Always Be Leaving
Your Mission: Build a “Self-Driving” Team
Dividing the Problem Space
Always Be Scaling
The Cycle of Success
Important Versus Urgent
Learn to Drop Balls
Protecting Your Energy
Conclusion
TL;DRs
Measuring Engineering Productivity
Why Should We Measure Engineering Productivity?
Triage: Is It Even Worth Measuring?
Selecting Meaningful Metrics with Goals and Signals
Goals
Signals
Metrics
Using Data to Validate Metrics
Taking Action and Tracking Results
Conclusion
TL;DRs
Processes
Style Guides and Rules
Why Have Rules?
Creating the Rules
Guiding Principles
The Style Guide
Changing the Rules
The Process
The Style Arbiters
Exceptions
Guidance
Applying the Rules
Error Checkers
Code Formatters
Conclusion
TL;DRs
Code Review
Code Review Flow
How Code Review Works at Google
Code Review Benefits
Code Correctness
Comprehension of Code
Code Consistency
Psychological and Cultural Benefits
Knowledge Sharing
Code Review Best Practices
Be Polite and Professional
Write Small Changes
Write Good Change Descriptions
Keep Reviewers to a Minimum
Automate Where Possible
Types of Code Reviews
Greenfield Code Reviews
Behavioral Changes, Improvements, and Optimizations
Bug Fixes and Rollbacks
Refactorings and Large-Scale Changes
Conclusion
TL;DRs
Documentation
What Qualifies as Documentation?
Why Is Documentation Needed?
Documentation Is Like Code
Know Your Audience
Types of Audiences
Documentation Types
Reference Documentation
Design Docs
Tutorials
Conceptual Documentation
Landing Pages
Documentation Reviews
Documentation Philosophy
WHO, WHAT, WHEN, WHERE, and WHY
The Beginning, Middle, and End
The Parameters of Good Documentation
Deprecating Documents
When Do You Need Technical Writers?
Conclusion
TL;DRs
Testing Overview
Why Do We Write Tests?
The Story of Google Web Server
Testing at the Speed of Modern Development
Write, Run, React
Benefits of Testing Code
Designing a Test Suite
Test Size
Test Scope
The Beyoncé Rule
A Note on Code Coverage
Testing at Google Scale
The Pitfalls of a Large Test Suite
History of Testing at Google
Orientation Classes
Test Certified
Testing on the Toilet
Testing Culture Today
The Limits of Automated Testing
Conclusion
TL;DRs
Unit Testing
The Importance of Maintainability
Preventing Brittle Tests
Strive for Unchanging Tests
Test via Public APIs
Test State, Not Interactions
Writing Clear Tests
Make Your Tests Complete and Concise
Test Behaviors, Not Methods
Don’t Put Logic in Tests
Write Clear Failure Messages
Tests and Code Sharing: DAMP, Not DRY
Shared Values
Shared Setup
Shared Helpers and Validation
Defining Test Infrastructure
Conclusion
TL;DRs
Test Doubles
The Impact of Test Doubles on Software Development
Test Doubles at Google
Basic Concepts
An Example Test Double
Seams
Mocking Frameworks
Techniques for Using Test Doubles
Faking
Stubbing
Interaction Testing
Real Implementations
Prefer Realism Over Isolation
How to Decide When to Use a Real Implementation
Faking
Why Are Fakes Important?
When Should Fakes Be Written?
The Fidelity of Fakes
Fakes Should Be Tested
What to Do If a Fake Is Not Available
Stubbing
The Dangers of Overusing Stubbing
When Is Stubbing Appropriate?
Interaction Testing
Prefer State Testing Over Interaction Testing
When Is Interaction Testing Appropriate?
Best Practices for Interaction Testing
Conclusion
TL;DRs
Larger Testing
What Are Larger Tests?
Fidelity
Common Gaps in Unit Tests
Why Not Have Larger Tests?
Larger Tests at Google
Larger Tests and Time
Larger Tests at Google Scale
Structure of a Large Test
The System Under Test
Test Data
Verification
Types of Larger Tests
Functional Testing of One or More Interacting Binaries
Browser and Device Testing
Performance, Load, and Stress testing
Deployment Configuration Testing
Exploratory Testing
A/B Diff Regression Testing
UAT
Probers and Canary Analysis
Disaster Recovery and Chaos Engineering
User Evaluation
Large Tests and the Developer Workflow
Authoring Large Tests
Running Large Tests
Owning Large Tests
Conclusion
TL;DRs
Deprecation
Why Deprecate?
Why Is Deprecation So Hard?
Deprecation During Design
Types of Deprecation
Advisory Deprecation
Compulsory Deprecation
Deprecation Warnings
Managing the Deprecation Process
Process Owners
Milestones
Deprecation Tooling
Conclusion
TL;DRs
Tools
Version Control and Branch Management
What Is Version Control?
Why Is Version Control Important?
Centralized VCS Versus Distributed VCS
Source of Truth
Version Control Versus Dependency Management
Branch Management
Work in Progress Is Akin to a Branch
Dev Branches
Release Branches
Version Control at Google
One Version
Scenario: Multiple Available Versions
The “One-Version” Rule
(Nearly) No Long-Lived Branches
What About Release Branches?
Monorepos
Future of Version Control
Conclusion
TL;DRs
Code Search
The Code Search UI
How Do Googlers Use Code Search?
Where?
What?
How?
Why?
Who and When?
Why a Separate Web Tool?
Scale
Zero Setup Global Code View
Specialization
Integration with Other Developer Tools
API Exposure
Impact of Scale on Design
Search Query Latency
Index Latency
Google’s Implementation
Search Index
Ranking
Selected Trade-Offs
Completeness: Repository at Head
Completeness: All Versus Most-Relevant Results
Completeness: Head Versus Branches Versus All History Versus Workspaces
Expressiveness: Token Versus Substring Versus Regex
Conclusion
TL;DRs
Build Systems and Build Philosophy
Purpose of a Build System
What Happens Without a Build System?
But All I Need Is a Compiler!
Shell Scripts to the Rescue?
Modern Build Systems
It’s All About Dependencies
Task-Based Build Systems
Artifact-Based Build Systems
Distributed Builds
Time, Scale, Trade-Offs
Dealing with Modules and Dependencies
Using Fine-Grained Modules and the 1:1:1 Rule
Minimizing Module Visibility
Managing Dependencies
Conclusion
TL;DRs
Critique: Google’s Code Review Tool
Code Review Tooling Principles
Code Review Flow
Notifications
Stage 1: Create a Change
Diffing
Analysis Results
Tight Tool Integration
Stage 2: Request Review
Stages 3 and 4: Understanding and Commenting on a Change
Commenting
Understanding the State of a Change
Stage 5: Change Approvals (Scoring a Change)
Stage 6: Committing a Change
After Commit: Tracking History
Conclusion
TL;DRs
Static Analysis
Characteristics of Effective Static Analysis
Scalability
Usability
Key Lessons in Making Static Analysis Work
Focus on Developer Happiness
Make Static Analysis a Part of the Core Developer Workflow
Empower Users to Contribute
Tricorder: Google’s Static Analysis Platform
Integrated Tools
Integrated Feedback Channels
Suggested Fixes
Per-Project Customization
Presubmits
Compiler Integration
Analysis While Editing and Browsing Code
Conclusion
TL;DRs
Dependency Management
Why Is Dependency Management So Difficult?
Conflicting Requirements and Diamond Dependencies
Importing Dependencies
Compatibility Promises
Considerations When Importing
How Google Handles Importing Dependencies
Dependency Management, In Theory
Nothing Changes (aka The Static Dependency Model)
Semantic Versioning
Bundled Distribution Models
Live at Head
The Limitations of SemVer
SemVer Might Overconstrain
SemVer Might Overpromise
Motivations
Minimum Version Selection
So, Does SemVer Work?
Dependency Management with Infinite Resources
Exporting Dependencies
Conclusion
TL;DRs
Large-Scale Changes
What Is a Large-Scale Change?
Who Deals with LSCs?
Barriers to Atomic Changes
Technical Limitations
Merge Conflicts
No Haunted Graveyards
Heterogeneity
Testing
Code Review
LSC Infrastructure
Policies and Culture
Codebase Insight
Change Management
Testing
Language Support
The LSC Process
Authorization
Change Creation
Sharding and Submitting
Cleanup
Conclusion
TL;DRs
Continuous Integration
CI Concepts
Fast Feedback Loops
Automation
Continuous Testing
CI Challenges
Hermetic Testing
CI at Google
CI Case Study: Google Takeout
But I Can’t Afford CI
Conclusion
TL;DRs
Continuous Delivery
Idioms of Continuous Delivery at Google
Velocity Is a Team Sport: How to Break Up a Deployment into Manageable Pieces
Evaluating Changes in Isolation: Flag-Guarding Features
Striving for Agility: Setting Up a Release Train
No Binary Is Perfect
Meet Your Release Deadline
Quality and User-Focus: Ship Only What Gets Used
Shifting Left: Making Data-Driven Decisions Earlier
Changing Team Culture: Building Discipline into Deployment
Conclusion
TL;DRs
Compute as a Service
Taming the Compute Environment
Automation of Toil
Containerization and Multitenancy
Summary
Writing Software for Managed Compute
Architecting for Failure
Batch Versus Serving
Managing State
Connecting to a Service
One-Off Code
CaaS Over Time and Scale
Containers as an Abstraction
One Service to Rule Them All
Submitted Configuration
Choosing a Compute Service
Centralization Versus Customization
Level of Abstraction: Serverless
Public Versus Private
Conclusion
TL;DRs
Conclusion
Afterword
Index