Then why do you often need external readability reviewers, especially if you or your team don't contribute to a codebase frequently? Having humans judge other humans' style is a given; it is part of the PR process. But why do you _also_ need someone with a readability rating as opposed to a code owner?
Note: I'm not a Googler nor a Xoogler, but my partner worked at Google for some years and so I've heard her complain a lot about Google and readability.
> But why do you _also_ need someone with a readability rating as opposed to a code owner
FWIW, these can be the same person - if the have appropriate readabilities. The problem is that it's not easy to get readability in the first place, so occasionally teams have to look for people outside.
Right but the fact that these are two separate roles strikes me both as bureaucratic overhead and a fundamental lack of trust in individual engineers. If I were approached to enact something like readability, I would veto it immediately. We hire engineers because we trust them. Part of that trust means that engineers occasionally wade into codebases they don't understand in languages they don't know and need to make contributions, but we trust them to not make a mess. Do some of them make messes? Undoubtedly. Is the magnitude of the mess, multiplied by the number of engineers, worth the creation of an entire readability certification process? I don't think so.
> Is the magnitude of the mess, multiplied by the number of engineers, worth the creation of an entire readability certification process? I don't think so.
And that's an acceptable trade-off for a different sized company. Google arguably has the biggest centralised codebase in the world, and simply has different requirements.
As other have said, most of the overhead is from attaining readability, not the requirement itself.
> And that's an acceptable trade-off for a different sized company. Google arguably has the biggest centralised codebase in the world, and simply has different requirements.
AWS does not operate this way nor do many CDNs operate this way nor do ISPs operate this way. There's other high scale businesses out there. Google isn't the only one. Using Google's scale to justify business practices is a self-fulfilling prophecy.
> AWS does not operate this way nor do many CDNs operate this way nor do ISPs operate this way.
I'm not denying that, but all the same, we're not talking about thousands of separate codebases. It's a single monorepo codebase with consistent styling, the advantage being that people can switch teams or contribute to other Google software under the same style guide.
The overhead isn't a bug, it's a core feature as consistent style makes it that much easier to switch teams.
It's also worth noting that Borgmon readability included an awful lot of "how not to shoot yourself in the foot in strange and unexpected ways". That required someone who had shown they knew of those strange and unexpected ways to review your code.
And I take it google has done wide ranging scientific studies about the variations in coding styles and language constructs that it is a secret advantage that they know how to write "readable" code? Implying they tried a bunch of diffrent ways until settling on the one true way that allows a diverse set of people with diverse experiences to read it?
Ever heard of COBOL?
Because readability has always been in the eye of the beholder, and codifying it makes it even worse.
Oh boy.
So, to start, we have done scientific studies about the costs and benefits of readability (as codified circa 2018, since that was when the last study was).
It was studied for multiple programming languages Google uses.
It was done, at my request, by the engineering productivity research team, who are experts in this kind of research - you can find public papers they publish[1].
For background: I was the person responsible for production programming languages, and I did this precisely because i did not feel at the time there were recent good studies as to whether readability was really worth the cost.
The answer is "yes, it is".
There is an upfront cost, but readability has a meaningful and net positive effect on engineering velocity.
It is large enough to make the cost back quite quickly.
One could go down the rabbit hole of seeing whether you improve (or make worse) the numbers by changing various parts of the style guides, but like I said, what is there now, has in fact been studied scientifically.
Honestly, i'm not sure why you think it wouldn't be. You come off as a little immature when you sort of just assume people have no idea what they are doing and don't think these things through. You are talking about a company with 60k+ engineers. Every hour of all-engineer time you waste is equivalent to having ~30 SWE do nothing for a year.
Maybe there are companies happy to do that. I worked at IBM for a few years when i was much younger ;). All I can say is that as long as i'm involved in developer tools at Google, i'm gonna try not to waste my customers time.
[1] I say this in the hopes you don't go making assumptions about whether the studies were done properly. They were properly controlled for tenure, number of reviews, change size, etc.
I'd love to see the research here. Because searching the software engineering Google papers from 2021-2021, the only somewhat relevant papers I found were case studies (such as [1] and [2]), which were based on Googlers' perception of their own processes. While self-perception can indeed by used to measure efficacy, there's a reason why it's used with great caution in surveys. I'll be honest, I have a hard time believing in readability myself.
We have never published the readability research. I could look into doing so, but it is awfully Google specific. Trying to come up with something generally applicable would be a very different project
I'm not a fan of readability exactly the way Google does it, but I'm pretty happy that Google insists on various aspects of it, like good identifier names.
I don't know of any research off hand, but I'm pretty sure the industry consensus is that good identifier names improve the quality of the code (Go style notwithstanding.) Readability is one way to training engineers to do it.
> Because readability has always been in the eye of the beholder, and codifying it makes it even worse.
This is empirically false. Consistency, even if it is unfavorable to your preferences, is superior to inconsistency. So a codified set of best practices is better than none at all.
There are part of Google's style guides that I would change if I could, but I also prefer having a style guide (and one that goes beyond things that are lintable) than none at all, because consistency across the codebase means that I can usually understand code at a glance, or if not, know at a glance that something unusual is happening. (this is in fact precisely the argument in favor of autoformatters like gofmt/black/prettier, but extended to softer concepts that can't always be formatted: consistent style, even if it isn't your favorite, is superior to inconsistent style).
> Consistency, even if it is unfavorable to your preferences, is superior to inconsistency. So a codified set of best practices is better than none at all.
Indeed, but at what cost? Every step of added gatekeeping reduces development velocity, and Google does not have a monopoly on high scale services (cough AWS cough). In an ideal world for every full-time dev writing code, you'd have a full-time reviewer whose job it is to simply review that dev's code. But the real world has budgets and deadlines, and humans become discouraged when changes take too long to complete. So is readability worth the added gatekeeping cost is the question.
You're ignoring the cost of unrestrained development without standards. The impression I've gotten is that a AWS papers over their lack of development gatekeeping with unreasonable oncall loads. All things considered, the "cost" of readability in terms of development velocity is fairly low. If anyone on your team has readability [in the language], they can review your code. This can be problem if only one person has it, but usually once one person gets through the process, you can get more people through fairly quickly.
I'm a bit of a polyglot at Google (I have readability in C++, python, and Javascript/Typescript, and am part-way through the Go and Java processes, which covers basically every popular language at the company), and IMO, the readability process for each language has been a net positive, although it was painful for a while when I had no readability and none of my coworkers did (but there are procedures for getting around that today).
For example, I "donate" my readability, and review random changes from people who don't have readability but need a quick turnaround or aren't yet working to gain readability.
> You're ignoring the cost of unrestrained development without standards.
I'm not. I'm not saying that code should be merged without any stylistic commentary; I'm simply proposing that stylistic commentary be provided by other stakeholders in the service you're contributing to, on top of whatever linter or sanitizer is being used. That is, after all, the goal of a human reviewer. To add _another_ layer of gatekeeping is what I'm questioning.
> The impression I've gotten is that a AWS papers over their lack of development gatekeeping with unreasonable oncall loads.
I don't agree with this perception at all. AWS spends a lot of time building frameworks and libraries which attempt to minimize human gatekeeping to only the necessary areas. So think retries, backoff, timeouts, circuit-breakers, etc. Instead of adding gatekeeping at the code level (with things like code style), instead gatekeeping is added at the architecture and operations stages, which is where most of the work of keeping a production level service up, available, and fast. Operational level gatekeeping in the bounds of a given organization is a much more neatly constrained problem than stylistic gatekeeping and this gatekeeping happens on a per-service basis rather than a per-CL/review basis. This keeps overhead a lot lower.
My impression from being outside of Google (and having a Xoogler for a partner) is that Google loves bureaucracy and process, and that engineers that continue to stay at Google are fine with navigating these processes. Its answer to problems is to add more bureaucracy, another item for the checklist that a person/committee/reviewer needs to check for. That reflex is hard to shake, IMO.
> That is, after all, the goal of a human reviewer. To add _another_ layer of gatekeeping is what I'm questioning.
I guess I'm not following the distinction. For most people, that's exactly what happens. Either you, or the someone on your direct team, will be the person with readability for your code. The readability process just ensures that you don't end up with silos where style diverges too much, and that the stakeholder who proposes stylistic commentary has a baseline knowlege of linguistic best practices.
With few exceptions, the thing people complain about is attaining readability themselves, not getting changes approved.
> I don't agree with this perception at all. AWS spends a lot of time building frameworks and libraries which attempt to minimize human gatekeeping to only the necessary areas. So think retries, backoff, timeouts, circuit-breakers, etc.
I think we're talking a bit past each other. These things are all table stakes defined by frameworks and updated by centralized processes as best practices change. Human reviewers, for the most part, are only going to say "why are you diverging from the default".
But importantly, that sidesteps my statement entirely, which, rephrasing, is that amazon carries along more technical debt than Google. Whether that's a good long-term business decision remains to be seen, but the impression I get from ex-amazon friends is that maintainability is less prioritized, and maintainability is one of the goals of the readability process. Global consistency helps anyone understand and improve anything else, and to an extend avoids haunted graveyards (though...not entirely).
> instead gatekeeping is added at the architecture and operations stages, which is where most of the work of keeping a production level service up, available, and fast. Operational level gatekeeping in the bounds of a given organization is a much more neatly constrained problem
I'm not sure what you mean, but if anything, this sounds more bureaucratic.
> My impression from being outside of Google (and having a Xoogler for a partner) is that Google loves bureaucracy and process, and that engineers that continue to stay at Google are fine with navigating these processes. Its answer to problems is to add more bureaucracy, another item for the checklist that a person/committee/reviewer needs to check for. That reflex is hard to shake, IMO.
FWIW I don't get this impression. It's honestly difficult for me to tell what you even are referring to. Like, other than readability and promo, there's very few processes or bits of bureaucracy that are consistent across the company, and those two have gotten decidedly more streamlined since I joined the company.
Well okay that's half true, I can think of a bunch of places where bureaucracy has increased, but they're all spurned by legislation.
> I guess I'm not following the distinction. For most people, that's exactly what happens. Either you, or the someone on your direct team, will be the person with readability for your code. The readability process just ensures that you don't end up with silos where style diverges too much, and that the stakeholder who proposes stylistic commentary has a baseline knowlege of linguistic best practices.
Separating these two out, even if one person may fulfill two roles, IMO adds to bureaucracy that I don't care for. Ramping up onto a new language sounds like a pretty fraught experience if nobody on your team has readability and the service you're contributing doesn't have a member who has the time to actually work with you, especially if your work is not relevant to theirs. They even touched upon this in the linked video. The whole idea that you can enter a bureaucratic catch-22 at a company seems like an anti-pattern to me, a moment to stop and think; a "modern" engineering organization should try its utmost to accelerate development, with only as many checks as needed for safety and no more.
> With few exceptions, the thing people complain about is attaining readability themselves, not getting changes approved.
That's what I mean by bureaucracy. Even having "another eye" on the code should work. Having a certification process for readability is even more bureaucracy.
> But importantly, that sidesteps my statement entirely, which, rephrasing, is that amazon carries along more technical debt than Google. Whether that's a good long-term business decision remains to be seen, but the impression I get from ex-amazon friends is that maintainability is less prioritized, and maintainability is one of the goals of the readability process. Global consistency helps anyone understand and improve anything else, and to an extend avoids haunted graveyards (though...not entirely).
Ah, that makes a lot more sense. In my career I've never worked at a place that prioritizes code style that much, and I've worked at other high scale places in the past. This seems like a Google specific need. You're right that global consistency helps to save off "there-be-dragons" codebases, but I feel like this is a case of YAGNI. Unless you have engineers constantly cross-cutting across the company, most engineers learn the style guidelines of a team/product/service by ramping up on the team. The optimization for global consistency seems premature to me.
> FWIW I don't get this impression. It's honestly difficult for me to tell what you even are referring to. Like, other than readability and promo, there's very few processes or bits of bureaucracy that are consistent across the company, and those two have gotten decidedly more streamlined since I joined the company.
But promos drive _so much_ of the company culture, at least from what my partner tells me. And the video in the OP certainly makes it sound like Google has lots of bureaucracy, though I'm not sure how many of the requirements outlined in the video were there for hyperbole or not. But especially for services that have low SLOs, it makes no sense to put thought into failover, evacuation, or anything like that.
> Separating these two out, even if one person may fulfill two roles, IMO adds to bureaucracy that I don't care for. Ramping up onto a new language sounds like a pretty fraught experience if nobody on your team has readability and the service you're contributing doesn't have a member who has the time to actually work with you, especially if your work is not relevant to theirs. They even touched upon this in the linked video. The whole idea that you can enter a bureaucratic catch-22 at a company seems like an anti-pattern to me, a moment to stop and think; a "modern" engineering organization should try its utmost to accelerate development, with only as many checks as needed for safety and no more.
Keep in mind this video is ten years old. While yes, trying to write something from scratch in a language no one on your team has any experience in can be fraught (although this raises other questions: what exactly are you doing, in every case I've used a new lang, my team has been able to find people to review if needed), even in that case, the organization has help for you (see again my prior comment about "donating" readability, and my understanding is that the readability granting process actively prioritizes people who "need" the readability because they have few potential reviewers).
> That's what I mean by bureaucracy. Even having "another eye" on the code should work. Having a certification process for readability is even more bureaucracy.
I think initially the primary driver of readability is C++ (but Java + Guice also...needs it). Take a look at https://abseil.io/tips/. That's 70 tips, which is around a third of the internal ones. I have C++ readability and have internalized, some of those. The readability granting process ensures your code is reviewed by someone who knows all (or nearly all) of those tips, and will recognize when you're doing wrong things in your code and help explain how or why you can improve. When I personally went through the C++ readability process, I got mentorship on how to fix bugs in my code, some of which a peer on my team would have caught, and some of which they wouldn't have. I can now recognize those bug prone patterns (which are hard to lint for, in my case they mostly had to do with parallelism) and know about tools to debug them (msan and asan, which an experienced C++ user should know, but on my team of people who had basically no C++ experience at Google? Nah).
That follows into other languages. Being granted readability means you have a familiarity with the language that suggests you'll be able to effectively mentor others and not mislead them. You don't have that otherwise.
> Unless you have engineers constantly cross-cutting across the company, most engineers learn the style guidelines of a team/product/service by ramping up on the team.
Google uses a single repo and builds stuff from HEAD. This has up- and downsides, but one of the upsides is that its very easy to investigate unusual behavior in your dependencies. I was doing something similar just this week, and fixed obscure bugs in like 7 other teams tools that were causing downstream problems that those teams weren't aware of. No bureaucracy. I found the bug, fixed them, and sent the change to the owning team. Imagine other situations, where I'd need to maintain a fork, or file a bug to have them fix it, or coordinate an update with them.
You'll pay the cost no matter what, choosing to do so in a way that also provides value and mentorship makes a lot of sense to me.
> But promos drive _so much_ of the company culture, at least from what my partner tells me. And the video in the OP certainly makes it sound like Google has lots of bureaucracy, though I'm not sure how many of the requirements outlined in the video were there for hyperbole or not.
I mean the entire thing is tongue in cheek and again is a decade old. I joined Google ~5 years ago, and even then most of the problems with resource acquisition were solved ("flex") and borgmon and its associated readability were gone, replaced with monarch, which is centralized and used a python-based (though still admittedly arcane) query language that doesn't require readability or managing your own instance. And since then things have gotten even more turnkey for any service that's...reasonably shaped. And, well, the level to which promo drives company culture is consistently overstated (at least in some ways).
Consistency is what you get when you have a defined rule set programmatically enforced. If your looking for "readability" via human judgment, then you get a very different result.
A programmatically enforced set of rules is certainly one way to get consistency, but it isn't the only way. You can achieve consistency through culture and training too, and sometimes that's the only way.
It isn't possible to statically analyze if a class/struct is a POD or if the methods enforce invariants. But it's often very easy to do so with a human eye. And there's value in the distinction!
Similarly, forcing someone to justify using a power-feature (operator overloading, templates, metaclasses, whatever) can only be done by a human. There may be cases where the power feature is warranted and the benefits outweigh the cost, but a linter can't know that. (and ultimately all of this comes back to: things look consistent, and when things are inconsistent, that's a strong signal that something unusual is happening and you should pay close attention)
What counts for readability is not set in stone by some language czar as the One True Way. Everyone knows the style guides can't be perfect which is why they're relatively mutable.
In any case, readability will comment on stuff that cannot easily be quantified, such as when to use a certain object hierarchy or dependency injection, etc...
It's more about good usage of idiomatic language constructs, which still requires good human judgement to evaluate.