Aaron N. Tubbs bio photo

Aaron N. Tubbs

Dragon chaser.

Twitter Facebook Google+ LinkedIn Github

This is a tall tale of two companies: On the surface, Monolithoogle bears a mild resemblance to Google. Distributazon has some things in common with Amazon. These similarities are anecdotal and superficial. Accuracy is not the goal. At best, I am attempting to develop hyperbolic caricatures of the companies as archetypes. The ideas here evolved from a conversation I had with a good friend.

This is the second installment of Invisible Corporations.

Meet Monolithoogle

Monolithoogle got its start by building the best search on the web. A decade and a half later, it’s still the best search on the web. This at first seems consistent with Monolithoogle’s stated mission, which is to “organize the world’s useful information and make it universally accessible and useful.” Like most companies, their actual mission is to make money.

Search is a vehicle for Monolithoogle’s core business: advertising. They are really good at ad placement and distribution. They are exceptionally good at monetizing their skill at this. That’s where their best talent is focused. They have some interesting side “businesses” including a mediocre social media platform, a couple of operating systems, and some self-driving cars.

Everybody knows these things about what Monolithoogle is and what it does, but the how is where it gets interesting. Monolithoogle’s core competency is the care and feeding of the biggest monolithic software platform in the world.

From the very beginning, they hired the best and the brightest. Many came with PhDs from Stanford or comparable education and experience. With great intellect came egos, control freaks, and sociopaths. Early gods imposed their rules on the system from the start. System rules were accompanied by process rules. Monolithoogle defines a way to do things, and that’s how things are done. Deviations from these ways of doing things are brought back to the flock.

The company has evolved and grown, but the guiding structures and principles remain constant.

A small handful of programming languages, libraries, and tools are permitted. Much is built inhouse from scratch. External technology is used sparingly and often with significant augmentation or restrictions. New engineers must undergo training and licensing in a particular language or technology before being allowed to work unsupervised. After a point, it becomes clear that it is not just that there are rules, but that it is made painful and infeasible to avoid following them.

These restrictions thrive because there are corresponding benefits.

Testers matrix report to product teams and testing teams. All code must be tested. Engineers are part of code testing, but it is not sufficient for only an engineer or product team to provide the testing. Code doesn’t go out unless it is tested, but there are plenty of resources at the company to make this possible. The testing team is also responsible for helping engineers do more testing themselves, and to make the testing they do more useful. Testing is mandated supported by all levels of the organization. It’s not a tax, it’s just part of doing business.

Once testing is in place, code deployment is an arduous process. It requires coordination with the design team to ensure application consistency. A dedicated group owns deployment of the code to production and is responsible for its roll-out and validation. Operational support is provided by this team; the product engineers developing the code need not fear being paged for routine concerns. This is a nice thing, but means that the operational engineers have exacting standards and requirements of their own, and are unwilling to accept a deviation.

With all of these matrix-reporting teams, it is perhaps obvious that the management structure at Monolithoogle is not simple. Of course, everybody has a line manager. These line managers are seen once per month under normal circumstances, since they manage dozens of employees each. At first this might seem odd, but these personnel managers are somewhat irrelevant in terms of day to day project work. Each engineer is responsible to one or more tech leads in direct and indirect fashions. Several of the major and minor gods frequently take personal interest in individual technical decisions or designs of a project. It is a blessing. Groups on the periphery will criticize and second-guess choices made by a given team as sport, but everybody generally gets along.

It may take forever to get a project to market, but its evaluation beforehand is sure to have been quite thorough. There is a lengthy pre-launch period for a project followed by a lengthy beta period followed by a long period of uncertainty whether or not one of the major gods will just elect to kill a project for lack of being on-platform or on-mission. This provides plenty of time for the employees on the project to jump ship to another team to save their career. As long as somebody pulls their weight at Monolithoogle, they need not fear layoffs.

Titles and compensation at Monolithoogle are based on peer feedback. This is not a company constrained by subjective evaluations from management. Instead, subjective evaluation by peers and timing and posturing from management is key. The politics can be exhausting and the particular peers deciding somebody’s fate are likely not direct peers of the employee in question. Early in the company’s history, careful playing of these politics catapulted several careers. Early fortunes in the wake of the IPO were defined by how well this game was played rather than how strong one’s contribution was. These games left a lasting shape on the organization that is now hard to alter.

Despite that, the review system attempts to be fair and objective in its intrinsic subjectivity. Difficult metrics are set for each employee, and only by greatly exceeding them can one progress significantly in stature and compensation. It may be a popularity contest, but it’s one that comes with clear rules for how to win.

But, enough about the people (as much as is possible if one indulges fully in Conway’s law).

Monolithoogle’s investment in tooling and platform borders on absurdity. Their platform is better than anything anybody has used or built anywhere else, ever. Countless dollars and engineer lifetimes have been spent in order to ensure this. Code builds are in practical terms instantaneous. Massive amounts of scaffolding are built up around source control to make interaction with it brilliantly fast and effortless. Code flows naturally from development environments to review systems to testing systems to production systems (of course each of these steps is gated by humans).

As mentioned, it’s up to the deployment and operations staff to validate the project. Once it’s validated, systems procurement is never a problem. A massive team exists that ensures sufficient quantities and locations of data centers are available to meet the needs of projects. Where these are and what they look like are irrelevant, beyond that they exist and are always available. Monolithoogle would not want a project team spending cycles thinking about anything but their business problems to solve.

While on that topic, the infrastructure is rather capable. The choices are limited, but any choice available is sufficient to solve a given problem. If it isn’t, optimization will be performed down to the kernel or hardware level to ensure it is. If for some reason it seems something is impossible, perceptions are appropriately corrected. There is no appetite for discussion of alternative solutions or platforms, because they clearly have not been scaled to Monolithoogle’s needs. Fundamental architectural changes involve the entirety of the company’s senior leadership and require years to resolve before the implementation of the changes can begin.

Messaging, data stores, calculations, distribution, deployment, localization, synchronization, consistency, scaling, and numerous other things are all solved problems. Each problem area is owned by a specific team whose mandate is to provide a standardized solution to the problem. Each solution must fit squarely into the grand architectural model and conform to the requirements of the layers above and below it. If a problem is not solved yet but is owned by a team, consuming teams will wait until that team solves that problem.

There’s a fantastic sense of order to it all. Monolithoogle is not plagued by indecisiveness or questions about ownership.

Of course, like any carefully layered model, implementations details bleed between the layers. This makes it hard to decouple the layers, and it makes it very difficult to change things. Getting all of those pieces and layers right is a hard problem. As a result, Monolithoogle has developed an extensive (that is, expensive) Pokemon collection. Top professors, language designers, kernel hackers, and systems programmers are certainly present. These folks bump into elder gods that did things like invent the languages, libraries, and operating systems on which the whole industry is based. At first this might seem silly and expensive, but the explanation is rather straightforward. First, nobody but the best and brightest are smart enough to actually further develop or scale Monolithoogle’s platform. The other is that it makes it clear that one must be at least a minor deity before considering confronting the gods or elder gods. This discourages upstart behavior and youthful attempts to deviate from the one true way.

The only downside to this is that these Pokemon occasionally go and build something that Monolithoogle never actually wanted, and that’s sort of a hassle to manage. It’s not easy to give an elder god orders.

Secrecy is a big part of the culture. This isn’t just about the outside world; internal secrecy is pervasive. Information leaks are tightly controlled and counter-intelligence operatives are on staff to police this. Even the most inane details are considered state secrets. Just try to find out how many lines of code the company maintains! Token things show up in research papers or are open sourced, but they are often out of date or not foundational portions of the core platform and infrastructure. Many compelling open source products on the market are not from Monolithoogle, but are in fact re-implemented versions of internal infrastructure by ex-employees.

So, let’s get back to the people. The people are pretty happy! All of their basic needs are provided for in the same way that their engineering needs are provided for. Food, healthcare, car care, and all sorts of other nonsense are just a given. This makes the cost of quitting high: One has to be willing to give up the platform, the tools, and a way of life. Taken to its extreme, individuals can find most of their basic needs met by the company. These are difficult things to give up. For those attempting to acquire Monolithoogle talent, competition on direct compensation alone is a foolish game.

Of course, people do quit. It’s a big company and it’s hard work to accomplish something. It’s even more work to change something. For those that want to make an impact or make it big, there is a lot of appeal to a smaller company or startup. But, on the whole, retention is pretty good. Those that stay may at times complain about aspects of the job, but they’re quick to add “but I couldn’t work anywhere else.” Many that escape the clutches and join a smaller firm or start their own company quickly find themselves attempting to replicate or re-implement platforms available at Monolithoogle. This can be a healthy strategy to make integration into Monolithoogle more straightforward when they are acquired.

In addition to retention, growth requires recruiting. Talent acquisitions (more on that soon) provide some scale, but the company has to get fresh and seasoned bodies in the door through other means. Monolithoogle has a lot of street cred among engineers (and others, which counts for something). Some of this comes from the things they do that don’t make money (e.g. X and Labs). Some of this comes from the cool papers and occasional open source project they release. Some of it comes from the nice work environments and perks. There is, as a result, a steady flow of applicants, but there is a lot more noise than signal. Like many companies, it survives on referrals, if for no other reason than that they’re statistically less likely to suck (read: they cost less to recruit). Candidates must enumerate the people they know inside the company so as to further broaden the criticism and feedback. Mistakes are expensive.

Monolithoogle has a famously difficult and lengthy interview process, sometimes involving several rounds of phone screens and exhausting on-site interviews. On-site interviews for other positions may be offered if a candidate wasn’t quite capable of the first role but there’s a better fit.

Even at Monolithoogle’s current scale, each new hire requires personal evaluation and sign-off by a major god. Packets are prepared for each candidate and they go through several committees of approval before being blessed or damned by said god (if they weren’t eliminated earlier in the process).

Recruitment is a lengthy and notoriously picky process, but insiders feel like there are fewer idiots around as a result. Most of the idiots are smart idiots, which is to say they have different opinions.

Monolithoogle’s Problems

On the surface, Monolithoogle seems like a pretty good company. Its employees are happy and challenged. It’s making money. There are great perks. Are there any downsides?

One may be Monolithoogle’s consumer experience. This should not be confused with Monolithoogle’s paying customers (the people that buy advertising). Most of the company’s products more complicated than search suck. Many that get a foothold and turn out to be off-mission down the road get killed, stranding users. Here’s the problem: The company’s mission and social platform are not designed with the consumer’s desires or needs in mind. It has to be just good enough to sell ads and maintain the brand equity.

The platform and culture (if they can even be considered distinct) have a certain rigidity that diminishes innovation. High-profile cultural (think 20% time) and organizational constructs exist (X, Labs) to create an illusion otherwise. Reality is that innovation comes from acquisitions. After all, they acquire a company every week. They have developed a sophisticated mechanism for finding, acquiring, and integrating these companies. Teams are in place to facilitate effective selective integration/divestment of talent, real estate, capital goods, and product. Products that are acquired are quickly brought on-platform, another problem found intractable by nearly all of the other companies in the world.

Another quirk is that Monolithoogle has had a hard time monetizing some of its core competencies. The true infrastructure on which the company runs (down to how their systems are built in data centers) is not for sale. Releasing or selling anything but the entire stack becomes somewhat meaningless. Selling the entire stack would diminish their competitive advantage. So they remain a successful advertising company and haven’t been able to sell the things they’re good at. This is not necessarily a bad thing, but it is a thing.

There are still some holdovers from the early days. Most companies reach a point where they are large enough that the old ways cease to work. Monolithoogle is different; it believes in, at all costs, scaling the monolith in every way possible. If something worked before, they can make it better, faster, and scale even bigger. The platform must support the requirements of anything that could run on it. The organization must maintain control. Nothing less will do.

Distributazon is Different

It may be a stretch to suggest Distributazon and Monolithoogle are polar opposites, but there are some rather substantial architectural, organizational, and cultural deviations. While it would be convenient to present another entire story in a vacuum, some shorthand can be leveraged by exploring just the differences.

Distributazon got its start as an online bookstore. Now, it’s an online everything store. Its mission is a mouthful: “to be Earth’s most customer-centric company, where customers can find and discover anything they might want to buy online, and endeavors to offer its customers the lowest possible prices.”

It’s barely making a profit because it is reinvesting all of its earnings. Distributazon makes a modest majority of its money by selling stuff and the rest by selling virtual stuff. In reality, it makes money on the margins of selling stuff and virtual stuff. It does this by having incredible storage and shipping infrastructure. At a scale no other company has attempted, it’s partnered with and constructed facilities near logistics providers to make this all incredibly smooth. One of its few large acquisitions is a profitable and successful company that just happens to have a large facility adjacent to the UPS worldport.

But, what makes Distributazon work isn’t that it has all of this infrastructure, it’s that it knows how to make that infrastructure work efficiently and play nice with all of the other infrastructure. Sure, there are box-packing algorithms and robots. The real innovation, however, is that they’ve built a huge and successful service oriented architecture.

Distributazon’s CEO famously dictated a massive architectural change. The quick version: All teams expose their data and functionality through service interfaces, these are the only ways teams communicate, and there’s no other IPC allowed. These services can be used by anybody, in or outside the company. The technology stack does not matter. At all.

Where Monolithoogle dictates the platform and technological choices, Distributazon does not care. As long as the functionality is self-contained, can be consumed by others, and can consume other functionality, and does so through a service interface, well, screw the details.

The implications of this are somewhat obvious: Where there’s one true way to do things at Monolithoogle, nobody cares how things are done at Distributazon. If a team wants to manage its own hardware, even, so be it. They can hire their own hardware people and systems people and network people and real estate people and build their own data center for their service.

Let’s just call each block of functionality and the associated technology and staff a silo. This can be used interchangeably with “a service” to a degree. Where Monolithoogle’s method of cooperation involves sticks, Distributazon’s method of cooperation involves carrots. Going back to the hardware example, one of Distributazon’s silos offers virtualized hardware as a service. They made it so awesome and easy that the other silos would be stupid to manage their own hardware … and so they don’t.

The counterpoint to this is that a silo needs to make something that doesn’t suck. Wheel invention is welcomed, but new wheels need to be big improvements over old ones. The alternative is that the silo ceases to exist. But, there’s no such thing as a sacred silo. The lifetime of every staffer, service, or silo is defined by how good they are versus the competition.

It should come as no surprise, then, that there’s a lot of competition at Distributazon. Silos aren’t encouraged to work together as much as compete existentially with each other. If a silo’s metrics aren’t looking good, just terminate the whole thing. Since everybody needs to design their silo to tolerate outages or failure of other silos, this turns out to scale pretty well organizationally just as it does technically.

From a software architecture standpoint, stability of the system comes from stability of the silos. Instability is difficult to debug and ownership is difficult to assign. To attempt to mitigate things, difficult requirements are imposed on silos regarding imposing (and enforcing) limits and providing (and honoring) service contracts. Silo owners learn quickly to test their services, as the silo developers own the support and operations. A bad service means lost sleep and a consistently bad service means no job. So, there are no mandates about things like testing and QA staff and quality and uptime, but it’s not surprising that silos will often invest in these things to avoid getting the whole team canned.

On this topic, metrics are easy to derive for a given silo. Usage is intrinsically baked into the software architecture. It’s easy to measure how often services are used, who uses them, how fast they are, and what their growth (or decline) in usage is over time. Budget is directly linked to this, and it’s up to the silo to figure out what that means for its inhabitants. Whether or not it’s fair is hard to say, but it’s in a very brutal sense objective.

Since each silo gets to figure out who it hires and how it runs, there’s a lot of variation in team make-up. It’s just as often a silo is run by a tester as it is by a software engineer as it is by a product lead or business analyze.

In general, this makes for a bunch of miserable employees. Few leave the company singing praises about its culture. At the same time, they all admire the intrinsic genius of the architecture: The company survives just fine if somebody does leave. A nice side effect of is that it doesn’t really matter who they hire: If a team doesn’t work out, they just try another one. If they want to offshore an extant or future silo, they just spin one up overseas and it plays by the same rules. Budgets and architecture and team composition are all incredibly efficient at self-governance as a result.

It may not be surprising, then, that attrition at the company is high. A heavy recruiting function (you guessed it, as a service) provides an ample supply of warm bodies. They use clever (if somewhat misleading) offers to attract college and industry talent that suggest growth and huge bonuses. Especially for folks that care only about the numbers on paper, they look especially attractive to somebody just leaving college. After reality sets in, their compensation is weaker than Monolithoogle’s.

The quality of candidates doesn’t matter that much. This makes recruiting easier. Screening out the bad candidates is a task for a given silo, but this process is incredibly efficient and quick due to the metric and service-based organization. If a team member doesn’t hit their metrics, HR as a service cans them. It’s really rather elegant. If a team member does well, they are rewarded efficiently and effectively, but not more than makes sense: There’s a silo that carefully calculates how much value an employee is providing and will ensure they’re not being paid drastically more than that in the long run.

It’s a relatively well-liked company from the outside but a relatively loathed company from the inside.

Distributazon’s street cred isn’t a great as Monolithoogle, but it’s not that bad. Since everybody in the world buys stuff from Distributazon, people not working at the company are pretty positive about it. For the engineering staff, the difficulty in ownership, infighting, and brutally metric-driven lifestyle is rough. The perks aren’t great, especially compared to somebody at Monolithoogle. So, attracting the top-tier talent and retaining them is challenging. Rather than be concerned by this, it’s embraced by the company. Arguably, the engineers that use Distributazon’s infrastructure from outside the company are even happier than those that use it internally.

Distributazon famously is stingy about its open source contributions, despite consuming open source software to build its silos. Strangely, this provides a connection with Monolithoogle, as much of Distributazon’s inner workings are purportedly secretive. Despite this, Distributazon is able to expose far more of its infrastructure to the outside world (by design), but only at the service interface level (yep, also by design). Don’t like it? Just roll an alternative for that service.

The products are not especially consistent, since they look like they were pieced together by several hundred different teams. This is true because this is, in fact, what happens. Consistency of experience, then, is incredibly poor versus somebody with platform lock-in and command and control like Monolithoogle. Luckily, none of the end users really seem to care.

There are a lot of things to like about the distributed nature of the architecture and organization. Acquisition integration is straightforward. A new product just goes service-oriented and plugs into the services provided by the company, and it’s off to the races. That the acquired company had people doing things before can remain that way if it makes sense. If it doesn’t make sense, that’s fine too.

With that said, Distributazon has acquired surprisingly few companies; when it has gone after opportunities it’s done so as a pragmatic way to grow into new markets or capabilities that were lacking. Tricks like talent and IP acquisition really aren’t Distributazon’s game. If some upstart wants to come in and reinvent how something is done, they are more than welcome to do so. If they suck at it, it’s easy to replace them. Everybody wins.

Monolithoogle exposed that as companies grow larger, the innovation stagnates and an easy way to get new ideas and talent is to buy it. Again, Distributazon’s organization and architecture combats this problem: The innovation hasn’t slowed down because it behaves like a bunch of small companies. If it decides it wants to start a hardware division or start selling any given part of its internal infrastructure or if it wants to break into the grocery market, it can. All the various silos have to do is make money and they’re golden. Killing products is never about whether or not a team is on-mission and everything to do with whether or not a team is hitting the relevant metrics to survive.

Time to market is short. Formally speaking, a silo can launch a project instantly as long as it honors the architectural rules and meets its service guarantees. Culturally this has some advantages as launches can be quick and frequent with a lot of iteration. There’s no overhead of coordinating with other teams, because the people that build the service own and support it and determine how best to deal with that problem. Anything they need from other services is negotiated at the service layer, so there’s no need to even meet with other people to launch a product.

At first it sounds like this is just a description of the software aspects of the company, but the notion of everything-is-a-service and metrics-driven-everything is pervasive. Distributazon has (you guessed it) a silo that models efficiency of warehouse staff as a service, taking into account the unique inputs for any given scenario. It knows how long it should take for a worker to get to a toilet, and it knows how long the average crap should be. If a worker isn’t hitting those metrics, they’re yanked and replaced with somebody who will.

Wrapping Up

Neither of these companies are better. Neither are good or bad. They’re very different. They’re both large companies that use a lot of technology and clever software engineering in support of their fundamental businesses (which are not software).

So, the interesting thing to do is to recap and tease out some of the differences along fundamental axes.

Software Architecture, Org Structure, and Innovation

Distributazon has org structure per silo. If there’s anything between the silos and the CEO, that probably depends on how big each silo is. The corporation is run like it was composed of a lot of smaller companies and startups. Innovation, time to market, and agility are not a concern. Acquisitions are infrequent and focus on accelerating growth into novel markets. Their software architecture is of distributed services.

Monolithoogle has a vast hierarchy. At first it looks somewhat flat on paper, but in reality the power is controlled by an elaborate matrix of management. Organic innovation has stagnated as the company has grown, so innovation comes via a healthy acquisition appetite. Their software architecture is incredibly monolithic and restrictive, with all of the benefits that introduces.

Talent

Monolithoogle hires and retains some of the best talent in the industry. Its biggest source of attrition in top talent are people that want to make impact and accomplish “big things.” Doing these things at the company are generally a thing of the past. Generally, talent at the company is happy. Performance measurement and compensation is determined by politics and popularity contests.

Distributazon hemorrhages staff at a rate only slightly lower than their ability to acquire it. Talent is generally miserable. Staff can accomplish things at virtually any scale, but they also quickly burn out or get canned. Performance measurement and compensation is determined by objective metrics.

Monolithoogle has to be incredibly selective and maintain great street cred to grow. Distributazon instead just needs to ensure it has a healthy volume of bodies coming in the door at all times. It has to ensure that culling the herd is similarly efficient.

Customers, “Product,” and Product

Monolithoogle’s money comes from customers (people that buy advertising) that prey on non-customers (normal human beings) that use their ostensible products. The non-customers are not the priority but the company still develops a fanatical following. The “product” for this audience is not on-mission and therefore is not important.

Distributazon’s money comes from customers that use their product to buy things. Their incentive is to make the product good for this audience.

The Outside World

Distributazon’s P/E ratio is comically high. It’s an order of magnitude higher than Monolithoogle. It’s reinvesting every cent it makes back into the business. Monolithoogle made bold claims about not caring about profitability, but it sure seems to like making a profit.

Both companies have mountains of admirers and critics. Monolithoogle seems more evil than they purport, and clearly does not make the “user” their top priority. Distributazon has singlehandedly destroyed every small town store that wasn’t already destroyed by Wal-Mart and annihilated the local bookstore because boy do people want meatspace books these days.

Distributazon is better about sharing its infrastructure with the world. Monolithoogle is better about trying to do nice things for the planet. Or at least to reduce its power bills.

Some Food for Thought

These invisible companies are the endpoints of an arbitrary continuum. Both are successful despite substantial differences in their approach.

A question to consider is how successful a company can be if it picks a point somewhere in the middle, instead. Some obvious questions develop from this:

  • Can a company pursue and retain top talent while simultaneously treating talent as easily replaceable?
  • Can a company attract top talent while sustaining a high rate of turnover?
  • Can organic innovation mix well with an aggressive schedule of acquisitions?
  • Can performance measurement and compensation be a mix of objective and subjective measurement?
  • Can a monolithic software architecture mesh with a distributed software architecture?
  • Can a monolithic organizational structure mix with a distributed software architecture (or vice-versa)?
  • Can a company succeed by making its employees somewhat happy, but not too happy?