Architecture principles

Macro-architecture is about the "architecture on a large scale": what subsystems exist, how do they work together, and what principles must all subsystems and teams adhere to.

1.A) Vertical system section

A "classic" architecture divides into layers and modules.

This is often reflected in the team structure: one or a few teams per layer. This division is not entirely unproblematic:

• For many requirements, the work of several teams must be closely coordinated because several layers are affected.
• The number of teams is usually difficult to scale. The number of developers per team anyway.
• Bottlenecks in single, central teams become a problem for other teams.
• The code base becomes very large very quickly. No one can see through the overall system anymore.
• The teams are closely linked by the common code base. Even updating a library in use can become a game of patience.

One could go on like this for a long time, but this is about the Lhotse architecture: and there we have decided on a vertical cut through the system.

Each green box is a standalone application with its own data management and front end. Some teams have more than one "vertical", but no vertical is developed by multiple teams.

Having their own data storage also means that verticals are not allowed to share state over the same database. A shared database would again create a tight coupling between the systems: they would have to agree on the schema of the data.

The internal construction of a vertical is entirely the responsibility of the team. In particular, there is no common code base. While this occasionally leads to parallel development, it does not require the divergent needs of multiple teams to be bottlenecked into a shared library.

1.B) Shared Nothing

The term "shared nothing" expresses that the subsystems among themselves, but also the individual instances of a clustered vertical, do not share a common state in the application. Thus, there are no in-memory caches that cluster nodes need to talk to each other about, no HTTP sessions that are replicated between instances, etc.

Since individual instances don't share anything, they don't need to communicate with each other, and no load balancer needs to take into account which "session" needs to be routed to which nodes (this is important for deployments, among other things, as well as for resiliency).

State in any form is therefore only allowed outside the system: In the browser, the database, a memcache, or in an HTTP cache. Problems with cache coherency are completely avoided in this way.

1.C) RESTful Architecture

Of course, the individual verticals must communicate with each other despite all their independence. There is also the need to provide technical interfaces for, say, external apps or the like. And then, of course, there is the integration of the subsystems into a common store GUI and the configuration of the store by a back-office application (aka "shopoffice").

To make all these things possible, we agreed on a REST architecture. REST doesn't mean "we exchange XML documents for HTTP and also use PUT and DELETE"; we take it a bit more seriously:

• Defined resources
• Use of defined Media-Types (if possible existing, otherwise Vendor-Specific)
• Respecting defined HTTP verbs, status codes and headers
• The unpronounceable HATEOAS principle (always reminds me of PCMCIA - People Can't Memorize Computer-Industries Acronyms), i.e. the use of links or "hyermedia controls" in resources to make the state transition possibilities (the ST in REST) navigable.
• Support for HTTP caches.

Among other things, this has resulted in an open source project http://github.com/otto-de/jsonhome that allows an application to publish or also consume a json-home document.

1.D) Data and data supply

Since the application is an online store, chances are that many subsystems need to know something about products. But if they do not use a common database, how will information be shared? We have two options for this:

• Creating redundancy via data supply processes.
• Ad-hoc requests via REST interface

Data supply processes are always asynchronous in the background. No customer should have to wait for such processes. Inconsistencies between the systems are deliberately accepted.

The accesses between the systems are always done via PULL mechanisms such as AtomPub feeds, because this way the coupling between the systems can be reduced. Since we have to expect inconsistencies anyway, a few seconds delay compared to an update via PUSH is not important.

If such inconsistencies are unacceptable, systems may exceptionally make ad hoc requests to other verticals; however, we try to avoid this because it involves tight coupling between systems.

However, the core of the architecture principle is that only one vertical is leading for all data. All other systems only access the leading vertical via REST interfaces and keep redundant data if needed. The "truth" about a datum is in the hands of the leading system.

2. micro-architecture

In contrast to macro-architecture, micro-architecture is about the "architecture in miniature", i.e. that of the individual verticals.