Whats wrong with Microservices?

This is the first post in a longer series on service oriented software architecture. The story begins with a retrospective of a large software project I was part of at the beginning of the century. The task was to implemented a new merchandise software suit for a large European retailer in line with their “Delta” service architecture.

Delta

The Delta service architecture was based on healthy design principles such as encapsulation, autonomy, independent deployment and contracts. The principles had been used on the business level and created nice fine granular, independent services with names like Merchandising Store, Assortment, Promotion, Retail Price, Store Replenishment, each service being responsible for a limited but clearly defined business capability. Remembering correctly there was more than 20 services to be made for the whole suite including buying and selling, representing a functional decomposition of the business.

Figure 1: The Delta Service Architecture

Since all services should be deployed in the same Enterprise Java production environment the team decided on a product line approach, and established a set of principles supported by common components such as the Object Relational Bridge for object persistence, JMS (Java Message Service) for asynchronous contracts processing, and implementation of domain logic using a rich object model guarded by transactional boundaries, an approach that turned out to work reasonable well from a technical point of view. Blueprint shown in figure 2.

Figure 2: Technical architecture anno 2003.

Today some of these choices might look weird, but there was no Cloud, no Spring framework, and no Docker when this was made. The client had in addition made several architectural decisions such as choosing application server, database and message broker, choices that reduced our ability to exploit the technology as we could have by starting out with a open source strategy.

One lesson learned here is, if possible, to avoid premature, political high profile commercial technology decisions. Such decisions always involve senior management as serious money change hands, and at the same time they need to be architectural healthy to stand their time.

Core Domain

The development team was new to the retail domain, and therefore we decided to begin with he simplest service, Retail Store. In retrospect this was a bad decision because it made us begin at the fringe of the domain instead of at the core.

Item is the core of retail. Deciding what items to sell in a particular store is the essence of the retailing business. A can of beer, the six pack and the case of six packs area all items, the same is a bundle of a pizza and a bottle of water. The effect being that there are tens of thousand items in play, some of them are seasonal, others geographical, and others are bundles and promotions such as 3 for 2.

Items can be nested structures organised into categories such as diary, meat, fish, fruit, and beverage. Which items are found in a particular store is defined by store type or format, size, season and geography. A simplified domain model is found in figure 3.

Figure 3: Retail assortment hierarchy from 35000 feet

When the work began we had a high level business service architecture, but we had no domain model showing the main business objects and their relationships to guide the work forward. The effect was that the team learnt the domain the hard way as it dug itself from the edges toward the core.

A fun fact is that the team met Eric Evans when he presented his book on Domain Driven Design in 2004 discovering that we had faced many of the same challenges as him. The difference was that he had been able to articulate those challenges and turn them into a book. Had the book been out earlier we would have been in a better position to ask hard questions related to our own approach.

Learnings

I have identified five major learnings from this endeavour that are relevant for those who considers to use Microservices as architectural style for their enterprise business application. At first glance the idea of small independent services sounds great, but it comes with some caveats and food for thought.

Firstly, a top down business capability based service decomposition without a thoroughly bottom-up analysis of the underpinning domain model is dangerous. In Domain-Driven Design speak this mean that the identification of bounded contexts require a top-dow, bottom-up, middle-out strategic design exercise since business capability and domain model boundaries are seldom the same. Cranking out those boundaries early is crucial for the systems architectural integrity. Its the key to evolution.

Secondly, begin with the core of the domain and work toward the edges. Retail is about the management of items, how to source them, and how to bring them to the appropriate shelves with the correct price given season, geography and campaigns. Beginning with the Retail Store service because it was simple was ok as a technical spike, but not as strategy.

Thirdly, fine granular services leads to exponential connectivity growth and a need to copy data between services. The number of connections in a graph grows according to f(n) = (n(n-1)/2). Therefore 5 services have 10 connections. Doubling to 10 services gives 45 connections, and a doubling to 20 services gives 190 connections and so on. The crux is to understand how many connections must be operational for the system as a whole to work and to balance this out with a healthy highway system providing the required transport capacity and end point management.

Fourthly, the development team was happy when the store service worked, but a single working service at the fringe of the domain does not serve the business. The crucial question to ask is what set of functionality must be present for the system as a whole to be useful? The ugly worst case answer to that is all, including a cross cutting user interface that we leave out for now. The lesson is that Microservices might give developers a perception of speed, but for the business who needs the whole operational the opposite might be the case. Therefore should the operational needs drive the architecture as functional wholes must be put into production.

Fifthly, the service architecture led to a distributed and fragmented domain model since service boundaries was not aligned with the underpinning domain model seen in figure 3. Price, Assortment and Promotion has the same data foundation and share 80% of the logic that we ended up replicating across those services.

To sum it all up, understand the whole before the parts and then carefully slice the whole into cohesive modules with well defined boundaries remembering that the business make their money from an operational whole, not from fragmented services that was easy to build.

Microservices

Microservices is an architectural style introduced around 2013 that promotes small autonomous services that work together, modelled around a business domain and supported by a set of principles or properties such as culture of automation, hide implementation details, decentralise all the things, independent deployment, consumer first, isolate failure, principles that are difficult to argue against, while a literal interpretation might cause more harm than needed.

Philosophically Microservice follows the principles of Cartesian reductionism, as did Lord Nelson in his divide and conquer strategy. The big difference, Lord Nelson’s task was to dismantle the French fleet, not to build a new fleet from the rubbles, and this difference coins IMHO the major challenge with the Microservice style. Its aimed at independent development and deployment of the parts pushing the assembly of the whole to operations. Some might argue that its then fixed by the DevOps model, but if there are 20 services supported by 20 teams the coordination problem is invitable.

Conclusion

Service oriented architectures and the Microservice architectural style offers opportunities for those who need independent deployment. For applications who do not need independent deployment a more cohesive or monolitic deployment approach might be better. Independent of style, the crux is to get the partitioning of the domain right at design time and in operations. The key question to answer is what must be operational as a functional whole?

This mean that the design time boundaries and the operational boundaries are not identical and for a solution to be successful the operational boundary is the most important. That said, to secure healthy operational boundaries, the internals of the system need to be well designed. Approaching a large complex domain with transactional scripts will most likely create problems.

In the next post the plan is to address how data management can be moved out of the functional services enabling data less services along the lines that data ages as wine, while software ages as fish …

See you all next time, any comments and questions are more than welcome.

Resurrection

Wellcome back to what I hope will be a living and more regular blog on software architecture and design challenges. A lot has happened since my last blogpost in 2016, and I find that the importance of holistic design and architectural thinking has increased over the years…

Topics that will be covered moving forward includes but are not restricted to micro-services, knowledge knowledge modelling, designing collaborative workspaces and last but not least, the lessons learned from building the OSDU Data Platform.

My objective is to bring some of the lessons learnt from spending decades working on implementing software solutions to business problems to a new generation of software practitioners and thinkers. A lot has happened since I started studying computer science in 1980, at the same time we circulate around the same problems though with much more powerful tools, and at times I wander if this has been for the good or the bad. Not that it’s not good to solve demanding problems, but that our methodologies and approaches does not develop fast enough to take advantage of the technology.

The first post will come in a week’s time and address what’s wrong with micro services. It’s a big mount full, but as I see it an important one to address. The question on the table is, what are the benefits of making a distributed networked solution to what in most cases are homogenous problems, and more importantly, what kind architecting can be applied to make it better.

Looking forward to seeing you again.

Boosting innovation through Open Source Software

It is common knowledge that a stalled aircraft falls like a stone. The same can be said about «stalled software», i.e. software where the developing entity has lost its ability to keep up with market needs.

We find «stalled software» in both enterprise legacy systems and in commercial industry specific solutions, particularly in industries characterised by shrinking and marginal markets.

One such marginal software market is the upstream oil and gas sector. This comes from the fact that there are a limited number of oil companies, and due to industrial consolidation the number is shrinking. Add then that most of these companies are very large and have a tradition for doing things in their own way.

The business model the software product industry has been built on new functionality was funded by the sales of todays offering to new customers and a share of the maintenance fee. The caveat that comes with a stalled market is that there are no new customers, with the effect that vendors must turn either to their existing customers for funding or to their shareholders.

Various models for joint development has been used, where the most used model is that the customer pays the vendor to develop something that is mutual beneficial, but where the vendor gets the ownership of the end product. This model is at its best weird.

A better model is to engage in collaborative development using an open source model. Parties contribute to build a shared platform that can work as a vehicle for innovation. Open Source provides the legal and technical framework for this.

The effect is that vendors, competitors and customers joins forces to boost innovation, collaborates on a common platform that enables them to innovate on areas where they want to be distinct for a fraction of the cost that comes with the current license model.

Automated planning and scheduling

It takes a radio signal between 8 and 22 minutes to travel the distance between Mars and Earth, giving a  roundtrip time between 16 and 44 minutes. The effect of this is that remote-control as we find it on the Earth does not work.

The consequence is that a vehicles like NASA´s Curiosity (picture) must  the able to sense its environment, analyse the operational situation and plan and execute actions in line with its mission assignment. This makes Curiosity an implementation of what I have classified as a system of action.

From a computer scientists perspective such systems  are built using  methodologies and technologies rooted in control theory, signal processing, pattern recognition and artifcial intelligence. Control theory is typically applied to solve sub-second vehicle control challenges and artificial intelligence underpins the vehicles tactical planning and decision making, including interaction with the human operator for strategic guidance.

The technologies used for planning and decission making falls into the fields of constraint programming, automated planning and multi-agents. Automated planning is the application of finding the best possible action, given the constraints of the situation and the objectives of the task at hand. The resulting logic is packaged into packages of collaborating agents facilitating a decoupled and flexible architecture.

It is though a paradox that while this class of technology is used to control space crafts on foreign planets, usage on similar problems here on planet Earth is more limited. That said, there is no doubt that more Earthly applications, it be optimisation of hospital operation rooms, supporting an drillers to mitigate problems avoiding to get stuck or to find the best possible action to take to maximise production while minimising energy usage are all examples of applications or problems that could benefit from being understood and solved as planning problems.

My claim is that software for automated planning and scheduling is one of the most underrated technologies in the market space. At the same time is it one of the most powerful technologies available. It provides beautiful solutions to a multitude of real world business problems. One such example from industry is the RDS robot.

Lets make 2016 the year of automated planning.

 

How companies become digital winners

This is the title of McKinsey podcast from the12th of January 2016 where they outline what they observe digital winners among large companies do. The first thing they point out is that incumbent companies should not try to mimic Uber, Netflix or any of the new digital only companies. The reason, they are few and they have very unique business models. Incumbent companies needs to leverage digital within their own context and as part of their business proposition.

What characterises the digital winners then? In short, four things:

  1. Alignment of strategy i.e. strategy includes digital
  2. Agile culture
  3. Capabilities
  4. Organisation, people and processes

Of these strategy comes first. Winners seams to organize their strategy around the following concerns:

  1. Who are my competitors in the new world? They might be others than what you are used to.
  2. How fast must we act? Understand the timing. Must we act bold and bravely or is a more slow response appropriate?
  3. Understand that digital is much more than online sales. You have to develop new value propositions, leverage data to use assets more efficiently and build new relationships with customers and partners.

Agile culture comes with something we might regard a paradox. Agility builds on a stable core processes. Its the stability that enables agility. This is also associated with the two-speed IT. One part focusing on stable core systems, the other on fast moving changes and experimentation.

When we come to capabilities there are four that stands out:

  1. Decision making, understanding that the least risky thing is to do something. Standstill is the most dangerous position. To often managers are seen acting too late.
  2. Connectivity, linking together products, goto market models and technology
  3. Radical cost reduction. Eliminate, simplify and automate processes.
  4. The two speed IT organisation, enabling new value prepositions

Finally we find the things digital winners don’t do. Firstly, they do not repurpose people. They seek external capabilities and they make decisions about what we should stop to do.

Secondly, they do not outsource, at least not the things they regard core. They hire and use mergers and & acquisitions to acquire competences and capabilities. They spinn of things that need to live on its own and they create incubation units when appropriate.

Thirdly, they experiment and mobilises behind winning models or approaches when they find them. Finally, there is no internal competition. In the cases that might be the case, the competing unit is spawn of and given an opportunity to prosper.

What we must remember; digitalisation is more than IT, its business.

Digitalisation – Creating a software organisation

Software is difficult to master, but at the same time is its mastery the key to competitive advantage in the digital world. Success depends basically from putting in place the following elements:

  • Know why you need to create your own software, and it should be about profit.
  • Make sure your requirements engineering capability is top notch.
  • Establish and nurture multi-skilled DevOps teams that contain domain knowledge.
  • Choose good programming languages, automated development environments and engineering practice.
  • Make sure its part of the firms technology branch.

For those who want to learn more, a more thorough story is told below:

The first thing to put in place is the rationale for doing it. The only valid reason is profit. I have chosen to talk in terms of profitability, as I see cost reduction as a mean to create profit.  There are basically three ways software can create profit:

  1. Create a product and license it to users for a fee.
  2. Create an online subscription service.
  3. Create an internal product or service that automates and optimises internal processes.

In addition there is a forth alternative; make internal services available for external customers for a fee. All these three options can use open source development and licensing models for cost reduction and sharing. How that can work will be addressed in a separate post.

So, knowing why you want to be good with software, lets take a look at what you will need to put in place.

Firstly, since software is systematised knowledge, and knowledge is perishable by nature your software will need to be continuously updated. This make your software outfit a centre of change and it needs to be designed in such way.

Secondly, to quote Steve Jobs: «the most important software you have, is the one you decided not to write». This mean that product management is crucial to succeed and it means that the product owners, to use a SCRUM term, most important word is NO. Being able to prioritise features in such way that released products or services continuously adds business value is crucial for the profitability of your investments. In terms of software engineering this mean that a good requirements process need to be in place.

Good requirements is the key to reduce rework. Therefore is it not enough to provide the right requirements, but the requirements need also to be right. If it cost $1 to fix a requirement the same defect will cost $20 during design build and most likely $200 or more in production. Good requirements engineering practice is the key to product profitability and product quality.

Thirdly, with a backlog of good quality requirements we need a skilled team of developers with good software engineering practices at their backbone. The best way to organise the product team(s) are as DevOps, where the same people is responsible for new features and operational stability / product quality. The team should consist of both subject matter specialists and software engineers, usability expertise and testing capabilities.

On the technical side I have a preference for strongly typed programming languages combined with Domain Driven Design, ensuring that the domain concepts at hand find its way into the source code, and that the source code is readable in terms of domain concepts. A banking systems operates with concepts such as accounts, interests rates, deposits and withdrawal.

To enhance product integrity develop libraries that are shared between various modules. Such libraries consist of functions and immutable value objects.

Fourthly, put in place an automated production lines for both development and operation. Instrument the running code with logging and use tools such as Splunk to analyse these logs in real time. It´s the only way to capture and understand the products operational behaviour.

Finally, how to organise? Dependent of size, but my recommendation is to make this entity part of the firms technological muscle and place it at the appropriate organisational level. The more important it is for the firms competitiveness the closer it should be to the CEO.

Digitalisation – Becoming a software company

Accepting that future success depends on how good your organisations becomes with software does not come easy for leaders who have succeeded in a traditional industry sector; it be oil, health care, engineering or manufacturing.The hardest part  is to understand  the practical implications with respect to what decisions and actions that must be made and carried out to succeed.

Before we try to answer what actions that must be taken, lets take a look at what software is and what its used for. Software is a representation of systemised knowledge that can be repeatably executed by a computer.

Knowledge mean anything from what is needed to control the temperature in a living room to pursuing strategies in war-games or even war it self and everything in between. One off the most common applications is to use software to strengthen human capabilities and senses, as is the case in medical ultrasound imaging. It is software that make it possible for doctors to look into the internals of a beating heart and to see if the heart chambers are performing as intended.

The same is true for seismic imaging where software algorithms are used to create images of 200 million old sediments, tousands of meters below todays surface.

Its by understanding that software is knowledge we understand why software is perishable and need to be continuously updated. The need for continuously updates includes also the software tools used to create software.  This make change the name of the software business. For sectors who strives to avoid change this feels scary.

At this point one might argue that we have created software programs for more than 50 years, what have changed? The answer to this is the cost of computers. Back in the days a computer was so expensive that only the most valuable problems where tried addressed, and programming was for the first of the few. Today computer hardware is pervasive and almost for free, making the start-up cost with global reach affordable for anybody. Think of the app-stores and the fact that smart teenagers with some luck become millionaires.

The effect of this is that software is leveraged at the edge of any business, forcing firms to systemise their intelectual property and knowledge and package it into software for sharing, sale or just internal use.

On the strategic level the implication is that vendors become competitors, customers might become suppliers and nobody knows how the market and the competition will look like down the road.

For senior management the solution is simple, they have to make software a first order thing to manage and create an organisational entity that is made for taking care of software at the edge of their business. Further they must create a culture for learing and continious change.

By not doing so they will loose opportunities they never new they had, and in the worst case their organisation will face extinction.

How such entity could be created will be addressed next.

Digitalisation – Reinventing the toolmaker

We the humans have tried to strengthen our capabilities and simplify our living since we took our first steps on the planet by creating tools. Historically our tools has been simple, an knife an ax or a hammer. With the industrial revolution they began to take the form of powerful machines,  machines that was directly controlled or steered by us.

Automation made it possible to delegate some of the required control to a computer, or to the «logical resolver» as some might like to call it. With the physical downsizing of digital computers and their  dramatically drop in cost, any machine can now be equipped with a computer enabling us to make smarter machines. To instruct these machines to do anything useful we have to develop software programs, something we have been striving to do for more than five decades.

The term «Internet of Things» mirrors the capability to equip smaller and smaller machines with more and more computing power, creating  machines that can be connected into clusters or networks of interconnected machines.

While the Internet of Things movement has focused on the physical things, we must also understand that there are even more machines without any physical form, the software robots.

(Be aware that our digital world diverse from Tolkien´s world. While Sauron could not excess his final power without physical form, requiring access to the ring of power, the formless robots of cyber space has no such limitation).

These logical machine´s or computer programs exists in the cyberspace and do a lot of work such as buying and selling stocks, oil cargos and optimising value chains. They perform the tasks they are programmed to do, including learning and adapting to a changing environment.

With digitalisation, firms will compete not only by providing the better physical machines, but also on their ability to develop and deploy logical machines, automating / actively supporting their processes, it be designing an oil well or diagnosing a cancer patient.

This is also the reason digitalisation will drive radical change to how we thing about tool making. Traditionally, disciplines, being a civil engineer, geologist or medical doctor have been driven by the possibility to perform their trade directly and they have developed the tools needed for such direct involvement, it be the scalpel or the screw driver.

With digitalisation, firms, it be hospitals, oil companies or car manufacturers will need to use their best experts not only to perform their trade directly, but also to involve them in the development and deployment of digital assets or assistants that can perform parts, if not all of their work, or directly support a more junior practician.

This represents a major shit in how we think of professions and professionals. Firstly, disciplines are forced to become tool makers. Secondly professionals are forced to work with other professionals creating tools. For many professionals this imply that they must spend time with programmers creating the software that captures their insights and knowledge so execution can be left to a computer.

The benefits from this change is that suddenly the expert is available 24 hours a day, He or she is not tired any more and their knowledge become institutionalised and available for other more junior practicians.

The downside is that professions need to change their way of thinking about their profession. They might also need to change behaviour and culture. They need to think of themselves as toolmakers, not only tool users.

Digitalisation- Outcome based services

One of the main takeaways from Industry of Things world in Berlin, September 2015  was the importance of outcome based contracts. The most used example of what this mean is found in the airline industry. Manufacturers such as General Electric and Rolls Royce have stopped selling aircraft engines to the airliners. They sell «thrust» and «hours in the air».

Their value proposition to the airliners are to make sure that the aircrafts can fly, and to reduce the total cost of ownership related to engine maintenance and repair. In the old days airliners bought engines, and then bought the same engines once again through a service agreement. In addition they all needed a large staff of mechanics to look after engines. By leasing engines on outcome based contracts all the practicalities are left to the vendors.

In capital intensive industries moving toward leasing models reduces risk and the need for upfront investment before a dollar is earned. With an outcome based leasing model the payment of machinery follows the income stream generated from using the actual machine. In this environment the manufacturer can optimise their products according to functional needs, not technical specifications. Customers choose the vendor with the best price / performance for the needed function.

It is digitalisation that make outcome based contracts feasible for more and more industries. When the manufacturer can monitor equipment performance and integrity in real-time, they can take more responsibility for when to service it, or even replace it before it breaks. Through standardisation and operational insight manufacturers can improve products and reduce cost.

Even conservative industries like oil and gas will not be spared from these effects. One example is drilling rigs. Today these are rented on daily rates, the poorer they perform the more the owner earns. With enhanced digitalisation it become possible for the rig contractor to offer outcome based contracts, where they are paid according to end product quality, avoidance of rework (technical side tracks) and undesired events (kicks). Digitalisation makes rig operations transparent for all involved actors. The key question to ask is what competence is needed to provide drilling as an outcome based service, and who is the end customer for such service.

Another example is subsea production. Today these facilities are acquired and installed by operating companies. In the future the manufacturer might take the responsibility for installing and operating the facility. In the end the traditional oil company leases the machinery to drain the reservoir. Again, the emerging questions are what does this change do with the existing company. What is the value proposition and what are the competencies required?

We can go from sector to sector and find similar situations. Digitalisation will have a disruptive effects on how value chains are organised, and one of the main drivers or enablers is the move toward outcome based contracts, moving us from CAPEX to OPEX.

Digitalisation – Building the needed capabilities

Some 20 years ago was I called to help two researchers, a physicist and a medical doctor with a poorly written C++ computer program at their hands. As the three of us sat and peered on the code the medic said: «You know, what happens now is rare phenomena here in Norway. Here we are, three professions working on a common problem as a team of equals».

That day I learnt a few things about tumours and magnetic tomography, they a few things on how to write faster and better C++ programs, but all of us learnt that solving a difficult problem benefits from a multi disciplined team.

Yesterday I wrote that digitalisation requires organisations to adopt Internet thinking and begin developing their digital capabilities and asset. The first capability they need to leverage is multi disciplined teams. In a true multi disciplined team all participants acknowledge the other disciplines and respect their uniqueness. It´s not about a customer / vendor or master / servant relationships,  its about peers working to solve a common problem.

Creating multi disciplined teams is hard, because most businesses are built around what is perceived as the most valuable discipline. Within healthcare and hospitals the medical doctor is at the top of the food chain. In oil companies geologists and geophysicists are the ones that rules the exploration department. Drillers and drilling engineers the drilling department and so on. In these type of cultures the final decision power is allocated to the what is perceived as the «leading» discipline. To succeed with digitalisation this kind of culture must come to and end.

Why must it come to and end, you now might ask? The answer to that is that digitalisation implies that the function or role of disciplines are changed. Digitalisation forces disciplines to create tools or services that make their skills and insights available in new ways. The geophysicist is not interpreting seismic images any more, their skills are needed to create software that interpret images. The same happens with any other discipline, it be medical doctor or a civil engineer.

With the multi disciplined team in place, whats next? Sustainable funding, as teams and digital assets eat money for breakfast. Compared with physical assets, that have a lifespan measured in years, a digital asset (in terms of software) need continuous investments to stay healthy and competitive. You are very much in a climate where «Who dares win» to quote David Sterling, the founder of the British SAS forces.

The practical consequence of this is less upfront work on formal business cases, but an adaptive learning approach where we want to fail early to make sure that good ideas are separated from the bad ones before the cost of failure skyrockets.

The third ingredient needed is adequate digital (software engineering) competence. You need software developers and software engineers. Here most companies faces another challenge, the good ones are rare. The one you need are the ones who can envision new capabilities and put in place the process, tools and team to make it fly.

Finally, what does Internet thinking mean? It boils down to a culture of sharing. Open source software. Create a community to solve the problem. This is how you scale the multi disciplinary team globally. The Internet is built on the simple belief that sharing is the fastest and most efficient way to create business. In this respect open source software does not mean free software, it mean that monetisation is done by other means than sale of licenses.

So summarised digitalisation depends on three things, multi disciplined teams, adequate funding, software engineering skills all wrapped into a sharing culture in line with the spirit of the Internet.

Good luck with your digitalisation journey.