Transcript
Brisals: I think most of you should recognize this movie or picture. Smiles all around. This is from, Mamma Mia! Here We Go Again. I think it's a mashup scene after the credits. This movie, I like a lot, of course, musical, so much color, vibrant colors, locations, sun, water, the star cast, so many things to entertain us. If you think of moviemaking, though, it has so many things to go through. Someone needs to come up with a story, write a script, find the producer, bring a director on board, find the stars, location, costume, so on and so forth.
It's a complicated process. Complexity is everywhere, not just moviemaking. Even conducting QCon is complicated. How they tackle the complexity and bring this experience to us is the success of it. It's in life, it's in software engineering as well. It is a fact. At the same time, when you talk to your team or engineers, you often see the fear, because they don't want to tackle complexity. They'll say, why don't you give it to that guy? Or, she knows everything, that's the best person. That is the way things are even in software engineering. My name is Sheen Brisals. I'm an AWS Serverless Hero. Recently, I wrote a book, "Serverless Development on AWS," with my colleague.
Complexity
Going back to filmmaking, it looks, everything is simple to us, when we watch a movie. They package everything together. We can call it a monolith, or whatever way we can call it. I see the way they present is a nicely done event driven architecture. Thing is, it's not just one big blob, though, there is an introduction. In many movies, especially the movies from India, they always introduce an interval. They build up the story, they hype up the interval, and leave you hanging on the suspense. Then there is the credits. Already a movie has been broken into few parts, we don't recognize, but it happens. Then, within each part or each interval, there are hundreds of scenes, simple scenes, too complicated scenes, all knitted together or edited together to bring us the entire experience.
The complexity is everywhere, but the mentality or the mindset, how we tackle is important. There's a great dialogue in that movie between a man and a woman, in the Mamma Mia movie. The man is always pessimistic. He says, everything is complicated. Life is so complicated, nothing is simple. The woman, though, more optimistic. She gives the right answer that everything can be simple if you break it down. If any of you read the book, "The Philosophy of Software Design," the author makes this statement. He says that the fundamental problem in engineering is problem decomposition, how we divide a problem into pieces.
This is so true everywhere. Say for example, if it comes to the film, we have a vision, the whole thing. Then we break it into different parts so we can focus. You know the saying, see the forest first before you look at the trees, exactly the same. Have the whole vision in your mind, then approach it in parts, focusing each bit. Usually, I like analogies. I usually tell this analogy. Say, you watch a night sky. It's a blanket of dots. That's it. No matter how many times you look at, you still get the same picture. Get a telescope, zoom in into one orange dot, what you see is like a galaxy.
Still, first see a blur. You keep going, you find suns and star patterns behind that galaxy. You keep going to find the planet. At some point, you will see the cloud formation and the landscape, not the AWS CloudFormation, but the nature cloud formation and the landscape. If you're lucky, keep going, somewhere you will find QEII, and in this room, you and me somewhere. This is the way engineers should approach when they are given a complex problem. They need to know how to make an entry into the problem, and then keep going.
Set Pieces
What is a set piece? I've taken it from the moviemaking. It also applies to theater, music, and other industries as well. Usually what happens is, when they plan a movie, they identify areas of the movie, they call set pieces, they are like standout. Say, for example, maybe a car chase, or a loud sequence, or a long drive, and things like that. They identify these parts of the movie, and they can then plan and film accordingly. They can do the rehearsal, similar to what we do, testing. They can also do these things in parallel in different locations, they can happen. This is the concept behind set pieces. Why does it matter? Because it has certain characteristics that we can apply to engineering, for example. It's part of the whole vision. That is like, when we have a big use case, this is part of it.
Then, you focus on something that you can manage. Then, you can plan, you can rehearse or test. Then, finally, you bring everything together to make the whole. You may be thinking, that's fine. Why in serverless? How do you apply this in serverless? To be fair, it's not specific to serverless. We can apply it to many things. However, there are reasons why this we can apply in a better way to serverless. I will go through three things. First thing is because the characteristics of serverless technology allows us to do that. Secondly, we can utilize the proven industry patterns and practices that everyone is familiar with. Finally, you can think of sustaining our application that we built. I will get to that later on, in the way I interpret sustainability when we build applications.
Serverless Characteristics
Let's look at serverless characteristics. Who is working with serverless or any familiarity? There are two ways I classify. First, I take the characteristics. These are the common things you hear from people, they often debate, and all sorts of things happen on the social media. It's a cloud computing model. It's part of the cloud setup. There is no server management. Then we pay for compute and storage, autoscales, high availability. These things matter when we architect for serverless because there are certain things taken care of by the provider for us. That means, we should be able to architect with that in mind. You don't need to unnecessarily add on scaling aspects or high availability, things like that.
For me, the benefits of serverless are crucial. Because of the way the serverless ecosystem is, we can optimize things at a granular level. For example, if you have a table, you can optimize that table different to another table. Depending on the use cases and scenarios, we have that flexibility. Due to that, we can go deeper in terms of the security and protecting the data and making privacy aspects and things. It is an ecosystem of managed services. This is like, where people who are new to serverless often confuse, they think that it's everything to do with Function as a Service or lambda functions, but isn't. They're just part of the whole system. You have so many managed services, they provide us the capabilities for us to build applications.
Then, an important aspect is the ability to iteratively and incrementally develop our applications. This, again, due to the nature of serverless characteristics, and also the way the ecosystem allows us. Most importantly, it brings in diversity into a team. I stress this a lot because there is always a consensus that a team is basically a few engineers doing programming. In the past, it used to be the case. You come in as a programming expert of a particular language. Serverless changed that mindset, because programming is part of it.
You need to know how to knit the services together, that means you need to know the infrastructure as code aspect. You need to know how to provision a database table. It could be a SQL or a NoSQL datastore. You need to know how to manage with queues, and how to set up your API authentication. There is no security expert sitting there helping you. It's all part of the engineer's day-to-day job. That's why it brings the diversity aspect into a team with different skills.
I mentioned the granularity and individuality. If you're someone new, so here's a sample small architecture piece from a simple setup. For example, I highlighted these three functions. If you look at the callouts, each function has different resource setup or characteristics. Their memory allocation is different. The timeouts, how long each can run, they're all different. When you have different tables, you can say you want to have on-demand scaling for one table, whereas you take control of the scaling for another table. Same with queues, you can decide how long in one queue, you want to retain the messages. Also, the polling interval, how fast you want to poll messages coming to the queue, or slow things down.
These are some of the aspects of serverless that we need to keep in mind when we architect application for serverless. The other thing is optimization at depth. For example, three data flow pipelines here. Say some data gets dropped into the source, it goes through the pipeline. The top one you see price changes data, and the bottom, product reviews, and the middle could be product description, updates, and things like that. Price changes, if you think of it, it's highly important data, so you want that data flowing as fast as possible.
Whereas product reviews, if I put a review, I should be ok that if it didn't appear for a day or two or even for a week. That means in this architecture, you can adjust the resources that you consume and architect so that you gain the cost or you gain sustainability. Whereas as the top one, you add more power to the resources so they perform in a better way. Let's quickly recap what we just mentioned. Think of the managed services aspect, the ephemeral characteristics of services, and then the granularity and the optimization options that we have with serverless, and the security, and the data privacy aspects.
Domain-Driven Design and Microservices
Let's look at domain-driven design and microservices. Who has read this blue book, "Domain-Driven Design?" I often say that what Eric Evans, author of this book, has given us is not a book on domain-driven design. He has given us an industry on domain-driven design, because at least a dozen or so books have been written to make us understand what he has written in DDD. There are conferences dedicated for DDD, people pay hundreds of dollars to attend. Then, in-person online courses. Why? Because there is value in the book, what he has given us. Now, domains, one of the aspects of domain-driven design was up until DDD, we've been building software forever.
We've been pulling software based on the functionality, or making libraries, modules, layers, TAs, all sorts of things, purely from the technology point of view. DDD came along, smashed everything, and he said, think organizational way, think organizations. Split your organization into domains, and think that way. Then you bring in the subdomains, break it down so then you have more visibility, more control. That happened. Most importantly, this is the best thing I like about DDD, the boundaries or the bounded context. This is so crucial for a successful serverless team or organization developing with serverless technologies. Because, of course, it matters to even other technologies, especially in a team environment, guarding boundaries is the most crucial and critical aspect.
When we get here, something else takes over, team topologies. It talks about the structure of different teams, stream-aligned teams, platform teams, these teams. If you focus stream-aligned teams, for example, so we have a boundary, we can now bring in a team to guard that boundary. They are the custodians of the bounded context. See how it's going. You break down the organization into domains and subdomains, and you identify the boundary, where according to DDD, ubiquitous language, the common language gets spoken, and it has a domain model. You have the responsibility as a team to protect the domain model. Any guesses who takes over from here? Microservices, obviously. Because now the team can build microservices and applications that reside within their boundaries. Of course, we will see how they interact later on.
This is why I said it's important whether we use serverless, or not, to capitalize on the proven practices and patterns in the industry as they evolve, to make use and get benefits. For example, DDD came in like 20-odd years ago. Microservices came later. Team topologies, kind of the recent one. The recent as few years ago. We can still bring everything together and work in harmony to make things happen. Just a quick recap as we focus on the serverless way of architecting. Domains, autonomy of teams, and the boundaries, and the independent microservices, and the contracts. Thoughts on these things should be in the mind of everyone who architects serverless applications. It's not necessarily the architects, anybody who architects serverless applications should think of this.
Sustain Your Applications
Let's move on to the next one, sustain your applications. If I asked you around to define sustainability, many of you will say the carbon footprint, or green initiative, or the planet. All fine. Sustainability as a definition is very generic. It's something that we keep it going with a little bit of nourishment or nutrition, so that it just doesn't die off. This is exactly the principle that we apply when it comes to our planet. We want us to keep going for our future generations. How does it relate to serverless, or software engineering, for example.
Let's go back to the old way of waterfall model. I'm sure many of you must have come across. Typically, it kicks off with the requirements, and then it keeps going with the different siloed phases, often taking weeks, or months, or maybe years to complete. The thing is, those days, if you've been through, when it comes to release phase, you just pushed out, you cut the CD or packed the software and send out. Then it gets pushed into some maintenance mode. Because only few fixes and enhancement happen after the release of a software. Those days, the maintenance teams are the most demotivated teams you can find in an organization, because they always get the backseat, and all the shiny microservices happen in here, the shiny team.
Let's think differently when it comes to serverless. What I call sustaining, because you start with an idea, you design, build, operate. Operate, in a sense, you deploy to cloud and look after it. You sustain because you started with MVP. It's not finished yet, you still need to keep it going. That's why I put another MVP on top to say that you start with a minimum viable product, but your goal is to make it most valuable product. For that, you need iteration. You need to iterate. When you do that, what you're doing is you are sustaining your product. That is the different meaning of sustainability coming here.
If you think of cloud. Cloud is basically formed of three things, if you leave the buildings and all sorts of things. Compute, and then you have storage, and network. Simple. The interplay of these three, all the services and things that we consume. The serverless, cloud is already in the picture, because it's part of the cloud. What we do with serverless development, we use cloud, and we build products using serverless technologies, using the processes to allow us to operate in the cloud successfully. What I call as a sustainability triangle in serverless.
Basically, we have the products, we have processes, and the cloud, and they form a sustainability triangle. The products, as I mentioned, these are the products, can sustain, can live longer. Processes are the processes that enhances to introduce this in a sustainable way of doing things and also operate sustainably in the cloud. We'll see what that means. When I was first talking about this one in a conference, and afterwards, someone said, you haven't mentioned what a sustainable product is. Then I started adding this slide. A sustainable product to me, it can mean many things.
For me, crucially, three important aspects, modularity, extensibility, and observability. These three things are crucial. I think they are also dependent. For example, if you have a modular product, it's highly likely that it can be extended. Then, if we have the better visibility of what's happening in your modular service, then you are in a better position to sustain it longer, because I always say that observing the things in a pond is way easier than the ocean. That's the mindset that we need to bring in, when we work with the serverless development or the services that we build.
Sustainable processes could be many things. This is where us coming in, as the mindset or the people or the developers or engineers, using the processes, and the cloud as their operating platform to gain us the advantage when it comes to sustaining the products and operating them sustainably. There are again, three different ways of looking at things, so our involvement is always there. When it comes to architecting, these aspects should be in mind, because the operating environment is cloud and how we operate. That's where the cloud aspect comes in. Some of the processes, for example, that enhance sustainability, lean principles.
Some of you must be familiar with lean principles as basically reducing waste. The value stream map is something very simple. Say, for example, you build a wonderful CI/CD pipeline. Push a button, goes through anywhere. If we have a process, it's like a QA check gate, that waits there for a day or two to get to the next stage of production, you're wasting a day or two there. That's where the lean principles say, you reduce waste. That's the aspect of the value stream. Then, being pragmatic with the iterative or agile development, starting something small, like I mentioned, MVP mindset and taking it forward.
This is the typical agile cycle. Then automation, I already mentioned, having the DevOps mindset, and continuous refactoring. I usually do a separate talk on continuous refactoring, and why we should do that. Because with modern technologies, things are changing every day. Cloud providers are releasing services and features day after day. That means we can't stand still after building an application, so we should be in a position to continuously evaluate and refactor, and to make things better for the future. That is one of the aspects of modern software engineering. Don't complain, like in the olden days, refactoring means legacy, you need to rewrite. No, we are enhancing or sustaining as we go. Growing serverless teams, sustaining teams is crucial.
The graphics is from my talk on how to grow serverless engineers. I think Gregor mentioned about the architect title, and not having the title. Something I always recommend engineers is to architect the solutions. For a long time, I stopped feeding, here's the architecture, go and build now. I sit with them, build the architecture. They say, "Sheen, I don't know how to do this." I say, "Study which part you don't know, go and find out." Then between me or other lead engineers, or whoever knows the architecture aspects, they evolve the architecture.
This is very important, especially in that serverless landscape. Also, in many organizations, you won't find server typical architects. Of course, you need to have this vision, the overall architecture, but we've gone away from the days of having a big A3 sheet with crisscrossing lines and boxes all over the place. We start with some logical view of top, then we identify the different parts to keep us going forward. Sustainability in the cloud is a shared one. As you work with a cloud provider, they come with certain sustainability aspects. This relates closely to our environment, for example.
They take care of green energy or powering their data centers accordingly. Then as customers or consumers, we have responsibilities, how we architect our solutions to gain the benefits of sustainability and have the contribution going via the provider to the wider world. This is again, an important aspect of architecting serverless applications. A quick recap on what we just covered. Thinking applications like the modular and extendable is crucial. Identify the processes that works for us. Then, have this shared responsibility understanding in mind.
Set Piece in Practice
Let's put everything in practice. It's not a complicated architecture, it's a simple one just to highlight some of the things we saw earlier. Here's a small reward system, very common. You go to an e-commerce website, you have rewards, vouchers, or codes you want to redeem. The website works with a content management system to load the reward data, for example. It has a backend service, typically, to validate the code and make the redemption. Then there may be a third-party application where some of the data gets stored as a ledger.
Let's say those two are third parties, we don't focus too much, and say, our domain, e-commerce, could be different in your cases. Let's for argument's sake, subdomain is customer, and we have a bounded context that's important: rewards. That's where the architecture diagram comes in. Traditional microservices approach usually considers one bounded context and big monolithic microservice. This has been the case, in many cases, because of the containerization, because everything gets built one. You come up with a monolith, microservice, a single pipeline, everything is fine.
With all the characteristics of serverless that we saw earlier, we can think in a different way. With the traditional versus microservices, when it comes to this scale, often we need to think of, if a particular piece of the application or service changes a lot, say, for example, reward redemption, business logic changes often because business rules change, why do we need to deploy the entire thing every time if it's just small, one part changing? Similarly, how do we have the deeper visibility of the operation of the entire thing, if there are critical parts, for example, the API serving the customer. That kind of critical. Again, another thing is the anti-corruption layer, the ACL.
Where do we fit that in? It's sad that many engineers and modern engineers don't know anti-corruption layer. I was talking to a few engineers a couple of weeks ago. We were discussing integrating with a third party or a different domain model. They came up with a solution, I said, we need to think of anti-corruption. They went completely silence. What is anti-corruption? No idea. These practices are very important, so that your domain model within your bounded context stays pure and perfect. It's not getting polluted or corrupted by the external things, and the developer experience. If there are four or five engineers in a team, how do we make sure they're not stepping each other's toes sharing the same code base, one service. They're all trying to deploy at the same time.
This is where probably we can bring in some of the thinking of identifying the pieces. For example, let's leave few of these things out and the external services, and also I'm going to leave out the website. Then, how can we look for areas that we can decouple and build as separate pieces. Say, for example, find the core services. In here, the backend service, whichever that is going to be, that is a core service that can stand on its own. Then, identifying the data flows. There are the data going from one system to the other, they can be clearly separated in many cases.
Identify those areas, so they can be developed, separate microservices, and have the different interaction patterns with others in the system. That is one way of looking at. Then, the anti-corruption layer. These are the protective measures to guard your domain model, how can we identify say, for example, content management system. If data model different to what the rewards bounded context model is, so that means, stop that. Do the transformation, translation, and push the data so that if you replay CMS or even the CRM, for example, you don't need to do too much to make the changes within the core model, because you have these layers taking care of all those aspects.
How do we piece these things together? We have a bounded context, and then we said, let's put some microservices. These are all smaller microservices, and they all connect each other. The lines here don't have any directions now, so that we will add soon. This is boxes and lines, the typical Gregor's masterpiece, but they add more value behind the scene as we start looking at each piece and the line. How do they connect? This is where engineers, again, I've seen struggling, because they don't get the concept of decoupling things or uncoupling things. If you look back, the filmmaking process.
I said, made of hundreds of scenes and sequence of scenes, how do we bring things together? Mainly two things, dialogue, carries over from one scene to the next one. Then the other one, music, background. Because that's always there, intermingling everything. What do we have in the world of microservices? You know the answer, APIs, and events, messages. I know people argue about events and messages, it's the same, but let's keep it separate. We have communication and contract. This is the reason why it is still possible when you break these things into different pieces, the score system works beautifully as one application.
If we add these aspects, then we can redraw the diagram with arrows. You identify the synchronous API invocation paths. For example, the website to the reward service. It has to be a synchronous request-response. Then, you identify, where can I bring in asynchrony, or event driven nature? Because there is no need for an anti-corruption layer to be coupled with the rewards, because it's like near real time. That can be event driven. You may have an event bus or event broker carrying that, that what you gain out of it is the flexibility to change the modularity and the extendibility that I mentioned earlier on. This happens. Then also, in the async world, you can also have webhooks. They come in, and then you can deal with incoming data or events. These are some of the ways of thinking the architecture when we deal with serverless applications, and taking the advantage of its characteristics and the patterns.
Serverless Microservices Approach
Typically, in a serverless world, this is how it will look like. The important thing to notice here is, all these microservices, they are living inside your bounded context. They don't cross the boundaries. That's where those communication and contracts I mentioned, they come in. Then you can have independent deployment pipelines. They can go happily to production without impacting anything else. This is the way and the power of breaking things down and making it more manageable for everyone, engineers and architects, everyone to deal with. Let's bring everything, make the whole. We have the whole boundary. What we need is we need an autonomous team.
This team, however you call, is two-pizza, one-pizza, three-pizza, depending on where we are, how much we eat, so that's the team. They own the microservices within the bounded context. That's important, that's the ownership. Everything that happens is their responsibility. Though I showed three or four microservices, in reality, there will be more. You need microservices to deal with report generation or data generation. You need microservices that deals with emailing out customers and taking the feedback back and things like that. These are the areas that we can easily decouple. When we build our application, that means we don't need to start with all these things at the same time.
Email can come in later, or you can do the report generation once you have the clarity on what data this bounded context deals with. Then, of course, as I mentioned, it's an autonomous team, they operate in their cloud account. This is important. I think many organizations are still going through this phase, not many organizations achieve this. This is crucial for velocity and flow for the team. The diagram basically what it means, when I work with teams, some of the teams I work with, this is what they do. They have their own account, their own repository. They don't deal with anything. If you want to talk to their services, there is an API, or the event flows, or the event broker, or the common event bus, so that is what we aim to build and architect with serverless.
Recap
When we think of the architecture through the serverless lens, or architect for serverless, think of the characteristics. Make use of, take advantage of the serverless characteristics. Bring in the patterns. Don't be shy to introduce anti-corruption layer or microservices to engineers. Let them learn. Importantly, team autonomy. Challenge engineers to confront complexity. For example, a couple of months ago, there was an engineer, he took over a particular piece of new work. He was going to bring up with some architecture diagram. He had no clue how to tackle, so he started drawing APIs and things.
I was like, how do you know you need an API here? He said, no, that system has an API. That's that system. Why don't you start with something like domain storytelling. Then, you draw the picture as a storyboard. Domain storytelling is a book actually, you can follow. It's nice to have that thinking. Then you explain it to everyone, stakeholders. You see that is good for the feature or the service that you're building, then you can slowly think in terms of the design and architecture. These are the ways that we need to feed all the good patterns and practices to engineers to bring in.
Key Takeaway
I leave you with this one. Has anyone read this book, "Ikigai?" This is completely unrelated to software engineering. This is all about life, how to live longer. I was astonished to see this same thinking in this book. It talks about, if there is a big goal in front of us, it can be drowning us. He said, rituals, like you can relate to our processes that helps us to take simple steps and achieve the goal. It then carries on saying this thing, when you're confronted, break it down, attack each part one by one. I only read this "Ikigai" in the last 2 months or something like that. I've been talking about the set pieces even before. I was happy that what we are trying to tackle in software engineering, already people are talking about and helping each of us to tackle in life. Bring all the ideas in together, and then let's build a set piece architecture.
See more presentations with transcripts
