Serverless Hexagonal Architecture — Python
This is a follow-up article from my previous — Serverless Then & Now. There I mentioned how in Serverless Engineering team, we have adopted more of a Hexagonal Architecture for Serverless and how it helped us to managed the DDD traits within Serverless. We use Python & Typescript in our team, and in this article I speak about how we did it using Python, you will need few prerequisites to follow along:
- Understanding of Hexagonal / Ports & Adapters Architecture
- Python installation with pyenv — recommended (manage multiple python versions efficiently)
- Python Poetry — Best dependency management for Python as far as I know
- Basic knowledge of AWS Python Powertools — Optional
Serverless Hexagon Design
- In Adapters — These are the different input types through which the application can be accessed, these can be async(events) or sync(APIs) like in the above case [I feel In-Ports (Interfaces) are optional]
- Domain / Business Layer — Core of your application with the business logic that just deals with the functionality and not the technology
- Out Port —Interfaces to the External adapters, very important as they are the abstraction that is going to leave protect your business layer from changes when the external adapter itself changes
- Out Adapters — All the Adapters that the application needs to successfully execute the business operations
Introduction & Setup
I would want to take up a simple Email Sending application for this demo, that is capable of sending a email when invoked via Http-Endpoint (sync) or from a Eventbridge message (async). The application will be responsible for :
- Sending email based on received inputs — via a http client endpoint
- Save tracking information on dynamodb, publish a email sent event
- respond synchronously if its a http request
Create a python environment & poetry app
pyenv install 3.9.13 # latest python 3.9 (lambda supports 3.9)
pyenv local 3.9.13 # set local terminal python to installed 3.9.x
poetry new serverless-hexagon-demo
poetry env use 3.9.13 # creates a virtualenv assigned to workspaceWe will need various dependencies for our sample project, here is the final pyproject.toml with the dev and package dependencies, clearly outlined.
Note: I’m adding boto3 and aws-lambda-powertools as dev dependency as boto3 is available in the lambda runtime and powertools can be added later as a layer.
Project Structure
Handler / Entry-Point
Lambda handler is fairly simple for an api app. Here we use the Powertools REST event-handler
In Adapter (REST routes)
Adapter is the place for validating, transforming the incoming requests, in this case REST (headers, query_params, body), etc. into respective domain models. Same applies to other adapters like DynamoDB, S3, Event-Bridge, etc.
Also, another important aspect here is declaring the manager / domain-logic here, as a global variable here, so that the same attributes gets reused in warm instances. (while the manager can be instantiated outside of the def here, the global variable helps with the unit tests mocking purposes)
Specifics & Important Tenets of Hex
Domain Manager
The domain manager is the key component of the hex architecture, it is the logic layer and should understand only business model. It is called ‘manager’ for a reason, delegates all the low level calls like database queries, rest-api calls to the out adapters while being responsible for the overall outcome of the business service.
Constructor — Pay close attention to the constructor, all of the out-ports needed for the business operations within the domain manager are present in the constructor but mentioned as Optional. This is by design, and that's how the hex is able to be developed as a singular application but deployed as multiple lambdas finally. Also notice that only ports (interfaces) are used for representing the outbound services, but not the adapters, this is by design too
Manager Instantiation — Manager is instantiated at the In-Adapter level (look at the router). Manager is capable of many things based on the design, but it operates contextually in coherence with the In-Adapter. If the In-Adapter that's invoking the manager is used for a business flow that is supposed to persist data in dynamoDB, then the ddb out-adapter is injected into the manager at the onset of the In-Adapter. The ingested Out-Adapter can be either-or (or) all-at-once depending on the business use-case.
Port & Out-Adapter(s)
Ports are plain & simple, but important. Ports play an important role in insulating the domain manager from the out adapter. The Out-Adapter is the one that deals with technology elements. Ports ensures that the domain manager is insulated from any changes to the outgoing adapters. It maintains the binding between the domain manager & the outgoing adapter. It is important to the name the port —BaseWeatherService, BaseRepository,etc. & its binding method with business-context like — getWeatherInfo, saveOrder, etc. Say, we’re updating the underlying dynamodb out-adapter to a serverless atlas (MongoDB) in the future, the port is going to ensure that the Domain Manager (or) its corresponding Unit Tests don’t get impacted during that time.
Out-Adapters, are technology oriented. They are responsible for the executing the technical services like S3, dynamo-db, Rest-API. Out-Adapters understand domain models, transform them to respective technical model, execute the operation and again return domain models back to the domain manager. On top of this, they also manage the lifecycle of the respective technical component involved. For, ex. A Rest-API adapter manages the session, client-certificate, auth-mechanism, etc. A relational database manages, the credentials, connection-pooling, cached-connections, etc. The Adapters should be leak-proof. Like the name suggests, the adapters should be acting as plug-n-play component within the hex.
Unit Testing Hex
Unit Testing the hex, will happen at 2 levels. They can be written in pytest
Unit Testing the In-Adapter → Domain Logic: Unit tests will have to be written for every type of lambda-handler within the hex. For this part of the Unit Test, the out adapters will have to be mocked. Since the out-adapters are guarded by a port, this becomes easy. We can write a bunch of Fake Services for the dynamo-db & the rest-api out adapters and test them by passing them at the handler level. Fake Adapters implement the port interfaces and return mock responses .
Unit Testing Out-Adapters:
- For Unit testing non AWS adapters, like
requestswe can use pytest-mock, here is a good blog I followed - AWS Service adapters are invoked via boto3 from within the lambda, these can be unit tested by the help of botocore-stubs, here is a good blog with examples
Packaging & Deploy
Most of the packaging & deploy logic needed for hexagonal lambdas are covered in my previous post.
Poetry & Layers helps a lot with the hex packaging, Since hex is a modular monolith, packaging all dependencies for various hex deployments is not the right approach. We can use respective layers, based on the lambda’s functionality. Most Serverless frameworks, AWS SAM, Serverless have good support for custom layers and including layers to lambdas. For this hex app, I had used AWS SAM. Also, Klayers is a cool choice, when you need any popular custom python package like NumPy, requests, etc.
Complete source code is available in this github repo. Thanks to Surendra keerthipati for helping with this hex demo.
Conclusion / TL;DR
- Hexagonal patterns are better suited for building Enterprise Serverless applications than Single Purpose functions, and helps a lot with teams following DDD. (Single purpose functions gets cumbersome with scale)
- Modern Python — Poetry for packaging, Typing with MyPy and the Powertools Python library from AWS makes Python a cool choice for building Serverless Lambdas in Hex pattern
- Teams following the Hexagonal design, will have to invest time in custom tooling and also extend libraries like Powertools for Python to improve their respect hex development lifecycle
