Story-Driven Service Design
From Feature Request to Minimum Viable API Product
Goal-oriented API design (“contract-first”) is a recommended practice in the Web API community; data modeling is equally important to balance API granularity and client-provider coupling. Finally, refactoring to API patterns has been researched recently. These three complementary practices form a mini-method whose steps are partially supported by API design tools.
Prerequisites: The instructions below assume that you have the MDSL Tools Version 5.4 (or higher) installed. Alternatively, an experimental Web Version of the MDSL Tools can be found online. [1]
Step 1: Write Integration Story
We start with a feature request, expressed as user story. The MDSL syntax for stories is similar to the agile format also supported in the Context Mapper Language (CML) for domain-driven design:
API description ReferenceManagementSystemscenario PaperPublishing
story PaperArchiving
a "Researcher"
wants to "create" a "PaperItem"
with "title" with "authors" with "venue" in a "PaperCollection"
so that "other researchers can find the referenced paper easily."
Rationale: “APIs should get to the POINT”, and starting with a story makes them purposeful. The linked blog post also lists some of the many books and blogs supporting this claim (under “What do other API designers recommend?”). Note that several variations of the story format exist, but a role-action-benefit triple is quite common.
Step 2: Transform Story to Endpoint Type Exposing Operation(s)
For now, let us assume that we can realize the story with a single API operation. Later on, we will decompose this operation and complement it with supporting ones.
The scenario-level transformation quick fix “Derive endpoint type supporting this scenario” in the MDSL plugin (or, in the MDSL Web Tools, the refactoring option “Add Endpoint with Operations for First Scenario and its First Story”, available on the second tab) analyzes the story and then adds the following candidate endpoint to the API description model:
endpoint type PaperPublishingRealizationEndpoint
supports scenario PaperPublishing
exposes
operation create with responsibility STATE_CREATION_OPERATION
expecting payload {
"paperItem":D,
"title":D, "authors":D, "venue":D,
"paperCollection":D
}
delivering payload "data":D<string>The story action was converted into an initial operation in the endpoint type; the objects from the story definition are available in the request message. The response message is just a stub that we will refine in the next step. Its data types are placeholders, for instance D (which is short for some untyped Data). In the MDSL documentation, the endpoint type syntax is explained in detail under “Service Endpoint Contracts in MDSL”.
Rationale: Adding this initial operation to the endpoint type (also known as service contract) ensures that all that we know so far is transferred from analysis to design. The operation serves as a starting point for the further design and implementation work required to realize the story.
Next, we can add operations via Microservice API Patterns decorators. Two endpoint-level transformation quick fixes are in action here: “Decorate as Information Holder Resource” assigns an architectural role to the endpoint (under “serves as”) and “Add operations common/typical for role stereotypes” provides two retrieval operations. The result is:
data type PaperPublishingRealizationEndpointDTO "paperPublishingRealizationEndpoint":Dendpoint type PaperPublishingRealizationEndpoint supports scenario PaperPublishing
serves as INFORMATION_HOLDER_RESOURCE
exposes
operation create with responsibility STATE_CREATION_OPERATION
expecting payload {
"paperItem":D,
"title":D, "authors":D, "venue":D,
"paperCollection":D}
delivering payload {"data":D<string>}
operation findAll with responsibility RETRIEVAL_OPERATION
expecting payload "query":{"queryFilter":MD<string>*}
delivering payload
"result":{"responseDTO":PaperPublishingRealizationEndpointDTO}*
operation findById with responsibility RETRIEVAL_OPERATION
expecting payload "resourceId":ID<int>
delivering payload
"responseDTO":PaperPublishingRealizationEndpointDTO
Note that STATE_CREATION_OPERATION and RETRIEVAL_OPERATION are two Microservice API Patterns (MAP) responsibility decorators, indicating the behavior of the operation (here: 1x write and 2x read access to API provider-side state). All available MAP decorators are listed in the “MDSL Grammar Quick Reference”.
Rationale: Making MDSL and the design transformations featured in this post aware of patterns speeds up the design work and removes technical risk (as a proven solution to a recurring problem is chosen and semi-automatically applied).
Step 3: Refine Data Types
The generated PaperPublishingRealizationEndpointDTO looks a bit odd and incomplete still. Let’s apply two transformation quick fixes to the data type definition to improve the data contract, “Add string as type” and “Include atomic parameter in key-value map” [2]:
data type PaperPublishingRealizationEndpointDTO
"mapOfPaperPublishingRealizationEndpointData":{
"key":ID<string>,
"paperPublishingRealizationEndpointData":D<string>}In the above snippet, the identifier of the generated DTO type structure has been changed to mapOfPaperPublishingRealizationEndpointData manually (this can be done via an “Rename Element” refactoring, provided by the underlying modeling frameworks EMF and Xtext).
Rationale: Key-value pairs are pretty common in programming languages and elsewhere (even in real life). Here, the "key" is specified to have an ID Identifier role, and the value is a plain string. This value can be structured too (similar to JSON object nesting).
The MDSL data modeling concepts are quite close to those in JSON and Jolie (curly braces!). “MDSL Data Contracts” is the reference page in the MDSL documentation.
Step 4: Refactor Operations to Improve Quality Properties
When large results sets are returned, the Pagination pattern is often applied to balance comprehensive information needs with network- and processing-efficient message sizes. Pagination is not only described as a pattern in MAP but also as an API refactoring called Introduce Pagination. The refactoring steps are supported in three MDSL transformation quick fixes that correspond to pattern variants. Apply “Introduce Offset-Based Pagination” to findAll in the endpoint (that was created in the previous step) to change the operation interface to:
operation findAll with responsibility RETRIEVAL_OPERATION
expecting payload "query":{"queryFilter":MD<string>*,
"limit":MD<int>, "offset":MD<int>}
delivering payload <<Pagination>> "result":{
"responseDTO":PaperPublishingRealizationEndpointDTO,
"offset-out":MD<int>, "limit-out":MD<int>,
"size":MD<int>, "self":L<string>, "next":L<string>}*Note the <<Pagination>> decorator and the additional parameters such as "limit".
When the amount of message exchanges should be reduced and larger messages are not an issue, you also may want to consider the Bundle Requests refactoring, typically applied to write operations. The MDSL Tools provide two separate transformations to refactor the request and response messages, “Bundle requests” and “Bundle responses”. The create operation originating from the story defined in Step 1 can serve as an example, realizing a bulk/batch upload here:
operation create with responsibility STATE_CREATION_OPERATION
expecting payload <<Request_Bundle>> {
{"paperItem":D, "title":D, "authors":D, "venue":D,
"paperCollection":D} }+
delivering payload <<Response_Bundle>> { {"data":D<string>} }+A third example of a quality refactoring is Add Wish List. Wish List is pattern to reduce the amount of response data returned (see here). We may want to apply it to the third operation in the endpoint, findById.
To do so, we first wrap the atomic request parameter and the type reference in the response message in a parameter tree (this can also be done automatically by the next transformation):
operation findById with responsibility RETRIEVAL_OPERATION
expecting payload "resourceIdWrapper":{"resourceId":ID<int>}
delivering payload "responseDTOWrapper":
{"responseDTO":PaperPublishingRealizationEndpointDTO}Now we can run the “Add Wish List” transformation on the operation to yield the following operation interface (note: the generated top-level message identifier was removed):
operation findById with responsibility RETRIEVAL_OPERATION
expecting payload {"resourceId":ID<int>,
<<Wish_List>> "desiredElements":MD<string>*}
delivering payload
{"responseDTO":PaperPublishingRealizationEndpointDTO}Outlook: Another twenty-something refactorings that match those in the refactoring catalog are available as well, and documented in the “MDSL Transformations” reference.
Step 5: Progress From Abstract Endpoint Types to HTTP Resources
MDSL is an intermediate, technology-neutral API contract language. But what we are really looking for is API Descriptions for popular integration technologies such as HTTP resource APIs.
HTTP resource APIs must respect the REST constraint “uniform interface”, which means that it is not possible to define the operations of a resource, identified by a URI, freely; only GET, POST, PUT, and in most environments, PATCH and DELETE are available. This constraint implies that an abstract endpoint that exposes more than five operations and/or multiple getters, cannot be translated to an HTTP resource API specification in a a straightforward way (this stands in contrast to APIs realized with gRPC Protocol Buffers, GraphQL, and so on). An explicit binding is required — sometimes more than one, as the relation from MDSL endpoint type to HTTP resource can be n:m.
In the MDSL Tools, multiple transformation quick fixes are available and in this transition; some manual work is required as well in this example. Let’s first create a binding (with the “Provide HTTP binding” quick fix on the endpoint) and then fix the reported mapping/binding problems , with “Move operation binding to new resource with URI template for PATH parameter” . We can also change the binding of the create operation from PUT to POST. The result of these three actions is:
API provider PaperPublishingRealizationEndpointProvider offers PaperPublishingRealizationEndpoint
at endpoint location "http://localhost:8080"
via protocol HTTP binding
resource PaperPublishingRealizationEndpointHome
at "/paperPublishingRealizationEndpointHome"
operation create to POST
element "paperItem" realized as BODY parameter
element "title" realized as BODY parameter
element "authors" realized as BODY parameter
element "venue" realized as BODY parameter
element "paperCollection" realized as BODY parameter
element "requestedPage" realized as BODY parameter
element "requestedPageSize" realized as BODY parameter
element "paperCollection" realized as BODY parameter
operation findAll to GET element "queryFilter" realized as QUERY parameter
resource PaperPublishingRealizationEndpointHome_findById
at "/paperPublishingRealizationEndpointHome/{resourceId}"
operation findById to GET element "resourceId" realized as PATH parameterMDSL language reference: “Protocol Bindings” page and “Support for HTTP Resource APIs”.
Step 6: Turn MDSL into OpenAPI and other Platform-Specific Interfaces
The MDSL that we yield when running through Steps 1 to 5 is available for download here.
Step 6a: OpenAPI for HTTP Resource APIs
In the MDSL Tools, a context menu option allows generating various target formats (select “MDSL” and then “Generate …”). [3]
In the Web-based Swagger editor, the OAS generated from the endpoint type and HTTP binding from the previous steps look like this (you may want to copy-paste it there and navigate around):
The paginated findAll operation appears as an HTTP GET method now. Metadata elements such as limit and offset control the pagination (as explained in the Pagination pattern):
Other target Interface Definition Languages (IDLs) such as gRPC Protocol Buffers, GraphQL schema and Jolie are supported as well (see here).
Rationale: With respect to the POINT principles for API design, the MDSL contracts with their various bindings promote a technology-neutral but still style-oriented approach (REST is one such integration architectural style).
The “MDSL Tools: Users Guide” explains the performed mapping/conversion in detail, and also show what to do with the generated specifications. See for instance “OpenAPI Specification Generator” [4].
Step 6b (optional): AsyncAPI (via AsyncMDSL)
AsyncAPI is supposed to do what OpenAPI does, but for APIs provided by messaging systems and their applications. There is an MDSL extension supporting such channel/queue interface definition on the abstract level; similar to core MDSL, it provides (integration) patterns decorators. It is called AsyncMDSL.
AsyncMDSL can be created from scratch — or added to our interface with a transformation. “Create AsyncMDSL Specification” is available in the “MDSL” menu (if the MDSL Tools have been installed into Eclipse). The channel that corresponds to the findAll operation looks as follows:
channel PaperPublishingRealizationEndpoint_findAll
request message findAllRequestChannel on path "/findAllRequestChannelPath"
expecting payload "query":{"queryFilter":MD<string>*}
reply message findAllReplyChannel on path "/findAllReplyChannelPath"
delivering payload "result":{"responseDTO":PaperPublishingRealizationEndpointDTO}*Rationale: Event-driven architectures and queue-based messaging are popular choices when integrating enterprise applications and service components. With AsyncMDSL, related modeling support is integrated into MDSL.
Another MDSL generator supports the transition from this abstract, pattern-oriented AsyncMDSL to AsyncAPI (“Generate AsyncAPI Specification”). The channel for findAll now looks like this:
findAllRequestChannel:
name: findAllRequestChannel
title: Find All Request Channel
description: |
No description specified
Request message. Reply message is *findAllReplyChannel*.
payload:
type: object
properties:
'queryFilter':
type: array
items:
type: string
findAllReplyChannel:
name: findAllReplyChannel
title: Find All Reply Channel
description: |
No description specified
Reply message. Request message is *findAllRequestChannel*.
payload:
type: array
items:
type: object
required:
- 'responseDTO'
properties:
'responseDTO':
$ref: '#/components/schemas/PaperPublishingRealizationEndpointDTO'The generated AsyncAPI is available for download as well. You can test it in the AsyncAPI Playground.
6c (optional): Java Modulith
If you are not in the mood for remoting and have decided to build a modular monolith instead,[5] you can also generate a set of Java classes and interfaces that are structured according to our intermediate API design.
Selecting “Generate Java Modulith” in the “MDSL” menu causes this code to be generated into the src-gen folder (multiple folders and files).
// ** service interface class:CreateResponseDataTypeList create(CreateRequestDataTypeList anonymousInput);
FindAllResponseDataTypeList findAll(FindAllRequestDataType anonymousInput);
PaperPublishingRealizationEndpointDTO findById(FindByIdRequestDataType anonymousInput);// ** DTOs used in interfacepublic class FindAllRequestDataType {
private List<String> queryFilter;
private Integer limit;
private Integer offset;
[...]
}public class FindAllResponseDataTypeList {
private List<FindAllResponseDataType> entries;
[...]
}public class FindAllResponseDataType {
private PaperPublishingRealizationEndpointDTO responseDTO;
private Integer offsetout;
private Integer limitout;
private Integer size;
private String self;
private String next;
[...]
}
The in parameters are called anonymousInput because the generator was unable to get a name from the input MDSL (which would be possible). The generated code comes with a very basic server stub implementation and a JUnit test case. So you can run it straight away.
Rationale: Remoting introduces overhead, which sometimes is not justified or not tolerable. A standalone Java program with local interfaces also is very easy to run and test, which supported rapid prototyping and continuous design refinement.
Step 7: From API Description to API Implementation(s)
See Step 7 of the post “Domain-Driven Service Design with Context Mapper and MDSL” for instructions how to:
- Create REST controllers in a Spring Boot application from the Step 6a OpenAPI.
- Complete this application and test it.
- Deploy it to a public cloud.
Wrap Up and Next Steps
This post demonstrated how to progress from requirement elicitation/analysis to contract-first service design in a few incremental steps. These steps are supported by several MDSL Tools. We were able to progress from analysis to early design and experimentation quite rapidly:
- Integration stories make sure that API operations are introduced for a reason, supporting the API first principle.
- Interface data modeling was underrated so far, but is important to get request and response messages content right; it is supported by transformations in the MDSL tools.
- An API design evolves continuously by refactoring them to patterns (which requires more than code refactoring).
- OpenAPI and other contract formats are generated from intermediate and final MDSL specifications (that come out of Steps 1 to 3).
- Mock implementations no longer have to be written manually but are generated too. Rapid API prototyping and testing is streamlined this way. Continuous API testing might then suggest further refactorings, leading back to earlier step in an iterative and incremental fashion.
The advantages of the resulting mini-method for service-oriented analysis and design include:
- An API feature requirement, expressed as story, is the starting point. Via a trace link, this goal is kept at hand while designing.
- We can experiment with different designs on a technology-independent, but still concrete level.
- Platform artifacts can be generated, which jump starts agile development.
Some of the consequences of leveraging such a tool-supported method are:
- A new language and supporting tools are involved, which causes some learning effort and context switches between tools while designing.
- Models (would) have to be reconciled if the tools were to support automated round-tripping.
- Modeling tools in general might be perceived as “anti agile” — if code is seen as the only abstraction that is required and suited.
Outlook: Having run through these seven steps, can we consider the API done? Well, not quite. The steps only yielded an early prototype and service stub. Next up would be providing backend connectivity, implementing API logic including transaction management, monitoring, backup and security design (to call out just a few particularly relevant design issues and stakeholder concerns). Our “Design Practice Repository (DPR)” can guide you through through some of the more advanced tasks.
Latest reference information for the MDSL transformations featured in this post (and all additional ones) can be found here.
Final thought: service and application design, implementation and integration might be business as usual for many of us, but never gets boring!
Contact me if you have comments (see links below).
– Olaf (a.k.a. socadk)
Note: A slightly longer version of his story is available at https://ozimmer.ch/practices/2022/01/20/StoryDrivenServiceDesignDemo.html
Acknowledgements
The refactorings featured in this post are joint work with my OST colleague Mirko Stocker. Mirko also reviewed an intermediate draft of this post.
AsyncMDSL and the AsyncAPI generator were developed by Giacomo Di Liberali in his master thesis at the University of Pisa.
While at OST, Stefan Kapferer originally developed several of the MDSL generators featured in this post.
Notes
[1] The Web version does not yet support all transformations. It also uses slightly different command names at present.
[2] This data-level transformation is not available in the MDSL Web Tools; as a workaround, you can copy the snippet from this post and paste it into the edit view that is available.
[3] This substep is also featured in another story, “Domain-Driven Service Design with Context Mapper and MDSL”.
[4] Many resources explaining how to use OpenAPI exist, for instance “3 Tools to Auto-Generate Client Libraries From OAS “.
[5] An IEEE Software Insights experience report “The Monolith Strikes Back: Why Istio Migrated From Microservices to a Monolithic Architecture” features a larger example of such decision.
© Olaf Zimmermann, 2022. All rights reserved.
