Here at Piclo, we’re on a journey to build a suite of effective and easy-to-use publicly-facing Application Programming Interfaces (APIs). We want to drive a significant proportion of System Operator (SO) and Flexibility Service Provider (FSP) interactions with the Piclo Flex platform to be via APIs. Why? In short, because users want to interact with Piclo Flex as efficiently and systematically as possible. You can read more about why APIs are the next step for flexibility markets in our previous blog.
There are various ways to approach the development and implementation of APIs. Many opt for an “implementation first” development approach, which consists of building the API first and then documenting it afterwards, maybe as an afterthought. However, we decided against this approach because we strongly feel that API specs, and by extension developer documentation, are key to building great APIs. As a result, our design, implementation and testing phases all revolve around the API specifications (specs), hence the “spec-driven development” term. This approach enables us to create APIs and the supporting API documentation with clearer requirements, better documentation, shorter feedback loops, and better team alignment.
This blog sheds light on the ideology we follow at Piclo when approaching the
four stages of spec-driven development: requirements gathering and design, implementation, testing and release. Within each of these, I’ll discuss how the API spec is front and centre and the benefits this ultimately brings to the end product - an API that can be successfully integrated by our consumers.
Requirements Gathering & Design
So, what do we need to kick-off this first step of spec-driven API development? In short, good functional requirements.
We need to know what the API is supposed to do, with further consideration for things like; what bit of Piclo functionality are we aiming to expose, what would a good interface look like for our consumers, and how does this fit into the wider ecosystem of APIs? If we have these well-defined, this puts us in a good position to begin designing an initial version of the API spec.
At Piclo we follow a RESTful design ethos, so this largely guides the nitty-gritty design decisions of the API spec. Combining this with the aforementioned functional requirements, we then take a collaborative approach - pulling in expertise from related product teams - to drafting out an initial version of the API spec. These are written following OpenAPI 3.0 specifications, which means we can leverage spec testing libraries (which I’ll touch upon later) and also serve them as our public API documentation using Redoc.
In following this process, the first key benefit of this approach is that it solidifies the functional requirements of the interface. Additionally, it also ensures the team is completely aligned with what we’re trying to achieve, before actually beginning any “physical” implementation work. Sounds simple, but the importance of this can often be underestimated.
Once we’re all aligned on what we’re building and have an initial version of the API spec written out, we commit this to our codebase. At first, we do not expose it publicly, but instead hide it behind a feature flag. Lower, internal environment versions of our docs site can have these feature flags enabled, so that developers’ can refer to the API specs for the next phase of development.
For some projects we’ve found it really advantageous to share an “initial” copy of API specs at this pre-implementation stage with external users, so we can get early feedback and alignment from them. This is useful on two fronts. Firstly, it means we can be confident that what we’re going to invest development time and effort building is aligned to the outcomes of all parties. And secondly, it gives our users the option to begin initial development on their side against the API spec, hence saving time by enabling development between both parties in parallel. Of course, this works best when both sides can remain agile to information discovered during implementation.
Implementation
I won’t go into much detail on our API implementation approach here, as it’s not overly relevant to the purpose of this piece, but the main takeaway is that after collaboratively writing the spec, implementation comes second.
Spec-driven development benefits implementation as once we have an initial version of the API spec defined, it forms a clear basis of what the developer is aiming for the API implementation to look and behave like. Having the API spec there as a ‘blueprint’ makes the developers life a lot easier when implementing the API itself, as there is little ambiguity of what the API should look and behave like when coupled with the functional requirements gathered previously.
However, that’s not to say the API Spec has to be followed precisely. Sometimes, during this phase, extra details emerge that were not evident in the design phase - and that’s ok! By delving into the codebase, or understanding service interactions better it’s fairly common to want to make small tweaks to both the API spec and implementation. For example, adding an extra field to the response object, or adding another parameter to filter the request by. That can be done here too.
Testing
So far, we have our API Spec and our API implementation. The API Spec has likely been iterated upon during the implementation phase previously, however, the crucial thing to point out at this stage is that the API Spec and the API implementation are effectively separate entities. We’ve relied solely on the developer following the Spec to create the API implementation. However, developers are fallible. This means there will likely be small inconsistencies lurking between the two, including in things like; data types, regex patterns, and required/ optional status of fields.
In order to sync up the API Spec and API implementation, we write Spec tests. At Piclo we use a Python library, Schemathesis, which reads our API spec and generates a bunch of test cases to check that the implementation behaves exactly as it’s stated in the documentation.
This is the third major benefit of API Spec-driven development: explicitly confirming that our API documentation matches the API implementation itself.
Release
Once we’re happy the spec tests are implemented and our documentation matches the API implementation, we can then expose this to our users. In practice, what this entails is to “turn on” a particular feature flag in the production environment - allowing the API to be reachable and displaying the spec on our docs site.
Summary
So there’s a whistle-stop tour of our 4-step approach to API Spec-driven development; design, implementation, spec testing and release and the 3 main benefits we leverage from this approach:
- It promotes establishing the functional requirements of the API first and foremost - we can’t create the spec without knowing exactly what we’re trying to do.
- Creating the API spec first, acts as a great and easily understandable “blueprint” for the implementing developers to aim for and also iterate upon.
- Lastly, and most importantly, with the help of some useful external tools, we can explicitly ascertain our publicly-facing documentation matches the API implementation itself.
All in all, we’ve found that these things add up to result in: clearer requirements, better documentation, shorter feedback loops, and better team alignment.
