I've realized a lot since I started my first job in 2017 as a Software Engineer. Most importantly, that the primary focus of a software engineer's job should be being as absolutely as boring as possible. At first this might seem like it doesn't make sense, what does boring even mean in the context of software design? Well, uniqueness is the enemy of any process. The more unique something is, the more complicated it is. Complicated can result in a magnitude of issues; knowledge siloing, large timelines to ship features, angry glares from an SRE team that has to help deploy the thing, and much more.
So, you might be wondering. Jared, what does being boring have to do with the title to begin with? With the rise of platform engineering, there's been a large shift towards building platforms instead of having a team of individuals be responsible for different parts of an application, now you can have one team owning their code end-to-end. Traditional SRE teams have ended up being split into a mix of SRE, operations, developer tooling, CI/CD, etc. that each provide platforms for developers. This has presented a pretty unique problem, how do these teams best integrate into the developer lifecycle? Through the repositories themselves.
Enter... Templating
While at Outreach, thanks to some amazing engineers I worked with (David de Regt and Ramesh VK) a tool called bootstrap was created. Bootstrap took the pretty traditional approach to, well, "bootstrapping" a repository by rendering templates. This is a pretty common pattern that we see across the ecosystem, either as Github Template Repositories or tools like cookiecutter. The problem with this model is that it only allows you to initialize something once. As soon it is ran, it's already stale, out of date, with no hope of being updated without a lot of effort later. In the middle of a monolith decomposition project, with the idea of 40-50 repositories being create at least, we needed a better way to solve this. The idea of "blocks" were born, a simple concept that would create a notion of "api boundaries" within files.
package main
func main() {
///StartBlock(name)
// Code inside of this block can be modified freely while
// code outside of this block is managed by bootstrap.
///EndBlock
}It may seem freedom restricting at first glance, but I've found that blocks are a powerful construct that can be used to bridge the gap between boilerplate and developer business logic. With this concept, we turned bootstrap into a smart templating system introducing the concept of native plugins enabling even more intelligence, such as AST powered rewrites of code imports for cases that wouldn't work in a pure template model.
Example: Golang Boilerplate
A great example of this problem is handling boilerplate for a Go application. Normally, you'll create a main package where you'll want to create some sort of logical flow to implement something, say a HTTP server.
package main
import "net/http"
func main() {
http.ListenAndServe(":8080", nil)
}Wow, super simple, right (aren't all small examples 😀). Now, throw in things like open telemetry tracing, metrics, config, and some other things... now it's not so simple.
package main
import (
"context"
"net/http"
"os"
)
// main is the entrypoint our service
func main() {
// exitCode support while allowing defers to function
exitCode := 1
defer func() {
if r := recover(); r != nil {
panic(r)
}
os.Exit(exitCode)
}()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
cfg, err := config.LoadConfig(ctx)
if err != nil {
log.Error(ctx, "failed to load config")
return
}
// Metrics
pexp, err := prometheus.New()
if err != nil {
log.Fatal(err)
}
provider := metric.NewMeterProvider(metric.WithReader(pexp))
meter := provider.Meter("myApp")
go serveMetrics()
if err := http.ListenAndServe(":8080", nil); err != nil {
log.Error(ctx, "error serving http: %v", err)
}
exitCode = 0
}
func serveMetrics() {
log.Printf("serving metrics at localhost:2223/metrics")
http.Handle("/metrics", promhttp.Handler())
if err := http.ListenAndServe(":2223", nil); err != nil {
fmt.Printf("error serving http: %v", err)
return
}
}That's also grossly understating where the developer's code would live too, as it's just a footnote. Most, if not all, of that could be templated But, how do you handle getting the developer owned code inside of there without causing issues? Blocks solve this problem easily.
package main
import (
"context"
"net/http"
"os"
)
// main is the entrypoint our service
func main() {
// exitCode support while allowing defers to function
exitCode := 1
defer func() {
if r := recover(); r != nil {
panic(r)
}
os.Exit(exitCode)
}()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
cfg, err := config.LoadConfig(ctx)
if err != nil {
log.Error(ctx, "failed to load config")
return
}
// Metrics
pexp, err := prometheus.New()
if err != nil {
log.Fatal(err)
}
provider := metric.NewMeterProvider(metric.WithReader(pexp))
meter := provider.Meter("myApp")
go serveMetrics()
// Add your business logic here.
//
// <<Stencil::Block(services)>>
if err := http.ListenAndServe(":8080", nil); err != nil {
log.Error(ctx, "error serving http: %v", err)
}
// <</Stencil::Block>>
exitCode = 0
}
func serveMetrics() {
log.Printf("serving metrics at localhost:2223/metrics")
http.Handle("/metrics", promhttp.Handler())
if err := http.ListenAndServe(":2223", nil); err != nil {
fmt.Printf("error serving http: %v", err)
return
}
}Note: You might be noticing some problems with the code above, don't worry, we did too! I'll talk about this more later.
Maturing bootstrap
Going back to our adventure rolling out bootstrap. After awhile, we realized that we had created a really powerful tool and had a desire to open-source what we created. Taking a look at some of the problems that we had encountered with the original tool, we created a new tool named stencil. Stencil was designed from the ground up to be open source and to support modules. As stencil usage grew across the company and the platform engineering model was adopted by other teams, there was a desire for distinct template ownership areas to be managed and released by those teams. We realized that we needed a module system and constructs to facilitate interoperability between those modules.
Looking back to the example shown earlier, we also noticed a few problems with the block only approach. What's around blocks is incredibly important. Developers would rely on variables and other things around the context of the blocks that we may not have necessarily wanted them to, which caused problems changing the code outside of it. Developers would also complain about large diffs in code reviews whenever they ran the tool. Out of this learning, we realized that blocks are most effective when combined with libraries. Let's go back to the example shown earlier.
package main
import (
"context"
"os"
"os/signal"
"github.com/getoutreach/gobox/pkg/async"
"github.com/getoutreach/stencil-golang/pkg/serviceactivities/automemlimit"
"github.com/getoutreach/stencil-golang/pkg/serviceactivities/gomaxprocs"
)
// main is the entrypoint for our service.
func main() {
defer exitcode.Exit()
ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
defer cancel()
cfg, err := config.LoadConfig(ctx)
if err != nil {}
acts := []async.Runner{
gomaxprocs.New(),
automemlimit.New(),
// <<Stencil::Block(services)>>
// !! Business logic implements the async.Runner interface and is
// !! allowed to be added here.
myapp.New(cfg),
// <</Stencil::Block>>
}
err = async.RunGroup(acts).Run(ctx)
if shutdown.HandleShutdownConditions(ctx, err) {
exitcode.Set(0)
}
}
Developers can now worry about satisfying one concrete interface and getting all other benefits provided by other teams and, most importantly, retaining full control of the entire lifecycle of their application still. Based on the block's location, they also are essentially only allowed to implement code that hooks into the async.Runner interface, ensuring that the bulk of their logic will only exist in files outside of the heavily restricted main.go.
The stencil-age
The rest is going to read as an advert for stencil, and to some extent it is, but I think it's important to detail some of the problems that we set off to solve with the design of stencil and what it brings to the table in this space.
Module Hooks
A problem presented by separate template modules is, how do you handle a file wanting to write to another or integrate with it? We solved this problem by introducing module hooks, dedicated integration points that can be written to by other templates.
Native Extensions
Mentioned earlier, bootstrap originally supported an AST-powered code rewriting system, during the migration to stencil it was decided to retire that system to reduce the complexity. However, added instead is the concept of native extensions that can be written in any language (though we primarily use Golang), and can provide functions callable through templates to implement custom logic. One example, is logic to merge go.mod files to handle user added modules and minimum versions managed by the original templates.
What's next?
Hopefully by now I've convinced you of the value of templating repositories and thinking about boilerplate as another API that should be provided to developers by a platform engineering team. Whenever you find yourself needing to do the same thing across n repositories, remember this post 😄. stencil lives on post-Outreach as a independently maintained open source project and we'd love to hear your feedback or receive contributions! Check out our documentation or Github for more information!