Continuing my adventures with Balena, an amazing IoT orchestration platform, I recently found myself wanting to test the stack of an IoT project locally. Tilt is a great platform for this, while it's generally oriented towards Kubernetes it also provides support for docker-compose. For those who aren't familiar with docker-compose, it's essentially schema that allows you to define a collection of docker containers and how to run them.
It's perfect for local development and is what Balena uses to deploy multiple containers onto a device. The downside to docker-compose is that it doesn't expose anything for automatically rebuilding a service. This means for each change in our application we need to rebuild it manually, and that's no fun. This is where Tilt comes in, using the
docker_compose function in our
Tiltfile we're able to have docker-compose just work.
This is great, but we still don't have our automatic rebuilds we want. We're able to achieve this by using the
docker_build function. Let's take a look at the changes.
If we run
tilt up we'll automatically get our docker image rebuilt on changes ?'
Using with Balena
For those of familiar with Balena you're probably aware of
balena push, which is a great command but it's not all that useful for trying to work with locally. Which is why we had to go build the
docker-compose setup system earlier. When I first got that system working I was frustrated with having to maintain two development systems. Our
docker-compose setup is further optimized to build binaries on the host and sync only those in, and the same was done with some hacky combination of
balena push sync + air automated rebuilds, and entr for automatic restarts without restarting the container. So, I set out to see how we could potentially hook this up to Balena.
Where's the Docker Host?
I started by looking at the balena-cli source code trying to reverse engineer the
balena push command. It turns out that the docker runtime is exposed at
<ip>:2375 (source). I gave this a shot by using
DOCKER_HOST=tcp://<device-ip>:2375 docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d96ab57fe037 d91873660990 "/bin/prometheus --c…" 36 minutes ago Up 36 minutes prometheus_4874803_2159222_f47abb6c669dd1f683fa440f17822a43
89b5eb8751a1 c3a2c27f36c9 "/usr/local/bin/my…" 36 minutes ago Restarting (2) 15 seconds ago my-service_4874801_2159222_f47abb6c669dd1f683fa440f17822a43
8052870f678a a1dc2ecec4c4 "/run.sh" 36 minutes ago Up 36 minutes grafana_4874804_2159222_f47abb6c669dd1f683fa440f17822a43
91d306f3860f registry2.balena-cloud.com/v2/85d77236b4d0fe3fc0e64c7ea43fc645 "/usr/src/app/entry.…" 2 hours ago Up 2 hours (healthy) balena_supervisor
Turns out, it's that easy. There's no authentication on the docker runtime on the
development images that Balena provides, which makes some sense given their disclaimer about development images not being safe for production usage.
Plugging it into Tilt
Taking a quick look at the images we can see that there's some special labels added to the images generated by Balena.
$ DOCKER_HOST=tcp://<device-ip>:2375 docker inspect <a-service-container-id> | jq .'.Config.Labels'
It turns out that the supervisor really likes these labels to be set, so I went ahead and created a
docker-compose.tilt.yml to be used for Tilt and inserted these labels in there for each of our service.
After that I noticed the supervisor was pretty angry about the container name format. This was easy to fix by using the
container_name attributed in the
docker-compose.tilt.yaml, so I went and added that to each of my services.
Now that we have our
docker-compose.tilt.yml all we need to do is get Tilt to use it. Simply modify the
docker_compose call to include it:
I wrote a quick Go script to do this, but an easier way is just to use the following one-liner (if there's a demand for this I'm happy to make this more portable):
That's it! Press
space to open the Web UI and you should be able to see your services in action.