How to Write an Nx Executor for Building Docker Images

Generating artifacts for my NestJs applications needed a common approach. Within Social Sprinkler, five backend services support user management, marketing / email, creating social posts, media, and publishing. The output of each service is a Docker image that eventually is deployed into Google Cloud Run.

Executors perform actions on code via a task name. With many deployable NestJs apps within my workspace, I wanted a common task that creates Docker images.

Goals

1. Understand Docker Files in the Workspace
2. What is an Executor?
3. Create the Executor
4. Node's child_process - The Secret Sauce

Workspace Docker Files

The end state is to have a Node image that can run my NestJs applications. In order to save package install time, I created a base.dockerfile under a docker directory.

The .dockerignore file skips copying certain directories and files into the image. My builds at the time of this writing happen on my laptop. So .env files, node_modules, .angular, and coverage directories are ignored.

The service images will be built on top of the base.dockerfile. To keep the image size small, dependencies are installed using the --omit=dev option skipping the install of devDependencies in the package.json.

Run the following command if you want to build the base.dockerfile. Get comfortable with the command because this will be used when the executor is created.

The api-app.dockerfile uses the mutable-ideas:${TAG} image. Each one of the ARG variables will be passed to our docker build command. Below explains what each build arg is.

The command below builds the api-app.dockerfile.

The build arguments are a good indicator of what our executor parameters might be. So let's dive into what is an Executor.

What is an Executor?

Executors run common tasks against code in the workspace. An Nx Workspace can contain multiple frameworks and coding languages. So a common task can be run with the respective cli for a particular project. Think of the common task of building a NestJs and Angular App. Same task, different CLIs called to operate on the code.

Imagine after the Docker images have been created, the next goal is to deploy the application to Google Cloud Run. The executor for the task build-docker-image would differ from the deploy task. Rather than the Docker cli, deploying to Google Cloud would require use of the Google Cloud cli, gcloud.

Now with the understanding of what an executor is, let's create one.

Create the Executor

Plugins are projects that contain executors, generators, and lint rules that run against your workspace's code.

Run the following command to create the plugin.

Create the Executor by running the command below.

With the custom executor in place, let's give it the power to Dockerize a NestJs application in the workspace.

Node's child_process - Secret Sauce

The base.dockerfile image needs to be rebuilt any time when the base.dockefile or the package.json file changes in a particular git commit. A unique hash of the files will serve as the tag of the image. Finally, the image needs to be built.

So how does the executor run the necessary CLI commands for Docker?

Child Process

The child_process module out of NodeJs provides the ability to spawn a subprocess. The exec method helps with asynchronous operations allowing a callback once the process terminates.

execPromise returns data back as string from the subprocess once it terminates. It also helps log any messages returned from the subprocess output.

With the ability to run subprocesses, utility classes that run a unique subprocess can be created to meet our goals of creating a Docker image.

So what other helpers should be created?

File Helper - Generate a Unique Tag

The fileHelper is responsible for generating the hash of base.dockerfile and package.json. This serves as the unique tag for the base.dockerfile. When generating the images for the NestJs applications, this tag will be sent along as a reference to pull the specific version of the base.dockerfile.

Tag Helper - Parse Project Tags

The tagHelper provides a bit of meta data around the project. Three tags are associated with all projects in my workspace to keep boundaries between projects, platform, type, domain. They're also handy when running tasks and filtering by a certain tag. I'll elaborate more on filtering by tag when running the tasks.

For the executor, the project tags are used to determine which Docker file to use based on the type tag.

Git Helper - Unique Tag for Commit Changes

gitHelper lists changed files and retrieves the latest commit hash. execPromise wraps the git client commands needed for the executor. The executor will focus on the commit hash as it will serve as the api-app.dockerfile tag. This way it's known which particular commit aligns with the image generated.

Docker Helper - Build, Tag, Push, Query

dockerHelper runs Docker commands necessary for generating the image. The executor calls the query and build methods. Depending on the type of project being built, the build arguments differ. The build arguments will be defined in the executor.

Docker Executor - Build the Image

With the helpers in place, the executor can start being put together. The two Docker images created will use the base.dockerfile and the api-app.dockerfile. Docker file names follow a naming convention of {{typeTag}}.dockerfile. Tagging the project will be covered in Defining a Task's Executor section.

Docker Image Executor

After the API project is built, the output directory to copy files into the image needs to be defined per executor's schema. The outputPath is an optional parameter as the base.dockerfile doesn't need it to create the image. The schema for the executor defines the outputPath which will copy the API project's build output.

The executor is supplied the schema and the executor context. The project configuration helps return the tags of the project. The executor checks the project configuration and parses the tags to provide the type tag. We're only interested in projects that have the type:api-app or type:base tags.

The Docker file, build arguments, and tagging differ between project types. The base.dockerfile does not require build arguments, so the imageName combines the name and unique hash of the package.json and base.dockerfile.

The api-app.dockerfile, requires the build arguments that were defined earlier in the table. The imageName combines the project name and the latest commit hash value.

With the executor written, how is it consumed?

Defining Common Tasks

Projects existing in the app folder of the workspace typically have a build task. A common task needs to be configured in the project.json with the executor to generate the Docker image. It'll be named build-docker-image and it will dependOn the build task for an api-app.

When projects are affected, specific tasks can be targeted to run. In the example below, affected projects are running the build-docker-image tasks for the differences between main and main~1 branches, with projects tagged with domain:social-sprinkler.

Using the tag option isn't necessary per the example below. It does open options for different build systems to define what should be built in the infrastructure based on particular tags. For instance, if projects in my Google Cloud infrastructure align to a particular domain and billing needs to be separated between the two, the tag option offers that filtering.

# FORMAT FOR RUNNING AFFECTED PROJECTS WITH A CERTAIN TASK
npx nx affected -t {{TASKNAME}} --base={{BRANCH_NAME}}~1 --head={{BRANCH_NAME} --tag=domain:{{DOMAIN_TAG}} --parallel=1

# EXAMPLE FOR RUNNING AFFECTED PROJECTS WITH A CERTAIN TASK
npx nx affected -t build-docker-image --base=main~1 --head=main --tag=domain:social-sprinkler --parallel=1

API App - Defining a Task's Executor

For any of the projects tagged with type:api-app, a common task defined with the executor is added. Per the example above, the task is named build-docker-image. The project.json defines the task, executor, and the parameters needed to build the Docker image. The task also depends on the build task being run first.

Workspace - Docker Task Executor

In the root of the workspace, another project.json file exists. The tasks existing in this directory run workspace level tasks. The base.dockerfile is the responsibility of the workspace to manage as changes to either the package.json or the base.dockerfile change the Docker image generated. A task named create-artifacts runs the create-docker-file task and the build-docker-image for all affected projects.

Creating the Images

Images can now created by running the command below.

# Templated Command
# npx nx run {{WORKSPACE_PROJECT_NAME}}:create-artifacts

# With mutable-ideas project name
npx nx run mutable-ideas:create-artifacts

Conclusion

Custom executors can be used to run tasks against the code in the repository. Node's child_process exec method opens the possibility to run CLI commands from git and docker enabling the process for generating images with a unique tagged image. Once these common tasks in projects have been defined, Nx's affected commands can target specific tasks to run and generate artifacts.