Google Cloud - Developer's Log - Moving to GCP Cloud Run From App Engine

After banging my head against the wall with a third party library, trying to update to version 8 of the Google Ads API, I lost faith in the 3rd provider's ability to mix well in my NX monorepository workspace.  While it may be a good lesson to keep my workspace updated with the latest code, the jump from version 4 to version 8, didn't seem that difficult on the surface.  Underneath that surface, where shared dependencies on GRPC for Firebase and the library left me stuck and cussing enough to make Black Beard blush.

This isn't a show stopper by any means.  When you've been a developer for 15+years, you become a polyglot of various languages.  Luckily, one of my implementations for this project uses Google Ads C# library, which again, I was forced down that path due to some third party library missing supported features.

Lesson learned, use the SDK from the service provider.  While Google Ads does allow for older versions of its API to be consumed, I understand their need to shut it down four versions later... totally my fault.

The C# project I created earlier generates different mutations to create the resources necessary for a large number of ads.  This relates to my post about importing 100+K Records in MySql On Google Cloud.  Thousands of permutations need to be created for all the Ads mutations, and the third library SDK didn't provide the ability to create a batch job.

With that project in hand, I created a console application that listens and watches for a number events.

1. Pulls messages from a Pub/Sub message queue (filtered subscription) to create the batch job
2. Polls the batch job to check when it's completed, every five minutes.
3. Pulls messages from the same queue (filtered subscription) to sync all the data from the batch job into the DB

In order to run this application, I packaged this into a Docker image.  With the Docker image, I created a Compute instance VM that creates a container from the image.

With the code in place and experience of creating a Docker image to be consumed, removing certain services from App Engine, where I presently have all my REST services living, needed to be moved.  The catch now, how do I route specifically to the different services in Google Cloud?  I have App Engine running my present REST API, a Task Service, and it also serves the UI.  I could use App Engine Flexible to run the image, but I don't want it running all the time if it's not in use.  With App Engine Flexible, it would always be running one instance.  And this is where Google Cloud Run is the better option.

So I have a few unknowns at this point:

1. How do I package this for Cloud Run?
2. What's configuration for Cloud Run like?  I reused an App Engine Template earlier used between multiple projects to stay in the free tier.
3. How am I going to route traffic differently between App Engine and Cloud Run?

Tasks

1. Move the Rest Services that were failing in Node to a WebApi implementation
2. Generate Docker Image to contain the WebApi Project to run services that NodeJs failed me
3. Deploy to Cloud Run
4. An Http Load Balancer will allow me to route traffic to the different services I need, so this will be a good option for App Engine / Cloud Run
5. Test, Test, Test

Move the Rest Services that were Failing in Node

With the other service in place that runs different tasks pushed from the queue, I initially have a WebApi project previously running to work locally.  Alongside, my inversion of control project that maps the contracts to the implementation can be shared between the two projects.  I'm not too concerned yet with the scoping / lifestyle of the object.

I need to make sure tokens are validated, so I added a Firebase token verification as an AuthorizationFilter, which then I wrapped in a TypeFilterAttribute so I could decorate the specific routes in the WebApi project, as not all routes / controllers will consume it.

Next, I created a GoogleAdsTokenFilter, checking the body of the request containing the token to grant access to Google Ads Api.  I added a filter here so I could add the the token to a customer context service that has a lifestyle scope to the request.

Problem

I ran into some issues with my GoogleAdsTokenFilter.  I essentially read the body of the request twice.  I read this stream in the filter, and as a parameter for the route using [FromBody].  The GoogleAdsTokenFilter exists to set the customer context from the request body, so in the pipeline of the request, it reads the entire stream of the body, leaving the position of the stream at the end rather than the beginning for the FromBody param attribute to consume.

I needed to set the stream back to the initial position for the route to read.  To accomplish this I need to to do two things, as I learned.

1. EnableBuffering so the stream could be re-read
2. Leave the StreamReader open in order for the body to be read again

Great example: Re-reading ASP.NET Core request

Generate the Docker Image and Push

While this should've been fairly easy, as I noted early I've done this before, I spent the last 15minutes struggles with an ambiguous error failed to compute cache key: "/bin/release/net5.0/publish" not found: not found.  WTF?  I copied the relative path from where it's located and tried to build under the directory the docker file existed.  This shouldn't be happening.

Problem

The .dockerignore file needed to be named dockerfile.dockerignore in the directory.  I'm no docker expert, but that seems odd given the explanation for building dotnet core web apps here: https://docs.docker.com/samples/dotnetcore/.  Either way, image created.  I noticed I named it dockerfile.dockerignore in my console application too.  Okay... whatever, in the name of progress, moving on.

I'll push the image to container registry so I can utilize in Google Cloud.  This deals with it's own set of complexities with logging in or using the correct IAM account.  The user uploading needs access to storage buckets.  Since I'm owner of the project, I'm able to write to whichever storage bucket I please.  For other developers that may exist on your team, or if you're doing this via a build script/job, use a service account and abide by the principle of least privilege.  Docker will also need to be configured, but all these things are beyond the scope of my rambling in this case.  Just so others know the cost of set up and time to do these things, I mention them.

In the end, docker image pushed successfully my project's container registry.  

Deploy to Cloud Run

This one could get a bit more difficult since I lack experience deploying to GCP's Cloud Run.  From what I witnessed earlier trying to configure this, it doesn't seem any more intimidating than setting up my VM with a docker container image.  Let's give this a whirl.

Pretty simple in the first two steps, configure the service name and select the docker image pushed to container registry, done.

The next step, I felt unsure because the docker image port may be exposed in the underlying image.  Luckily, it exposes port 8080, so no change required for step 3 on the port.  I don't need to pass any arguments to the container commands / arguments since the entrypoint remains the same for the image.

The autoscaling will be the part to pay attention.  Billing increases if I allow too many instances, and if I scale to the present 100 instances required, I have GOOD problems, or a shit number of requests I didn't want.  I haven't looked at Cloud Run's ability to prevent denial of service attacks and am curious of the strategy(ies) to resolve.

I added the environment variables similar to my VM implementation, and set the service account to my App Engine account.

Starting of the container failed.  There's an issue with the port.  So aspnet core uses the ASPNETCORE_URLS environment variable to map the ports.  I'll set that and update the container port and redeploy.

Still running into an issue.  Using the Firebase SDK, I accidentally packaged my .env into the image.  I'm able to tell considering the GOOGLE_APPLICATION_CREDENTIALS reads from a path that exists on my local environment.  I added the .env not be added to the image within my .dockerignore file.  Not sure why it's still getting packaged into the image.  I need to figure out if there's a way to ignore this file when I use dotnet publish.

Got it, the .csproj configuration allows me to ignore the .env file from being copied on publish.

<ItemGroup>
    <None Update=".env">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
</ItemGroup>

I deployed a new image and changed the container image to the new version in Cloud Build.  It appears to be running.

I tested the changes by hitting the URL with expected authorization headers and request body to my services.  It kept failing, but I was no longer receiving a response with the Google Robot.  While I thought I had opened the service to the public, I hadn't.  I updated the permissions to allow all users and received an unexpected error, but a familiar one from my service.

I missed a couple of environment variables that were contributing to a couple of other errors.  Those were added and requests were responding as expected with the content.

Setting up the Load Balancer

Now that my services are scattered through different service areas of GCP, a load balancer needs to direct traffic to the appropriate service based on the requested routes.

I setup a static IP address for both IP4 and IP6.  I then configured this in the DNS host.

It appears I can make requests on the http protocol but not https via the custom domain.  In this case, I'm using a custom domain since this is a development environment.

In a future iteration of this, I may want to create an Identity Aware proxy that ensures certain calls made to certain load balancer routes are authorized.  As I go toward a more microservice architecture, I don't want to have to redeploy each app every time I make a change to the authorization check.

It looks like there's an issue with the GCP generated certificate.

Not necessarily an issue with the certificate but rather the configuration within DNS.  Initially, the IP address proxied through the provider emitting a different IP address configured within my load balancer.  Time to test the front end.

Everything appears to be functioning properly.  I'm unable to see the client list names or retrieve a dependent list.  I just need to redeploy the frontend in order to show the new changes.

As I progress in this project, I plan on moving the frontend code to a storage bucket and create a backend for it.  I do need to figure out how not to cache the index.html file as this will contain all new references to static resources.

Test, Test, Test

After several tests of the UI, one minor issue, but the data returned from the API services works.  Since my backend service works with another Google API, the performance within their network works pretty well.

Conclusion

Moving over to Cloud Run proved to be no worse than setting up App Engine.  Since the services I consume exist in different Serverless offerings in GCP, I needed a strategy that would balance and route the different requests for the different services and utilized an Http Load balancer.  This posed it's own unrelated GCP challenges when I chose a GCP manage certificate for the domain name, as my DNS provider proxied requests, differing IP addresses.

I uncovered a number of improvements I can make for future iterations.

1. Develop a pipeline that creates a new image and deploys to Cloud Run
2. With the load balancer in place, an Identity Proxy will be explored to handle authorization
3. I will be moving away from Node for my backend services that need to talk to other Google APIs, particularly if they aren't Google generated libraries.