Artifact Management on Google Cloud

Artifacts

The output of your software build process commonly produces an artifact, maybe a Docker image or similar, that then needs to be pushed / stored somewhere for deployment.

Historically on Google Cloud, you could use Container Registry. While you still can, there are various pointers in the documentation asking folks to move to Artifact Registry. You can also enable Google’s behind-the-scenes proxy for “moving” your images / making them available in Artifact Registry but that is just too much “magic” for me.

Exploring Artifact Registry a bit and some of the additional features has resulted in various learnings that this post explores and shares.

What about that API?

The Artifact Registry API is interesting. At first glance, it looks like it would have everything that you could imagine needing with all the expected CRUD (Create, Read, Update, and Delete) operations for repositories, images, files, etc. In most cases, it seems to serve a lot of needs but there are some interesting oddities and things to note depending on your expectations.

Let’s Talk Tags

Let's Talk Tags

With Container Registry, it utilized Docker’s built-in functionality to add a latest tag if you didn’t specify one on upload. This is interesting and various downstream services may rely on this functionality. For example, external vulnerability scanning software commonly relies on a “standard” tag depending on implementation. Google’s Artifact Registry does not have this same behavior nor does it appear possible to enable. Thus, if you expect this tag to exist, you’ll likely be disappointed. Of course you can update your processes that push images to push any tag you wish but out-of-the-box this is a change in behavior that was not clearly documented.

Let’s Talk Repositories vs. Registries

With Container Registry, your images are stored in one of four possible multi-regional registries (cloud storage buckets). This is fairly simple to follow and is mimicked by the hostname in the image. Google’s Artifact Registry is quite a bit different with both single region and multi-region options. Depending on how you utilize artifact storage this can get quite complicated (and confusing) if you have one project specifying us and another using us-central as an example.

A related complexity (or confusion) are the image paths.

With Container Registry, it was fairly simple with:
```
gcr.io / <some repository> / <images>
```

With Artifact Registry this can be quite different:

us-docker.pkg.dev / <some registry> / <sub-folder: maybe a service or team> / <images>

This extra level completely makes sense for an organization but does result in an extra level of complexity for artifact management, integrations (both custom and out-of-the-box), and API calls.

It became very clear to me quite early in Artifact Registry usage that it would become a wild west if we didn’t set standards through shared modules / interfaces. Whereas with Container Registry’s “repository” typically having a single meaning, these paths could have multiple meanings: services, teams, security profiles, etc.

Trying Out That API

Like all other Google Cloud APIs, it is a fairly standard looking interface, implemented in various client libraries for Python, Go, etc.

For a variety of reasons here, it seems that the Python Client Library is more usable than the Go Client library for this type of service (and many others on Google Cloud).

To start, you might want to list images (assuming you have uploaded some to test):

artifactregistry_v1.ListDockerImagesRequest(parent=’projects/<project_id>/locations/-’)

This will throw a 400, which is relatively odd.

After engaging with Google Cloud support, they confirmed this is “unexpected” and said there is an internal Feature Request opened to fix it:

“I have tested and confirmed this behavior myself.
Upon searching internally about why it is not working when asking to show
all of the locations during one call, I could find a Feature Request
opened for it. Right now, this action is only possible through the
CLI with the `gcloud artifacts repositories list` , however I agree
that an API method equivalent would be useful.”

This isn’t the end of the world but it does make this quite annoying at a scale of hundreds of images, in hundreds of registries.

No worries, we can just list locations and then iterate / insert those:

ERROR: module 'google.cloud.artifactregistry_v1' has no attribute 'ListLocationsRequest'

I guess the implementation forgot about this type of request for the client libraries.

Not a problem, you can use the discovery APIs and the REST interface to make this request:

with build('artifactregistry', 'v1', credentials=credentials) as service:
        request = service.projects().locations().list(name=f'projects/{project}')

Excellent, now you can access the locations we will need to utilize to list specific images in a location.

This does beg the question, did anyone actually try to use these client libraries to date? Or, is this just too early for this product even though it is the provider recommendation?

Removing & Adding Your Own Tags

Being able to remove / add tags at a global level to both scan and implement company policy is fairly important with any artifact storage mechanism. Sometimes it is not desired to attempt and do this through a client-side module (e.g. global build configuration, global CI configuration). In this case, I wanted to address this as a post-push mechanism / service enforcement. There are various reasons for this but this allows development to occur without getting blocked or even a need to understand some of the complexity of secure supply chains as is the case with this use case.

To scan all Docker images in Artifact Registry as discussed above and be able to make per-location decisions requires a mixture of client libraries and REST calls.

A snippet that iterates a given set of projects, locations, repositories, and images:

# Iterate projects
for project_id in project_ids:
    location_paths = []
    repositories   = []
    location_paths = list_locations(credentials=credentials, project=project_id)

    # Gather repos for all locations
    for location in location_paths:
        repositories_in_location = list_repositories(client=client_artifactregistry, location=location)
        repositories.append(repositories_in_location) if repositories_in_location else None
    repositories_flattened = flatten(repositories)

    # Gather images for each repository
    for repository in repositories_flattened:
        images_per_repo = []
        docker_images_in_repo = list_images(client=client_artifactregistry, repository=repository)

        if len(docker_images_in_repo) > 0:
            images_per_repo.append(docker_images_in_repo)
            images_flattened = flatten(images_per_repo)

            # {'image_short_name': [{image_details}, {image_details}]}
            image_objects = split_images_per_repo(images_flattened)

            # Iterate short names, which are "services" in our world
            for service in image_objects:
                # Sort images by `upload_time`, in reverse for the most recent being the first element
                images_sorted_desc = sorted(image_objects[service], key=lambda x: x['upload_time'], reverse=True)

                # Does any image have the `latest` tag?
                image_with_latest_tag = ''
                for obj in images_sorted_desc:
                    if 'latest' in obj['tags']:
                        image_with_latest_tag = obj['name']

If you have multiple services / teams (sub-folders) in a repository, you will likely need to group them by the image name itself:

for image in images:
    # Split the name up to the `@` symbol as our key
    image_name = image['name'].split('@')[0]

The upload_time returned can be easily converted to a timestamp:

for response in page_result:
    images.append({
        'name': response.name,
        'tags': response.tags,
        'upload_time': datetime.datetime.fromtimestamp(response.upload_time.timestamp()).timestamp()
    })

These are the bigger structural pieces needed if you have a similar type of use case and wanted to get started with the Artifact Registry APIs. This isn’t meant as a complete example as everyone’s use case and implementation is likely very different but enough to get started.

How-to Actually Create & Delete Tags

Interestingly, creating and deleting tags in Artifact Registry is actually more complicated than I originally expected. There are a couple reasons for this:

The APIs (and Client Libraries) lack clear examples of full requests.
There are virtually no full examples in existence as public / open source contributions.
These APIs are unlike most other GCP APIs.

First, you need permissions like any GCP API. For me, this was most easily accomplished with a Custom Role and dedicated Service Account. If you haven’t read our previous post on Custom Roles you may want to check it out here.

If you are utilizing Terraform, it may look something like this:

resource "google_organization_iam_custom_role" "artifact-registry-foobar" {
  role_id = "ArtifactRegistryFooBar"
  org_id  = data.google_project.org.org_id
  title   = "Artifact Registry FooBar"
  permissions = [
    "artifactregistry.tags.create",
    "artifactregistry.tags.delete",
    "artifactregistry.tags.list"
  ]
}

And associating a dedicated Service Account to it:

binding {
    members = [
      "serviceAccount:artifact-registry-foobar@artifact-registry-foobar.iam.gserviceaccount.com",
    ]
    role = google_organization_iam_custom_role.artifact-registry-foobar.id
  }

With the appropriate permissions in place, now comes the tricky part - figuring out how to call these APIs (or utilizing the Client Libraries). If this is your first time, you may want to start with the create and delete REST APIs for this functionality as the documentation is a bit easier to parse but still isn’t entirely helpful.

If you want to jump right into some initial questions you may have (as did I):

Is packages in the parent parameter literal packages or is this a placeholder for the type of registry?
- This is literal packages.
What is a tagId? Is this the tag name?
- No, it is not the tag name. I am actually still not sure what it is and how it impacts anything nor can I find it documented anywhere.
How do I specify the tag?
- You follow the JSON specification here. You provide the name and version in a very specific format (more on this below).
Is it correct that the delete call uses name as the parameter and not parent as the create call does?
- Yes, that is correct.

To create a tag, you need to format a couple pieces. You cannot reuse the format of the name that you may have used as returned from the images API as the name.

Let’s jump into creating a tag on an Artifact Registry Docker Image:

name = tag_obj['name']
name_in_tag_format = name.replace('dockerImages', 'packages')
name_split         = name_in_tag_format.split('@')
name_with_sha      = name_split[0] + '/versions/' + name_split[1]
name_without_sha   = name_split[0]

try:
    request = artifactregistry_v1.CreateTagRequest(
        parent=name_without_sha,
        tag=artifactregistry_v1.Tag(
            name=name_without_sha + '/tags/<your fancy tag>',
            version=name_with_sha
        ),
        tag_id='<your tag id>'
    )
    client.create_tag(request=request)
except exceptions.AlreadyExists:
    print('add_tag(): Tag already exists, skipping...')

A couple things to note here:

The name must be formatted removing dockerImages and replacing it with packages.
To tag a version, you have to add the /versions/ to the string before the SHA.
You have to specify a tag_id which I am not actually sure what it implies.
You can catch a variety of errors, such as AlreadyExists if you choose.

Removing a tag is a bit simpler but still has its oddities:

name = tag_name
name_in_tag_format = name.replace('dockerImages', 'packages')
name_split         = name_in_tag_format.split('@')
name_without_sha   = name_split[0]

try:
    request = artifactregistry_v1.DeleteTagRequest(name=name_without_sha + '/tags/latest')
    client.delete_tag(request=request)
except Exception as err:
    print('remove_tag() ERROR:', err)

A tag removal takes far less construction and is a bit easier to parse.

Conclusion

Using Artifact Registry has a lot of benefits if you need them but it is a bit hard to recommend to folks if all they need is some basic Docker image storage well integrated into the Google Cloud ecosystem (or otherwise).

Container Registry is generally much easier to work with and more mature all around but specifically with their API and external integrations. Multiple 3rd-parties, while they have basic Artifact Registry integrations, all seem to fall short and I suspect this is due to some of the complexities discussed above and their lack of dedication to usable client libraries and APIs. It is also much harder to work with Artifact Registry even just for basic monitoring, scanning, and updating of tags.

Providers that recommend a solution that isn’t completed / prime-time is a great reminder to thoroughly test and give all services a good shakedown before trusting their recommendation. At Unit 410 we like to constantly be trying new services, implementations, and seeing if there is a benefit (or detriment) so that we have a formed opinion.

If you like this sort-of work, feel free to get in touch with us.

We are currently hiring for a Cryptocurrency Engineer in / around our Austin, TX location.