Dynamic Storage in Action: Scaling Storage for Nexus Repository 3

Repository Manager | Reading time: 10 minutes

In this guide

Introduction

 

Let’s say you’re a system administrator managing hundreds – perhaps thousands – of hosted applications on Nexus Repository. So you’re likely to be familiar with the pitfalls of managing disk capacity. Constant builds and commits to your internal repositories not only consume the storage while the application grows but also it affects system performance.

Before you reach full capacity, you can configure the repository manager so that its blob stores scale to new storage locations. We call this dynamic storage. It allows you to combine multiple blob stores into a single group, then connect the group to another storage location. In essence, dynamic storage expands your local instance to new disks. Then you can migrate data to new locations.

Let’s take a closer look at the capability and how it could be a beneficial component to your storage management strategy.

Prerequisites

Before you dive into this guide we recommend that take a look at the:

These guides give you the lowdown on common terminologies, the relationship between the directories that maintain storage, and some basic tips on estimating blob stores by size.

Desired Outcomes

At the end of this guide, you will be able to:

  • Understand how blob store groups are advantageous to storage management.
  • Create blob store groups to add endpoints for additional storage.
  • Evaluate the use of soft quotas and how they elevate awareness of space issues in Nexus Repository.
  • Demonstrate the process to create a dynamic blob store group.

Dynamic Storage with Blob Store Groups

It’s common for repository administrators to – at some point – encounter space issues that result from repeated builds, lofty images, and continuous deployments to the application. This is where the process of merging one or more local blob stores to a blob store group comes in handy.

In the worst case scenario, you’ll exceed the limit of storage in the repository manager without advance notification. At this point, you’ll more than likely receive error messages such as the one depicted in this knowledge base article.

At this stage of repository knowledge it’s likely you have existing repositories on your local instance. So the next step is to promote an existing blob store to a group. Blob store groups allow repositories to store artifacts across multiple storage locations. By assigning additional storage to the group you can dynamically expand your storage especially when your team continues to grow in size.

Meeting Storage Needs

Planning a strategy to extend storage resources to all applications your fellow developers is not easy. Part of scaling extra storage is to consider the frequency of output of team product.

When your CI/CD pipeline matures the size of your blob stores inevitably grows, too. That’s because blob stores – the directories that contain component data in your repositories – increase in size from repeated, continuous builds. You assign multiple repositories to one blob store. However, the binaries deployed to that blob store can fill up the directory, increasing the risk of a full disk.

In order to scale your repositories and assigned blob stores you’ll need to convert the existing blob stores to blob store groups. To do this you’ll need to do the following:

  1. Attach network storage to your local repository manager.
  2. Convert your local blob stores into a blob store group.
  3. Create a new blob store on the newly attached media.
  4. Add a new blob store to the blob store group.

So let’s take a look at your blob stores on an installation before you attach additional resources.

On this local install the list shows there’s nearly 53 gigabytes of available space for each of the configured blob stores. This seems to imply that your repository may have access to twice the amount of space. It’s not the case. In fact, the Available space value at 52.9 GB means the figure is the total amount of blob storage across the entire local installation. This is because both blob stores were created on the same local disk.

As seen here you have a finite amount of space, so we recommend you employ a strategy to help you stay alert on storage constraints. On the Blob Stores screen in the UI you get a clear picture on the potential space provided for components and on your local installation. So when all those hefty builds – like Docker images – start coming through, you’ll need to perform additional configuration in you blob stores to avoid reaching the maximum.

Using Soft Quotas for Disk Management

Running out of disk space can be catastrophic for your Nexus Repository Manager and lead to a corruption of your internal component database. To avoid this you can apply a disk quota on your blob stores. In this version of Nexus Repository an administrator like you can place a soft quota on an individual blob store or a dynamic group.

When you activate a soft quota on a blob store it places a limit on the size or remaining disk space for that blob store or even a dynamic group. Currently hard quotas or limits don’t exist, where if maximum storage is reached you team won’t be able to write to the blob store directory any more.

A soft quota, on the other hand, acts like a grace period. It’s a way for you to get an early warning on the space taken up in a blob store group (or individual blob store). This type of provisional warning provides an opportunity to act before the disk fills up with contents.

Soft quota includes a couple parameters:

  • Space remaining - Based on the set value this warning will alerts you on the remaining disk space available in the blob store group. For example, if you have one gigabyte of disk space and your blob store capacity is 650MB, you receive a warning when you have 350MB remaining.

  • Space used - When you set this value you can receive a message on the total amount of space blob store group consumes. So, if you set the quota to alert you at the halfway point of your one gigabyte maximum, you’ll receive a warning when you reach 500MB total space used.

You can get reports, messaging, and warnings of blob store health from a few areas:

  • Status - On this screen – found under the Support menu – a comment will appear in the Message column if you violate the set quota.

  • API - On the API screen, under System:

  • Enter the blob store id name.

  • Then click the Execute button.

  • Review the Server response section. It displays a human-readable message, stating either amount of space remaining or used.

  • CLI - From your terminal you can also review the status of your blob store groups, by ID.

Bear in mind though that when a soft quota is assigned, producers on your team could continue to write to disk without fail.

Blob Store Groups and Repositories

Let’s start with an illustration of a local file blob store. In this depiction, we created a blob store called test-npm-directory. This directory is slowly being consumed by content. An additional drive is needed to give the storage some elasticity.

In the diagram above:

  • The Available space of the blob store – test-npm-store – is 50 gigabytes.
  • The repository Repo_hosted_1 – on the same disk – consumes 40 gigabytes of space.
  • The repository Repo_hosted_2 – also on the same disk – consumes 9 gigabytes of space.

The repositories that store the binaries are getting too large. So in order to connect more space you need to convert the local directory to a dynamic group. This action logically separates the connection between ../sonatype-work/nexus3 and ../sonatype-work/nexus3/blobs/<blob-id> – originally located on the local disk – from Repo_hosted_1 and Repo_hosted_2. When the blob group is formed and the two hosted repositories will point to this new dynamic location.

In the dynamic group you can promote extra blob stores into the group. For example you can add another disk with 100 gigabytes of space. On top of the 50 gigs, already allocated to the group, the hosted repositories now have access to 150 total gigabytes. Any new component builds deployed or distributed to either blob store, which now has 3 times the space after the group and attached network storage is applied to the repository manager.

Create a Dynamic Group with Storage Limits

In this demonstration, convert a blob store to a group. Then apply a soft quota to the group you create. For the purposes of achieving an outcome that you can quickly detect in the console log, the quota limit will be small.

Convert the Blob Store to a Group

In this example the blob store file1 is used by one repository.

To add more storage to your blob store perform the following steps:

  1. Go to the Blob Stores screen from the Administration panel on the left
  2. Choose a Blob Store with a File storage Type, e.g file1.
  3. Click the Promote to group button.

When you promote the blob store, file1, it now becomes a group that you can extend to additional storage media. On this new screen a Members section appears. You can add additional blob store files moving them from the Available field to the Selected field.

When you navigate back to the Blob Stores screen the new group is listed with the suffix “-promoted”. In other words the group file1-promoted is now listed among your other blob stores.

Next, choose a Fill Policy. The attributes from this selection allows your team of producers to write to a certain blob store in your group. You can choose from:

  • Round Robin - this policy opens up writes to all Selected blob stores in the group. When you have multiple blob stores in your group all requests to deploy, modify, or write to the repository will pass through each blob equally.

  • Write to First - this policy limits all writes to the first Selected blob store. If have a new blob store on an empty disk this policy enables all writes to occur on the new media drive.

Add a Soft Quota to the Group

In the new Group blob store, file1-promoted, manage your disk quota by reviewing the amount of space used on disk. Follow the steps:

  1. Check the Enable Soft Quota box.
  2. Select Space used from the Type of Quota dropdown.
  3. Add a small value to the Quota Limit in MB dropdown, e.g. 1.
  4. Click Save to add the quota to this blob store.

Next test this configuration out.

  1. Upload a file that exceeds 1MB.
  2. Review the quota warning. You can either check the logs, use the API interface, or the Status screen.

Remove a Blob Store from the Group

If you have an old disk with a blob store no longer needed, you can schedule a task – Admin - Remove a member from a blob store group – to remove it. It’s no different from other tasks. You can either run it manually or automate it with a cron expression which creates a firing schedule.

To do this:

  1. Select Tasks from the menu.
  2. Select Admin - Remove a member from a blob store group.
  3. Identify the task in the Task Name field.
  4. Enter the Group.
  5. Enter the Blob store member.
  6. Select Task frequency.
  7. Click Create task to complete the task configuration.
  8. Select the Task Name from the list.
  9. Click Run to execute the task.

When the task is executed it copies all contents from the blob store member to the next available member in the group. In other words, the process changes the state of the unwanted member to read-only. After it redistributes the blobs the member is removed. The blob store group then returns to its original state, with one less member.

What’s Next?

Now you know how to create blob store groups with a soft quota and attach them to external media. As mentioned earlier, you can connect your groups to network storage or to cloud services. The next guide will cover the guidelines and steps to move the blob store contents from one external file system to another.