Archiving Cold Research Data
Table of Contents
About the cloud archive service
What archive storage is (and is not)
Preparing collections for archive
Other options available at UNC-CH
About our cloud archive service
Research data volumes continue to grow faster than campus storage capacity. RC's enterprise storage systems are designed for active, high-performance research computing — they are expensive to operate and heavily subscribed. Using them to hold data that no one is actively touching is not sustainable.
For data that must be retained but is no longer in active use ("cold" data), ITS Research Computing offers Microsoft Azure cloud archive storage for approved groups who have data collections that are:
- genuinely cold: not in need of retrieval for at least a year, ideally longer
- curated: reviewed so that only data that truly needs to be kept is archived
- well-organized, with a meaningful structure, naming conventions, and optimized for future retrieval
- packaged into tar bundles accompanied by listing files and a brief README (see Preparing collections for archive)
There is no single size threshold for using the archive. What matters is not how much data you have, but whether the data is archive-ready — appropriately curated, genuinely cold, and worth the effort of packaging and long-term retention. When those conditions are met, archiving should happen. When they are not, more curation is needed first — or the data belongs on active storage, or it should be deleted.
There are typically no up-front charges for basic cloud archive storage. Retrieval, however, incurs real costs — particularly early or frequent retrieval — as bringing data back from the archive tier is billed and not immediate.
At this time we do not allow direct access to archived data; writes and retrieval requests are handled by Research Computing.
What archive storage is (and is not)
Archive storage is much less expensive per terabyte than enterprise storage, but it works very differently from the filesystems you are used to. Key realities:
It is offline-like storage. Archived data cannot be opened, listed, or read in place. To access anything, it must first be "rehydrated" (restored from the archive tier to an accessible tier in the cloud), which can take hours or longer, and then transferred back to campus.
Retrieval has costs. Rehydration and egress (download back to campus) both incur charges. Retrieving large amounts of data can incur significant expenses.
There is a minimum commitment. Azure archive storage has a 180-day minimum storage period. Data deleted or retrieved before 180 days is still billed as if it were stored the full period.
It is write-once by design. Archive works best for data that is packaged once, stored for years, and rarely retrieved.
Archive storage is appropriate for data you must keep (grant, journal publication, ex-student, regulatory retention requirements, irreplaceable raw data) but do not expect to use.
Is your data archive-ready?
Archive-readiness — not data size — is what determines whether archiving is appropriate. Before requesting archive, your team should be able to answer yes to the following:
Will this data sit untouched for at least a year? If there is a realistic chance you (or a student, or a collaborator) will want some of it back within the next 6–12 months, it is not archive-ready. Requests to pull data back 30 or 60 days after archiving defeat the purpose.
Has the data been curated? Someone who knows the nature of the data has reviewed it and removed what does not need to be kept: scratch files, intermediate results that can be regenerated, duplicate copies, old software builds, conda environments, caches, core dumps, etc. Deleting is free; archiving junk wastes shared resources, inflates future retrieval costs, and buries data that actually matters.
Can anything be deleted instead? The cheapest data to manage is data that no longer exists. If the data can be regenerated from raw inputs and documented code, consider archiving only the raw inputs and the code. We strongly encourage deletion wherever appropriate.
Is there a designated owner? A named person (typically the PI) who understands what the data is, why it is being kept, and how long it must be retained, and who can authorize any future retrieval or deletion?
Is it organized for partial retrieval? Data must be packaged so that a future retrieval can target a small, relevant slice rather than the entire archive.
If the answer to any of these is no, the data needs more curation work before archiving — or it should be deleted.
Preparing collections for archive
Archive storage holds packaged bundles, not browsable directory trees. Good packaging determines how much a future retrieval will cost and how long it will take. Storage archives are "object stores", not file systems.
Each collection requires four things:
Tar bundles. Package the data into manageably-sized tar files, split along lines that match how the data might be needed later (by project, experiment, instrument run, or year). A future need should be met by retrieving one or two bundles, not everything.
A listing file for each bundle. Listing files (or sidecar files) stay on locally accessible storage and identify the contents of archived bundles, enabling us to identify which bundle holds a given file without rehydrating anything.
A checksum for each bundle, used to verify the archived copy — and any future retrieval — against the original.
A README describing what the data is, who owns it, the retention requirement it satisfies, and a review/expiration date.
Example, for a directory /proj/tarheellab/stuff_to_archive (the -C option stores relative paths, so the bundle extracts cleanly anywhere):
tar -czvf stuff_to_archive.tar.gz -C /proj/tarheellab stuff_to_archive
tar -tvf stuff_to_archive.tar.gz > stuff_to_archive_tarlist.txt
sha256sum stuff_to_archive.tar.gz > stuff_to_archive.tar.gz.sha256
Do not delete the campus copy until the archived copy has been verified against its checksum. Keep the listing and checksum files on accessible storage.
The habits that make data archive-ready — organizing, documenting, and pruning as you go — are the same habits that make data manageable throughout a project lifecycle.
Getting started
Cloud archive storage requests are evaluated on a case-by-case basis. Before acting on data moves, we first need to discuss the nature of the data and user expectations. We'll walk through the archive-readiness questions above with your group, assess the candidate collection, and determine if it's organized in a logical manner, formatted correctly, and plan how it will be curated. This is not gatekeeping for its own sake — it protects your group from retrieval costs and protects shared campus resources.
To get started, please contact us at research [at] unc.edu
Retrieval
Retrieval is expected to be rare. When it is needed, contact us with the specific tar bundles required (identified from your listing files). Rehydration typically takes up to a day or more, plus transfer time back to campus. Rehydration and download costs are the responsibility of the data owner. Retrieving an entire archive is expensive and slow — this is why curation and packaging discipline up front are essential.
Costs
Research Computing does not re-charge for archival storage, however we reserve the right to pass through data repatriation expenses if one must retrieve large volumes of data or frequently access archived data. ITS-RC pays approximately $15k per PB per year to hold onto archived data on behalf of research at Carolina.
Other options available at UNC-CH
In addition to Research Computing's cloud archive storage services, please note that the following services should be checked to see if they are more appropriate for your data and storage requirements:
RMDC: The Research Data Management Core. Tools and services to support data management and sharing, including consultations and training, data curation and archiving, and tools and infrastructure. https://researchdata.unc.edu/
CDR: The Carolina Digital Repository. The Carolina Digital Repository (CDR) is a digital archive for scholarly materials produced by members of the University of North Carolina at Chapel Hill community. The main goal of the CDR is to keep UNC digital scholarly output safe, accessible and discoverable for as long as needed. https://cdr.lib.unc.edu
See also
Introduction to the topic of digital preservation: https://en.wikipedia.org/wiki/Digital_preservation
ITS tips on avoiding digital clutter: https://its.unc.edu/2024/08/29/dont-be-a-data-hoarder/