A unified CAS nodes and key-value database, using on-disk storage for both. More...

#include "llvm/CAS/UnifiedOnDiskCache.h"

Public Types
using	ValueBytes = std::array<char, sizeof(uint64_t)>

Public Member Functions
OnDiskGraphDB &	getGraphDB ()
	The `OnDiskGraphDB` instance for the open directory.
const OnDiskGraphDB &	getGraphDB () const
	The `OnDiskGraphDB` instance for the open directory.
OnDiskKeyValueDB &	getKeyValueDB ()
	The `OnDiskGraphDB` instance for the open directory.
const OnDiskKeyValueDB &	getKeyValueDB () const
	The `OnDiskGraphDB` instance for the open directory.
LLVM_ABI Error	validateActionCache () const
	Validate the action cache only.
LLVM_ABI Error	close (bool CheckSizeLimit=true)
	This is called implicitly at destruction time, so it is not required for a client to call this.
LLVM_ABI void	setSizeLimit (std::optional< uint64_t > SizeLimit)
	Set the size for limiting growth.
LLVM_ABI uint64_t	getStorageSize () const
LLVM_ABI bool	hasExceededSizeLimit () const
bool	needsGarbageCollection () const
LLVM_ABI Error	collectGarbage ()
	Remove unused data from the current UnifiedOnDiskCache.
LLVM_ABI	~UnifiedOnDiskCache ()

Static Public Member Functions
static LLVM_ABI Expected< std::unique_ptr< UnifiedOnDiskCache > >	open (StringRef Path, std::optional< uint64_t > SizeLimit, StringRef HashName, unsigned HashByteSize, OnDiskGraphDB::FaultInPolicy FaultInPolicy=OnDiskGraphDB::FaultInPolicy::FullTree)
	Open a `UnifiedOnDiskCache` instance for a directory.
static LLVM_ABI Expected< ValidationResult >	validateIfNeeded (StringRef Path, StringRef HashName, unsigned HashByteSize, bool CheckHash, OnDiskGraphDB::HashingFuncT HashFn, bool ForceValidation)
	Validate the data in `Path` in-process, if it has not been validated since the last system boot.
static LLVM_ABI Expected< ValidationResult >	recover (StringRef Path)
	Recover from invalid data in `Path` after a failed `validateIfNeeded`, by marking all the data for garbage collection.
static LLVM_ABI Error	collectGarbage (StringRef Path, ondisk::OnDiskCASLogger *Logger=nullptr)
	Remove any unused data from the directory at `Path`.
static LLVM_ABI ObjectID	getObjectIDFromValue (ArrayRef< char > Value)
	Helper function to convert the value stored in KeyValueDB and ObjectID.
static LLVM_ABI ValueBytes	getValueFromObjectID (ObjectID ID)

Friends
class	OnDiskGraphDB
class	OnDiskKeyValueDB

Detailed Description

A unified CAS nodes and key-value database, using on-disk storage for both.

It manages storage growth and provides APIs for garbage collection.

High-level properties:

While UnifiedOnDiskCache is open on a directory, by any process, the storage size in that directory will keep growing unrestricted. For data to become eligible for garbage-collection there should be no open instances of UnifiedOnDiskCache for that directory, by any process.
Garbage-collection needs to be triggered explicitly by the client. It can be triggered on a directory concurrently, at any time and by any process, without affecting any active readers/writers, in the same process or other processes.

Usage patterns should be that an instance of UnifiedOnDiskCache is open for a limited period of time, e.g. for the duration of a build operation. For long-living processes that need periodic access to a UnifiedOnDiskCache, the client should devise a scheme where access is performed within some defined period. For example, if a service is designed to continuously wait for requests that access a UnifiedOnDiskCache, it could keep the instance alive while new requests are coming in but close it after a time period in which there are no new requests.

Definition at line 41 of file UnifiedOnDiskCache.h.

Member Typedef Documentation

◆ ValueBytes

using llvm::cas::ondisk::UnifiedOnDiskCache::ValueBytes = std::array<char, sizeof(uint64_t)>

Definition at line 163 of file UnifiedOnDiskCache.h.

Constructor & Destructor Documentation

◆ ~UnifiedOnDiskCache()

UnifiedOnDiskCache::~UnifiedOnDiskCache ( )

Definition at line 695 of file UnifiedOnDiskCache.cpp.

References close(), and llvm::consumeError().

Member Function Documentation

◆ close()

Error UnifiedOnDiskCache::close ( bool CheckSizeLimit = true )

This is called implicitly at destruction time, so it is not required for a client to call this.

After calling close the only method that is valid to call is needsGarbageCollection.

Parameters

CheckSizeLimit if true it will check whether the primary store has exceeded its intended size limit. If false the check is skipped even if a SizeLimit was passed to the open call.

Definition at line 640 of file UnifiedOnDiskCache.cpp.

References assert(), llvm::sys::fs::closeFile(), llvm::sys::fs::convertFDToNativeFile(), llvm::sys::fs::create_directory(), llvm::createFileError(), llvm::sys::fs::Exclusive, llvm::sys::path::get_separator(), getNextDBDirName(), hasExceededSizeLimit(), llvm::no_lock_available, llvm::sys::sandbox::scopedDisable(), llvm::Error::success(), llvm::cas::ondisk::tryLockFileThreadSafe(), and llvm::cas::ondisk::unlockFileThreadSafe().

Referenced by ~UnifiedOnDiskCache().

◆ collectGarbage() [1/2]

Error UnifiedOnDiskCache::collectGarbage ( )

Remove unused data from the current UnifiedOnDiskCache.

Definition at line 715 of file UnifiedOnDiskCache.cpp.

References collectGarbage().

Referenced by collectGarbage().

◆ collectGarbage() [2/2]

Error UnifiedOnDiskCache::collectGarbage	(	StringRef	Path,
		ondisk::OnDiskCASLogger *	Logger = nullptr )

static

Remove any unused data from the directory at Path.

If there are no such data the operation is a no-op.

This can be called concurrently, regardless of whether there is an open UnifiedOnDiskCache instance or not; it has no effect on readers/writers in the same process or other processes.

It is recommended that garbage-collection is triggered concurrently in the background, so that it has minimal effect on the workload of the process.

Definition at line 697 of file UnifiedOnDiskCache.cpp.

References llvm::sys::path::append(), llvm::createFileError(), getAllGarbageDirs(), llvm::sys::fs::remove_directories(), llvm::sys::path::remove_filename(), and llvm::Error::success().

◆ getGraphDB() [1/2]

OnDiskGraphDB & llvm::cas::ondisk::UnifiedOnDiskCache::getGraphDB ( )

inline

The OnDiskGraphDB instance for the open directory.

Definition at line 44 of file UnifiedOnDiskCache.h.

References OnDiskGraphDB.

◆ getGraphDB() [2/2]

const OnDiskGraphDB & llvm::cas::ondisk::UnifiedOnDiskCache::getGraphDB ( ) const

inline

The OnDiskGraphDB instance for the open directory.

Definition at line 47 of file UnifiedOnDiskCache.h.

References OnDiskGraphDB.

◆ getKeyValueDB() [1/2]

OnDiskKeyValueDB & llvm::cas::ondisk::UnifiedOnDiskCache::getKeyValueDB ( )

inline

The OnDiskGraphDB instance for the open directory.

Definition at line 50 of file UnifiedOnDiskCache.h.

References OnDiskKeyValueDB.

Referenced by validateActionCache().

◆ getKeyValueDB() [2/2]

const OnDiskKeyValueDB & llvm::cas::ondisk::UnifiedOnDiskCache::getKeyValueDB ( ) const

inline

The OnDiskGraphDB instance for the open directory.

Definition at line 53 of file UnifiedOnDiskCache.h.

References OnDiskKeyValueDB.

◆ getObjectIDFromValue()

ObjectID UnifiedOnDiskCache::getObjectIDFromValue ( ArrayRef< char > Value )

static

Helper function to convert the value stored in KeyValueDB and ObjectID.

Definition at line 112 of file UnifiedOnDiskCache.cpp.

References assert(), llvm::cas::ondisk::ObjectID::fromOpaqueData(), and llvm::support::endian::read64le().

Referenced by validateOnDiskKeyValueDB().

◆ getStorageSize()

uint64_t UnifiedOnDiskCache::getStorageSize ( ) const

Returns: the storage size of the cache data.

Definition at line 602 of file UnifiedOnDiskCache.cpp.

◆ getValueFromObjectID()

UnifiedOnDiskCache::ValueBytes UnifiedOnDiskCache::getValueFromObjectID ( ObjectID ID )

static

Definition at line 119 of file UnifiedOnDiskCache.cpp.

References llvm::support::endian::write64le().

◆ hasExceededSizeLimit()

bool UnifiedOnDiskCache::hasExceededSizeLimit ( ) const

Returns: whether the primary store has exceeded the intended size limit. This can return false even if the overall size of the opened directory is over the SizeLimit passed to open. To know whether garbage collection needs to be triggered or not, call needsGarbaseCollection.

Definition at line 615 of file UnifiedOnDiskCache.cpp.

Referenced by close().

◆ needsGarbageCollection()

bool llvm::cas::ondisk::UnifiedOnDiskCache::needsGarbageCollection ( ) const

inline

Returns: whether there are unused data that can be deleted using a collectGarbage call.

Definition at line 143 of file UnifiedOnDiskCache.h.

◆ open()

Expected< std::unique_ptr< UnifiedOnDiskCache > > UnifiedOnDiskCache::open	(	StringRef	Path,
		std::optional< uint64_t >	SizeLimit,
		StringRef	HashName,
		unsigned	HashByteSize,
		OnDiskGraphDB::FaultInPolicy	FaultInPolicy = OnDiskGraphDB::FaultInPolicy::FullTree )

static

Open a UnifiedOnDiskCache instance for a directory.

Parameters

Path	directory for the on-disk database. The directory will be created if it doesn't exist.
SizeLimit	Optional size for limiting growth. This has an effect for when the instance is closed.
HashName	Identifier name for the hashing algorithm that is going to be used.
HashByteSize	Size for the object digest hash bytes.
FaultInPolicy	Controls how nodes are copied to primary store. This is recorded at creation time and subsequent opens need to pass the same policy otherwise the `open` will fail.

If there is only one directory open databases on it. If there are 2 or more directories, get the most recent directories and chain them, with the most recent being the primary one. The remaining directories are unused data than can be garbage-collected.

Definition at line 505 of file UnifiedOnDiskCache.cpp.

References llvm::sys::path::append(), assert(), llvm::sys::fs::CD_OpenAlways, llvm::sys::fs::create_directories(), llvm::createFileError(), DBDirPrefix, getAllDBDirs(), llvm::cas::ondisk::lockFileThreadSafe(), llvm::sys::fs::OF_None, llvm::cas::ondisk::OnDiskGraphDB::open(), llvm::cas::ondisk::OnDiskKeyValueDB::open(), llvm::sys::fs::openFileForReadWrite(), llvm::cas::ondisk::OnDiskCASLogger::openIfEnabled(), llvm::sys::sandbox::scopedDisable(), and llvm::sys::fs::Shared.

Referenced by llvm::cas::builtin::createBuiltinUnifiedOnDiskCache(), and validateInProcess().

◆ recover()

Expected< ValidationResult > UnifiedOnDiskCache::recover ( StringRef Path )

static

Recover from invalid data in Path after a failed validateIfNeeded, by marking all the data for garbage collection.

Recovery requires exclusive access to the CAS and it is an error to attempt recovery if there is concurrent use of the CAS.

Recovery is serialized with validateIfNeeded, and only happens if the last validation failed or crashed. If the data has been recovered or validated successfully since, e.g. by a concurrent process, recovery is skipped.

Parameters

Path	directory for the on-disk database.

Returns: Recovered if the data has been cleared, Skipped if recovery is not needed, or an Error if recovery cannot be performed.

Definition at line 461 of file UnifiedOnDiskCache.cpp.

References getCachedBootTime(), markAllDBDirsCorrupt(), llvm::cas::ondisk::OnDiskCASLogger::openIfEnabled(), llvm::cas::Recovered, llvm::cas::Skipped, and llvm::toStringWithoutConsuming().

Referenced by llvm::cas::recoverOnDiskUnifiedCASDatabases().

◆ setSizeLimit()

void UnifiedOnDiskCache::setSizeLimit ( std::optional< uint64_t > SizeLimit )

Set the size for limiting growth.

This has an effect for when the instance is closed.

Definition at line 598 of file UnifiedOnDiskCache.cpp.

◆ validateActionCache()

Error UnifiedOnDiskCache::validateActionCache ( ) const

Validate the action cache only.

Definition at line 219 of file UnifiedOnDiskCache.cpp.

References getKeyValueDB(), and llvm::cas::ondisk::OnDiskKeyValueDB::validate().

◆ validateIfNeeded()

Expected< ValidationResult > UnifiedOnDiskCache::validateIfNeeded	(	StringRef	Path,
		StringRef	HashName,
		unsigned	HashByteSize,
		bool	CheckHash,
		OnDiskGraphDB::HashingFuncT	HashFn,
		bool	ForceValidation )

static

Validate the data in Path in-process, if it has not been validated since the last system boot.

A successful validation is recorded so that subsequent calls can skip it; a failed or crashed one is recorded as pending for recover, and is not skipped by subsequent calls. Where the boot time is not known validation is never skipped.

Clients that want to be resilient to unexpected crashes during validation may call this from a separate process (e.g. via llvm-cas -validate-if-needed) and call recover if it fails.

Parameters

Path	directory for the on-disk database.
HashName	Identifier name for the hashing algorithm that is going to be used.
HashByteSize	Size for the object digest hash bytes.
CheckHash	Whether to validate hashes match the data.
ForceValidation	Whether to force validation to occur even if it should not be necessary.

Returns: Valid if the data is valid, Skipped if validation is not needed, or an Error if validation cannot be performed or the data is invalid.

Definition at line 411 of file UnifiedOnDiskCache.cpp.

References getCachedBootTime(), llvm::cas::ondisk::OnDiskCASLogger::openIfEnabled(), llvm::cas::Skipped, llvm::toStringWithoutConsuming(), llvm::cas::Valid, and validateInProcess().

Referenced by llvm::cas::validateOnDiskUnifiedCASDatabasesIfNeeded().