llvm::cas::ondisk::OnDiskGraphDB Class Reference

On-disk CAS nodes database, independent of a particular hashing algorithm. More...

#include "llvm/CAS/OnDiskGraphDB.h"

Classes
struct	IndexProxy
	Proxy for an on-disk index record. More...

Public Types
enum class	FaultInPolicy { SingleNode , FullTree }
	How to fault-in nodes if an upstream database is used. More...
using	HashingFuncT
	Hashing function type for validation.

Public Member Functions
LLVM_ABI_FOR_TEST Error	store (ObjectID ID, ArrayRef< ObjectID > Refs, ArrayRef< char > Data)
	Associate data & references with a particular object ID.
LLVM_ABI_FOR_TEST Expected< std::optional< ObjectHandle > >	load (ObjectID Ref)
ArrayRef< uint8_t >	getDigest (ObjectID Ref) const
LLVM_ABI_FOR_TEST Expected< ObjectID >	getReference (ArrayRef< uint8_t > Hash)
	Form a reference for the provided hash.
LLVM_ABI_FOR_TEST std::optional< ObjectID >	getExistingReference (ArrayRef< uint8_t > Digest, bool CheckUpstream=true)
	Get an existing reference to the object Digest.
Expected< bool >	isMaterialized (ObjectID Ref)
	Check whether the object associated with Ref is stored in the CAS.
bool	containsObject (ObjectID Ref, bool CheckUpstream=true) const
	Check whether the object associated with Ref is stored in the CAS.
LLVM_ABI_FOR_TEST ArrayRef< char >	getObjectData (ObjectHandle Node) const
object_refs_range	getObjectRefs (ObjectHandle Node) const
LLVM_ABI_FOR_TEST size_t	getStorageSize () const
unsigned	getHardStorageLimitUtilization () const
void	print (raw_ostream &OS) const
Error	validate (bool Deep, HashingFuncT Hasher) const
	Validate the OnDiskGraphDB.
LLVM_ABI_FOR_TEST Error	validateObjectID (ObjectID ID)
	Checks that ID exists in the index.
LLVM_ABI_FOR_TEST	~OnDiskGraphDB ()

Static Public Member Functions
static LLVM_ABI_FOR_TEST Expected< std::unique_ptr< OnDiskGraphDB > >	open (StringRef Path, StringRef HashName, unsigned HashByteSize, OnDiskGraphDB *UpstreamDB=nullptr, std::shared_ptr< OnDiskCASLogger > Logger=nullptr, FaultInPolicy Policy=FaultInPolicy::FullTree)
	Open the on-disk store from a directory.

Detailed Description

On-disk CAS nodes database, independent of a particular hashing algorithm.

Definition at line 258 of file OnDiskGraphDB.h.

Member Typedef Documentation

HashingFuncT

using llvm::cas::ondisk::OnDiskGraphDB::HashingFuncT

Initial value:

 function_ref<void(
      ArrayRef<ArrayRef<uint8_t>>, ArrayRef<char>, SmallVectorImpl<uint8_t> &)>

Hashing function type for validation.

Definition at line 332 of file OnDiskGraphDB.h.

Member Enumeration Documentation

FaultInPolicy

enum class llvm::cas::ondisk::OnDiskGraphDB::FaultInPolicy

strong

How to fault-in nodes if an upstream database is used.

Enumerator
SingleNode	Copy only the requested node.
FullTree	Copy the the entire graph of a node.

Definition at line 348 of file OnDiskGraphDB.h.

Constructor & Destructor Documentation

~OnDiskGraphDB()

OnDiskGraphDB::~OnDiskGraphDB

(

)

Definition at line 1676 of file OnDiskGraphDB.cpp.

Member Function Documentation

containsObject()

bool llvm::cas::ondisk::OnDiskGraphDB::containsObject ( ObjectID Ref, bool CheckUpstream = true ) const

inline

Check whether the object associated with Ref is stored in the CAS.

Note that this function does not fault-in.

Definition at line 292 of file OnDiskGraphDB.h.

References llvm::consumeError(), llvm_unreachable, and llvm::Ref.

getDigest()

ArrayRef< uint8_t > llvm::cas::ondisk::OnDiskGraphDB::getDigest ( ObjectID Ref ) const

inline

Returns

the hash bytes digest for the object reference.

Definition at line 271 of file OnDiskGraphDB.h.

References llvm::cantFail(), getDigest(), and llvm::Ref.

Referenced by getDigest(), load(), store(), and validateObjectID().

getExistingReference()

std::optional< ObjectID > OnDiskGraphDB::getExistingReference	(	ArrayRef< uint8_t >	Digest,
		bool	CheckUpstream = true )

Get an existing reference to the object Digest.

Returns nullopt if the object is not stored in this CAS.

Definition at line 1157 of file OnDiskGraphDB.cpp.

References llvm::expectedToOptional(), I, LLVM_UNLIKELY, P, and llvm::Ref.

getHardStorageLimitUtilization()

unsigned OnDiskGraphDB::getHardStorageLimitUtilization

(

)

const

Returns

The precentage of space utilization of hard space limits.

Return value is an integer between 0 and 100 for percentage.

Definition at line 1589 of file OnDiskGraphDB.cpp.

getObjectData()

ArrayRef< char > OnDiskGraphDB::getObjectData

(

ObjectHandle

Node

)

const

Returns

the data part of the provided object handle.

Definition at line 1220 of file OnDiskGraphDB.cpp.

References assert(), and getContentFromHandle().

getObjectRefs()

object_refs_range llvm::cas::ondisk::OnDiskGraphDB::getObjectRefs ( ObjectHandle Node ) const

inline

Returns

the object referenced by the provided object handle.

Definition at line 313 of file OnDiskGraphDB.h.

References llvm::cas::ondisk::InternalRefArrayRef::begin(), llvm::cas::ondisk::InternalRefArrayRef::end(), and llvm::make_range().

getReference()

Expected< ObjectID > OnDiskGraphDB::getReference

(

ArrayRef< uint8_t >

Hash

)

Form a reference for the provided hash.

The reference can be used as part of a CAS object even if it's not associated with an object yet.

Definition at line 1145 of file OnDiskGraphDB.cpp.

References I, and LLVM_UNLIKELY.

getStorageSize()

size_t OnDiskGraphDB::getStorageSize

(

)

const

Returns

Total size of stored objects.

NOTE: There's a possibility that the returned size is not including a large object if the process crashed right at the point of inserting it.

Definition at line 1585 of file OnDiskGraphDB.cpp.

isMaterialized()

Expected< bool > OnDiskGraphDB::isMaterialized

(

ObjectID

Ref

)

Check whether the object associated with Ref is stored in the CAS.

Note that this function will fault-in according to the policy.

Definition at line 1297 of file OnDiskGraphDB.cpp.

References llvm_unreachable, and llvm::Ref.

load()

Expected< std::optional< ObjectHandle > > OnDiskGraphDB::load

(

ObjectID

Ref

)

Returns

nullopt if the object associated with Ref does not exist.

Definition at line 1236 of file OnDiskGraphDB.cpp.

References llvm::sys::fs::closeFile(), createCorruptObjectError(), llvm::createFileError(), llvm::cas::ondisk::ObjectHandle::fromFileOffset(), llvm::cas::ondisk::ObjectHandle::fromMemory(), getDigest(), I, llvm_unreachable, llvm::sys::fs::openNativeFileForRead(), llvm::sys::fs::mapped_file_region::readonly, llvm::Ref, llvm::sys::sandbox::scopedDisable(), and llvm::sys::fs::status().

open()

Expected< std::unique_ptr< OnDiskGraphDB > > OnDiskGraphDB::open ( StringRef Path, StringRef HashName, unsigned HashByteSize, OnDiskGraphDB * UpstreamDB = nullptr, std::shared_ptr< OnDiskCASLogger > Logger = nullptr, FaultInPolicy Policy = FaultInPolicy::FullTree )

static

Open the on-disk store from a directory.

Parameters

Path	directory for the on-disk store. The directory will be created if it doesn't exist.
HashName	Identifier name for the hashing algorithm that is going to be used.
HashByteSize	Size for the object digest hash bytes.
UpstreamDB	Optional on-disk store to be used for faulting-in nodes if they don't exist in the primary store. The upstream store is only used for reading nodes, new nodes are only written to the primary store. User need to make sure UpstreamDB outlives current instance of OnDiskGraphDB and the common usage is to have an UnifiedOnDiskCache to manage both.
Policy	If UpstreamDB is provided, controls how nodes are copied to primary store. This is recorded at creation time and subsequent opens need to pass the same policy otherwise the open will fail.

Definition at line 1596 of file OnDiskGraphDB.cpp.

References llvm::sys::path::append(), llvm::argument_out_of_domain, llvm::cas::ondisk::CASFormatVersion, llvm::cas::OnDiskDataAllocator::create(), llvm::cas::OnDiskTrieRawHashMap::create(), llvm::sys::fs::create_directories(), llvm::createFileError(), llvm::createStringError(), DataPoolFilePrefix, DataPoolTableName, llvm::cas::ondisk::getOverriddenMaxMappingSize(), IndexFilePrefix, IndexTableName, SingleNode, and llvm::cas::ondisk::useSmallMappingSize().

Referenced by llvm::cas::ondisk::UnifiedOnDiskCache::open().

print()

void OnDiskGraphDB::print

(

raw_ostream &

OS

)

const

Definition at line 1062 of file OnDiskGraphDB.cpp.

References assert(), D(), llvm::Data, llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::isAligned(), llvm::Align::Of(), llvm::Offset, llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::sort(), and llvm::toString().

store()

Error OnDiskGraphDB::store	(	ObjectID	ID,
		ArrayRef< ObjectID >	Refs,
		ArrayRef< char >	Data )

Associate data & references with a particular object ID.

If there is already a record for this object the operation is a no-op.

Parameters

ID	the object ID to associate the data & references with.
Refs	references
Data	data buffer.

Definition at line 1450 of file OnDiskGraphDB.cpp.

References llvm::Alloc, assert(), createCorruptObjectError(), createTempFile(), llvm::Data, llvm::dbgs(), llvm::ArrayRef< T >::empty(), llvm::cas::FileOffset::get(), getDigest(), llvm::handleErrors(), I, LLVM_DEBUG, LLVM_UNLIKELY, P, llvm::Ref, Size, and llvm::Error::success().

validate()

Error OnDiskGraphDB::validate	(	bool	Deep,
		HashingFuncT	Hasher ) const

Validate the OnDiskGraphDB.

Parameters

Deep	if true, rehash all the objects to ensure no data corruption in stored objects, otherwise just validate the structure of CAS database.
Hasher	is the hashing function used for objects inside CAS.

Definition at line 899 of file OnDiskGraphDB.cpp.

References llvm::arrayRefFromStringRef(), llvm::createStringError(), D(), llvm::ArrayRef< T >::drop_back(), llvm::sys::fs::exists(), llvm::MemoryBuffer::getFile(), llvm::cas::ondisk::InternalRef::getFromOffset(), I, llvm::illegal_byte_sequence, llvm::isAligned(), llvm_unreachable, llvm::Align::Of(), llvm::Offset, llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::Ref, llvm::Error::success(), llvm::toHex(), llvm::toString(), and llvm::utohexstr().

validateObjectID()

Error OnDiskGraphDB::validateObjectID

(

ObjectID

ID

)

Checks that ID exists in the index.

It is allowed to not have data associated with it.

Definition at line 1032 of file OnDiskGraphDB.cpp.

References llvm::createStringError(), getDigest(), llvm::cas::ondisk::ObjectID::getOpaqueData(), I, llvm::illegal_byte_sequence, llvm::cas::ondisk::OnDiskGraphDB::IndexProxy::Offset, P, llvm::Error::success(), llvm::toHex(), llvm::toString(), and llvm::utohexstr().

---

The documentation for this class was generated from the following files:

• include/llvm/CAS/OnDiskGraphDB.h
• lib/CAS/OnDiskGraphDB.cpp