Content-addressable storage for objects. More...

#include "llvm/CAS/ObjectStore.h"

Inheritance diagram for llvm::cas::ObjectStore:

Public Member Functions
virtual Expected< CASID >	parseID (StringRef ID)=0
	Get a `CASID` from a `ID`, which should have been generated by CASID::print().
virtual Expected< ObjectRef >	store (ArrayRef< ObjectRef > Refs, ArrayRef< char > Data)=0
	Store object into ObjectStore.
virtual CASID	getID (ObjectRef Ref) const =0
	Get an ID for `Ref`.
virtual Expected< ObjectRef >	storeFromFile (StringRef Path)
	Stores the data of a file into ObjectStore.
virtual Error	exportDataToFile (ObjectHandle Node, StringRef Path) const
	Exports the data of an object to a file path.
virtual std::optional< ObjectRef >	getReference (const CASID &ID) const =0
	Get an existing reference to the object called `ID`.
virtual Expected< bool >	isMaterialized (ObjectRef Ref) const =0
virtual Error	validateObject (const CASID &ID)=0
	Validate the underlying object referred by CASID.
virtual Error	validate (bool CheckHash) const =0
	Validate the entire ObjectStore.
LLVM_ABI_FOR_TEST Expected< ObjectProxy >	createProxy (ArrayRef< ObjectRef > Refs, StringRef Data)
	Helper functions to store object and returns a ObjectProxy.
Expected< ObjectRef >	storeFromString (ArrayRef< ObjectRef > Refs, StringRef String)
	Store object from StringRef.
Expected< ObjectRef >	storeFromOpenFile (sys::fs::file_t FD, std::optional< sys::fs::file_status > Status=std::nullopt)
	Default implementation reads `FD` and calls storeNode().
LLVM_ABI Expected< ObjectProxy >	getProxy (const CASID &ID)
	Create ObjectProxy from CASID. If the object doesn't exist, get an error.
LLVM_ABI Expected< ObjectProxy >	getProxy (ObjectRef Ref)
	Create ObjectProxy from ObjectRef.
Expected< std::optional< ObjectProxy > >	getProxyIfExists (ObjectRef Ref)
uint64_t	readData (ObjectHandle Node, raw_ostream &OS, uint64_t Offset=0, uint64_t MaxBytes=-1ULL) const
	Read the data from `Data` into `OS`.
virtual Error	setSizeLimit (std::optional< uint64_t > SizeLimit)
	Set the size for limiting growth of on-disk storage.
virtual Expected< std::optional< uint64_t > >	getStorageSize () const
virtual Error	pruneStorageData ()
	Prune local storage to reduce its size according to the desired size limit.
Error	validateTree (ObjectRef Ref)
	Validate the whole node tree.
LLVM_ABI Expected< ObjectRef >	importObject (ObjectStore &Upstream, ObjectRef Other)
	Import object from another CAS.
virtual void	print (raw_ostream &) const
	Print the ObjectStore internals for debugging purpose.
void	dump () const
const CASContext &	getContext () const
	Get CASContext.
virtual	~ObjectStore ()=default

Static Public Member Functions
static Error	createUnknownObjectError (const CASID &ID)

Protected Member Functions
virtual Expected< std::optional< ObjectHandle > >	loadIfExists (ObjectRef Ref)=0
	Load the object referenced by `Ref`.
Expected< ObjectHandle >	load (ObjectRef Ref)
	Like `loadIfExists` but returns an error if the object is missing.
virtual uint64_t	getDataSize (ObjectHandle Node) const =0
	Get the size of some data.
virtual Error	forEachRef (ObjectHandle Node, function_ref< Error(ObjectRef)> Callback) const =0
	Methods for handling objects.
virtual ObjectRef	readRef (ObjectHandle Node, size_t I) const =0
virtual size_t	getNumRefs (ObjectHandle Node) const =0
virtual ArrayRef< char >	getData (ObjectHandle Node, bool RequiresNullTerminator=false) const =0
virtual Expected< ObjectRef >	storeFromOpenFileImpl (sys::fs::file_t FD, std::optional< sys::fs::file_status > Status)
	Get ObjectRef from open file.
StringRef	getDataString (ObjectHandle Node)
	Get a lifetime-extended StringRef pointing at `Data`.
std::unique_ptr< MemoryBuffer >	getMemoryBuffer (ObjectHandle Node, StringRef Name="", bool RequiresNullTerminator=true)
	Get a lifetime-extended MemoryBuffer pointing at `Data`.
virtual void	readRefs (ObjectHandle Node, SmallVectorImpl< ObjectRef > &Refs) const
	Read all the refs from object in a SmallVector.
	ObjectStore (const CASContext &Context)

Friends
class	ObjectProxy

Detailed Description

Content-addressable storage for objects.

Conceptually, objects are stored in a "unique set".

Objects are immutable ("value objects") that are defined by their content. They are implicitly deduplicated by content.
Each object has a unique identifier (UID) that's derived from its content, called a CASID.
- This UID is a fixed-size (strong) hash of the transitive content of a CAS object.
- It's comparable between any two CAS instances that have the same CASIDContext::getHashSchemaIdentifier().
- The UID can be printed (e.g., CASID::toString()) and it can parsed by the same or a different CAS instance with ObjectStore::parseID().
An object can be looked up by content or by UID.
- store() is "get-or-create" methods, writing an object if it doesn't exist yet, and return a ref to it in any case.
- loadObject(const CASID&) looks up an object by its UID.
Objects can reference other objects, forming an arbitrary DAG.

The ObjectStore interface has a few ways of referencing objects:

ObjectRef encapsulates a reference to something in the CAS. It is an opaque type that references an object inside a specific CAS. It is implementation defined if the underlying object exists or not for an ObjectRef, and it can used to speed up CAS lookup as an implementation detail. However, you don't know anything about the underlying objects. "Loading" the object is a separate step that may not have happened yet, and which can fail (e.g. due to filesystem corruption) or introduce latency (if downloading from a remote store).
ObjectHandle encapulates a loaded object in the CAS. You need one of these to inspect the content of an object: to look at its stored data and references. This is internal to CAS implementation and not availble from CAS public APIs.
CASID: the UID for an object in the CAS, obtained through ObjectStore::getID() or ObjectStore::parseID(). This is a valid CAS identifier, but may reference an object that is unknown to this CAS instance.
ObjectProxy pairs an ObjectHandle (subclass) with a ObjectStore, and wraps access APIs to avoid having to pass extra parameters. It is the object used for accessing underlying data and refs by CAS users.

Both ObjectRef and ObjectHandle are lightweight, wrapping a uint64_t and are only valid with the associated ObjectStore instance.

There are a few options for accessing content of objects, with different lifetime tradeoffs:

getData() accesses data without exposing lifetime at all.
getMemoryBuffer() returns a MemoryBuffer whose lifetime is independent of the CAS (it can live longer).
getDataString() return StringRef with lifetime is guaranteed to last as long as ObjectStore.
readRef() and forEachRef() iterate through the references in an object. There is no lifetime assumption.

Definition at line 90 of file ObjectStore.h.

Constructor & Destructor Documentation

◆ ~ObjectStore()

virtual llvm::cas::ObjectStore::~ObjectStore ( )

virtualdefault

◆ ObjectStore()

llvm::cas::ObjectStore::ObjectStore ( const CASContext & Context )

inlineprotected

Definition at line 288 of file ObjectStore.h.

Referenced by llvm::cas::builtin::BuiltinCAS::BuiltinCAS(), and importObject().

Member Function Documentation

◆ createProxy()

Expected< ObjectProxy > ObjectStore::createProxy	(	ArrayRef< ObjectRef >	Refs,
		StringRef	Data )

Helper functions to store object and returns a ObjectProxy.

Definition at line 117 of file ObjectStore.cpp.

References llvm::arrayRefFromStringRef(), llvm::Data, getProxy(), llvm::Ref, and store().

Referenced by llvm::cas::NamedValuesSchema::construct().

◆ createUnknownObjectError()

Error ObjectStore::createUnknownObjectError ( const CASID & ID )

static

Definition at line 112 of file ObjectStore.cpp.

References llvm::createStringError().

Referenced by getProxy().

◆ dump()

LLVM_DUMP_METHOD void ObjectStore::dump ( ) const

Definition at line 27 of file ObjectStore.cpp.

References llvm::dbgs(), LLVM_DUMP_METHOD, and print().

◆ exportDataToFile()

Error ObjectStore::exportDataToFile	(	ObjectHandle	Node,
		StringRef	Path ) const

virtual

Exports the data of an object to a file path.

It does not include any references of the object.

An underlying implementation could perform optimizations that reduce I/O and disk space consumption.

Parameters

Node	the object to read data from.
FilePath	the path of the file data.

Definition at line 148 of file ObjectStore.cpp.

References llvm::sys::path::append(), llvm::SmallVectorImpl< T >::clear(), llvm::createFileError(), llvm::sys::fs::createUniqueFile(), llvm::Data, llvm::SmallVectorTemplateCommon< T, typename >::empty(), getData(), llvm::sys::path::parent_path(), llvm::sys::fs::remove(), llvm::sys::fs::rename(), llvm::scope_exit(), llvm::sys::sandbox::scopedDisable(), and llvm::Error::success().

◆ forEachRef()

virtual Error llvm::cas::ObjectStore::forEachRef	(	ObjectHandle	Node,
		function_ref< Error(ObjectRef)>	Callback ) const

protectedpure virtual

Methods for handling objects.

CAS implementations need to override to provide functions to access stored CAS objects and references.

Referenced by readRefs(), and validateTree().

◆ getContext()

const CASContext & llvm::cas::ObjectStore::getContext ( ) const

inline

Get CASContext.

Definition at line 283 of file ObjectStore.h.

Referenced by llvm::cas::builtin::BuiltinCAS::parseID().

◆ getData()

virtual ArrayRef< char > llvm::cas::ObjectStore::getData	(	ObjectHandle	Node,
		bool	RequiresNullTerminator = false ) const

protectedpure virtual

Implemented in llvm::cas::builtin::BuiltinCAS.

Referenced by exportDataToFile(), getDataString(), getMemoryBuffer(), importObject(), and readData().

◆ getDataSize()

virtual uint64_t llvm::cas::ObjectStore::getDataSize ( ObjectHandle Node ) const

protectedpure virtual

Get the size of some data.

Implemented in llvm::cas::builtin::BuiltinCAS.

◆ getDataString()

StringRef llvm::cas::ObjectStore::getDataString ( ObjectHandle Node )

inlineprotected

Get a lifetime-extended StringRef pointing at Data.

Depending on the CAS implementation, this may involve in-memory storage overhead.

Definition at line 176 of file ObjectStore.h.

References getData(), and llvm::toStringRef().

◆ getID()

virtual CASID llvm::cas::ObjectStore::getID ( ObjectRef Ref ) const

pure virtual

Get an ID for Ref.

References llvm::Ref.

Referenced by llvm::cas::ObjectProxy::getReferenceID(), load(), and validateTree().

◆ getMemoryBuffer()

std::unique_ptr< MemoryBuffer > ObjectStore::getMemoryBuffer	(	ObjectHandle	Node,
		StringRef	Name = "",
		bool	RequiresNullTerminator = true )

protected

Get a lifetime-extended MemoryBuffer pointing at Data.

Depending on the CAS implementation, this may involve in-memory storage overhead.

Definition at line 71 of file ObjectStore.cpp.

References getData(), llvm::MemoryBuffer::getMemBuffer(), and llvm::toStringRef().

◆ getNumRefs()

virtual size_t llvm::cas::ObjectStore::getNumRefs ( ObjectHandle Node ) const

protectedpure virtual

Referenced by importObject().

◆ getProxy() [1/2]

Expected< ObjectProxy > ObjectStore::getProxy ( const CASID & ID )

Create ObjectProxy from CASID. If the object doesn't exist, get an error.

Definition at line 86 of file ObjectStore.cpp.

References createUnknownObjectError(), getProxy(), getReference(), and llvm::Ref.

Referenced by createProxy(), getProxy(), and llvm::cas::NamedValuesSchema::load().

◆ getProxy() [2/2]

Expected< ObjectProxy > ObjectStore::getProxy ( ObjectRef Ref )

Create ObjectProxy from ObjectRef.

If the object can't be loaded, get an error.

Definition at line 94 of file ObjectStore.cpp.

References H, llvm::cas::ObjectProxy::load(), load, and llvm::Ref.

◆ getProxyIfExists()

Expected< std::optional< ObjectProxy > > ObjectStore::getProxyIfExists ( ObjectRef Ref )

Returns: std::nullopt if the object is missing from the CAS.

Definition at line 103 of file ObjectStore.cpp.

References H, llvm::cas::ObjectProxy::load(), loadIfExists(), and llvm::Ref.

◆ getReference()

virtual std::optional< ObjectRef > llvm::cas::ObjectStore::getReference ( const CASID & ID ) const

pure virtual

Get an existing reference to the object called ID.

Returns None if the object is not stored in this CAS.

Referenced by getProxy(), and llvm::cas::builtin::BuiltinCAS::validateObject().

◆ getStorageSize()

virtual Expected< std::optional< uint64_t > > llvm::cas::ObjectStore::getStorageSize ( ) const

inlinevirtual

Returns: the storage size of the on-disk CAS data.

Implementations that don't have an implementation for this should return std::nullopt.

Definition at line 260 of file ObjectStore.h.

◆ importObject()

Expected< ObjectRef > ObjectStore::importObject	(	ObjectStore &	Upstream,
		ObjectRef	Other )

Import object from another CAS.

This will import the full tree from the other CAS.

Keeps track of the state of visitation for current node and all of its parents. Upstream Cursor holds information only from upstream CAS.

PrimaryNodeStack holds the ObjectRef of the current CAS, with nodes either just stored in the CAS or nodes already exists in the current CAS.

A map from upstream ObjectRef to current ObjectRef.

Definition at line 203 of file ObjectStore.cpp.

References llvm::ArrayRef(), assert(), llvm::SmallVectorTemplateCommon< T, typename >::back(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::find(), llvm::SmallVectorTemplateCommon< T, typename >::front(), getData(), getNumRefs(), I, LLVM_UNLIKELY, load(), ObjectStore(), llvm::Other, llvm::SmallVectorTemplateBase< T, bool >::pop_back(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), readRef(), llvm::Ref, llvm::SmallVectorTemplateCommon< T, typename >::size(), store(), llvm::SmallVectorImpl< T >::truncate(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::try_emplace().

◆ isMaterialized()

virtual Expected< bool > llvm::cas::ObjectStore::isMaterialized ( ObjectRef Ref ) const

pure virtual

Returns: true if the object is directly available from the local CAS, for implementations that have this kind of distinction.

References llvm::Ref.

◆ load()

Expected< ObjectHandle > ObjectStore::load ( ObjectRef Ref )

protected

Like loadIfExists but returns an error if the object is missing.

Definition at line 60 of file ObjectStore.cpp.

References llvm::createStringError(), getID(), llvm::invalid_argument, loadIfExists(), llvm::Ref, and llvm::toString().

Referenced by importObject().

◆ loadIfExists()

virtual Expected< std::optional< ObjectHandle > > llvm::cas::ObjectStore::loadIfExists ( ObjectRef Ref )

protectedpure virtual

Load the object referenced by Ref.

Errors if the object cannot be loaded.

Returns: std::nullopt if the object is missing from the CAS.

References llvm::Ref.

Referenced by getProxyIfExists(), and load().

◆ parseID()

virtual Expected< CASID > llvm::cas::ObjectStore::parseID ( StringRef ID )

pure virtual

Get a CASID from a ID, which should have been generated by CASID::print().

This succeeds as long as validateID() would pass. The object may be unknown to this CAS instance.

TODO: Remove, and update callers to use validateID() or extractHashFromID().

Implemented in llvm::cas::builtin::BuiltinCAS.

◆ print()

virtual void llvm::cas::ObjectStore::print ( raw_ostream & ) const

inlinevirtual

Print the ObjectStore internals for debugging purpose.

Definition at line 279 of file ObjectStore.h.

Referenced by dump().

◆ pruneStorageData()

virtual Error llvm::cas::ObjectStore::pruneStorageData ( )

inlinevirtual

Prune local storage to reduce its size according to the desired size limit.

Pruning can happen concurrently with other operations.

Implementations may leave this unimplemented.

Definition at line 268 of file ObjectStore.h.

References llvm::Error::success().

◆ readData()

uint64_t llvm::cas::ObjectStore::readData	(	ObjectHandle	Node,
		raw_ostream &	OS,
		uint64_t	Offset = 0,
		uint64_t	MaxBytes = -1ULL ) const

inline

Read the data from Data into OS.

Definition at line 239 of file ObjectStore.h.

References assert(), llvm::Data, getData(), llvm::Offset, and llvm::toStringRef().

◆ readRef()

virtual ObjectRef llvm::cas::ObjectStore::readRef	(	ObjectHandle	Node,
		size_t	I ) const

protectedpure virtual

References I.

Referenced by importObject().

◆ readRefs()

void ObjectStore::readRefs	(	ObjectHandle	Node,
		SmallVectorImpl< ObjectRef > &	Refs ) const

protectedvirtual

Read all the refs from object in a SmallVector.

Definition at line 78 of file ObjectStore.cpp.

References llvm::consumeError(), forEachRef(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::Ref, and llvm::Error::success().

◆ setSizeLimit()

virtual Error llvm::cas::ObjectStore::setSizeLimit ( std::optional< uint64_t > SizeLimit )

inlinevirtual

Set the size for limiting growth of on-disk storage.

This has an effect for when the instance is closed.

Implementations may leave this unimplemented.

Definition at line 252 of file ObjectStore.h.

References SizeLimit, and llvm::Error::success().

◆ store()

virtual Expected< ObjectRef > llvm::cas::ObjectStore::store	(	ArrayRef< ObjectRef >	Refs,
		ArrayRef< char >	Data )

pure virtual

Store object into ObjectStore.

Implemented in llvm::cas::builtin::BuiltinCAS.

References llvm::Data.

Referenced by createProxy(), importObject(), storeFromOpenFileImpl(), and storeFromString().

◆ storeFromFile()

Expected< ObjectRef > ObjectStore::storeFromFile ( StringRef Path )

virtual

Stores the data of a file into ObjectStore.

An underlying implementation could perform optimizations that reduce I/O and disk space consumption.

If there are any concurrent modifications to the file, the contents in the CAS may be corrupt.

Parameters

FilePath the path of the file data.

Definition at line 138 of file ObjectStore.cpp.

References llvm::sys::fs::closeFile(), llvm::sys::fs::openNativeFileForRead(), llvm::scope_exit(), llvm::sys::sandbox::scopedDisable(), and storeFromOpenFile().

◆ storeFromOpenFile()

Expected< ObjectRef > llvm::cas::ObjectStore::storeFromOpenFile	(	sys::fs::file_t	FD,
		std::optional< sys::fs::file_status >	Status = std::nullopt )

inline

Default implementation reads FD and calls storeNode().

Does not take ownership of FD; the caller is responsible for closing it.

If Status is sent in it is to be treated as a hint. Implementations must protect against the file size potentially growing after the status was taken (i.e., they cannot assume that an mmap will be null-terminated where Status implies).

Returns the CASID and the size of the file.

Definition at line 222 of file ObjectStore.h.

References storeFromOpenFileImpl().

Referenced by storeFromFile().

◆ storeFromOpenFileImpl()

Expected< ObjectRef > ObjectStore::storeFromOpenFileImpl	(	sys::fs::file_t	FD,
		std::optional< sys::fs::file_status >	Status )

protectedvirtual

Get ObjectRef from open file.

Definition at line 126 of file ObjectStore.cpp.

References llvm::arrayRefFromStringRef(), llvm::errorCodeToError(), llvm::MemoryBuffer::getOpenFile(), Size, and store().

Referenced by storeFromOpenFile().

◆ storeFromString()

Expected< ObjectRef > llvm::cas::ObjectStore::storeFromString	(	ArrayRef< ObjectRef >	Refs,
		StringRef	String )

inline

Store object from StringRef.

Definition at line 207 of file ObjectStore.h.

References llvm::arrayRefFromStringRef(), store(), and llvm::String.

◆ validate()

virtual Error llvm::cas::ObjectStore::validate ( bool CheckHash ) const

pure virtual

Validate the entire ObjectStore.

◆ validateObject()

virtual Error llvm::cas::ObjectStore::validateObject ( const CASID & ID )

pure virtual

Validate the underlying object referred by CASID.

Implemented in llvm::cas::builtin::BuiltinCAS.

Referenced by validateTree().

◆ validateTree()

Error ObjectStore::validateTree ( ObjectRef Ref )

Validate the whole node tree.

Definition at line 179 of file ObjectStore.cpp.

References llvm::SmallVectorTemplateCommon< T, typename >::empty(), forEachRef(), getID(), I, llvm::detail::DenseSetImpl< ValueT, MapTy, ValueInfoT >::insert(), load, llvm::SmallVectorImpl< T >::pop_back_val(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::Ref, llvm::Error::success(), and validateObject().

◆ ObjectProxy

friend class ObjectProxy

friend

Definition at line 91 of file ObjectStore.h.

References ObjectProxy.

Referenced by ObjectProxy.

The documentation for this class was generated from the following files:

include/llvm/CAS/ObjectStore.h
lib/CAS/ObjectStore.cpp

Public Member Functions

Static Public Member Functions

Protected Member Functions

Friends

Detailed Description

Constructor & Destructor Documentation

◆ ~ObjectStore()

◆ ObjectStore()

Member Function Documentation

◆ createProxy()

◆ createUnknownObjectError()

◆ dump()

◆ exportDataToFile()

◆ forEachRef()

◆ getContext()

◆ getData()

◆ getDataSize()

◆ getDataString()

◆ getID()

◆ getMemoryBuffer()

◆ getNumRefs()

◆ getProxy() [1/2]

◆ getProxy() [2/2]

◆ getProxyIfExists()

◆ getReference()

◆ getStorageSize()

◆ importObject()

◆ isMaterialized()

◆ load()

◆ loadIfExists()

◆ parseID()

◆ print()

◆ pruneStorageData()

◆ readData()

◆ readRef()

◆ readRefs()

◆ setSizeLimit()

◆ store()

◆ storeFromFile()

◆ storeFromOpenFile()

◆ storeFromOpenFileImpl()

◆ storeFromString()

◆ validate()

◆ validateObject()

◆ validateTree()

Friends And Related Symbol Documentation

◆ ObjectProxy