Content-addressable storage for objects. More...

#include "llvm/CAS/ObjectStore.h"

Inheritance diagram for llvm::cas::ObjectStore:

Public Member Functions
virtual Expected< CASID >	parseID (StringRef ID)=0
	Get a `CASID` from a `ID`, which should have been generated by CASID::print().
virtual Expected< ObjectRef >	store (ArrayRef< ObjectRef > Refs, ArrayRef< char > Data)=0
	Store object into ObjectStore.
virtual CASID	getID (ObjectRef Ref) const =0
	Get an ID for `Ref`.
virtual std::optional< ObjectRef >	getReference (const CASID &ID) const =0
	Get an existing reference to the object called `ID`.
virtual Expected< bool >	isMaterialized (ObjectRef Ref) const =0
virtual Error	validate (const CASID &ID)=0
	Validate the underlying object referred by CASID.
Expected< ObjectProxy >	createProxy (ArrayRef< ObjectRef > Refs, StringRef Data)
	Helper functions to store object and returns a ObjectProxy.
Expected< ObjectRef >	storeFromString (ArrayRef< ObjectRef > Refs, StringRef String)
	Store object from StringRef.
Expected< ObjectRef >	storeFromOpenFile (sys::fs::file_t FD, std::optional< sys::fs::file_status > Status=std::nullopt)
	Default implementation reads `FD` and calls storeNode().
Expected< ObjectProxy >	getProxy (const CASID &ID)
	Create ObjectProxy from CASID. If the object doesn't exist, get an error.
Expected< ObjectProxy >	getProxy (ObjectRef Ref)
	Create ObjectProxy from ObjectRef.
Expected< std::optional< ObjectProxy > >	getProxyIfExists (ObjectRef Ref)
uint64_t	readData (ObjectHandle Node, raw_ostream &OS, uint64_t Offset=0, uint64_t MaxBytes=-1ULL) const
	Read the data from `Data` into `OS`.
Error	validateTree (ObjectRef Ref)
	Validate the whole node tree.
const CASContext &	getContext () const
	Get CASContext.
virtual	~ObjectStore ()=default

Static Public Member Functions
static Error	createUnknownObjectError (const CASID &ID)

Protected Member Functions
virtual Expected< std::optional< ObjectHandle > >	loadIfExists (ObjectRef Ref)=0
	Load the object referenced by `Ref`.
Expected< ObjectHandle >	load (ObjectRef Ref)
	Like `loadIfExists` but returns an error if the object is missing.
virtual uint64_t	getDataSize (ObjectHandle Node) const =0
	Get the size of some data.
virtual Error	forEachRef (ObjectHandle Node, function_ref< Error(ObjectRef)> Callback) const =0
	Methods for handling objects.
virtual ObjectRef	readRef (ObjectHandle Node, size_t I) const =0
virtual size_t	getNumRefs (ObjectHandle Node) const =0
virtual ArrayRef< char >	getData (ObjectHandle Node, bool RequiresNullTerminator=false) const =0
virtual Expected< ObjectRef >	storeFromOpenFileImpl (sys::fs::file_t FD, std::optional< sys::fs::file_status > Status)
	Get ObjectRef from open file.
StringRef	getDataString (ObjectHandle Node)
	Get a lifetime-extended StringRef pointing at `Data`.
std::unique_ptr< MemoryBuffer >	getMemoryBuffer (ObjectHandle Node, StringRef Name="", bool RequiresNullTerminator=true)
	Get a lifetime-extended MemoryBuffer pointing at `Data`.
virtual void	readRefs (ObjectHandle Node, SmallVectorImpl< ObjectRef > &Refs) const
	Read all the refs from object in a SmallVector.
	ObjectStore (const CASContext &Context)

Friends
class	ObjectProxy

Detailed Description

Content-addressable storage for objects.

Conceptually, objects are stored in a "unique set".

Objects are immutable ("value objects") that are defined by their content. They are implicitly deduplicated by content.
Each object has a unique identifier (UID) that's derived from its content, called a CASID.
- This UID is a fixed-size (strong) hash of the transitive content of a CAS object.
- It's comparable between any two CAS instances that have the same CASIDContext::getHashSchemaIdentifier().
- The UID can be printed (e.g., CASID::toString()) and it can parsed by the same or a different CAS instance with ObjectStore::parseID().
An object can be looked up by content or by UID.
- store() is "get-or-create" methods, writing an object if it doesn't exist yet, and return a ref to it in any case.
- loadObject(const CASID&) looks up an object by its UID.
Objects can reference other objects, forming an arbitrary DAG.

The ObjectStore interface has a few ways of referencing objects:

ObjectRef encapsulates a reference to something in the CAS. It is an opaque type that references an object inside a specific CAS. It is implementation defined if the underlying object exists or not for an ObjectRef, and it can used to speed up CAS lookup as an implementation detail. However, you don't know anything about the underlying objects. "Loading" the object is a separate step that may not have happened yet, and which can fail (e.g. due to filesystem corruption) or introduce latency (if downloading from a remote store).
ObjectHandle encapulates a loaded object in the CAS. You need one of these to inspect the content of an object: to look at its stored data and references. This is internal to CAS implementation and not availble from CAS public APIs.
CASID: the UID for an object in the CAS, obtained through ObjectStore::getID() or ObjectStore::parseID(). This is a valid CAS identifier, but may reference an object that is unknown to this CAS instance.
ObjectProxy pairs an ObjectHandle (subclass) with a ObjectStore, and wraps access APIs to avoid having to pass extra parameters. It is the object used for accessing underlying data and refs by CAS users.

Both ObjectRef and ObjectHandle are lightweight, wrapping a uint64_t and are only valid with the associated ObjectStore instance.

There are a few options for accessing content of objects, with different lifetime tradeoffs:

getData() accesses data without exposing lifetime at all.
getMemoryBuffer() returns a MemoryBuffer whose lifetime is independent of the CAS (it can live longer).
getDataString() return StringRef with lifetime is guaranteed to last as long as ObjectStore.
readRef() and forEachRef() iterate through the references in an object. There is no lifetime assumption.

Definition at line 85 of file ObjectStore.h.

Constructor & Destructor Documentation

◆ ~ObjectStore()

virtual llvm::cas::ObjectStore::~ObjectStore ( )

virtualdefault

◆ ObjectStore()

llvm::cas::ObjectStore::ObjectStore ( const CASContext & Context )

inlineprotected

Definition at line 227 of file ObjectStore.h.

Referenced by llvm::cas::builtin::BuiltinCAS::BuiltinCAS().

Member Function Documentation

◆ createProxy()

Expected< ObjectProxy > ObjectStore::createProxy	(	ArrayRef< ObjectRef >	Refs,
		StringRef	Data )

Helper functions to store object and returns a ObjectProxy.

Definition at line 113 of file ObjectStore.cpp.

References llvm::arrayRefFromStringRef(), llvm::Data, getProxy(), llvm::Ref, and store().

◆ createUnknownObjectError()

Error ObjectStore::createUnknownObjectError ( const CASID & ID )

static

Definition at line 108 of file ObjectStore.cpp.

References llvm::createStringError().

Referenced by getProxy().

◆ forEachRef()

virtual Error llvm::cas::ObjectStore::forEachRef	(	ObjectHandle	Node,
		function_ref< Error(ObjectRef)>	Callback ) const

protectedpure virtual

Methods for handling objects.

CAS implementations need to override to provide functions to access stored CAS objects and references.

Referenced by readRefs(), and validateTree().

◆ getContext()

const CASContext & llvm::cas::ObjectStore::getContext ( ) const

inline

Get CASContext.

Definition at line 222 of file ObjectStore.h.

Referenced by llvm::cas::builtin::BuiltinCAS::parseID().

◆ getData()

virtual ArrayRef< char > llvm::cas::ObjectStore::getData	(	ObjectHandle	Node,
		bool	RequiresNullTerminator = false ) const

protectedpure virtual

Implemented in llvm::cas::builtin::BuiltinCAS.

Referenced by getDataString(), getMemoryBuffer(), and readData().

◆ getDataSize()

virtual uint64_t llvm::cas::ObjectStore::getDataSize ( ObjectHandle Node ) const

protectedpure virtual

Get the size of some data.

Implemented in llvm::cas::builtin::BuiltinCAS.

◆ getDataString()

StringRef llvm::cas::ObjectStore::getDataString ( ObjectHandle Node )

inlineprotected

Get a lifetime-extended StringRef pointing at Data.

Depending on the CAS implementation, this may involve in-memory storage overhead.

Definition at line 147 of file ObjectStore.h.

References getData(), and llvm::toStringRef().

◆ getID()

virtual CASID llvm::cas::ObjectStore::getID ( ObjectRef Ref ) const

pure virtual

Get an ID for Ref.

References llvm::Ref.

Referenced by llvm::cas::ObjectProxy::getReferenceID(), load(), and validateTree().

◆ getMemoryBuffer()

std::unique_ptr< MemoryBuffer > ObjectStore::getMemoryBuffer	(	ObjectHandle	Node,
		StringRef	Name = "",
		bool	RequiresNullTerminator = true )

protected

Get a lifetime-extended MemoryBuffer pointing at Data.

Depending on the CAS implementation, this may involve in-memory storage overhead.

Definition at line 67 of file ObjectStore.cpp.

References getData(), llvm::MemoryBuffer::getMemBuffer(), and llvm::toStringRef().

◆ getNumRefs()

virtual size_t llvm::cas::ObjectStore::getNumRefs ( ObjectHandle Node ) const

protectedpure virtual

◆ getProxy() [1/2]

Expected< ObjectProxy > ObjectStore::getProxy ( const CASID & ID )

Create ObjectProxy from CASID. If the object doesn't exist, get an error.

Definition at line 82 of file ObjectStore.cpp.

References createUnknownObjectError(), getProxy(), getReference(), and llvm::Ref.

Referenced by createProxy(), and getProxy().

◆ getProxy() [2/2]

Expected< ObjectProxy > ObjectStore::getProxy ( ObjectRef Ref )

Create ObjectProxy from ObjectRef.

If the object can't be loaded, get an error.

Definition at line 90 of file ObjectStore.cpp.

References H, llvm::cas::ObjectProxy::load(), load, and llvm::Ref.

◆ getProxyIfExists()

Expected< std::optional< ObjectProxy > > ObjectStore::getProxyIfExists ( ObjectRef Ref )

Returns: std::nullopt if the object is missing from the CAS.

Definition at line 99 of file ObjectStore.cpp.

References H, llvm::cas::ObjectProxy::load(), loadIfExists(), and llvm::Ref.

◆ getReference()

virtual std::optional< ObjectRef > llvm::cas::ObjectStore::getReference ( const CASID & ID ) const

pure virtual

Get an existing reference to the object called ID.

Returns None if the object is not stored in this CAS.

Referenced by getProxy(), and llvm::cas::builtin::BuiltinCAS::validate().

◆ isMaterialized()

virtual Expected< bool > llvm::cas::ObjectStore::isMaterialized ( ObjectRef Ref ) const

pure virtual

Returns: true if the object is directly available from the local CAS, for implementations that have this kind of distinction.

References llvm::Ref.

◆ load()

Expected< ObjectHandle > ObjectStore::load ( ObjectRef Ref )

protected

Like loadIfExists but returns an error if the object is missing.

Definition at line 56 of file ObjectStore.cpp.

References llvm::createStringError(), getID(), llvm::invalid_argument, loadIfExists(), llvm::Ref, and llvm::toString().

◆ loadIfExists()

virtual Expected< std::optional< ObjectHandle > > llvm::cas::ObjectStore::loadIfExists ( ObjectRef Ref )

protectedpure virtual

Load the object referenced by Ref.

Errors if the object cannot be loaded.

Returns: std::nullopt if the object is missing from the CAS.

References llvm::Ref.

Referenced by getProxyIfExists(), and load().

◆ parseID()

virtual Expected< CASID > llvm::cas::ObjectStore::parseID ( StringRef ID )

pure virtual

Get a CASID from a ID, which should have been generated by CASID::print().

This succeeds as long as validateID() would pass. The object may be unknown to this CAS instance.

TODO: Remove, and update callers to use validateID() or extractHashFromID().

Implemented in llvm::cas::builtin::BuiltinCAS.

◆ readData()

uint64_t llvm::cas::ObjectStore::readData	(	ObjectHandle	Node,
		raw_ostream &	OS,
		uint64_t	Offset = 0,
		uint64_t	MaxBytes = -1ULL ) const

inline

Read the data from Data into OS.

Definition at line 209 of file ObjectStore.h.

References assert(), llvm::Data, getData(), llvm::Offset, and llvm::toStringRef().

◆ readRef()

virtual ObjectRef llvm::cas::ObjectStore::readRef	(	ObjectHandle	Node,
		size_t	I ) const

protectedpure virtual

References I.

◆ readRefs()

void ObjectStore::readRefs	(	ObjectHandle	Node,
		SmallVectorImpl< ObjectRef > &	Refs ) const

protectedvirtual

Read all the refs from object in a SmallVector.

Definition at line 74 of file ObjectStore.cpp.

References llvm::consumeError(), forEachRef(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::Ref, and llvm::Error::success().

◆ store()

virtual Expected< ObjectRef > llvm::cas::ObjectStore::store	(	ArrayRef< ObjectRef >	Refs,
		ArrayRef< char >	Data )

pure virtual

Store object into ObjectStore.

Implemented in llvm::cas::builtin::BuiltinCAS.

References llvm::Data.

Referenced by createProxy(), storeFromOpenFileImpl(), and storeFromString().

◆ storeFromOpenFile()

Expected< ObjectRef > llvm::cas::ObjectStore::storeFromOpenFile	(	sys::fs::file_t	FD,
		std::optional< sys::fs::file_status >	Status = std::nullopt )

inline

Default implementation reads FD and calls storeNode().

Does not take ownership of FD; the caller is responsible for closing it.

If Status is sent in it is to be treated as a hint. Implementations must protect against the file size potentially growing after the status was taken (i.e., they cannot assume that an mmap will be null-terminated where Status implies).

Returns the CASID and the size of the file.

Definition at line 192 of file ObjectStore.h.

References storeFromOpenFileImpl().

◆ storeFromOpenFileImpl()

Expected< ObjectRef > ObjectStore::storeFromOpenFileImpl	(	sys::fs::file_t	FD,
		std::optional< sys::fs::file_status >	Status )

protectedvirtual

Get ObjectRef from open file.

Definition at line 122 of file ObjectStore.cpp.

References llvm::arrayRefFromStringRef(), llvm::errorCodeToError(), llvm::MemoryBuffer::getOpenFile(), Size, and store().

Referenced by storeFromOpenFile().

◆ storeFromString()

Expected< ObjectRef > llvm::cas::ObjectStore::storeFromString	(	ArrayRef< ObjectRef >	Refs,
		StringRef	String )

inline

Store object from StringRef.

Definition at line 177 of file ObjectStore.h.

References llvm::arrayRefFromStringRef(), store(), and llvm::String.

◆ validate()

virtual Error llvm::cas::ObjectStore::validate ( const CASID & ID )

pure virtual

Validate the underlying object referred by CASID.

Implemented in llvm::cas::builtin::BuiltinCAS.

Referenced by validateTree().

◆ validateTree()

Error ObjectStore::validateTree ( ObjectRef Ref )

Validate the whole node tree.

Definition at line 134 of file ObjectStore.cpp.

References llvm::SmallVectorTemplateCommon< T, typename >::empty(), forEachRef(), getID(), I, llvm::detail::DenseSetImpl< ValueT, MapTy, ValueInfoT >::insert(), load, llvm::SmallVectorImpl< T >::pop_back_val(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::Ref, llvm::Error::success(), llvm::Expected< T >::takeError(), and validate().

◆ ObjectProxy

friend class ObjectProxy

friend

Definition at line 86 of file ObjectStore.h.

References ObjectProxy.

Referenced by ObjectProxy.

The documentation for this class was generated from the following files:

include/llvm/CAS/ObjectStore.h
lib/CAS/ObjectStore.cpp

Public Member Functions

Static Public Member Functions

Protected Member Functions

Friends

Detailed Description

Constructor & Destructor Documentation

◆ ~ObjectStore()

◆ ObjectStore()

Member Function Documentation

◆ createProxy()

◆ createUnknownObjectError()

◆ forEachRef()

◆ getContext()

◆ getData()

◆ getDataSize()

◆ getDataString()

◆ getID()

◆ getMemoryBuffer()

◆ getNumRefs()

◆ getProxy() [1/2]

◆ getProxy() [2/2]

◆ getProxyIfExists()

◆ getReference()

◆ isMaterialized()

◆ load()

◆ loadIfExists()

◆ parseID()

◆ readData()

◆ readRef()

◆ readRefs()

◆ store()

◆ storeFromOpenFile()

◆ storeFromOpenFileImpl()

◆ storeFromString()

◆ validate()

◆ validateTree()

Friends And Related Symbol Documentation

◆ ObjectProxy