OnDiskGraphDB.cpp File Reference

This file implements OnDiskGraphDB, an on-disk CAS nodes database, independent of a particular hashing algorithm. More...

#include "llvm/CAS/OnDiskGraphDB.h"
#include "OnDiskCommon.h"
#include "llvm/ADT/DenseMap.h"
#include "llvm/ADT/ScopeExit.h"
#include "llvm/ADT/StringExtras.h"
#include "llvm/CAS/OnDiskDataAllocator.h"
#include "llvm/CAS/OnDiskTrieRawHashMap.h"
#include "llvm/Support/Alignment.h"
#include "llvm/Support/Compiler.h"
#include "llvm/Support/Errc.h"
#include "llvm/Support/Error.h"
#include "llvm/Support/ErrorHandling.h"
#include "llvm/Support/FileSystem.h"
#include "llvm/Support/MemoryBuffer.h"
#include "llvm/Support/Path.h"
#include "llvm/Support/Process.h"
#include <atomic>
#include <mutex>
#include <optional>

Go to the source code of this file.

Classes
struct	llvm::cas::ondisk::OnDiskGraphDB::IndexProxy
	Proxy for an on-disk index record. More...

Macros
#define	DEBUG_TYPE   "on-disk-cas"

Functions
static Error	createCorruptObjectError (Expected< ArrayRef< uint8_t > > ID)
static OnDiskContent	getContentFromHandle (const OnDiskDataAllocator &DataPool, ObjectHandle OH)
static Expected< MappedTempFile >	createTempFile (StringRef FinalPath, uint64_t Size)
static size_t	getPageSize ()

Variables
static constexpr StringLiteral	IndexTableName = "llvm.cas.index"
static constexpr StringLiteral	DataPoolTableName = "llvm.cas.data"
static constexpr StringLiteral	IndexFilePrefix = "index."
static constexpr StringLiteral	DataPoolFilePrefix = "data."
static constexpr StringLiteral	FilePrefixObject = "obj."
static constexpr StringLiteral	FilePrefixLeaf = "leaf."
static constexpr StringLiteral	FilePrefixLeaf0 = "leaf+0."

Detailed Description

This file implements OnDiskGraphDB, an on-disk CAS nodes database, independent of a particular hashing algorithm.

It only needs to be configured for the hash size and controls the schema of the storage.

OnDiskGraphDB defines:

• How the data is stored inside database, either as a standalone file, or allocated inside a datapool.
• How references to other objects inside the same database is stored. They are stored as internal references, instead of full hash value to save space.
• How to chain databases together and import objects from upstream databases.

Here's a top-level description of the current layout:

• db/index.<version>: a file for the "index" table, named by IndexTableName and managed by TrieRawHashMap. The contents are 8B that are accessed atomically, describing the object kind and where/how it's stored (including an optional file offset). See TrieRecord for more details.
• db/data.<version>: a file for the "data" table, named by DataPoolTableName and managed by DataStore. New objects within TrieRecord::MaxEmbeddedSize are inserted here as TrieRecord::StorageKind::DataPool.
• db/obj.<offset>.<version>: a file storing an object outside the main "data" table, named by its offset into the "index" table, with the format of TrieRecord::StorageKind::Standalone.
• db/leaf.<offset>.<version>: a file storing a leaf node outside the main "data" table, named by its offset into the "index" table, with the format of TrieRecord::StorageKind::StandaloneLeaf.
• db/leaf+0.<offset>.<version>: a file storing a null-terminated leaf object outside the main "data" table, named by its offset into the "index" table, with the format of TrieRecord::StorageKind::StandaloneLeaf0.

Definition in file OnDiskGraphDB.cpp.

Macro Definition Documentation

DEBUG_TYPE

#define DEBUG_TYPE   "on-disk-cas"

Definition at line 67 of file OnDiskGraphDB.cpp.

Function Documentation

createCorruptObjectError()

Error createCorruptObjectError ( Expected< ArrayRef< uint8_t > > ID )

static

Definition at line 83 of file OnDiskGraphDB.cpp.

References llvm::createStringError(), llvm::invalid_argument, and llvm::toHex().

Referenced by llvm::cas::ondisk::OnDiskGraphDB::load(), and llvm::cas::ondisk::OnDiskGraphDB::store().

createTempFile()

Expected< MappedTempFile > createTempFile ( StringRef FinalPath, uint64_t Size )

static

Definition at line 1338 of file OnDiskGraphDB.cpp.

References assert(), llvm::sys::fs::convertFDToNativeFile(), llvm::createFileError(), E(), llvm::cas::ondisk::preallocateFileTail(), llvm::sys::fs::mapped_file_region::readwrite, llvm::sys::fs::resize_file_before_mapping_readwrite(), and Size.

Referenced by llvm::cas::ondisk::OnDiskGraphDB::store().

getContentFromHandle()

OnDiskContent getContentFromHandle ( const OnDiskDataAllocator & DataPool, ObjectHandle OH )

static

Definition at line 1166 of file OnDiskGraphDB.cpp.

References assert(), llvm::cantFail(), and llvm::Data.

Referenced by llvm::cas::ondisk::OnDiskGraphDB::getObjectData().

getPageSize()

size_t getPageSize ( )

static

Definition at line 1360 of file OnDiskGraphDB.cpp.

References llvm::sys::Process::getPageSizeEstimate(), and PageSize.

Variable Documentation

DataPoolFilePrefix

StringLiteral DataPoolFilePrefix = "data."

staticconstexpr

Definition at line 77 of file OnDiskGraphDB.cpp.

Referenced by llvm::cas::ondisk::OnDiskGraphDB::open().

DataPoolTableName

StringLiteral DataPoolTableName = "llvm.cas.data"

staticconstexpr

Definition at line 74 of file OnDiskGraphDB.cpp.

Referenced by llvm::cas::ondisk::OnDiskGraphDB::open().

FilePrefixLeaf

StringLiteral FilePrefixLeaf = "leaf."

staticconstexpr

Definition at line 80 of file OnDiskGraphDB.cpp.

FilePrefixLeaf0

StringLiteral FilePrefixLeaf0 = "leaf+0."

staticconstexpr

Definition at line 81 of file OnDiskGraphDB.cpp.

FilePrefixObject

StringLiteral FilePrefixObject = "obj."

staticconstexpr

Definition at line 79 of file OnDiskGraphDB.cpp.

IndexFilePrefix

StringLiteral IndexFilePrefix = "index."

staticconstexpr

Definition at line 76 of file OnDiskGraphDB.cpp.

Referenced by llvm::cas::ondisk::OnDiskGraphDB::open().

IndexTableName

StringLiteral IndexTableName = "llvm.cas.index"

staticconstexpr

Definition at line 73 of file OnDiskGraphDB.cpp.

Referenced by llvm::cas::ondisk::OnDiskGraphDB::open().