LLVM 18.0.0git
GsymCreator.h
Go to the documentation of this file.
1//===- GsymCreator.h --------------------------------------------*- C++ -*-===//
2//
3// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
4// See https://llvm.org/LICENSE.txt for license information.
5// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
6//
7//===----------------------------------------------------------------------===//
8
9#ifndef LLVM_DEBUGINFO_GSYM_GSYMCREATOR_H
10#define LLVM_DEBUGINFO_GSYM_GSYMCREATOR_H
11
12#include <functional>
13#include <memory>
14#include <mutex>
15#include <thread>
16
18#include "llvm/ADT/ArrayRef.h"
19#include "llvm/ADT/StringSet.h"
23#include "llvm/Support/Endian.h"
24#include "llvm/Support/Error.h"
25#include "llvm/Support/Path.h"
26
27namespace llvm {
28
29namespace gsym {
30class FileWriter;
31
32/// GsymCreator is used to emit GSYM data to a stand alone file or section
33/// within a file.
34///
35/// The GsymCreator is designed to be used in 3 stages:
36/// - Create FunctionInfo objects and add them
37/// - Finalize the GsymCreator object
38/// - Save to file or section
39///
40/// The first stage involves creating FunctionInfo objects from another source
41/// of information like compiler debug info metadata, DWARF or Breakpad files.
42/// Any strings in the FunctionInfo or contained information, like InlineInfo
43/// or LineTable objects, should get the string table offsets by calling
44/// GsymCreator::insertString(...). Any file indexes that are needed should be
45/// obtained by calling GsymCreator::insertFile(...). All of the function calls
46/// in GsymCreator are thread safe. This allows multiple threads to create and
47/// add FunctionInfo objects while parsing debug information.
48///
49/// Once all of the FunctionInfo objects have been added, the
50/// GsymCreator::finalize(...) must be called prior to saving. This function
51/// will sort the FunctionInfo objects, finalize the string table, and do any
52/// other passes on the information needed to prepare the information to be
53/// saved.
54///
55/// Once the object has been finalized, it can be saved to a file or section.
56///
57/// ENCODING
58///
59/// GSYM files are designed to be memory mapped into a process as shared, read
60/// only data, and used as is.
61///
62/// The GSYM file format when in a stand alone file consists of:
63/// - Header
64/// - Address Table
65/// - Function Info Offsets
66/// - File Table
67/// - String Table
68/// - Function Info Data
69///
70/// HEADER
71///
72/// The header is fully described in "llvm/DebugInfo/GSYM/Header.h".
73///
74/// ADDRESS TABLE
75///
76/// The address table immediately follows the header in the file and consists
77/// of Header.NumAddresses address offsets. These offsets are sorted and can be
78/// binary searched for efficient lookups. Addresses in the address table are
79/// stored as offsets from a 64 bit base address found in Header.BaseAddress.
80/// This allows the address table to contain 8, 16, or 32 offsets. This allows
81/// the address table to not require full 64 bit addresses for each address.
82/// The resulting GSYM size is smaller and causes fewer pages to be touched
83/// during address lookups when the address table is smaller. The size of the
84/// address offsets in the address table is specified in the header in
85/// Header.AddrOffSize. The first offset in the address table is aligned to
86/// Header.AddrOffSize alignment to ensure efficient access when loaded into
87/// memory.
88///
89/// FUNCTION INFO OFFSETS TABLE
90///
91/// The function info offsets table immediately follows the address table and
92/// consists of Header.NumAddresses 32 bit file offsets: one for each address
93/// in the address table. This data is aligned to a 4 byte boundary. The
94/// offsets in this table are the relative offsets from the start offset of the
95/// GSYM header and point to the function info data for each address in the
96/// address table. Keeping this data separate from the address table helps to
97/// reduce the number of pages that are touched when address lookups occur on a
98/// GSYM file.
99///
100/// FILE TABLE
101///
102/// The file table immediately follows the function info offsets table. The
103/// encoding of the FileTable is:
104///
105/// struct FileTable {
106/// uint32_t Count;
107/// FileEntry Files[];
108/// };
109///
110/// The file table starts with a 32 bit count of the number of files that are
111/// used in all of the function info, followed by that number of FileEntry
112/// structures. The file table is aligned to a 4 byte boundary, Each file in
113/// the file table is represented with a FileEntry structure.
114/// See "llvm/DebugInfo/GSYM/FileEntry.h" for details.
115///
116/// STRING TABLE
117///
118/// The string table follows the file table in stand alone GSYM files and
119/// contains all strings for everything contained in the GSYM file. Any string
120/// data should be added to the string table and any references to strings
121/// inside GSYM information must be stored as 32 bit string table offsets into
122/// this string table. The string table always starts with an empty string at
123/// offset zero and is followed by any strings needed by the GSYM information.
124/// The start of the string table is not aligned to any boundary.
125///
126/// FUNCTION INFO DATA
127///
128/// The function info data is the payload that contains information about the
129/// address that is being looked up. It contains all of the encoded
130/// FunctionInfo objects. Each encoded FunctionInfo's data is pointed to by an
131/// entry in the Function Info Offsets Table. For details on the exact encoding
132/// of FunctionInfo objects, see "llvm/DebugInfo/GSYM/FunctionInfo.h".
134 // Private member variables require Mutex protections
135 mutable std::mutex Mutex;
136 std::vector<FunctionInfo> Funcs;
137 StringTableBuilder StrTab;
138 StringSet<> StringStorage;
140 // Needed for mapping string offsets back to the string stored in \a StrTab.
142 std::vector<llvm::gsym::FileEntry> Files;
143 std::vector<uint8_t> UUID;
144 std::optional<AddressRanges> ValidTextRanges;
145 std::optional<uint64_t> BaseAddress;
146 bool IsSegment = false;
147 bool Finalized = false;
148 bool Quiet;
149
150
151 /// Get the first function start address.
152 ///
153 /// \returns The start address of the first FunctionInfo or std::nullopt if
154 /// there are no function infos.
155 std::optional<uint64_t> getFirstFunctionAddress() const;
156
157 /// Get the last function address.
158 ///
159 /// \returns The start address of the last FunctionInfo or std::nullopt if
160 /// there are no function infos.
161 std::optional<uint64_t> getLastFunctionAddress() const;
162
163 /// Get the base address to use for this GSYM file.
164 ///
165 /// \returns The base address to put into the header and to use when creating
166 /// the address offset table or std::nullpt if there are no valid
167 /// function infos or if the base address wasn't specified.
168 std::optional<uint64_t> getBaseAddress() const;
169
170 /// Get the size of an address offset in the address offset table.
171 ///
172 /// GSYM files store offsets from the base address in the address offset table
173 /// and we store the size of the address offsets in the GSYM header. This
174 /// function will calculate the size in bytes of these address offsets based
175 /// on the current contents of the GSYM file.
176 ///
177 /// \returns The size in byets of the address offsets.
178 uint8_t getAddressOffsetSize() const;
179
180 /// Get the maximum address offset for the current address offset size.
181 ///
182 /// This is used when creating the address offset table to ensure we have
183 /// values that are in range so we don't end up truncating address offsets
184 /// when creating GSYM files as the code evolves.
185 ///
186 /// \returns The maximum address offset value that will be encoded into a GSYM
187 /// file.
188 uint64_t getMaxAddressOffset() const;
189
190 /// Calculate the byte size of the GSYM header and tables sizes.
191 ///
192 /// This function will calculate the exact size in bytes of the encocded GSYM
193 /// for the following items:
194 /// - The GSYM header
195 /// - The Address offset table
196 /// - The Address info offset table
197 /// - The file table
198 /// - The string table
199 ///
200 /// This is used to help split GSYM files into segments.
201 ///
202 /// \returns Size in bytes the GSYM header and tables.
203 uint64_t calculateHeaderAndTableSize() const;
204
205 /// Copy a FunctionInfo from the \a SrcGC GSYM creator into this creator.
206 ///
207 /// Copy the function info and only the needed files and strings and add a
208 /// converted FunctionInfo into this object. This is used to segment GSYM
209 /// files into separate files while only transferring the files and strings
210 /// that are needed from \a SrcGC.
211 ///
212 /// \param SrcGC The source gsym creator to copy from.
213 /// \param FuncInfoIdx The function info index within \a SrcGC to copy.
214 /// \returns The number of bytes it will take to encode the function info in
215 /// this GsymCreator. This helps calculate the size of the current GSYM
216 /// segment file.
217 uint64_t copyFunctionInfo(const GsymCreator &SrcGC, size_t FuncInfoIdx);
218
219 /// Copy a string from \a SrcGC into this object.
220 ///
221 /// Copy a string from \a SrcGC by string table offset into this GSYM creator.
222 /// If a string has already been copied, the uniqued string table offset will
223 /// be returned, otherwise the string will be copied and a unique offset will
224 /// be returned.
225 ///
226 /// \param SrcGC The source gsym creator to copy from.
227 /// \param StrOff The string table offset from \a SrcGC to copy.
228 /// \returns The new string table offset of the string within this object.
229 uint32_t copyString(const GsymCreator &SrcGC, uint32_t StrOff);
230
231 /// Copy a file from \a SrcGC into this object.
232 ///
233 /// Copy a file from \a SrcGC by file index into this GSYM creator. Files
234 /// consist of two string table entries, one for the directory and one for the
235 /// filename, this function will copy any needed strings ensure the file is
236 /// uniqued within this object. If a file already exists in this GSYM creator
237 /// the uniqued index will be returned, else the stirngs will be copied and
238 /// the new file index will be returned.
239 ///
240 /// \param SrcGC The source gsym creator to copy from.
241 /// \param FileIdx The 1 based file table index within \a SrcGC to copy. A
242 /// file index of zero will always return zero as the zero is a reserved file
243 /// index that means no file.
244 /// \returns The new file index of the file within this object.
245 uint32_t copyFile(const GsymCreator &SrcGC, uint32_t FileIdx);
246
247 /// Inserts a FileEntry into the file table.
248 ///
249 /// This is used to insert a file entry in a thread safe way into this object.
250 ///
251 /// \param FE A file entry object that contains valid string table offsets
252 /// from this object already.
253 uint32_t insertFileEntry(FileEntry FE);
254
255 /// Fixup any string and file references by updating any file indexes and
256 /// strings offsets in the InlineInfo parameter.
257 ///
258 /// When copying InlineInfo entries, we can simply make a copy of the object
259 /// and then fixup the files and strings for efficiency.
260 ///
261 /// \param SrcGC The source gsym creator to copy from.
262 /// \param II The inline info that contains file indexes and string offsets
263 /// that come from \a SrcGC. The entries will be updated by coping any files
264 /// and strings over into this object.
265 void fixupInlineInfo(const GsymCreator &SrcGC, InlineInfo &II);
266
267 /// Save this GSYM file into segments that are roughly \a SegmentSize in size.
268 ///
269 /// When segemented GSYM files are saved to disk, they will use \a Path as a
270 /// prefix and then have the first function info address appended to the path
271 /// when each segment is saved. Each segmented GSYM file has a only the
272 /// strings and files that are needed to save the function infos that are in
273 /// each segment. These smaller files are easy to compress and download
274 /// separately and allow for efficient lookups with very large GSYM files and
275 /// segmenting them allows servers to download only the segments that are
276 /// needed.
277 ///
278 /// \param Path The path prefix to use when saving the GSYM files.
279 /// \param ByteOrder The endianness to use when saving the file.
280 /// \param SegmentSize The size in bytes to segment the GSYM file into.
281 llvm::Error saveSegments(StringRef Path, llvm::endianness ByteOrder,
282 uint64_t SegmentSize) const;
283
284 /// Let this creator know that this is a segment of another GsymCreator.
285 ///
286 /// When we have a segment, we know that function infos will be added in
287 /// ascending address range order without having to be finalized. We also
288 /// don't need to sort and unique entries during the finalize function call.
289 void setIsSegment() {
290 IsSegment = true;
291 }
292
293public:
294 GsymCreator(bool Quiet = false);
295
296 /// Save a GSYM file to a stand alone file.
297 ///
298 /// \param Path The file path to save the GSYM file to.
299 /// \param ByteOrder The endianness to use when saving the file.
300 /// \param SegmentSize The size in bytes to segment the GSYM file into. If
301 /// this option is set this function will create N segments
302 /// that are all around \a SegmentSize bytes in size. This
303 /// allows a very large GSYM file to be broken up into
304 /// shards. Each GSYM file will have its own file table,
305 /// and string table that only have the files and strings
306 /// needed for the shared. If this argument has no value,
307 /// a single GSYM file that contains all function
308 /// information will be created.
309 /// \returns An error object that indicates success or failure of the save.
311 std::optional<uint64_t> SegmentSize = std::nullopt) const;
312
313 /// Encode a GSYM into the file writer stream at the current position.
314 ///
315 /// \param O The stream to save the binary data to
316 /// \returns An error object that indicates success or failure of the save.
317 llvm::Error encode(FileWriter &O) const;
318
319 /// Insert a string into the GSYM string table.
320 ///
321 /// All strings used by GSYM files must be uniqued by adding them to this
322 /// string pool and using the returned offset for any string values.
323 ///
324 /// \param S The string to insert into the string table.
325 /// \param Copy If true, then make a backing copy of the string. If false,
326 /// the string is owned by another object that will stay around
327 /// long enough for the GsymCreator to save the GSYM file.
328 /// \returns The unique 32 bit offset into the string table.
329 uint32_t insertString(StringRef S, bool Copy = true);
330
331 /// Insert a file into this GSYM creator.
332 ///
333 /// Inserts a file by adding a FileEntry into the "Files" member variable if
334 /// the file has not already been added. The file path is split into
335 /// directory and filename which are both added to the string table. This
336 /// allows paths to be stored efficiently by reusing the directories that are
337 /// common between multiple files.
338 ///
339 /// \param Path The path to the file to insert.
340 /// \param Style The path style for the "Path" parameter.
341 /// \returns The unique file index for the inserted file.
344
345 /// Add a function info to this GSYM creator.
346 ///
347 /// All information in the FunctionInfo object must use the
348 /// GsymCreator::insertString(...) function when creating string table
349 /// offsets for names and other strings.
350 ///
351 /// \param FI The function info object to emplace into our functions list.
352 void addFunctionInfo(FunctionInfo &&FI);
353
354 /// Finalize the data in the GSYM creator prior to saving the data out.
355 ///
356 /// Finalize must be called after all FunctionInfo objects have been added
357 /// and before GsymCreator::save() is called.
358 ///
359 /// \param OS Output stream to report duplicate function infos, overlapping
360 /// function infos, and function infos that were merged or removed.
361 /// \returns An error object that indicates success or failure of the
362 /// finalize.
364
365 /// Set the UUID value.
366 ///
367 /// \param UUIDBytes The new UUID bytes.
369 UUID.assign(UUIDBytes.begin(), UUIDBytes.end());
370 }
371
372 /// Thread safe iteration over all function infos.
373 ///
374 /// \param Callback A callback function that will get called with each
375 /// FunctionInfo. If the callback returns false, stop iterating.
377 std::function<bool(FunctionInfo &)> const &Callback);
378
379 /// Thread safe const iteration over all function infos.
380 ///
381 /// \param Callback A callback function that will get called with each
382 /// FunctionInfo. If the callback returns false, stop iterating.
384 std::function<bool(const FunctionInfo &)> const &Callback) const;
385
386 /// Get the current number of FunctionInfo objects contained in this
387 /// object.
388 size_t getNumFunctionInfos() const;
389
390 /// Set valid .text address ranges that all functions must be contained in.
392 ValidTextRanges = TextRanges;
393 }
394
395 /// Get the valid text ranges.
396 const std::optional<AddressRanges> GetValidTextRanges() const {
397 return ValidTextRanges;
398 }
399
400 /// Check if an address is a valid code address.
401 ///
402 /// Any functions whose addresses do not exist within these function bounds
403 /// will not be converted into the final GSYM. This allows the object file
404 /// to figure out the valid file address ranges of all the code sections
405 /// and ensure we don't add invalid functions to the final output. Many
406 /// linkers have issues when dead stripping functions from DWARF debug info
407 /// where they set the DW_AT_low_pc to zero, but newer DWARF has the
408 /// DW_AT_high_pc as an offset from the DW_AT_low_pc and these size
409 /// attributes have no relocations that can be applied. This results in DWARF
410 /// where many functions have an DW_AT_low_pc of zero and a valid offset size
411 /// for DW_AT_high_pc. If we extract all valid ranges from an object file
412 /// that are marked with executable permissions, we can properly ensure that
413 /// these functions are removed.
414 ///
415 /// \param Addr An address to check.
416 ///
417 /// \returns True if the address is in the valid text ranges or if no valid
418 /// text ranges have been set, false otherwise.
419 bool IsValidTextAddress(uint64_t Addr) const;
420
421 /// Set the base address to use for the GSYM file.
422 ///
423 /// Setting the base address to use for the GSYM file. Object files typically
424 /// get loaded from a base address when the OS loads them into memory. Using
425 /// GSYM files for symbolication becomes easier if the base address in the
426 /// GSYM header is the same address as it allows addresses to be easily slid
427 /// and allows symbolication without needing to find the original base
428 /// address in the original object file.
429 ///
430 /// \param Addr The address to use as the base address of the GSYM file
431 /// when it is saved to disk.
433 BaseAddress = Addr;
434 }
435
436 /// Whether the transformation should be quiet, i.e. not output warnings.
437 bool isQuiet() const { return Quiet; }
438
439
440 /// Create a segmented GSYM creator starting with function info index
441 /// \a FuncIdx.
442 ///
443 /// This function will create a GsymCreator object that will encode into
444 /// roughly \a SegmentSize bytes and return it. It is used by the private
445 /// saveSegments(...) function and also is used by the GSYM unit tests to test
446 /// segmenting of GSYM files. The returned GsymCreator can be finalized and
447 /// encoded.
448 ///
449 /// \param [in] SegmentSize The size in bytes to roughly segment the GSYM file
450 /// into.
451 /// \param [in,out] FuncIdx The index of the first function info to encode
452 /// into the returned GsymCreator. This index will be updated so it can be
453 /// used in subsequent calls to this function to allow more segments to be
454 /// created.
455 /// \returns An expected unique pointer to a GsymCreator or an error. The
456 /// returned unique pointer can be NULL if there are no more functions to
457 /// encode.
459 createSegment(uint64_t SegmentSize, size_t &FuncIdx) const;
460};
461
462} // namespace gsym
463} // namespace llvm
464
465#endif // LLVM_DEBUGINFO_GSYM_GSYMCREATOR_H
arc branch finalize
uint64_t Addr
raw_pwrite_stream & OS
StringSet - A set-like wrapper for the StringMap.
std::pair< llvm::MachO::Target, std::string > UUID
The AddressRanges class helps normalize address range collections.
ArrayRef - Represent a constant reference to an array (0 or more elements consecutively in memory),...
Definition: ArrayRef.h:41
iterator end() const
Definition: ArrayRef.h:154
iterator begin() const
Definition: ArrayRef.h:153
Lightweight error class with error context and mandatory checking.
Definition: Error.h:160
Tagged union holding either a T or a Error.
Definition: Error.h:474
StringRef - Represent a constant reference to a string, i.e.
Definition: StringRef.h:50
StringSet - A wrapper for StringMap that provides set-like functionality.
Definition: StringSet.h:23
Utility for building string tables with deduplicated suffixes.
A simplified binary data writer class that doesn't require targets, target definitions,...
Definition: FileWriter.h:29
GsymCreator is used to emit GSYM data to a stand alone file or section within a file.
Definition: GsymCreator.h:133
void addFunctionInfo(FunctionInfo &&FI)
Add a function info to this GSYM creator.
uint32_t insertString(StringRef S, bool Copy=true)
Insert a string into the GSYM string table.
void setBaseAddress(uint64_t Addr)
Set the base address to use for the GSYM file.
Definition: GsymCreator.h:432
const std::optional< AddressRanges > GetValidTextRanges() const
Get the valid text ranges.
Definition: GsymCreator.h:396
llvm::Expected< std::unique_ptr< GsymCreator > > createSegment(uint64_t SegmentSize, size_t &FuncIdx) const
Create a segmented GSYM creator starting with function info index FuncIdx.
llvm::Error save(StringRef Path, llvm::endianness ByteOrder, std::optional< uint64_t > SegmentSize=std::nullopt) const
Save a GSYM file to a stand alone file.
Definition: GsymCreator.cpp:64
llvm::Error encode(FileWriter &O) const
Encode a GSYM into the file writer stream at the current position.
Definition: GsymCreator.cpp:76
bool isQuiet() const
Whether the transformation should be quiet, i.e. not output warnings.
Definition: GsymCreator.h:437
void setUUID(llvm::ArrayRef< uint8_t > UUIDBytes)
Set the UUID value.
Definition: GsymCreator.h:368
void SetValidTextRanges(AddressRanges &TextRanges)
Set valid .text address ranges that all functions must be contained in.
Definition: GsymCreator.h:391
uint32_t insertFile(StringRef Path, sys::path::Style Style=sys::path::Style::native)
Insert a file into this GSYM creator.
Definition: GsymCreator.cpp:28
size_t getNumFunctionInfos() const
Get the current number of FunctionInfo objects contained in this object.
bool IsValidTextAddress(uint64_t Addr) const
Check if an address is a valid code address.
void forEachFunctionInfo(std::function< bool(FunctionInfo &)> const &Callback)
Thread safe iteration over all function infos.
This class implements an extremely fast bulk output stream that can only output to a stream.
Definition: raw_ostream.h:52
This is an optimization pass for GlobalISel generic memory operations.
Definition: AddressRanges.h:18
endianness
Definition: bit.h:70
Files in GSYM are contained in FileEntry structs where we split the directory and basename into two d...
Definition: FileEntry.h:24
Function information in GSYM files encodes information for one contiguous address range.
Definition: FunctionInfo.h:88
Inline information stores the name of the inline function along with an array of address ranges.
Definition: InlineInfo.h:59