LLVM 19.0.0git
MemorySanitizer.cpp
Go to the documentation of this file.
1//===- MemorySanitizer.cpp - detector of uninitialized reads --------------===//
2//
3// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
4// See https://llvm.org/LICENSE.txt for license information.
5// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
6//
7//===----------------------------------------------------------------------===//
8//
9/// \file
10/// This file is a part of MemorySanitizer, a detector of uninitialized
11/// reads.
12///
13/// The algorithm of the tool is similar to Memcheck
14/// (http://goo.gl/QKbem). We associate a few shadow bits with every
15/// byte of the application memory, poison the shadow of the malloc-ed
16/// or alloca-ed memory, load the shadow bits on every memory read,
17/// propagate the shadow bits through some of the arithmetic
18/// instruction (including MOV), store the shadow bits on every memory
19/// write, report a bug on some other instructions (e.g. JMP) if the
20/// associated shadow is poisoned.
21///
22/// But there are differences too. The first and the major one:
23/// compiler instrumentation instead of binary instrumentation. This
24/// gives us much better register allocation, possible compiler
25/// optimizations and a fast start-up. But this brings the major issue
26/// as well: msan needs to see all program events, including system
27/// calls and reads/writes in system libraries, so we either need to
28/// compile *everything* with msan or use a binary translation
29/// component (e.g. DynamoRIO) to instrument pre-built libraries.
30/// Another difference from Memcheck is that we use 8 shadow bits per
31/// byte of application memory and use a direct shadow mapping. This
32/// greatly simplifies the instrumentation code and avoids races on
33/// shadow updates (Memcheck is single-threaded so races are not a
34/// concern there. Memcheck uses 2 shadow bits per byte with a slow
35/// path storage that uses 8 bits per byte).
36///
37/// The default value of shadow is 0, which means "clean" (not poisoned).
38///
39/// Every module initializer should call __msan_init to ensure that the
40/// shadow memory is ready. On error, __msan_warning is called. Since
41/// parameters and return values may be passed via registers, we have a
42/// specialized thread-local shadow for return values
43/// (__msan_retval_tls) and parameters (__msan_param_tls).
44///
45/// Origin tracking.
46///
47/// MemorySanitizer can track origins (allocation points) of all uninitialized
48/// values. This behavior is controlled with a flag (msan-track-origins) and is
49/// disabled by default.
50///
51/// Origins are 4-byte values created and interpreted by the runtime library.
52/// They are stored in a second shadow mapping, one 4-byte value for 4 bytes
53/// of application memory. Propagation of origins is basically a bunch of
54/// "select" instructions that pick the origin of a dirty argument, if an
55/// instruction has one.
56///
57/// Every 4 aligned, consecutive bytes of application memory have one origin
58/// value associated with them. If these bytes contain uninitialized data
59/// coming from 2 different allocations, the last store wins. Because of this,
60/// MemorySanitizer reports can show unrelated origins, but this is unlikely in
61/// practice.
62///
63/// Origins are meaningless for fully initialized values, so MemorySanitizer
64/// avoids storing origin to memory when a fully initialized value is stored.
65/// This way it avoids needless overwriting origin of the 4-byte region on
66/// a short (i.e. 1 byte) clean store, and it is also good for performance.
67///
68/// Atomic handling.
69///
70/// Ideally, every atomic store of application value should update the
71/// corresponding shadow location in an atomic way. Unfortunately, atomic store
72/// of two disjoint locations can not be done without severe slowdown.
73///
74/// Therefore, we implement an approximation that may err on the safe side.
75/// In this implementation, every atomically accessed location in the program
76/// may only change from (partially) uninitialized to fully initialized, but
77/// not the other way around. We load the shadow _after_ the application load,
78/// and we store the shadow _before_ the app store. Also, we always store clean
79/// shadow (if the application store is atomic). This way, if the store-load
80/// pair constitutes a happens-before arc, shadow store and load are correctly
81/// ordered such that the load will get either the value that was stored, or
82/// some later value (which is always clean).
83///
84/// This does not work very well with Compare-And-Swap (CAS) and
85/// Read-Modify-Write (RMW) operations. To follow the above logic, CAS and RMW
86/// must store the new shadow before the app operation, and load the shadow
87/// after the app operation. Computers don't work this way. Current
88/// implementation ignores the load aspect of CAS/RMW, always returning a clean
89/// value. It implements the store part as a simple atomic store by storing a
90/// clean shadow.
91///
92/// Instrumenting inline assembly.
93///
94/// For inline assembly code LLVM has little idea about which memory locations
95/// become initialized depending on the arguments. It can be possible to figure
96/// out which arguments are meant to point to inputs and outputs, but the
97/// actual semantics can be only visible at runtime. In the Linux kernel it's
98/// also possible that the arguments only indicate the offset for a base taken
99/// from a segment register, so it's dangerous to treat any asm() arguments as
100/// pointers. We take a conservative approach generating calls to
101/// __msan_instrument_asm_store(ptr, size)
102/// , which defer the memory unpoisoning to the runtime library.
103/// The latter can perform more complex address checks to figure out whether
104/// it's safe to touch the shadow memory.
105/// Like with atomic operations, we call __msan_instrument_asm_store() before
106/// the assembly call, so that changes to the shadow memory will be seen by
107/// other threads together with main memory initialization.
108///
109/// KernelMemorySanitizer (KMSAN) implementation.
110///
111/// The major differences between KMSAN and MSan instrumentation are:
112/// - KMSAN always tracks the origins and implies msan-keep-going=true;
113/// - KMSAN allocates shadow and origin memory for each page separately, so
114/// there are no explicit accesses to shadow and origin in the
115/// instrumentation.
116/// Shadow and origin values for a particular X-byte memory location
117/// (X=1,2,4,8) are accessed through pointers obtained via the
118/// __msan_metadata_ptr_for_load_X(ptr)
119/// __msan_metadata_ptr_for_store_X(ptr)
120/// functions. The corresponding functions check that the X-byte accesses
121/// are possible and returns the pointers to shadow and origin memory.
122/// Arbitrary sized accesses are handled with:
123/// __msan_metadata_ptr_for_load_n(ptr, size)
124/// __msan_metadata_ptr_for_store_n(ptr, size);
125/// Note that the sanitizer code has to deal with how shadow/origin pairs
126/// returned by the these functions are represented in different ABIs. In
127/// the X86_64 ABI they are returned in RDX:RAX, and in the SystemZ ABI they
128/// are written to memory pointed to by a hidden parameter.
129/// - TLS variables are stored in a single per-task struct. A call to a
130/// function __msan_get_context_state() returning a pointer to that struct
131/// is inserted into every instrumented function before the entry block;
132/// - __msan_warning() takes a 32-bit origin parameter;
133/// - local variables are poisoned with __msan_poison_alloca() upon function
134/// entry and unpoisoned with __msan_unpoison_alloca() before leaving the
135/// function;
136/// - the pass doesn't declare any global variables or add global constructors
137/// to the translation unit.
138///
139/// Also, KMSAN currently ignores uninitialized memory passed into inline asm
140/// calls, making sure we're on the safe side wrt. possible false positives.
141///
142/// KernelMemorySanitizer only supports X86_64 and SystemZ at the moment.
143///
144//
145// FIXME: This sanitizer does not yet handle scalable vectors
146//
147//===----------------------------------------------------------------------===//
148
150#include "llvm/ADT/APInt.h"
151#include "llvm/ADT/ArrayRef.h"
152#include "llvm/ADT/DenseMap.h"
154#include "llvm/ADT/SetVector.h"
155#include "llvm/ADT/SmallVector.h"
157#include "llvm/ADT/StringRef.h"
161#include "llvm/IR/Argument.h"
163#include "llvm/IR/Attributes.h"
164#include "llvm/IR/BasicBlock.h"
165#include "llvm/IR/CallingConv.h"
166#include "llvm/IR/Constant.h"
167#include "llvm/IR/Constants.h"
168#include "llvm/IR/DataLayout.h"
169#include "llvm/IR/DerivedTypes.h"
170#include "llvm/IR/Function.h"
171#include "llvm/IR/GlobalValue.h"
173#include "llvm/IR/IRBuilder.h"
174#include "llvm/IR/InlineAsm.h"
175#include "llvm/IR/InstVisitor.h"
176#include "llvm/IR/InstrTypes.h"
177#include "llvm/IR/Instruction.h"
178#include "llvm/IR/Instructions.h"
180#include "llvm/IR/Intrinsics.h"
181#include "llvm/IR/IntrinsicsX86.h"
182#include "llvm/IR/MDBuilder.h"
183#include "llvm/IR/Module.h"
184#include "llvm/IR/Type.h"
185#include "llvm/IR/Value.h"
186#include "llvm/IR/ValueMap.h"
189#include "llvm/Support/Casting.h"
191#include "llvm/Support/Debug.h"
200#include <algorithm>
201#include <cassert>
202#include <cstddef>
203#include <cstdint>
204#include <memory>
205#include <string>
206#include <tuple>
207
208using namespace llvm;
209
210#define DEBUG_TYPE "msan"
211
212DEBUG_COUNTER(DebugInsertCheck, "msan-insert-check",
213 "Controls which checks to insert");
214
215static const unsigned kOriginSize = 4;
218
219// These constants must be kept in sync with the ones in msan.h.
220static const unsigned kParamTLSSize = 800;
221static const unsigned kRetvalTLSSize = 800;
222
223// Accesses sizes are powers of two: 1, 2, 4, 8.
224static const size_t kNumberOfAccessSizes = 4;
225
226/// Track origins of uninitialized values.
227///
228/// Adds a section to MemorySanitizer report that points to the allocation
229/// (stack or heap) the uninitialized bits came from originally.
231 "msan-track-origins",
232 cl::desc("Track origins (allocation sites) of poisoned memory"), cl::Hidden,
233 cl::init(0));
234
235static cl::opt<bool> ClKeepGoing("msan-keep-going",
236 cl::desc("keep going after reporting a UMR"),
237 cl::Hidden, cl::init(false));
238
239static cl::opt<bool>
240 ClPoisonStack("msan-poison-stack",
241 cl::desc("poison uninitialized stack variables"), cl::Hidden,
242 cl::init(true));
243
245 "msan-poison-stack-with-call",
246 cl::desc("poison uninitialized stack variables with a call"), cl::Hidden,
247 cl::init(false));
248
250 "msan-poison-stack-pattern",
251 cl::desc("poison uninitialized stack variables with the given pattern"),
252 cl::Hidden, cl::init(0xff));
253
254static cl::opt<bool>
255 ClPrintStackNames("msan-print-stack-names",
256 cl::desc("Print name of local stack variable"),
257 cl::Hidden, cl::init(true));
258
259static cl::opt<bool> ClPoisonUndef("msan-poison-undef",
260 cl::desc("poison undef temps"), cl::Hidden,
261 cl::init(true));
262
263static cl::opt<bool>
264 ClHandleICmp("msan-handle-icmp",
265 cl::desc("propagate shadow through ICmpEQ and ICmpNE"),
266 cl::Hidden, cl::init(true));
267
268static cl::opt<bool>
269 ClHandleICmpExact("msan-handle-icmp-exact",
270 cl::desc("exact handling of relational integer ICmp"),
271 cl::Hidden, cl::init(false));
272
274 "msan-handle-lifetime-intrinsics",
275 cl::desc(
276 "when possible, poison scoped variables at the beginning of the scope "
277 "(slower, but more precise)"),
278 cl::Hidden, cl::init(true));
279
280// When compiling the Linux kernel, we sometimes see false positives related to
281// MSan being unable to understand that inline assembly calls may initialize
282// local variables.
283// This flag makes the compiler conservatively unpoison every memory location
284// passed into an assembly call. Note that this may cause false positives.
285// Because it's impossible to figure out the array sizes, we can only unpoison
286// the first sizeof(type) bytes for each type* pointer.
288 "msan-handle-asm-conservative",
289 cl::desc("conservative handling of inline assembly"), cl::Hidden,
290 cl::init(true));
291
292// This flag controls whether we check the shadow of the address
293// operand of load or store. Such bugs are very rare, since load from
294// a garbage address typically results in SEGV, but still happen
295// (e.g. only lower bits of address are garbage, or the access happens
296// early at program startup where malloc-ed memory is more likely to
297// be zeroed. As of 2012-08-28 this flag adds 20% slowdown.
299 "msan-check-access-address",
300 cl::desc("report accesses through a pointer which has poisoned shadow"),
301 cl::Hidden, cl::init(true));
302
304 "msan-eager-checks",
305 cl::desc("check arguments and return values at function call boundaries"),
306 cl::Hidden, cl::init(false));
307
309 "msan-dump-strict-instructions",
310 cl::desc("print out instructions with default strict semantics"),
311 cl::Hidden, cl::init(false));
312
314 "msan-instrumentation-with-call-threshold",
315 cl::desc(
316 "If the function being instrumented requires more than "
317 "this number of checks and origin stores, use callbacks instead of "
318 "inline checks (-1 means never use callbacks)."),
319 cl::Hidden, cl::init(3500));
320
321static cl::opt<bool>
322 ClEnableKmsan("msan-kernel",
323 cl::desc("Enable KernelMemorySanitizer instrumentation"),
324 cl::Hidden, cl::init(false));
325
326static cl::opt<bool>
327 ClDisableChecks("msan-disable-checks",
328 cl::desc("Apply no_sanitize to the whole file"), cl::Hidden,
329 cl::init(false));
330
331static cl::opt<bool>
332 ClCheckConstantShadow("msan-check-constant-shadow",
333 cl::desc("Insert checks for constant shadow values"),
334 cl::Hidden, cl::init(true));
335
336// This is off by default because of a bug in gold:
337// https://sourceware.org/bugzilla/show_bug.cgi?id=19002
338static cl::opt<bool>
339 ClWithComdat("msan-with-comdat",
340 cl::desc("Place MSan constructors in comdat sections"),
341 cl::Hidden, cl::init(false));
342
343// These options allow to specify custom memory map parameters
344// See MemoryMapParams for details.
345static cl::opt<uint64_t> ClAndMask("msan-and-mask",
346 cl::desc("Define custom MSan AndMask"),
347 cl::Hidden, cl::init(0));
348
349static cl::opt<uint64_t> ClXorMask("msan-xor-mask",
350 cl::desc("Define custom MSan XorMask"),
351 cl::Hidden, cl::init(0));
352
353static cl::opt<uint64_t> ClShadowBase("msan-shadow-base",
354 cl::desc("Define custom MSan ShadowBase"),
355 cl::Hidden, cl::init(0));
356
357static cl::opt<uint64_t> ClOriginBase("msan-origin-base",
358 cl::desc("Define custom MSan OriginBase"),
359 cl::Hidden, cl::init(0));
360
361static cl::opt<int>
362 ClDisambiguateWarning("msan-disambiguate-warning-threshold",
363 cl::desc("Define threshold for number of checks per "
364 "debug location to force origin update."),
365 cl::Hidden, cl::init(3));
366
367const char kMsanModuleCtorName[] = "msan.module_ctor";
368const char kMsanInitName[] = "__msan_init";
369
370namespace {
371
372// Memory map parameters used in application-to-shadow address calculation.
373// Offset = (Addr & ~AndMask) ^ XorMask
374// Shadow = ShadowBase + Offset
375// Origin = OriginBase + Offset
376struct MemoryMapParams {
377 uint64_t AndMask;
378 uint64_t XorMask;
379 uint64_t ShadowBase;
380 uint64_t OriginBase;
381};
382
383struct PlatformMemoryMapParams {
384 const MemoryMapParams *bits32;
385 const MemoryMapParams *bits64;
386};
387
388} // end anonymous namespace
389
390// i386 Linux
391static const MemoryMapParams Linux_I386_MemoryMapParams = {
392 0x000080000000, // AndMask
393 0, // XorMask (not used)
394 0, // ShadowBase (not used)
395 0x000040000000, // OriginBase
396};
397
398// x86_64 Linux
399static const MemoryMapParams Linux_X86_64_MemoryMapParams = {
400 0, // AndMask (not used)
401 0x500000000000, // XorMask
402 0, // ShadowBase (not used)
403 0x100000000000, // OriginBase
404};
405
406// mips64 Linux
407static const MemoryMapParams Linux_MIPS64_MemoryMapParams = {
408 0, // AndMask (not used)
409 0x008000000000, // XorMask
410 0, // ShadowBase (not used)
411 0x002000000000, // OriginBase
412};
413
414// ppc64 Linux
415static const MemoryMapParams Linux_PowerPC64_MemoryMapParams = {
416 0xE00000000000, // AndMask
417 0x100000000000, // XorMask
418 0x080000000000, // ShadowBase
419 0x1C0000000000, // OriginBase
420};
421
422// s390x Linux
423static const MemoryMapParams Linux_S390X_MemoryMapParams = {
424 0xC00000000000, // AndMask
425 0, // XorMask (not used)
426 0x080000000000, // ShadowBase
427 0x1C0000000000, // OriginBase
428};
429
430// aarch64 Linux
431static const MemoryMapParams Linux_AArch64_MemoryMapParams = {
432 0, // AndMask (not used)
433 0x0B00000000000, // XorMask
434 0, // ShadowBase (not used)
435 0x0200000000000, // OriginBase
436};
437
438// loongarch64 Linux
439static const MemoryMapParams Linux_LoongArch64_MemoryMapParams = {
440 0, // AndMask (not used)
441 0x500000000000, // XorMask
442 0, // ShadowBase (not used)
443 0x100000000000, // OriginBase
444};
445
446// aarch64 FreeBSD
447static const MemoryMapParams FreeBSD_AArch64_MemoryMapParams = {
448 0x1800000000000, // AndMask
449 0x0400000000000, // XorMask
450 0x0200000000000, // ShadowBase
451 0x0700000000000, // OriginBase
452};
453
454// i386 FreeBSD
455static const MemoryMapParams FreeBSD_I386_MemoryMapParams = {
456 0x000180000000, // AndMask
457 0x000040000000, // XorMask
458 0x000020000000, // ShadowBase
459 0x000700000000, // OriginBase
460};
461
462// x86_64 FreeBSD
463static const MemoryMapParams FreeBSD_X86_64_MemoryMapParams = {
464 0xc00000000000, // AndMask
465 0x200000000000, // XorMask
466 0x100000000000, // ShadowBase
467 0x380000000000, // OriginBase
468};
469
470// x86_64 NetBSD
471static const MemoryMapParams NetBSD_X86_64_MemoryMapParams = {
472 0, // AndMask
473 0x500000000000, // XorMask
474 0, // ShadowBase
475 0x100000000000, // OriginBase
476};
477
478static const PlatformMemoryMapParams Linux_X86_MemoryMapParams = {
481};
482
483static const PlatformMemoryMapParams Linux_MIPS_MemoryMapParams = {
484 nullptr,
486};
487
488static const PlatformMemoryMapParams Linux_PowerPC_MemoryMapParams = {
489 nullptr,
491};
492
493static const PlatformMemoryMapParams Linux_S390_MemoryMapParams = {
494 nullptr,
496};
497
498static const PlatformMemoryMapParams Linux_ARM_MemoryMapParams = {
499 nullptr,
501};
502
503static const PlatformMemoryMapParams Linux_LoongArch_MemoryMapParams = {
504 nullptr,
506};
507
508static const PlatformMemoryMapParams FreeBSD_ARM_MemoryMapParams = {
509 nullptr,
511};
512
513static const PlatformMemoryMapParams FreeBSD_X86_MemoryMapParams = {
516};
517
518static const PlatformMemoryMapParams NetBSD_X86_MemoryMapParams = {
519 nullptr,
521};
522
523namespace {
524
525/// Instrument functions of a module to detect uninitialized reads.
526///
527/// Instantiating MemorySanitizer inserts the msan runtime library API function
528/// declarations into the module if they don't exist already. Instantiating
529/// ensures the __msan_init function is in the list of global constructors for
530/// the module.
531class MemorySanitizer {
532public:
533 MemorySanitizer(Module &M, MemorySanitizerOptions Options)
534 : CompileKernel(Options.Kernel), TrackOrigins(Options.TrackOrigins),
535 Recover(Options.Recover), EagerChecks(Options.EagerChecks) {
536 initializeModule(M);
537 }
538
539 // MSan cannot be moved or copied because of MapParams.
540 MemorySanitizer(MemorySanitizer &&) = delete;
541 MemorySanitizer &operator=(MemorySanitizer &&) = delete;
542 MemorySanitizer(const MemorySanitizer &) = delete;
543 MemorySanitizer &operator=(const MemorySanitizer &) = delete;
544
545 bool sanitizeFunction(Function &F, TargetLibraryInfo &TLI);
546
547private:
548 friend struct MemorySanitizerVisitor;
549 friend struct VarArgHelperBase;
550 friend struct VarArgAMD64Helper;
551 friend struct VarArgMIPS64Helper;
552 friend struct VarArgAArch64Helper;
553 friend struct VarArgPowerPC64Helper;
554 friend struct VarArgSystemZHelper;
555
556 void initializeModule(Module &M);
557 void initializeCallbacks(Module &M, const TargetLibraryInfo &TLI);
558 void createKernelApi(Module &M, const TargetLibraryInfo &TLI);
559 void createUserspaceApi(Module &M, const TargetLibraryInfo &TLI);
560
561 template <typename... ArgsTy>
562 FunctionCallee getOrInsertMsanMetadataFunction(Module &M, StringRef Name,
563 ArgsTy... Args);
564
565 /// True if we're compiling the Linux kernel.
566 bool CompileKernel;
567 /// Track origins (allocation points) of uninitialized values.
568 int TrackOrigins;
569 bool Recover;
570 bool EagerChecks;
571
572 Triple TargetTriple;
573 LLVMContext *C;
574 Type *IntptrTy; ///< Integer type with the size of a ptr in default AS.
575 Type *OriginTy;
576 PointerType *PtrTy; ///< Integer type with the size of a ptr in default AS.
577
578 // XxxTLS variables represent the per-thread state in MSan and per-task state
579 // in KMSAN.
580 // For the userspace these point to thread-local globals. In the kernel land
581 // they point to the members of a per-task struct obtained via a call to
582 // __msan_get_context_state().
583
584 /// Thread-local shadow storage for function parameters.
585 Value *ParamTLS;
586
587 /// Thread-local origin storage for function parameters.
588 Value *ParamOriginTLS;
589
590 /// Thread-local shadow storage for function return value.
591 Value *RetvalTLS;
592
593 /// Thread-local origin storage for function return value.
594 Value *RetvalOriginTLS;
595
596 /// Thread-local shadow storage for in-register va_arg function.
597 Value *VAArgTLS;
598
599 /// Thread-local shadow storage for in-register va_arg function.
600 Value *VAArgOriginTLS;
601
602 /// Thread-local shadow storage for va_arg overflow area.
603 Value *VAArgOverflowSizeTLS;
604
605 /// Are the instrumentation callbacks set up?
606 bool CallbacksInitialized = false;
607
608 /// The run-time callback to print a warning.
609 FunctionCallee WarningFn;
610
611 // These arrays are indexed by log2(AccessSize).
612 FunctionCallee MaybeWarningFn[kNumberOfAccessSizes];
613 FunctionCallee MaybeStoreOriginFn[kNumberOfAccessSizes];
614
615 /// Run-time helper that generates a new origin value for a stack
616 /// allocation.
617 FunctionCallee MsanSetAllocaOriginWithDescriptionFn;
618 // No description version
619 FunctionCallee MsanSetAllocaOriginNoDescriptionFn;
620
621 /// Run-time helper that poisons stack on function entry.
622 FunctionCallee MsanPoisonStackFn;
623
624 /// Run-time helper that records a store (or any event) of an
625 /// uninitialized value and returns an updated origin id encoding this info.
626 FunctionCallee MsanChainOriginFn;
627
628 /// Run-time helper that paints an origin over a region.
629 FunctionCallee MsanSetOriginFn;
630
631 /// MSan runtime replacements for memmove, memcpy and memset.
632 FunctionCallee MemmoveFn, MemcpyFn, MemsetFn;
633
634 /// KMSAN callback for task-local function argument shadow.
635 StructType *MsanContextStateTy;
636 FunctionCallee MsanGetContextStateFn;
637
638 /// Functions for poisoning/unpoisoning local variables
639 FunctionCallee MsanPoisonAllocaFn, MsanUnpoisonAllocaFn;
640
641 /// Pair of shadow/origin pointers.
642 Type *MsanMetadata;
643
644 /// Each of the MsanMetadataPtrXxx functions returns a MsanMetadata.
645 FunctionCallee MsanMetadataPtrForLoadN, MsanMetadataPtrForStoreN;
646 FunctionCallee MsanMetadataPtrForLoad_1_8[4];
647 FunctionCallee MsanMetadataPtrForStore_1_8[4];
648 FunctionCallee MsanInstrumentAsmStoreFn;
649
650 /// Storage for return values of the MsanMetadataPtrXxx functions.
651 Value *MsanMetadataAlloca;
652
653 /// Helper to choose between different MsanMetadataPtrXxx().
654 FunctionCallee getKmsanShadowOriginAccessFn(bool isStore, int size);
655
656 /// Memory map parameters used in application-to-shadow calculation.
657 const MemoryMapParams *MapParams;
658
659 /// Custom memory map parameters used when -msan-shadow-base or
660 // -msan-origin-base is provided.
661 MemoryMapParams CustomMapParams;
662
663 MDNode *ColdCallWeights;
664
665 /// Branch weights for origin store.
666 MDNode *OriginStoreWeights;
667};
668
669void insertModuleCtor(Module &M) {
672 /*InitArgTypes=*/{},
673 /*InitArgs=*/{},
674 // This callback is invoked when the functions are created the first
675 // time. Hook them into the global ctors list in that case:
676 [&](Function *Ctor, FunctionCallee) {
677 if (!ClWithComdat) {
678 appendToGlobalCtors(M, Ctor, 0);
679 return;
680 }
681 Comdat *MsanCtorComdat = M.getOrInsertComdat(kMsanModuleCtorName);
682 Ctor->setComdat(MsanCtorComdat);
683 appendToGlobalCtors(M, Ctor, 0, Ctor);
684 });
685}
686
687template <class T> T getOptOrDefault(const cl::opt<T> &Opt, T Default) {
688 return (Opt.getNumOccurrences() > 0) ? Opt : Default;
689}
690
691} // end anonymous namespace
692
694 bool EagerChecks)
695 : Kernel(getOptOrDefault(ClEnableKmsan, K)),
696 TrackOrigins(getOptOrDefault(ClTrackOrigins, Kernel ? 2 : TO)),
697 Recover(getOptOrDefault(ClKeepGoing, Kernel || R)),
698 EagerChecks(getOptOrDefault(ClEagerChecks, EagerChecks)) {}
699
702 bool Modified = false;
703 if (!Options.Kernel) {
704 insertModuleCtor(M);
705 Modified = true;
706 }
707
708 auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
709 for (Function &F : M) {
710 if (F.empty())
711 continue;
712 MemorySanitizer Msan(*F.getParent(), Options);
713 Modified |=
714 Msan.sanitizeFunction(F, FAM.getResult<TargetLibraryAnalysis>(F));
715 }
716
717 if (!Modified)
718 return PreservedAnalyses::all();
719
721 // GlobalsAA is considered stateless and does not get invalidated unless
722 // explicitly invalidated; PreservedAnalyses::none() is not enough. Sanitizers
723 // make changes that require GlobalsAA to be invalidated.
724 PA.abandon<GlobalsAA>();
725 return PA;
726}
727
729 raw_ostream &OS, function_ref<StringRef(StringRef)> MapClassName2PassName) {
731 OS, MapClassName2PassName);
732 OS << '<';
733 if (Options.Recover)
734 OS << "recover;";
735 if (Options.Kernel)
736 OS << "kernel;";
737 if (Options.EagerChecks)
738 OS << "eager-checks;";
739 OS << "track-origins=" << Options.TrackOrigins;
740 OS << '>';
741}
742
743/// Create a non-const global initialized with the given string.
744///
745/// Creates a writable global for Str so that we can pass it to the
746/// run-time lib. Runtime uses first 4 bytes of the string to store the
747/// frame ID, so the string needs to be mutable.
749 StringRef Str) {
750 Constant *StrConst = ConstantDataArray::getString(M.getContext(), Str);
751 return new GlobalVariable(M, StrConst->getType(), /*isConstant=*/true,
752 GlobalValue::PrivateLinkage, StrConst, "");
753}
754
755template <typename... ArgsTy>
757MemorySanitizer::getOrInsertMsanMetadataFunction(Module &M, StringRef Name,
758 ArgsTy... Args) {
759 if (TargetTriple.getArch() == Triple::systemz) {
760 // SystemZ ABI: shadow/origin pair is returned via a hidden parameter.
761 return M.getOrInsertFunction(Name, Type::getVoidTy(*C),
762 PointerType::get(MsanMetadata, 0),
763 std::forward<ArgsTy>(Args)...);
764 }
765
766 return M.getOrInsertFunction(Name, MsanMetadata,
767 std::forward<ArgsTy>(Args)...);
768}
769
770/// Create KMSAN API callbacks.
771void MemorySanitizer::createKernelApi(Module &M, const TargetLibraryInfo &TLI) {
772 IRBuilder<> IRB(*C);
773
774 // These will be initialized in insertKmsanPrologue().
775 RetvalTLS = nullptr;
776 RetvalOriginTLS = nullptr;
777 ParamTLS = nullptr;
778 ParamOriginTLS = nullptr;
779 VAArgTLS = nullptr;
780 VAArgOriginTLS = nullptr;
781 VAArgOverflowSizeTLS = nullptr;
782
783 WarningFn = M.getOrInsertFunction("__msan_warning",
784 TLI.getAttrList(C, {0}, /*Signed=*/false),
785 IRB.getVoidTy(), IRB.getInt32Ty());
786
787 // Requests the per-task context state (kmsan_context_state*) from the
788 // runtime library.
789 MsanContextStateTy = StructType::get(
790 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
791 ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8),
792 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
793 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8), /* va_arg_origin */
794 IRB.getInt64Ty(), ArrayType::get(OriginTy, kParamTLSSize / 4), OriginTy,
795 OriginTy);
796 MsanGetContextStateFn = M.getOrInsertFunction(
797 "__msan_get_context_state", PointerType::get(MsanContextStateTy, 0));
798
799 MsanMetadata = StructType::get(PointerType::get(IRB.getInt8Ty(), 0),
800 PointerType::get(IRB.getInt32Ty(), 0));
801
802 for (int ind = 0, size = 1; ind < 4; ind++, size <<= 1) {
803 std::string name_load =
804 "__msan_metadata_ptr_for_load_" + std::to_string(size);
805 std::string name_store =
806 "__msan_metadata_ptr_for_store_" + std::to_string(size);
807 MsanMetadataPtrForLoad_1_8[ind] = getOrInsertMsanMetadataFunction(
808 M, name_load, PointerType::get(IRB.getInt8Ty(), 0));
809 MsanMetadataPtrForStore_1_8[ind] = getOrInsertMsanMetadataFunction(
810 M, name_store, PointerType::get(IRB.getInt8Ty(), 0));
811 }
812
813 MsanMetadataPtrForLoadN = getOrInsertMsanMetadataFunction(
814 M, "__msan_metadata_ptr_for_load_n", PointerType::get(IRB.getInt8Ty(), 0),
815 IRB.getInt64Ty());
816 MsanMetadataPtrForStoreN = getOrInsertMsanMetadataFunction(
817 M, "__msan_metadata_ptr_for_store_n",
818 PointerType::get(IRB.getInt8Ty(), 0), IRB.getInt64Ty());
819
820 // Functions for poisoning and unpoisoning memory.
821 MsanPoisonAllocaFn = M.getOrInsertFunction(
822 "__msan_poison_alloca", IRB.getVoidTy(), PtrTy, IntptrTy, PtrTy);
823 MsanUnpoisonAllocaFn = M.getOrInsertFunction(
824 "__msan_unpoison_alloca", IRB.getVoidTy(), PtrTy, IntptrTy);
825}
826
828 return M.getOrInsertGlobal(Name, Ty, [&] {
829 return new GlobalVariable(M, Ty, false, GlobalVariable::ExternalLinkage,
830 nullptr, Name, nullptr,
832 });
833}
834
835/// Insert declarations for userspace-specific functions and globals.
836void MemorySanitizer::createUserspaceApi(Module &M, const TargetLibraryInfo &TLI) {
837 IRBuilder<> IRB(*C);
838
839 // Create the callback.
840 // FIXME: this function should have "Cold" calling conv,
841 // which is not yet implemented.
842 if (TrackOrigins) {
843 StringRef WarningFnName = Recover ? "__msan_warning_with_origin"
844 : "__msan_warning_with_origin_noreturn";
845 WarningFn = M.getOrInsertFunction(WarningFnName,
846 TLI.getAttrList(C, {0}, /*Signed=*/false),
847 IRB.getVoidTy(), IRB.getInt32Ty());
848 } else {
849 StringRef WarningFnName =
850 Recover ? "__msan_warning" : "__msan_warning_noreturn";
851 WarningFn = M.getOrInsertFunction(WarningFnName, IRB.getVoidTy());
852 }
853
854 // Create the global TLS variables.
855 RetvalTLS =
856 getOrInsertGlobal(M, "__msan_retval_tls",
857 ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8));
858
859 RetvalOriginTLS = getOrInsertGlobal(M, "__msan_retval_origin_tls", OriginTy);
860
861 ParamTLS =
862 getOrInsertGlobal(M, "__msan_param_tls",
863 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
864
865 ParamOriginTLS =
866 getOrInsertGlobal(M, "__msan_param_origin_tls",
867 ArrayType::get(OriginTy, kParamTLSSize / 4));
868
869 VAArgTLS =
870 getOrInsertGlobal(M, "__msan_va_arg_tls",
871 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
872
873 VAArgOriginTLS =
874 getOrInsertGlobal(M, "__msan_va_arg_origin_tls",
875 ArrayType::get(OriginTy, kParamTLSSize / 4));
876
877 VAArgOverflowSizeTLS =
878 getOrInsertGlobal(M, "__msan_va_arg_overflow_size_tls", IRB.getInt64Ty());
879
880 for (size_t AccessSizeIndex = 0; AccessSizeIndex < kNumberOfAccessSizes;
881 AccessSizeIndex++) {
882 unsigned AccessSize = 1 << AccessSizeIndex;
883 std::string FunctionName = "__msan_maybe_warning_" + itostr(AccessSize);
884 MaybeWarningFn[AccessSizeIndex] = M.getOrInsertFunction(
885 FunctionName, TLI.getAttrList(C, {0, 1}, /*Signed=*/false),
886 IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), IRB.getInt32Ty());
887
888 FunctionName = "__msan_maybe_store_origin_" + itostr(AccessSize);
889 MaybeStoreOriginFn[AccessSizeIndex] = M.getOrInsertFunction(
890 FunctionName, TLI.getAttrList(C, {0, 2}, /*Signed=*/false),
891 IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), PtrTy,
892 IRB.getInt32Ty());
893 }
894
895 MsanSetAllocaOriginWithDescriptionFn =
896 M.getOrInsertFunction("__msan_set_alloca_origin_with_descr",
897 IRB.getVoidTy(), PtrTy, IntptrTy, PtrTy, PtrTy);
898 MsanSetAllocaOriginNoDescriptionFn =
899 M.getOrInsertFunction("__msan_set_alloca_origin_no_descr",
900 IRB.getVoidTy(), PtrTy, IntptrTy, PtrTy);
901 MsanPoisonStackFn = M.getOrInsertFunction("__msan_poison_stack",
902 IRB.getVoidTy(), PtrTy, IntptrTy);
903}
904
905/// Insert extern declaration of runtime-provided functions and globals.
906void MemorySanitizer::initializeCallbacks(Module &M, const TargetLibraryInfo &TLI) {
907 // Only do this once.
908 if (CallbacksInitialized)
909 return;
910
911 IRBuilder<> IRB(*C);
912 // Initialize callbacks that are common for kernel and userspace
913 // instrumentation.
914 MsanChainOriginFn = M.getOrInsertFunction(
915 "__msan_chain_origin",
916 TLI.getAttrList(C, {0}, /*Signed=*/false, /*Ret=*/true), IRB.getInt32Ty(),
917 IRB.getInt32Ty());
918 MsanSetOriginFn = M.getOrInsertFunction(
919 "__msan_set_origin", TLI.getAttrList(C, {2}, /*Signed=*/false),
920 IRB.getVoidTy(), PtrTy, IntptrTy, IRB.getInt32Ty());
921 MemmoveFn =
922 M.getOrInsertFunction("__msan_memmove", PtrTy, PtrTy, PtrTy, IntptrTy);
923 MemcpyFn =
924 M.getOrInsertFunction("__msan_memcpy", PtrTy, PtrTy, PtrTy, IntptrTy);
925 MemsetFn = M.getOrInsertFunction("__msan_memset",
926 TLI.getAttrList(C, {1}, /*Signed=*/true),
927 PtrTy, PtrTy, IRB.getInt32Ty(), IntptrTy);
928
929 MsanInstrumentAsmStoreFn =
930 M.getOrInsertFunction("__msan_instrument_asm_store", IRB.getVoidTy(),
931 PointerType::get(IRB.getInt8Ty(), 0), IntptrTy);
932
933 if (CompileKernel) {
934 createKernelApi(M, TLI);
935 } else {
936 createUserspaceApi(M, TLI);
937 }
938 CallbacksInitialized = true;
939}
940
941FunctionCallee MemorySanitizer::getKmsanShadowOriginAccessFn(bool isStore,
942 int size) {
943 FunctionCallee *Fns =
944 isStore ? MsanMetadataPtrForStore_1_8 : MsanMetadataPtrForLoad_1_8;
945 switch (size) {
946 case 1:
947 return Fns[0];
948 case 2:
949 return Fns[1];
950 case 4:
951 return Fns[2];
952 case 8:
953 return Fns[3];
954 default:
955 return nullptr;
956 }
957}
958
959/// Module-level initialization.
960///
961/// inserts a call to __msan_init to the module's constructor list.
962void MemorySanitizer::initializeModule(Module &M) {
963 auto &DL = M.getDataLayout();
964
965 TargetTriple = Triple(M.getTargetTriple());
966
967 bool ShadowPassed = ClShadowBase.getNumOccurrences() > 0;
968 bool OriginPassed = ClOriginBase.getNumOccurrences() > 0;
969 // Check the overrides first
970 if (ShadowPassed || OriginPassed) {
971 CustomMapParams.AndMask = ClAndMask;
972 CustomMapParams.XorMask = ClXorMask;
973 CustomMapParams.ShadowBase = ClShadowBase;
974 CustomMapParams.OriginBase = ClOriginBase;
975 MapParams = &CustomMapParams;
976 } else {
977 switch (TargetTriple.getOS()) {
978 case Triple::FreeBSD:
979 switch (TargetTriple.getArch()) {
980 case Triple::aarch64:
981 MapParams = FreeBSD_ARM_MemoryMapParams.bits64;
982 break;
983 case Triple::x86_64:
984 MapParams = FreeBSD_X86_MemoryMapParams.bits64;
985 break;
986 case Triple::x86:
987 MapParams = FreeBSD_X86_MemoryMapParams.bits32;
988 break;
989 default:
990 report_fatal_error("unsupported architecture");
991 }
992 break;
993 case Triple::NetBSD:
994 switch (TargetTriple.getArch()) {
995 case Triple::x86_64:
996 MapParams = NetBSD_X86_MemoryMapParams.bits64;
997 break;
998 default:
999 report_fatal_error("unsupported architecture");
1000 }
1001 break;
1002 case Triple::Linux:
1003 switch (TargetTriple.getArch()) {
1004 case Triple::x86_64:
1005 MapParams = Linux_X86_MemoryMapParams.bits64;
1006 break;
1007 case Triple::x86:
1008 MapParams = Linux_X86_MemoryMapParams.bits32;
1009 break;
1010 case Triple::mips64:
1011 case Triple::mips64el:
1012 MapParams = Linux_MIPS_MemoryMapParams.bits64;
1013 break;
1014 case Triple::ppc64:
1015 case Triple::ppc64le:
1016 MapParams = Linux_PowerPC_MemoryMapParams.bits64;
1017 break;
1018 case Triple::systemz:
1019 MapParams = Linux_S390_MemoryMapParams.bits64;
1020 break;
1021 case Triple::aarch64:
1022 case Triple::aarch64_be:
1023 MapParams = Linux_ARM_MemoryMapParams.bits64;
1024 break;
1026 MapParams = Linux_LoongArch_MemoryMapParams.bits64;
1027 break;
1028 default:
1029 report_fatal_error("unsupported architecture");
1030 }
1031 break;
1032 default:
1033 report_fatal_error("unsupported operating system");
1034 }
1035 }
1036
1037 C = &(M.getContext());
1038 IRBuilder<> IRB(*C);
1039 IntptrTy = IRB.getIntPtrTy(DL);
1040 OriginTy = IRB.getInt32Ty();
1041 PtrTy = IRB.getPtrTy();
1042
1043 ColdCallWeights = MDBuilder(*C).createBranchWeights(1, 1000);
1044 OriginStoreWeights = MDBuilder(*C).createBranchWeights(1, 1000);
1045
1046 if (!CompileKernel) {
1047 if (TrackOrigins)
1048 M.getOrInsertGlobal("__msan_track_origins", IRB.getInt32Ty(), [&] {
1049 return new GlobalVariable(
1050 M, IRB.getInt32Ty(), true, GlobalValue::WeakODRLinkage,
1051 IRB.getInt32(TrackOrigins), "__msan_track_origins");
1052 });
1053
1054 if (Recover)
1055 M.getOrInsertGlobal("__msan_keep_going", IRB.getInt32Ty(), [&] {
1056 return new GlobalVariable(M, IRB.getInt32Ty(), true,
1057 GlobalValue::WeakODRLinkage,
1058 IRB.getInt32(Recover), "__msan_keep_going");
1059 });
1060 }
1061}
1062
1063namespace {
1064
1065/// A helper class that handles instrumentation of VarArg
1066/// functions on a particular platform.
1067///
1068/// Implementations are expected to insert the instrumentation
1069/// necessary to propagate argument shadow through VarArg function
1070/// calls. Visit* methods are called during an InstVisitor pass over
1071/// the function, and should avoid creating new basic blocks. A new
1072/// instance of this class is created for each instrumented function.
1073struct VarArgHelper {
1074 virtual ~VarArgHelper() = default;
1075
1076 /// Visit a CallBase.
1077 virtual void visitCallBase(CallBase &CB, IRBuilder<> &IRB) = 0;
1078
1079 /// Visit a va_start call.
1080 virtual void visitVAStartInst(VAStartInst &I) = 0;
1081
1082 /// Visit a va_copy call.
1083 virtual void visitVACopyInst(VACopyInst &I) = 0;
1084
1085 /// Finalize function instrumentation.
1086 ///
1087 /// This method is called after visiting all interesting (see above)
1088 /// instructions in a function.
1089 virtual void finalizeInstrumentation() = 0;
1090};
1091
1092struct MemorySanitizerVisitor;
1093
1094} // end anonymous namespace
1095
1096static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
1097 MemorySanitizerVisitor &Visitor);
1098
1099static unsigned TypeSizeToSizeIndex(TypeSize TS) {
1100 if (TS.isScalable())
1101 // Scalable types unconditionally take slowpaths.
1102 return kNumberOfAccessSizes;
1103 unsigned TypeSizeFixed = TS.getFixedValue();
1104 if (TypeSizeFixed <= 8)
1105 return 0;
1106 return Log2_32_Ceil((TypeSizeFixed + 7) / 8);
1107}
1108
1109namespace {
1110
1111/// Helper class to attach debug information of the given instruction onto new
1112/// instructions inserted after.
1113class NextNodeIRBuilder : public IRBuilder<> {
1114public:
1115 explicit NextNodeIRBuilder(Instruction *IP) : IRBuilder<>(IP->getNextNode()) {
1116 SetCurrentDebugLocation(IP->getDebugLoc());
1117 }
1118};
1119
1120/// This class does all the work for a given function. Store and Load
1121/// instructions store and load corresponding shadow and origin
1122/// values. Most instructions propagate shadow from arguments to their
1123/// return values. Certain instructions (most importantly, BranchInst)
1124/// test their argument shadow and print reports (with a runtime call) if it's
1125/// non-zero.
1126struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
1127 Function &F;
1128 MemorySanitizer &MS;
1129 SmallVector<PHINode *, 16> ShadowPHINodes, OriginPHINodes;
1130 ValueMap<Value *, Value *> ShadowMap, OriginMap;
1131 std::unique_ptr<VarArgHelper> VAHelper;
1132 const TargetLibraryInfo *TLI;
1133 Instruction *FnPrologueEnd;
1134
1135 // The following flags disable parts of MSan instrumentation based on
1136 // exclusion list contents and command-line options.
1137 bool InsertChecks;
1138 bool PropagateShadow;
1139 bool PoisonStack;
1140 bool PoisonUndef;
1141
1142 struct ShadowOriginAndInsertPoint {
1143 Value *Shadow;
1144 Value *Origin;
1145 Instruction *OrigIns;
1146
1147 ShadowOriginAndInsertPoint(Value *S, Value *O, Instruction *I)
1148 : Shadow(S), Origin(O), OrigIns(I) {}
1149 };
1151 DenseMap<const DILocation *, int> LazyWarningDebugLocationCount;
1152 bool InstrumentLifetimeStart = ClHandleLifetimeIntrinsics;
1156 int64_t SplittableBlocksCount = 0;
1157
1158 MemorySanitizerVisitor(Function &F, MemorySanitizer &MS,
1159 const TargetLibraryInfo &TLI)
1160 : F(F), MS(MS), VAHelper(CreateVarArgHelper(F, MS, *this)), TLI(&TLI) {
1161 bool SanitizeFunction =
1162 F.hasFnAttribute(Attribute::SanitizeMemory) && !ClDisableChecks;
1163 InsertChecks = SanitizeFunction;
1164 PropagateShadow = SanitizeFunction;
1165 PoisonStack = SanitizeFunction && ClPoisonStack;
1166 PoisonUndef = SanitizeFunction && ClPoisonUndef;
1167
1168 // In the presence of unreachable blocks, we may see Phi nodes with
1169 // incoming nodes from such blocks. Since InstVisitor skips unreachable
1170 // blocks, such nodes will not have any shadow value associated with them.
1171 // It's easier to remove unreachable blocks than deal with missing shadow.
1173
1174 MS.initializeCallbacks(*F.getParent(), TLI);
1175 FnPrologueEnd = IRBuilder<>(F.getEntryBlock().getFirstNonPHI())
1176 .CreateIntrinsic(Intrinsic::donothing, {}, {});
1177
1178 if (MS.CompileKernel) {
1179 IRBuilder<> IRB(FnPrologueEnd);
1180 insertKmsanPrologue(IRB);
1181 }
1182
1183 LLVM_DEBUG(if (!InsertChecks) dbgs()
1184 << "MemorySanitizer is not inserting checks into '"
1185 << F.getName() << "'\n");
1186 }
1187
1188 bool instrumentWithCalls(Value *V) {
1189 // Constants likely will be eliminated by follow-up passes.
1190 if (isa<Constant>(V))
1191 return false;
1192
1193 ++SplittableBlocksCount;
1195 SplittableBlocksCount > ClInstrumentationWithCallThreshold;
1196 }
1197
1198 bool isInPrologue(Instruction &I) {
1199 return I.getParent() == FnPrologueEnd->getParent() &&
1200 (&I == FnPrologueEnd || I.comesBefore(FnPrologueEnd));
1201 }
1202
1203 // Creates a new origin and records the stack trace. In general we can call
1204 // this function for any origin manipulation we like. However it will cost
1205 // runtime resources. So use this wisely only if it can provide additional
1206 // information helpful to a user.
1207 Value *updateOrigin(Value *V, IRBuilder<> &IRB) {
1208 if (MS.TrackOrigins <= 1)
1209 return V;
1210 return IRB.CreateCall(MS.MsanChainOriginFn, V);
1211 }
1212
1213 Value *originToIntptr(IRBuilder<> &IRB, Value *Origin) {
1214 const DataLayout &DL = F.getParent()->getDataLayout();
1215 unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1216 if (IntptrSize == kOriginSize)
1217 return Origin;
1218 assert(IntptrSize == kOriginSize * 2);
1219 Origin = IRB.CreateIntCast(Origin, MS.IntptrTy, /* isSigned */ false);
1220 return IRB.CreateOr(Origin, IRB.CreateShl(Origin, kOriginSize * 8));
1221 }
1222
1223 /// Fill memory range with the given origin value.
1224 void paintOrigin(IRBuilder<> &IRB, Value *Origin, Value *OriginPtr,
1225 TypeSize TS, Align Alignment) {
1226 const DataLayout &DL = F.getParent()->getDataLayout();
1227 const Align IntptrAlignment = DL.getABITypeAlign(MS.IntptrTy);
1228 unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1229 assert(IntptrAlignment >= kMinOriginAlignment);
1230 assert(IntptrSize >= kOriginSize);
1231
1232 // Note: The loop based formation works for fixed length vectors too,
1233 // however we prefer to unroll and specialize alignment below.
1234 if (TS.isScalable()) {
1235 Value *Size = IRB.CreateTypeSize(IRB.getInt32Ty(), TS);
1236 Value *RoundUp = IRB.CreateAdd(Size, IRB.getInt32(kOriginSize - 1));
1237 Value *End = IRB.CreateUDiv(RoundUp, IRB.getInt32(kOriginSize));
1238 auto [InsertPt, Index] =
1240 IRB.SetInsertPoint(InsertPt);
1241
1242 Value *GEP = IRB.CreateGEP(MS.OriginTy, OriginPtr, Index);
1244 return;
1245 }
1246
1247 unsigned Size = TS.getFixedValue();
1248
1249 unsigned Ofs = 0;
1250 Align CurrentAlignment = Alignment;
1251 if (Alignment >= IntptrAlignment && IntptrSize > kOriginSize) {
1252 Value *IntptrOrigin = originToIntptr(IRB, Origin);
1253 Value *IntptrOriginPtr =
1254 IRB.CreatePointerCast(OriginPtr, PointerType::get(MS.IntptrTy, 0));
1255 for (unsigned i = 0; i < Size / IntptrSize; ++i) {
1256 Value *Ptr = i ? IRB.CreateConstGEP1_32(MS.IntptrTy, IntptrOriginPtr, i)
1257 : IntptrOriginPtr;
1258 IRB.CreateAlignedStore(IntptrOrigin, Ptr, CurrentAlignment);
1259 Ofs += IntptrSize / kOriginSize;
1260 CurrentAlignment = IntptrAlignment;
1261 }
1262 }
1263
1264 for (unsigned i = Ofs; i < (Size + kOriginSize - 1) / kOriginSize; ++i) {
1265 Value *GEP =
1266 i ? IRB.CreateConstGEP1_32(MS.OriginTy, OriginPtr, i) : OriginPtr;
1267 IRB.CreateAlignedStore(Origin, GEP, CurrentAlignment);
1268 CurrentAlignment = kMinOriginAlignment;
1269 }
1270 }
1271
1272 void storeOrigin(IRBuilder<> &IRB, Value *Addr, Value *Shadow, Value *Origin,
1273 Value *OriginPtr, Align Alignment) {
1274 const DataLayout &DL = F.getParent()->getDataLayout();
1275 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1276 TypeSize StoreSize = DL.getTypeStoreSize(Shadow->getType());
1277 Value *ConvertedShadow = convertShadowToScalar(Shadow, IRB);
1278 if (auto *ConstantShadow = dyn_cast<Constant>(ConvertedShadow)) {
1279 if (!ClCheckConstantShadow || ConstantShadow->isZeroValue()) {
1280 // Origin is not needed: value is initialized or const shadow is
1281 // ignored.
1282 return;
1283 }
1284 if (llvm::isKnownNonZero(ConvertedShadow, DL)) {
1285 // Copy origin as the value is definitely uninitialized.
1286 paintOrigin(IRB, updateOrigin(Origin, IRB), OriginPtr, StoreSize,
1287 OriginAlignment);
1288 return;
1289 }
1290 // Fallback to runtime check, which still can be optimized out later.
1291 }
1292
1293 TypeSize TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
1294 unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1295 if (instrumentWithCalls(ConvertedShadow) &&
1296 SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
1297 FunctionCallee Fn = MS.MaybeStoreOriginFn[SizeIndex];
1298 Value *ConvertedShadow2 =
1299 IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1300 CallBase *CB = IRB.CreateCall(Fn, {ConvertedShadow2, Addr, Origin});
1301 CB->addParamAttr(0, Attribute::ZExt);
1302 CB->addParamAttr(2, Attribute::ZExt);
1303 } else {
1304 Value *Cmp = convertToBool(ConvertedShadow, IRB, "_mscmp");
1306 Cmp, &*IRB.GetInsertPoint(), false, MS.OriginStoreWeights);
1307 IRBuilder<> IRBNew(CheckTerm);
1308 paintOrigin(IRBNew, updateOrigin(Origin, IRBNew), OriginPtr, StoreSize,
1309 OriginAlignment);
1310 }
1311 }
1312
1313 void materializeStores() {
1314 for (StoreInst *SI : StoreList) {
1315 IRBuilder<> IRB(SI);
1316 Value *Val = SI->getValueOperand();
1317 Value *Addr = SI->getPointerOperand();
1318 Value *Shadow = SI->isAtomic() ? getCleanShadow(Val) : getShadow(Val);
1319 Value *ShadowPtr, *OriginPtr;
1320 Type *ShadowTy = Shadow->getType();
1321 const Align Alignment = SI->getAlign();
1322 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1323 std::tie(ShadowPtr, OriginPtr) =
1324 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ true);
1325
1326 StoreInst *NewSI = IRB.CreateAlignedStore(Shadow, ShadowPtr, Alignment);
1327 LLVM_DEBUG(dbgs() << " STORE: " << *NewSI << "\n");
1328 (void)NewSI;
1329
1330 if (SI->isAtomic())
1331 SI->setOrdering(addReleaseOrdering(SI->getOrdering()));
1332
1333 if (MS.TrackOrigins && !SI->isAtomic())
1334 storeOrigin(IRB, Addr, Shadow, getOrigin(Val), OriginPtr,
1335 OriginAlignment);
1336 }
1337 }
1338
1339 // Returns true if Debug Location curresponds to multiple warnings.
1340 bool shouldDisambiguateWarningLocation(const DebugLoc &DebugLoc) {
1341 if (MS.TrackOrigins < 2)
1342 return false;
1343
1344 if (LazyWarningDebugLocationCount.empty())
1345 for (const auto &I : InstrumentationList)
1346 ++LazyWarningDebugLocationCount[I.OrigIns->getDebugLoc()];
1347
1348 return LazyWarningDebugLocationCount[DebugLoc] >= ClDisambiguateWarning;
1349 }
1350
1351 /// Helper function to insert a warning at IRB's current insert point.
1352 void insertWarningFn(IRBuilder<> &IRB, Value *Origin) {
1353 if (!Origin)
1354 Origin = (Value *)IRB.getInt32(0);
1355 assert(Origin->getType()->isIntegerTy());
1356
1357 if (shouldDisambiguateWarningLocation(IRB.getCurrentDebugLocation())) {
1358 // Try to create additional origin with debug info of the last origin
1359 // instruction. It may provide additional information to the user.
1360 if (Instruction *OI = dyn_cast_or_null<Instruction>(Origin)) {
1361 assert(MS.TrackOrigins);
1362 auto NewDebugLoc = OI->getDebugLoc();
1363 // Origin update with missing or the same debug location provides no
1364 // additional value.
1365 if (NewDebugLoc && NewDebugLoc != IRB.getCurrentDebugLocation()) {
1366 // Insert update just before the check, so we call runtime only just
1367 // before the report.
1368 IRBuilder<> IRBOrigin(&*IRB.GetInsertPoint());
1369 IRBOrigin.SetCurrentDebugLocation(NewDebugLoc);
1370 Origin = updateOrigin(Origin, IRBOrigin);
1371 }
1372 }
1373 }
1374
1375 if (MS.CompileKernel || MS.TrackOrigins)
1376 IRB.CreateCall(MS.WarningFn, Origin)->setCannotMerge();
1377 else
1378 IRB.CreateCall(MS.WarningFn)->setCannotMerge();
1379 // FIXME: Insert UnreachableInst if !MS.Recover?
1380 // This may invalidate some of the following checks and needs to be done
1381 // at the very end.
1382 }
1383
1384 void materializeOneCheck(IRBuilder<> &IRB, Value *ConvertedShadow,
1385 Value *Origin) {
1386 const DataLayout &DL = F.getParent()->getDataLayout();
1387 TypeSize TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
1388 unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1389 if (instrumentWithCalls(ConvertedShadow) &&
1390 SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
1391 FunctionCallee Fn = MS.MaybeWarningFn[SizeIndex];
1392 Value *ConvertedShadow2 =
1393 IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1394 CallBase *CB = IRB.CreateCall(
1395 Fn, {ConvertedShadow2,
1396 MS.TrackOrigins && Origin ? Origin : (Value *)IRB.getInt32(0)});
1397 CB->addParamAttr(0, Attribute::ZExt);
1398 CB->addParamAttr(1, Attribute::ZExt);
1399 } else {
1400 Value *Cmp = convertToBool(ConvertedShadow, IRB, "_mscmp");
1402 Cmp, &*IRB.GetInsertPoint(),
1403 /* Unreachable */ !MS.Recover, MS.ColdCallWeights);
1404
1405 IRB.SetInsertPoint(CheckTerm);
1406 insertWarningFn(IRB, Origin);
1407 LLVM_DEBUG(dbgs() << " CHECK: " << *Cmp << "\n");
1408 }
1409 }
1410
1411 void materializeInstructionChecks(
1412 ArrayRef<ShadowOriginAndInsertPoint> InstructionChecks) {
1413 const DataLayout &DL = F.getParent()->getDataLayout();
1414 // Disable combining in some cases. TrackOrigins checks each shadow to pick
1415 // correct origin.
1416 bool Combine = !MS.TrackOrigins;
1417 Instruction *Instruction = InstructionChecks.front().OrigIns;
1418 Value *Shadow = nullptr;
1419 for (const auto &ShadowData : InstructionChecks) {
1420 assert(ShadowData.OrigIns == Instruction);
1422
1423 Value *ConvertedShadow = ShadowData.Shadow;
1424
1425 if (auto *ConstantShadow = dyn_cast<Constant>(ConvertedShadow)) {
1426 if (!ClCheckConstantShadow || ConstantShadow->isZeroValue()) {
1427 // Skip, value is initialized or const shadow is ignored.
1428 continue;
1429 }
1430 if (llvm::isKnownNonZero(ConvertedShadow, DL)) {
1431 // Report as the value is definitely uninitialized.
1432 insertWarningFn(IRB, ShadowData.Origin);
1433 if (!MS.Recover)
1434 return; // Always fail and stop here, not need to check the rest.
1435 // Skip entire instruction,
1436 continue;
1437 }
1438 // Fallback to runtime check, which still can be optimized out later.
1439 }
1440
1441 if (!Combine) {
1442 materializeOneCheck(IRB, ConvertedShadow, ShadowData.Origin);
1443 continue;
1444 }
1445
1446 if (!Shadow) {
1447 Shadow = ConvertedShadow;
1448 continue;
1449 }
1450
1451 Shadow = convertToBool(Shadow, IRB, "_mscmp");
1452 ConvertedShadow = convertToBool(ConvertedShadow, IRB, "_mscmp");
1453 Shadow = IRB.CreateOr(Shadow, ConvertedShadow, "_msor");
1454 }
1455
1456 if (Shadow) {
1457 assert(Combine);
1459 materializeOneCheck(IRB, Shadow, nullptr);
1460 }
1461 }
1462
1463 void materializeChecks() {
1464 llvm::stable_sort(InstrumentationList,
1465 [](const ShadowOriginAndInsertPoint &L,
1466 const ShadowOriginAndInsertPoint &R) {
1467 return L.OrigIns < R.OrigIns;
1468 });
1469
1470 for (auto I = InstrumentationList.begin();
1471 I != InstrumentationList.end();) {
1472 auto J =
1473 std::find_if(I + 1, InstrumentationList.end(),
1474 [L = I->OrigIns](const ShadowOriginAndInsertPoint &R) {
1475 return L != R.OrigIns;
1476 });
1477 // Process all checks of instruction at once.
1478 materializeInstructionChecks(ArrayRef<ShadowOriginAndInsertPoint>(I, J));
1479 I = J;
1480 }
1481
1482 LLVM_DEBUG(dbgs() << "DONE:\n" << F);
1483 }
1484
1485 // Returns the last instruction in the new prologue
1486 void insertKmsanPrologue(IRBuilder<> &IRB) {
1487 Value *ContextState = IRB.CreateCall(MS.MsanGetContextStateFn, {});
1488 Constant *Zero = IRB.getInt32(0);
1489 MS.ParamTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1490 {Zero, IRB.getInt32(0)}, "param_shadow");
1491 MS.RetvalTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1492 {Zero, IRB.getInt32(1)}, "retval_shadow");
1493 MS.VAArgTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1494 {Zero, IRB.getInt32(2)}, "va_arg_shadow");
1495 MS.VAArgOriginTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1496 {Zero, IRB.getInt32(3)}, "va_arg_origin");
1497 MS.VAArgOverflowSizeTLS =
1498 IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1499 {Zero, IRB.getInt32(4)}, "va_arg_overflow_size");
1500 MS.ParamOriginTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1501 {Zero, IRB.getInt32(5)}, "param_origin");
1502 MS.RetvalOriginTLS =
1503 IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1504 {Zero, IRB.getInt32(6)}, "retval_origin");
1505 if (MS.TargetTriple.getArch() == Triple::systemz)
1506 MS.MsanMetadataAlloca = IRB.CreateAlloca(MS.MsanMetadata, 0u);
1507 }
1508
1509 /// Add MemorySanitizer instrumentation to a function.
1510 bool runOnFunction() {
1511 // Iterate all BBs in depth-first order and create shadow instructions
1512 // for all instructions (where applicable).
1513 // For PHI nodes we create dummy shadow PHIs which will be finalized later.
1514 for (BasicBlock *BB : depth_first(FnPrologueEnd->getParent()))
1515 visit(*BB);
1516
1517 // Finalize PHI nodes.
1518 for (PHINode *PN : ShadowPHINodes) {
1519 PHINode *PNS = cast<PHINode>(getShadow(PN));
1520 PHINode *PNO = MS.TrackOrigins ? cast<PHINode>(getOrigin(PN)) : nullptr;
1521 size_t NumValues = PN->getNumIncomingValues();
1522 for (size_t v = 0; v < NumValues; v++) {
1523 PNS->addIncoming(getShadow(PN, v), PN->getIncomingBlock(v));
1524 if (PNO)
1525 PNO->addIncoming(getOrigin(PN, v), PN->getIncomingBlock(v));
1526 }
1527 }
1528
1529 VAHelper->finalizeInstrumentation();
1530
1531 // Poison llvm.lifetime.start intrinsics, if we haven't fallen back to
1532 // instrumenting only allocas.
1533 if (InstrumentLifetimeStart) {
1534 for (auto Item : LifetimeStartList) {
1535 instrumentAlloca(*Item.second, Item.first);
1536 AllocaSet.remove(Item.second);
1537 }
1538 }
1539 // Poison the allocas for which we didn't instrument the corresponding
1540 // lifetime intrinsics.
1541 for (AllocaInst *AI : AllocaSet)
1542 instrumentAlloca(*AI);
1543
1544 // Insert shadow value checks.
1545 materializeChecks();
1546
1547 // Delayed instrumentation of StoreInst.
1548 // This may not add new address checks.
1549 materializeStores();
1550
1551 return true;
1552 }
1553
1554 /// Compute the shadow type that corresponds to a given Value.
1555 Type *getShadowTy(Value *V) { return getShadowTy(V->getType()); }
1556
1557 /// Compute the shadow type that corresponds to a given Type.
1558 Type *getShadowTy(Type *OrigTy) {
1559 if (!OrigTy->isSized()) {
1560 return nullptr;
1561 }
1562 // For integer type, shadow is the same as the original type.
1563 // This may return weird-sized types like i1.
1564 if (IntegerType *IT = dyn_cast<IntegerType>(OrigTy))
1565 return IT;
1566 const DataLayout &DL = F.getParent()->getDataLayout();
1567 if (VectorType *VT = dyn_cast<VectorType>(OrigTy)) {
1568 uint32_t EltSize = DL.getTypeSizeInBits(VT->getElementType());
1569 return VectorType::get(IntegerType::get(*MS.C, EltSize),
1570 VT->getElementCount());
1571 }
1572 if (ArrayType *AT = dyn_cast<ArrayType>(OrigTy)) {
1573 return ArrayType::get(getShadowTy(AT->getElementType()),
1574 AT->getNumElements());
1575 }
1576 if (StructType *ST = dyn_cast<StructType>(OrigTy)) {
1578 for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
1579 Elements.push_back(getShadowTy(ST->getElementType(i)));
1580 StructType *Res = StructType::get(*MS.C, Elements, ST->isPacked());
1581 LLVM_DEBUG(dbgs() << "getShadowTy: " << *ST << " ===> " << *Res << "\n");
1582 return Res;
1583 }
1584 uint32_t TypeSize = DL.getTypeSizeInBits(OrigTy);
1585 return IntegerType::get(*MS.C, TypeSize);
1586 }
1587
1588 /// Extract combined shadow of struct elements as a bool
1589 Value *collapseStructShadow(StructType *Struct, Value *Shadow,
1590 IRBuilder<> &IRB) {
1591 Value *FalseVal = IRB.getIntN(/* width */ 1, /* value */ 0);
1592 Value *Aggregator = FalseVal;
1593
1594 for (unsigned Idx = 0; Idx < Struct->getNumElements(); Idx++) {
1595 // Combine by ORing together each element's bool shadow
1596 Value *ShadowItem = IRB.CreateExtractValue(Shadow, Idx);
1597 Value *ShadowBool = convertToBool(ShadowItem, IRB);
1598
1599 if (Aggregator != FalseVal)
1600 Aggregator = IRB.CreateOr(Aggregator, ShadowBool);
1601 else
1602 Aggregator = ShadowBool;
1603 }
1604
1605 return Aggregator;
1606 }
1607
1608 // Extract combined shadow of array elements
1609 Value *collapseArrayShadow(ArrayType *Array, Value *Shadow,
1610 IRBuilder<> &IRB) {
1611 if (!Array->getNumElements())
1612 return IRB.getIntN(/* width */ 1, /* value */ 0);
1613
1614 Value *FirstItem = IRB.CreateExtractValue(Shadow, 0);
1615 Value *Aggregator = convertShadowToScalar(FirstItem, IRB);
1616
1617 for (unsigned Idx = 1; Idx < Array->getNumElements(); Idx++) {
1618 Value *ShadowItem = IRB.CreateExtractValue(Shadow, Idx);
1619 Value *ShadowInner = convertShadowToScalar(ShadowItem, IRB);
1620 Aggregator = IRB.CreateOr(Aggregator, ShadowInner);
1621 }
1622 return Aggregator;
1623 }
1624
1625 /// Convert a shadow value to it's flattened variant. The resulting
1626 /// shadow may not necessarily have the same bit width as the input
1627 /// value, but it will always be comparable to zero.
1628 Value *convertShadowToScalar(Value *V, IRBuilder<> &IRB) {
1629 if (StructType *Struct = dyn_cast<StructType>(V->getType()))
1630 return collapseStructShadow(Struct, V, IRB);
1631 if (ArrayType *Array = dyn_cast<ArrayType>(V->getType()))
1632 return collapseArrayShadow(Array, V, IRB);
1633 if (isa<VectorType>(V->getType())) {
1634 if (isa<ScalableVectorType>(V->getType()))
1635 return convertShadowToScalar(IRB.CreateOrReduce(V), IRB);
1636 unsigned BitWidth =
1637 V->getType()->getPrimitiveSizeInBits().getFixedValue();
1638 return IRB.CreateBitCast(V, IntegerType::get(*MS.C, BitWidth));
1639 }
1640 return V;
1641 }
1642
1643 // Convert a scalar value to an i1 by comparing with 0
1644 Value *convertToBool(Value *V, IRBuilder<> &IRB, const Twine &name = "") {
1645 Type *VTy = V->getType();
1646 if (!VTy->isIntegerTy())
1647 return convertToBool(convertShadowToScalar(V, IRB), IRB, name);
1648 if (VTy->getIntegerBitWidth() == 1)
1649 // Just converting a bool to a bool, so do nothing.
1650 return V;
1651 return IRB.CreateICmpNE(V, ConstantInt::get(VTy, 0), name);
1652 }
1653
1654 Type *ptrToIntPtrType(Type *PtrTy) const {
1655 if (VectorType *VectTy = dyn_cast<VectorType>(PtrTy)) {
1656 return VectorType::get(ptrToIntPtrType(VectTy->getElementType()),
1657 VectTy->getElementCount());
1658 }
1659 assert(PtrTy->isIntOrPtrTy());
1660 return MS.IntptrTy;
1661 }
1662
1663 Type *getPtrToShadowPtrType(Type *IntPtrTy, Type *ShadowTy) const {
1664 if (VectorType *VectTy = dyn_cast<VectorType>(IntPtrTy)) {
1665 return VectorType::get(
1666 getPtrToShadowPtrType(VectTy->getElementType(), ShadowTy),
1667 VectTy->getElementCount());
1668 }
1669 assert(IntPtrTy == MS.IntptrTy);
1670 return PointerType::get(*MS.C, 0);
1671 }
1672
1673 Constant *constToIntPtr(Type *IntPtrTy, uint64_t C) const {
1674 if (VectorType *VectTy = dyn_cast<VectorType>(IntPtrTy)) {
1676 VectTy->getElementCount(), constToIntPtr(VectTy->getElementType(), C));
1677 }
1678 assert(IntPtrTy == MS.IntptrTy);
1679 return ConstantInt::get(MS.IntptrTy, C);
1680 }
1681
1682 /// Compute the integer shadow offset that corresponds to a given
1683 /// application address.
1684 ///
1685 /// Offset = (Addr & ~AndMask) ^ XorMask
1686 /// Addr can be a ptr or <N x ptr>. In both cases ShadowTy the shadow type of
1687 /// a single pointee.
1688 /// Returns <shadow_ptr, origin_ptr> or <<N x shadow_ptr>, <N x origin_ptr>>.
1689 Value *getShadowPtrOffset(Value *Addr, IRBuilder<> &IRB) {
1690 Type *IntptrTy = ptrToIntPtrType(Addr->getType());
1691 Value *OffsetLong = IRB.CreatePointerCast(Addr, IntptrTy);
1692
1693 if (uint64_t AndMask = MS.MapParams->AndMask)
1694 OffsetLong = IRB.CreateAnd(OffsetLong, constToIntPtr(IntptrTy, ~AndMask));
1695
1696 if (uint64_t XorMask = MS.MapParams->XorMask)
1697 OffsetLong = IRB.CreateXor(OffsetLong, constToIntPtr(IntptrTy, XorMask));
1698 return OffsetLong;
1699 }
1700
1701 /// Compute the shadow and origin addresses corresponding to a given
1702 /// application address.
1703 ///
1704 /// Shadow = ShadowBase + Offset
1705 /// Origin = (OriginBase + Offset) & ~3ULL
1706 /// Addr can be a ptr or <N x ptr>. In both cases ShadowTy the shadow type of
1707 /// a single pointee.
1708 /// Returns <shadow_ptr, origin_ptr> or <<N x shadow_ptr>, <N x origin_ptr>>.
1709 std::pair<Value *, Value *>
1710 getShadowOriginPtrUserspace(Value *Addr, IRBuilder<> &IRB, Type *ShadowTy,
1711 MaybeAlign Alignment) {
1712 VectorType *VectTy = dyn_cast<VectorType>(Addr->getType());
1713 if (!VectTy) {
1714 assert(Addr->getType()->isPointerTy());
1715 } else {
1716 assert(VectTy->getElementType()->isPointerTy());
1717 }
1718 Type *IntptrTy = ptrToIntPtrType(Addr->getType());
1719 Value *ShadowOffset = getShadowPtrOffset(Addr, IRB);
1720 Value *ShadowLong = ShadowOffset;
1721 if (uint64_t ShadowBase = MS.MapParams->ShadowBase) {
1722 ShadowLong =
1723 IRB.CreateAdd(ShadowLong, constToIntPtr(IntptrTy, ShadowBase));
1724 }
1725 Value *ShadowPtr = IRB.CreateIntToPtr(
1726 ShadowLong, getPtrToShadowPtrType(IntptrTy, ShadowTy));
1727
1728 Value *OriginPtr = nullptr;
1729 if (MS.TrackOrigins) {
1730 Value *OriginLong = ShadowOffset;
1731 uint64_t OriginBase = MS.MapParams->OriginBase;
1732 if (OriginBase != 0)
1733 OriginLong =
1734 IRB.CreateAdd(OriginLong, constToIntPtr(IntptrTy, OriginBase));
1735 if (!Alignment || *Alignment < kMinOriginAlignment) {
1737 OriginLong = IRB.CreateAnd(OriginLong, constToIntPtr(IntptrTy, ~Mask));
1738 }
1739 OriginPtr = IRB.CreateIntToPtr(
1740 OriginLong, getPtrToShadowPtrType(IntptrTy, MS.OriginTy));
1741 }
1742 return std::make_pair(ShadowPtr, OriginPtr);
1743 }
1744
1745 template <typename... ArgsTy>
1746 Value *createMetadataCall(IRBuilder<> &IRB, FunctionCallee Callee,
1747 ArgsTy... Args) {
1748 if (MS.TargetTriple.getArch() == Triple::systemz) {
1749 IRB.CreateCall(Callee,
1750 {MS.MsanMetadataAlloca, std::forward<ArgsTy>(Args)...});
1751 return IRB.CreateLoad(MS.MsanMetadata, MS.MsanMetadataAlloca);
1752 }
1753
1754 return IRB.CreateCall(Callee, {std::forward<ArgsTy>(Args)...});
1755 }
1756
1757 std::pair<Value *, Value *> getShadowOriginPtrKernelNoVec(Value *Addr,
1758 IRBuilder<> &IRB,
1759 Type *ShadowTy,
1760 bool isStore) {
1761 Value *ShadowOriginPtrs;
1762 const DataLayout &DL = F.getParent()->getDataLayout();
1763 TypeSize Size = DL.getTypeStoreSize(ShadowTy);
1764
1765 FunctionCallee Getter = MS.getKmsanShadowOriginAccessFn(isStore, Size);
1766 Value *AddrCast =
1767 IRB.CreatePointerCast(Addr, PointerType::get(IRB.getInt8Ty(), 0));
1768 if (Getter) {
1769 ShadowOriginPtrs = createMetadataCall(IRB, Getter, AddrCast);
1770 } else {
1771 Value *SizeVal = ConstantInt::get(MS.IntptrTy, Size);
1772 ShadowOriginPtrs = createMetadataCall(
1773 IRB,
1774 isStore ? MS.MsanMetadataPtrForStoreN : MS.MsanMetadataPtrForLoadN,
1775 AddrCast, SizeVal);
1776 }
1777 Value *ShadowPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 0);
1778 ShadowPtr = IRB.CreatePointerCast(ShadowPtr, PointerType::get(ShadowTy, 0));
1779 Value *OriginPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 1);
1780
1781 return std::make_pair(ShadowPtr, OriginPtr);
1782 }
1783
1784 /// Addr can be a ptr or <N x ptr>. In both cases ShadowTy the shadow type of
1785 /// a single pointee.
1786 /// Returns <shadow_ptr, origin_ptr> or <<N x shadow_ptr>, <N x origin_ptr>>.
1787 std::pair<Value *, Value *> getShadowOriginPtrKernel(Value *Addr,
1788 IRBuilder<> &IRB,
1789 Type *ShadowTy,
1790 bool isStore) {
1791 VectorType *VectTy = dyn_cast<VectorType>(Addr->getType());
1792 if (!VectTy) {
1793 assert(Addr->getType()->isPointerTy());
1794 return getShadowOriginPtrKernelNoVec(Addr, IRB, ShadowTy, isStore);
1795 }
1796
1797 // TODO: Support callbacs with vectors of addresses.
1798 unsigned NumElements = cast<FixedVectorType>(VectTy)->getNumElements();
1799 Value *ShadowPtrs = ConstantInt::getNullValue(
1800 FixedVectorType::get(IRB.getPtrTy(), NumElements));
1801 Value *OriginPtrs = nullptr;
1802 if (MS.TrackOrigins)
1803 OriginPtrs = ConstantInt::getNullValue(
1804 FixedVectorType::get(IRB.getPtrTy(), NumElements));
1805 for (unsigned i = 0; i < NumElements; ++i) {
1806 Value *OneAddr =
1807 IRB.CreateExtractElement(Addr, ConstantInt::get(IRB.getInt32Ty(), i));
1808 auto [ShadowPtr, OriginPtr] =
1809 getShadowOriginPtrKernelNoVec(OneAddr, IRB, ShadowTy, isStore);
1810
1811 ShadowPtrs = IRB.CreateInsertElement(
1812 ShadowPtrs, ShadowPtr, ConstantInt::get(IRB.getInt32Ty(), i));
1813 if (MS.TrackOrigins)
1814 OriginPtrs = IRB.CreateInsertElement(
1815 OriginPtrs, OriginPtr, ConstantInt::get(IRB.getInt32Ty(), i));
1816 }
1817 return {ShadowPtrs, OriginPtrs};
1818 }
1819
1820 std::pair<Value *, Value *> getShadowOriginPtr(Value *Addr, IRBuilder<> &IRB,
1821 Type *ShadowTy,
1822 MaybeAlign Alignment,
1823 bool isStore) {
1824 if (MS.CompileKernel)
1825 return getShadowOriginPtrKernel(Addr, IRB, ShadowTy, isStore);
1826 return getShadowOriginPtrUserspace(Addr, IRB, ShadowTy, Alignment);
1827 }
1828
1829 /// Compute the shadow address for a given function argument.
1830 ///
1831 /// Shadow = ParamTLS+ArgOffset.
1832 Value *getShadowPtrForArgument(IRBuilder<> &IRB, int ArgOffset) {
1833 Value *Base = IRB.CreatePointerCast(MS.ParamTLS, MS.IntptrTy);
1834 if (ArgOffset)
1835 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1836 return IRB.CreateIntToPtr(Base, IRB.getPtrTy(0), "_msarg");
1837 }
1838
1839 /// Compute the origin address for a given function argument.
1840 Value *getOriginPtrForArgument(IRBuilder<> &IRB, int ArgOffset) {
1841 if (!MS.TrackOrigins)
1842 return nullptr;
1843 Value *Base = IRB.CreatePointerCast(MS.ParamOriginTLS, MS.IntptrTy);
1844 if (ArgOffset)
1845 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1846 return IRB.CreateIntToPtr(Base, IRB.getPtrTy(0), "_msarg_o");
1847 }
1848
1849 /// Compute the shadow address for a retval.
1850 Value *getShadowPtrForRetval(IRBuilder<> &IRB) {
1851 return IRB.CreatePointerCast(MS.RetvalTLS, IRB.getPtrTy(0), "_msret");
1852 }
1853
1854 /// Compute the origin address for a retval.
1855 Value *getOriginPtrForRetval() {
1856 // We keep a single origin for the entire retval. Might be too optimistic.
1857 return MS.RetvalOriginTLS;
1858 }
1859
1860 /// Set SV to be the shadow value for V.
1861 void setShadow(Value *V, Value *SV) {
1862 assert(!ShadowMap.count(V) && "Values may only have one shadow");
1863 ShadowMap[V] = PropagateShadow ? SV : getCleanShadow(V);
1864 }
1865
1866 /// Set Origin to be the origin value for V.
1867 void setOrigin(Value *V, Value *Origin) {
1868 if (!MS.TrackOrigins)
1869 return;
1870 assert(!OriginMap.count(V) && "Values may only have one origin");
1871 LLVM_DEBUG(dbgs() << "ORIGIN: " << *V << " ==> " << *Origin << "\n");
1872 OriginMap[V] = Origin;
1873 }
1874
1875 Constant *getCleanShadow(Type *OrigTy) {
1876 Type *ShadowTy = getShadowTy(OrigTy);
1877 if (!ShadowTy)
1878 return nullptr;
1879 return Constant::getNullValue(ShadowTy);
1880 }
1881
1882 /// Create a clean shadow value for a given value.
1883 ///
1884 /// Clean shadow (all zeroes) means all bits of the value are defined
1885 /// (initialized).
1886 Constant *getCleanShadow(Value *V) { return getCleanShadow(V->getType()); }
1887
1888 /// Create a dirty shadow of a given shadow type.
1889 Constant *getPoisonedShadow(Type *ShadowTy) {
1890 assert(ShadowTy);
1891 if (isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy))
1892 return Constant::getAllOnesValue(ShadowTy);
1893 if (ArrayType *AT = dyn_cast<ArrayType>(ShadowTy)) {
1894 SmallVector<Constant *, 4> Vals(AT->getNumElements(),
1895 getPoisonedShadow(AT->getElementType()));
1896 return ConstantArray::get(AT, Vals);
1897 }
1898 if (StructType *ST = dyn_cast<StructType>(ShadowTy)) {
1900 for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
1901 Vals.push_back(getPoisonedShadow(ST->getElementType(i)));
1902 return ConstantStruct::get(ST, Vals);
1903 }
1904 llvm_unreachable("Unexpected shadow type");
1905 }
1906
1907 /// Create a dirty shadow for a given value.
1908 Constant *getPoisonedShadow(Value *V) {
1909 Type *ShadowTy = getShadowTy(V);
1910 if (!ShadowTy)
1911 return nullptr;
1912 return getPoisonedShadow(ShadowTy);
1913 }
1914
1915 /// Create a clean (zero) origin.
1916 Value *getCleanOrigin() { return Constant::getNullValue(MS.OriginTy); }
1917
1918 /// Get the shadow value for a given Value.
1919 ///
1920 /// This function either returns the value set earlier with setShadow,
1921 /// or extracts if from ParamTLS (for function arguments).
1922 Value *getShadow(Value *V) {
1923 if (Instruction *I = dyn_cast<Instruction>(V)) {
1924 if (!PropagateShadow || I->getMetadata(LLVMContext::MD_nosanitize))
1925 return getCleanShadow(V);
1926 // For instructions the shadow is already stored in the map.
1927 Value *Shadow = ShadowMap[V];
1928 if (!Shadow) {
1929 LLVM_DEBUG(dbgs() << "No shadow: " << *V << "\n" << *(I->getParent()));
1930 (void)I;
1931 assert(Shadow && "No shadow for a value");
1932 }
1933 return Shadow;
1934 }
1935 if (UndefValue *U = dyn_cast<UndefValue>(V)) {
1936 Value *AllOnes = (PropagateShadow && PoisonUndef) ? getPoisonedShadow(V)
1937 : getCleanShadow(V);
1938 LLVM_DEBUG(dbgs() << "Undef: " << *U << " ==> " << *AllOnes << "\n");
1939 (void)U;
1940 return AllOnes;
1941 }
1942 if (Argument *A = dyn_cast<Argument>(V)) {
1943 // For arguments we compute the shadow on demand and store it in the map.
1944 Value *&ShadowPtr = ShadowMap[V];
1945 if (ShadowPtr)
1946 return ShadowPtr;
1947 Function *F = A->getParent();
1948 IRBuilder<> EntryIRB(FnPrologueEnd);
1949 unsigned ArgOffset = 0;
1950 const DataLayout &DL = F->getParent()->getDataLayout();
1951 for (auto &FArg : F->args()) {
1952 if (!FArg.getType()->isSized()) {
1953 LLVM_DEBUG(dbgs() << "Arg is not sized\n");
1954 continue;
1955 }
1956
1957 unsigned Size = FArg.hasByValAttr()
1958 ? DL.getTypeAllocSize(FArg.getParamByValType())
1959 : DL.getTypeAllocSize(FArg.getType());
1960
1961 if (A == &FArg) {
1962 bool Overflow = ArgOffset + Size > kParamTLSSize;
1963 if (FArg.hasByValAttr()) {
1964 // ByVal pointer itself has clean shadow. We copy the actual
1965 // argument shadow to the underlying memory.
1966 // Figure out maximal valid memcpy alignment.
1967 const Align ArgAlign = DL.getValueOrABITypeAlignment(
1968 FArg.getParamAlign(), FArg.getParamByValType());
1969 Value *CpShadowPtr, *CpOriginPtr;
1970 std::tie(CpShadowPtr, CpOriginPtr) =
1971 getShadowOriginPtr(V, EntryIRB, EntryIRB.getInt8Ty(), ArgAlign,
1972 /*isStore*/ true);
1973 if (!PropagateShadow || Overflow) {
1974 // ParamTLS overflow.
1975 EntryIRB.CreateMemSet(
1976 CpShadowPtr, Constant::getNullValue(EntryIRB.getInt8Ty()),
1977 Size, ArgAlign);
1978 } else {
1979 Value *Base = getShadowPtrForArgument(EntryIRB, ArgOffset);
1980 const Align CopyAlign = std::min(ArgAlign, kShadowTLSAlignment);
1981 Value *Cpy = EntryIRB.CreateMemCpy(CpShadowPtr, CopyAlign, Base,
1982 CopyAlign, Size);
1983 LLVM_DEBUG(dbgs() << " ByValCpy: " << *Cpy << "\n");
1984 (void)Cpy;
1985
1986 if (MS.TrackOrigins) {
1987 Value *OriginPtr =
1988 getOriginPtrForArgument(EntryIRB, ArgOffset);
1989 // FIXME: OriginSize should be:
1990 // alignTo(V % kMinOriginAlignment + Size, kMinOriginAlignment)
1991 unsigned OriginSize = alignTo(Size, kMinOriginAlignment);
1992 EntryIRB.CreateMemCpy(
1993 CpOriginPtr,
1994 /* by getShadowOriginPtr */ kMinOriginAlignment, OriginPtr,
1995 /* by origin_tls[ArgOffset] */ kMinOriginAlignment,
1996 OriginSize);
1997 }
1998 }
1999 }
2000
2001 if (!PropagateShadow || Overflow || FArg.hasByValAttr() ||
2002 (MS.EagerChecks && FArg.hasAttribute(Attribute::NoUndef))) {
2003 ShadowPtr = getCleanShadow(V);
2004 setOrigin(A, getCleanOrigin());
2005 } else {
2006 // Shadow over TLS
2007 Value *Base = getShadowPtrForArgument(EntryIRB, ArgOffset);
2008 ShadowPtr = EntryIRB.CreateAlignedLoad(getShadowTy(&FArg), Base,
2010 if (MS.TrackOrigins) {
2011 Value *OriginPtr =
2012 getOriginPtrForArgument(EntryIRB, ArgOffset);
2013 setOrigin(A, EntryIRB.CreateLoad(MS.OriginTy, OriginPtr));
2014 }
2015 }
2017 << " ARG: " << FArg << " ==> " << *ShadowPtr << "\n");
2018 break;
2019 }
2020
2021 ArgOffset += alignTo(Size, kShadowTLSAlignment);
2022 }
2023 assert(ShadowPtr && "Could not find shadow for an argument");
2024 return ShadowPtr;
2025 }
2026 // For everything else the shadow is zero.
2027 return getCleanShadow(V);
2028 }
2029
2030 /// Get the shadow for i-th argument of the instruction I.
2031 Value *getShadow(Instruction *I, int i) {
2032 return getShadow(I->getOperand(i));
2033 }
2034
2035 /// Get the origin for a value.
2036 Value *getOrigin(Value *V) {
2037 if (!MS.TrackOrigins)
2038 return nullptr;
2039 if (!PropagateShadow || isa<Constant>(V) || isa<InlineAsm>(V))
2040 return getCleanOrigin();
2041 assert((isa<Instruction>(V) || isa<Argument>(V)) &&
2042 "Unexpected value type in getOrigin()");
2043 if (Instruction *I = dyn_cast<Instruction>(V)) {
2044 if (I->getMetadata(LLVMContext::MD_nosanitize))
2045 return getCleanOrigin();
2046 }
2047 Value *Origin = OriginMap[V];
2048 assert(Origin && "Missing origin");
2049 return Origin;
2050 }
2051
2052 /// Get the origin for i-th argument of the instruction I.
2053 Value *getOrigin(Instruction *I, int i) {
2054 return getOrigin(I->getOperand(i));
2055 }
2056
2057 /// Remember the place where a shadow check should be inserted.
2058 ///
2059 /// This location will be later instrumented with a check that will print a
2060 /// UMR warning in runtime if the shadow value is not 0.
2061 void insertShadowCheck(Value *Shadow, Value *Origin, Instruction *OrigIns) {
2062 assert(Shadow);
2063 if (!InsertChecks)
2064 return;
2065
2066 if (!DebugCounter::shouldExecute(DebugInsertCheck)) {
2067 LLVM_DEBUG(dbgs() << "Skipping check of " << *Shadow << " before "
2068 << *OrigIns << "\n");
2069 return;
2070 }
2071#ifndef NDEBUG
2072 Type *ShadowTy = Shadow->getType();
2073 assert((isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy) ||
2074 isa<StructType>(ShadowTy) || isa<ArrayType>(ShadowTy)) &&
2075 "Can only insert checks for integer, vector, and aggregate shadow "
2076 "types");
2077#endif
2078 InstrumentationList.push_back(
2079 ShadowOriginAndInsertPoint(Shadow, Origin, OrigIns));
2080 }
2081
2082 /// Remember the place where a shadow check should be inserted.
2083 ///
2084 /// This location will be later instrumented with a check that will print a
2085 /// UMR warning in runtime if the value is not fully defined.
2086 void insertShadowCheck(Value *Val, Instruction *OrigIns) {
2087 assert(Val);
2088 Value *Shadow, *Origin;
2090 Shadow = getShadow(Val);
2091 if (!Shadow)
2092 return;
2093 Origin = getOrigin(Val);
2094 } else {
2095 Shadow = dyn_cast_or_null<Instruction>(getShadow(Val));
2096 if (!Shadow)
2097 return;
2098 Origin = dyn_cast_or_null<Instruction>(getOrigin(Val));
2099 }
2100 insertShadowCheck(Shadow, Origin, OrigIns);
2101 }
2102
2104 switch (a) {
2105 case AtomicOrdering::NotAtomic:
2106 return AtomicOrdering::NotAtomic;
2107 case AtomicOrdering::Unordered:
2108 case AtomicOrdering::Monotonic:
2109 case AtomicOrdering::Release:
2110 return AtomicOrdering::Release;
2111 case AtomicOrdering::Acquire:
2112 case AtomicOrdering::AcquireRelease:
2113 return AtomicOrdering::AcquireRelease;
2114 case AtomicOrdering::SequentiallyConsistent:
2115 return AtomicOrdering::SequentiallyConsistent;
2116 }
2117 llvm_unreachable("Unknown ordering");
2118 }
2119
2120 Value *makeAddReleaseOrderingTable(IRBuilder<> &IRB) {
2121 constexpr int NumOrderings = (int)AtomicOrderingCABI::seq_cst + 1;
2122 uint32_t OrderingTable[NumOrderings] = {};
2123
2124 OrderingTable[(int)AtomicOrderingCABI::relaxed] =
2125 OrderingTable[(int)AtomicOrderingCABI::release] =
2126 (int)AtomicOrderingCABI::release;
2127 OrderingTable[(int)AtomicOrderingCABI::consume] =
2128 OrderingTable[(int)AtomicOrderingCABI::acquire] =
2129 OrderingTable[(int)AtomicOrderingCABI::acq_rel] =
2130 (int)AtomicOrderingCABI::acq_rel;
2131 OrderingTable[(int)AtomicOrderingCABI::seq_cst] =
2132 (int)AtomicOrderingCABI::seq_cst;
2133
2135 ArrayRef(OrderingTable, NumOrderings));
2136 }
2137
2139 switch (a) {
2140 case AtomicOrdering::NotAtomic:
2141 return AtomicOrdering::NotAtomic;
2142 case AtomicOrdering::Unordered:
2143 case AtomicOrdering::Monotonic:
2144 case AtomicOrdering::Acquire:
2145 return AtomicOrdering::Acquire;
2146 case AtomicOrdering::Release:
2147 case AtomicOrdering::AcquireRelease:
2148 return AtomicOrdering::AcquireRelease;
2149 case AtomicOrdering::SequentiallyConsistent:
2150 return AtomicOrdering::SequentiallyConsistent;
2151 }
2152 llvm_unreachable("Unknown ordering");
2153 }
2154
2155 Value *makeAddAcquireOrderingTable(IRBuilder<> &IRB) {
2156 constexpr int NumOrderings = (int)AtomicOrderingCABI::seq_cst + 1;
2157 uint32_t OrderingTable[NumOrderings] = {};
2158
2159 OrderingTable[(int)AtomicOrderingCABI::relaxed] =
2160 OrderingTable[(int)AtomicOrderingCABI::acquire] =
2161 OrderingTable[(int)AtomicOrderingCABI::consume] =
2162 (int)AtomicOrderingCABI::acquire;
2163 OrderingTable[(int)AtomicOrderingCABI::release] =
2164 OrderingTable[(int)AtomicOrderingCABI::acq_rel] =
2165 (int)AtomicOrderingCABI::acq_rel;
2166 OrderingTable[(int)AtomicOrderingCABI::seq_cst] =
2167 (int)AtomicOrderingCABI::seq_cst;
2168
2170 ArrayRef(OrderingTable, NumOrderings));
2171 }
2172
2173 // ------------------- Visitors.
2174 using InstVisitor<MemorySanitizerVisitor>::visit;
2175 void visit(Instruction &I) {
2176 if (I.getMetadata(LLVMContext::MD_nosanitize))
2177 return;
2178 // Don't want to visit if we're in the prologue
2179 if (isInPrologue(I))
2180 return;
2182 }
2183
2184 /// Instrument LoadInst
2185 ///
2186 /// Loads the corresponding shadow and (optionally) origin.
2187 /// Optionally, checks that the load address is fully defined.
2188 void visitLoadInst(LoadInst &I) {
2189 assert(I.getType()->isSized() && "Load type must have size");
2190 assert(!I.getMetadata(LLVMContext::MD_nosanitize));
2191 NextNodeIRBuilder IRB(&I);
2192 Type *ShadowTy = getShadowTy(&I);
2193 Value *Addr = I.getPointerOperand();
2194 Value *ShadowPtr = nullptr, *OriginPtr = nullptr;
2195 const Align Alignment = I.getAlign();
2196 if (PropagateShadow) {
2197 std::tie(ShadowPtr, OriginPtr) =
2198 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
2199 setShadow(&I,
2200 IRB.CreateAlignedLoad(ShadowTy, ShadowPtr, Alignment, "_msld"));
2201 } else {
2202 setShadow(&I, getCleanShadow(&I));
2203 }
2204
2206 insertShadowCheck(I.getPointerOperand(), &I);
2207
2208 if (I.isAtomic())
2209 I.setOrdering(addAcquireOrdering(I.getOrdering()));
2210
2211 if (MS.TrackOrigins) {
2212 if (PropagateShadow) {
2213 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
2214 setOrigin(
2215 &I, IRB.CreateAlignedLoad(MS.OriginTy, OriginPtr, OriginAlignment));
2216 } else {
2217 setOrigin(&I, getCleanOrigin());
2218 }
2219 }
2220 }
2221
2222 /// Instrument StoreInst
2223 ///
2224 /// Stores the corresponding shadow and (optionally) origin.
2225 /// Optionally, checks that the store address is fully defined.
2226 void visitStoreInst(StoreInst &I) {
2227 StoreList.push_back(&I);
2229 insertShadowCheck(I.getPointerOperand(), &I);
2230 }
2231
2232 void handleCASOrRMW(Instruction &I) {
2233 assert(isa<AtomicRMWInst>(I) || isa<AtomicCmpXchgInst>(I));
2234
2235 IRBuilder<> IRB(&I);
2236 Value *Addr = I.getOperand(0);
2237 Value *Val = I.getOperand(1);
2238 Value *ShadowPtr = getShadowOriginPtr(Addr, IRB, getShadowTy(Val), Align(1),
2239 /*isStore*/ true)
2240 .first;
2241
2243 insertShadowCheck(Addr, &I);
2244
2245 // Only test the conditional argument of cmpxchg instruction.
2246 // The other argument can potentially be uninitialized, but we can not
2247 // detect this situation reliably without possible false positives.
2248 if (isa<AtomicCmpXchgInst>(I))
2249 insertShadowCheck(Val, &I);
2250
2251 IRB.CreateStore(getCleanShadow(Val), ShadowPtr);
2252
2253 setShadow(&I, getCleanShadow(&I));
2254 setOrigin(&I, getCleanOrigin());
2255 }
2256
2257 void visitAtomicRMWInst(AtomicRMWInst &I) {
2258 handleCASOrRMW(I);
2259 I.setOrdering(addReleaseOrdering(I.getOrdering()));
2260 }
2261
2262 void visitAtomicCmpXchgInst(AtomicCmpXchgInst &I) {
2263 handleCASOrRMW(I);
2264 I.setSuccessOrdering(addReleaseOrdering(I.getSuccessOrdering()));
2265 }
2266
2267 // Vector manipulation.
2268 void visitExtractElementInst(ExtractElementInst &I) {
2269 insertShadowCheck(I.getOperand(1), &I);
2270 IRBuilder<> IRB(&I);
2271 setShadow(&I, IRB.CreateExtractElement(getShadow(&I, 0), I.getOperand(1),
2272 "_msprop"));
2273 setOrigin(&I, getOrigin(&I, 0));
2274 }
2275
2276 void visitInsertElementInst(InsertElementInst &I) {
2277 insertShadowCheck(I.getOperand(2), &I);
2278 IRBuilder<> IRB(&I);
2279 auto *Shadow0 = getShadow(&I, 0);
2280 auto *Shadow1 = getShadow(&I, 1);
2281 setShadow(&I, IRB.CreateInsertElement(Shadow0, Shadow1, I.getOperand(2),
2282 "_msprop"));
2283 setOriginForNaryOp(I);
2284 }
2285
2286 void visitShuffleVectorInst(ShuffleVectorInst &I) {
2287 IRBuilder<> IRB(&I);
2288 auto *Shadow0 = getShadow(&I, 0);
2289 auto *Shadow1 = getShadow(&I, 1);
2290 setShadow(&I, IRB.CreateShuffleVector(Shadow0, Shadow1, I.getShuffleMask(),
2291 "_msprop"));
2292 setOriginForNaryOp(I);
2293 }
2294
2295 // Casts.
2296 void visitSExtInst(SExtInst &I) {
2297 IRBuilder<> IRB(&I);
2298 setShadow(&I, IRB.CreateSExt(getShadow(&I, 0), I.getType(), "_msprop"));
2299 setOrigin(&I, getOrigin(&I, 0));
2300 }
2301
2302 void visitZExtInst(ZExtInst &I) {
2303 IRBuilder<> IRB(&I);
2304 setShadow(&I, IRB.CreateZExt(getShadow(&I, 0), I.getType(), "_msprop"));
2305 setOrigin(&I, getOrigin(&I, 0));
2306 }
2307
2308 void visitTruncInst(TruncInst &I) {
2309 IRBuilder<> IRB(&I);
2310 setShadow(&I, IRB.CreateTrunc(getShadow(&I, 0), I.getType(), "_msprop"));
2311 setOrigin(&I, getOrigin(&I, 0));
2312 }
2313
2314 void visitBitCastInst(BitCastInst &I) {
2315 // Special case: if this is the bitcast (there is exactly 1 allowed) between
2316 // a musttail call and a ret, don't instrument. New instructions are not
2317 // allowed after a musttail call.
2318 if (auto *CI = dyn_cast<CallInst>(I.getOperand(0)))
2319 if (CI->isMustTailCall())
2320 return;
2321 IRBuilder<> IRB(&I);
2322 setShadow(&I, IRB.CreateBitCast(getShadow(&I, 0), getShadowTy(&I)));
2323 setOrigin(&I, getOrigin(&I, 0));
2324 }
2325
2326 void visitPtrToIntInst(PtrToIntInst &I) {
2327 IRBuilder<> IRB(&I);
2328 setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
2329 "_msprop_ptrtoint"));
2330 setOrigin(&I, getOrigin(&I, 0));
2331 }
2332
2333 void visitIntToPtrInst(IntToPtrInst &I) {
2334 IRBuilder<> IRB(&I);
2335 setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
2336 "_msprop_inttoptr"));
2337 setOrigin(&I, getOrigin(&I, 0));
2338 }
2339
2340 void visitFPToSIInst(CastInst &I) { handleShadowOr(I); }
2341 void visitFPToUIInst(CastInst &I) { handleShadowOr(I); }
2342 void visitSIToFPInst(CastInst &I) { handleShadowOr(I); }
2343 void visitUIToFPInst(CastInst &I) { handleShadowOr(I); }
2344 void visitFPExtInst(CastInst &I) { handleShadowOr(I); }
2345 void visitFPTruncInst(CastInst &I) { handleShadowOr(I); }
2346
2347 /// Propagate shadow for bitwise AND.
2348 ///
2349 /// This code is exact, i.e. if, for example, a bit in the left argument
2350 /// is defined and 0, then neither the value not definedness of the
2351 /// corresponding bit in B don't affect the resulting shadow.
2352 void visitAnd(BinaryOperator &I) {
2353 IRBuilder<> IRB(&I);
2354 // "And" of 0 and a poisoned value results in unpoisoned value.
2355 // 1&1 => 1; 0&1 => 0; p&1 => p;
2356 // 1&0 => 0; 0&0 => 0; p&0 => 0;
2357 // 1&p => p; 0&p => 0; p&p => p;
2358 // S = (S1 & S2) | (V1 & S2) | (S1 & V2)
2359 Value *S1 = getShadow(&I, 0);
2360 Value *S2 = getShadow(&I, 1);
2361 Value *V1 = I.getOperand(0);
2362 Value *V2 = I.getOperand(1);
2363 if (V1->getType() != S1->getType()) {
2364 V1 = IRB.CreateIntCast(V1, S1->getType(), false);
2365 V2 = IRB.CreateIntCast(V2, S2->getType(), false);
2366 }
2367 Value *S1S2 = IRB.CreateAnd(S1, S2);
2368 Value *V1S2 = IRB.CreateAnd(V1, S2);
2369 Value *S1V2 = IRB.CreateAnd(S1, V2);
2370 setShadow(&I, IRB.CreateOr({S1S2, V1S2, S1V2}));
2371 setOriginForNaryOp(I);
2372 }
2373
2374 void visitOr(BinaryOperator &I) {
2375 IRBuilder<> IRB(&I);
2376 // "Or" of 1 and a poisoned value results in unpoisoned value.
2377 // 1|1 => 1; 0|1 => 1; p|1 => 1;
2378 // 1|0 => 1; 0|0 => 0; p|0 => p;
2379 // 1|p => 1; 0|p => p; p|p => p;
2380 // S = (S1 & S2) | (~V1 & S2) | (S1 & ~V2)
2381 Value *S1 = getShadow(&I, 0);
2382 Value *S2 = getShadow(&I, 1);
2383 Value *V1 = IRB.CreateNot(I.getOperand(0));
2384 Value *V2 = IRB.CreateNot(I.getOperand(1));
2385 if (V1->getType() != S1->getType()) {
2386 V1 = IRB.CreateIntCast(V1, S1->getType(), false);
2387 V2 = IRB.CreateIntCast(V2, S2->getType(), false);
2388 }
2389 Value *S1S2 = IRB.CreateAnd(S1, S2);
2390 Value *V1S2 = IRB.CreateAnd(V1, S2);
2391 Value *S1V2 = IRB.CreateAnd(S1, V2);
2392 setShadow(&I, IRB.CreateOr({S1S2, V1S2, S1V2}));
2393 setOriginForNaryOp(I);
2394 }
2395
2396 /// Default propagation of shadow and/or origin.
2397 ///
2398 /// This class implements the general case of shadow propagation, used in all
2399 /// cases where we don't know and/or don't care about what the operation
2400 /// actually does. It converts all input shadow values to a common type
2401 /// (extending or truncating as necessary), and bitwise OR's them.
2402 ///
2403 /// This is much cheaper than inserting checks (i.e. requiring inputs to be
2404 /// fully initialized), and less prone to false positives.
2405 ///
2406 /// This class also implements the general case of origin propagation. For a
2407 /// Nary operation, result origin is set to the origin of an argument that is
2408 /// not entirely initialized. If there is more than one such arguments, the
2409 /// rightmost of them is picked. It does not matter which one is picked if all
2410 /// arguments are initialized.
2411 template <bool CombineShadow> class Combiner {
2412 Value *Shadow = nullptr;
2413 Value *Origin = nullptr;
2414 IRBuilder<> &IRB;
2415 MemorySanitizerVisitor *MSV;
2416
2417 public:
2418 Combiner(MemorySanitizerVisitor *MSV, IRBuilder<> &IRB)
2419 : IRB(IRB), MSV(MSV) {}
2420
2421 /// Add a pair of shadow and origin values to the mix.
2422 Combiner &Add(Value *OpShadow, Value *OpOrigin) {
2423 if (CombineShadow) {
2424 assert(OpShadow);
2425 if (!Shadow)
2426 Shadow = OpShadow;
2427 else {
2428 OpShadow = MSV->CreateShadowCast(IRB, OpShadow, Shadow->getType());
2429 Shadow = IRB.CreateOr(Shadow, OpShadow, "_msprop");
2430 }
2431 }
2432
2433 if (MSV->MS.TrackOrigins) {
2434 assert(OpOrigin);
2435 if (!Origin) {
2436 Origin = OpOrigin;
2437 } else {
2438 Constant *ConstOrigin = dyn_cast<Constant>(OpOrigin);
2439 // No point in adding something that might result in 0 origin value.
2440 if (!ConstOrigin || !ConstOrigin->isNullValue()) {
2441 Value *Cond = MSV->convertToBool(OpShadow, IRB);
2442 Origin = IRB.CreateSelect(Cond, OpOrigin, Origin);
2443 }
2444 }
2445 }
2446 return *this;
2447 }
2448
2449 /// Add an application value to the mix.
2450 Combiner &Add(Value *V) {
2451 Value *OpShadow = MSV->getShadow(V);
2452 Value *OpOrigin = MSV->MS.TrackOrigins ? MSV->getOrigin(V) : nullptr;
2453 return Add(OpShadow, OpOrigin);
2454 }
2455
2456 /// Set the current combined values as the given instruction's shadow
2457 /// and origin.
2458 void Done(Instruction *I) {
2459 if (CombineShadow) {
2460 assert(Shadow);
2461 Shadow = MSV->CreateShadowCast(IRB, Shadow, MSV->getShadowTy(I));
2462 MSV->setShadow(I, Shadow);
2463 }
2464 if (MSV->MS.TrackOrigins) {
2465 assert(Origin);
2466 MSV->setOrigin(I, Origin);
2467 }
2468 }
2469 };
2470
2471 using ShadowAndOriginCombiner = Combiner<true>;
2472 using OriginCombiner = Combiner<false>;
2473
2474 /// Propagate origin for arbitrary operation.
2475 void setOriginForNaryOp(Instruction &I) {
2476 if (!MS.TrackOrigins)
2477 return;
2478 IRBuilder<> IRB(&I);
2479 OriginCombiner OC(this, IRB);
2480 for (Use &Op : I.operands())
2481 OC.Add(Op.get());
2482 OC.Done(&I);
2483 }
2484
2485 size_t VectorOrPrimitiveTypeSizeInBits(Type *Ty) {
2486 assert(!(Ty->isVectorTy() && Ty->getScalarType()->isPointerTy()) &&
2487 "Vector of pointers is not a valid shadow type");
2488 return Ty->isVectorTy() ? cast<FixedVectorType>(Ty)->getNumElements() *
2490 : Ty->getPrimitiveSizeInBits();
2491 }
2492
2493 /// Cast between two shadow types, extending or truncating as
2494 /// necessary.
2495 Value *CreateShadowCast(IRBuilder<> &IRB, Value *V, Type *dstTy,
2496 bool Signed = false) {
2497 Type *srcTy = V->getType();
2498 size_t srcSizeInBits = VectorOrPrimitiveTypeSizeInBits(srcTy);
2499 size_t dstSizeInBits = VectorOrPrimitiveTypeSizeInBits(dstTy);
2500 if (srcSizeInBits > 1 && dstSizeInBits == 1)
2501 return IRB.CreateICmpNE(V, getCleanShadow(V));
2502
2503 if (dstTy->isIntegerTy() && srcTy->isIntegerTy())
2504 return IRB.CreateIntCast(V, dstTy, Signed);
2505 if (dstTy->isVectorTy() && srcTy->isVectorTy() &&
2506 cast<VectorType>(dstTy)->getElementCount() ==
2507 cast<VectorType>(srcTy)->getElementCount())
2508 return IRB.CreateIntCast(V, dstTy, Signed);
2509 Value *V1 = IRB.CreateBitCast(V, Type::getIntNTy(*MS.C, srcSizeInBits));
2510 Value *V2 =
2511 IRB.CreateIntCast(V1, Type::getIntNTy(*MS.C, dstSizeInBits), Signed);
2512 return IRB.CreateBitCast(V2, dstTy);
2513 // TODO: handle struct types.
2514 }
2515
2516 /// Cast an application value to the type of its own shadow.
2517 Value *CreateAppToShadowCast(IRBuilder<> &IRB, Value *V) {
2518 Type *ShadowTy = getShadowTy(V);
2519 if (V->getType() == ShadowTy)
2520 return V;
2521 if (V->getType()->isPtrOrPtrVectorTy())
2522 return IRB.CreatePtrToInt(V, ShadowTy);
2523 else
2524 return IRB.CreateBitCast(V, ShadowTy);
2525 }
2526
2527 /// Propagate shadow for arbitrary operation.
2528 void handleShadowOr(Instruction &I) {
2529 IRBuilder<> IRB(&I);
2530 ShadowAndOriginCombiner SC(this, IRB);
2531 for (Use &Op : I.operands())
2532 SC.Add(Op.get());
2533 SC.Done(&I);
2534 }
2535
2536 void visitFNeg(UnaryOperator &I) { handleShadowOr(I); }
2537
2538 // Handle multiplication by constant.
2539 //
2540 // Handle a special case of multiplication by constant that may have one or
2541 // more zeros in the lower bits. This makes corresponding number of lower bits
2542 // of the result zero as well. We model it by shifting the other operand
2543 // shadow left by the required number of bits. Effectively, we transform
2544 // (X * (A * 2**B)) to ((X << B) * A) and instrument (X << B) as (Sx << B).
2545 // We use multiplication by 2**N instead of shift to cover the case of
2546 // multiplication by 0, which may occur in some elements of a vector operand.
2547 void handleMulByConstant(BinaryOperator &I, Constant *ConstArg,
2548 Value *OtherArg) {
2549 Constant *ShadowMul;
2550 Type *Ty = ConstArg->getType();
2551 if (auto *VTy = dyn_cast<VectorType>(Ty)) {
2552 unsigned NumElements = cast<FixedVectorType>(VTy)->getNumElements();
2553 Type *EltTy = VTy->getElementType();
2555 for (unsigned Idx = 0; Idx < NumElements; ++Idx) {
2556 if (ConstantInt *Elt =
2557 dyn_cast<ConstantInt>(ConstArg->getAggregateElement(Idx))) {
2558 const APInt &V = Elt->getValue();
2559 APInt V2 = APInt(V.getBitWidth(), 1) << V.countr_zero();
2560 Elements.push_back(ConstantInt::get(EltTy, V2));
2561 } else {
2562 Elements.push_back(ConstantInt::get(EltTy, 1));
2563 }
2564 }
2565 ShadowMul = ConstantVector::get(Elements);
2566 } else {
2567 if (ConstantInt *Elt = dyn_cast<ConstantInt>(ConstArg)) {
2568 const APInt &V = Elt->getValue();
2569 APInt V2 = APInt(V.getBitWidth(), 1) << V.countr_zero();
2570 ShadowMul = ConstantInt::get(Ty, V2);
2571 } else {
2572 ShadowMul = ConstantInt::get(Ty, 1);
2573 }
2574 }
2575
2576 IRBuilder<> IRB(&I);
2577 setShadow(&I,
2578 IRB.CreateMul(getShadow(OtherArg), ShadowMul, "msprop_mul_cst"));
2579 setOrigin(&I, getOrigin(OtherArg));
2580 }
2581
2582 void visitMul(BinaryOperator &I) {
2583 Constant *constOp0 = dyn_cast<Constant>(I.getOperand(0));
2584 Constant *constOp1 = dyn_cast<Constant>(I.getOperand(1));
2585 if (constOp0 && !constOp1)
2586 handleMulByConstant(I, constOp0, I.getOperand(1));
2587 else if (constOp1 && !constOp0)
2588 handleMulByConstant(I, constOp1, I.getOperand(0));
2589 else
2590 handleShadowOr(I);
2591 }
2592
2593 void visitFAdd(BinaryOperator &I) { handleShadowOr(I); }
2594 void visitFSub(BinaryOperator &I) { handleShadowOr(I); }
2595 void visitFMul(BinaryOperator &I) { handleShadowOr(I); }
2596 void visitAdd(BinaryOperator &I) { handleShadowOr(I); }
2597 void visitSub(BinaryOperator &I) { handleShadowOr(I); }
2598 void visitXor(BinaryOperator &I) { handleShadowOr(I); }
2599
2600 void handleIntegerDiv(Instruction &I) {
2601 IRBuilder<> IRB(&I);
2602 // Strict on the second argument.
2603 insertShadowCheck(I.getOperand(1), &I);
2604 setShadow(&I, getShadow(&I, 0));
2605 setOrigin(&I, getOrigin(&I, 0));
2606 }
2607
2608 void visitUDiv(BinaryOperator &I) { handleIntegerDiv(I); }
2609 void visitSDiv(BinaryOperator &I) { handleIntegerDiv(I); }
2610 void visitURem(BinaryOperator &I) { handleIntegerDiv(I); }
2611 void visitSRem(BinaryOperator &I) { handleIntegerDiv(I); }
2612
2613 // Floating point division is side-effect free. We can not require that the
2614 // divisor is fully initialized and must propagate shadow. See PR37523.
2615 void visitFDiv(BinaryOperator &I) { handleShadowOr(I); }
2616 void visitFRem(BinaryOperator &I) { handleShadowOr(I); }
2617
2618 /// Instrument == and != comparisons.
2619 ///
2620 /// Sometimes the comparison result is known even if some of the bits of the
2621 /// arguments are not.
2622 void handleEqualityComparison(ICmpInst &I) {
2623 IRBuilder<> IRB(&I);
2624 Value *A = I.getOperand(0);
2625 Value *B = I.getOperand(1);
2626 Value *Sa = getShadow(A);
2627 Value *Sb = getShadow(B);
2628
2629 // Get rid of pointers and vectors of pointers.
2630 // For ints (and vectors of ints), types of A and Sa match,
2631 // and this is a no-op.
2632 A = IRB.CreatePointerCast(A, Sa->getType());
2633 B = IRB.CreatePointerCast(B, Sb->getType());
2634
2635 // A == B <==> (C = A^B) == 0
2636 // A != B <==> (C = A^B) != 0
2637 // Sc = Sa | Sb
2638 Value *C = IRB.CreateXor(A, B);
2639 Value *Sc = IRB.CreateOr(Sa, Sb);
2640 // Now dealing with i = (C == 0) comparison (or C != 0, does not matter now)
2641 // Result is defined if one of the following is true
2642 // * there is a defined 1 bit in C
2643 // * C is fully defined
2644 // Si = !(C & ~Sc) && Sc
2646 Value *MinusOne = Constant::getAllOnesValue(Sc->getType());
2647 Value *LHS = IRB.CreateICmpNE(Sc, Zero);
2648 Value *RHS =
2649 IRB.CreateICmpEQ(IRB.CreateAnd(IRB.CreateXor(Sc, MinusOne), C), Zero);
2650 Value *Si = IRB.CreateAnd(LHS, RHS);
2651 Si->setName("_msprop_icmp");
2652 setShadow(&I, Si);
2653 setOriginForNaryOp(I);
2654 }
2655
2656 /// Build the lowest possible value of V, taking into account V's
2657 /// uninitialized bits.
2658 Value *getLowestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2659 bool isSigned) {
2660 if (isSigned) {
2661 // Split shadow into sign bit and other bits.
2662 Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2663 Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2664 // Maximise the undefined shadow bit, minimize other undefined bits.
2665 return IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaOtherBits)),
2666 SaSignBit);
2667 } else {
2668 // Minimize undefined bits.
2669 return IRB.CreateAnd(A, IRB.CreateNot(Sa));
2670 }
2671 }
2672
2673 /// Build the highest possible value of V, taking into account V's
2674 /// uninitialized bits.
2675 Value *getHighestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2676 bool isSigned) {
2677 if (isSigned) {
2678 // Split shadow into sign bit and other bits.
2679 Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2680 Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2681 // Minimise the undefined shadow bit, maximise other undefined bits.
2682 return IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaSignBit)),
2683 SaOtherBits);
2684 } else {
2685 // Maximize undefined bits.
2686 return IRB.CreateOr(A, Sa);
2687 }
2688 }
2689
2690 /// Instrument relational comparisons.
2691 ///
2692 /// This function does exact shadow propagation for all relational
2693 /// comparisons of integers, pointers and vectors of those.
2694 /// FIXME: output seems suboptimal when one of the operands is a constant
2695 void handleRelationalComparisonExact(ICmpInst &I) {
2696 IRBuilder<> IRB(&I);
2697 Value *A = I.getOperand(0);
2698 Value *B = I.getOperand(1);
2699 Value *Sa = getShadow(A);
2700 Value *Sb = getShadow(B);
2701
2702 // Get rid of pointers and vectors of pointers.
2703 // For ints (and vectors of ints), types of A and Sa match,
2704 // and this is a no-op.
2705 A = IRB.CreatePointerCast(A, Sa->getType());
2706 B = IRB.CreatePointerCast(B, Sb->getType());
2707
2708 // Let [a0, a1] be the interval of possible values of A, taking into account
2709 // its undefined bits. Let [b0, b1] be the interval of possible values of B.
2710 // Then (A cmp B) is defined iff (a0 cmp b1) == (a1 cmp b0).
2711 bool IsSigned = I.isSigned();
2712 Value *S1 = IRB.CreateICmp(I.getPredicate(),
2713 getLowestPossibleValue(IRB, A, Sa, IsSigned),
2714 getHighestPossibleValue(IRB, B, Sb, IsSigned));
2715 Value *S2 = IRB.CreateICmp(I.getPredicate(),
2716 getHighestPossibleValue(IRB, A, Sa, IsSigned),
2717 getLowestPossibleValue(IRB, B, Sb, IsSigned));
2718 Value *Si = IRB.CreateXor(S1, S2);
2719 setShadow(&I, Si);
2720 setOriginForNaryOp(I);
2721 }
2722
2723 /// Instrument signed relational comparisons.
2724 ///
2725 /// Handle sign bit tests: x<0, x>=0, x<=-1, x>-1 by propagating the highest
2726 /// bit of the shadow. Everything else is delegated to handleShadowOr().
2727 void handleSignedRelationalComparison(ICmpInst &I) {
2728 Constant *constOp;
2729 Value *op = nullptr;
2731 if ((constOp = dyn_cast<Constant>(I.getOperand(1)))) {
2732 op = I.getOperand(0);
2733 pre = I.getPredicate();
2734 } else if ((constOp = dyn_cast<Constant>(I.getOperand(0)))) {
2735 op = I.getOperand(1);
2736 pre = I.getSwappedPredicate();
2737 } else {
2738 handleShadowOr(I);
2739 return;
2740 }
2741
2742 if ((constOp->isNullValue() &&
2743 (pre == CmpInst::ICMP_SLT || pre == CmpInst::ICMP_SGE)) ||
2744 (constOp->isAllOnesValue() &&
2745 (pre == CmpInst::ICMP_SGT || pre == CmpInst::ICMP_SLE))) {
2746 IRBuilder<> IRB(&I);
2747 Value *Shadow = IRB.CreateICmpSLT(getShadow(op), getCleanShadow(op),
2748 "_msprop_icmp_s");
2749 setShadow(&I, Shadow);
2750 setOrigin(&I, getOrigin(op));
2751 } else {
2752 handleShadowOr(I);
2753 }
2754 }
2755
2756 void visitICmpInst(ICmpInst &I) {
2757 if (!ClHandleICmp) {
2758 handleShadowOr(I);
2759 return;
2760 }
2761 if (I.isEquality()) {
2762 handleEqualityComparison(I);
2763 return;
2764 }
2765
2766 assert(I.isRelational());
2767 if (ClHandleICmpExact) {
2768 handleRelationalComparisonExact(I);
2769 return;
2770 }
2771 if (I.isSigned()) {
2772 handleSignedRelationalComparison(I);
2773 return;
2774 }
2775
2776 assert(I.isUnsigned());
2777 if ((isa<Constant>(I.getOperand(0)) || isa<Constant>(I.getOperand(1)))) {
2778 handleRelationalComparisonExact(I);
2779 return;
2780 }
2781
2782 handleShadowOr(I);
2783 }
2784
2785 void visitFCmpInst(FCmpInst &I) { handleShadowOr(I); }
2786
2787 void handleShift(BinaryOperator &I) {
2788 IRBuilder<> IRB(&I);
2789 // If any of the S2 bits are poisoned, the whole thing is poisoned.
2790 // Otherwise perform the same shift on S1.
2791 Value *S1 = getShadow(&I, 0);
2792 Value *S2 = getShadow(&I, 1);
2793 Value *S2Conv =
2794 IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)), S2->getType());
2795 Value *V2 = I.getOperand(1);
2796 Value *Shift = IRB.CreateBinOp(I.getOpcode(), S1, V2);
2797 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2798 setOriginForNaryOp(I);
2799 }
2800
2801 void visitShl(BinaryOperator &I) { handleShift(I); }
2802 void visitAShr(BinaryOperator &I) { handleShift(I); }
2803 void visitLShr(BinaryOperator &I) { handleShift(I); }
2804
2805 void handleFunnelShift(IntrinsicInst &I) {
2806 IRBuilder<> IRB(&I);
2807 // If any of the S2 bits are poisoned, the whole thing is poisoned.
2808 // Otherwise perform the same shift on S0 and S1.
2809 Value *S0 = getShadow(&I, 0);
2810 Value *S1 = getShadow(&I, 1);
2811 Value *S2 = getShadow(&I, 2);
2812 Value *S2Conv =
2813 IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)), S2->getType());
2814 Value *V2 = I.getOperand(2);
2816 I.getModule(), I.getIntrinsicID(), S2Conv->getType());
2817 Value *Shift = IRB.CreateCall(Intrin, {S0, S1, V2});
2818 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2819 setOriginForNaryOp(I);
2820 }
2821
2822 /// Instrument llvm.memmove
2823 ///
2824 /// At this point we don't know if llvm.memmove will be inlined or not.
2825 /// If we don't instrument it and it gets inlined,
2826 /// our interceptor will not kick in and we will lose the memmove.
2827 /// If we instrument the call here, but it does not get inlined,
2828 /// we will memove the shadow twice: which is bad in case
2829 /// of overlapping regions. So, we simply lower the intrinsic to a call.
2830 ///
2831 /// Similar situation exists for memcpy and memset.
2832 void visitMemMoveInst(MemMoveInst &I) {
2833 getShadow(I.getArgOperand(1)); // Ensure shadow initialized
2834 IRBuilder<> IRB(&I);
2835 IRB.CreateCall(MS.MemmoveFn,
2836 {I.getArgOperand(0), I.getArgOperand(1),
2837 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2838 I.eraseFromParent();
2839 }
2840
2841 /// Instrument memcpy
2842 ///
2843 /// Similar to memmove: avoid copying shadow twice. This is somewhat
2844 /// unfortunate as it may slowdown small constant memcpys.
2845 /// FIXME: consider doing manual inline for small constant sizes and proper
2846 /// alignment.
2847 ///
2848 /// Note: This also handles memcpy.inline, which promises no calls to external
2849 /// functions as an optimization. However, with instrumentation enabled this
2850 /// is difficult to promise; additionally, we know that the MSan runtime
2851 /// exists and provides __msan_memcpy(). Therefore, we assume that with
2852 /// instrumentation it's safe to turn memcpy.inline into a call to
2853 /// __msan_memcpy(). Should this be wrong, such as when implementing memcpy()
2854 /// itself, instrumentation should be disabled with the no_sanitize attribute.
2855 void visitMemCpyInst(MemCpyInst &I) {
2856 getShadow(I.getArgOperand(1)); // Ensure shadow initialized
2857 IRBuilder<> IRB(&I);
2858 IRB.CreateCall(MS.MemcpyFn,
2859 {I.getArgOperand(0), I.getArgOperand(1),
2860 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2861 I.eraseFromParent();
2862 }
2863
2864 // Same as memcpy.
2865 void visitMemSetInst(MemSetInst &I) {
2866 IRBuilder<> IRB(&I);
2867 IRB.CreateCall(
2868 MS.MemsetFn,
2869 {I.getArgOperand(0),
2870 IRB.CreateIntCast(I.getArgOperand(1), IRB.getInt32Ty(), false),
2871 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2872 I.eraseFromParent();
2873 }
2874
2875 void visitVAStartInst(VAStartInst &I) { VAHelper->visitVAStartInst(I); }
2876
2877 void visitVACopyInst(VACopyInst &I) { VAHelper->visitVACopyInst(I); }
2878
2879 /// Handle vector store-like intrinsics.
2880 ///
2881 /// Instrument intrinsics that look like a simple SIMD store: writes memory,
2882 /// has 1 pointer argument and 1 vector argument, returns void.
2883 bool handleVectorStoreIntrinsic(IntrinsicInst &I) {
2884 IRBuilder<> IRB(&I);
2885 Value *Addr = I.getArgOperand(0);
2886 Value *Shadow = getShadow(&I, 1);
2887 Value *ShadowPtr, *OriginPtr;
2888
2889 // We don't know the pointer alignment (could be unaligned SSE store!).
2890 // Have to assume to worst case.
2891 std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
2892 Addr, IRB, Shadow->getType(), Align(1), /*isStore*/ true);
2893 IRB.CreateAlignedStore(Shadow, ShadowPtr, Align(1));
2894
2896 insertShadowCheck(Addr, &I);
2897
2898 // FIXME: factor out common code from materializeStores
2899 if (MS.TrackOrigins)
2900 IRB.CreateStore(getOrigin(&I, 1), OriginPtr);
2901 return true;
2902 }
2903
2904 /// Handle vector load-like intrinsics.
2905 ///
2906 /// Instrument intrinsics that look like a simple SIMD load: reads memory,
2907 /// has 1 pointer argument, returns a vector.
2908 bool handleVectorLoadIntrinsic(IntrinsicInst &I) {
2909 IRBuilder<> IRB(&I);
2910 Value *Addr = I.getArgOperand(0);
2911
2912 Type *ShadowTy = getShadowTy(&I);
2913 Value *ShadowPtr = nullptr, *OriginPtr = nullptr;
2914 if (PropagateShadow) {
2915 // We don't know the pointer alignment (could be unaligned SSE load!).
2916 // Have to assume to worst case.
2917 const Align Alignment = Align(1);
2918 std::tie(ShadowPtr, OriginPtr) =
2919 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
2920 setShadow(&I,
2921 IRB.CreateAlignedLoad(ShadowTy, ShadowPtr, Alignment, "_msld"));
2922 } else {
2923 setShadow(&I, getCleanShadow(&I));
2924 }
2925
2927 insertShadowCheck(Addr, &I);
2928
2929 if (MS.TrackOrigins) {
2930 if (PropagateShadow)
2931 setOrigin(&I, IRB.CreateLoad(MS.OriginTy, OriginPtr));
2932 else
2933 setOrigin(&I, getCleanOrigin());
2934 }
2935 return true;
2936 }
2937
2938 /// Handle (SIMD arithmetic)-like intrinsics.
2939 ///
2940 /// Instrument intrinsics with any number of arguments of the same type,
2941 /// equal to the return type. The type should be simple (no aggregates or
2942 /// pointers; vectors are fine).
2943 /// Caller guarantees that this intrinsic does not access memory.
2944 bool maybeHandleSimpleNomemIntrinsic(IntrinsicInst &I) {
2945 Type *RetTy = I.getType();
2946 if (!(RetTy->isIntOrIntVectorTy() || RetTy->isFPOrFPVectorTy() ||
2947 RetTy->isX86_MMXTy()))
2948 return false;
2949
2950 unsigned NumArgOperands = I.arg_size();
2951 for (unsigned i = 0; i < NumArgOperands; ++i) {
2952 Type *Ty = I.getArgOperand(i)->getType();
2953 if (Ty != RetTy)
2954 return false;
2955 }
2956
2957 IRBuilder<> IRB(&I);
2958 ShadowAndOriginCombiner SC(this, IRB);
2959 for (unsigned i = 0; i < NumArgOperands; ++i)
2960 SC.Add(I.getArgOperand(i));
2961 SC.Done(&I);
2962
2963 return true;
2964 }
2965
2966 /// Heuristically instrument unknown intrinsics.
2967 ///
2968 /// The main purpose of this code is to do something reasonable with all
2969 /// random intrinsics we might encounter, most importantly - SIMD intrinsics.
2970 /// We recognize several classes of intrinsics by their argument types and
2971 /// ModRefBehaviour and apply special instrumentation when we are reasonably
2972 /// sure that we know what the intrinsic does.
2973 ///
2974 /// We special-case intrinsics where this approach fails. See llvm.bswap
2975 /// handling as an example of that.
2976 bool handleUnknownIntrinsic(IntrinsicInst &I) {
2977 unsigned NumArgOperands = I.arg_size();
2978 if (NumArgOperands == 0)
2979 return false;
2980
2981 if (NumArgOperands == 2 && I.getArgOperand(0)->getType()->isPointerTy() &&
2982 I.getArgOperand(1)->getType()->isVectorTy() &&
2983 I.getType()->isVoidTy() && !I.onlyReadsMemory()) {
2984 // This looks like a vector store.
2985 return handleVectorStoreIntrinsic(I);
2986 }
2987
2988 if (NumArgOperands == 1 && I.getArgOperand(0)->getType()->isPointerTy() &&
2989 I.getType()->isVectorTy() && I.onlyReadsMemory()) {
2990 // This looks like a vector load.
2991 return handleVectorLoadIntrinsic(I);
2992 }
2993
2994 if (I.doesNotAccessMemory())
2995 if (maybeHandleSimpleNomemIntrinsic(I))
2996 return true;
2997
2998 // FIXME: detect and handle SSE maskstore/maskload
2999 return false;
3000 }
3001
3002 void handleInvariantGroup(IntrinsicInst &I) {
3003 setShadow(&I, getShadow(&I, 0));
3004 setOrigin(&I, getOrigin(&I, 0));
3005 }
3006
3007 void handleLifetimeStart(IntrinsicInst &I) {
3008 if (!PoisonStack)
3009 return;
3010 AllocaInst *AI = llvm::findAllocaForValue(I.getArgOperand(1));
3011 if (!AI)
3012 InstrumentLifetimeStart = false;
3013 LifetimeStartList.push_back(std::make_pair(&I, AI));
3014 }
3015
3016 void handleBswap(IntrinsicInst &I) {
3017 IRBuilder<> IRB(&I);
3018 Value *Op = I.getArgOperand(0);
3019 Type *OpType = Op->getType();
3021 F.getParent(), Intrinsic::bswap, ArrayRef(&OpType, 1));
3022 setShadow(&I, IRB.CreateCall(BswapFunc, getShadow(Op)));
3023 setOrigin(&I, getOrigin(Op));
3024 }
3025
3026 void handleCountZeroes(IntrinsicInst &I) {
3027 IRBuilder<> IRB(&I);
3028 Value *Src = I.getArgOperand(0);
3029
3030 // Set the Output shadow based on input Shadow
3031 Value *BoolShadow = IRB.CreateIsNotNull(getShadow(Src), "_mscz_bs");
3032
3033 // If zero poison is requested, mix in with the shadow
3034 Constant *IsZeroPoison = cast<Constant>(I.getOperand(1));
3035 if (!IsZeroPoison->isZeroValue()) {
3036 Value *BoolZeroPoison = IRB.CreateIsNull(Src, "_mscz_bzp");
3037 BoolShadow = IRB.CreateOr(BoolShadow, BoolZeroPoison, "_mscz_bs");
3038 }
3039
3040 Value *OutputShadow =
3041 IRB.CreateSExt(BoolShadow, getShadowTy(Src), "_mscz_os");
3042
3043 setShadow(&I, OutputShadow);
3044 setOriginForNaryOp(I);
3045 }
3046
3047 // Instrument vector convert intrinsic.
3048 //
3049 // This function instruments intrinsics like cvtsi2ss:
3050 // %Out = int_xxx_cvtyyy(%ConvertOp)
3051 // or
3052 // %Out = int_xxx_cvtyyy(%CopyOp, %ConvertOp)
3053 // Intrinsic converts \p NumUsedElements elements of \p ConvertOp to the same
3054 // number \p Out elements, and (if has 2 arguments) copies the rest of the
3055 // elements from \p CopyOp.
3056 // In most cases conversion involves floating-point value which may trigger a
3057 // hardware exception when not fully initialized. For this reason we require
3058 // \p ConvertOp[0:NumUsedElements] to be fully initialized and trap otherwise.
3059 // We copy the shadow of \p CopyOp[NumUsedElements:] to \p
3060 // Out[NumUsedElements:]. This means that intrinsics without \p CopyOp always
3061 // return a fully initialized value.
3062 void handleVectorConvertIntrinsic(IntrinsicInst &I, int NumUsedElements,
3063 bool HasRoundingMode = false) {
3064 IRBuilder<> IRB(&I);
3065 Value *CopyOp, *ConvertOp;
3066
3067 assert((!HasRoundingMode ||
3068 isa<ConstantInt>(I.getArgOperand(I.arg_size() - 1))) &&
3069 "Invalid rounding mode");
3070
3071 switch (I.arg_size() - HasRoundingMode) {
3072 case 2:
3073 CopyOp = I.getArgOperand(0);
3074 ConvertOp = I.getArgOperand(1);
3075 break;
3076 case 1:
3077 ConvertOp = I.getArgOperand(0);
3078 CopyOp = nullptr;
3079 break;
3080 default:
3081 llvm_unreachable("Cvt intrinsic with unsupported number of arguments.");
3082 }
3083
3084 // The first *NumUsedElements* elements of ConvertOp are converted to the
3085 // same number of output elements. The rest of the output is copied from
3086 // CopyOp, or (if not available) filled with zeroes.
3087 // Combine shadow for elements of ConvertOp that are used in this operation,
3088 // and insert a check.
3089 // FIXME: consider propagating shadow of ConvertOp, at least in the case of
3090 // int->any conversion.
3091 Value *ConvertShadow = getShadow(ConvertOp);
3092 Value *AggShadow = nullptr;
3093 if (ConvertOp->getType()->isVectorTy()) {
3094 AggShadow = IRB.CreateExtractElement(
3095 ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), 0));
3096 for (int i = 1; i < NumUsedElements; ++i) {
3097 Value *MoreShadow = IRB.CreateExtractElement(
3098 ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), i));
3099 AggShadow = IRB.CreateOr(AggShadow, MoreShadow);
3100 }
3101 } else {
3102 AggShadow = ConvertShadow;
3103 }
3104 assert(AggShadow->getType()->isIntegerTy());
3105 insertShadowCheck(AggShadow, getOrigin(ConvertOp), &I);
3106
3107 // Build result shadow by zero-filling parts of CopyOp shadow that come from
3108 // ConvertOp.
3109 if (CopyOp) {
3110 assert(CopyOp->getType() == I.getType());
3111 assert(CopyOp->getType()->isVectorTy());
3112 Value *ResultShadow = getShadow(CopyOp);
3113 Type *EltTy = cast<VectorType>(ResultShadow->getType())->getElementType();
3114 for (int i = 0; i < NumUsedElements; ++i) {
3115 ResultShadow = IRB.CreateInsertElement(
3116 ResultShadow, ConstantInt::getNullValue(EltTy),
3117 ConstantInt::get(IRB.getInt32Ty(), i));
3118 }
3119 setShadow(&I, ResultShadow);
3120 setOrigin(&I, getOrigin(CopyOp));
3121 } else {
3122 setShadow(&I, getCleanShadow(&I));
3123 setOrigin(&I, getCleanOrigin());
3124 }
3125 }
3126
3127 // Given a scalar or vector, extract lower 64 bits (or less), and return all
3128 // zeroes if it is zero, and all ones otherwise.
3129 Value *Lower64ShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
3130 if (S->getType()->isVectorTy())
3131 S = CreateShadowCast(IRB, S, IRB.getInt64Ty(), /* Signed */ true);
3132 assert(S->getType()->getPrimitiveSizeInBits() <= 64);
3133 Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
3134 return CreateShadowCast(IRB, S2, T, /* Signed */ true);
3135 }
3136
3137 // Given a vector, extract its first element, and return all
3138 // zeroes if it is zero, and all ones otherwise.
3139 Value *LowerElementShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
3140 Value *S1 = IRB.CreateExtractElement(S, (uint64_t)0);
3141 Value *S2 = IRB.CreateICmpNE(S1, getCleanShadow(S1));
3142 return CreateShadowCast(IRB, S2, T, /* Signed */ true);
3143 }
3144
3145 Value *VariableShadowExtend(IRBuilder<> &IRB, Value *S) {
3146 Type *T = S->getType();
3147 assert(T->isVectorTy());
3148 Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
3149 return IRB.CreateSExt(S2, T);
3150 }
3151
3152 // Instrument vector shift intrinsic.
3153 //
3154 // This function instruments intrinsics like int_x86_avx2_psll_w.
3155 // Intrinsic shifts %In by %ShiftSize bits.
3156 // %ShiftSize may be a vector. In that case the lower 64 bits determine shift
3157 // size, and the rest is ignored. Behavior is defined even if shift size is
3158 // greater than register (or field) width.
3159 void handleVectorShiftIntrinsic(IntrinsicInst &I, bool Variable) {
3160 assert(I.arg_size() == 2);
3161 IRBuilder<> IRB(&I);
3162 // If any of the S2 bits are poisoned, the whole thing is poisoned.
3163 // Otherwise perform the same shift on S1.
3164 Value *S1 = getShadow(&I, 0);
3165 Value *S2 = getShadow(&I, 1);
3166 Value *S2Conv = Variable ? VariableShadowExtend(IRB, S2)
3167 : Lower64ShadowExtend(IRB, S2, getShadowTy(&I));
3168 Value *V1 = I.getOperand(0);
3169 Value *V2 = I.getOperand(1);
3170 Value *Shift = IRB.CreateCall(I.getFunctionType(), I.getCalledOperand(),
3171 {IRB.CreateBitCast(S1, V1->getType()), V2});
3172 Shift = IRB.CreateBitCast(Shift, getShadowTy(&I));
3173 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
3174 setOriginForNaryOp(I);
3175 }
3176
3177 // Get an X86_MMX-sized vector type.
3178 Type *getMMXVectorTy(unsigned EltSizeInBits) {
3179 const unsigned X86_MMXSizeInBits = 64;
3180 assert(EltSizeInBits != 0 && (X86_MMXSizeInBits % EltSizeInBits) == 0 &&
3181 "Illegal MMX vector element size");
3182 return FixedVectorType::get(IntegerType::get(*MS.C, EltSizeInBits),
3183 X86_MMXSizeInBits / EltSizeInBits);
3184 }
3185
3186 // Returns a signed counterpart for an (un)signed-saturate-and-pack
3187 // intrinsic.
3188 Intrinsic::ID getSignedPackIntrinsic(Intrinsic::ID id) {
3189 switch (id) {
3190 case Intrinsic::x86_sse2_packsswb_128:
3191 case Intrinsic::x86_sse2_packuswb_128:
3192 return Intrinsic::x86_sse2_packsswb_128;
3193
3194 case Intrinsic::x86_sse2_packssdw_128:
3195 case Intrinsic::x86_sse41_packusdw:
3196 return Intrinsic::x86_sse2_packssdw_128;
3197
3198 case Intrinsic::x86_avx2_packsswb:
3199 case Intrinsic::x86_avx2_packuswb:
3200 return Intrinsic::x86_avx2_packsswb;
3201
3202 case Intrinsic::x86_avx2_packssdw:
3203 case Intrinsic::x86_avx2_packusdw:
3204 return Intrinsic::x86_avx2_packssdw;
3205
3206 case Intrinsic::x86_mmx_packsswb:
3207 case Intrinsic::x86_mmx_packuswb:
3208 return Intrinsic::x86_mmx_packsswb;
3209
3210 case Intrinsic::x86_mmx_packssdw:
3211 return Intrinsic::x86_mmx_packssdw;
3212 default:
3213 llvm_unreachable("unexpected intrinsic id");
3214 }
3215 }
3216
3217 // Instrument vector pack intrinsic.
3218 //
3219 // This function instruments intrinsics like x86_mmx_packsswb, that
3220 // packs elements of 2 input vectors into half as many bits with saturation.
3221 // Shadow is propagated with the signed variant of the same intrinsic applied
3222 // to sext(Sa != zeroinitializer), sext(Sb != zeroinitializer).
3223 // EltSizeInBits is used only for x86mmx arguments.
3224 void handleVectorPackIntrinsic(IntrinsicInst &I, unsigned EltSizeInBits = 0) {
3225 assert(I.arg_size() == 2);
3226 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
3227 IRBuilder<> IRB(&I);
3228 Value *S1 = getShadow(&I, 0);
3229 Value *S2 = getShadow(&I, 1);
3230 assert(isX86_MMX || S1->getType()->isVectorTy());
3231
3232 // SExt and ICmpNE below must apply to individual elements of input vectors.
3233 // In case of x86mmx arguments, cast them to appropriate vector types and
3234 // back.
3235 Type *T = isX86_MMX ? getMMXVectorTy(EltSizeInBits) : S1->getType();
3236 if (isX86_MMX) {
3237 S1 = IRB.CreateBitCast(S1, T);
3238 S2 = IRB.CreateBitCast(S2, T);
3239 }
3240 Value *S1_ext =
3242 Value *S2_ext =
3244 if (isX86_MMX) {
3245 Type *X86_MMXTy = Type::getX86_MMXTy(*MS.C);
3246 S1_ext = IRB.CreateBitCast(S1_ext, X86_MMXTy);
3247 S2_ext = IRB.CreateBitCast(S2_ext, X86_MMXTy);
3248 }
3249
3251 F.getParent(), getSignedPackIntrinsic(I.getIntrinsicID()));
3252
3253 Value *S =
3254 IRB.CreateCall(ShadowFn, {S1_ext, S2_ext}, "_msprop_vector_pack");
3255 if (isX86_MMX)
3256 S = IRB.CreateBitCast(S, getShadowTy(&I));
3257 setShadow(&I, S);
3258 setOriginForNaryOp(I);
3259 }
3260
3261 // Instrument sum-of-absolute-differences intrinsic.
3262 void handleVectorSadIntrinsic(IntrinsicInst &I) {
3263 const unsigned SignificantBitsPerResultElement = 16;
3264 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
3265 Type *ResTy = isX86_MMX ? IntegerType::get(*MS.C, 64) : I.getType();
3266 unsigned ZeroBitsPerResultElement =
3267 ResTy->getScalarSizeInBits() - SignificantBitsPerResultElement;
3268
3269 IRBuilder<> IRB(&I);
3270 auto *Shadow0 = getShadow(&I, 0);
3271 auto *Shadow1 = getShadow(&I, 1);
3272 Value *S = IRB.CreateOr(Shadow0, Shadow1);
3273 S = IRB.CreateBitCast(S, ResTy);
3274 S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
3275 ResTy);
3276 S = IRB.CreateLShr(S, ZeroBitsPerResultElement);
3277 S = IRB.CreateBitCast(S, getShadowTy(&I));
3278 setShadow(&I, S);
3279 setOriginForNaryOp(I);
3280 }
3281
3282 // Instrument multiply-add intrinsic.
3283 void handleVectorPmaddIntrinsic(IntrinsicInst &I,
3284 unsigned EltSizeInBits = 0) {
3285 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
3286 Type *ResTy = isX86_MMX ? getMMXVectorTy(EltSizeInBits * 2) : I.getType();
3287 IRBuilder<> IRB(&I);
3288 auto *Shadow0 = getShadow(&I, 0);
3289 auto *Shadow1 = getShadow(&I, 1);
3290 Value *S = IRB.CreateOr(Shadow0, Shadow1);
3291 S = IRB.CreateBitCast(S, ResTy);
3292 S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
3293 ResTy);
3294 S = IRB.CreateBitCast(S, getShadowTy(&I));
3295 setShadow(&I, S);
3296 setOriginForNaryOp(I);
3297 }
3298
3299 // Instrument compare-packed intrinsic.
3300 // Basically, an or followed by sext(icmp ne 0) to end up with all-zeros or
3301 // all-ones shadow.
3302 void handleVectorComparePackedIntrinsic(IntrinsicInst &I) {
3303 IRBuilder<> IRB(&I);
3304 Type *ResTy = getShadowTy(&I);
3305 auto *Shadow0 = getShadow(&I, 0);
3306 auto *Shadow1 = getShadow(&I, 1);
3307 Value *S0 = IRB.CreateOr(Shadow0, Shadow1);
3308 Value *S = IRB.CreateSExt(
3309 IRB.CreateICmpNE(S0, Constant::getNullValue(ResTy)), ResTy);
3310 setShadow(&I, S);
3311 setOriginForNaryOp(I);
3312 }
3313
3314 // Instrument compare-scalar intrinsic.
3315 // This handles both cmp* intrinsics which return the result in the first
3316 // element of a vector, and comi* which return the result as i32.
3317 void handleVectorCompareScalarIntrinsic(IntrinsicInst &I) {
3318 IRBuilder<> IRB(&I);
3319 auto *Shadow0 = getShadow(&I, 0);
3320 auto *Shadow1 = getShadow(&I, 1);
3321 Value *S0 = IRB.CreateOr(Shadow0, Shadow1);
3322 Value *S = LowerElementShadowExtend(IRB, S0, getShadowTy(&I));
3323 setShadow(&I, S);
3324 setOriginForNaryOp(I);
3325 }
3326
3327 // Instrument generic vector reduction intrinsics
3328 // by ORing together all their fields.
3329 void handleVectorReduceIntrinsic(IntrinsicInst &I) {
3330 IRBuilder<> IRB(&I);
3331 Value *S = IRB.CreateOrReduce(getShadow(&I, 0));
3332 setShadow(&I, S);
3333 setOrigin(&I, getOrigin(&I, 0));
3334 }
3335
3336 // Instrument vector.reduce.or intrinsic.
3337 // Valid (non-poisoned) set bits in the operand pull low the
3338 // corresponding shadow bits.
3339 void handleVectorReduceOrIntrinsic(IntrinsicInst &I) {
3340 IRBuilder<> IRB(&I);
3341 Value *OperandShadow = getShadow(&I, 0);
3342 Value *OperandUnsetBits = IRB.CreateNot(I.getOperand(0));
3343 Value *OperandUnsetOrPoison = IRB.CreateOr(OperandUnsetBits, OperandShadow);
3344 // Bit N is clean if any field's bit N is 1 and unpoison
3345 Value *OutShadowMask = IRB.CreateAndReduce(OperandUnsetOrPoison);
3346 // Otherwise, it is clean if every field's bit N is unpoison
3347 Value *OrShadow = IRB.CreateOrReduce(OperandShadow);
3348 Value *S = IRB.CreateAnd(OutShadowMask, OrShadow);
3349
3350 setShadow(&I, S);
3351 setOrigin(&I, getOrigin(&I, 0));
3352 }
3353
3354 // Instrument vector.reduce.and intrinsic.
3355 // Valid (non-poisoned) unset bits in the operand pull down the
3356 // corresponding shadow bits.
3357 void handleVectorReduceAndIntrinsic(IntrinsicInst &I) {
3358 IRBuilder<> IRB(&I);
3359 Value *OperandShadow = getShadow(&I, 0);
3360 Value *OperandSetOrPoison = IRB.CreateOr(I.getOperand(0), OperandShadow);
3361 // Bit N is clean if any field's bit N is 0 and unpoison
3362 Value *OutShadowMask = IRB.CreateAndReduce(OperandSetOrPoison);
3363 // Otherwise, it is clean if every field's bit N is unpoison
3364 Value *OrShadow = IRB.CreateOrReduce(OperandShadow);
3365 Value *S = IRB.CreateAnd(OutShadowMask, OrShadow);
3366
3367 setShadow(&I, S);
3368 setOrigin(&I, getOrigin(&I, 0));
3369 }
3370
3371 void handleStmxcsr(IntrinsicInst &I) {
3372 IRBuilder<> IRB(&I);
3373 Value *Addr = I.getArgOperand(0);
3374 Type *Ty = IRB.getInt32Ty();
3375 Value *ShadowPtr =
3376 getShadowOriginPtr(Addr, IRB, Ty, Align(1), /*isStore*/ true).first;
3377
3378 IRB.CreateStore(getCleanShadow(Ty), ShadowPtr);
3379
3381 insertShadowCheck(Addr, &I);
3382 }
3383
3384 void handleLdmxcsr(IntrinsicInst &I) {
3385 if (!InsertChecks)
3386 return;
3387
3388 IRBuilder<> IRB(&I);
3389 Value *Addr = I.getArgOperand(0);
3390 Type *Ty = IRB.getInt32Ty();
3391 const Align Alignment = Align(1);
3392 Value *ShadowPtr, *OriginPtr;
3393 std::tie(ShadowPtr, OriginPtr) =
3394 getShadowOriginPtr(Addr, IRB, Ty, Alignment, /*isStore*/ false);
3395
3397 insertShadowCheck(Addr, &I);
3398
3399 Value *Shadow = IRB.CreateAlignedLoad(Ty, ShadowPtr, Alignment, "_ldmxcsr");
3400 Value *Origin = MS.TrackOrigins ? IRB.CreateLoad(MS.OriginTy, OriginPtr)
3401 : getCleanOrigin();
3402 insertShadowCheck(Shadow, Origin, &I);
3403 }
3404
3405 void handleMaskedExpandLoad(IntrinsicInst &I) {
3406 IRBuilder<> IRB(&I);
3407 Value *Ptr = I.getArgOperand(0);
3408 Value *Mask = I.getArgOperand(1);
3409 Value *PassThru = I.getArgOperand(2);
3410
3412 insertShadowCheck(Ptr, &I);
3413 insertShadowCheck(Mask, &I);
3414 }
3415
3416 if (!PropagateShadow) {
3417 setShadow(&I, getCleanShadow(&I));
3418 setOrigin(&I, getCleanOrigin());
3419 return;
3420 }
3421
3422 Type *ShadowTy = getShadowTy(&I);
3423 Type *ElementShadowTy = cast<VectorType>(ShadowTy)->getElementType();
3424 auto [ShadowPtr, OriginPtr] =
3425 getShadowOriginPtr(Ptr, IRB, ElementShadowTy, {}, /*isStore*/ false);
3426
3427 Value *Shadow = IRB.CreateMaskedExpandLoad(
3428 ShadowTy, ShadowPtr, Mask, getShadow(PassThru), "_msmaskedexpload");
3429
3430 setShadow(&I, Shadow);
3431
3432 // TODO: Store origins.
3433 setOrigin(&I, getCleanOrigin());
3434 }
3435
3436 void handleMaskedCompressStore(IntrinsicInst &I) {
3437 IRBuilder<> IRB(&I);
3438 Value *Values = I.getArgOperand(0);
3439 Value *Ptr = I.getArgOperand(1);
3440 Value *Mask = I.getArgOperand(2);
3441
3443 insertShadowCheck(Ptr, &I);
3444 insertShadowCheck(Mask, &I);
3445 }
3446
3447 Value *Shadow = getShadow(Values);
3448 Type *ElementShadowTy =
3449 getShadowTy(cast<VectorType>(Values->getType())->getElementType());
3450 auto [ShadowPtr, OriginPtrs] =
3451 getShadowOriginPtr(Ptr, IRB, ElementShadowTy, {}, /*isStore*/ true);
3452
3453 IRB.CreateMaskedCompressStore(Shadow, ShadowPtr, Mask);
3454
3455 // TODO: Store origins.
3456 }
3457
3458 void handleMaskedGather(IntrinsicInst &I) {
3459 IRBuilder<> IRB(&I);
3460 Value *Ptrs = I.getArgOperand(0);
3461 const Align Alignment(
3462 cast<ConstantInt>(I.getArgOperand(1))->getZExtValue());
3463 Value *Mask = I.getArgOperand(2);
3464 Value *PassThru = I.getArgOperand(3);
3465
3466 Type *PtrsShadowTy = getShadowTy(Ptrs);
3468 insertShadowCheck(Mask, &I);
3469 Value *MaskedPtrShadow = IRB.CreateSelect(
3470 Mask, getShadow(Ptrs), Constant::getNullValue((PtrsShadowTy)),
3471 "_msmaskedptrs");
3472 insertShadowCheck(MaskedPtrShadow, getOrigin(Ptrs), &I);
3473 }
3474
3475 if (!PropagateShadow) {
3476 setShadow(&I, getCleanShadow(&I));
3477 setOrigin(&I, getCleanOrigin());
3478 return;
3479 }
3480
3481 Type *ShadowTy = getShadowTy(&I);
3482 Type *ElementShadowTy = cast<VectorType>(ShadowTy)->getElementType();
3483 auto [ShadowPtrs, OriginPtrs] = getShadowOriginPtr(
3484 Ptrs, IRB, ElementShadowTy, Alignment, /*isStore*/ false);
3485
3486 Value *Shadow =
3487 IRB.CreateMaskedGather(ShadowTy, ShadowPtrs, Alignment, Mask,
3488 getShadow(PassThru), "_msmaskedgather");
3489
3490 setShadow(&I, Shadow);
3491
3492 // TODO: Store origins.
3493 setOrigin(&I, getCleanOrigin());
3494 }
3495
3496 void handleMaskedScatter(IntrinsicInst &I) {
3497 IRBuilder<> IRB(&I);
3498 Value *Values = I.getArgOperand(0);
3499 Value *Ptrs = I.getArgOperand(1);
3500 const Align Alignment(
3501 cast<ConstantInt>(I.getArgOperand(2))->getZExtValue());
3502 Value *Mask = I.getArgOperand(3);
3503
3504 Type *PtrsShadowTy = getShadowTy(Ptrs);
3506 insertShadowCheck(Mask, &I);
3507 Value *MaskedPtrShadow = IRB.CreateSelect(
3508 Mask, getShadow(Ptrs), Constant::getNullValue((PtrsShadowTy)),
3509 "_msmaskedptrs");
3510 insertShadowCheck(MaskedPtrShadow, getOrigin(Ptrs), &I);
3511 }
3512
3513 Value *Shadow = getShadow(Values);
3514 Type *ElementShadowTy =
3515 getShadowTy(cast<VectorType>(Values->getType())->getElementType());
3516 auto [ShadowPtrs, OriginPtrs] = getShadowOriginPtr(
3517 Ptrs, IRB, ElementShadowTy, Alignment, /*isStore*/ true);
3518
3519 IRB.CreateMaskedScatter(Shadow, ShadowPtrs, Alignment, Mask);
3520
3521 // TODO: Store origin.
3522 }
3523
3524 void handleMaskedStore(IntrinsicInst &I) {
3525 IRBuilder<> IRB(&I);
3526 Value *V = I.getArgOperand(0);
3527 Value *Ptr = I.getArgOperand(1);
3528 const Align Alignment(
3529 cast<ConstantInt>(I.getArgOperand(2))->getZExtValue());
3530 Value *Mask = I.getArgOperand(3);
3531 Value *Shadow = getShadow(V);
3532
3534 insertShadowCheck(Ptr, &I);
3535 insertShadowCheck(Mask, &I);
3536 }
3537
3538 Value *ShadowPtr;
3539 Value *OriginPtr;
3540 std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
3541 Ptr, IRB, Shadow->getType(), Alignment, /*isStore*/ true);
3542
3543 IRB.CreateMaskedStore(Shadow, ShadowPtr, Alignment, Mask);
3544
3545 if (!MS.TrackOrigins)
3546 return;
3547
3548 auto &DL = F.getParent()->getDataLayout();
3549 paintOrigin(IRB, getOrigin(V), OriginPtr,
3550 DL.getTypeStoreSize(Shadow->getType()),
3551 std::max(Alignment, kMinOriginAlignment));
3552 }
3553
3554 void handleMaskedLoad(IntrinsicInst &I) {
3555 IRBuilder<> IRB(&I);
3556 Value *Ptr = I.getArgOperand(0);
3557 const Align Alignment(
3558 cast<ConstantInt>(I.getArgOperand(1))->getZExtValue());
3559 Value *Mask = I.getArgOperand(2);
3560 Value *PassThru = I.getArgOperand(3);
3561
3563 insertShadowCheck(Ptr, &I);
3564 insertShadowCheck(Mask, &I);
3565 }
3566
3567 if (!PropagateShadow) {
3568 setShadow(&I, getCleanShadow(&I));
3569 setOrigin(&I, getCleanOrigin());
3570 return;
3571 }
3572
3573 Type *ShadowTy = getShadowTy(&I);
3574 Value *ShadowPtr, *OriginPtr;
3575 std::tie(ShadowPtr, OriginPtr) =
3576 getShadowOriginPtr(Ptr, IRB, ShadowTy, Alignment, /*isStore*/ false);
3577 setShadow(&I, IRB.CreateMaskedLoad(ShadowTy, ShadowPtr, Alignment, Mask,
3578 getShadow(PassThru), "_msmaskedld"));
3579
3580 if (!MS.TrackOrigins)
3581 return;
3582
3583 // Choose between PassThru's and the loaded value's origins.
3584 Value *MaskedPassThruShadow = IRB.CreateAnd(
3585 getShadow(PassThru), IRB.CreateSExt(IRB.CreateNeg(Mask), ShadowTy));
3586
3587 Value *NotNull = convertToBool(MaskedPassThruShadow, IRB, "_mscmp");
3588
3589 Value *PtrOrigin = IRB.CreateLoad(MS.OriginTy, OriginPtr);
3590 Value *Origin = IRB.CreateSelect(NotNull, getOrigin(PassThru), PtrOrigin);
3591
3592 setOrigin(&I, Origin);
3593 }
3594
3595 // Instrument BMI / BMI2 intrinsics.
3596 // All of these intrinsics are Z = I(X, Y)
3597 // where the types of all operands and the result match, and are either i32 or
3598 // i64. The following instrumentation happens to work for all of them:
3599 // Sz = I(Sx, Y) | (sext (Sy != 0))
3600 void handleBmiIntrinsic(IntrinsicInst &I) {
3601 IRBuilder<> IRB(&I);
3602 Type *ShadowTy = getShadowTy(&I);
3603
3604 // If any bit of the mask operand is poisoned, then the whole thing is.
3605 Value *SMask = getShadow(&I, 1);
3606 SMask = IRB.CreateSExt(IRB.CreateICmpNE(SMask, getCleanShadow(ShadowTy)),
3607 ShadowTy);
3608 // Apply the same intrinsic to the shadow of the first operand.
3609 Value *S = IRB.CreateCall(I.getCalledFunction(),
3610 {getShadow(&I, 0), I.getOperand(1)});
3611 S = IRB.CreateOr(SMask, S);
3612 setShadow(&I, S);
3613 setOriginForNaryOp(I);
3614 }
3615
3616 SmallVector<int, 8> getPclmulMask(unsigned Width, bool OddElements) {
3618 for (unsigned X = OddElements ? 1 : 0; X < Width; X += 2) {
3619 Mask.append(2, X);
3620 }
3621 return Mask;
3622 }
3623
3624 // Instrument pclmul intrinsics.
3625 // These intrinsics operate either on odd or on even elements of the input
3626 // vectors, depending on the constant in the 3rd argument, ignoring the rest.
3627 // Replace the unused elements with copies of the used ones, ex:
3628 // (0, 1, 2, 3) -> (0, 0, 2, 2) (even case)
3629 // or
3630 // (0, 1, 2, 3) -> (1, 1, 3, 3) (odd case)
3631 // and then apply the usual shadow combining logic.
3632 void handlePclmulIntrinsic(IntrinsicInst &I) {
3633 IRBuilder<> IRB(&I);
3634 unsigned Width =
3635 cast<FixedVectorType>(I.getArgOperand(0)->getType())->getNumElements();
3636 assert(isa<ConstantInt>(I.getArgOperand(2)) &&
3637 "pclmul 3rd operand must be a constant");
3638 unsigned Imm = cast<ConstantInt>(I.getArgOperand(2))->getZExtValue();
3639 Value *Shuf0 = IRB.CreateShuffleVector(getShadow(&I, 0),
3640 getPclmulMask(Width, Imm & 0x01));
3641 Value *Shuf1 = IRB.CreateShuffleVector(getShadow(&I, 1),
3642 getPclmulMask(Width, Imm & 0x10));
3643 ShadowAndOriginCombiner SOC(this, IRB);
3644 SOC.Add(Shuf0, getOrigin(&I, 0));
3645 SOC.Add(Shuf1, getOrigin(&I, 1));
3646 SOC.Done(&I);
3647 }
3648
3649 // Instrument _mm_*_sd|ss intrinsics
3650 void handleUnarySdSsIntrinsic(IntrinsicInst &I) {
3651 IRBuilder<> IRB(&I);
3652 unsigned Width =
3653 cast<FixedVectorType>(I.getArgOperand(0)->getType())->getNumElements();
3654 Value *First = getShadow(&I, 0);
3655 Value *Second = getShadow(&I, 1);
3656 // First element of second operand, remaining elements of first operand
3658 Mask.push_back(Width);
3659 for (unsigned i = 1; i < Width; i++)
3660 Mask.push_back(i);
3661 Value *Shadow = IRB.CreateShuffleVector(First, Second, Mask);
3662
3663 setShadow(&I, Shadow);
3664 setOriginForNaryOp(I);
3665 }
3666
3667 void handleVtestIntrinsic(IntrinsicInst &I) {
3668 IRBuilder<> IRB(&I);
3669 Value *Shadow0 = getShadow(&I, 0);
3670 Value *Shadow1 = getShadow(&I, 1);
3671 Value *Or = IRB.CreateOr(Shadow0, Shadow1);
3672 Value *NZ = IRB.CreateICmpNE(Or, Constant::getNullValue(Or->getType()));
3673 Value *Scalar = convertShadowToScalar(NZ, IRB);
3674 Value *Shadow = IRB.CreateZExt(Scalar, getShadowTy(&I));
3675
3676 setShadow(&I, Shadow);
3677 setOriginForNaryOp(I);
3678 }
3679
3680 void handleBinarySdSsIntrinsic(IntrinsicInst &I) {
3681 IRBuilder<> IRB(&I);
3682 unsigned Width =
3683 cast<FixedVectorType>(I.getArgOperand(0)->getType())->getNumElements();
3684 Value *First = getShadow(&I, 0);
3685 Value *Second = getShadow(&I, 1);
3686 Value *OrShadow = IRB.CreateOr(First, Second);
3687 // First element of both OR'd together, remaining elements of first operand
3689 Mask.push_back(Width);
3690 for (unsigned i = 1; i < Width; i++)
3691 Mask.push_back(i);
3692 Value *Shadow = IRB.CreateShuffleVector(First, OrShadow, Mask);
3693
3694 setShadow(&I, Shadow);
3695 setOriginForNaryOp(I);
3696 }
3697
3698 // Instrument abs intrinsic.
3699 // handleUnknownIntrinsic can't handle it because of the last
3700 // is_int_min_poison argument which does not match the result type.
3701 void handleAbsIntrinsic(IntrinsicInst &I) {
3702 assert(I.getType()->isIntOrIntVectorTy());
3703 assert(I.getArgOperand(0)->getType() == I.getType());
3704
3705 // FIXME: Handle is_int_min_poison.
3706 IRBuilder<> IRB(&I);
3707 setShadow(&I, getShadow(&I, 0));
3708 setOrigin(&I, getOrigin(&I, 0));
3709 }
3710
3711 void handleIsFpClass(IntrinsicInst &I) {
3712 IRBuilder<> IRB(&I);
3713 Value *Shadow = getShadow(&I, 0);
3714 setShadow(&I, IRB.CreateICmpNE(Shadow, getCleanShadow(Shadow)));
3715 setOrigin(&I, getOrigin(&I, 0));
3716 }
3717
3718 void handleArithmeticWithOverflow(IntrinsicInst &I) {
3719 IRBuilder<> IRB(&I);
3720 Value *Shadow0 = getShadow(&I, 0);
3721 Value *Shadow1 = getShadow(&I, 1);
3722 Value *ShadowElt0 = IRB.CreateOr(Shadow0, Shadow1);
3723 Value *ShadowElt1 =
3724 IRB.CreateICmpNE(ShadowElt0, getCleanShadow(ShadowElt0));
3725
3726 Value *Shadow = PoisonValue::get(getShadowTy(&I));
3727 Shadow = IRB.CreateInsertValue(Shadow, ShadowElt0, 0);
3728 Shadow = IRB.CreateInsertValue(Shadow, ShadowElt1, 1);
3729
3730 setShadow(&I, Shadow);
3731 setOriginForNaryOp(I);
3732 }
3733
3734 void visitIntrinsicInst(IntrinsicInst &I) {
3735 switch (I.getIntrinsicID()) {
3736 case Intrinsic::uadd_with_overflow:
3737 case Intrinsic::sadd_with_overflow:
3738 case Intrinsic::usub_with_overflow:
3739 case Intrinsic::ssub_with_overflow:
3740 case Intrinsic::umul_with_overflow:
3741 case Intrinsic::smul_with_overflow:
3742 handleArithmeticWithOverflow(I);
3743 break;
3744 case Intrinsic::abs:
3745 handleAbsIntrinsic(I);
3746 break;
3747 case Intrinsic::is_fpclass:
3748 handleIsFpClass(I);
3749 break;
3750 case Intrinsic::lifetime_start:
3751 handleLifetimeStart(I);
3752 break;
3753 case Intrinsic::launder_invariant_group:
3754 case Intrinsic::strip_invariant_group:
3755 handleInvariantGroup(I);
3756 break;
3757 case Intrinsic::bswap:
3758 handleBswap(I);
3759 break;
3760 case Intrinsic::ctlz:
3761 case Intrinsic::cttz:
3762 handleCountZeroes(I);
3763 break;
3764 case Intrinsic::masked_compressstore:
3765 handleMaskedCompressStore(I);
3766 break;
3767 case Intrinsic::masked_expandload:
3768 handleMaskedExpandLoad(I);
3769 break;
3770 case Intrinsic::masked_gather:
3771 handleMaskedGather(I);
3772 break;
3773 case Intrinsic::masked_scatter:
3774 handleMaskedScatter(I);
3775 break;
3776 case Intrinsic::masked_store:
3777 handleMaskedStore(I);
3778 break;
3779 case Intrinsic::masked_load:
3780 handleMaskedLoad(I);
3781 break;
3782 case Intrinsic::vector_reduce_and:
3783 handleVectorReduceAndIntrinsic(I);
3784 break;
3785 case Intrinsic::vector_reduce_or:
3786 handleVectorReduceOrIntrinsic(I);
3787 break;
3788 case Intrinsic::vector_reduce_add:
3789 case Intrinsic::vector_reduce_xor:
3790 case Intrinsic::vector_reduce_mul:
3791 handleVectorReduceIntrinsic(I);
3792 break;
3793 case Intrinsic::x86_sse_stmxcsr:
3794 handleStmxcsr(I);
3795 break;
3796 case Intrinsic::x86_sse_ldmxcsr:
3797 handleLdmxcsr(I);
3798 break;
3799 case Intrinsic::x86_avx512_vcvtsd2usi64:
3800 case Intrinsic::x86_avx512_vcvtsd2usi32:
3801 case Intrinsic::x86_avx512_vcvtss2usi64:
3802 case Intrinsic::x86_avx512_vcvtss2usi32:
3803 case Intrinsic::x86_avx512_cvttss2usi64:
3804 case Intrinsic::x86_avx512_cvttss2usi:
3805 case Intrinsic::x86_avx512_cvttsd2usi64:
3806 case Intrinsic::x86_avx512_cvttsd2usi:
3807 case Intrinsic::x86_avx512_cvtusi2ss:
3808 case Intrinsic::x86_avx512_cvtusi642sd:
3809 case Intrinsic::x86_avx512_cvtusi642ss:
3810 handleVectorConvertIntrinsic(I, 1, true);
3811 break;
3812 case Intrinsic::x86_sse2_cvtsd2si64:
3813 case Intrinsic::x86_sse2_cvtsd2si:
3814 case Intrinsic::x86_sse2_cvtsd2ss:
3815 case Intrinsic::x86_sse2_cvttsd2si64:
3816 case Intrinsic::x86_sse2_cvttsd2si:
3817 case Intrinsic::x86_sse_cvtss2si64:
3818 case Intrinsic::x86_sse_cvtss2si:
3819 case Intrinsic::x86_sse_cvttss2si64:
3820 case Intrinsic::x86_sse_cvttss2si:
3821 handleVectorConvertIntrinsic(I, 1);
3822 break;
3823 case Intrinsic::x86_sse_cvtps2pi:
3824 case Intrinsic::x86_sse_cvttps2pi:
3825 handleVectorConvertIntrinsic(I, 2);
3826 break;
3827
3828 case Intrinsic::x86_avx512_psll_w_512:
3829 case Intrinsic::x86_avx512_psll_d_512:
3830 case Intrinsic::x86_avx512_psll_q_512:
3831 case Intrinsic::x86_avx512_pslli_w_512:
3832 case Intrinsic::x86_avx512_pslli_d_512:
3833 case Intrinsic::x86_avx512_pslli_q_512:
3834 case Intrinsic::x86_avx512_psrl_w_512:
3835 case Intrinsic::x86_avx512_psrl_d_512:
3836 case Intrinsic::x86_avx512_psrl_q_512:
3837 case Intrinsic::x86_avx512_psra_w_512:
3838 case Intrinsic::x86_avx512_psra_d_512:
3839 case Intrinsic::x86_avx512_psra_q_512:
3840 case Intrinsic::x86_avx512_psrli_w_512:
3841 case Intrinsic::x86_avx512_psrli_d_512:
3842 case Intrinsic::x86_avx512_psrli_q_512:
3843 case Intrinsic::x86_avx512_psrai_w_512:
3844 case Intrinsic::x86_avx512_psrai_d_512:
3845 case Intrinsic::x86_avx512_psrai_q_512:
3846 case Intrinsic::x86_avx512_psra_q_256:
3847 case Intrinsic::x86_avx512_psra_q_128:
3848 case Intrinsic::x86_avx512_psrai_q_256:
3849 case Intrinsic::x86_avx512_psrai_q_128:
3850 case Intrinsic::x86_avx2_psll_w:
3851 case Intrinsic::x86_avx2_psll_d:
3852 case Intrinsic::x86_avx2_psll_q:
3853 case Intrinsic::x86_avx2_pslli_w:
3854 case Intrinsic::x86_avx2_pslli_d:
3855 case Intrinsic::x86_avx2_pslli_q:
3856 case Intrinsic::x86_avx2_psrl_w:
3857 case Intrinsic::x86_avx2_psrl_d:
3858 case Intrinsic::x86_avx2_psrl_q:
3859 case Intrinsic::x86_avx2_psra_w:
3860 case Intrinsic::x86_avx2_psra_d:
3861 case Intrinsic::x86_avx2_psrli_w:
3862 case Intrinsic::x86_avx2_psrli_d:
3863 case Intrinsic::x86_avx2_psrli_q:
3864 case Intrinsic::x86_avx2_psrai_w:
3865 case Intrinsic::x86_avx2_psrai_d:
3866 case Intrinsic::x86_sse2_psll_w:
3867 case Intrinsic::x86_sse2_psll_d:
3868 case Intrinsic::x86_sse2_psll_q:
3869 case Intrinsic::x86_sse2_pslli_w:
3870 case Intrinsic::x86_sse2_pslli_d:
3871 case Intrinsic::x86_sse2_pslli_q:
3872 case Intrinsic::x86_sse2_psrl_w:
3873 case Intrinsic::x86_sse2_psrl_d:
3874 case Intrinsic::x86_sse2_psrl_q:
3875 case Intrinsic::x86_sse2_psra_w:
3876 case Intrinsic::x86_sse2_psra_d:
3877 case Intrinsic::x86_sse2_psrli_w:
3878 case Intrinsic::x86_sse2_psrli_d:
3879 case Intrinsic::x86_sse2_psrli_q:
3880 case Intrinsic::x86_sse2_psrai_w:
3881 case Intrinsic::x86_sse2_psrai_d:
3882 case Intrinsic::x86_mmx_psll_w:
3883 case Intrinsic::x86_mmx_psll_d:
3884 case Intrinsic::x86_mmx_psll_q:
3885 case Intrinsic::x86_mmx_pslli_w:
3886 case Intrinsic::x86_mmx_pslli_d:
3887 case Intrinsic::x86_mmx_pslli_q:
3888 case Intrinsic::x86_mmx_psrl_w:
3889 case Intrinsic::x86_mmx_psrl_d:
3890 case Intrinsic::x86_mmx_psrl_q:
3891 case Intrinsic::x86_mmx_psra_w:
3892 case Intrinsic::x86_mmx_psra_d:
3893 case Intrinsic::x86_mmx_psrli_w:
3894 case Intrinsic::x86_mmx_psrli_d:
3895 case Intrinsic::x86_mmx_psrli_q:
3896 case Intrinsic::x86_mmx_psrai_w:
3897 case Intrinsic::x86_mmx_psrai_d:
3898 handleVectorShiftIntrinsic(I, /* Variable */ false);
3899 break;
3900 case Intrinsic::x86_avx2_psllv_d:
3901 case Intrinsic::x86_avx2_psllv_d_256:
3902 case Intrinsic::x86_avx512_psllv_d_512:
3903 case Intrinsic::x86_avx2_psllv_q:
3904 case Intrinsic::x86_avx2_psllv_q_256:
3905 case Intrinsic::x86_avx512_psllv_q_512:
3906 case Intrinsic::x86_avx2_psrlv_d:
3907 case Intrinsic::x86_avx2_psrlv_d_256:
3908 case Intrinsic::x86_avx512_psrlv_d_512:
3909 case Intrinsic::x86_avx2_psrlv_q:
3910 case Intrinsic::x86_avx2_psrlv_q_256:
3911 case Intrinsic::x86_avx512_psrlv_q_512:
3912 case Intrinsic::x86_avx2_psrav_d:
3913 case Intrinsic::x86_avx2_psrav_d_256:
3914 case Intrinsic::x86_avx512_psrav_d_512:
3915 case Intrinsic::x86_avx512_psrav_q_128:
3916 case Intrinsic::x86_avx512_psrav_q_256:
3917 case Intrinsic::x86_avx512_psrav_q_512:
3918 handleVectorShiftIntrinsic(I, /* Variable */ true);
3919 break;
3920
3921 case Intrinsic::x86_sse2_packsswb_128:
3922 case Intrinsic::x86_sse2_packssdw_128:
3923 case Intrinsic::x86_sse2_packuswb_128:
3924 case Intrinsic::x86_sse41_packusdw:
3925 case Intrinsic::x86_avx2_packsswb:
3926 case Intrinsic::x86_avx2_packssdw:
3927 case Intrinsic::x86_avx2_packuswb:
3928 case Intrinsic::x86_avx2_packusdw:
3929 handleVectorPackIntrinsic(I);
3930 break;
3931
3932 case Intrinsic::x86_mmx_packsswb:
3933 case Intrinsic::x86_mmx_packuswb:
3934 handleVectorPackIntrinsic(I, 16);
3935 break;
3936
3937 case Intrinsic::x86_mmx_packssdw:
3938 handleVectorPackIntrinsic(I, 32);
3939 break;
3940
3941 case Intrinsic::x86_mmx_psad_bw:
3942 case Intrinsic::x86_sse2_psad_bw:
3943 case Intrinsic::x86_avx2_psad_bw:
3944 handleVectorSadIntrinsic(I);
3945 break;
3946
3947 case Intrinsic::x86_sse2_pmadd_wd:
3948 case Intrinsic::x86_avx2_pmadd_wd:
3949 case Intrinsic::x86_ssse3_pmadd_ub_sw_128:
3950 case Intrinsic::x86_avx2_pmadd_ub_sw:
3951 handleVectorPmaddIntrinsic(I);
3952 break;
3953
3954 case Intrinsic::x86_ssse3_pmadd_ub_sw:
3955 handleVectorPmaddIntrinsic(I, 8);
3956 break;
3957
3958 case Intrinsic::x86_mmx_pmadd_wd:
3959 handleVectorPmaddIntrinsic(I, 16);
3960 break;
3961
3962 case Intrinsic::x86_sse_cmp_ss:
3963 case Intrinsic::x86_sse2_cmp_sd:
3964 case Intrinsic::x86_sse_comieq_ss:
3965 case Intrinsic::x86_sse_comilt_ss:
3966 case Intrinsic::x86_sse_comile_ss:
3967 case Intrinsic::x86_sse_comigt_ss:
3968 case Intrinsic::x86_sse_comige_ss:
3969 case Intrinsic::x86_sse_comineq_ss:
3970 case Intrinsic::x86_sse_ucomieq_ss:
3971 case Intrinsic::x86_sse_ucomilt_ss:
3972 case Intrinsic::x86_sse_ucomile_ss:
3973 case Intrinsic::x86_sse_ucomigt_ss:
3974 case Intrinsic::x86_sse_ucomige_ss:
3975 case Intrinsic::x86_sse_ucomineq_ss:
3976 case Intrinsic::x86_sse2_comieq_sd:
3977 case Intrinsic::x86_sse2_comilt_sd:
3978 case Intrinsic::x86_sse2_comile_sd:
3979 case Intrinsic::x86_sse2_comigt_sd:
3980 case Intrinsic::x86_sse2_comige_sd:
3981 case Intrinsic::x86_sse2_comineq_sd:
3982 case Intrinsic::x86_sse2_ucomieq_sd:
3983 case Intrinsic::x86_sse2_ucomilt_sd:
3984 case Intrinsic::x86_sse2_ucomile_sd:
3985 case Intrinsic::x86_sse2_ucomigt_sd:
3986 case Intrinsic::x86_sse2_ucomige_sd:
3987 case Intrinsic::x86_sse2_ucomineq_sd:
3988 handleVectorCompareScalarIntrinsic(I);
3989 break;
3990
3991 case Intrinsic::x86_avx_cmp_pd_256:
3992 case Intrinsic::x86_avx_cmp_ps_256:
3993 case Intrinsic::x86_sse2_cmp_pd:
3994 case Intrinsic::x86_sse_cmp_ps:
3995 handleVectorComparePackedIntrinsic(I);
3996 break;
3997
3998 case Intrinsic::x86_bmi_bextr_32:
3999 case Intrinsic::x86_bmi_bextr_64:
4000 case Intrinsic::x86_bmi_bzhi_32:
4001 case Intrinsic::x86_bmi_bzhi_64:
4002 case Intrinsic::x86_bmi_pdep_32:
4003 case Intrinsic::x86_bmi_pdep_64:
4004 case Intrinsic::x86_bmi_pext_32:
4005 case Intrinsic::x86_bmi_pext_64:
4006 handleBmiIntrinsic(I);
4007 break;
4008
4009 case Intrinsic::x86_pclmulqdq:
4010 case Intrinsic::x86_pclmulqdq_256:
4011 case Intrinsic::x86_pclmulqdq_512:
4012 handlePclmulIntrinsic(I);
4013 break;
4014
4015 case Intrinsic::x86_sse41_round_sd:
4016 case Intrinsic::x86_sse41_round_ss:
4017 handleUnarySdSsIntrinsic(I);
4018 break;
4019 case Intrinsic::x86_sse2_max_sd:
4020 case Intrinsic::x86_sse_max_ss:
4021 case Intrinsic::x86_sse2_min_sd:
4022 case Intrinsic::x86_sse_min_ss:
4023 handleBinarySdSsIntrinsic(I);
4024 break;
4025
4026 case Intrinsic::x86_avx_vtestc_pd:
4027 case Intrinsic::x86_avx_vtestc_pd_256:
4028 case Intrinsic::x86_avx_vtestc_ps:
4029 case Intrinsic::x86_avx_vtestc_ps_256:
4030 case Intrinsic::x86_avx_vtestnzc_pd:
4031 case Intrinsic::x86_avx_vtestnzc_pd_256:
4032 case Intrinsic::x86_avx_vtestnzc_ps:
4033 case Intrinsic::x86_avx_vtestnzc_ps_256:
4034 case Intrinsic::x86_avx_vtestz_pd:
4035 case Intrinsic::x86_avx_vtestz_pd_256:
4036 case Intrinsic::x86_avx_vtestz_ps:
4037 case Intrinsic::x86_avx_vtestz_ps_256:
4038 case Intrinsic::x86_avx_ptestc_256:
4039 case Intrinsic::x86_avx_ptestnzc_256:
4040 case Intrinsic::x86_avx_ptestz_256:
4041 case Intrinsic::x86_sse41_ptestc:
4042 case Intrinsic::x86_sse41_ptestnzc:
4043 case Intrinsic::x86_sse41_ptestz:
4044 handleVtestIntrinsic(I);
4045 break;
4046
4047 case Intrinsic::fshl:
4048 case Intrinsic::fshr:
4049 handleFunnelShift(I);
4050 break;
4051
4052 case Intrinsic::is_constant:
4053 // The result of llvm.is.constant() is always defined.
4054 setShadow(&I, getCleanShadow(&I));
4055 setOrigin(&I, getCleanOrigin());
4056 break;
4057
4058 default:
4059 if (!handleUnknownIntrinsic(I))
4060 visitInstruction(I);
4061 break;
4062 }
4063 }
4064
4065 void visitLibAtomicLoad(CallBase &CB) {
4066 // Since we use getNextNode here, we can't have CB terminate the BB.
4067 assert(isa<CallInst>(CB));
4068
4069 IRBuilder<> IRB(&CB);
4070 Value *Size = CB.getArgOperand(0);
4071 Value *SrcPtr = CB.getArgOperand(1);
4072 Value *DstPtr = CB.getArgOperand(2);
4073 Value *Ordering = CB.getArgOperand(3);
4074 // Convert the call to have at least Acquire ordering to make sure
4075 // the shadow operations aren't reordered before it.
4076 Value *NewOrdering =
4077 IRB.CreateExtractElement(makeAddAcquireOrderingTable(IRB), Ordering);
4078 CB.setArgOperand(3, NewOrdering);
4079
4080 NextNodeIRBuilder NextIRB(&CB);
4081 Value *SrcShadowPtr, *SrcOriginPtr;
4082 std::tie(SrcShadowPtr, SrcOriginPtr) =
4083 getShadowOriginPtr(SrcPtr, NextIRB, NextIRB.getInt8Ty(), Align(1),
4084 /*isStore*/ false);
4085 Value *DstShadowPtr =
4086 getShadowOriginPtr(DstPtr, NextIRB, NextIRB.getInt8Ty(), Align(1),
4087 /*isStore*/ true)
4088 .first;
4089
4090 NextIRB.CreateMemCpy(DstShadowPtr, Align(1), SrcShadowPtr, Align(1), Size);
4091 if (MS.TrackOrigins) {
4092 Value *SrcOrigin = NextIRB.CreateAlignedLoad(MS.OriginTy, SrcOriginPtr,
4094 Value *NewOrigin = updateOrigin(SrcOrigin, NextIRB);
4095 NextIRB.CreateCall(MS.MsanSetOriginFn, {DstPtr, Size, NewOrigin});
4096 }
4097 }
4098
4099 void visitLibAtomicStore(CallBase &CB) {
4100 IRBuilder<> IRB(&CB);
4101 Value *Size = CB.getArgOperand(0);
4102 Value *DstPtr = CB.getArgOperand(2);
4103 Value *Ordering = CB.getArgOperand(3);
4104 // Convert the call to have at least Release ordering to make sure
4105 // the shadow operations aren't reordered after it.
4106 Value *NewOrdering =
4107 IRB.CreateExtractElement(makeAddReleaseOrderingTable(IRB), Ordering);
4108 CB.setArgOperand(3, NewOrdering);
4109
4110 Value *DstShadowPtr =
4111 getShadowOriginPtr(DstPtr, IRB, IRB.getInt8Ty(), Align(1),
4112 /*isStore*/ true)
4113 .first;
4114
4115 // Atomic store always paints clean shadow/origin. See file header.
4116 IRB.CreateMemSet(DstShadowPtr, getCleanShadow(IRB.getInt8Ty()), Size,
4117 Align(1));
4118 }
4119
4120 void visitCallBase(CallBase &CB) {
4121 assert(!CB.getMetadata(LLVMContext::MD_nosanitize));
4122 if (CB.isInlineAsm()) {
4123 // For inline asm (either a call to asm function, or callbr instruction),
4124 // do the usual thing: check argument shadow and mark all outputs as
4125 // clean. Note that any side effects of the inline asm that are not
4126 // immediately visible in its constraints are not handled.
4128 visitAsmInstruction(CB);
4129 else
4130 visitInstruction(CB);
4131 return;
4132 }
4133 LibFunc LF;
4134 if (TLI->getLibFunc(CB, LF)) {
4135 // libatomic.a functions need to have special handling because there isn't
4136 // a good way to intercept them or compile the library with
4137 // instrumentation.
4138 switch (LF) {
4139 case LibFunc_atomic_load:
4140 if (!isa<CallInst>(CB)) {
4141 llvm::errs() << "MSAN -- cannot instrument invoke of libatomic load."
4142 "Ignoring!\n";
4143 break;
4144 }
4145 visitLibAtomicLoad(CB);
4146 return;
4147 case LibFunc_atomic_store:
4148 visitLibAtomicStore(CB);
4149 return;
4150 default:
4151 break;
4152 }
4153 }
4154
4155 if (auto *Call = dyn_cast<CallInst>(&CB)) {
4156 assert(!isa<IntrinsicInst>(Call) && "intrinsics are handled elsewhere");
4157
4158 // We are going to insert code that relies on the fact that the callee
4159 // will become a non-readonly function after it is instrumented by us. To
4160 // prevent this code from being optimized out, mark that function
4161 // non-readonly in advance.
4162 // TODO: We can likely do better than dropping memory() completely here.
4164 B.addAttribute(Attribute::Memory).addAttribute(Attribute::Speculatable);
4165
4166 Call->removeFnAttrs(B);
4167 if (Function *Func = Call->getCalledFunction()) {
4168 Func->removeFnAttrs(B);
4169 }
4170
4172 }
4173 IRBuilder<> IRB(&CB);
4174 bool MayCheckCall = MS.EagerChecks;
4175 if (Function *Func = CB.getCalledFunction()) {
4176 // __sanitizer_unaligned_{load,store} functions may be called by users
4177 // and always expects shadows in the TLS. So don't check them.
4178 MayCheckCall &= !Func->getName().starts_with("__sanitizer_unaligned_");
4179 }
4180
4181 unsigned ArgOffset = 0;
4182 LLVM_DEBUG(dbgs() << " CallSite: " << CB << "\n");
4183 for (const auto &[i, A] : llvm::enumerate(CB.args())) {
4184 if (!A->getType()->isSized()) {
4185 LLVM_DEBUG(dbgs() << "Arg " << i << " is not sized: " << CB << "\n");
4186 continue;
4187 }
4188 unsigned Size = 0;
4189 const DataLayout &DL = F.getParent()->getDataLayout();
4190
4191 bool ByVal = CB.paramHasAttr(i, Attribute::ByVal);
4192 bool NoUndef = CB.paramHasAttr(i, Attribute::NoUndef);
4193 bool EagerCheck = MayCheckCall && !ByVal && NoUndef;
4194
4195 if (EagerCheck) {
4196 insertShadowCheck(A, &CB);
4197 Size = DL.getTypeAllocSize(A->getType());
4198 } else {
4199 Value *Store = nullptr;
4200 // Compute the Shadow for arg even if it is ByVal, because
4201 // in that case getShadow() will copy the actual arg shadow to
4202 // __msan_param_tls.
4203 Value *ArgShadow = getShadow(A);
4204 Value *ArgShadowBase = getShadowPtrForArgument(IRB, ArgOffset);
4205 LLVM_DEBUG(dbgs() << " Arg#" << i << ": " << *A
4206 << " Shadow: " << *ArgShadow << "\n");
4207 if (ByVal) {
4208 // ByVal requires some special handling as it's too big for a single
4209 // load
4210 assert(A->getType()->isPointerTy() &&
4211 "ByVal argument is not a pointer!");
4212 Size = DL.getTypeAllocSize(CB.getParamByValType(i));
4213 if (ArgOffset + Size > kParamTLSSize)
4214 break;
4215 const MaybeAlign ParamAlignment(CB.getParamAlign(i));
4216 MaybeAlign Alignment = std::nullopt;
4217 if (ParamAlignment)
4218 Alignment = std::min(*ParamAlignment, kShadowTLSAlignment);
4219 Value *AShadowPtr, *AOriginPtr;
4220 std::tie(AShadowPtr, AOriginPtr) =
4221 getShadowOriginPtr(A, IRB, IRB.getInt8Ty(), Alignment,
4222 /*isStore*/ false);
4223 if (!PropagateShadow) {
4224 Store = IRB.CreateMemSet(ArgShadowBase,
4226 Size, Alignment);
4227 } else {
4228 Store = IRB.CreateMemCpy(ArgShadowBase, Alignment, AShadowPtr,
4229 Alignment, Size);
4230 if (MS.TrackOrigins) {
4231 Value *ArgOriginBase = getOriginPtrForArgument(IRB, ArgOffset);
4232 // FIXME: OriginSize should be:
4233 // alignTo(A % kMinOriginAlignment + Size, kMinOriginAlignment)
4234 unsigned OriginSize = alignTo(Size, kMinOriginAlignment);
4235 IRB.CreateMemCpy(
4236 ArgOriginBase,
4237 /* by origin_tls[ArgOffset] */ kMinOriginAlignment,
4238 AOriginPtr,
4239 /* by getShadowOriginPtr */ kMinOriginAlignment, OriginSize);
4240 }
4241 }
4242 } else {
4243 // Any other parameters mean we need bit-grained tracking of uninit
4244 // data
4245 Size = DL.getTypeAllocSize(A->getType());
4246 if (ArgOffset + Size > kParamTLSSize)
4247 break;
4248 Store = IRB.CreateAlignedStore(ArgShadow, ArgShadowBase,
4250 Constant *Cst = dyn_cast<Constant>(ArgShadow);
4251 if (MS.TrackOrigins && !(Cst && Cst->isNullValue())) {
4252 IRB.CreateStore(getOrigin(A),
4253 getOriginPtrForArgument(IRB, ArgOffset));
4254 }
4255 }
4256 (void)Store;
4257 assert(Store != nullptr);
4258 LLVM_DEBUG(dbgs() << " Param:" << *Store << "\n");
4259 }
4260 assert(Size != 0);
4261 ArgOffset += alignTo(Size, kShadowTLSAlignment);
4262 }
4263 LLVM_DEBUG(dbgs() << " done with call args\n");
4264
4265 FunctionType *FT = CB.getFunctionType();
4266 if (FT->isVarArg()) {
4267 VAHelper->visitCallBase(CB, IRB);
4268 }
4269
4270 // Now, get the shadow for the RetVal.
4271 if (!CB.getType()->isSized())
4272 return;
4273 // Don't emit the epilogue for musttail call returns.
4274 if (isa<CallInst>(CB) && cast<CallInst>(CB).isMustTailCall())
4275 return;
4276
4277 if (MayCheckCall && CB.hasRetAttr(Attribute::NoUndef)) {
4278 setShadow(&CB, getCleanShadow(&CB));
4279 setOrigin(&CB, getCleanOrigin());
4280 return;
4281 }
4282
4283 IRBuilder<> IRBBefore(&CB);
4284 // Until we have full dynamic coverage, make sure the retval shadow is 0.
4285 Value *Base = getShadowPtrForRetval(IRBBefore);
4286 IRBBefore.CreateAlignedStore(getCleanShadow(&CB), Base,
4288 BasicBlock::iterator NextInsn;
4289 if (isa<CallInst>(CB)) {
4290 NextInsn = ++CB.getIterator();
4291 assert(NextInsn != CB.getParent()->end());
4292 } else {
4293 BasicBlock *NormalDest = cast<InvokeInst>(CB).getNormalDest();
4294 if (!NormalDest->getSinglePredecessor()) {
4295 // FIXME: this case is tricky, so we are just conservative here.
4296 // Perhaps we need to split the edge between this BB and NormalDest,
4297 // but a naive attempt to use SplitEdge leads to a crash.
4298 setShadow(&CB, getCleanShadow(&CB));
4299 setOrigin(&CB, getCleanOrigin());
4300 return;
4301 }
4302 // FIXME: NextInsn is likely in a basic block that has not been visited
4303 // yet. Anything inserted there will be instrumented by MSan later!
4304 NextInsn = NormalDest->getFirstInsertionPt();
4305 assert(NextInsn != NormalDest->end() &&
4306 "Could not find insertion point for retval shadow load");
4307 }
4308 IRBuilder<> IRBAfter(&*NextInsn);
4309 Value *RetvalShadow = IRBAfter.CreateAlignedLoad(
4310 getShadowTy(&CB), getShadowPtrForRetval(IRBAfter),
4311 kShadowTLSAlignment, "_msret");
4312 setShadow(&CB, RetvalShadow);
4313 if (MS.TrackOrigins)
4314 setOrigin(&CB, IRBAfter.CreateLoad(MS.OriginTy,
4315 getOriginPtrForRetval()));
4316 }
4317
4318 bool isAMustTailRetVal(Value *RetVal) {
4319 if (auto *I = dyn_cast<BitCastInst>(RetVal)) {
4320 RetVal = I->getOperand(0);
4321 }
4322 if (auto *I = dyn_cast<CallInst>(RetVal)) {
4323 return I->isMustTailCall();
4324 }
4325 return false;
4326 }
4327
4328 void visitReturnInst(ReturnInst &I) {
4329 IRBuilder<> IRB(&I);
4330 Value *RetVal = I.getReturnValue();
4331 if (!RetVal)
4332 return;
4333 // Don't emit the epilogue for musttail call returns.
4334 if (isAMustTailRetVal(RetVal))
4335 return;
4336 Value *ShadowPtr = getShadowPtrForRetval(IRB);
4337 bool HasNoUndef = F.hasRetAttribute(Attribute::NoUndef);
4338 bool StoreShadow = !(MS.EagerChecks && HasNoUndef);
4339 // FIXME: Consider using SpecialCaseList to specify a list of functions that
4340 // must always return fully initialized values. For now, we hardcode "main".
4341 bool EagerCheck = (MS.EagerChecks && HasNoUndef) || (F.getName() == "main");
4342
4343 Value *Shadow = getShadow(RetVal);
4344 bool StoreOrigin = true;
4345 if (EagerCheck) {
4346 insertShadowCheck(RetVal, &I);
4347 Shadow = getCleanShadow(RetVal);
4348 StoreOrigin = false;
4349 }
4350
4351 // The caller may still expect information passed over TLS if we pass our
4352 // check
4353 if (StoreShadow) {
4354 IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment);
4355 if (MS.TrackOrigins && StoreOrigin)
4356 IRB.CreateStore(getOrigin(RetVal), getOriginPtrForRetval());
4357 }
4358 }
4359
4360 void visitPHINode(PHINode &I) {
4361 IRBuilder<> IRB(&I);
4362 if (!PropagateShadow) {
4363 setShadow(&I, getCleanShadow(&I));
4364 setOrigin(&I, getCleanOrigin());
4365 return;
4366 }
4367
4368 ShadowPHINodes.push_back(&I);
4369 setShadow(&I, IRB.CreatePHI(getShadowTy(&I), I.getNumIncomingValues(),
4370 "_msphi_s"));
4371 if (MS.TrackOrigins)
4372 setOrigin(
4373 &I, IRB.CreatePHI(MS.OriginTy, I.getNumIncomingValues(), "_msphi_o"));
4374 }
4375
4376 Value *getLocalVarIdptr(AllocaInst &I) {
4377 ConstantInt *IntConst =
4378 ConstantInt::get(Type::getInt32Ty((*F.getParent()).getContext()), 0);
4379 return new GlobalVariable(*F.getParent(), IntConst->getType(),
4380 /*isConstant=*/false, GlobalValue::PrivateLinkage,
4381 IntConst);
4382 }
4383
4384 Value *getLocalVarDescription(AllocaInst &I) {
4385 return createPrivateConstGlobalForString(*F.getParent(), I.getName());
4386 }
4387
4388 void poisonAllocaUserspace(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
4389 if (PoisonStack && ClPoisonStackWithCall) {
4390 IRB.CreateCall(MS.MsanPoisonStackFn, {&I, Len});
4391 } else {
4392 Value *ShadowBase, *OriginBase;
4393 std::tie(ShadowBase, OriginBase) = getShadowOriginPtr(
4394 &I, IRB, IRB.getInt8Ty(), Align(1), /*isStore*/ true);
4395
4396 Value *PoisonValue = IRB.getInt8(PoisonStack ? ClPoisonStackPattern : 0);
4397 IRB.CreateMemSet(ShadowBase, PoisonValue, Len, I.getAlign());
4398 }
4399
4400 if (PoisonStack && MS.TrackOrigins) {
4401 Value *Idptr = getLocalVarIdptr(I);
4402 if (ClPrintStackNames) {
4403 Value *Descr = getLocalVarDescription(I);
4404 IRB.CreateCall(MS.MsanSetAllocaOriginWithDescriptionFn,
4405 {&I, Len, Idptr, Descr});
4406 } else {
4407 IRB.CreateCall(MS.MsanSetAllocaOriginNoDescriptionFn, {&I, Len, Idptr});
4408 }
4409 }
4410 }
4411
4412 void poisonAllocaKmsan(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
4413 Value *Descr = getLocalVarDescription(I);
4414 if (PoisonStack) {
4415 IRB.CreateCall(MS.MsanPoisonAllocaFn, {&I, Len, Descr});
4416 } else {
4417 IRB.CreateCall(MS.MsanUnpoisonAllocaFn, {&I, Len});
4418 }
4419 }
4420
4421 void instrumentAlloca(AllocaInst &I, Instruction *InsPoint = nullptr) {
4422 if (!InsPoint)
4423 InsPoint = &I;
4424 NextNodeIRBuilder IRB(InsPoint);
4425 const DataLayout &DL = F.getParent()->getDataLayout();
4426 uint64_t TypeSize = DL.getTypeAllocSize(I.getAllocatedType());
4427 Value *Len = ConstantInt::get(MS.IntptrTy, TypeSize);
4428 if (I.isArrayAllocation())
4429 Len = IRB.CreateMul(Len,
4430 IRB.CreateZExtOrTrunc(I.getArraySize(), MS.IntptrTy));
4431
4432 if (MS.CompileKernel)
4433 poisonAllocaKmsan(I, IRB, Len);
4434 else
4435 poisonAllocaUserspace(I, IRB, Len);
4436 }
4437
4438 void visitAllocaInst(AllocaInst &I) {
4439 setShadow(&I, getCleanShadow(&I));
4440 setOrigin(&I, getCleanOrigin());
4441 // We'll get to this alloca later unless it's poisoned at the corresponding
4442 // llvm.lifetime.start.
4443 AllocaSet.insert(&I);
4444 }
4445
4446 void visitSelectInst(SelectInst &I) {
4447 IRBuilder<> IRB(&I);
4448 // a = select b, c, d
4449 Value *B = I.getCondition();
4450 Value *C = I.getTrueValue();
4451 Value *D = I.getFalseValue();
4452 Value *Sb = getShadow(B);
4453 Value *Sc = getShadow(C);
4454 Value *Sd = getShadow(D);
4455
4456 // Result shadow if condition shadow is 0.
4457 Value *Sa0 = IRB.CreateSelect(B, Sc, Sd);
4458 Value *Sa1;
4459 if (I.getType()->isAggregateType()) {
4460 // To avoid "sign extending" i1 to an arbitrary aggregate type, we just do
4461 // an extra "select". This results in much more compact IR.
4462 // Sa = select Sb, poisoned, (select b, Sc, Sd)
4463 Sa1 = getPoisonedShadow(getShadowTy(I.getType()));
4464 } else {
4465 // Sa = select Sb, [ (c^d) | Sc | Sd ], [ b ? Sc : Sd ]
4466 // If Sb (condition is poisoned), look for bits in c and d that are equal
4467 // and both unpoisoned.
4468 // If !Sb (condition is unpoisoned), simply pick one of Sc and Sd.
4469
4470 // Cast arguments to shadow-compatible type.
4471 C = CreateAppToShadowCast(IRB, C);
4472 D = CreateAppToShadowCast(IRB, D);
4473
4474 // Result shadow if condition shadow is 1.
4475 Sa1 = IRB.CreateOr({IRB.CreateXor(C, D), Sc, Sd});
4476 }
4477 Value *Sa = IRB.CreateSelect(Sb, Sa1, Sa0, "_msprop_select");
4478 setShadow(&I, Sa);
4479 if (MS.TrackOrigins) {
4480 // Origins are always i32, so any vector conditions must be flattened.
4481 // FIXME: consider tracking vector origins for app vectors?
4482 if (B->getType()->isVectorTy()) {
4483 B = convertToBool(B, IRB);
4484 Sb = convertToBool(Sb, IRB);
4485 }
4486 // a = select b, c, d
4487 // Oa = Sb ? Ob : (b ? Oc : Od)
4488 setOrigin(
4489 &I, IRB.CreateSelect(Sb, getOrigin(I.getCondition()),
4490 IRB.CreateSelect(B, getOrigin(I.getTrueValue()),
4491 getOrigin(I.getFalseValue()))));
4492 }
4493 }
4494
4495 void visitLandingPadInst(LandingPadInst &I) {
4496 // Do nothing.
4497 // See https://github.com/google/sanitizers/issues/504
4498 setShadow(&I, getCleanShadow(&I));
4499 setOrigin(&I, getCleanOrigin());
4500 }
4501
4502 void visitCatchSwitchInst(CatchSwitchInst &I) {
4503 setShadow(&I, getCleanShadow(&I));
4504 setOrigin(&I, getCleanOrigin());
4505 }
4506
4507 void visitFuncletPadInst(FuncletPadInst &I) {
4508 setShadow(&I, getCleanShadow(&I));
4509 setOrigin(&I, getCleanOrigin());
4510 }
4511
4512 void visitGetElementPtrInst(GetElementPtrInst &I) { handleShadowOr(I); }
4513
4514 void visitExtractValueInst(ExtractValueInst &I) {
4515 IRBuilder<> IRB(&I);
4516 Value *Agg = I.getAggregateOperand();
4517 LLVM_DEBUG(dbgs() << "ExtractValue: " << I << "\n");
4518 Value *AggShadow = getShadow(Agg);
4519 LLVM_DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
4520 Value *ResShadow = IRB.CreateExtractValue(AggShadow, I.getIndices());
4521 LLVM_DEBUG(dbgs() << " ResShadow: " << *ResShadow << "\n");
4522 setShadow(&I, ResShadow);
4523 setOriginForNaryOp(I);
4524 }
4525
4526 void visitInsertValueInst(InsertValueInst &I) {
4527 IRBuilder<> IRB(&I);
4528 LLVM_DEBUG(dbgs() << "InsertValue: " << I << "\n");
4529 Value *AggShadow = getShadow(I.getAggregateOperand());
4530 Value *InsShadow = getShadow(I.getInsertedValueOperand());
4531 LLVM_DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
4532 LLVM_DEBUG(dbgs() << " InsShadow: " << *InsShadow << "\n");
4533 Value *Res = IRB.CreateInsertValue(AggShadow, InsShadow, I.getIndices());
4534 LLVM_DEBUG(dbgs() << " Res: " << *Res << "\n");
4535 setShadow(&I, Res);
4536 setOriginForNaryOp(I);
4537 }
4538
4539 void dumpInst(Instruction &I) {
4540 if (CallInst *CI = dyn_cast<CallInst>(&I)) {
4541 errs() << "ZZZ call " << CI->getCalledFunction()->getName() << "\n";
4542 } else {
4543 errs() << "ZZZ " << I.getOpcodeName() << "\n";
4544 }
4545 errs() << "QQQ " << I << "\n";
4546 }
4547
4548 void visitResumeInst(ResumeInst &I) {
4549 LLVM_DEBUG(dbgs() << "Resume: " << I << "\n");
4550 // Nothing to do here.
4551 }
4552
4553 void visitCleanupReturnInst(CleanupReturnInst &CRI) {
4554 LLVM_DEBUG(dbgs() << "CleanupReturn: " << CRI << "\n");
4555 // Nothing to do here.
4556 }
4557
4558 void visitCatchReturnInst(CatchReturnInst &CRI) {
4559 LLVM_DEBUG(dbgs() << "CatchReturn: " << CRI << "\n");
4560 // Nothing to do here.
4561 }
4562
4563 void instrumentAsmArgument(Value *Operand, Type *ElemTy, Instruction &I,
4564 IRBuilder<> &IRB, const DataLayout &DL,
4565 bool isOutput) {
4566 // For each assembly argument, we check its value for being initialized.
4567 // If the argument is a pointer, we assume it points to a single element
4568 // of the corresponding type (or to a 8-byte word, if the type is unsized).
4569 // Each such pointer is instrumented with a call to the runtime library.
4570 Type *OpType = Operand->getType();
4571 // Check the operand value itself.
4572 insertShadowCheck(Operand, &I);
4573 if (!OpType->isPointerTy() || !isOutput) {
4574 assert(!isOutput);
4575 return;
4576 }
4577 if (!ElemTy->isSized())
4578 return;
4579 auto Size = DL.getTypeStoreSize(ElemTy);
4580 Value *SizeVal = IRB.CreateTypeSize(MS.IntptrTy, Size);
4581 if (MS.CompileKernel) {
4582 IRB.CreateCall(MS.MsanInstrumentAsmStoreFn, {Operand, SizeVal});
4583 } else {
4584 // ElemTy, derived from elementtype(), does not encode the alignment of
4585 // the pointer. Conservatively assume that the shadow memory is unaligned.
4586 // When Size is large, avoid StoreInst as it would expand to many
4587 // instructions.
4588 auto [ShadowPtr, _] =
4589 getShadowOriginPtrUserspace(Operand, IRB, IRB.getInt8Ty(), Align(1));
4590 if (Size <= 32)
4591 IRB.CreateAlignedStore(getCleanShadow(ElemTy), ShadowPtr, Align(1));
4592 else
4593 IRB.CreateMemSet(ShadowPtr, ConstantInt::getNullValue(IRB.getInt8Ty()),
4594 SizeVal, Align(1));
4595 }
4596 }
4597
4598 /// Get the number of output arguments returned by pointers.
4599 int getNumOutputArgs(InlineAsm *IA, CallBase *CB) {
4600 int NumRetOutputs = 0;
4601 int NumOutputs = 0;
4602 Type *RetTy = cast<Value>(CB)->getType();
4603 if (!RetTy->isVoidTy()) {
4604 // Register outputs are returned via the CallInst return value.
4605 auto *ST = dyn_cast<StructType>(RetTy);
4606 if (ST)
4607 NumRetOutputs = ST->getNumElements();
4608 else
4609 NumRetOutputs = 1;
4610 }
4611 InlineAsm::ConstraintInfoVector Constraints = IA->ParseConstraints();
4612 for (const InlineAsm::ConstraintInfo &Info : Constraints) {
4613 switch (Info.Type) {
4615 NumOutputs++;
4616 break;
4617 default:
4618 break;
4619 }
4620 }
4621 return NumOutputs - NumRetOutputs;
4622 }
4623
4624 void visitAsmInstruction(Instruction &I) {
4625 // Conservative inline assembly handling: check for poisoned shadow of
4626 // asm() arguments, then unpoison the result and all the memory locations
4627 // pointed to by those arguments.
4628 // An inline asm() statement in C++ contains lists of input and output
4629 // arguments used by the assembly code. These are mapped to operands of the
4630 // CallInst as follows:
4631 // - nR register outputs ("=r) are returned by value in a single structure
4632 // (SSA value of the CallInst);
4633 // - nO other outputs ("=m" and others) are returned by pointer as first
4634 // nO operands of the CallInst;
4635 // - nI inputs ("r", "m" and others) are passed to CallInst as the
4636 // remaining nI operands.
4637 // The total number of asm() arguments in the source is nR+nO+nI, and the
4638 // corresponding CallInst has nO+nI+1 operands (the last operand is the
4639 // function to be called).
4640 const DataLayout &DL = F.getParent()->getDataLayout();
4641 CallBase *CB = cast<CallBase>(&I);
4642 IRBuilder<> IRB(&I);
4643 InlineAsm *IA = cast<InlineAsm>(CB->getCalledOperand());
4644 int OutputArgs = getNumOutputArgs(IA, CB);
4645 // The last operand of a CallInst is the function itself.
4646 int NumOperands = CB->getNumOperands() - 1;
4647
4648 // Check input arguments. Doing so before unpoisoning output arguments, so
4649 // that we won't overwrite uninit values before checking them.
4650 for (int i = OutputArgs; i < NumOperands; i++) {
4651 Value *Operand = CB->getOperand(i);
4652 instrumentAsmArgument(Operand, CB->getParamElementType(i), I, IRB, DL,
4653 /*isOutput*/ false);
4654 }
4655 // Unpoison output arguments. This must happen before the actual InlineAsm
4656 // call, so that the shadow for memory published in the asm() statement
4657 // remains valid.
4658 for (int i = 0; i < OutputArgs; i++) {
4659 Value *Operand = CB->getOperand(i);
4660 instrumentAsmArgument(Operand, CB->getParamElementType(i), I, IRB, DL,
4661 /*isOutput*/ true);
4662 }
4663
4664 setShadow(&I, getCleanShadow(&I));
4665 setOrigin(&I, getCleanOrigin());
4666 }
4667
4668 void visitFreezeInst(FreezeInst &I) {
4669 // Freeze always returns a fully defined value.
4670 setShadow(&I, getCleanShadow(&I));
4671 setOrigin(&I, getCleanOrigin());
4672 }
4673
4674 void visitInstruction(Instruction &I) {
4675 // Everything else: stop propagating and check for poisoned shadow.
4677 dumpInst(I);
4678 LLVM_DEBUG(dbgs() << "DEFAULT: " << I << "\n");
4679 for (size_t i = 0, n = I.getNumOperands(); i < n; i++) {
4680 Value *Operand = I.getOperand(i);
4681 if (Operand->getType()->isSized())
4682 insertShadowCheck(Operand, &I);
4683 }
4684 setShadow(&I, getCleanShadow(&I));
4685 setOrigin(&I, getCleanOrigin());
4686 }
4687};
4688
4689struct VarArgHelperBase : public VarArgHelper {
4690 Function &F;
4691 MemorySanitizer &MS;
4692 MemorySanitizerVisitor &MSV;
4693 SmallVector<CallInst *, 16> VAStartInstrumentationList;
4694 const unsigned VAListTagSize;
4695
4696 VarArgHelperBase(Function &F, MemorySanitizer &MS,
4697 MemorySanitizerVisitor &MSV, unsigned VAListTagSize)
4698 : F(F), MS(MS), MSV(MSV), VAListTagSize(VAListTagSize) {}
4699
4700 Value *getShadowAddrForVAArgument(IRBuilder<> &IRB, unsigned ArgOffset) {
4701 Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4702 return IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4703 }
4704
4705 /// Compute the shadow address for a given va_arg.
4706 Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4707 unsigned ArgOffset) {
4708 Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4709 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4710 return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4711 "_msarg_va_s");
4712 }
4713
4714 /// Compute the shadow address for a given va_arg.
4715 Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4716 unsigned ArgOffset, unsigned ArgSize) {
4717 // Make sure we don't overflow __msan_va_arg_tls.
4718 if (ArgOffset + ArgSize > kParamTLSSize)
4719 return nullptr;
4720 return getShadowPtrForVAArgument(Ty, IRB, ArgOffset);
4721 }
4722
4723 /// Compute the origin address for a given va_arg.
4724 Value *getOriginPtrForVAArgument(IRBuilder<> &IRB, int ArgOffset) {
4725 Value *Base = IRB.CreatePointerCast(MS.VAArgOriginTLS, MS.IntptrTy);
4726 // getOriginPtrForVAArgument() is always called after
4727 // getShadowPtrForVAArgument(), so __msan_va_arg_origin_tls can never
4728 // overflow.
4729 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4730 return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0),
4731 "_msarg_va_o");
4732 }
4733
4734 void CleanUnusedTLS(IRBuilder<> &IRB, Value *ShadowBase,
4735 unsigned BaseOffset) {
4736 // The tails of __msan_va_arg_tls is not large enough to fit full
4737 // value shadow, but it will be copied to backup anyway. Make it
4738 // clean.
4739 if (BaseOffset >= kParamTLSSize)
4740 return;
4741 Value *TailSize =
4742 ConstantInt::getSigned(IRB.getInt32Ty(), kParamTLSSize - BaseOffset);
4743 IRB.CreateMemSet(ShadowBase, ConstantInt::getNullValue(IRB.getInt8Ty()),
4744 TailSize, Align(8));
4745 }
4746
4747 void unpoisonVAListTagForInst(IntrinsicInst &I) {
4748 IRBuilder<> IRB(&I);
4749 Value *VAListTag = I.getArgOperand(0);
4750 const Align Alignment = Align(8);
4751 auto [ShadowPtr, OriginPtr] = MSV.getShadowOriginPtr(
4752 VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4753 // Unpoison the whole __va_list_tag.
4754 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4755 VAListTagSize, Alignment, false);
4756 }
4757
4758 void visitVAStartInst(VAStartInst &I) override {
4759 if (F.getCallingConv() == CallingConv::Win64)
4760 return;
4761 VAStartInstrumentationList.push_back(&I);
4762 unpoisonVAListTagForInst(I);
4763 }
4764
4765 void visitVACopyInst(VACopyInst &I) override {
4766 if (F.getCallingConv() == CallingConv::Win64)
4767 return;
4768 unpoisonVAListTagForInst(I);
4769 }
4770};
4771
4772/// AMD64-specific implementation of VarArgHelper.
4773struct VarArgAMD64Helper : public VarArgHelperBase {
4774 // An unfortunate workaround for asymmetric lowering of va_arg stuff.
4775 // See a comment in visitCallBase for more details.
4776 static const unsigned AMD64GpEndOffset = 48; // AMD64 ABI Draft 0.99.6 p3.5.7
4777 static const unsigned AMD64FpEndOffsetSSE = 176;
4778 // If SSE is disabled, fp_offset in va_list is zero.
4779 static const unsigned AMD64FpEndOffsetNoSSE = AMD64GpEndOffset;
4780
4781 unsigned AMD64FpEndOffset;
4782 AllocaInst *VAArgTLSCopy = nullptr;
4783 AllocaInst *VAArgTLSOriginCopy = nullptr;
4784 Value *VAArgOverflowSize = nullptr;
4785
4786 enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory };
4787
4788 VarArgAMD64Helper(Function &F, MemorySanitizer &MS,
4789 MemorySanitizerVisitor &MSV)
4790 : VarArgHelperBase(F, MS, MSV, /*VAListTagSize=*/24) {
4791 AMD64FpEndOffset = AMD64FpEndOffsetSSE;
4792 for (const auto &Attr : F.getAttributes().getFnAttrs()) {
4793 if (Attr.isStringAttribute() &&
4794 (Attr.getKindAsString() == "target-features")) {
4795 if (Attr.getValueAsString().contains("-sse"))
4796 AMD64FpEndOffset = AMD64FpEndOffsetNoSSE;
4797 break;
4798 }
4799 }
4800 }
4801
4802 ArgKind classifyArgument(Value *arg) {
4803 // A very rough approximation of X86_64 argument classification rules.
4804 Type *T = arg->getType();
4805 if (T->isX86_FP80Ty())
4806 return AK_Memory;
4807 if (T->isFPOrFPVectorTy() || T->isX86_MMXTy())
4808 return AK_FloatingPoint;
4809 if (T->isIntegerTy() && T->getPrimitiveSizeInBits() <= 64)
4810 return AK_GeneralPurpose;
4811 if (T->isPointerTy())
4812 return AK_GeneralPurpose;
4813 return AK_Memory;
4814 }
4815
4816 // For VarArg functions, store the argument shadow in an ABI-specific format
4817 // that corresponds to va_list layout.
4818 // We do this because Clang lowers va_arg in the frontend, and this pass
4819 // only sees the low level code that deals with va_list internals.
4820 // A much easier alternative (provided that Clang emits va_arg instructions)
4821 // would have been to associate each live instance of va_list with a copy of
4822 // MSanParamTLS, and extract shadow on va_arg() call in the argument list
4823 // order.
4824 void visitCallBase(CallBase &CB, IRBuilder<> &IRB) override {
4825 unsigned GpOffset = 0;
4826 unsigned FpOffset = AMD64GpEndOffset;
4827 unsigned OverflowOffset = AMD64FpEndOffset;
4828 const DataLayout &DL = F.getParent()->getDataLayout();
4829
4830 for (const auto &[ArgNo, A] : llvm::enumerate(CB.args())) {
4831 bool IsFixed = ArgNo < CB.getFunctionType()->getNumParams();
4832 bool IsByVal = CB.paramHasAttr(ArgNo, Attribute::ByVal);
4833 if (IsByVal) {
4834 // ByVal arguments always go to the overflow area.
4835 // Fixed arguments passed through the overflow area will be stepped
4836 // over by va_start, so don't count them towards the offset.
4837 if (IsFixed)
4838 continue;
4839 assert(A->getType()->isPointerTy());
4840 Type *RealTy = CB.getParamByValType(ArgNo);
4841 uint64_t ArgSize = DL.getTypeAllocSize(RealTy);
4842 uint64_t AlignedSize = alignTo(ArgSize, 8);
4843 unsigned BaseOffset = OverflowOffset;
4844 Value *ShadowBase =
4845 getShadowPtrForVAArgument(RealTy, IRB, OverflowOffset);
4846 Value *OriginBase = nullptr;
4847 if (MS.TrackOrigins)
4848 OriginBase = getOriginPtrForVAArgument(IRB, OverflowOffset);
4849 OverflowOffset += AlignedSize;
4850
4851 if (OverflowOffset > kParamTLSSize) {
4852 CleanUnusedTLS(IRB, ShadowBase, BaseOffset);
4853 continue; // We have no space to copy shadow there.
4854 }
4855
4856 Value *ShadowPtr, *OriginPtr;
4857 std::tie(ShadowPtr, OriginPtr) =
4858 MSV.getShadowOriginPtr(A, IRB, IRB.getInt8Ty(), kShadowTLSAlignment,
4859 /*isStore*/ false);
4860 IRB.CreateMemCpy(ShadowBase, kShadowTLSAlignment, ShadowPtr,
4861 kShadowTLSAlignment, ArgSize);
4862 if (MS.TrackOrigins)
4863 IRB.CreateMemCpy(OriginBase, kShadowTLSAlignment, OriginPtr,
4864 kShadowTLSAlignment, ArgSize);
4865 } else {
4866 ArgKind AK = classifyArgument(A);
4867 if (AK == AK_GeneralPurpose && GpOffset >= AMD64GpEndOffset)
4868 AK = AK_Memory;
4869 if (AK == AK_FloatingPoint && FpOffset >= AMD64FpEndOffset)
4870 AK = AK_Memory;
4871 Value *ShadowBase, *OriginBase = nullptr;
4872 switch (AK) {
4873 case AK_GeneralPurpose:
4874 ShadowBase = getShadowPtrForVAArgument(A->getType(), IRB, GpOffset);
4875 if (MS.TrackOrigins)
4876 OriginBase = getOriginPtrForVAArgument(IRB, GpOffset);
4877 GpOffset += 8;
4878 assert(GpOffset <= kParamTLSSize);
4879 break;
4880 case AK_FloatingPoint:
4881 ShadowBase = getShadowPtrForVAArgument(A->getType(), IRB, FpOffset);
4882 if (MS.TrackOrigins)
4883 OriginBase = getOriginPtrForVAArgument(IRB, FpOffset);
4884 FpOffset += 16;
4885 assert(FpOffset <= kParamTLSSize);
4886 break;
4887 case AK_Memory:
4888 if (IsFixed)
4889 continue;
4890 uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
4891 uint64_t AlignedSize = alignTo(ArgSize, 8);
4892 unsigned BaseOffset = OverflowOffset;
4893 ShadowBase =
4894 getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset);
4895 if (MS.TrackOrigins) {
4896 OriginBase = getOriginPtrForVAArgument(IRB, OverflowOffset);
4897 }
4898 OverflowOffset += AlignedSize;
4899 if (OverflowOffset > kParamTLSSize) {
4900 // We have no space to copy shadow there.
4901 CleanUnusedTLS(IRB, ShadowBase, BaseOffset);
4902 continue;
4903 }
4904 }
4905 // Take fixed arguments into account for GpOffset and FpOffset,
4906 // but don't actually store shadows for them.
4907 // TODO(glider): don't call get*PtrForVAArgument() for them.
4908 if (IsFixed)
4909 continue;
4910 Value *Shadow = MSV.getShadow(A);
4911 IRB.CreateAlignedStore(Shadow, ShadowBase, kShadowTLSAlignment);
4912 if (MS.TrackOrigins) {
4913 Value *Origin = MSV.getOrigin(A);
4914 TypeSize StoreSize = DL.getTypeStoreSize(Shadow->getType());
4915 MSV.paintOrigin(IRB, Origin, OriginBase, StoreSize,
4917 }
4918 }
4919 }
4920 Constant *OverflowSize =
4921 ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AMD64FpEndOffset);
4922 IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
4923 }
4924
4925 void finalizeInstrumentation() override {
4926 assert(!VAArgOverflowSize && !VAArgTLSCopy &&
4927 "finalizeInstrumentation called twice");
4928 if (!VAStartInstrumentationList.empty()) {
4929 // If there is a va_start in this function, make a backup copy of
4930 // va_arg_tls somewhere in the function entry block.
4931 IRBuilder<> IRB(MSV.FnPrologueEnd);
4932 VAArgOverflowSize =
4933 IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
4934 Value *CopySize = IRB.CreateAdd(
4935 ConstantInt::get(MS.IntptrTy, AMD64FpEndOffset), VAArgOverflowSize);
4936 VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4937 VAArgTLSCopy->setAlignment(kShadowTLSAlignment);
4938 IRB.CreateMemSet(VAArgTLSCopy, Constant::getNullValue(IRB.getInt8Ty()),
4939 CopySize, kShadowTLSAlignment, false);
4940
4941