LLVM 19.0.0git
MemorySanitizer.cpp
Go to the documentation of this file.
1//===- MemorySanitizer.cpp - detector of uninitialized reads --------------===//
2//
3// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
4// See https://llvm.org/LICENSE.txt for license information.
5// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
6//
7//===----------------------------------------------------------------------===//
8//
9/// \file
10/// This file is a part of MemorySanitizer, a detector of uninitialized
11/// reads.
12///
13/// The algorithm of the tool is similar to Memcheck
14/// (http://goo.gl/QKbem). We associate a few shadow bits with every
15/// byte of the application memory, poison the shadow of the malloc-ed
16/// or alloca-ed memory, load the shadow bits on every memory read,
17/// propagate the shadow bits through some of the arithmetic
18/// instruction (including MOV), store the shadow bits on every memory
19/// write, report a bug on some other instructions (e.g. JMP) if the
20/// associated shadow is poisoned.
21///
22/// But there are differences too. The first and the major one:
23/// compiler instrumentation instead of binary instrumentation. This
24/// gives us much better register allocation, possible compiler
25/// optimizations and a fast start-up. But this brings the major issue
26/// as well: msan needs to see all program events, including system
27/// calls and reads/writes in system libraries, so we either need to
28/// compile *everything* with msan or use a binary translation
29/// component (e.g. DynamoRIO) to instrument pre-built libraries.
30/// Another difference from Memcheck is that we use 8 shadow bits per
31/// byte of application memory and use a direct shadow mapping. This
32/// greatly simplifies the instrumentation code and avoids races on
33/// shadow updates (Memcheck is single-threaded so races are not a
34/// concern there. Memcheck uses 2 shadow bits per byte with a slow
35/// path storage that uses 8 bits per byte).
36///
37/// The default value of shadow is 0, which means "clean" (not poisoned).
38///
39/// Every module initializer should call __msan_init to ensure that the
40/// shadow memory is ready. On error, __msan_warning is called. Since
41/// parameters and return values may be passed via registers, we have a
42/// specialized thread-local shadow for return values
43/// (__msan_retval_tls) and parameters (__msan_param_tls).
44///
45/// Origin tracking.
46///
47/// MemorySanitizer can track origins (allocation points) of all uninitialized
48/// values. This behavior is controlled with a flag (msan-track-origins) and is
49/// disabled by default.
50///
51/// Origins are 4-byte values created and interpreted by the runtime library.
52/// They are stored in a second shadow mapping, one 4-byte value for 4 bytes
53/// of application memory. Propagation of origins is basically a bunch of
54/// "select" instructions that pick the origin of a dirty argument, if an
55/// instruction has one.
56///
57/// Every 4 aligned, consecutive bytes of application memory have one origin
58/// value associated with them. If these bytes contain uninitialized data
59/// coming from 2 different allocations, the last store wins. Because of this,
60/// MemorySanitizer reports can show unrelated origins, but this is unlikely in
61/// practice.
62///
63/// Origins are meaningless for fully initialized values, so MemorySanitizer
64/// avoids storing origin to memory when a fully initialized value is stored.
65/// This way it avoids needless overwriting origin of the 4-byte region on
66/// a short (i.e. 1 byte) clean store, and it is also good for performance.
67///
68/// Atomic handling.
69///
70/// Ideally, every atomic store of application value should update the
71/// corresponding shadow location in an atomic way. Unfortunately, atomic store
72/// of two disjoint locations can not be done without severe slowdown.
73///
74/// Therefore, we implement an approximation that may err on the safe side.
75/// In this implementation, every atomically accessed location in the program
76/// may only change from (partially) uninitialized to fully initialized, but
77/// not the other way around. We load the shadow _after_ the application load,
78/// and we store the shadow _before_ the app store. Also, we always store clean
79/// shadow (if the application store is atomic). This way, if the store-load
80/// pair constitutes a happens-before arc, shadow store and load are correctly
81/// ordered such that the load will get either the value that was stored, or
82/// some later value (which is always clean).
83///
84/// This does not work very well with Compare-And-Swap (CAS) and
85/// Read-Modify-Write (RMW) operations. To follow the above logic, CAS and RMW
86/// must store the new shadow before the app operation, and load the shadow
87/// after the app operation. Computers don't work this way. Current
88/// implementation ignores the load aspect of CAS/RMW, always returning a clean
89/// value. It implements the store part as a simple atomic store by storing a
90/// clean shadow.
91///
92/// Instrumenting inline assembly.
93///
94/// For inline assembly code LLVM has little idea about which memory locations
95/// become initialized depending on the arguments. It can be possible to figure
96/// out which arguments are meant to point to inputs and outputs, but the
97/// actual semantics can be only visible at runtime. In the Linux kernel it's
98/// also possible that the arguments only indicate the offset for a base taken
99/// from a segment register, so it's dangerous to treat any asm() arguments as
100/// pointers. We take a conservative approach generating calls to
101/// __msan_instrument_asm_store(ptr, size)
102/// , which defer the memory unpoisoning to the runtime library.
103/// The latter can perform more complex address checks to figure out whether
104/// it's safe to touch the shadow memory.
105/// Like with atomic operations, we call __msan_instrument_asm_store() before
106/// the assembly call, so that changes to the shadow memory will be seen by
107/// other threads together with main memory initialization.
108///
109/// KernelMemorySanitizer (KMSAN) implementation.
110///
111/// The major differences between KMSAN and MSan instrumentation are:
112/// - KMSAN always tracks the origins and implies msan-keep-going=true;
113/// - KMSAN allocates shadow and origin memory for each page separately, so
114/// there are no explicit accesses to shadow and origin in the
115/// instrumentation.
116/// Shadow and origin values for a particular X-byte memory location
117/// (X=1,2,4,8) are accessed through pointers obtained via the
118/// __msan_metadata_ptr_for_load_X(ptr)
119/// __msan_metadata_ptr_for_store_X(ptr)
120/// functions. The corresponding functions check that the X-byte accesses
121/// are possible and returns the pointers to shadow and origin memory.
122/// Arbitrary sized accesses are handled with:
123/// __msan_metadata_ptr_for_load_n(ptr, size)
124/// __msan_metadata_ptr_for_store_n(ptr, size);
125/// Note that the sanitizer code has to deal with how shadow/origin pairs
126/// returned by the these functions are represented in different ABIs. In
127/// the X86_64 ABI they are returned in RDX:RAX, in PowerPC64 they are
128/// returned in r3 and r4, and in the SystemZ ABI they are written to memory
129/// pointed to by a hidden parameter.
130/// - TLS variables are stored in a single per-task struct. A call to a
131/// function __msan_get_context_state() returning a pointer to that struct
132/// is inserted into every instrumented function before the entry block;
133/// - __msan_warning() takes a 32-bit origin parameter;
134/// - local variables are poisoned with __msan_poison_alloca() upon function
135/// entry and unpoisoned with __msan_unpoison_alloca() before leaving the
136/// function;
137/// - the pass doesn't declare any global variables or add global constructors
138/// to the translation unit.
139///
140/// Also, KMSAN currently ignores uninitialized memory passed into inline asm
141/// calls, making sure we're on the safe side wrt. possible false positives.
142///
143/// KernelMemorySanitizer only supports X86_64, SystemZ and PowerPC64 at the
144/// moment.
145///
146//
147// FIXME: This sanitizer does not yet handle scalable vectors
148//
149//===----------------------------------------------------------------------===//
150
152#include "llvm/ADT/APInt.h"
153#include "llvm/ADT/ArrayRef.h"
154#include "llvm/ADT/DenseMap.h"
156#include "llvm/ADT/SetVector.h"
157#include "llvm/ADT/SmallPtrSet.h"
158#include "llvm/ADT/SmallVector.h"
160#include "llvm/ADT/StringRef.h"
164#include "llvm/IR/Argument.h"
166#include "llvm/IR/Attributes.h"
167#include "llvm/IR/BasicBlock.h"
168#include "llvm/IR/CallingConv.h"
169#include "llvm/IR/Constant.h"
170#include "llvm/IR/Constants.h"
171#include "llvm/IR/DataLayout.h"
172#include "llvm/IR/DerivedTypes.h"
173#include "llvm/IR/Function.h"
174#include "llvm/IR/GlobalValue.h"
176#include "llvm/IR/IRBuilder.h"
177#include "llvm/IR/InlineAsm.h"
178#include "llvm/IR/InstVisitor.h"
179#include "llvm/IR/InstrTypes.h"
180#include "llvm/IR/Instruction.h"
181#include "llvm/IR/Instructions.h"
183#include "llvm/IR/Intrinsics.h"
184#include "llvm/IR/IntrinsicsX86.h"
185#include "llvm/IR/MDBuilder.h"
186#include "llvm/IR/Module.h"
187#include "llvm/IR/Type.h"
188#include "llvm/IR/Value.h"
189#include "llvm/IR/ValueMap.h"
192#include "llvm/Support/Casting.h"
194#include "llvm/Support/Debug.h"
203#include <algorithm>
204#include <cassert>
205#include <cstddef>
206#include <cstdint>
207#include <memory>
208#include <string>
209#include <tuple>
210
211using namespace llvm;
212
213#define DEBUG_TYPE "msan"
214
215DEBUG_COUNTER(DebugInsertCheck, "msan-insert-check",
216 "Controls which checks to insert");
217
218DEBUG_COUNTER(DebugInstrumentInstruction, "msan-instrument-instruction",
219 "Controls which instruction to instrument");
220
221static const unsigned kOriginSize = 4;
224
225// These constants must be kept in sync with the ones in msan.h.
226static const unsigned kParamTLSSize = 800;
227static const unsigned kRetvalTLSSize = 800;
228
229// Accesses sizes are powers of two: 1, 2, 4, 8.
230static const size_t kNumberOfAccessSizes = 4;
231
232/// Track origins of uninitialized values.
233///
234/// Adds a section to MemorySanitizer report that points to the allocation
235/// (stack or heap) the uninitialized bits came from originally.
237 "msan-track-origins",
238 cl::desc("Track origins (allocation sites) of poisoned memory"), cl::Hidden,
239 cl::init(0));
240
241static cl::opt<bool> ClKeepGoing("msan-keep-going",
242 cl::desc("keep going after reporting a UMR"),
243 cl::Hidden, cl::init(false));
244
245static cl::opt<bool>
246 ClPoisonStack("msan-poison-stack",
247 cl::desc("poison uninitialized stack variables"), cl::Hidden,
248 cl::init(true));
249
251 "msan-poison-stack-with-call",
252 cl::desc("poison uninitialized stack variables with a call"), cl::Hidden,
253 cl::init(false));
254
256 "msan-poison-stack-pattern",
257 cl::desc("poison uninitialized stack variables with the given pattern"),
258 cl::Hidden, cl::init(0xff));
259
260static cl::opt<bool>
261 ClPrintStackNames("msan-print-stack-names",
262 cl::desc("Print name of local stack variable"),
263 cl::Hidden, cl::init(true));
264
265static cl::opt<bool> ClPoisonUndef("msan-poison-undef",
266 cl::desc("poison undef temps"), cl::Hidden,
267 cl::init(true));
268
269static cl::opt<bool>
270 ClHandleICmp("msan-handle-icmp",
271 cl::desc("propagate shadow through ICmpEQ and ICmpNE"),
272 cl::Hidden, cl::init(true));
273
274static cl::opt<bool>
275 ClHandleICmpExact("msan-handle-icmp-exact",
276 cl::desc("exact handling of relational integer ICmp"),
277 cl::Hidden, cl::init(false));
278
280 "msan-handle-lifetime-intrinsics",
281 cl::desc(
282 "when possible, poison scoped variables at the beginning of the scope "
283 "(slower, but more precise)"),
284 cl::Hidden, cl::init(true));
285
286// When compiling the Linux kernel, we sometimes see false positives related to
287// MSan being unable to understand that inline assembly calls may initialize
288// local variables.
289// This flag makes the compiler conservatively unpoison every memory location
290// passed into an assembly call. Note that this may cause false positives.
291// Because it's impossible to figure out the array sizes, we can only unpoison
292// the first sizeof(type) bytes for each type* pointer.
294 "msan-handle-asm-conservative",
295 cl::desc("conservative handling of inline assembly"), cl::Hidden,
296 cl::init(true));
297
298// This flag controls whether we check the shadow of the address
299// operand of load or store. Such bugs are very rare, since load from
300// a garbage address typically results in SEGV, but still happen
301// (e.g. only lower bits of address are garbage, or the access happens
302// early at program startup where malloc-ed memory is more likely to
303// be zeroed. As of 2012-08-28 this flag adds 20% slowdown.
305 "msan-check-access-address",
306 cl::desc("report accesses through a pointer which has poisoned shadow"),
307 cl::Hidden, cl::init(true));
308
310 "msan-eager-checks",
311 cl::desc("check arguments and return values at function call boundaries"),
312 cl::Hidden, cl::init(false));
313
315 "msan-dump-strict-instructions",
316 cl::desc("print out instructions with default strict semantics"),
317 cl::Hidden, cl::init(false));
318
320 "msan-instrumentation-with-call-threshold",
321 cl::desc(
322 "If the function being instrumented requires more than "
323 "this number of checks and origin stores, use callbacks instead of "
324 "inline checks (-1 means never use callbacks)."),
325 cl::Hidden, cl::init(3500));
326
327static cl::opt<bool>
328 ClEnableKmsan("msan-kernel",
329 cl::desc("Enable KernelMemorySanitizer instrumentation"),
330 cl::Hidden, cl::init(false));
331
332static cl::opt<bool>
333 ClDisableChecks("msan-disable-checks",
334 cl::desc("Apply no_sanitize to the whole file"), cl::Hidden,
335 cl::init(false));
336
337static cl::opt<bool>
338 ClCheckConstantShadow("msan-check-constant-shadow",
339 cl::desc("Insert checks for constant shadow values"),
340 cl::Hidden, cl::init(true));
341
342// This is off by default because of a bug in gold:
343// https://sourceware.org/bugzilla/show_bug.cgi?id=19002
344static cl::opt<bool>
345 ClWithComdat("msan-with-comdat",
346 cl::desc("Place MSan constructors in comdat sections"),
347 cl::Hidden, cl::init(false));
348
349// These options allow to specify custom memory map parameters
350// See MemoryMapParams for details.
351static cl::opt<uint64_t> ClAndMask("msan-and-mask",
352 cl::desc("Define custom MSan AndMask"),
353 cl::Hidden, cl::init(0));
354
355static cl::opt<uint64_t> ClXorMask("msan-xor-mask",
356 cl::desc("Define custom MSan XorMask"),
357 cl::Hidden, cl::init(0));
358
359static cl::opt<uint64_t> ClShadowBase("msan-shadow-base",
360 cl::desc("Define custom MSan ShadowBase"),
361 cl::Hidden, cl::init(0));
362
363static cl::opt<uint64_t> ClOriginBase("msan-origin-base",
364 cl::desc("Define custom MSan OriginBase"),
365 cl::Hidden, cl::init(0));
366
367static cl::opt<int>
368 ClDisambiguateWarning("msan-disambiguate-warning-threshold",
369 cl::desc("Define threshold for number of checks per "
370 "debug location to force origin update."),
371 cl::Hidden, cl::init(3));
372
373const char kMsanModuleCtorName[] = "msan.module_ctor";
374const char kMsanInitName[] = "__msan_init";
375
376namespace {
377
378// Memory map parameters used in application-to-shadow address calculation.
379// Offset = (Addr & ~AndMask) ^ XorMask
380// Shadow = ShadowBase + Offset
381// Origin = OriginBase + Offset
382struct MemoryMapParams {
383 uint64_t AndMask;
384 uint64_t XorMask;
385 uint64_t ShadowBase;
386 uint64_t OriginBase;
387};
388
389struct PlatformMemoryMapParams {
390 const MemoryMapParams *bits32;
391 const MemoryMapParams *bits64;
392};
393
394} // end anonymous namespace
395
396// i386 Linux
397static const MemoryMapParams Linux_I386_MemoryMapParams = {
398 0x000080000000, // AndMask
399 0, // XorMask (not used)
400 0, // ShadowBase (not used)
401 0x000040000000, // OriginBase
402};
403
404// x86_64 Linux
405static const MemoryMapParams Linux_X86_64_MemoryMapParams = {
406 0, // AndMask (not used)
407 0x500000000000, // XorMask
408 0, // ShadowBase (not used)
409 0x100000000000, // OriginBase
410};
411
412// mips64 Linux
413static const MemoryMapParams Linux_MIPS64_MemoryMapParams = {
414 0, // AndMask (not used)
415 0x008000000000, // XorMask
416 0, // ShadowBase (not used)
417 0x002000000000, // OriginBase
418};
419
420// ppc64 Linux
421static const MemoryMapParams Linux_PowerPC64_MemoryMapParams = {
422 0xE00000000000, // AndMask
423 0x100000000000, // XorMask
424 0x080000000000, // ShadowBase
425 0x1C0000000000, // OriginBase
426};
427
428// s390x Linux
429static const MemoryMapParams Linux_S390X_MemoryMapParams = {
430 0xC00000000000, // AndMask
431 0, // XorMask (not used)
432 0x080000000000, // ShadowBase
433 0x1C0000000000, // OriginBase
434};
435
436// aarch64 Linux
437static const MemoryMapParams Linux_AArch64_MemoryMapParams = {
438 0, // AndMask (not used)
439 0x0B00000000000, // XorMask
440 0, // ShadowBase (not used)
441 0x0200000000000, // OriginBase
442};
443
444// loongarch64 Linux
445static const MemoryMapParams Linux_LoongArch64_MemoryMapParams = {
446 0, // AndMask (not used)
447 0x500000000000, // XorMask
448 0, // ShadowBase (not used)
449 0x100000000000, // OriginBase
450};
451
452// aarch64 FreeBSD
453static const MemoryMapParams FreeBSD_AArch64_MemoryMapParams = {
454 0x1800000000000, // AndMask
455 0x0400000000000, // XorMask
456 0x0200000000000, // ShadowBase
457 0x0700000000000, // OriginBase
458};
459
460// i386 FreeBSD
461static const MemoryMapParams FreeBSD_I386_MemoryMapParams = {
462 0x000180000000, // AndMask
463 0x000040000000, // XorMask
464 0x000020000000, // ShadowBase
465 0x000700000000, // OriginBase
466};
467
468// x86_64 FreeBSD
469static const MemoryMapParams FreeBSD_X86_64_MemoryMapParams = {
470 0xc00000000000, // AndMask
471 0x200000000000, // XorMask
472 0x100000000000, // ShadowBase
473 0x380000000000, // OriginBase
474};
475
476// x86_64 NetBSD
477static const MemoryMapParams NetBSD_X86_64_MemoryMapParams = {
478 0, // AndMask
479 0x500000000000, // XorMask
480 0, // ShadowBase
481 0x100000000000, // OriginBase
482};
483
484static const PlatformMemoryMapParams Linux_X86_MemoryMapParams = {
487};
488
489static const PlatformMemoryMapParams Linux_MIPS_MemoryMapParams = {
490 nullptr,
492};
493
494static const PlatformMemoryMapParams Linux_PowerPC_MemoryMapParams = {
495 nullptr,
497};
498
499static const PlatformMemoryMapParams Linux_S390_MemoryMapParams = {
500 nullptr,
502};
503
504static const PlatformMemoryMapParams Linux_ARM_MemoryMapParams = {
505 nullptr,
507};
508
509static const PlatformMemoryMapParams Linux_LoongArch_MemoryMapParams = {
510 nullptr,
512};
513
514static const PlatformMemoryMapParams FreeBSD_ARM_MemoryMapParams = {
515 nullptr,
517};
518
519static const PlatformMemoryMapParams FreeBSD_X86_MemoryMapParams = {
522};
523
524static const PlatformMemoryMapParams NetBSD_X86_MemoryMapParams = {
525 nullptr,
527};
528
529namespace {
530
531/// Instrument functions of a module to detect uninitialized reads.
532///
533/// Instantiating MemorySanitizer inserts the msan runtime library API function
534/// declarations into the module if they don't exist already. Instantiating
535/// ensures the __msan_init function is in the list of global constructors for
536/// the module.
537class MemorySanitizer {
538public:
539 MemorySanitizer(Module &M, MemorySanitizerOptions Options)
540 : CompileKernel(Options.Kernel), TrackOrigins(Options.TrackOrigins),
541 Recover(Options.Recover), EagerChecks(Options.EagerChecks) {
542 initializeModule(M);
543 }
544
545 // MSan cannot be moved or copied because of MapParams.
546 MemorySanitizer(MemorySanitizer &&) = delete;
547 MemorySanitizer &operator=(MemorySanitizer &&) = delete;
548 MemorySanitizer(const MemorySanitizer &) = delete;
549 MemorySanitizer &operator=(const MemorySanitizer &) = delete;
550
551 bool sanitizeFunction(Function &F, TargetLibraryInfo &TLI);
552
553private:
554 friend struct MemorySanitizerVisitor;
555 friend struct VarArgHelperBase;
556 friend struct VarArgAMD64Helper;
557 friend struct VarArgMIPS64Helper;
558 friend struct VarArgAArch64Helper;
559 friend struct VarArgPowerPC64Helper;
560 friend struct VarArgSystemZHelper;
561
562 void initializeModule(Module &M);
563 void initializeCallbacks(Module &M, const TargetLibraryInfo &TLI);
564 void createKernelApi(Module &M, const TargetLibraryInfo &TLI);
565 void createUserspaceApi(Module &M, const TargetLibraryInfo &TLI);
566
567 template <typename... ArgsTy>
568 FunctionCallee getOrInsertMsanMetadataFunction(Module &M, StringRef Name,
569 ArgsTy... Args);
570
571 /// True if we're compiling the Linux kernel.
572 bool CompileKernel;
573 /// Track origins (allocation points) of uninitialized values.
574 int TrackOrigins;
575 bool Recover;
576 bool EagerChecks;
577
578 Triple TargetTriple;
579 LLVMContext *C;
580 Type *IntptrTy; ///< Integer type with the size of a ptr in default AS.
581 Type *OriginTy;
582 PointerType *PtrTy; ///< Integer type with the size of a ptr in default AS.
583
584 // XxxTLS variables represent the per-thread state in MSan and per-task state
585 // in KMSAN.
586 // For the userspace these point to thread-local globals. In the kernel land
587 // they point to the members of a per-task struct obtained via a call to
588 // __msan_get_context_state().
589
590 /// Thread-local shadow storage for function parameters.
591 Value *ParamTLS;
592
593 /// Thread-local origin storage for function parameters.
594 Value *ParamOriginTLS;
595
596 /// Thread-local shadow storage for function return value.
597 Value *RetvalTLS;
598
599 /// Thread-local origin storage for function return value.
600 Value *RetvalOriginTLS;
601
602 /// Thread-local shadow storage for in-register va_arg function.
603 Value *VAArgTLS;
604
605 /// Thread-local shadow storage for in-register va_arg function.
606 Value *VAArgOriginTLS;
607
608 /// Thread-local shadow storage for va_arg overflow area.
609 Value *VAArgOverflowSizeTLS;
610
611 /// Are the instrumentation callbacks set up?
612 bool CallbacksInitialized = false;
613
614 /// The run-time callback to print a warning.
615 FunctionCallee WarningFn;
616
617 // These arrays are indexed by log2(AccessSize).
618 FunctionCallee MaybeWarningFn[kNumberOfAccessSizes];
619 FunctionCallee MaybeStoreOriginFn[kNumberOfAccessSizes];
620
621 /// Run-time helper that generates a new origin value for a stack
622 /// allocation.
623 FunctionCallee MsanSetAllocaOriginWithDescriptionFn;
624 // No description version
625 FunctionCallee MsanSetAllocaOriginNoDescriptionFn;
626
627 /// Run-time helper that poisons stack on function entry.
628 FunctionCallee MsanPoisonStackFn;
629
630 /// Run-time helper that records a store (or any event) of an
631 /// uninitialized value and returns an updated origin id encoding this info.
632 FunctionCallee MsanChainOriginFn;
633
634 /// Run-time helper that paints an origin over a region.
635 FunctionCallee MsanSetOriginFn;
636
637 /// MSan runtime replacements for memmove, memcpy and memset.
638 FunctionCallee MemmoveFn, MemcpyFn, MemsetFn;
639
640 /// KMSAN callback for task-local function argument shadow.
641 StructType *MsanContextStateTy;
642 FunctionCallee MsanGetContextStateFn;
643
644 /// Functions for poisoning/unpoisoning local variables
645 FunctionCallee MsanPoisonAllocaFn, MsanUnpoisonAllocaFn;
646
647 /// Pair of shadow/origin pointers.
648 Type *MsanMetadata;
649
650 /// Each of the MsanMetadataPtrXxx functions returns a MsanMetadata.
651 FunctionCallee MsanMetadataPtrForLoadN, MsanMetadataPtrForStoreN;
652 FunctionCallee MsanMetadataPtrForLoad_1_8[4];
653 FunctionCallee MsanMetadataPtrForStore_1_8[4];
654 FunctionCallee MsanInstrumentAsmStoreFn;
655
656 /// Storage for return values of the MsanMetadataPtrXxx functions.
657 Value *MsanMetadataAlloca;
658
659 /// Helper to choose between different MsanMetadataPtrXxx().
660 FunctionCallee getKmsanShadowOriginAccessFn(bool isStore, int size);
661
662 /// Memory map parameters used in application-to-shadow calculation.
663 const MemoryMapParams *MapParams;
664
665 /// Custom memory map parameters used when -msan-shadow-base or
666 // -msan-origin-base is provided.
667 MemoryMapParams CustomMapParams;
668
669 MDNode *ColdCallWeights;
670
671 /// Branch weights for origin store.
672 MDNode *OriginStoreWeights;
673};
674
675void insertModuleCtor(Module &M) {
678 /*InitArgTypes=*/{},
679 /*InitArgs=*/{},
680 // This callback is invoked when the functions are created the first
681 // time. Hook them into the global ctors list in that case:
682 [&](Function *Ctor, FunctionCallee) {
683 if (!ClWithComdat) {
684 appendToGlobalCtors(M, Ctor, 0);
685 return;
686 }
687 Comdat *MsanCtorComdat = M.getOrInsertComdat(kMsanModuleCtorName);
688 Ctor->setComdat(MsanCtorComdat);
689 appendToGlobalCtors(M, Ctor, 0, Ctor);
690 });
691}
692
693template <class T> T getOptOrDefault(const cl::opt<T> &Opt, T Default) {
694 return (Opt.getNumOccurrences() > 0) ? Opt : Default;
695}
696
697} // end anonymous namespace
698
700 bool EagerChecks)
701 : Kernel(getOptOrDefault(ClEnableKmsan, K)),
702 TrackOrigins(getOptOrDefault(ClTrackOrigins, Kernel ? 2 : TO)),
703 Recover(getOptOrDefault(ClKeepGoing, Kernel || R)),
704 EagerChecks(getOptOrDefault(ClEagerChecks, EagerChecks)) {}
705
708 bool Modified = false;
709 if (!Options.Kernel) {
710 insertModuleCtor(M);
711 Modified = true;
712 }
713
714 auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
715 for (Function &F : M) {
716 if (F.empty())
717 continue;
718 MemorySanitizer Msan(*F.getParent(), Options);
719 Modified |=
720 Msan.sanitizeFunction(F, FAM.getResult<TargetLibraryAnalysis>(F));
721 }
722
723 if (!Modified)
724 return PreservedAnalyses::all();
725
727 // GlobalsAA is considered stateless and does not get invalidated unless
728 // explicitly invalidated; PreservedAnalyses::none() is not enough. Sanitizers
729 // make changes that require GlobalsAA to be invalidated.
730 PA.abandon<GlobalsAA>();
731 return PA;
732}
733
735 raw_ostream &OS, function_ref<StringRef(StringRef)> MapClassName2PassName) {
737 OS, MapClassName2PassName);
738 OS << '<';
739 if (Options.Recover)
740 OS << "recover;";
741 if (Options.Kernel)
742 OS << "kernel;";
743 if (Options.EagerChecks)
744 OS << "eager-checks;";
745 OS << "track-origins=" << Options.TrackOrigins;
746 OS << '>';
747}
748
749/// Create a non-const global initialized with the given string.
750///
751/// Creates a writable global for Str so that we can pass it to the
752/// run-time lib. Runtime uses first 4 bytes of the string to store the
753/// frame ID, so the string needs to be mutable.
755 StringRef Str) {
756 Constant *StrConst = ConstantDataArray::getString(M.getContext(), Str);
757 return new GlobalVariable(M, StrConst->getType(), /*isConstant=*/true,
758 GlobalValue::PrivateLinkage, StrConst, "");
759}
760
761template <typename... ArgsTy>
763MemorySanitizer::getOrInsertMsanMetadataFunction(Module &M, StringRef Name,
764 ArgsTy... Args) {
765 if (TargetTriple.getArch() == Triple::systemz) {
766 // SystemZ ABI: shadow/origin pair is returned via a hidden parameter.
767 return M.getOrInsertFunction(Name, Type::getVoidTy(*C),
768 PointerType::get(MsanMetadata, 0),
769 std::forward<ArgsTy>(Args)...);
770 }
771
772 return M.getOrInsertFunction(Name, MsanMetadata,
773 std::forward<ArgsTy>(Args)...);
774}
775
776/// Create KMSAN API callbacks.
777void MemorySanitizer::createKernelApi(Module &M, const TargetLibraryInfo &TLI) {
778 IRBuilder<> IRB(*C);
779
780 // These will be initialized in insertKmsanPrologue().
781 RetvalTLS = nullptr;
782 RetvalOriginTLS = nullptr;
783 ParamTLS = nullptr;
784 ParamOriginTLS = nullptr;
785 VAArgTLS = nullptr;
786 VAArgOriginTLS = nullptr;
787 VAArgOverflowSizeTLS = nullptr;
788
789 WarningFn = M.getOrInsertFunction("__msan_warning",
790 TLI.getAttrList(C, {0}, /*Signed=*/false),
791 IRB.getVoidTy(), IRB.getInt32Ty());
792
793 // Requests the per-task context state (kmsan_context_state*) from the
794 // runtime library.
795 MsanContextStateTy = StructType::get(
796 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
797 ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8),
798 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
799 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8), /* va_arg_origin */
800 IRB.getInt64Ty(), ArrayType::get(OriginTy, kParamTLSSize / 4), OriginTy,
801 OriginTy);
802 MsanGetContextStateFn = M.getOrInsertFunction(
803 "__msan_get_context_state", PointerType::get(MsanContextStateTy, 0));
804
805 MsanMetadata = StructType::get(PointerType::get(IRB.getInt8Ty(), 0),
806 PointerType::get(IRB.getInt32Ty(), 0));
807
808 for (int ind = 0, size = 1; ind < 4; ind++, size <<= 1) {
809 std::string name_load =
810 "__msan_metadata_ptr_for_load_" + std::to_string(size);
811 std::string name_store =
812 "__msan_metadata_ptr_for_store_" + std::to_string(size);
813 MsanMetadataPtrForLoad_1_8[ind] = getOrInsertMsanMetadataFunction(
814 M, name_load, PointerType::get(IRB.getInt8Ty(), 0));
815 MsanMetadataPtrForStore_1_8[ind] = getOrInsertMsanMetadataFunction(
816 M, name_store, PointerType::get(IRB.getInt8Ty(), 0));
817 }
818
819 MsanMetadataPtrForLoadN = getOrInsertMsanMetadataFunction(
820 M, "__msan_metadata_ptr_for_load_n", PointerType::get(IRB.getInt8Ty(), 0),
821 IRB.getInt64Ty());
822 MsanMetadataPtrForStoreN = getOrInsertMsanMetadataFunction(
823 M, "__msan_metadata_ptr_for_store_n",
824 PointerType::get(IRB.getInt8Ty(), 0), IRB.getInt64Ty());
825
826 // Functions for poisoning and unpoisoning memory.
827 MsanPoisonAllocaFn = M.getOrInsertFunction(
828 "__msan_poison_alloca", IRB.getVoidTy(), PtrTy, IntptrTy, PtrTy);
829 MsanUnpoisonAllocaFn = M.getOrInsertFunction(
830 "__msan_unpoison_alloca", IRB.getVoidTy(), PtrTy, IntptrTy);
831}
832
834 return M.getOrInsertGlobal(Name, Ty, [&] {
835 return new GlobalVariable(M, Ty, false, GlobalVariable::ExternalLinkage,
836 nullptr, Name, nullptr,
838 });
839}
840
841/// Insert declarations for userspace-specific functions and globals.
842void MemorySanitizer::createUserspaceApi(Module &M, const TargetLibraryInfo &TLI) {
843 IRBuilder<> IRB(*C);
844
845 // Create the callback.
846 // FIXME: this function should have "Cold" calling conv,
847 // which is not yet implemented.
848 if (TrackOrigins) {
849 StringRef WarningFnName = Recover ? "__msan_warning_with_origin"
850 : "__msan_warning_with_origin_noreturn";
851 WarningFn = M.getOrInsertFunction(WarningFnName,
852 TLI.getAttrList(C, {0}, /*Signed=*/false),
853 IRB.getVoidTy(), IRB.getInt32Ty());
854 } else {
855 StringRef WarningFnName =
856 Recover ? "__msan_warning" : "__msan_warning_noreturn";
857 WarningFn = M.getOrInsertFunction(WarningFnName, IRB.getVoidTy());
858 }
859
860 // Create the global TLS variables.
861 RetvalTLS =
862 getOrInsertGlobal(M, "__msan_retval_tls",
863 ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8));
864
865 RetvalOriginTLS = getOrInsertGlobal(M, "__msan_retval_origin_tls", OriginTy);
866
867 ParamTLS =
868 getOrInsertGlobal(M, "__msan_param_tls",
869 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
870
871 ParamOriginTLS =
872 getOrInsertGlobal(M, "__msan_param_origin_tls",
873 ArrayType::get(OriginTy, kParamTLSSize / 4));
874
875 VAArgTLS =
876 getOrInsertGlobal(M, "__msan_va_arg_tls",
877 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
878
879 VAArgOriginTLS =
880 getOrInsertGlobal(M, "__msan_va_arg_origin_tls",
881 ArrayType::get(OriginTy, kParamTLSSize / 4));
882
883 VAArgOverflowSizeTLS =
884 getOrInsertGlobal(M, "__msan_va_arg_overflow_size_tls", IRB.getInt64Ty());
885
886 for (size_t AccessSizeIndex = 0; AccessSizeIndex < kNumberOfAccessSizes;
887 AccessSizeIndex++) {
888 unsigned AccessSize = 1 << AccessSizeIndex;
889 std::string FunctionName = "__msan_maybe_warning_" + itostr(AccessSize);
890 MaybeWarningFn[AccessSizeIndex] = M.getOrInsertFunction(
891 FunctionName, TLI.getAttrList(C, {0, 1}, /*Signed=*/false),
892 IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), IRB.getInt32Ty());
893
894 FunctionName = "__msan_maybe_store_origin_" + itostr(AccessSize);
895 MaybeStoreOriginFn[AccessSizeIndex] = M.getOrInsertFunction(
896 FunctionName, TLI.getAttrList(C, {0, 2}, /*Signed=*/false),
897 IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), PtrTy,
898 IRB.getInt32Ty());
899 }
900
901 MsanSetAllocaOriginWithDescriptionFn =
902 M.getOrInsertFunction("__msan_set_alloca_origin_with_descr",
903 IRB.getVoidTy(), PtrTy, IntptrTy, PtrTy, PtrTy);
904 MsanSetAllocaOriginNoDescriptionFn =
905 M.getOrInsertFunction("__msan_set_alloca_origin_no_descr",
906 IRB.getVoidTy(), PtrTy, IntptrTy, PtrTy);
907 MsanPoisonStackFn = M.getOrInsertFunction("__msan_poison_stack",
908 IRB.getVoidTy(), PtrTy, IntptrTy);
909}
910
911/// Insert extern declaration of runtime-provided functions and globals.
912void MemorySanitizer::initializeCallbacks(Module &M, const TargetLibraryInfo &TLI) {
913 // Only do this once.
914 if (CallbacksInitialized)
915 return;
916
917 IRBuilder<> IRB(*C);
918 // Initialize callbacks that are common for kernel and userspace
919 // instrumentation.
920 MsanChainOriginFn = M.getOrInsertFunction(
921 "__msan_chain_origin",
922 TLI.getAttrList(C, {0}, /*Signed=*/false, /*Ret=*/true), IRB.getInt32Ty(),
923 IRB.getInt32Ty());
924 MsanSetOriginFn = M.getOrInsertFunction(
925 "__msan_set_origin", TLI.getAttrList(C, {2}, /*Signed=*/false),
926 IRB.getVoidTy(), PtrTy, IntptrTy, IRB.getInt32Ty());
927 MemmoveFn =
928 M.getOrInsertFunction("__msan_memmove", PtrTy, PtrTy, PtrTy, IntptrTy);
929 MemcpyFn =
930 M.getOrInsertFunction("__msan_memcpy", PtrTy, PtrTy, PtrTy, IntptrTy);
931 MemsetFn = M.getOrInsertFunction("__msan_memset",
932 TLI.getAttrList(C, {1}, /*Signed=*/true),
933 PtrTy, PtrTy, IRB.getInt32Ty(), IntptrTy);
934
935 MsanInstrumentAsmStoreFn =
936 M.getOrInsertFunction("__msan_instrument_asm_store", IRB.getVoidTy(),
937 PointerType::get(IRB.getInt8Ty(), 0), IntptrTy);
938
939 if (CompileKernel) {
940 createKernelApi(M, TLI);
941 } else {
942 createUserspaceApi(M, TLI);
943 }
944 CallbacksInitialized = true;
945}
946
947FunctionCallee MemorySanitizer::getKmsanShadowOriginAccessFn(bool isStore,
948 int size) {
949 FunctionCallee *Fns =
950 isStore ? MsanMetadataPtrForStore_1_8 : MsanMetadataPtrForLoad_1_8;
951 switch (size) {
952 case 1:
953 return Fns[0];
954 case 2:
955 return Fns[1];
956 case 4:
957 return Fns[2];
958 case 8:
959 return Fns[3];
960 default:
961 return nullptr;
962 }
963}
964
965/// Module-level initialization.
966///
967/// inserts a call to __msan_init to the module's constructor list.
968void MemorySanitizer::initializeModule(Module &M) {
969 auto &DL = M.getDataLayout();
970
971 TargetTriple = Triple(M.getTargetTriple());
972
973 bool ShadowPassed = ClShadowBase.getNumOccurrences() > 0;
974 bool OriginPassed = ClOriginBase.getNumOccurrences() > 0;
975 // Check the overrides first
976 if (ShadowPassed || OriginPassed) {
977 CustomMapParams.AndMask = ClAndMask;
978 CustomMapParams.XorMask = ClXorMask;
979 CustomMapParams.ShadowBase = ClShadowBase;
980 CustomMapParams.OriginBase = ClOriginBase;
981 MapParams = &CustomMapParams;
982 } else {
983 switch (TargetTriple.getOS()) {
984 case Triple::FreeBSD:
985 switch (TargetTriple.getArch()) {
986 case Triple::aarch64:
987 MapParams = FreeBSD_ARM_MemoryMapParams.bits64;
988 break;
989 case Triple::x86_64:
990 MapParams = FreeBSD_X86_MemoryMapParams.bits64;
991 break;
992 case Triple::x86:
993 MapParams = FreeBSD_X86_MemoryMapParams.bits32;
994 break;
995 default:
996 report_fatal_error("unsupported architecture");
997 }
998 break;
999 case Triple::NetBSD:
1000 switch (TargetTriple.getArch()) {
1001 case Triple::x86_64:
1002 MapParams = NetBSD_X86_MemoryMapParams.bits64;
1003 break;
1004 default:
1005 report_fatal_error("unsupported architecture");
1006 }
1007 break;
1008 case Triple::Linux:
1009 switch (TargetTriple.getArch()) {
1010 case Triple::x86_64:
1011 MapParams = Linux_X86_MemoryMapParams.bits64;
1012 break;
1013 case Triple::x86:
1014 MapParams = Linux_X86_MemoryMapParams.bits32;
1015 break;
1016 case Triple::mips64:
1017 case Triple::mips64el:
1018 MapParams = Linux_MIPS_MemoryMapParams.bits64;
1019 break;
1020 case Triple::ppc64:
1021 case Triple::ppc64le:
1022 MapParams = Linux_PowerPC_MemoryMapParams.bits64;
1023 break;
1024 case Triple::systemz:
1025 MapParams = Linux_S390_MemoryMapParams.bits64;
1026 break;
1027 case Triple::aarch64:
1028 case Triple::aarch64_be:
1029 MapParams = Linux_ARM_MemoryMapParams.bits64;
1030 break;
1032 MapParams = Linux_LoongArch_MemoryMapParams.bits64;
1033 break;
1034 default:
1035 report_fatal_error("unsupported architecture");
1036 }
1037 break;
1038 default:
1039 report_fatal_error("unsupported operating system");
1040 }
1041 }
1042
1043 C = &(M.getContext());
1044 IRBuilder<> IRB(*C);
1045 IntptrTy = IRB.getIntPtrTy(DL);
1046 OriginTy = IRB.getInt32Ty();
1047 PtrTy = IRB.getPtrTy();
1048
1049 ColdCallWeights = MDBuilder(*C).createUnlikelyBranchWeights();
1050 OriginStoreWeights = MDBuilder(*C).createUnlikelyBranchWeights();
1051
1052 if (!CompileKernel) {
1053 if (TrackOrigins)
1054 M.getOrInsertGlobal("__msan_track_origins", IRB.getInt32Ty(), [&] {
1055 return new GlobalVariable(
1056 M, IRB.getInt32Ty(), true, GlobalValue::WeakODRLinkage,
1057 IRB.getInt32(TrackOrigins), "__msan_track_origins");
1058 });
1059
1060 if (Recover)
1061 M.getOrInsertGlobal("__msan_keep_going", IRB.getInt32Ty(), [&] {
1062 return new GlobalVariable(M, IRB.getInt32Ty(), true,
1063 GlobalValue::WeakODRLinkage,
1064 IRB.getInt32(Recover), "__msan_keep_going");
1065 });
1066 }
1067}
1068
1069namespace {
1070
1071/// A helper class that handles instrumentation of VarArg
1072/// functions on a particular platform.
1073///
1074/// Implementations are expected to insert the instrumentation
1075/// necessary to propagate argument shadow through VarArg function
1076/// calls. Visit* methods are called during an InstVisitor pass over
1077/// the function, and should avoid creating new basic blocks. A new
1078/// instance of this class is created for each instrumented function.
1079struct VarArgHelper {
1080 virtual ~VarArgHelper() = default;
1081
1082 /// Visit a CallBase.
1083 virtual void visitCallBase(CallBase &CB, IRBuilder<> &IRB) = 0;
1084
1085 /// Visit a va_start call.
1086 virtual void visitVAStartInst(VAStartInst &I) = 0;
1087
1088 /// Visit a va_copy call.
1089 virtual void visitVACopyInst(VACopyInst &I) = 0;
1090
1091 /// Finalize function instrumentation.
1092 ///
1093 /// This method is called after visiting all interesting (see above)
1094 /// instructions in a function.
1095 virtual void finalizeInstrumentation() = 0;
1096};
1097
1098struct MemorySanitizerVisitor;
1099
1100} // end anonymous namespace
1101
1102static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
1103 MemorySanitizerVisitor &Visitor);
1104
1105static unsigned TypeSizeToSizeIndex(TypeSize TS) {
1106 if (TS.isScalable())
1107 // Scalable types unconditionally take slowpaths.
1108 return kNumberOfAccessSizes;
1109 unsigned TypeSizeFixed = TS.getFixedValue();
1110 if (TypeSizeFixed <= 8)
1111 return 0;
1112 return Log2_32_Ceil((TypeSizeFixed + 7) / 8);
1113}
1114
1115namespace {
1116
1117/// Helper class to attach debug information of the given instruction onto new
1118/// instructions inserted after.
1119class NextNodeIRBuilder : public IRBuilder<> {
1120public:
1121 explicit NextNodeIRBuilder(Instruction *IP) : IRBuilder<>(IP->getNextNode()) {
1122 SetCurrentDebugLocation(IP->getDebugLoc());
1123 }
1124};
1125
1126/// This class does all the work for a given function. Store and Load
1127/// instructions store and load corresponding shadow and origin
1128/// values. Most instructions propagate shadow from arguments to their
1129/// return values. Certain instructions (most importantly, BranchInst)
1130/// test their argument shadow and print reports (with a runtime call) if it's
1131/// non-zero.
1132struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
1133 Function &F;
1134 MemorySanitizer &MS;
1135 SmallVector<PHINode *, 16> ShadowPHINodes, OriginPHINodes;
1136 ValueMap<Value *, Value *> ShadowMap, OriginMap;
1137 std::unique_ptr<VarArgHelper> VAHelper;
1138 const TargetLibraryInfo *TLI;
1139 Instruction *FnPrologueEnd;
1141
1142 // The following flags disable parts of MSan instrumentation based on
1143 // exclusion list contents and command-line options.
1144 bool InsertChecks;
1145 bool PropagateShadow;
1146 bool PoisonStack;
1147 bool PoisonUndef;
1148
1149 struct ShadowOriginAndInsertPoint {
1150 Value *Shadow;
1151 Value *Origin;
1152 Instruction *OrigIns;
1153
1154 ShadowOriginAndInsertPoint(Value *S, Value *O, Instruction *I)
1155 : Shadow(S), Origin(O), OrigIns(I) {}
1156 };
1158 DenseMap<const DILocation *, int> LazyWarningDebugLocationCount;
1159 bool InstrumentLifetimeStart = ClHandleLifetimeIntrinsics;
1163 int64_t SplittableBlocksCount = 0;
1164
1165 MemorySanitizerVisitor(Function &F, MemorySanitizer &MS,
1166 const TargetLibraryInfo &TLI)
1167 : F(F), MS(MS), VAHelper(CreateVarArgHelper(F, MS, *this)), TLI(&TLI) {
1168 bool SanitizeFunction =
1169 F.hasFnAttribute(Attribute::SanitizeMemory) && !ClDisableChecks;
1170 InsertChecks = SanitizeFunction;
1171 PropagateShadow = SanitizeFunction;
1172 PoisonStack = SanitizeFunction && ClPoisonStack;
1173 PoisonUndef = SanitizeFunction && ClPoisonUndef;
1174
1175 // In the presence of unreachable blocks, we may see Phi nodes with
1176 // incoming nodes from such blocks. Since InstVisitor skips unreachable
1177 // blocks, such nodes will not have any shadow value associated with them.
1178 // It's easier to remove unreachable blocks than deal with missing shadow.
1180
1181 MS.initializeCallbacks(*F.getParent(), TLI);
1182 FnPrologueEnd = IRBuilder<>(F.getEntryBlock().getFirstNonPHI())
1183 .CreateIntrinsic(Intrinsic::donothing, {}, {});
1184
1185 if (MS.CompileKernel) {
1186 IRBuilder<> IRB(FnPrologueEnd);
1187 insertKmsanPrologue(IRB);
1188 }
1189
1190 LLVM_DEBUG(if (!InsertChecks) dbgs()
1191 << "MemorySanitizer is not inserting checks into '"
1192 << F.getName() << "'\n");
1193 }
1194
1195 bool instrumentWithCalls(Value *V) {
1196 // Constants likely will be eliminated by follow-up passes.
1197 if (isa<Constant>(V))
1198 return false;
1199
1200 ++SplittableBlocksCount;
1202 SplittableBlocksCount > ClInstrumentationWithCallThreshold;
1203 }
1204
1205 bool isInPrologue(Instruction &I) {
1206 return I.getParent() == FnPrologueEnd->getParent() &&
1207 (&I == FnPrologueEnd || I.comesBefore(FnPrologueEnd));
1208 }
1209
1210 // Creates a new origin and records the stack trace. In general we can call
1211 // this function for any origin manipulation we like. However it will cost
1212 // runtime resources. So use this wisely only if it can provide additional
1213 // information helpful to a user.
1214 Value *updateOrigin(Value *V, IRBuilder<> &IRB) {
1215 if (MS.TrackOrigins <= 1)
1216 return V;
1217 return IRB.CreateCall(MS.MsanChainOriginFn, V);
1218 }
1219
1220 Value *originToIntptr(IRBuilder<> &IRB, Value *Origin) {
1221 const DataLayout &DL = F.getDataLayout();
1222 unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1223 if (IntptrSize == kOriginSize)
1224 return Origin;
1225 assert(IntptrSize == kOriginSize * 2);
1226 Origin = IRB.CreateIntCast(Origin, MS.IntptrTy, /* isSigned */ false);
1227 return IRB.CreateOr(Origin, IRB.CreateShl(Origin, kOriginSize * 8));
1228 }
1229
1230 /// Fill memory range with the given origin value.
1231 void paintOrigin(IRBuilder<> &IRB, Value *Origin, Value *OriginPtr,
1232 TypeSize TS, Align Alignment) {
1233 const DataLayout &DL = F.getDataLayout();
1234 const Align IntptrAlignment = DL.getABITypeAlign(MS.IntptrTy);
1235 unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1236 assert(IntptrAlignment >= kMinOriginAlignment);
1237 assert(IntptrSize >= kOriginSize);
1238
1239 // Note: The loop based formation works for fixed length vectors too,
1240 // however we prefer to unroll and specialize alignment below.
1241 if (TS.isScalable()) {
1242 Value *Size = IRB.CreateTypeSize(MS.IntptrTy, TS);
1243 Value *RoundUp =
1244 IRB.CreateAdd(Size, ConstantInt::get(MS.IntptrTy, kOriginSize - 1));
1245 Value *End =
1246 IRB.CreateUDiv(RoundUp, ConstantInt::get(MS.IntptrTy, kOriginSize));
1247 auto [InsertPt, Index] =
1249 IRB.SetInsertPoint(InsertPt);
1250
1251 Value *GEP = IRB.CreateGEP(MS.OriginTy, OriginPtr, Index);
1253 return;
1254 }
1255
1256 unsigned Size = TS.getFixedValue();
1257
1258 unsigned Ofs = 0;
1259 Align CurrentAlignment = Alignment;
1260 if (Alignment >= IntptrAlignment && IntptrSize > kOriginSize) {
1261 Value *IntptrOrigin = originToIntptr(IRB, Origin);
1262 Value *IntptrOriginPtr =
1263 IRB.CreatePointerCast(OriginPtr, PointerType::get(MS.IntptrTy, 0));
1264 for (unsigned i = 0; i < Size / IntptrSize; ++i) {
1265 Value *Ptr = i ? IRB.CreateConstGEP1_32(MS.IntptrTy, IntptrOriginPtr, i)
1266 : IntptrOriginPtr;
1267 IRB.CreateAlignedStore(IntptrOrigin, Ptr, CurrentAlignment);
1268 Ofs += IntptrSize / kOriginSize;
1269 CurrentAlignment = IntptrAlignment;
1270 }
1271 }
1272
1273 for (unsigned i = Ofs; i < (Size + kOriginSize - 1) / kOriginSize; ++i) {
1274 Value *GEP =
1275 i ? IRB.CreateConstGEP1_32(MS.OriginTy, OriginPtr, i) : OriginPtr;
1276 IRB.CreateAlignedStore(Origin, GEP, CurrentAlignment);
1277 CurrentAlignment = kMinOriginAlignment;
1278 }
1279 }
1280
1281 void storeOrigin(IRBuilder<> &IRB, Value *Addr, Value *Shadow, Value *Origin,
1282 Value *OriginPtr, Align Alignment) {
1283 const DataLayout &DL = F.getDataLayout();
1284 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1285 TypeSize StoreSize = DL.getTypeStoreSize(Shadow->getType());
1286 // ZExt cannot convert between vector and scalar
1287 Value *ConvertedShadow = convertShadowToScalar(Shadow, IRB);
1288 if (auto *ConstantShadow = dyn_cast<Constant>(ConvertedShadow)) {
1289 if (!ClCheckConstantShadow || ConstantShadow->isZeroValue()) {
1290 // Origin is not needed: value is initialized or const shadow is
1291 // ignored.
1292 return;
1293 }
1294 if (llvm::isKnownNonZero(ConvertedShadow, DL)) {
1295 // Copy origin as the value is definitely uninitialized.
1296 paintOrigin(IRB, updateOrigin(Origin, IRB), OriginPtr, StoreSize,
1297 OriginAlignment);
1298 return;
1299 }
1300 // Fallback to runtime check, which still can be optimized out later.
1301 }
1302
1303 TypeSize TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
1304 unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1305 if (instrumentWithCalls(ConvertedShadow) &&
1306 SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
1307 FunctionCallee Fn = MS.MaybeStoreOriginFn[SizeIndex];
1308 Value *ConvertedShadow2 =
1309 IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1310 CallBase *CB = IRB.CreateCall(Fn, {ConvertedShadow2, Addr, Origin});
1311 CB->addParamAttr(0, Attribute::ZExt);
1312 CB->addParamAttr(2, Attribute::ZExt);
1313 } else {
1314 Value *Cmp = convertToBool(ConvertedShadow, IRB, "_mscmp");
1316 Cmp, &*IRB.GetInsertPoint(), false, MS.OriginStoreWeights);
1317 IRBuilder<> IRBNew(CheckTerm);
1318 paintOrigin(IRBNew, updateOrigin(Origin, IRBNew), OriginPtr, StoreSize,
1319 OriginAlignment);
1320 }
1321 }
1322
1323 void materializeStores() {
1324 for (StoreInst *SI : StoreList) {
1325 IRBuilder<> IRB(SI);
1326 Value *Val = SI->getValueOperand();
1327 Value *Addr = SI->getPointerOperand();
1328 Value *Shadow = SI->isAtomic() ? getCleanShadow(Val) : getShadow(Val);
1329 Value *ShadowPtr, *OriginPtr;
1330 Type *ShadowTy = Shadow->getType();
1331 const Align Alignment = SI->getAlign();
1332 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1333 std::tie(ShadowPtr, OriginPtr) =
1334 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ true);
1335
1336 StoreInst *NewSI = IRB.CreateAlignedStore(Shadow, ShadowPtr, Alignment);
1337 LLVM_DEBUG(dbgs() << " STORE: " << *NewSI << "\n");
1338 (void)NewSI;
1339
1340 if (SI->isAtomic())
1341 SI->setOrdering(addReleaseOrdering(SI->getOrdering()));
1342
1343 if (MS.TrackOrigins && !SI->isAtomic())
1344 storeOrigin(IRB, Addr, Shadow, getOrigin(Val), OriginPtr,
1345 OriginAlignment);
1346 }
1347 }
1348
1349 // Returns true if Debug Location corresponds to multiple warnings.
1350 bool shouldDisambiguateWarningLocation(const DebugLoc &DebugLoc) {
1351 if (MS.TrackOrigins < 2)
1352 return false;
1353
1354 if (LazyWarningDebugLocationCount.empty())
1355 for (const auto &I : InstrumentationList)
1356 ++LazyWarningDebugLocationCount[I.OrigIns->getDebugLoc()];
1357
1358 return LazyWarningDebugLocationCount[DebugLoc] >= ClDisambiguateWarning;
1359 }
1360
1361 /// Helper function to insert a warning at IRB's current insert point.
1362 void insertWarningFn(IRBuilder<> &IRB, Value *Origin) {
1363 if (!Origin)
1364 Origin = (Value *)IRB.getInt32(0);
1365 assert(Origin->getType()->isIntegerTy());
1366
1367 if (shouldDisambiguateWarningLocation(IRB.getCurrentDebugLocation())) {
1368 // Try to create additional origin with debug info of the last origin
1369 // instruction. It may provide additional information to the user.
1370 if (Instruction *OI = dyn_cast_or_null<Instruction>(Origin)) {
1371 assert(MS.TrackOrigins);
1372 auto NewDebugLoc = OI->getDebugLoc();
1373 // Origin update with missing or the same debug location provides no
1374 // additional value.
1375 if (NewDebugLoc && NewDebugLoc != IRB.getCurrentDebugLocation()) {
1376 // Insert update just before the check, so we call runtime only just
1377 // before the report.
1378 IRBuilder<> IRBOrigin(&*IRB.GetInsertPoint());
1379 IRBOrigin.SetCurrentDebugLocation(NewDebugLoc);
1380 Origin = updateOrigin(Origin, IRBOrigin);
1381 }
1382 }
1383 }
1384
1385 if (MS.CompileKernel || MS.TrackOrigins)
1386 IRB.CreateCall(MS.WarningFn, Origin)->setCannotMerge();
1387 else
1388 IRB.CreateCall(MS.WarningFn)->setCannotMerge();
1389 // FIXME: Insert UnreachableInst if !MS.Recover?
1390 // This may invalidate some of the following checks and needs to be done
1391 // at the very end.
1392 }
1393
1394 void materializeOneCheck(IRBuilder<> &IRB, Value *ConvertedShadow,
1395 Value *Origin) {
1396 const DataLayout &DL = F.getDataLayout();
1397 TypeSize TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
1398 unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1399 if (instrumentWithCalls(ConvertedShadow) &&
1400 SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
1401 FunctionCallee Fn = MS.MaybeWarningFn[SizeIndex];
1402 // ZExt cannot convert between vector and scalar
1403 ConvertedShadow = convertShadowToScalar(ConvertedShadow, IRB);
1404 Value *ConvertedShadow2 =
1405 IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1406 CallBase *CB = IRB.CreateCall(
1407 Fn, {ConvertedShadow2,
1408 MS.TrackOrigins && Origin ? Origin : (Value *)IRB.getInt32(0)});
1409 CB->addParamAttr(0, Attribute::ZExt);
1410 CB->addParamAttr(1, Attribute::ZExt);
1411 } else {
1412 Value *Cmp = convertToBool(ConvertedShadow, IRB, "_mscmp");
1414 Cmp, &*IRB.GetInsertPoint(),
1415 /* Unreachable */ !MS.Recover, MS.ColdCallWeights);
1416
1417 IRB.SetInsertPoint(CheckTerm);
1418 insertWarningFn(IRB, Origin);
1419 LLVM_DEBUG(dbgs() << " CHECK: " << *Cmp << "\n");
1420 }
1421 }
1422
1423 void materializeInstructionChecks(
1424 ArrayRef<ShadowOriginAndInsertPoint> InstructionChecks) {
1425 const DataLayout &DL = F.getDataLayout();
1426 // Disable combining in some cases. TrackOrigins checks each shadow to pick
1427 // correct origin.
1428 bool Combine = !MS.TrackOrigins;
1429 Instruction *Instruction = InstructionChecks.front().OrigIns;
1430 Value *Shadow = nullptr;
1431 for (const auto &ShadowData : InstructionChecks) {
1432 assert(ShadowData.OrigIns == Instruction);
1434
1435 Value *ConvertedShadow = ShadowData.Shadow;
1436
1437 if (auto *ConstantShadow = dyn_cast<Constant>(ConvertedShadow)) {
1438 if (!ClCheckConstantShadow || ConstantShadow->isZeroValue()) {
1439 // Skip, value is initialized or const shadow is ignored.
1440 continue;
1441 }
1442 if (llvm::isKnownNonZero(ConvertedShadow, DL)) {
1443 // Report as the value is definitely uninitialized.
1444 insertWarningFn(IRB, ShadowData.Origin);
1445 if (!MS.Recover)
1446 return; // Always fail and stop here, not need to check the rest.
1447 // Skip entire instruction,
1448 continue;
1449 }
1450 // Fallback to runtime check, which still can be optimized out later.
1451 }
1452
1453 if (!Combine) {
1454 materializeOneCheck(IRB, ConvertedShadow, ShadowData.Origin);
1455 continue;
1456 }
1457
1458 if (!Shadow) {
1459 Shadow = ConvertedShadow;
1460 continue;
1461 }
1462
1463 Shadow = convertToBool(Shadow, IRB, "_mscmp");
1464 ConvertedShadow = convertToBool(ConvertedShadow, IRB, "_mscmp");
1465 Shadow = IRB.CreateOr(Shadow, ConvertedShadow, "_msor");
1466 }
1467
1468 if (Shadow) {
1469 assert(Combine);
1471 materializeOneCheck(IRB, Shadow, nullptr);
1472 }
1473 }
1474
1475 void materializeChecks() {
1476#ifndef NDEBUG
1477 // For assert below.
1479#endif
1480
1481 for (auto I = InstrumentationList.begin();
1482 I != InstrumentationList.end();) {
1483 auto OrigIns = I->OrigIns;
1484 // Checks are grouped by the original instruction. We call all
1485 // `insertShadowCheck` for an instruction at once.
1486 assert(Done.insert(OrigIns).second);
1487 auto J = std::find_if(I + 1, InstrumentationList.end(),
1488 [OrigIns](const ShadowOriginAndInsertPoint &R) {
1489 return OrigIns != R.OrigIns;
1490 });
1491 // Process all checks of instruction at once.
1492 materializeInstructionChecks(ArrayRef<ShadowOriginAndInsertPoint>(I, J));
1493 I = J;
1494 }
1495
1496 LLVM_DEBUG(dbgs() << "DONE:\n" << F);
1497 }
1498
1499 // Returns the last instruction in the new prologue
1500 void insertKmsanPrologue(IRBuilder<> &IRB) {
1501 Value *ContextState = IRB.CreateCall(MS.MsanGetContextStateFn, {});
1502 Constant *Zero = IRB.getInt32(0);
1503 MS.ParamTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1504 {Zero, IRB.getInt32(0)}, "param_shadow");
1505 MS.RetvalTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1506 {Zero, IRB.getInt32(1)}, "retval_shadow");
1507 MS.VAArgTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1508 {Zero, IRB.getInt32(2)}, "va_arg_shadow");
1509 MS.VAArgOriginTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1510 {Zero, IRB.getInt32(3)}, "va_arg_origin");
1511 MS.VAArgOverflowSizeTLS =
1512 IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1513 {Zero, IRB.getInt32(4)}, "va_arg_overflow_size");
1514 MS.ParamOriginTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1515 {Zero, IRB.getInt32(5)}, "param_origin");
1516 MS.RetvalOriginTLS =
1517 IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1518 {Zero, IRB.getInt32(6)}, "retval_origin");
1519 if (MS.TargetTriple.getArch() == Triple::systemz)
1520 MS.MsanMetadataAlloca = IRB.CreateAlloca(MS.MsanMetadata, 0u);
1521 }
1522
1523 /// Add MemorySanitizer instrumentation to a function.
1524 bool runOnFunction() {
1525 // Iterate all BBs in depth-first order and create shadow instructions
1526 // for all instructions (where applicable).
1527 // For PHI nodes we create dummy shadow PHIs which will be finalized later.
1528 for (BasicBlock *BB : depth_first(FnPrologueEnd->getParent()))
1529 visit(*BB);
1530
1531 // `visit` above only collects instructions. Process them after iterating
1532 // CFG to avoid requirement on CFG transformations.
1533 for (Instruction *I : Instructions)
1535
1536 // Finalize PHI nodes.
1537 for (PHINode *PN : ShadowPHINodes) {
1538 PHINode *PNS = cast<PHINode>(getShadow(PN));
1539 PHINode *PNO = MS.TrackOrigins ? cast<PHINode>(getOrigin(PN)) : nullptr;
1540 size_t NumValues = PN->getNumIncomingValues();
1541 for (size_t v = 0; v < NumValues; v++) {
1542 PNS->addIncoming(getShadow(PN, v), PN->getIncomingBlock(v));
1543 if (PNO)
1544 PNO->addIncoming(getOrigin(PN, v), PN->getIncomingBlock(v));
1545 }
1546 }
1547
1548 VAHelper->finalizeInstrumentation();
1549
1550 // Poison llvm.lifetime.start intrinsics, if we haven't fallen back to
1551 // instrumenting only allocas.
1552 if (InstrumentLifetimeStart) {
1553 for (auto Item : LifetimeStartList) {
1554 instrumentAlloca(*Item.second, Item.first);
1555 AllocaSet.remove(Item.second);
1556 }
1557 }
1558 // Poison the allocas for which we didn't instrument the corresponding
1559 // lifetime intrinsics.
1560 for (AllocaInst *AI : AllocaSet)
1561 instrumentAlloca(*AI);
1562
1563 // Insert shadow value checks.
1564 materializeChecks();
1565
1566 // Delayed instrumentation of StoreInst.
1567 // This may not add new address checks.
1568 materializeStores();
1569
1570 return true;
1571 }
1572
1573 /// Compute the shadow type that corresponds to a given Value.
1574 Type *getShadowTy(Value *V) { return getShadowTy(V->getType()); }
1575
1576 /// Compute the shadow type that corresponds to a given Type.
1577 Type *getShadowTy(Type *OrigTy) {
1578 if (!OrigTy->isSized()) {
1579 return nullptr;
1580 }
1581 // For integer type, shadow is the same as the original type.
1582 // This may return weird-sized types like i1.
1583 if (IntegerType *IT = dyn_cast<IntegerType>(OrigTy))
1584 return IT;
1585 const DataLayout &DL = F.getDataLayout();
1586 if (VectorType *VT = dyn_cast<VectorType>(OrigTy)) {
1587 uint32_t EltSize = DL.getTypeSizeInBits(VT->getElementType());
1588 return VectorType::get(IntegerType::get(*MS.C, EltSize),
1589 VT->getElementCount());
1590 }
1591 if (ArrayType *AT = dyn_cast<ArrayType>(OrigTy)) {
1592 return ArrayType::get(getShadowTy(AT->getElementType()),
1593 AT->getNumElements());
1594 }
1595 if (StructType *ST = dyn_cast<StructType>(OrigTy)) {
1597 for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
1598 Elements.push_back(getShadowTy(ST->getElementType(i)));
1599 StructType *Res = StructType::get(*MS.C, Elements, ST->isPacked());
1600 LLVM_DEBUG(dbgs() << "getShadowTy: " << *ST << " ===> " << *Res << "\n");
1601 return Res;
1602 }
1603 uint32_t TypeSize = DL.getTypeSizeInBits(OrigTy);
1604 return IntegerType::get(*MS.C, TypeSize);
1605 }
1606
1607 /// Extract combined shadow of struct elements as a bool
1608 Value *collapseStructShadow(StructType *Struct, Value *Shadow,
1609 IRBuilder<> &IRB) {
1610 Value *FalseVal = IRB.getIntN(/* width */ 1, /* value */ 0);
1611 Value *Aggregator = FalseVal;
1612
1613 for (unsigned Idx = 0; Idx < Struct->getNumElements(); Idx++) {
1614 // Combine by ORing together each element's bool shadow
1615 Value *ShadowItem = IRB.CreateExtractValue(Shadow, Idx);
1616 Value *ShadowBool = convertToBool(ShadowItem, IRB);
1617
1618 if (Aggregator != FalseVal)
1619 Aggregator = IRB.CreateOr(Aggregator, ShadowBool);
1620 else
1621 Aggregator = ShadowBool;
1622 }
1623
1624 return Aggregator;
1625 }
1626
1627 // Extract combined shadow of array elements
1628 Value *collapseArrayShadow(ArrayType *Array, Value *Shadow,
1629 IRBuilder<> &IRB) {
1630 if (!Array->getNumElements())
1631 return IRB.getIntN(/* width */ 1, /* value */ 0);
1632
1633 Value *FirstItem = IRB.CreateExtractValue(Shadow, 0);
1634 Value *Aggregator = convertShadowToScalar(FirstItem, IRB);
1635
1636 for (unsigned Idx = 1; Idx < Array->getNumElements(); Idx++) {
1637 Value *ShadowItem = IRB.CreateExtractValue(Shadow, Idx);
1638 Value *ShadowInner = convertShadowToScalar(ShadowItem, IRB);
1639 Aggregator = IRB.CreateOr(Aggregator, ShadowInner);
1640 }
1641 return Aggregator;
1642 }
1643
1644 /// Convert a shadow value to it's flattened variant. The resulting
1645 /// shadow may not necessarily have the same bit width as the input
1646 /// value, but it will always be comparable to zero.
1647 Value *convertShadowToScalar(Value *V, IRBuilder<> &IRB) {
1648 if (StructType *Struct = dyn_cast<StructType>(V->getType()))
1649 return collapseStructShadow(Struct, V, IRB);
1650 if (ArrayType *Array = dyn_cast<ArrayType>(V->getType()))
1651 return collapseArrayShadow(Array, V, IRB);
1652 if (isa<VectorType>(V->getType())) {
1653 if (isa<ScalableVectorType>(V->getType()))
1654 return convertShadowToScalar(IRB.CreateOrReduce(V), IRB);
1655 unsigned BitWidth =
1656 V->getType()->getPrimitiveSizeInBits().getFixedValue();
1657 return IRB.CreateBitCast(V, IntegerType::get(*MS.C, BitWidth));
1658 }
1659 return V;
1660 }
1661
1662 // Convert a scalar value to an i1 by comparing with 0
1663 Value *convertToBool(Value *V, IRBuilder<> &IRB, const Twine &name = "") {
1664 Type *VTy = V->getType();
1665 if (!VTy->isIntegerTy())
1666 return convertToBool(convertShadowToScalar(V, IRB), IRB, name);
1667 if (VTy->getIntegerBitWidth() == 1)
1668 // Just converting a bool to a bool, so do nothing.
1669 return V;
1670 return IRB.CreateICmpNE(V, ConstantInt::get(VTy, 0), name);
1671 }
1672
1673 Type *ptrToIntPtrType(Type *PtrTy) const {
1674 if (VectorType *VectTy = dyn_cast<VectorType>(PtrTy)) {
1675 return VectorType::get(ptrToIntPtrType(VectTy->getElementType()),
1676 VectTy->getElementCount());
1677 }
1678 assert(PtrTy->isIntOrPtrTy());
1679 return MS.IntptrTy;
1680 }
1681
1682 Type *getPtrToShadowPtrType(Type *IntPtrTy, Type *ShadowTy) const {
1683 if (VectorType *VectTy = dyn_cast<VectorType>(IntPtrTy)) {
1684 return VectorType::get(
1685 getPtrToShadowPtrType(VectTy->getElementType(), ShadowTy),
1686 VectTy->getElementCount());
1687 }
1688 assert(IntPtrTy == MS.IntptrTy);
1689 return PointerType::get(*MS.C, 0);
1690 }
1691
1692 Constant *constToIntPtr(Type *IntPtrTy, uint64_t C) const {
1693 if (VectorType *VectTy = dyn_cast<VectorType>(IntPtrTy)) {
1695 VectTy->getElementCount(), constToIntPtr(VectTy->getElementType(), C));
1696 }
1697 assert(IntPtrTy == MS.IntptrTy);
1698 return ConstantInt::get(MS.IntptrTy, C);
1699 }
1700
1701 /// Compute the integer shadow offset that corresponds to a given
1702 /// application address.
1703 ///
1704 /// Offset = (Addr & ~AndMask) ^ XorMask
1705 /// Addr can be a ptr or <N x ptr>. In both cases ShadowTy the shadow type of
1706 /// a single pointee.
1707 /// Returns <shadow_ptr, origin_ptr> or <<N x shadow_ptr>, <N x origin_ptr>>.
1708 Value *getShadowPtrOffset(Value *Addr, IRBuilder<> &IRB) {
1709 Type *IntptrTy = ptrToIntPtrType(Addr->getType());
1710 Value *OffsetLong = IRB.CreatePointerCast(Addr, IntptrTy);
1711
1712 if (uint64_t AndMask = MS.MapParams->AndMask)
1713 OffsetLong = IRB.CreateAnd(OffsetLong, constToIntPtr(IntptrTy, ~AndMask));
1714
1715 if (uint64_t XorMask = MS.MapParams->XorMask)
1716 OffsetLong = IRB.CreateXor(OffsetLong, constToIntPtr(IntptrTy, XorMask));
1717 return OffsetLong;
1718 }
1719
1720 /// Compute the shadow and origin addresses corresponding to a given
1721 /// application address.
1722 ///
1723 /// Shadow = ShadowBase + Offset
1724 /// Origin = (OriginBase + Offset) & ~3ULL
1725 /// Addr can be a ptr or <N x ptr>. In both cases ShadowTy the shadow type of
1726 /// a single pointee.
1727 /// Returns <shadow_ptr, origin_ptr> or <<N x shadow_ptr>, <N x origin_ptr>>.
1728 std::pair<Value *, Value *>
1729 getShadowOriginPtrUserspace(Value *Addr, IRBuilder<> &IRB, Type *ShadowTy,
1730 MaybeAlign Alignment) {
1731 VectorType *VectTy = dyn_cast<VectorType>(Addr->getType());
1732 if (!VectTy) {
1733 assert(Addr->getType()->isPointerTy());
1734 } else {
1735 assert(VectTy->getElementType()->isPointerTy());
1736 }
1737 Type *IntptrTy = ptrToIntPtrType(Addr->getType());
1738 Value *ShadowOffset = getShadowPtrOffset(Addr, IRB);
1739 Value *ShadowLong = ShadowOffset;
1740 if (uint64_t ShadowBase = MS.MapParams->ShadowBase) {
1741 ShadowLong =
1742 IRB.CreateAdd(ShadowLong, constToIntPtr(IntptrTy, ShadowBase));
1743 }
1744 Value *ShadowPtr = IRB.CreateIntToPtr(
1745 ShadowLong, getPtrToShadowPtrType(IntptrTy, ShadowTy));
1746
1747 Value *OriginPtr = nullptr;
1748 if (MS.TrackOrigins) {
1749 Value *OriginLong = ShadowOffset;
1750 uint64_t OriginBase = MS.MapParams->OriginBase;
1751 if (OriginBase != 0)
1752 OriginLong =
1753 IRB.CreateAdd(OriginLong, constToIntPtr(IntptrTy, OriginBase));
1754 if (!Alignment || *Alignment < kMinOriginAlignment) {
1756 OriginLong = IRB.CreateAnd(OriginLong, constToIntPtr(IntptrTy, ~Mask));
1757 }
1758 OriginPtr = IRB.CreateIntToPtr(
1759 OriginLong, getPtrToShadowPtrType(IntptrTy, MS.OriginTy));
1760 }
1761 return std::make_pair(ShadowPtr, OriginPtr);
1762 }
1763
1764 template <typename... ArgsTy>
1765 Value *createMetadataCall(IRBuilder<> &IRB, FunctionCallee Callee,
1766 ArgsTy... Args) {
1767 if (MS.TargetTriple.getArch() == Triple::systemz) {
1768 IRB.CreateCall(Callee,
1769 {MS.MsanMetadataAlloca, std::forward<ArgsTy>(Args)...});
1770 return IRB.CreateLoad(MS.MsanMetadata, MS.MsanMetadataAlloca);
1771 }
1772
1773 return IRB.CreateCall(Callee, {std::forward<ArgsTy>(Args)...});
1774 }
1775
1776 std::pair<Value *, Value *> getShadowOriginPtrKernelNoVec(Value *Addr,
1777 IRBuilder<> &IRB,
1778 Type *ShadowTy,
1779 bool isStore) {
1780 Value *ShadowOriginPtrs;
1781 const DataLayout &DL = F.getDataLayout();
1782 TypeSize Size = DL.getTypeStoreSize(ShadowTy);
1783
1784 FunctionCallee Getter = MS.getKmsanShadowOriginAccessFn(isStore, Size);
1785 Value *AddrCast =
1786 IRB.CreatePointerCast(Addr, PointerType::get(IRB.getInt8Ty(), 0));
1787 if (Getter) {
1788 ShadowOriginPtrs = createMetadataCall(IRB, Getter, AddrCast);
1789 } else {
1790 Value *SizeVal = ConstantInt::get(MS.IntptrTy, Size);
1791 ShadowOriginPtrs = createMetadataCall(
1792 IRB,
1793 isStore ? MS.MsanMetadataPtrForStoreN : MS.MsanMetadataPtrForLoadN,
1794 AddrCast, SizeVal);
1795 }
1796 Value *ShadowPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 0);
1797 ShadowPtr = IRB.CreatePointerCast(ShadowPtr, PointerType::get(ShadowTy, 0));
1798 Value *OriginPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 1);
1799
1800 return std::make_pair(ShadowPtr, OriginPtr);
1801 }
1802
1803 /// Addr can be a ptr or <N x ptr>. In both cases ShadowTy the shadow type of
1804 /// a single pointee.
1805 /// Returns <shadow_ptr, origin_ptr> or <<N x shadow_ptr>, <N x origin_ptr>>.
1806 std::pair<Value *, Value *> getShadowOriginPtrKernel(Value *Addr,
1807 IRBuilder<> &IRB,
1808 Type *ShadowTy,
1809 bool isStore) {
1810 VectorType *VectTy = dyn_cast<VectorType>(Addr->getType());
1811 if (!VectTy) {
1812 assert(Addr->getType()->isPointerTy());
1813 return getShadowOriginPtrKernelNoVec(Addr, IRB, ShadowTy, isStore);
1814 }
1815
1816 // TODO: Support callbacs with vectors of addresses.
1817 unsigned NumElements = cast<FixedVectorType>(VectTy)->getNumElements();
1818 Value *ShadowPtrs = ConstantInt::getNullValue(
1819 FixedVectorType::get(IRB.getPtrTy(), NumElements));
1820 Value *OriginPtrs = nullptr;
1821 if (MS.TrackOrigins)
1822 OriginPtrs = ConstantInt::getNullValue(
1823 FixedVectorType::get(IRB.getPtrTy(), NumElements));
1824 for (unsigned i = 0; i < NumElements; ++i) {
1825 Value *OneAddr =
1826 IRB.CreateExtractElement(Addr, ConstantInt::get(IRB.getInt32Ty(), i));
1827 auto [ShadowPtr, OriginPtr] =
1828 getShadowOriginPtrKernelNoVec(OneAddr, IRB, ShadowTy, isStore);
1829
1830 ShadowPtrs = IRB.CreateInsertElement(
1831 ShadowPtrs, ShadowPtr, ConstantInt::get(IRB.getInt32Ty(), i));
1832 if (MS.TrackOrigins)
1833 OriginPtrs = IRB.CreateInsertElement(
1834 OriginPtrs, OriginPtr, ConstantInt::get(IRB.getInt32Ty(), i));
1835 }
1836 return {ShadowPtrs, OriginPtrs};
1837 }
1838
1839 std::pair<Value *, Value *> getShadowOriginPtr(Value *Addr, IRBuilder<> &IRB,
1840 Type *ShadowTy,
1841 MaybeAlign Alignment,
1842 bool isStore) {
1843 if (MS.CompileKernel)
1844 return getShadowOriginPtrKernel(Addr, IRB, ShadowTy, isStore);
1845 return getShadowOriginPtrUserspace(Addr, IRB, ShadowTy, Alignment);
1846 }
1847
1848 /// Compute the shadow address for a given function argument.
1849 ///
1850 /// Shadow = ParamTLS+ArgOffset.
1851 Value *getShadowPtrForArgument(IRBuilder<> &IRB, int ArgOffset) {
1852 Value *Base = IRB.CreatePointerCast(MS.ParamTLS, MS.IntptrTy);
1853 if (ArgOffset)
1854 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1855 return IRB.CreateIntToPtr(Base, IRB.getPtrTy(0), "_msarg");
1856 }
1857
1858 /// Compute the origin address for a given function argument.
1859 Value *getOriginPtrForArgument(IRBuilder<> &IRB, int ArgOffset) {
1860 if (!MS.TrackOrigins)
1861 return nullptr;
1862 Value *Base = IRB.CreatePointerCast(MS.ParamOriginTLS, MS.IntptrTy);
1863 if (ArgOffset)
1864 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1865 return IRB.CreateIntToPtr(Base, IRB.getPtrTy(0), "_msarg_o");
1866 }
1867
1868 /// Compute the shadow address for a retval.
1869 Value *getShadowPtrForRetval(IRBuilder<> &IRB) {
1870 return IRB.CreatePointerCast(MS.RetvalTLS, IRB.getPtrTy(0), "_msret");
1871 }
1872
1873 /// Compute the origin address for a retval.
1874 Value *getOriginPtrForRetval() {
1875 // We keep a single origin for the entire retval. Might be too optimistic.
1876 return MS.RetvalOriginTLS;
1877 }
1878
1879 /// Set SV to be the shadow value for V.
1880 void setShadow(Value *V, Value *SV) {
1881 assert(!ShadowMap.count(V) && "Values may only have one shadow");
1882 ShadowMap[V] = PropagateShadow ? SV : getCleanShadow(V);
1883 }
1884
1885 /// Set Origin to be the origin value for V.
1886 void setOrigin(Value *V, Value *Origin) {
1887 if (!MS.TrackOrigins)
1888 return;
1889 assert(!OriginMap.count(V) && "Values may only have one origin");
1890 LLVM_DEBUG(dbgs() << "ORIGIN: " << *V << " ==> " << *Origin << "\n");
1891 OriginMap[V] = Origin;
1892 }
1893
1894 Constant *getCleanShadow(Type *OrigTy) {
1895 Type *ShadowTy = getShadowTy(OrigTy);
1896 if (!ShadowTy)
1897 return nullptr;
1898 return Constant::getNullValue(ShadowTy);
1899 }
1900
1901 /// Create a clean shadow value for a given value.
1902 ///
1903 /// Clean shadow (all zeroes) means all bits of the value are defined
1904 /// (initialized).
1905 Constant *getCleanShadow(Value *V) { return getCleanShadow(V->getType()); }
1906
1907 /// Create a dirty shadow of a given shadow type.
1908 Constant *getPoisonedShadow(Type *ShadowTy) {
1909 assert(ShadowTy);
1910 if (isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy))
1911 return Constant::getAllOnesValue(ShadowTy);
1912 if (ArrayType *AT = dyn_cast<ArrayType>(ShadowTy)) {
1913 SmallVector<Constant *, 4> Vals(AT->getNumElements(),
1914 getPoisonedShadow(AT->getElementType()));
1915 return ConstantArray::get(AT, Vals);
1916 }
1917 if (StructType *ST = dyn_cast<StructType>(ShadowTy)) {
1919 for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
1920 Vals.push_back(getPoisonedShadow(ST->getElementType(i)));
1921 return ConstantStruct::get(ST, Vals);
1922 }
1923 llvm_unreachable("Unexpected shadow type");
1924 }
1925
1926 /// Create a dirty shadow for a given value.
1927 Constant *getPoisonedShadow(Value *V) {
1928 Type *ShadowTy = getShadowTy(V);
1929 if (!ShadowTy)
1930 return nullptr;
1931 return getPoisonedShadow(ShadowTy);
1932 }
1933
1934 /// Create a clean (zero) origin.
1935 Value *getCleanOrigin() { return Constant::getNullValue(MS.OriginTy); }
1936
1937 /// Get the shadow value for a given Value.
1938 ///
1939 /// This function either returns the value set earlier with setShadow,
1940 /// or extracts if from ParamTLS (for function arguments).
1941 Value *getShadow(Value *V) {
1942 if (Instruction *I = dyn_cast<Instruction>(V)) {
1943 if (!PropagateShadow || I->getMetadata(LLVMContext::MD_nosanitize))
1944 return getCleanShadow(V);
1945 // For instructions the shadow is already stored in the map.
1946 Value *Shadow = ShadowMap[V];
1947 if (!Shadow) {
1948 LLVM_DEBUG(dbgs() << "No shadow: " << *V << "\n" << *(I->getParent()));
1949 (void)I;
1950 assert(Shadow && "No shadow for a value");
1951 }
1952 return Shadow;
1953 }
1954 if (UndefValue *U = dyn_cast<UndefValue>(V)) {
1955 Value *AllOnes = (PropagateShadow && PoisonUndef) ? getPoisonedShadow(V)
1956 : getCleanShadow(V);
1957 LLVM_DEBUG(dbgs() << "Undef: " << *U << " ==> " << *AllOnes << "\n");
1958 (void)U;
1959 return AllOnes;
1960 }
1961 if (Argument *A = dyn_cast<Argument>(V)) {
1962 // For arguments we compute the shadow on demand and store it in the map.
1963 Value *&ShadowPtr = ShadowMap[V];
1964 if (ShadowPtr)
1965 return ShadowPtr;
1966 Function *F = A->getParent();
1967 IRBuilder<> EntryIRB(FnPrologueEnd);
1968 unsigned ArgOffset = 0;
1969 const DataLayout &DL = F->getDataLayout();
1970 for (auto &FArg : F->args()) {
1971 if (!FArg.getType()->isSized() || FArg.getType()->isScalableTy()) {
1972 LLVM_DEBUG(dbgs() << (FArg.getType()->isScalableTy()
1973 ? "vscale not fully supported\n"
1974 : "Arg is not sized\n"));
1975 if (A == &FArg) {
1976 ShadowPtr = getCleanShadow(V);
1977 setOrigin(A, getCleanOrigin());
1978 break;
1979 }
1980 continue;
1981 }
1982
1983 unsigned Size = FArg.hasByValAttr()
1984 ? DL.getTypeAllocSize(FArg.getParamByValType())
1985 : DL.getTypeAllocSize(FArg.getType());
1986
1987 if (A == &FArg) {
1988 bool Overflow = ArgOffset + Size > kParamTLSSize;
1989 if (FArg.hasByValAttr()) {
1990 // ByVal pointer itself has clean shadow. We copy the actual
1991 // argument shadow to the underlying memory.
1992 // Figure out maximal valid memcpy alignment.
1993 const Align ArgAlign = DL.getValueOrABITypeAlignment(
1994 FArg.getParamAlign(), FArg.getParamByValType());
1995 Value *CpShadowPtr, *CpOriginPtr;
1996 std::tie(CpShadowPtr, CpOriginPtr) =
1997 getShadowOriginPtr(V, EntryIRB, EntryIRB.getInt8Ty(), ArgAlign,
1998 /*isStore*/ true);
1999 if (!PropagateShadow || Overflow) {
2000 // ParamTLS overflow.
2001 EntryIRB.CreateMemSet(
2002 CpShadowPtr, Constant::getNullValue(EntryIRB.getInt8Ty()),
2003 Size, ArgAlign);
2004 } else {
2005 Value *Base = getShadowPtrForArgument(EntryIRB, ArgOffset);
2006 const Align CopyAlign = std::min(ArgAlign, kShadowTLSAlignment);
2007 Value *Cpy = EntryIRB.CreateMemCpy(CpShadowPtr, CopyAlign, Base,
2008 CopyAlign, Size);
2009 LLVM_DEBUG(dbgs() << " ByValCpy: " << *Cpy << "\n");
2010 (void)Cpy;
2011
2012 if (MS.TrackOrigins) {
2013 Value *OriginPtr =
2014 getOriginPtrForArgument(EntryIRB, ArgOffset);
2015 // FIXME: OriginSize should be:
2016 // alignTo(V % kMinOriginAlignment + Size, kMinOriginAlignment)
2017 unsigned OriginSize = alignTo(Size, kMinOriginAlignment);
2018 EntryIRB.CreateMemCpy(
2019 CpOriginPtr,
2020 /* by getShadowOriginPtr */ kMinOriginAlignment, OriginPtr,
2021 /* by origin_tls[ArgOffset] */ kMinOriginAlignment,
2022 OriginSize);
2023 }
2024 }
2025 }
2026
2027 if (!PropagateShadow || Overflow || FArg.hasByValAttr() ||
2028 (MS.EagerChecks && FArg.hasAttribute(Attribute::NoUndef))) {
2029 ShadowPtr = getCleanShadow(V);
2030 setOrigin(A, getCleanOrigin());
2031 } else {
2032 // Shadow over TLS
2033 Value *Base = getShadowPtrForArgument(EntryIRB, ArgOffset);
2034 ShadowPtr = EntryIRB.CreateAlignedLoad(getShadowTy(&FArg), Base,
2036 if (MS.TrackOrigins) {
2037 Value *OriginPtr =
2038 getOriginPtrForArgument(EntryIRB, ArgOffset);
2039 setOrigin(A, EntryIRB.CreateLoad(MS.OriginTy, OriginPtr));
2040 }
2041 }
2043 << " ARG: " << FArg << " ==> " << *ShadowPtr << "\n");
2044 break;
2045 }
2046
2047 ArgOffset += alignTo(Size, kShadowTLSAlignment);
2048 }
2049 assert(ShadowPtr && "Could not find shadow for an argument");
2050 return ShadowPtr;
2051 }
2052 // For everything else the shadow is zero.
2053 return getCleanShadow(V);
2054 }
2055
2056 /// Get the shadow for i-th argument of the instruction I.
2057 Value *getShadow(Instruction *I, int i) {
2058 return getShadow(I->getOperand(i));
2059 }
2060
2061 /// Get the origin for a value.
2062 Value *getOrigin(Value *V) {
2063 if (!MS.TrackOrigins)
2064 return nullptr;
2065 if (!PropagateShadow || isa<Constant>(V) || isa<InlineAsm>(V))
2066 return getCleanOrigin();
2067 assert((isa<Instruction>(V) || isa<Argument>(V)) &&
2068 "Unexpected value type in getOrigin()");
2069 if (Instruction *I = dyn_cast<Instruction>(V)) {
2070 if (I->getMetadata(LLVMContext::MD_nosanitize))
2071 return getCleanOrigin();
2072 }
2073 Value *Origin = OriginMap[V];
2074 assert(Origin && "Missing origin");
2075 return Origin;
2076 }
2077
2078 /// Get the origin for i-th argument of the instruction I.
2079 Value *getOrigin(Instruction *I, int i) {
2080 return getOrigin(I->getOperand(i));
2081 }
2082
2083 /// Remember the place where a shadow check should be inserted.
2084 ///
2085 /// This location will be later instrumented with a check that will print a
2086 /// UMR warning in runtime if the shadow value is not 0.
2087 void insertShadowCheck(Value *Shadow, Value *Origin, Instruction *OrigIns) {
2088 assert(Shadow);
2089 if (!InsertChecks)
2090 return;
2091
2092 if (!DebugCounter::shouldExecute(DebugInsertCheck)) {
2093 LLVM_DEBUG(dbgs() << "Skipping check of " << *Shadow << " before "
2094 << *OrigIns << "\n");
2095 return;
2096 }
2097#ifndef NDEBUG
2098 Type *ShadowTy = Shadow->getType();
2099 assert((isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy) ||
2100 isa<StructType>(ShadowTy) || isa<ArrayType>(ShadowTy)) &&
2101 "Can only insert checks for integer, vector, and aggregate shadow "
2102 "types");
2103#endif
2104 InstrumentationList.push_back(
2105 ShadowOriginAndInsertPoint(Shadow, Origin, OrigIns));
2106 }
2107
2108 /// Remember the place where a shadow check should be inserted.
2109 ///
2110 /// This location will be later instrumented with a check that will print a
2111 /// UMR warning in runtime if the value is not fully defined.
2112 void insertShadowCheck(Value *Val, Instruction *OrigIns) {
2113 assert(Val);
2114 Value *Shadow, *Origin;
2116 Shadow = getShadow(Val);
2117 if (!Shadow)
2118 return;
2119 Origin = getOrigin(Val);
2120 } else {
2121 Shadow = dyn_cast_or_null<Instruction>(getShadow(Val));
2122 if (!Shadow)
2123 return;
2124 Origin = dyn_cast_or_null<Instruction>(getOrigin(Val));
2125 }
2126 insertShadowCheck(Shadow, Origin, OrigIns);
2127 }
2128
2130 switch (a) {
2131 case AtomicOrdering::NotAtomic:
2132 return AtomicOrdering::NotAtomic;
2133 case AtomicOrdering::Unordered:
2134 case AtomicOrdering::Monotonic:
2135 case AtomicOrdering::Release:
2136 return AtomicOrdering::Release;
2137 case AtomicOrdering::Acquire:
2138 case AtomicOrdering::AcquireRelease:
2139 return AtomicOrdering::AcquireRelease;
2140 case AtomicOrdering::SequentiallyConsistent:
2141 return AtomicOrdering::SequentiallyConsistent;
2142 }
2143 llvm_unreachable("Unknown ordering");
2144 }
2145
2146 Value *makeAddReleaseOrderingTable(IRBuilder<> &IRB) {
2147 constexpr int NumOrderings = (int)AtomicOrderingCABI::seq_cst + 1;
2148 uint32_t OrderingTable[NumOrderings] = {};
2149
2150 OrderingTable[(int)AtomicOrderingCABI::relaxed] =
2151 OrderingTable[(int)AtomicOrderingCABI::release] =
2152 (int)AtomicOrderingCABI::release;
2153 OrderingTable[(int)AtomicOrderingCABI::consume] =
2154 OrderingTable[(int)AtomicOrderingCABI::acquire] =
2155 OrderingTable[(int)AtomicOrderingCABI::acq_rel] =
2156 (int)AtomicOrderingCABI::acq_rel;
2157 OrderingTable[(int)AtomicOrderingCABI::seq_cst] =
2158 (int)AtomicOrderingCABI::seq_cst;
2159
2160 return ConstantDataVector::get(IRB.getContext(), OrderingTable);
2161 }
2162
2164 switch (a) {
2165 case AtomicOrdering::NotAtomic:
2166 return AtomicOrdering::NotAtomic;
2167 case AtomicOrdering::Unordered:
2168 case AtomicOrdering::Monotonic:
2169 case AtomicOrdering::Acquire:
2170 return AtomicOrdering::Acquire;
2171 case AtomicOrdering::Release:
2172 case AtomicOrdering::AcquireRelease:
2173 return AtomicOrdering::AcquireRelease;
2174 case AtomicOrdering::SequentiallyConsistent:
2175 return AtomicOrdering::SequentiallyConsistent;
2176 }
2177 llvm_unreachable("Unknown ordering");
2178 }
2179
2180 Value *makeAddAcquireOrderingTable(IRBuilder<> &IRB) {
2181 constexpr int NumOrderings = (int)AtomicOrderingCABI::seq_cst + 1;
2182 uint32_t OrderingTable[NumOrderings] = {};
2183
2184 OrderingTable[(int)AtomicOrderingCABI::relaxed] =
2185 OrderingTable[(int)AtomicOrderingCABI::acquire] =
2186 OrderingTable[(int)AtomicOrderingCABI::consume] =
2187 (int)AtomicOrderingCABI::acquire;
2188 OrderingTable[(int)AtomicOrderingCABI::release] =
2189 OrderingTable[(int)AtomicOrderingCABI::acq_rel] =
2190 (int)AtomicOrderingCABI::acq_rel;
2191 OrderingTable[(int)AtomicOrderingCABI::seq_cst] =
2192 (int)AtomicOrderingCABI::seq_cst;
2193
2194 return ConstantDataVector::get(IRB.getContext(), OrderingTable);
2195 }
2196
2197 // ------------------- Visitors.
2198 using InstVisitor<MemorySanitizerVisitor>::visit;
2199 void visit(Instruction &I) {
2200 if (I.getMetadata(LLVMContext::MD_nosanitize))
2201 return;
2202 // Don't want to visit if we're in the prologue
2203 if (isInPrologue(I))
2204 return;
2205 if (!DebugCounter::shouldExecute(DebugInstrumentInstruction)) {
2206 LLVM_DEBUG(dbgs() << "Skipping instruction: " << I << "\n");
2207 // We still need to set the shadow and origin to clean values.
2208 setShadow(&I, getCleanShadow(&I));
2209 setOrigin(&I, getCleanOrigin());
2210 return;
2211 }
2212
2213 Instructions.push_back(&I);
2214 }
2215
2216 /// Instrument LoadInst
2217 ///
2218 /// Loads the corresponding shadow and (optionally) origin.
2219 /// Optionally, checks that the load address is fully defined.
2220 void visitLoadInst(LoadInst &I) {
2221 assert(I.getType()->isSized() && "Load type must have size");
2222 assert(!I.getMetadata(LLVMContext::MD_nosanitize));
2223 NextNodeIRBuilder IRB(&I);
2224 Type *ShadowTy = getShadowTy(&I);
2225 Value *Addr = I.getPointerOperand();
2226 Value *ShadowPtr = nullptr, *OriginPtr = nullptr;
2227 const Align Alignment = I.getAlign();
2228 if (PropagateShadow) {
2229 std::tie(ShadowPtr, OriginPtr) =
2230 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
2231 setShadow(&I,
2232 IRB.CreateAlignedLoad(ShadowTy, ShadowPtr, Alignment, "_msld"));
2233 } else {
2234 setShadow(&I, getCleanShadow(&I));
2235 }
2236
2238 insertShadowCheck(I.getPointerOperand(), &I);
2239
2240 if (I.isAtomic())
2241 I.setOrdering(addAcquireOrdering(I.getOrdering()));
2242
2243 if (MS.TrackOrigins) {
2244 if (PropagateShadow) {
2245 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
2246 setOrigin(
2247 &I, IRB.CreateAlignedLoad(MS.OriginTy, OriginPtr, OriginAlignment));
2248 } else {
2249 setOrigin(&I, getCleanOrigin());
2250 }
2251 }
2252 }
2253
2254 /// Instrument StoreInst
2255 ///
2256 /// Stores the corresponding shadow and (optionally) origin.
2257 /// Optionally, checks that the store address is fully defined.
2258 void visitStoreInst(StoreInst &I) {
2259 StoreList.push_back(&I);
2261 insertShadowCheck(I.getPointerOperand(), &I);
2262 }
2263
2264 void handleCASOrRMW(Instruction &I) {
2265 assert(isa<AtomicRMWInst>(I) || isa<AtomicCmpXchgInst>(I));
2266
2267 IRBuilder<> IRB(&I);
2268 Value *Addr = I.getOperand(0);
2269 Value *Val = I.getOperand(1);
2270 Value *ShadowPtr = getShadowOriginPtr(Addr, IRB, getShadowTy(Val), Align(1),
2271 /*isStore*/ true)
2272 .first;
2273
2275 insertShadowCheck(Addr, &I);
2276
2277 // Only test the conditional argument of cmpxchg instruction.
2278 // The other argument can potentially be uninitialized, but we can not
2279 // detect this situation reliably without possible false positives.
2280 if (isa<AtomicCmpXchgInst>(I))
2281 insertShadowCheck(Val, &I);
2282
2283 IRB.CreateStore(getCleanShadow(Val), ShadowPtr);
2284
2285 setShadow(&I, getCleanShadow(&I));
2286 setOrigin(&I, getCleanOrigin());
2287 }
2288
2289 void visitAtomicRMWInst(AtomicRMWInst &I) {
2290 handleCASOrRMW(I);
2291 I.setOrdering(addReleaseOrdering(I.getOrdering()));
2292 }
2293
2294 void visitAtomicCmpXchgInst(AtomicCmpXchgInst &I) {
2295 handleCASOrRMW(I);
2296 I.setSuccessOrdering(addReleaseOrdering(I.getSuccessOrdering()));
2297 }
2298
2299 // Vector manipulation.
2300 void visitExtractElementInst(ExtractElementInst &I) {
2301 insertShadowCheck(I.getOperand(1), &I);
2302 IRBuilder<> IRB(&I);
2303 setShadow(&I, IRB.CreateExtractElement(getShadow(&I, 0), I.getOperand(1),
2304 "_msprop"));
2305 setOrigin(&I, getOrigin(&I, 0));
2306 }
2307
2308 void visitInsertElementInst(InsertElementInst &I) {
2309 insertShadowCheck(I.getOperand(2), &I);
2310 IRBuilder<> IRB(&I);
2311 auto *Shadow0 = getShadow(&I, 0);
2312 auto *Shadow1 = getShadow(&I, 1);
2313 setShadow(&I, IRB.CreateInsertElement(Shadow0, Shadow1, I.getOperand(2),
2314 "_msprop"));
2315 setOriginForNaryOp(I);
2316 }
2317
2318 void visitShuffleVectorInst(ShuffleVectorInst &I) {
2319 IRBuilder<> IRB(&I);
2320 auto *Shadow0 = getShadow(&I, 0);
2321 auto *Shadow1 = getShadow(&I, 1);
2322 setShadow(&I, IRB.CreateShuffleVector(Shadow0, Shadow1, I.getShuffleMask(),
2323 "_msprop"));
2324 setOriginForNaryOp(I);
2325 }
2326
2327 // Casts.
2328 void visitSExtInst(SExtInst &I) {
2329 IRBuilder<> IRB(&I);
2330 setShadow(&I, IRB.CreateSExt(getShadow(&I, 0), I.getType(), "_msprop"));
2331 setOrigin(&I, getOrigin(&I, 0));
2332 }
2333
2334 void visitZExtInst(ZExtInst &I) {
2335 IRBuilder<> IRB(&I);
2336 setShadow(&I, IRB.CreateZExt(getShadow(&I, 0), I.getType(), "_msprop"));
2337 setOrigin(&I, getOrigin(&I, 0));
2338 }
2339
2340 void visitTruncInst(TruncInst &I) {
2341 IRBuilder<> IRB(&I);
2342 setShadow(&I, IRB.CreateTrunc(getShadow(&I, 0), I.getType(), "_msprop"));
2343 setOrigin(&I, getOrigin(&I, 0));
2344 }
2345
2346 void visitBitCastInst(BitCastInst &I) {
2347 // Special case: if this is the bitcast (there is exactly 1 allowed) between
2348 // a musttail call and a ret, don't instrument. New instructions are not
2349 // allowed after a musttail call.
2350 if (auto *CI = dyn_cast<CallInst>(I.getOperand(0)))
2351 if (CI->isMustTailCall())
2352 return;
2353 IRBuilder<> IRB(&I);
2354 setShadow(&I, IRB.CreateBitCast(getShadow(&I, 0), getShadowTy(&I)));
2355 setOrigin(&I, getOrigin(&I, 0));
2356 }
2357
2358 void visitPtrToIntInst(PtrToIntInst &I) {
2359 IRBuilder<> IRB(&I);
2360 setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
2361 "_msprop_ptrtoint"));
2362 setOrigin(&I, getOrigin(&I, 0));
2363 }
2364
2365 void visitIntToPtrInst(IntToPtrInst &I) {
2366 IRBuilder<> IRB(&I);
2367 setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
2368 "_msprop_inttoptr"));
2369 setOrigin(&I, getOrigin(&I, 0));
2370 }
2371
2372 void visitFPToSIInst(CastInst &I) { handleShadowOr(I); }
2373 void visitFPToUIInst(CastInst &I) { handleShadowOr(I); }
2374 void visitSIToFPInst(CastInst &I) { handleShadowOr(I); }
2375 void visitUIToFPInst(CastInst &I) { handleShadowOr(I); }
2376 void visitFPExtInst(CastInst &I) { handleShadowOr(I); }
2377 void visitFPTruncInst(CastInst &I) { handleShadowOr(I); }
2378
2379 /// Propagate shadow for bitwise AND.
2380 ///
2381 /// This code is exact, i.e. if, for example, a bit in the left argument
2382 /// is defined and 0, then neither the value not definedness of the
2383 /// corresponding bit in B don't affect the resulting shadow.
2384 void visitAnd(BinaryOperator &I) {
2385 IRBuilder<> IRB(&I);
2386 // "And" of 0 and a poisoned value results in unpoisoned value.
2387 // 1&1 => 1; 0&1 => 0; p&1 => p;
2388 // 1&0 => 0; 0&0 => 0; p&0 => 0;
2389 // 1&p => p; 0&p => 0; p&p => p;
2390 // S = (S1 & S2) | (V1 & S2) | (S1 & V2)
2391 Value *S1 = getShadow(&I, 0);
2392 Value *S2 = getShadow(&I, 1);
2393 Value *V1 = I.getOperand(0);
2394 Value *V2 = I.getOperand(1);
2395 if (V1->getType() != S1->getType()) {
2396 V1 = IRB.CreateIntCast(V1, S1->getType(), false);
2397 V2 = IRB.CreateIntCast(V2, S2->getType(), false);
2398 }
2399 Value *S1S2 = IRB.CreateAnd(S1, S2);
2400 Value *V1S2 = IRB.CreateAnd(V1, S2);
2401 Value *S1V2 = IRB.CreateAnd(S1, V2);
2402 setShadow(&I, IRB.CreateOr({S1S2, V1S2, S1V2}));
2403 setOriginForNaryOp(I);
2404 }
2405
2406 void visitOr(BinaryOperator &I) {
2407 IRBuilder<> IRB(&I);
2408 // "Or" of 1 and a poisoned value results in unpoisoned value.
2409 // 1|1 => 1; 0|1 => 1; p|1 => 1;
2410 // 1|0 => 1; 0|0 => 0; p|0 => p;
2411 // 1|p => 1; 0|p => p; p|p => p;
2412 // S = (S1 & S2) | (~V1 & S2) | (S1 & ~V2)
2413 Value *S1 = getShadow(&I, 0);
2414 Value *S2 = getShadow(&I, 1);
2415 Value *V1 = IRB.CreateNot(I.getOperand(0));
2416 Value *V2 = IRB.CreateNot(I.getOperand(1));
2417 if (V1->getType() != S1->getType()) {
2418 V1 = IRB.CreateIntCast(V1, S1->getType(), false);
2419 V2 = IRB.CreateIntCast(V2, S2->getType(), false);
2420 }
2421 Value *S1S2 = IRB.CreateAnd(S1, S2);
2422 Value *V1S2 = IRB.CreateAnd(V1, S2);
2423 Value *S1V2 = IRB.CreateAnd(S1, V2);
2424 setShadow(&I, IRB.CreateOr({S1S2, V1S2, S1V2}));
2425 setOriginForNaryOp(I);
2426 }
2427
2428 /// Default propagation of shadow and/or origin.
2429 ///
2430 /// This class implements the general case of shadow propagation, used in all
2431 /// cases where we don't know and/or don't care about what the operation
2432 /// actually does. It converts all input shadow values to a common type
2433 /// (extending or truncating as necessary), and bitwise OR's them.
2434 ///
2435 /// This is much cheaper than inserting checks (i.e. requiring inputs to be
2436 /// fully initialized), and less prone to false positives.
2437 ///
2438 /// This class also implements the general case of origin propagation. For a
2439 /// Nary operation, result origin is set to the origin of an argument that is
2440 /// not entirely initialized. If there is more than one such arguments, the
2441 /// rightmost of them is picked. It does not matter which one is picked if all
2442 /// arguments are initialized.
2443 template <bool CombineShadow> class Combiner {
2444 Value *Shadow = nullptr;
2445 Value *Origin = nullptr;
2446 IRBuilder<> &IRB;
2447 MemorySanitizerVisitor *MSV;
2448
2449 public:
2450 Combiner(MemorySanitizerVisitor *MSV, IRBuilder<> &IRB)
2451 : IRB(IRB), MSV(MSV) {}
2452
2453 /// Add a pair of shadow and origin values to the mix.
2454 Combiner &Add(Value *OpShadow, Value *OpOrigin) {
2455 if (CombineShadow) {
2456 assert(OpShadow);
2457 if (!Shadow)
2458 Shadow = OpShadow;
2459 else {
2460 OpShadow = MSV->CreateShadowCast(IRB, OpShadow, Shadow->getType());
2461 Shadow = IRB.CreateOr(Shadow, OpShadow, "_msprop");
2462 }
2463 }
2464
2465 if (MSV->MS.TrackOrigins) {
2466 assert(OpOrigin);
2467 if (!Origin) {
2468 Origin = OpOrigin;
2469 } else {
2470 Constant *ConstOrigin = dyn_cast<Constant>(OpOrigin);
2471 // No point in adding something that might result in 0 origin value.
2472 if (!ConstOrigin || !ConstOrigin->isNullValue()) {
2473 Value *Cond = MSV->convertToBool(OpShadow, IRB);
2474 Origin = IRB.CreateSelect(Cond, OpOrigin, Origin);
2475 }
2476 }
2477 }
2478 return *this;
2479 }
2480
2481 /// Add an application value to the mix.
2482 Combiner &Add(Value *V) {
2483 Value *OpShadow = MSV->getShadow(V);
2484 Value *OpOrigin = MSV->MS.TrackOrigins ? MSV->getOrigin(V) : nullptr;
2485 return Add(OpShadow, OpOrigin);
2486 }
2487
2488 /// Set the current combined values as the given instruction's shadow
2489 /// and origin.
2490 void Done(Instruction *I) {
2491 if (CombineShadow) {
2492 assert(Shadow);
2493 Shadow = MSV->CreateShadowCast(IRB, Shadow, MSV->getShadowTy(I));
2494 MSV->setShadow(I, Shadow);
2495 }
2496 if (MSV->MS.TrackOrigins) {
2497 assert(Origin);
2498 MSV->setOrigin(I, Origin);
2499 }
2500 }
2501 };
2502
2503 using ShadowAndOriginCombiner = Combiner<true>;
2504 using OriginCombiner = Combiner<false>;
2505
2506 /// Propagate origin for arbitrary operation.
2507 void setOriginForNaryOp(Instruction &I) {
2508 if (!MS.TrackOrigins)
2509 return;
2510 IRBuilder<> IRB(&I);
2511 OriginCombiner OC(this, IRB);
2512 for (Use &Op : I.operands())
2513 OC.Add(Op.get());
2514 OC.Done(&I);
2515 }
2516
2517 size_t VectorOrPrimitiveTypeSizeInBits(Type *Ty) {
2518 assert(!(Ty->isVectorTy() && Ty->getScalarType()->isPointerTy()) &&
2519 "Vector of pointers is not a valid shadow type");
2520 return Ty->isVectorTy() ? cast<FixedVectorType>(Ty)->getNumElements() *
2522 : Ty->getPrimitiveSizeInBits();
2523 }
2524
2525 /// Cast between two shadow types, extending or truncating as
2526 /// necessary.
2527 Value *CreateShadowCast(IRBuilder<> &IRB, Value *V, Type *dstTy,
2528 bool Signed = false) {
2529 Type *srcTy = V->getType();
2530 if (srcTy == dstTy)
2531 return V;
2532 size_t srcSizeInBits = VectorOrPrimitiveTypeSizeInBits(srcTy);
2533 size_t dstSizeInBits = VectorOrPrimitiveTypeSizeInBits(dstTy);
2534 if (srcSizeInBits > 1 && dstSizeInBits == 1)
2535 return IRB.CreateICmpNE(V, getCleanShadow(V));
2536
2537 if (dstTy->isIntegerTy() && srcTy->isIntegerTy())
2538 return IRB.CreateIntCast(V, dstTy, Signed);
2539 if (dstTy->isVectorTy() && srcTy->isVectorTy() &&
2540 cast<VectorType>(dstTy)->getElementCount() ==
2541 cast<VectorType>(srcTy)->getElementCount())
2542 return IRB.CreateIntCast(V, dstTy, Signed);
2543 Value *V1 = IRB.CreateBitCast(V, Type::getIntNTy(*MS.C, srcSizeInBits));
2544 Value *V2 =
2545 IRB.CreateIntCast(V1, Type::getIntNTy(*MS.C, dstSizeInBits), Signed);
2546 return IRB.CreateBitCast(V2, dstTy);
2547 // TODO: handle struct types.
2548 }
2549
2550 /// Cast an application value to the type of its own shadow.
2551 Value *CreateAppToShadowCast(IRBuilder<> &IRB, Value *V) {
2552 Type *ShadowTy = getShadowTy(V);
2553 if (V->getType() == ShadowTy)
2554 return V;
2555 if (V->getType()->isPtrOrPtrVectorTy())
2556 return IRB.CreatePtrToInt(V, ShadowTy);
2557 else
2558 return IRB.CreateBitCast(V, ShadowTy);
2559 }
2560
2561 /// Propagate shadow for arbitrary operation.
2562 void handleShadowOr(Instruction &I) {
2563 IRBuilder<> IRB(&I);
2564 ShadowAndOriginCombiner SC(this, IRB);
2565 for (Use &Op : I.operands())
2566 SC.Add(Op.get());
2567 SC.Done(&I);
2568 }
2569
2570 void visitFNeg(UnaryOperator &I) { handleShadowOr(I); }
2571
2572 // Handle multiplication by constant.
2573 //
2574 // Handle a special case of multiplication by constant that may have one or
2575 // more zeros in the lower bits. This makes corresponding number of lower bits
2576 // of the result zero as well. We model it by shifting the other operand
2577 // shadow left by the required number of bits. Effectively, we transform
2578 // (X * (A * 2**B)) to ((X << B) * A) and instrument (X << B) as (Sx << B).
2579 // We use multiplication by 2**N instead of shift to cover the case of
2580 // multiplication by 0, which may occur in some elements of a vector operand.
2581 void handleMulByConstant(BinaryOperator &I, Constant *ConstArg,
2582 Value *OtherArg) {
2583 Constant *ShadowMul;
2584 Type *Ty = ConstArg->getType();
2585 if (auto *VTy = dyn_cast<VectorType>(Ty)) {
2586 unsigned NumElements = cast<FixedVectorType>(VTy)->getNumElements();
2587 Type *EltTy = VTy->getElementType();
2589 for (unsigned Idx = 0; Idx < NumElements; ++Idx) {
2590 if (ConstantInt *Elt =
2591 dyn_cast<ConstantInt>(ConstArg->getAggregateElement(Idx))) {
2592 const APInt &V = Elt->getValue();
2593 APInt V2 = APInt(V.getBitWidth(), 1) << V.countr_zero();
2594 Elements.push_back(ConstantInt::get(EltTy, V2));
2595 } else {
2596 Elements.push_back(ConstantInt::get(EltTy, 1));
2597 }
2598 }
2599 ShadowMul = ConstantVector::get(Elements);
2600 } else {
2601 if (ConstantInt *Elt = dyn_cast<ConstantInt>(ConstArg)) {
2602 const APInt &V = Elt->getValue();
2603 APInt V2 = APInt(V.getBitWidth(), 1) << V.countr_zero();
2604 ShadowMul = ConstantInt::get(Ty, V2);
2605 } else {
2606 ShadowMul = ConstantInt::get(Ty, 1);
2607 }
2608 }
2609
2610 IRBuilder<> IRB(&I);
2611 setShadow(&I,
2612 IRB.CreateMul(getShadow(OtherArg), ShadowMul, "msprop_mul_cst"));
2613 setOrigin(&I, getOrigin(OtherArg));
2614 }
2615
2616 void visitMul(BinaryOperator &I) {
2617 Constant *constOp0 = dyn_cast<Constant>(I.getOperand(0));
2618 Constant *constOp1 = dyn_cast<Constant>(I.getOperand(1));
2619 if (constOp0 && !constOp1)
2620 handleMulByConstant(I, constOp0, I.getOperand(1));
2621 else if (constOp1 && !constOp0)
2622 handleMulByConstant(I, constOp1, I.getOperand(0));
2623 else
2624 handleShadowOr(I);
2625 }
2626
2627 void visitFAdd(BinaryOperator &I) { handleShadowOr(I); }
2628 void visitFSub(BinaryOperator &I) { handleShadowOr(I); }
2629 void visitFMul(BinaryOperator &I) { handleShadowOr(I); }
2630 void visitAdd(BinaryOperator &I) { handleShadowOr(I); }
2631 void visitSub(BinaryOperator &I) { handleShadowOr(I); }
2632 void visitXor(BinaryOperator &I) { handleShadowOr(I); }
2633
2634 void handleIntegerDiv(Instruction &I) {
2635 IRBuilder<> IRB(&I);
2636 // Strict on the second argument.
2637 insertShadowCheck(I.getOperand(1), &I);
2638 setShadow(&I, getShadow(&I, 0));
2639 setOrigin(&I, getOrigin(&I, 0));
2640 }
2641
2642 void visitUDiv(BinaryOperator &I) { handleIntegerDiv(I); }
2643 void visitSDiv(BinaryOperator &I) { handleIntegerDiv(I); }
2644 void visitURem(BinaryOperator &I) { handleIntegerDiv(I); }
2645 void visitSRem(BinaryOperator &I) { handleIntegerDiv(I); }
2646
2647 // Floating point division is side-effect free. We can not require that the
2648 // divisor is fully initialized and must propagate shadow. See PR37523.
2649 void visitFDiv(BinaryOperator &I) { handleShadowOr(I); }
2650 void visitFRem(BinaryOperator &I) { handleShadowOr(I); }
2651
2652 /// Instrument == and != comparisons.
2653 ///
2654 /// Sometimes the comparison result is known even if some of the bits of the
2655 /// arguments are not.
2656 void handleEqualityComparison(ICmpInst &I) {
2657 IRBuilder<> IRB(&I);
2658 Value *A = I.getOperand(0);
2659 Value *B = I.getOperand(1);
2660 Value *Sa = getShadow(A);
2661 Value *Sb = getShadow(B);
2662
2663 // Get rid of pointers and vectors of pointers.
2664 // For ints (and vectors of ints), types of A and Sa match,
2665 // and this is a no-op.
2666 A = IRB.CreatePointerCast(A, Sa->getType());
2667 B = IRB.CreatePointerCast(B, Sb->getType());
2668
2669 // A == B <==> (C = A^B) == 0
2670 // A != B <==> (C = A^B) != 0
2671 // Sc = Sa | Sb
2672 Value *C = IRB.CreateXor(A, B);
2673 Value *Sc = IRB.CreateOr(Sa, Sb);
2674 // Now dealing with i = (C == 0) comparison (or C != 0, does not matter now)
2675 // Result is defined if one of the following is true
2676 // * there is a defined 1 bit in C
2677 // * C is fully defined
2678 // Si = !(C & ~Sc) && Sc
2680 Value *MinusOne = Constant::getAllOnesValue(Sc->getType());
2681 Value *LHS = IRB.CreateICmpNE(Sc, Zero);
2682 Value *RHS =
2683 IRB.CreateICmpEQ(IRB.CreateAnd(IRB.CreateXor(Sc, MinusOne), C), Zero);
2684 Value *Si = IRB.CreateAnd(LHS, RHS);
2685 Si->setName("_msprop_icmp");
2686 setShadow(&I, Si);
2687 setOriginForNaryOp(I);
2688 }
2689
2690 /// Build the lowest possible value of V, taking into account V's
2691 /// uninitialized bits.
2692 Value *getLowestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2693 bool isSigned) {
2694 if (isSigned) {
2695 // Split shadow into sign bit and other bits.
2696 Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2697 Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2698 // Maximise the undefined shadow bit, minimize other undefined bits.
2699 return IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaOtherBits)),
2700 SaSignBit);
2701 } else {
2702 // Minimize undefined bits.
2703 return IRB.CreateAnd(A, IRB.CreateNot(Sa));
2704 }
2705 }
2706
2707 /// Build the highest possible value of V, taking into account V's
2708 /// uninitialized bits.
2709 Value *getHighestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2710 bool isSigned) {
2711 if (isSigned) {
2712 // Split shadow into sign bit and other bits.
2713 Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2714 Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2715 // Minimise the undefined shadow bit, maximise other undefined bits.
2716 return IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaSignBit)),
2717 SaOtherBits);
2718 } else {
2719 // Maximize undefined bits.
2720 return IRB.CreateOr(A, Sa);
2721 }
2722 }
2723
2724 /// Instrument relational comparisons.
2725 ///
2726 /// This function does exact shadow propagation for all relational
2727 /// comparisons of integers, pointers and vectors of those.
2728 /// FIXME: output seems suboptimal when one of the operands is a constant
2729 void handleRelationalComparisonExact(ICmpInst &I) {
2730 IRBuilder<> IRB(&I);
2731 Value *A = I.getOperand(0);
2732 Value *B = I.getOperand(1);
2733 Value *Sa = getShadow(A);
2734 Value *Sb = getShadow(B);
2735
2736 // Get rid of pointers and vectors of pointers.
2737 // For ints (and vectors of ints), types of A and Sa match,
2738 // and this is a no-op.
2739 A = IRB.CreatePointerCast(A, Sa->getType());
2740 B = IRB.CreatePointerCast(B, Sb->getType());
2741
2742 // Let [a0, a1] be the interval of possible values of A, taking into account
2743 // its undefined bits. Let [b0, b1] be the interval of possible values of B.
2744 // Then (A cmp B) is defined iff (a0 cmp b1) == (a1 cmp b0).
2745 bool IsSigned = I.isSigned();
2746 Value *S1 = IRB.CreateICmp(I.getPredicate(),
2747 getLowestPossibleValue(IRB, A, Sa, IsSigned),
2748 getHighestPossibleValue(IRB, B, Sb, IsSigned));
2749 Value *S2 = IRB.CreateICmp(I.getPredicate(),
2750 getHighestPossibleValue(IRB, A, Sa, IsSigned),
2751 getLowestPossibleValue(IRB, B, Sb, IsSigned));
2752 Value *Si = IRB.CreateXor(S1, S2);
2753 setShadow(&I, Si);
2754 setOriginForNaryOp(I);
2755 }
2756
2757 /// Instrument signed relational comparisons.
2758 ///
2759 /// Handle sign bit tests: x<0, x>=0, x<=-1, x>-1 by propagating the highest
2760 /// bit of the shadow. Everything else is delegated to handleShadowOr().
2761 void handleSignedRelationalComparison(ICmpInst &I) {
2762 Constant *constOp;
2763 Value *op = nullptr;
2765 if ((constOp = dyn_cast<Constant>(I.getOperand(1)))) {
2766 op = I.getOperand(0);
2767 pre = I.getPredicate();
2768 } else if ((constOp = dyn_cast<Constant>(I.getOperand(0)))) {
2769 op = I.getOperand(1);
2770 pre = I.getSwappedPredicate();
2771 } else {
2772 handleShadowOr(I);
2773 return;
2774 }
2775
2776 if ((constOp->isNullValue() &&
2777 (pre == CmpInst::ICMP_SLT || pre == CmpInst::ICMP_SGE)) ||
2778 (constOp->isAllOnesValue() &&
2779 (pre == CmpInst::ICMP_SGT || pre == CmpInst::ICMP_SLE))) {
2780 IRBuilder<> IRB(&I);
2781 Value *Shadow = IRB.CreateICmpSLT(getShadow(op), getCleanShadow(op),
2782 "_msprop_icmp_s");
2783 setShadow(&I, Shadow);
2784 setOrigin(&I, getOrigin(op));
2785 } else {
2786 handleShadowOr(I);
2787 }
2788 }
2789
2790 void visitICmpInst(ICmpInst &I) {
2791 if (!ClHandleICmp) {
2792 handleShadowOr(I);
2793 return;
2794 }
2795 if (I.isEquality()) {
2796 handleEqualityComparison(I);
2797 return;
2798 }
2799
2800 assert(I.isRelational());
2801 if (ClHandleICmpExact) {
2802 handleRelationalComparisonExact(I);
2803 return;
2804 }
2805 if (I.isSigned()) {
2806 handleSignedRelationalComparison(I);
2807 return;
2808 }
2809
2810 assert(I.isUnsigned());
2811 if ((isa<Constant>(I.getOperand(0)) || isa<Constant>(I.getOperand(1)))) {
2812 handleRelationalComparisonExact(I);
2813 return;
2814 }
2815
2816 handleShadowOr(I);
2817 }
2818
2819 void visitFCmpInst(FCmpInst &I) { handleShadowOr(I); }
2820
2821 void handleShift(BinaryOperator &I) {
2822 IRBuilder<> IRB(&I);
2823 // If any of the S2 bits are poisoned, the whole thing is poisoned.
2824 // Otherwise perform the same shift on S1.
2825 Value *S1 = getShadow(&I, 0);
2826 Value *S2 = getShadow(&I, 1);
2827 Value *S2Conv =
2828 IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)), S2->getType());
2829 Value *V2 = I.getOperand(1);
2830 Value *Shift = IRB.CreateBinOp(I.getOpcode(), S1, V2);
2831 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2832 setOriginForNaryOp(I);
2833 }
2834
2835 void visitShl(BinaryOperator &I) { handleShift(I); }
2836 void visitAShr(BinaryOperator &I) { handleShift(I); }
2837 void visitLShr(BinaryOperator &I) { handleShift(I); }
2838
2839 void handleFunnelShift(IntrinsicInst &I) {
2840 IRBuilder<> IRB(&I);
2841 // If any of the S2 bits are poisoned, the whole thing is poisoned.
2842 // Otherwise perform the same shift on S0 and S1.
2843 Value *S0 = getShadow(&I, 0);
2844 Value *S1 = getShadow(&I, 1);
2845 Value *S2 = getShadow(&I, 2);
2846 Value *S2Conv =
2847 IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)), S2->getType());
2848 Value *V2 = I.getOperand(2);
2850 I.getModule(), I.getIntrinsicID(), S2Conv->getType());
2851 Value *Shift = IRB.CreateCall(Intrin, {S0, S1, V2});
2852 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2853 setOriginForNaryOp(I);
2854 }
2855
2856 /// Instrument llvm.memmove
2857 ///
2858 /// At this point we don't know if llvm.memmove will be inlined or not.
2859 /// If we don't instrument it and it gets inlined,
2860 /// our interceptor will not kick in and we will lose the memmove.
2861 /// If we instrument the call here, but it does not get inlined,
2862 /// we will memove the shadow twice: which is bad in case
2863 /// of overlapping regions. So, we simply lower the intrinsic to a call.
2864 ///
2865 /// Similar situation exists for memcpy and memset.
2866 void visitMemMoveInst(MemMoveInst &I) {
2867 getShadow(I.getArgOperand(1)); // Ensure shadow initialized
2868 IRBuilder<> IRB(&I);
2869 IRB.CreateCall(MS.MemmoveFn,
2870 {I.getArgOperand(0), I.getArgOperand(1),
2871 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2872 I.eraseFromParent();
2873 }
2874
2875 /// Instrument memcpy
2876 ///
2877 /// Similar to memmove: avoid copying shadow twice. This is somewhat
2878 /// unfortunate as it may slowdown small constant memcpys.
2879 /// FIXME: consider doing manual inline for small constant sizes and proper
2880 /// alignment.
2881 ///
2882 /// Note: This also handles memcpy.inline, which promises no calls to external
2883 /// functions as an optimization. However, with instrumentation enabled this
2884 /// is difficult to promise; additionally, we know that the MSan runtime
2885 /// exists and provides __msan_memcpy(). Therefore, we assume that with
2886 /// instrumentation it's safe to turn memcpy.inline into a call to
2887 /// __msan_memcpy(). Should this be wrong, such as when implementing memcpy()
2888 /// itself, instrumentation should be disabled with the no_sanitize attribute.
2889 void visitMemCpyInst(MemCpyInst &I) {
2890 getShadow(I.getArgOperand(1)); // Ensure shadow initialized
2891 IRBuilder<> IRB(&I);
2892 IRB.CreateCall(MS.MemcpyFn,
2893 {I.getArgOperand(0), I.getArgOperand(1),
2894 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2895 I.eraseFromParent();
2896 }
2897
2898 // Same as memcpy.
2899 void visitMemSetInst(MemSetInst &I) {
2900 IRBuilder<> IRB(&I);
2901 IRB.CreateCall(
2902 MS.MemsetFn,
2903 {I.getArgOperand(0),
2904 IRB.CreateIntCast(I.getArgOperand(1), IRB.getInt32Ty(), false),
2905 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2906 I.eraseFromParent();
2907 }
2908
2909 void visitVAStartInst(VAStartInst &I) { VAHelper->visitVAStartInst(I); }
2910
2911 void visitVACopyInst(VACopyInst &I) { VAHelper->visitVACopyInst(I); }
2912
2913 /// Handle vector store-like intrinsics.
2914 ///
2915 /// Instrument intrinsics that look like a simple SIMD store: writes memory,
2916 /// has 1 pointer argument and 1 vector argument, returns void.
2917 bool handleVectorStoreIntrinsic(IntrinsicInst &I) {
2918 IRBuilder<> IRB(&I);
2919 Value *Addr = I.getArgOperand(0);
2920 Value *Shadow = getShadow(&I, 1);
2921 Value *ShadowPtr, *OriginPtr;
2922
2923 // We don't know the pointer alignment (could be unaligned SSE store!).
2924 // Have to assume to worst case.
2925 std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
2926 Addr, IRB, Shadow->getType(), Align(1), /*isStore*/ true);
2927 IRB.CreateAlignedStore(Shadow, ShadowPtr, Align(1));
2928
2930 insertShadowCheck(Addr, &I);
2931
2932 // FIXME: factor out common code from materializeStores
2933 if (MS.TrackOrigins)
2934 IRB.CreateStore(getOrigin(&I, 1), OriginPtr);
2935 return true;
2936 }
2937
2938 /// Handle vector load-like intrinsics.
2939 ///
2940 /// Instrument intrinsics that look like a simple SIMD load: reads memory,
2941 /// has 1 pointer argument, returns a vector.
2942 bool handleVectorLoadIntrinsic(IntrinsicInst &I) {
2943 IRBuilder<> IRB(&I);
2944 Value *Addr = I.getArgOperand(0);
2945
2946 Type *ShadowTy = getShadowTy(&I);
2947 Value *ShadowPtr = nullptr, *OriginPtr = nullptr;
2948 if (PropagateShadow) {
2949 // We don't know the pointer alignment (could be unaligned SSE load!).
2950 // Have to assume to worst case.
2951 const Align Alignment = Align(1);
2952 std::tie(ShadowPtr, OriginPtr) =
2953 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
2954 setShadow(&I,
2955 IRB.CreateAlignedLoad(ShadowTy, ShadowPtr, Alignment, "_msld"));
2956 } else {
2957 setShadow(&I, getCleanShadow(&I));
2958 }
2959
2961 insertShadowCheck(Addr, &I);
2962
2963 if (MS.TrackOrigins) {
2964 if (PropagateShadow)
2965 setOrigin(&I, IRB.CreateLoad(MS.OriginTy, OriginPtr));
2966 else
2967 setOrigin(&I, getCleanOrigin());
2968 }
2969 return true;
2970 }
2971
2972 /// Handle (SIMD arithmetic)-like intrinsics.
2973 ///
2974 /// Instrument intrinsics with any number of arguments of the same type,
2975 /// equal to the return type. The type should be simple (no aggregates or
2976 /// pointers; vectors are fine).
2977 /// Caller guarantees that this intrinsic does not access memory.
2978 bool maybeHandleSimpleNomemIntrinsic(IntrinsicInst &I) {
2979 Type *RetTy = I.getType();
2980 if (!(RetTy->isIntOrIntVectorTy() || RetTy->isFPOrFPVectorTy() ||
2981 RetTy->isX86_MMXTy()))
2982 return false;
2983
2984 unsigned NumArgOperands = I.arg_size();
2985 for (unsigned i = 0; i < NumArgOperands; ++i) {
2986 Type *Ty = I.getArgOperand(i)->getType();
2987 if (Ty != RetTy)
2988 return false;
2989 }
2990
2991 IRBuilder<> IRB(&I);
2992 ShadowAndOriginCombiner SC(this, IRB);
2993 for (unsigned i = 0; i < NumArgOperands; ++i)
2994 SC.Add(I.getArgOperand(i));
2995 SC.Done(&I);
2996
2997 return true;
2998 }
2999
3000 /// Heuristically instrument unknown intrinsics.
3001 ///
3002 /// The main purpose of this code is to do something reasonable with all
3003 /// random intrinsics we might encounter, most importantly - SIMD intrinsics.
3004 /// We recognize several classes of intrinsics by their argument types and
3005 /// ModRefBehaviour and apply special instrumentation when we are reasonably
3006 /// sure that we know what the intrinsic does.
3007 ///
3008 /// We special-case intrinsics where this approach fails. See llvm.bswap
3009 /// handling as an example of that.
3010 bool handleUnknownIntrinsic(IntrinsicInst &I) {
3011 unsigned NumArgOperands = I.arg_size();
3012 if (NumArgOperands == 0)
3013 return false;
3014
3015 if (NumArgOperands == 2 && I.getArgOperand(0)->getType()->isPointerTy() &&
3016 I.getArgOperand(1)->getType()->isVectorTy() &&
3017 I.getType()->isVoidTy() && !I.onlyReadsMemory()) {
3018 // This looks like a vector store.
3019 return handleVectorStoreIntrinsic(I);
3020 }
3021
3022 if (NumArgOperands == 1 && I.getArgOperand(0)->getType()->isPointerTy() &&
3023 I.getType()->isVectorTy() && I.onlyReadsMemory()) {
3024 // This looks like a vector load.
3025 return handleVectorLoadIntrinsic(I);
3026 }
3027
3028 if (I.doesNotAccessMemory())
3029 if (maybeHandleSimpleNomemIntrinsic(I))
3030 return true;
3031
3032 // FIXME: detect and handle SSE maskstore/maskload
3033 return false;
3034 }
3035
3036 void handleInvariantGroup(IntrinsicInst &I) {
3037 setShadow(&I, getShadow(&I, 0));
3038 setOrigin(&I, getOrigin(&I, 0));
3039 }
3040
3041 void handleLifetimeStart(IntrinsicInst &I) {
3042 if (!PoisonStack)
3043 return;
3044 AllocaInst *AI = llvm::findAllocaForValue(I.getArgOperand(1));
3045 if (!AI)
3046 InstrumentLifetimeStart = false;
3047 LifetimeStartList.push_back(std::make_pair(&I, AI));
3048 }
3049
3050 void handleBswap(IntrinsicInst &I) {
3051 IRBuilder<> IRB(&I);
3052 Value *Op = I.getArgOperand(0);
3053 Type *OpType = Op->getType();
3055 F.getParent(), Intrinsic::bswap, ArrayRef(&OpType, 1));
3056 setShadow(&I, IRB.CreateCall(BswapFunc, getShadow(Op)));
3057 setOrigin(&I, getOrigin(Op));
3058 }
3059
3060 void handleCountZeroes(IntrinsicInst &I) {
3061 IRBuilder<> IRB(&I);
3062 Value *Src = I.getArgOperand(0);
3063
3064 // Set the Output shadow based on input Shadow
3065 Value *BoolShadow = IRB.CreateIsNotNull(getShadow(Src), "_mscz_bs");
3066
3067 // If zero poison is requested, mix in with the shadow
3068 Constant *IsZeroPoison = cast<Constant>(I.getOperand(1));
3069 if (!IsZeroPoison->isZeroValue()) {
3070 Value *BoolZeroPoison = IRB.CreateIsNull(Src, "_mscz_bzp");
3071 BoolShadow = IRB.CreateOr(BoolShadow, BoolZeroPoison, "_mscz_bs");
3072 }
3073
3074 Value *OutputShadow =
3075 IRB.CreateSExt(BoolShadow, getShadowTy(Src), "_mscz_os");
3076
3077 setShadow(&I, OutputShadow);
3078 setOriginForNaryOp(I);
3079 }
3080
3081 // Instrument vector convert intrinsic.
3082 //
3083 // This function instruments intrinsics like cvtsi2ss:
3084 // %Out = int_xxx_cvtyyy(%ConvertOp)
3085 // or
3086 // %Out = int_xxx_cvtyyy(%CopyOp, %ConvertOp)
3087 // Intrinsic converts \p NumUsedElements elements of \p ConvertOp to the same
3088 // number \p Out elements, and (if has 2 arguments) copies the rest of the
3089 // elements from \p CopyOp.
3090 // In most cases conversion involves floating-point value which may trigger a
3091 // hardware exception when not fully initialized. For this reason we require
3092 // \p ConvertOp[0:NumUsedElements] to be fully initialized and trap otherwise.
3093 // We copy the shadow of \p CopyOp[NumUsedElements:] to \p
3094 // Out[NumUsedElements:]. This means that intrinsics without \p CopyOp always
3095 // return a fully initialized value.
3096 void handleVectorConvertIntrinsic(IntrinsicInst &I, int NumUsedElements,
3097 bool HasRoundingMode = false) {
3098 IRBuilder<> IRB(&I);
3099 Value *CopyOp, *ConvertOp;
3100
3101 assert((!HasRoundingMode ||
3102 isa<ConstantInt>(I.getArgOperand(I.arg_size() - 1))) &&
3103 "Invalid rounding mode");
3104
3105 switch (I.arg_size() - HasRoundingMode) {
3106 case 2:
3107 CopyOp = I.getArgOperand(0);
3108 ConvertOp = I.getArgOperand(1);
3109 break;
3110 case 1:
3111 ConvertOp = I.getArgOperand(0);
3112 CopyOp = nullptr;
3113 break;
3114 default:
3115 llvm_unreachable("Cvt intrinsic with unsupported number of arguments.");
3116 }
3117
3118 // The first *NumUsedElements* elements of ConvertOp are converted to the
3119 // same number of output elements. The rest of the output is copied from
3120 // CopyOp, or (if not available) filled with zeroes.
3121 // Combine shadow for elements of ConvertOp that are used in this operation,
3122 // and insert a check.
3123 // FIXME: consider propagating shadow of ConvertOp, at least in the case of
3124 // int->any conversion.
3125 Value *ConvertShadow = getShadow(ConvertOp);
3126 Value *AggShadow = nullptr;
3127 if (ConvertOp->getType()->isVectorTy()) {
3128 AggShadow = IRB.CreateExtractElement(
3129 ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), 0));
3130 for (int i = 1; i < NumUsedElements; ++i) {
3131 Value *MoreShadow = IRB.CreateExtractElement(
3132 ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), i));
3133 AggShadow = IRB.CreateOr(AggShadow, MoreShadow);
3134 }
3135 } else {
3136 AggShadow = ConvertShadow;
3137 }
3138 assert(AggShadow->getType()->isIntegerTy());
3139 insertShadowCheck(AggShadow, getOrigin(ConvertOp), &I);
3140
3141 // Build result shadow by zero-filling parts of CopyOp shadow that come from
3142 // ConvertOp.
3143 if (CopyOp) {
3144 assert(CopyOp->getType() == I.getType());
3145 assert(CopyOp->getType()->isVectorTy());
3146 Value *ResultShadow = getShadow(CopyOp);
3147 Type *EltTy = cast<VectorType>(ResultShadow->getType())->getElementType();
3148 for (int i = 0; i < NumUsedElements; ++i) {
3149 ResultShadow = IRB.CreateInsertElement(
3150 ResultShadow, ConstantInt::getNullValue(EltTy),
3151 ConstantInt::get(IRB.getInt32Ty(), i));
3152 }
3153 setShadow(&I, ResultShadow);
3154 setOrigin(&I, getOrigin(CopyOp));
3155 } else {
3156 setShadow(&I, getCleanShadow(&I));
3157 setOrigin(&I, getCleanOrigin());
3158 }
3159 }
3160
3161 // Given a scalar or vector, extract lower 64 bits (or less), and return all
3162 // zeroes if it is zero, and all ones otherwise.
3163 Value *Lower64ShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
3164 if (S->getType()->isVectorTy())
3165 S = CreateShadowCast(IRB, S, IRB.getInt64Ty(), /* Signed */ true);
3166 assert(S->getType()->getPrimitiveSizeInBits() <= 64);
3167 Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
3168 return CreateShadowCast(IRB, S2, T, /* Signed */ true);
3169 }
3170
3171 // Given a vector, extract its first element, and return all
3172 // zeroes if it is zero, and all ones otherwise.
3173 Value *LowerElementShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
3174 Value *S1 = IRB.CreateExtractElement(S, (uint64_t)0);
3175 Value *S2 = IRB.CreateICmpNE(S1, getCleanShadow(S1));
3176 return CreateShadowCast(IRB, S2, T, /* Signed */ true);
3177 }
3178
3179 Value *VariableShadowExtend(IRBuilder<> &IRB, Value *S) {
3180 Type *T = S->getType();
3181 assert(T->isVectorTy());
3182 Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
3183 return IRB.CreateSExt(S2, T);
3184 }
3185
3186 // Instrument vector shift intrinsic.
3187 //
3188 // This function instruments intrinsics like int_x86_avx2_psll_w.
3189 // Intrinsic shifts %In by %ShiftSize bits.
3190 // %ShiftSize may be a vector. In that case the lower 64 bits determine shift
3191 // size, and the rest is ignored. Behavior is defined even if shift size is
3192 // greater than register (or field) width.
3193 void handleVectorShiftIntrinsic(IntrinsicInst &I, bool Variable) {
3194 assert(I.arg_size() == 2);
3195 IRBuilder<> IRB(&I);
3196 // If any of the S2 bits are poisoned, the whole thing is poisoned.
3197 // Otherwise perform the same shift on S1.
3198 Value *S1 = getShadow(&I, 0);
3199 Value *S2 = getShadow(&I, 1);
3200 Value *S2Conv = Variable ? VariableShadowExtend(IRB, S2)
3201 : Lower64ShadowExtend(IRB, S2, getShadowTy(&I));
3202 Value *V1 = I.getOperand(0);
3203 Value *V2 = I.getOperand(1);
3204 Value *Shift = IRB.CreateCall(I.getFunctionType(), I.getCalledOperand(),
3205 {IRB.CreateBitCast(S1, V1->getType()), V2});
3206 Shift = IRB.CreateBitCast(Shift, getShadowTy(&I));
3207 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
3208 setOriginForNaryOp(I);
3209 }
3210
3211 // Get an X86_MMX-sized vector type.
3212 Type *getMMXVectorTy(unsigned EltSizeInBits) {
3213 const unsigned X86_MMXSizeInBits = 64;
3214 assert(EltSizeInBits != 0 && (X86_MMXSizeInBits % EltSizeInBits) == 0 &&
3215 "Illegal MMX vector element size");
3216 return FixedVectorType::get(IntegerType::get(*MS.C, EltSizeInBits),
3217 X86_MMXSizeInBits / EltSizeInBits);
3218 }
3219
3220 // Returns a signed counterpart for an (un)signed-saturate-and-pack
3221 // intrinsic.
3222 Intrinsic::ID getSignedPackIntrinsic(Intrinsic::ID id) {
3223 switch (id) {
3224 case Intrinsic::x86_sse2_packsswb_128:
3225 case Intrinsic::x86_sse2_packuswb_128:
3226 return Intrinsic::x86_sse2_packsswb_128;
3227
3228 case Intrinsic::x86_sse2_packssdw_128:
3229 case Intrinsic::x86_sse41_packusdw:
3230 return Intrinsic::x86_sse2_packssdw_128;
3231
3232 case Intrinsic::x86_avx2_packsswb:
3233 case Intrinsic::x86_avx2_packuswb:
3234 return Intrinsic::x86_avx2_packsswb;
3235
3236 case Intrinsic::x86_avx2_packssdw:
3237 case Intrinsic::x86_avx2_packusdw:
3238 return Intrinsic::x86_avx2_packssdw;
3239
3240 case Intrinsic::x86_mmx_packsswb:
3241 case Intrinsic::x86_mmx_packuswb:
3242 return Intrinsic::x86_mmx_packsswb;
3243
3244 case Intrinsic::x86_mmx_packssdw:
3245 return Intrinsic::x86_mmx_packssdw;
3246 default:
3247 llvm_unreachable("unexpected intrinsic id");
3248 }
3249 }
3250
3251 // Instrument vector pack intrinsic.
3252 //
3253 // This function instruments intrinsics like x86_mmx_packsswb, that
3254 // packs elements of 2 input vectors into half as many bits with saturation.
3255 // Shadow is propagated with the signed variant of the same intrinsic applied
3256 // to sext(Sa != zeroinitializer), sext(Sb != zeroinitializer).
3257 // EltSizeInBits is used only for x86mmx arguments.
3258 void handleVectorPackIntrinsic(IntrinsicInst &I, unsigned EltSizeInBits = 0) {
3259 assert(I.arg_size() == 2);
3260 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
3261 IRBuilder<> IRB(&I);
3262 Value *S1 = getShadow(&I, 0);
3263 Value *S2 = getShadow(&I, 1);
3264 assert(isX86_MMX || S1->getType()->isVectorTy());
3265
3266 // SExt and ICmpNE below must apply to individual elements of input vectors.
3267 // In case of x86mmx arguments, cast them to appropriate vector types and
3268 // back.
3269 Type *T = isX86_MMX ? getMMXVectorTy(EltSizeInBits) : S1->getType();
3270 if (isX86_MMX) {
3271 S1 = IRB.CreateBitCast(S1, T);
3272 S2 = IRB.CreateBitCast(S2, T);
3273 }
3274 Value *S1_ext =
3276 Value *S2_ext =
3278 if (isX86_MMX) {
3279 Type *X86_MMXTy = Type::getX86_MMXTy(*MS.C);
3280 S1_ext = IRB.CreateBitCast(S1_ext, X86_MMXTy);
3281 S2_ext = IRB.CreateBitCast(S2_ext, X86_MMXTy);
3282 }
3283
3285 F.getParent(), getSignedPackIntrinsic(I.getIntrinsicID()));
3286
3287 Value *S =
3288 IRB.CreateCall(ShadowFn, {S1_ext, S2_ext}, "_msprop_vector_pack");
3289 if (isX86_MMX)
3290 S = IRB.CreateBitCast(S, getShadowTy(&I));
3291 setShadow(&I, S);
3292 setOriginForNaryOp(I);
3293 }
3294
3295 // Convert `Mask` into `<n x i1>`.
3296 Constant *createDppMask(unsigned Width, unsigned Mask) {
3298 for (auto &M : R) {
3299 M = ConstantInt::getBool(F.getContext(), Mask & 1);
3300 Mask >>= 1;
3301 }
3302 return ConstantVector::get(R);
3303 }
3304
3305 // Calculate output shadow as array of booleans `<n x i1>`, assuming if any
3306 // arg is poisoned, entire dot product is poisoned.
3307 Value *findDppPoisonedOutput(IRBuilder<> &IRB, Value *S, unsigned SrcMask,
3308 unsigned DstMask) {
3309 const unsigned Width =
3310 cast<FixedVectorType>(S->getType())->getNumElements();
3311
3312 S = IRB.CreateSelect(createDppMask(Width, SrcMask), S,
3314 Value *SElem = IRB.CreateOrReduce(S);
3315 Value *IsClean = IRB.CreateIsNull(SElem, "_msdpp");
3316 Value *DstMaskV = createDppMask(Width, DstMask);
3317
3318 return IRB.CreateSelect(
3319 IsClean, Constant::getNullValue(DstMaskV->getType()), DstMaskV);
3320 }
3321
3322 // See `Intel Intrinsics Guide` for `_dp_p*` instructions.
3323 //
3324 // 2 and 4 element versions produce single scalar of dot product, and then
3325 // puts it into elements of output vector, selected by 4 lowest bits of the
3326 // mask. Top 4 bits of the mask control which elements of input to use for dot
3327 // product.
3328 //
3329 // 8 element version mask still has only 4 bit for input, and 4 bit for output
3330 // mask. According to the spec it just operates as 4 element version on first
3331 // 4 elements of inputs and output, and then on last 4 elements of inputs and
3332 // output.
3333 void handleDppIntrinsic(IntrinsicInst &I) {
3334 IRBuilder<> IRB(&I);
3335
3336 Value *S0 = getShadow(&I, 0);
3337 Value *S1 = getShadow(&I, 1);
3338 Value *S = IRB.CreateOr(S0, S1);
3339
3340 const unsigned Width =
3341 cast<FixedVectorType>(S->getType())->getNumElements();
3342 assert(Width == 2 || Width == 4 || Width == 8);
3343
3344 const unsigned Mask = cast<ConstantInt>(I.getArgOperand(2))->getZExtValue();
3345 const unsigned SrcMask = Mask >> 4;
3346 const unsigned DstMask = Mask & 0xf;
3347
3348 // Calculate shadow as `<n x i1>`.
3349 Value *SI1 = findDppPoisonedOutput(IRB, S, SrcMask, DstMask);
3350 if (Width == 8) {
3351 // First 4 elements of shadow are already calculated. `makeDppShadow`
3352 // operats on 32 bit masks, so we can just shift masks, and repeat.
3353 SI1 = IRB.CreateOr(
3354 SI1, findDppPoisonedOutput(IRB, S, SrcMask << 4, DstMask << 4));
3355 }
3356 // Extend to real size of shadow, poisoning either all or none bits of an
3357 // element.
3358 S = IRB.CreateSExt(SI1, S->getType(), "_msdpp");
3359
3360 setShadow(&I, S);
3361 setOriginForNaryOp(I);
3362 }
3363
3364 Value *convertBlendvToSelectMask(IRBuilder<> &IRB, Value *C) {
3365 C = CreateAppToShadowCast(IRB, C);
3366 FixedVectorType *FVT = cast<FixedVectorType>(C->getType());
3367 unsigned ElSize = FVT->getElementType()->getPrimitiveSizeInBits();
3368 C = IRB.CreateAShr(C, ElSize - 1);
3369 FVT = FixedVectorType::get(IRB.getInt1Ty(), FVT->getNumElements());
3370 return IRB.CreateTrunc(C, FVT);
3371 }
3372
3373 // `blendv(f, t, c)` is effectively `select(c[top_bit], t, f)`.
3374 void handleBlendvIntrinsic(IntrinsicInst &I) {
3375 Value *C = I.getOperand(2);
3376 Value *T = I.getOperand(1);
3377 Value *F = I.getOperand(0);
3378
3379 Value *Sc = getShadow(&I, 2);
3380 Value *Oc = MS.TrackOrigins ? getOrigin(C) : nullptr;
3381
3382 {
3383 IRBuilder<> IRB(&I);
3384 // Extract top bit from condition and its shadow.
3385 C = convertBlendvToSelectMask(IRB, C);
3386 Sc = convertBlendvToSelectMask(IRB, Sc);
3387
3388 setShadow(C, Sc);
3389 setOrigin(C, Oc);
3390 }
3391
3392 handleSelectLikeInst(I, C, T, F);
3393 }
3394
3395 // Instrument sum-of-absolute-differences intrinsic.
3396 void handleVectorSadIntrinsic(IntrinsicInst &I) {
3397 const unsigned SignificantBitsPerResultElement = 16;
3398 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
3399 Type *ResTy = isX86_MMX ? IntegerType::get(*MS.C, 64) : I.getType();
3400 unsigned ZeroBitsPerResultElement =
3401 ResTy->getScalarSizeInBits() - SignificantBitsPerResultElement;
3402
3403 IRBuilder<> IRB(&I);
3404 auto *Shadow0 = getShadow(&I, 0);
3405 auto *Shadow1 = getShadow(&I, 1);
3406 Value *S = IRB.CreateOr(Shadow0, Shadow1);
3407 S = IRB.CreateBitCast(S, ResTy);
3408 S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
3409 ResTy);
3410 S = IRB.CreateLShr(S, ZeroBitsPerResultElement);
3411 S = IRB.CreateBitCast(S, getShadowTy(&I));
3412 setShadow(&I, S);
3413 setOriginForNaryOp(I);
3414 }
3415
3416 // Instrument multiply-add intrinsic.
3417 void handleVectorPmaddIntrinsic(IntrinsicInst &I,
3418 unsigned EltSizeInBits = 0) {
3419 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
3420 Type *ResTy = isX86_MMX ? getMMXVectorTy(EltSizeInBits * 2) : I.getType();
3421 IRBuilder<> IRB(&I);
3422 auto *Shadow0 = getShadow(&I, 0);
3423 auto *Shadow1 = getShadow(&I, 1);
3424 Value *S = IRB.CreateOr(Shadow0, Shadow1);
3425 S = IRB.CreateBitCast(S, ResTy);
3426 S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
3427 ResTy);
3428 S = IRB.CreateBitCast(S, getShadowTy(&I));
3429 setShadow(&I, S);
3430 setOriginForNaryOp(I);
3431 }
3432
3433 // Instrument compare-packed intrinsic.
3434 // Basically, an or followed by sext(icmp ne 0) to end up with all-zeros or
3435 // all-ones shadow.
3436 void handleVectorComparePackedIntrinsic(IntrinsicInst &I) {
3437 IRBuilder<> IRB(&I);
3438 Type *ResTy = getShadowTy(&I);
3439 auto *Shadow0 = getShadow(&I, 0);
3440 auto *Shadow1 = getShadow(&I, 1);
3441 Value *S0 = IRB.CreateOr(Shadow0, Shadow1);
3442 Value *S = IRB.CreateSExt(
3443 IRB.CreateICmpNE(S0, Constant::getNullValue(ResTy)), ResTy);
3444 setShadow(&I, S);
3445 setOriginForNaryOp(I);
3446 }
3447
3448 // Instrument compare-scalar intrinsic.
3449 // This handles both cmp* intrinsics which return the result in the first
3450 // element of a vector, and comi* which return the result as i32.
3451 void handleVectorCompareScalarIntrinsic(IntrinsicInst &I) {
3452 IRBuilder<> IRB(&I);
3453 auto *Shadow0 = getShadow(&I, 0);
3454 auto *Shadow1 = getShadow(&I, 1);
3455 Value *S0 = IRB.CreateOr(Shadow0, Shadow1);
3456 Value *S = LowerElementShadowExtend(IRB, S0, getShadowTy(&I));
3457 setShadow(&I, S);
3458 setOriginForNaryOp(I);
3459 }
3460
3461 // Instrument generic vector reduction intrinsics
3462 // by ORing together all their fields.
3463 void handleVectorReduceIntrinsic(IntrinsicInst &I) {
3464 IRBuilder<> IRB(&I);
3465 Value *S = IRB.CreateOrReduce(getShadow(&I, 0));
3466 setShadow(&I, S);
3467 setOrigin(&I, getOrigin(&I, 0));
3468 }
3469
3470 // Instrument vector.reduce.or intrinsic.
3471 // Valid (non-poisoned) set bits in the operand pull low the
3472 // corresponding shadow bits.
3473 void handleVectorReduceOrIntrinsic(IntrinsicInst &I) {
3474 IRBuilder<> IRB(&I);
3475 Value *OperandShadow = getShadow(&I, 0);
3476 Value *OperandUnsetBits = IRB.CreateNot(I.getOperand(0));
3477 Value *OperandUnsetOrPoison = IRB.CreateOr(OperandUnsetBits, OperandShadow);
3478 // Bit N is clean if any field's bit N is 1 and unpoison
3479 Value *OutShadowMask = IRB.CreateAndReduce(OperandUnsetOrPoison);
3480 // Otherwise, it is clean if every field's bit N is unpoison
3481 Value *OrShadow = IRB.CreateOrReduce(OperandShadow);
3482 Value *S = IRB.CreateAnd(OutShadowMask, OrShadow);
3483
3484 setShadow(&I, S);
3485 setOrigin(&I, getOrigin(&I, 0));
3486 }
3487
3488 // Instrument vector.reduce.and intrinsic.
3489 // Valid (non-poisoned) unset bits in the operand pull down the
3490 // corresponding shadow bits.
3491 void handleVectorReduceAndIntrinsic(IntrinsicInst &I) {
3492 IRBuilder<> IRB(&I);
3493 Value *OperandShadow = getShadow(&I, 0);
3494 Value *OperandSetOrPoison = IRB.CreateOr(I.getOperand(0), OperandShadow);
3495 // Bit N is clean if any field's bit N is 0 and unpoison
3496 Value *OutShadowMask = IRB.CreateAndReduce(OperandSetOrPoison);
3497 // Otherwise, it is clean if every field's bit N is unpoison
3498 Value *OrShadow = IRB.CreateOrReduce(OperandShadow);
3499 Value *S = IRB.CreateAnd(OutShadowMask, OrShadow);
3500
3501 setShadow(&I, S);
3502 setOrigin(&I, getOrigin(&I, 0));
3503 }
3504
3505 void handleStmxcsr(IntrinsicInst &I) {
3506 IRBuilder<> IRB(&I);
3507 Value *Addr = I.getArgOperand(0);
3508 Type *Ty = IRB.getInt32Ty();
3509 Value *ShadowPtr =
3510 getShadowOriginPtr(Addr, IRB, Ty, Align(1), /*isStore*/ true).first;
3511
3512 IRB.CreateStore(getCleanShadow(Ty), ShadowPtr);
3513
3515 insertShadowCheck(Addr, &I);
3516 }
3517
3518 void handleLdmxcsr(IntrinsicInst &I) {
3519 if (!InsertChecks)
3520 return;
3521
3522 IRBuilder<> IRB(&I);
3523 Value *Addr = I.getArgOperand(0);
3524 Type *Ty = IRB.getInt32Ty();
3525 const Align Alignment = Align(1);
3526 Value *ShadowPtr, *OriginPtr;
3527 std::tie(ShadowPtr, OriginPtr) =
3528 getShadowOriginPtr(Addr, IRB, Ty, Alignment, /*isStore*/ false);
3529
3531 insertShadowCheck(Addr, &I);
3532
3533 Value *Shadow = IRB.CreateAlignedLoad(Ty, ShadowPtr, Alignment, "_ldmxcsr");
3534 Value *Origin = MS.TrackOrigins ? IRB.CreateLoad(MS.OriginTy, OriginPtr)
3535 : getCleanOrigin();
3536 insertShadowCheck(Shadow, Origin, &I);
3537 }
3538
3539 void handleMaskedExpandLoad(IntrinsicInst &I) {
3540 IRBuilder<> IRB(&I);
3541 Value *Ptr = I.getArgOperand(0);
3542 Value *Mask = I.getArgOperand(1);
3543 Value *PassThru = I.getArgOperand(2);
3544
3546 insertShadowCheck(Ptr, &I);
3547 insertShadowCheck(Mask, &I);
3548 }
3549
3550 if (!PropagateShadow) {
3551 setShadow(&I, getCleanShadow(&I));
3552 setOrigin(&I, getCleanOrigin());
3553 return;
3554 }
3555
3556 Type *ShadowTy = getShadowTy(&I);
3557 Type *ElementShadowTy = cast<VectorType>(ShadowTy)->getElementType();
3558 auto [ShadowPtr, OriginPtr] =
3559 getShadowOriginPtr(Ptr, IRB, ElementShadowTy, {}, /*isStore*/ false);
3560
3561 Value *Shadow = IRB.CreateMaskedExpandLoad(
3562 ShadowTy, ShadowPtr, Mask, getShadow(PassThru), "_msmaskedexpload");
3563
3564 setShadow(&I, Shadow);
3565
3566 // TODO: Store origins.
3567 setOrigin(&I, getCleanOrigin());
3568 }
3569
3570 void handleMaskedCompressStore(IntrinsicInst &I) {
3571 IRBuilder<> IRB(&I);
3572 Value *Values = I.getArgOperand(0);
3573 Value *Ptr = I.getArgOperand(1);
3574 Value *Mask = I.getArgOperand(2);
3575
3577 insertShadowCheck(Ptr, &I);
3578 insertShadowCheck(Mask, &I);
3579 }
3580
3581 Value *Shadow = getShadow(Values);
3582 Type *ElementShadowTy =
3583 getShadowTy(cast<VectorType>(Values->getType())->getElementType());
3584 auto [ShadowPtr, OriginPtrs] =
3585 getShadowOriginPtr(Ptr, IRB, ElementShadowTy, {}, /*isStore*/ true);
3586
3587 IRB.CreateMaskedCompressStore(Shadow, ShadowPtr, Mask);
3588
3589 // TODO: Store origins.
3590 }
3591
3592 void handleMaskedGather(IntrinsicInst &I) {
3593 IRBuilder<> IRB(&I);
3594 Value *Ptrs = I.getArgOperand(0);
3595 const Align Alignment(
3596 cast<ConstantInt>(I.getArgOperand(1))->getZExtValue());
3597 Value *Mask = I.getArgOperand(2);
3598 Value *PassThru = I.getArgOperand(3);
3599
3600 Type *PtrsShadowTy = getShadowTy(Ptrs);
3602 insertShadowCheck(Mask, &I);
3603 Value *MaskedPtrShadow = IRB.CreateSelect(
3604 Mask, getShadow(Ptrs), Constant::getNullValue((PtrsShadowTy)),
3605 "_msmaskedptrs");
3606 insertShadowCheck(MaskedPtrShadow, getOrigin(Ptrs), &I);
3607 }
3608
3609 if (!PropagateShadow) {
3610 setShadow(&I, getCleanShadow(&I));
3611 setOrigin(&I, getCleanOrigin());
3612 return;
3613 }
3614
3615 Type *ShadowTy = getShadowTy(&I);
3616 Type *ElementShadowTy = cast<VectorType>(ShadowTy)->getElementType();
3617 auto [ShadowPtrs, OriginPtrs] = getShadowOriginPtr(
3618 Ptrs, IRB, ElementShadowTy, Alignment, /*isStore*/ false);
3619
3620 Value *Shadow =
3621 IRB.CreateMaskedGather(ShadowTy, ShadowPtrs, Alignment, Mask,
3622 getShadow(PassThru), "_msmaskedgather");
3623
3624 setShadow(&I, Shadow);
3625
3626 // TODO: Store origins.
3627 setOrigin(&I, getCleanOrigin());
3628 }
3629
3630 void handleMaskedScatter(IntrinsicInst &I) {
3631 IRBuilder<> IRB(&I);
3632 Value *Values = I.getArgOperand(0);
3633 Value *Ptrs = I.getArgOperand(1);
3634 const Align Alignment(
3635 cast<ConstantInt>(I.getArgOperand(2))->getZExtValue());
3636 Value *Mask = I.getArgOperand(3);
3637
3638 Type *PtrsShadowTy = getShadowTy(Ptrs);
3640 insertShadowCheck(Mask, &I);
3641 Value *MaskedPtrShadow = IRB.CreateSelect(
3642 Mask, getShadow(Ptrs), Constant::getNullValue((PtrsShadowTy)),
3643 "_msmaskedptrs");
3644 insertShadowCheck(MaskedPtrShadow, getOrigin(Ptrs), &I);
3645 }
3646
3647 Value *Shadow = getShadow(Values);
3648 Type *ElementShadowTy =
3649 getShadowTy(cast<VectorType>(Values->getType())->getElementType());
3650 auto [ShadowPtrs, OriginPtrs] = getShadowOriginPtr(
3651 Ptrs, IRB, ElementShadowTy, Alignment, /*isStore*/ true);
3652
3653 IRB.CreateMaskedScatter(Shadow, ShadowPtrs, Alignment, Mask);
3654
3655 // TODO: Store origin.
3656 }
3657
3658 void handleMaskedStore(IntrinsicInst &I) {
3659 IRBuilder<> IRB(&I);
3660 Value *V = I.getArgOperand(0);
3661 Value *Ptr = I.getArgOperand(1);
3662 const Align Alignment(
3663 cast<ConstantInt>(I.getArgOperand(2))->getZExtValue());
3664 Value *Mask = I.getArgOperand(3);
3665 Value *Shadow = getShadow(V);
3666
3668 insertShadowCheck(Ptr, &I);
3669 insertShadowCheck(Mask, &I);
3670 }
3671
3672 Value *ShadowPtr;
3673 Value *OriginPtr;
3674 std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
3675 Ptr, IRB, Shadow->getType(), Alignment, /*isStore*/ true);
3676
3677 IRB.CreateMaskedStore(Shadow, ShadowPtr, Alignment, Mask);
3678
3679 if (!MS.TrackOrigins)
3680 return;
3681
3682 auto &DL = F.getDataLayout();
3683 paintOrigin(IRB, getOrigin(V), OriginPtr,
3684 DL.getTypeStoreSize(Shadow->getType()),
3685 std::max(Alignment, kMinOriginAlignment));
3686 }
3687
3688 void handleMaskedLoad(IntrinsicInst &I) {
3689 IRBuilder<> IRB(&I);
3690 Value *Ptr = I.getArgOperand(0);
3691 const Align Alignment(
3692 cast<ConstantInt>(I.getArgOperand(1))->getZExtValue());
3693 Value *Mask = I.getArgOperand(2);
3694 Value *PassThru = I.getArgOperand(3);
3695
3697 insertShadowCheck(Ptr, &I);
3698 insertShadowCheck(Mask, &I);
3699 }
3700
3701 if (!PropagateShadow) {
3702 setShadow(&I, getCleanShadow(&I));
3703 setOrigin(&I, getCleanOrigin());
3704 return;
3705 }
3706
3707 Type *ShadowTy = getShadowTy(&I);
3708 Value *ShadowPtr, *OriginPtr;
3709 std::tie(ShadowPtr, OriginPtr) =
3710 getShadowOriginPtr(Ptr, IRB, ShadowTy, Alignment, /*isStore*/ false);
3711 setShadow(&I, IRB.CreateMaskedLoad(ShadowTy, ShadowPtr, Alignment, Mask,
3712 getShadow(PassThru), "_msmaskedld"));
3713
3714 if (!MS.TrackOrigins)
3715 return;
3716
3717 // Choose between PassThru's and the loaded value's origins.
3718 Value *MaskedPassThruShadow = IRB.CreateAnd(
3719 getShadow(PassThru), IRB.CreateSExt(IRB.CreateNeg(Mask), ShadowTy));
3720
3721 Value *NotNull = convertToBool(MaskedPassThruShadow, IRB, "_mscmp");
3722
3723 Value *PtrOrigin = IRB.CreateLoad(MS.OriginTy, OriginPtr);
3724 Value *Origin = IRB.CreateSelect(NotNull, getOrigin(PassThru), PtrOrigin);
3725
3726 setOrigin(&I, Origin);
3727 }
3728
3729 // Instrument BMI / BMI2 intrinsics.
3730 // All of these intrinsics are Z = I(X, Y)
3731 // where the types of all operands and the result match, and are either i32 or
3732 // i64. The following instrumentation happens to work for all of them:
3733 // Sz = I(Sx, Y) | (sext (Sy != 0))
3734 void handleBmiIntrinsic(IntrinsicInst &I) {
3735 IRBuilder<> IRB(&I);
3736 Type *ShadowTy = getShadowTy(&I);
3737
3738 // If any bit of the mask operand is poisoned, then the whole thing is.
3739 Value *SMask = getShadow(&I, 1);
3740 SMask = IRB.CreateSExt(IRB.CreateICmpNE(SMask, getCleanShadow(ShadowTy)),
3741 ShadowTy);
3742 // Apply the same intrinsic to the shadow of the first operand.
3743 Value *S = IRB.CreateCall(I.getCalledFunction(),
3744 {getShadow(&I, 0), I.getOperand(1)});
3745 S = IRB.CreateOr(SMask, S);
3746 setShadow(&I, S);
3747 setOriginForNaryOp(I);
3748 }
3749
3750 static SmallVector<int, 8> getPclmulMask(unsigned Width, bool OddElements) {
3752 for (unsigned X = OddElements ? 1 : 0; X < Width; X += 2) {
3753 Mask.append(2, X);
3754 }
3755 return Mask;
3756 }
3757
3758 // Instrument pclmul intrinsics.
3759 // These intrinsics operate either on odd or on even elements of the input
3760 // vectors, depending on the constant in the 3rd argument, ignoring the rest.
3761 // Replace the unused elements with copies of the used ones, ex:
3762 // (0, 1, 2, 3) -> (0, 0, 2, 2) (even case)
3763 // or
3764 // (0, 1, 2, 3) -> (1, 1, 3, 3) (odd case)
3765 // and then apply the usual shadow combining logic.
3766 void handlePclmulIntrinsic(IntrinsicInst &I) {
3767 IRBuilder<> IRB(&I);
3768 unsigned Width =
3769 cast<FixedVectorType>(I.getArgOperand(0)->getType())->getNumElements();
3770 assert(isa<ConstantInt>(I.getArgOperand(2)) &&
3771 "pclmul 3rd operand must be a constant");
3772 unsigned Imm = cast<ConstantInt>(I.getArgOperand(2))->getZExtValue();
3773 Value *Shuf0 = IRB.CreateShuffleVector(getShadow(&I, 0),
3774 getPclmulMask(Width, Imm & 0x01));
3775 Value *Shuf1 = IRB.CreateShuffleVector(getShadow(&I, 1),
3776 getPclmulMask(Width, Imm & 0x10));
3777 ShadowAndOriginCombiner SOC(this, IRB);
3778 SOC.Add(Shuf0, getOrigin(&I, 0));
3779 SOC.Add(Shuf1, getOrigin(&I, 1));
3780 SOC.Done(&I);
3781 }
3782
3783 // Instrument _mm_*_sd|ss intrinsics
3784 void handleUnarySdSsIntrinsic(IntrinsicInst &I) {
3785 IRBuilder<> IRB(&I);
3786 unsigned Width =
3787 cast<FixedVectorType>(I.getArgOperand(0)->getType())->getNumElements();
3788 Value *First = getShadow(&I, 0);
3789 Value *Second = getShadow(&I, 1);
3790 // First element of second operand, remaining elements of first operand
3792 Mask.push_back(Width);
3793 for (unsigned i = 1; i < Width; i++)
3794 Mask.push_back(i);
3795 Value *Shadow = IRB.CreateShuffleVector(First, Second, Mask);
3796
3797 setShadow(&I, Shadow);
3798 setOriginForNaryOp(I);
3799 }
3800
3801 void handleVtestIntrinsic(IntrinsicInst &I) {
3802 IRBuilder<> IRB(&I);
3803 Value *Shadow0 = getShadow(&I, 0);
3804 Value *Shadow1 = getShadow(&I, 1);
3805 Value *Or = IRB.CreateOr(Shadow0, Shadow1);
3806 Value *NZ = IRB.CreateICmpNE(Or, Constant::getNullValue(Or->getType()));
3807 Value *Scalar = convertShadowToScalar(NZ, IRB);
3808 Value *Shadow = IRB.CreateZExt(Scalar, getShadowTy(&I));
3809
3810 setShadow(&I, Shadow);
3811 setOriginForNaryOp(I);
3812 }
3813
3814 void handleBinarySdSsIntrinsic(IntrinsicInst &I) {
3815 IRBuilder<> IRB(&I);
3816 unsigned Width =
3817 cast<FixedVectorType>(I.getArgOperand(0)->getType())->getNumElements();
3818 Value *First = getShadow(&I, 0);
3819 Value *Second = getShadow(&I, 1);
3820 Value *OrShadow = IRB.CreateOr(First, Second);
3821 // First element of both OR'd together, remaining elements of first operand
3823 Mask.push_back(Width);
3824 for (unsigned i = 1; i < Width; i++)
3825 Mask.push_back(i);
3826 Value *Shadow = IRB.CreateShuffleVector(First, OrShadow, Mask);
3827
3828 setShadow(&I, Shadow);
3829 setOriginForNaryOp(I);
3830 }
3831
3832 // Instrument abs intrinsic.
3833 // handleUnknownIntrinsic can't handle it because of the last
3834 // is_int_min_poison argument which does not match the result type.
3835 void handleAbsIntrinsic(IntrinsicInst &I) {
3836 assert(I.getType()->isIntOrIntVectorTy());
3837 assert(I.getArgOperand(0)->getType() == I.getType());
3838
3839 // FIXME: Handle is_int_min_poison.
3840 IRBuilder<> IRB(&I);
3841 setShadow(&I, getShadow(&I, 0));
3842 setOrigin(&I, getOrigin(&I, 0));
3843 }
3844
3845 void handleIsFpClass(IntrinsicInst &I) {
3846 IRBuilder<> IRB(&I);
3847 Value *Shadow = getShadow(&I, 0);
3848 setShadow(&I, IRB.CreateICmpNE(Shadow, getCleanShadow(Shadow)));
3849 setOrigin(&I, getOrigin(&I, 0));
3850 }
3851
3852 void handleArithmeticWithOverflow(IntrinsicInst &I) {
3853 IRBuilder<> IRB(&I);
3854 Value *Shadow0 = getShadow(&I, 0);
3855 Value *Shadow1 = getShadow(&I, 1);
3856 Value *ShadowElt0 = IRB.CreateOr(Shadow0, Shadow1);
3857 Value *ShadowElt1 =
3858 IRB.CreateICmpNE(ShadowElt0, getCleanShadow(ShadowElt0));
3859
3860 Value *Shadow = PoisonValue::get(getShadowTy(&I));
3861 Shadow = IRB.CreateInsertValue(Shadow, ShadowElt0, 0);
3862 Shadow = IRB.CreateInsertValue(Shadow, ShadowElt1, 1);
3863
3864 setShadow(&I, Shadow);
3865 setOriginForNaryOp(I);
3866 }
3867
3868 void visitIntrinsicInst(IntrinsicInst &I) {
3869 switch (I.getIntrinsicID()) {
3870 case Intrinsic::uadd_with_overflow:
3871 case Intrinsic::sadd_with_overflow:
3872 case Intrinsic::usub_with_overflow:
3873 case Intrinsic::ssub_with_overflow:
3874 case Intrinsic::umul_with_overflow:
3875 case Intrinsic::smul_with_overflow:
3876 handleArithmeticWithOverflow(I);
3877 break;
3878 case Intrinsic::abs:
3879 handleAbsIntrinsic(I);
3880 break;
3881 case Intrinsic::is_fpclass:
3882 handleIsFpClass(I);
3883 break;
3884 case Intrinsic::lifetime_start:
3885 handleLifetimeStart(I);
3886 break;
3887 case Intrinsic::launder_invariant_group:
3888 case Intrinsic::strip_invariant_group:
3889 handleInvariantGroup(I);
3890 break;
3891 case Intrinsic::bswap:
3892 handleBswap(I);
3893 break;
3894 case Intrinsic::ctlz:
3895 case Intrinsic::cttz:
3896 handleCountZeroes(I);
3897 break;
3898 case Intrinsic::masked_compressstore:
3899 handleMaskedCompressStore(I);
3900 break;
3901 case Intrinsic::masked_expandload:
3902 handleMaskedExpandLoad(I);
3903 break;
3904 case Intrinsic::masked_gather:
3905 handleMaskedGather(I);
3906 break;
3907 case Intrinsic::masked_scatter:
3908 handleMaskedScatter(I);
3909 break;
3910 case Intrinsic::masked_store:
3911 handleMaskedStore(I);
3912 break;
3913 case Intrinsic::masked_load:
3914 handleMaskedLoad(I);
3915 break;
3916 case Intrinsic::vector_reduce_and:
3917 handleVectorReduceAndIntrinsic(I);
3918 break;
3919 case Intrinsic::vector_reduce_or:
3920 handleVectorReduceOrIntrinsic(I);
3921 break;
3922 case Intrinsic::vector_reduce_add:
3923 case Intrinsic::vector_reduce_xor:
3924 case Intrinsic::vector_reduce_mul:
3925 handleVectorReduceIntrinsic(I);
3926 break;
3927 case Intrinsic::x86_sse_stmxcsr:
3928 handleStmxcsr(I);
3929 break;
3930 case Intrinsic::x86_sse_ldmxcsr:
3931 handleLdmxcsr(I);
3932 break;
3933 case Intrinsic::x86_avx512_vcvtsd2usi64:
3934 case Intrinsic::x86_avx512_vcvtsd2usi32:
3935 case Intrinsic::x86_avx512_vcvtss2usi64:
3936 case Intrinsic::x86_avx512_vcvtss2usi32:
3937 case Intrinsic::x86_avx512_cvttss2usi64:
3938 case Intrinsic::x86_avx512_cvttss2usi:
3939 case Intrinsic::x86_avx512_cvttsd2usi64:
3940 case Intrinsic::x86_avx512_cvttsd2usi:
3941 case Intrinsic::x86_avx512_cvtusi2ss:
3942 case Intrinsic::x86_avx512_cvtusi642sd:
3943 case Intrinsic::x86_avx512_cvtusi642ss:
3944 handleVectorConvertIntrinsic(I, 1, true);
3945 break;
3946 case Intrinsic::x86_sse2_cvtsd2si64:
3947 case Intrinsic::x86_sse2_cvtsd2si:
3948 case Intrinsic::x86_sse2_cvtsd2ss:
3949 case Intrinsic::x86_sse2_cvttsd2si64:
3950 case Intrinsic::x86_sse2_cvttsd2si:
3951 case Intrinsic::x86_sse_cvtss2si64:
3952 case Intrinsic::x86_sse_cvtss2si:
3953 case Intrinsic::x86_sse_cvttss2si64:
3954 case Intrinsic::x86_sse_cvttss2si:
3955 handleVectorConvertIntrinsic(I, 1);
3956 break;
3957 case Intrinsic::x86_sse_cvtps2pi:
3958 case Intrinsic::x86_sse_cvttps2pi:
3959 handleVectorConvertIntrinsic(I, 2);
3960 break;
3961
3962 case Intrinsic::x86_avx512_psll_w_512:
3963 case Intrinsic::x86_avx512_psll_d_512:
3964 case Intrinsic::x86_avx512_psll_q_512:
3965 case Intrinsic::x86_avx512_pslli_w_512:
3966 case Intrinsic::x86_avx512_pslli_d_512:
3967 case Intrinsic::x86_avx512_pslli_q_512:
3968 case Intrinsic::x86_avx512_psrl_w_512:
3969 case Intrinsic::x86_avx512_psrl_d_512:
3970 case Intrinsic::x86_avx512_psrl_q_512:
3971 case Intrinsic::x86_avx512_psra_w_512:
3972 case Intrinsic::x86_avx512_psra_d_512:
3973 case Intrinsic::x86_avx512_psra_q_512:
3974 case Intrinsic::x86_avx512_psrli_w_512:
3975 case Intrinsic::x86_avx512_psrli_d_512:
3976 case Intrinsic::x86_avx512_psrli_q_512:
3977 case Intrinsic::x86_avx512_psrai_w_512:
3978 case Intrinsic::x86_avx512_psrai_d_512:
3979 case Intrinsic::x86_avx512_psrai_q_512:
3980 case Intrinsic::x86_avx512_psra_q_256:
3981 case Intrinsic::x86_avx512_psra_q_128:
3982 case Intrinsic::x86_avx512_psrai_q_256:
3983 case Intrinsic::x86_avx512_psrai_q_128:
3984 case Intrinsic::x86_avx2_psll_w:
3985 case Intrinsic::x86_avx2_psll_d:
3986 case Intrinsic::x86_avx2_psll_q:
3987 case Intrinsic::x86_avx2_pslli_w:
3988 case Intrinsic::x86_avx2_pslli_d:
3989 case Intrinsic::x86_avx2_pslli_q:
3990 case Intrinsic::x86_avx2_psrl_w:
3991 case Intrinsic::x86_avx2_psrl_d:
3992 case Intrinsic::x86_avx2_psrl_q:
3993 case Intrinsic::x86_avx2_psra_w:
3994 case Intrinsic::x86_avx2_psra_d:
3995 case Intrinsic::x86_avx2_psrli_w:
3996 case Intrinsic::x86_avx2_psrli_d:
3997 case Intrinsic::x86_avx2_psrli_q:
3998 case Intrinsic::x86_avx2_psrai_w:
3999 case Intrinsic::x86_avx2_psrai_d:
4000 case Intrinsic::x86_sse2_psll_w:
4001 case Intrinsic::x86_sse2_psll_d:
4002 case Intrinsic::x86_sse2_psll_q:
4003 case Intrinsic::x86_sse2_pslli_w:
4004 case Intrinsic::x86_sse2_pslli_d:
4005 case Intrinsic::x86_sse2_pslli_q:
4006 case Intrinsic::x86_sse2_psrl_w:
4007 case Intrinsic::x86_sse2_psrl_d:
4008 case Intrinsic::x86_sse2_psrl_q:
4009 case Intrinsic::x86_sse2_psra_w:
4010 case Intrinsic::x86_sse2_psra_d:
4011 case Intrinsic::x86_sse2_psrli_w:
4012 case Intrinsic::x86_sse2_psrli_d:
4013 case Intrinsic::x86_sse2_psrli_q:
4014 case Intrinsic::x86_sse2_psrai_w:
4015 case Intrinsic::x86_sse2_psrai_d:
4016 case Intrinsic::x86_mmx_psll_w:
4017 case Intrinsic::x86_mmx_psll_d:
4018 case Intrinsic::x86_mmx_psll_q:
4019 case Intrinsic::x86_mmx_pslli_w:
4020 case Intrinsic::x86_mmx_pslli_d:
4021 case Intrinsic::x86_mmx_pslli_q:
4022 case Intrinsic::x86_mmx_psrl_w:
4023 case Intrinsic::x86_mmx_psrl_d:
4024 case Intrinsic::x86_mmx_psrl_q:
4025 case Intrinsic::x86_mmx_psra_w:
4026 case Intrinsic::x86_mmx_psra_d:
4027 case Intrinsic::x86_mmx_psrli_w:
4028 case Intrinsic::x86_mmx_psrli_d:
4029 case Intrinsic::x86_mmx_psrli_q:
4030 case Intrinsic::x86_mmx_psrai_w:
4031 case Intrinsic::x86_mmx_psrai_d:
4032 handleVectorShiftIntrinsic(I, /* Variable */ false);
4033 break;
4034 case Intrinsic::x86_avx2_psllv_d:
4035 case Intrinsic::x86_avx2_psllv_d_256:
4036 case Intrinsic::x86_avx512_psllv_d_512:
4037 case Intrinsic::x86_avx2_psllv_q:
4038 case Intrinsic::x86_avx2_psllv_q_256:
4039 case Intrinsic::x86_avx512_psllv_q_512:
4040 case Intrinsic::x86_avx2_psrlv_d:
4041 case Intrinsic::x86_avx2_psrlv_d_256:
4042 case Intrinsic::x86_avx512_psrlv_d_512:
4043 case Intrinsic::x86_avx2_psrlv_q:
4044 case Intrinsic::x86_avx2_psrlv_q_256:
4045 case Intrinsic::x86_avx512_psrlv_q_512:
4046 case Intrinsic::x86_avx2_psrav_d:
4047 case Intrinsic::x86_avx2_psrav_d_256:
4048 case Intrinsic::x86_avx512_psrav_d_512:
4049 case Intrinsic::x86_avx512_psrav_q_128:
4050 case Intrinsic::x86_avx512_psrav_q_256:
4051 case Intrinsic::x86_avx512_psrav_q_512:
4052 handleVectorShiftIntrinsic(I, /* Variable */ true);
4053 break;
4054
4055 case Intrinsic::x86_sse2_packsswb_128:
4056 case Intrinsic::x86_sse2_packssdw_128:
4057 case Intrinsic::x86_sse2_packuswb_128:
4058 case Intrinsic::x86_sse41_packusdw:
4059 case Intrinsic::x86_avx2_packsswb:
4060 case Intrinsic::x86_avx2_packssdw:
4061 case Intrinsic::x86_avx2_packuswb:
4062 case Intrinsic::x86_avx2_packusdw:
4063 handleVectorPackIntrinsic(I);
4064 break;
4065
4066 case Intrinsic::x86_sse41_pblendvb:
4067 case Intrinsic::x86_sse41_blendvpd:
4068 case Intrinsic::x86_sse41_blendvps:
4069 case Intrinsic::x86_avx_blendv_pd_256:
4070 case Intrinsic::x86_avx_blendv_ps_256:
4071 case Intrinsic::x86_avx2_pblendvb:
4072 handleBlendvIntrinsic(I);
4073 break;
4074
4075 case Intrinsic::x86_avx_dp_ps_256:
4076 case Intrinsic::x86_sse41_dppd:
4077 case Intrinsic::x86_sse41_dpps:
4078 handleDppIntrinsic(I);
4079 break;
4080
4081 case Intrinsic::x86_mmx_packsswb:
4082 case Intrinsic::x86_mmx_packuswb:
4083 handleVectorPackIntrinsic(I, 16);
4084 break;
4085
4086 case Intrinsic::x86_mmx_packssdw:
4087 handleVectorPackIntrinsic(I, 32);
4088 break;
4089
4090 case Intrinsic::x86_mmx_psad_bw:
4091 case Intrinsic::x86_sse2_psad_bw:
4092 case Intrinsic::x86_avx2_psad_bw:
4093 handleVectorSadIntrinsic(I);
4094 break;
4095
4096 case Intrinsic::x86_sse2_pmadd_wd:
4097 case Intrinsic::x86_avx2_pmadd_wd:
4098 case Intrinsic::x86_ssse3_pmadd_ub_sw_128:
4099 case Intrinsic::x86_avx2_pmadd_ub_sw:
4100 handleVectorPmaddIntrinsic(I);
4101 break;
4102
4103 case Intrinsic::x86_ssse3_pmadd_ub_sw:
4104 handleVectorPmaddIntrinsic(I, 8);
4105 break;
4106
4107 case Intrinsic::x86_mmx_pmadd_wd:
4108 handleVectorPmaddIntrinsic(I, 16);
4109 break;
4110
4111 case Intrinsic::x86_sse_cmp_ss:
4112 case Intrinsic::x86_sse2_cmp_sd:
4113 case Intrinsic::x86_sse_comieq_ss:
4114 case Intrinsic::x86_sse_comilt_ss:
4115 case Intrinsic::x86_sse_comile_ss:
4116 case Intrinsic::x86_sse_comigt_ss:
4117 case Intrinsic::x86_sse_comige_ss:
4118 case Intrinsic::x86_sse_comineq_ss:
4119 case Intrinsic::x86_sse_ucomieq_ss:
4120 case Intrinsic::x86_sse_ucomilt_ss:
4121 case Intrinsic::x86_sse_ucomile_ss:
4122 case Intrinsic::x86_sse_ucomigt_ss:
4123 case Intrinsic::x86_sse_ucomige_ss:
4124 case Intrinsic::x86_sse_ucomineq_ss:
4125 case Intrinsic::x86_sse2_comieq_sd:
4126 case Intrinsic::x86_sse2_comilt_sd:
4127 case Intrinsic::x86_sse2_comile_sd:
4128 case Intrinsic::x86_sse2_comigt_sd:
4129 case Intrinsic::x86_sse2_comige_sd:
4130 case Intrinsic::x86_sse2_comineq_sd:
4131 case Intrinsic::x86_sse2_ucomieq_sd:
4132 case Intrinsic::x86_sse2_ucomilt_sd:
4133 case Intrinsic::x86_sse2_ucomile_sd:
4134 case Intrinsic::x86_sse2_ucomigt_sd:
4135 case Intrinsic::x86_sse2_ucomige_sd:
4136 case Intrinsic::x86_sse2_ucomineq_sd:
4137 handleVectorCompareScalarIntrinsic(I);
4138 break;
4139
4140 case Intrinsic::x86_avx_cmp_pd_256:
4141 case Intrinsic::x86_avx_cmp_ps_256:
4142 case Intrinsic::x86_sse2_cmp_pd:
4143 case Intrinsic::x86_sse_cmp_ps:
4144 handleVectorComparePackedIntrinsic(I);
4145 break;
4146
4147 case Intrinsic::x86_bmi_bextr_32:
4148 case Intrinsic::x86_bmi_bextr_64:
4149 case Intrinsic::x86_bmi_bzhi_32:
4150 case Intrinsic::x86_bmi_bzhi_64:
4151 case Intrinsic::x86_bmi_pdep_32:
4152 case Intrinsic::x86_bmi_pdep_64:
4153 case Intrinsic::x86_bmi_pext_32:
4154 case Intrinsic::x86_bmi_pext_64:
4155 handleBmiIntrinsic(I);
4156 break;
4157
4158 case Intrinsic::x86_pclmulqdq:
4159 case Intrinsic::x86_pclmulqdq_256:
4160 case Intrinsic::x86_pclmulqdq_512:
4161 handlePclmulIntrinsic(I);
4162 break;
4163
4164 case Intrinsic::x86_sse41_round_sd:
4165 case Intrinsic::x86_sse41_round_ss:
4166 handleUnarySdSsIntrinsic(I);
4167 break;
4168 case Intrinsic::x86_sse2_max_sd:
4169 case Intrinsic::x86_sse_max_ss:
4170 case Intrinsic::x86_sse2_min_sd:
4171 case Intrinsic::x86_sse_min_ss:
4172 handleBinarySdSsIntrinsic(I);
4173 break;
4174
4175 case Intrinsic::x86_avx_vtestc_pd:
4176 case Intrinsic::x86_avx_vtestc_pd_256:
4177 case Intrinsic::x86_avx_vtestc_ps:
4178 case Intrinsic::x86_avx_vtestc_ps_256:
4179 case Intrinsic::x86_avx_vtestnzc_pd:
4180 case Intrinsic::x86_avx_vtestnzc_pd_256:
4181 case Intrinsic::x86_avx_vtestnzc_ps:
4182 case Intrinsic::x86_avx_vtestnzc_ps_256:
4183 case Intrinsic::x86_avx_vtestz_pd:
4184 case Intrinsic::x86_avx_vtestz_pd_256:
4185 case Intrinsic::x86_avx_vtestz_ps:
4186 case Intrinsic::x86_avx_vtestz_ps_256:
4187 case Intrinsic::x86_avx_ptestc_256:
4188 case Intrinsic::x86_avx_ptestnzc_256:
4189 case Intrinsic::x86_avx_ptestz_256:
4190 case Intrinsic::x86_sse41_ptestc:
4191 case Intrinsic::x86_sse41_ptestnzc:
4192 case Intrinsic::x86_sse41_ptestz:
4193 handleVtestIntrinsic(I);
4194 break;
4195
4196 case Intrinsic::fshl:
4197 case Intrinsic::fshr:
4198 handleFunnelShift(I);
4199 break;
4200
4201 case Intrinsic::is_constant:
4202 // The result of llvm.is.constant() is always defined.
4203 setShadow(&I, getCleanShadow(&I));
4204 setOrigin(&I, getCleanOrigin());
4205 break;
4206
4207 default:
4208 if (!handleUnknownIntrinsic(I))
4209 visitInstruction(I);
4210 break;
4211 }
4212 }
4213
4214 void visitLibAtomicLoad(CallBase &CB) {
4215 // Since we use getNextNode here, we can't have CB terminate the BB.
4216 assert(isa<CallInst>(CB));
4217
4218 IRBuilder<> IRB(&CB);
4219 Value *Size = CB.getArgOperand(0);
4220 Value *SrcPtr = CB.getArgOperand(1);
4221 Value *DstPtr = CB.getArgOperand(2);
4222 Value *Ordering = CB.getArgOperand(3);
4223 // Convert the call to have at least Acquire ordering to make sure
4224 // the shadow operations aren't reordered before it.
4225 Value *NewOrdering =
4226 IRB.CreateExtractElement(makeAddAcquireOrderingTable(IRB), Ordering);
4227 CB.setArgOperand(3, NewOrdering);
4228
4229 NextNodeIRBuilder NextIRB(&CB);
4230 Value *SrcShadowPtr, *SrcOriginPtr;
4231 std::tie(SrcShadowPtr, SrcOriginPtr) =
4232 getShadowOriginPtr(SrcPtr, NextIRB, NextIRB.getInt8Ty(), Align(1),
4233 /*isStore*/ false);
4234 Value *DstShadowPtr =
4235 getShadowOriginPtr(DstPtr, NextIRB, NextIRB.getInt8Ty(), Align(1),
4236 /*isStore*/ true)
4237 .first;
4238
4239 NextIRB.CreateMemCpy(DstShadowPtr, Align(1), SrcShadowPtr, Align(1), Size);
4240 if (MS.TrackOrigins) {
4241 Value *SrcOrigin = NextIRB.CreateAlignedLoad(MS.OriginTy, SrcOriginPtr,
4243 Value *NewOrigin = updateOrigin(SrcOrigin, NextIRB);
4244 NextIRB.CreateCall(MS.MsanSetOriginFn, {DstPtr, Size, NewOrigin});
4245 }
4246 }
4247
4248 void visitLibAtomicStore(CallBase &CB) {
4249 IRBuilder<> IRB(&CB);
4250 Value *Size = CB.getArgOperand(0);
4251 Value *DstPtr = CB.getArgOperand(2);
4252 Value *Ordering = CB.getArgOperand(3);
4253 // Convert the call to have at least Release ordering to make sure
4254 // the shadow operations aren't reordered after it.
4255 Value *NewOrdering =
4256 IRB.CreateExtractElement(makeAddReleaseOrderingTable(IRB), Ordering);
4257 CB.setArgOperand(3, NewOrdering);
4258
4259 Value *DstShadowPtr =
4260 getShadowOriginPtr(DstPtr, IRB, IRB.getInt8Ty(), Align(1),
4261 /*isStore*/ true)
4262 .first;
4263
4264 // Atomic store always paints clean shadow/origin. See file header.
4265 IRB.CreateMemSet(DstShadowPtr, getCleanShadow(IRB.getInt8Ty()), Size,
4266 Align(1));
4267 }
4268
4269 void visitCallBase(CallBase &CB) {
4270 assert(!CB.getMetadata(LLVMContext::MD_nosanitize));
4271 if (CB.isInlineAsm()) {
4272 // For inline asm (either a call to asm function, or callbr instruction),
4273 // do the usual thing: check argument shadow and mark all outputs as
4274 // clean. Note that any side effects of the inline asm that are not
4275 // immediately visible in its constraints are not handled.
4277 visitAsmInstruction(CB);
4278 else
4279 visitInstruction(CB);
4280 return;
4281 }
4282 LibFunc LF;
4283 if (TLI->getLibFunc(CB, LF)) {
4284 // libatomic.a functions need to have special handling because there isn't
4285 // a good way to intercept them or compile the library with
4286 // instrumentation.
4287 switch (LF) {
4288 case LibFunc_atomic_load:
4289 if (!isa<CallInst>(CB)) {
4290 llvm::errs() << "MSAN -- cannot instrument invoke of libatomic load."
4291 "Ignoring!\n";
4292 break;
4293 }
4294 visitLibAtomicLoad(CB);
4295 return;
4296 case LibFunc_atomic_store:
4297 visitLibAtomicStore(CB);
4298 return;
4299 default:
4300 break;
4301 }
4302 }
4303
4304 if (auto *Call = dyn_cast<CallInst>(&CB)) {
4305 assert(!isa<IntrinsicInst>(Call) && "intrinsics are handled elsewhere");
4306
4307 // We are going to insert code that relies on the fact that the callee
4308 // will become a non-readonly function after it is instrumented by us. To
4309 // prevent this code from being optimized out, mark that function
4310 // non-readonly in advance.
4311 // TODO: We can likely do better than dropping memory() completely here.
4313 B.addAttribute(Attribute::Memory).addAttribute(Attribute::Speculatable);
4314
4315 Call->removeFnAttrs(B);
4316 if (Function *Func = Call->getCalledFunction()) {
4317 Func->removeFnAttrs(B);
4318 }
4319
4321 }
4322 IRBuilder<> IRB(&CB);
4323 bool MayCheckCall = MS.EagerChecks;
4324 if (Function *Func = CB.getCalledFunction()) {
4325 // __sanitizer_unaligned_{load,store} functions may be called by users
4326 // and always expects shadows in the TLS. So don't check them.
4327 MayCheckCall &= !Func->getName().starts_with("__sanitizer_unaligned_");
4328 }
4329
4330 unsigned ArgOffset = 0;
4331 LLVM_DEBUG(dbgs() << " CallSite: " << CB << "\n");
4332 for (const auto &[i, A] : llvm::enumerate(CB.args())) {
4333 if (!A->getType()->isSized()) {
4334 LLVM_DEBUG(dbgs() << "Arg " << i << " is not sized: " << CB << "\n");
4335 continue;
4336 }
4337
4338 if (A->getType()->isScalableTy()) {
4339 LLVM_DEBUG(dbgs() << "Arg " << i << " is vscale: " << CB << "\n");
4340 // Handle as noundef, but don't reserve tls slots.
4341 insertShadowCheck(A, &CB);
4342 continue;
4343 }
4344
4345 unsigned Size = 0;
4346 const DataLayout &DL = F.getDataLayout();
4347
4348 bool ByVal = CB.paramHasAttr(i, Attribute::ByVal);
4349 bool NoUndef = CB.paramHasAttr(i, Attribute::NoUndef);
4350 bool EagerCheck = MayCheckCall && !ByVal && NoUndef;
4351
4352 if (EagerCheck) {
4353 insertShadowCheck(A, &CB);
4354 Size = DL.getTypeAllocSize(A->getType());
4355 } else {
4356 Value *Store = nullptr;
4357 // Compute the Shadow for arg even if it is ByVal, because
4358 // in that case getShadow() will copy the actual arg shadow to
4359 // __msan_param_tls.
4360 Value *ArgShadow = getShadow(A);
4361 Value *ArgShadowBase = getShadowPtrForArgument(IRB, ArgOffset);
4362 LLVM_DEBUG(dbgs() << " Arg#" << i << ": " << *A
4363 << " Shadow: " << *ArgShadow << "\n");
4364 if (ByVal) {
4365 // ByVal requires some special handling as it's too big for a single
4366 // load
4367 assert(A->getType()->isPointerTy() &&
4368 "ByVal argument is not a pointer!");
4369 Size = DL.getTypeAllocSize(CB.getParamByValType(i));
4370 if (ArgOffset + Size > kParamTLSSize)
4371 break;
4372 const MaybeAlign ParamAlignment(CB.getParamAlign(i));
4373 MaybeAlign Alignment = std::nullopt;
4374 if (ParamAlignment)
4375 Alignment = std::min(*ParamAlignment, kShadowTLSAlignment);
4376 Value *AShadowPtr, *AOriginPtr;
4377 std::tie(AShadowPtr, AOriginPtr) =
4378 getShadowOriginPtr(A, IRB, IRB.getInt8Ty(), Alignment,
4379 /*isStore*/ false);
4380 if (!PropagateShadow) {
4381 Store = IRB.CreateMemSet(ArgShadowBase,
4383 Size, Alignment);
4384 } else {
4385 Store = IRB.CreateMemCpy(ArgShadowBase, Alignment, AShadowPtr,
4386 Alignment, Size);
4387 if (MS.TrackOrigins) {
4388 Value *ArgOriginBase = getOriginPtrForArgument(IRB, ArgOffset);
4389 // FIXME: OriginSize should be:
4390 // alignTo(A % kMinOriginAlignment + Size, kMinOriginAlignment)
4391 unsigned OriginSize = alignTo(Size, kMinOriginAlignment);
4392 IRB.CreateMemCpy(
4393 ArgOriginBase,
4394 /* by origin_tls[ArgOffset] */ kMinOriginAlignment,
4395 AOriginPtr,
4396 /* by getShadowOriginPtr */ kMinOriginAlignment, OriginSize);
4397 }
4398 }
4399 } else {
4400 // Any other parameters mean we need bit-grained tracking of uninit
4401 // data
4402 Size = DL.getTypeAllocSize(A->getType());
4403 if (ArgOffset + Size > kParamTLSSize)
4404 break;
4405 Store = IRB.CreateAlignedStore(ArgShadow, ArgShadowBase,
4407 Constant *Cst = dyn_cast<Constant>(ArgShadow);
4408 if (MS.TrackOrigins && !(Cst && Cst->isNullValue())) {
4409 IRB.CreateStore(getOrigin(A),
4410 getOriginPtrForArgument(IRB, ArgOffset));
4411 }
4412 }
4413 (void)Store;
4414 assert(Store != nullptr);
4415 LLVM_DEBUG(dbgs() << " Param:" << *Store << "\n");
4416 }
4417 assert(Size != 0);
4418 ArgOffset += alignTo(Size, kShadowTLSAlignment);
4419 }
4420 LLVM_DEBUG(dbgs() << " done with call args\n");
4421
4422 FunctionType *FT = CB.getFunctionType();
4423 if (FT->isVarArg()) {
4424 VAHelper->visitCallBase(CB, IRB);
4425 }
4426
4427 // Now, get the shadow for the RetVal.
4428 if (!CB.getType()->isSized())
4429 return;
4430 // Don't emit the epilogue for musttail call returns.
4431 if (isa<CallInst>(CB) && cast<CallInst>(CB).isMustTailCall())
4432 return;
4433
4434 if (MayCheckCall && CB.hasRetAttr(Attribute::NoUndef)) {
4435 setShadow(&CB, getCleanShadow(&CB));
4436 setOrigin(&CB, getCleanOrigin());
4437 return;
4438 }
4439
4440 IRBuilder<> IRBBefore(&CB);
4441 // Until we have full dynamic coverage, make sure the retval shadow is 0.
4442 Value *Base = getShadowPtrForRetval(IRBBefore);
4443 IRBBefore.CreateAlignedStore(getCleanShadow(&CB), Base,
4445 BasicBlock::iterator NextInsn;
4446 if (isa<CallInst>(CB)) {
4447 NextInsn = ++CB.getIterator();
4448 assert(NextInsn != CB.getParent()->end());
4449 } else {
4450 BasicBlock *NormalDest = cast<InvokeInst>(CB).getNormalDest();
4451 if (!NormalDest->getSinglePredecessor()) {
4452 // FIXME: this case is tricky, so we are just conservative here.
4453 // Perhaps we need to split the edge between this BB and NormalDest,
4454 // but a naive attempt to use SplitEdge leads to a crash.
4455 setShadow(&CB, getCleanShadow(&CB));
4456 setOrigin(&CB, getCleanOrigin());
4457 return;
4458 }
4459 // FIXME: NextInsn is likely in a basic block that has not been visited
4460 // yet. Anything inserted there will be instrumented by MSan later!
4461 NextInsn = NormalDest->getFirstInsertionPt();
4462 assert(NextInsn != NormalDest->end() &&
4463 "Could not find insertion point for retval shadow load");
4464 }
4465 IRBuilder<> IRBAfter(&*NextInsn);
4466 Value *RetvalShadow = IRBAfter.CreateAlignedLoad(
4467 getShadowTy(&CB), getShadowPtrForRetval(IRBAfter),
4468 kShadowTLSAlignment, "_msret");
4469 setShadow(&CB, RetvalShadow);
4470 if (MS.TrackOrigins)
4471 setOrigin(&CB, IRBAfter.CreateLoad(MS.OriginTy,
4472 getOriginPtrForRetval()));
4473 }
4474
4475 bool isAMustTailRetVal(Value *RetVal) {
4476 if (auto *I = dyn_cast<BitCastInst>(RetVal)) {
4477 RetVal = I->getOperand(0);
4478 }
4479 if (auto *I = dyn_cast<CallInst>(RetVal)) {
4480 return I->isMustTailCall();
4481 }
4482 return false;
4483 }
4484
4485 void visitReturnInst(ReturnInst &I) {
4486 IRBuilder<> IRB(&I);
4487 Value *RetVal = I.getReturnValue();
4488 if (!RetVal)
4489 return;
4490 // Don't emit the epilogue for musttail call returns.
4491 if (isAMustTailRetVal(RetVal))
4492 return;
4493 Value *ShadowPtr = getShadowPtrForRetval(IRB);
4494 bool HasNoUndef = F.hasRetAttribute(Attribute::NoUndef);
4495 bool StoreShadow = !(MS.EagerChecks && HasNoUndef);
4496 // FIXME: Consider using SpecialCaseList to specify a list of functions that
4497 // must always return fully initialized values. For now, we hardcode "main".
4498 bool EagerCheck = (MS.EagerChecks && HasNoUndef) || (F.getName() == "main");
4499
4500 Value *Shadow = getShadow(RetVal);
4501 bool StoreOrigin = true;
4502 if (EagerCheck) {
4503 insertShadowCheck(RetVal, &I);
4504 Shadow = getCleanShadow(RetVal);
4505 StoreOrigin = false;
4506 }
4507
4508 // The caller may still expect information passed over TLS if we pass our
4509 // check
4510 if (StoreShadow) {
4511 IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment);
4512 if (MS.TrackOrigins && StoreOrigin)
4513 IRB.CreateStore(getOrigin(RetVal), getOriginPtrForRetval());
4514 }
4515 }
4516
4517 void visitPHINode(PHINode &I) {
4518 IRBuilder<> IRB(&I);
4519 if (!PropagateShadow) {
4520 setShadow(&I, getCleanShadow(&I));
4521 setOrigin(&I, getCleanOrigin());
4522 return;
4523 }
4524
4525 ShadowPHINodes.push_back(&I);
4526 setShadow(&I, IRB.CreatePHI(getShadowTy(&I), I.getNumIncomingValues(),
4527 "_msphi_s"));
4528 if (MS.TrackOrigins)
4529 setOrigin(
4530 &I, IRB.CreatePHI(MS.OriginTy, I.getNumIncomingValues(), "_msphi_o"));
4531 }
4532
4533 Value *getLocalVarIdptr(AllocaInst &I) {
4534 ConstantInt *IntConst =
4535 ConstantInt::get(Type::getInt32Ty((*F.getParent()).getContext()), 0);
4536 return new GlobalVariable(*F.getParent(), IntConst->getType(),
4537 /*isConstant=*/false, GlobalValue::PrivateLinkage,
4538 IntConst);
4539 }
4540
4541 Value *getLocalVarDescription(AllocaInst &I) {
4542 return createPrivateConstGlobalForString(*F.getParent(), I.getName());
4543 }
4544
4545 void poisonAllocaUserspace(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
4546 if (PoisonStack && ClPoisonStackWithCall) {
4547 IRB.CreateCall(MS.MsanPoisonStackFn, {&I, Len});
4548 } else {
4549 Value *ShadowBase, *OriginBase;
4550 std::tie(ShadowBase, OriginBase) = getShadowOriginPtr(
4551 &I, IRB, IRB.getInt8Ty(), Align(1), /*isStore*/ true);
4552
4553 Value *PoisonValue = IRB.getInt8(PoisonStack ? ClPoisonStackPattern : 0);
4554 IRB.CreateMemSet(ShadowBase, PoisonValue, Len, I.getAlign());
4555 }
4556
4557 if (PoisonStack && MS.TrackOrigins) {
4558 Value *Idptr = getLocalVarIdptr(I);
4559 if (ClPrintStackNames) {
4560 Value *Descr = getLocalVarDescription(I);
4561 IRB.CreateCall(MS.MsanSetAllocaOriginWithDescriptionFn,
4562 {&I, Len, Idptr, Descr});
4563 } else {
4564 IRB.CreateCall(MS.MsanSetAllocaOriginNoDescriptionFn, {&I, Len, Idptr});
4565 }
4566 }
4567 }
4568
4569 void poisonAllocaKmsan(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
4570 Value *Descr = getLocalVarDescription(I);
4571 if (PoisonStack) {
4572 IRB.CreateCall(MS.MsanPoisonAllocaFn, {&I, Len, Descr});
4573 } else {
4574 IRB.CreateCall(MS.MsanUnpoisonAllocaFn, {&I, Len});
4575 }
4576 }
4577
4578 void instrumentAlloca(AllocaInst &I, Instruction *InsPoint = nullptr) {
4579 if (!InsPoint)
4580 InsPoint = &I;
4581 NextNodeIRBuilder IRB(InsPoint);
4582 const DataLayout &DL = F.getDataLayout();
4583 TypeSize TS = DL.getTypeAllocSize(I.getAllocatedType());
4584 Value *Len = IRB.CreateTypeSize(MS.IntptrTy, TS);
4585 if (I.isArrayAllocation())
4586 Len = IRB.CreateMul(Len,
4587 IRB.CreateZExtOrTrunc(I.getArraySize(), MS.IntptrTy));
4588
4589 if (MS.CompileKernel)
4590 poisonAllocaKmsan(I, IRB, Len);
4591 else
4592 poisonAllocaUserspace(I, IRB, Len);
4593 }
4594
4595 void visitAllocaInst(AllocaInst &I) {
4596 setShadow(&I, getCleanShadow(&I));
4597 setOrigin(&I, getCleanOrigin());
4598 // We'll get to this alloca later unless it's poisoned at the corresponding
4599 // llvm.lifetime.start.
4600 AllocaSet.insert(&I);
4601 }
4602
4603 void visitSelectInst(SelectInst &I) {
4604 // a = select b, c, d
4605 Value *B = I.getCondition();
4606 Value *C = I.getTrueValue();
4607 Value *D = I.getFalseValue();
4608
4609 handleSelectLikeInst(I, B, C, D);
4610 }
4611
4612 void handleSelectLikeInst(Instruction &I, Value *B, Value *C, Value *D) {
4613 IRBuilder<> IRB(&I);
4614
4615 Value *Sb = getShadow(B);
4616 Value *Sc = getShadow(C);
4617 Value *Sd = getShadow(D);
4618
4619 Value *Ob = MS.TrackOrigins ? getOrigin(B) : nullptr;
4620 Value *Oc = MS.TrackOrigins ? getOrigin(C) : nullptr;
4621 Value *Od = MS.TrackOrigins ? getOrigin(D) : nullptr;
4622
4623 // Result shadow if condition shadow is 0.
4624 Value *Sa0 = IRB.CreateSelect(B, Sc, Sd);
4625 Value *Sa1;
4626 if (I.getType()->isAggregateType()) {
4627 // To avoid "sign extending" i1 to an arbitrary aggregate type, we just do
4628 // an extra "select". This results in much more compact IR.
4629 // Sa = select Sb, poisoned, (select b, Sc, Sd)
4630 Sa1 = getPoisonedShadow(getShadowTy(I.getType()));
4631 } else {
4632 // Sa = select Sb, [ (c^d) | Sc | Sd ], [ b ? Sc : Sd ]
4633 // If Sb (condition is poisoned), look for bits in c and d that are equal
4634 // and both unpoisoned.
4635 // If !Sb (condition is unpoisoned), simply pick one of Sc and Sd.
4636
4637 // Cast arguments to shadow-compatible type.
4638 C = CreateAppToShadowCast(IRB, C);
4639 D = CreateAppToShadowCast(IRB, D);
4640
4641 // Result shadow if condition shadow is 1.
4642 Sa1 = IRB.CreateOr({IRB.CreateXor(C, D), Sc, Sd});
4643 }
4644 Value *Sa = IRB.CreateSelect(Sb, Sa1, Sa0, "_msprop_select");
4645 setShadow(&I, Sa);
4646 if (MS.TrackOrigins) {
4647 // Origins are always i32, so any vector conditions must be flattened.
4648 // FIXME: consider tracking vector origins for app vectors?
4649 if (B->getType()->isVectorTy()) {
4650 B = convertToBool(B, IRB);
4651 Sb = convertToBool(Sb, IRB);
4652 }
4653 // a = select b, c, d
4654 // Oa = Sb ? Ob : (b ? Oc : Od)
4655 setOrigin(&I, IRB.CreateSelect(Sb, Ob, IRB.CreateSelect(B, Oc, Od)));
4656 }
4657 }
4658
4659 void visitLandingPadInst(LandingPadInst &I) {
4660 // Do nothing.
4661 // See https://github.com/google/sanitizers/issues/504
4662 setShadow(&I, getCleanShadow(&I));
4663 setOrigin(&I, getCleanOrigin());
4664 }
4665
4666 void visitCatchSwitchInst(CatchSwitchInst &I) {
4667 setShadow(&I, getCleanShadow(&I));
4668 setOrigin(&I, getCleanOrigin());
4669 }
4670
4671 void visitFuncletPadInst(FuncletPadInst &I) {
4672 setShadow(&I, getCleanShadow(&I));
4673 setOrigin(&I, getCleanOrigin());
4674 }
4675
4676 void visitGetElementPtrInst(GetElementPtrInst &I) { handleShadowOr(I); }
4677
4678 void visitExtractValueInst(ExtractValueInst &I) {
4679 IRBuilder<> IRB(&I);
4680 Value *Agg = I.getAggregateOperand();
4681 LLVM_DEBUG(dbgs() << "ExtractValue: " << I << "\n");
4682 Value *AggShadow = getShadow(Agg);
4683 LLVM_DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
4684 Value *ResShadow = IRB.CreateExtractValue(AggShadow, I.getIndices());
4685 LLVM_DEBUG(dbgs() << " ResShadow: " << *ResShadow << "\n");
4686 setShadow(&I, ResShadow);
4687 setOriginForNaryOp(I);
4688 }
4689
4690 void visitInsertValueInst(InsertValueInst &I) {
4691 IRBuilder<> IRB(&I);
4692 LLVM_DEBUG(dbgs() << "InsertValue: " << I << "\n");
4693 Value *AggShadow = getShadow(I.getAggregateOperand());
4694 Value *InsShadow = getShadow(I.getInsertedValueOperand());
4695 LLVM_DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
4696 LLVM_DEBUG(dbgs() << " InsShadow: " << *InsShadow << "\n");
4697 Value *Res = IRB.CreateInsertValue(AggShadow, InsShadow, I.getIndices());
4698 LLVM_DEBUG(dbgs() << " Res: " << *Res << "\n");
4699 setShadow(&I, Res);
4700 setOriginForNaryOp(I);
4701 }
4702
4703 void dumpInst(Instruction &I) {
4704 if (CallInst *CI = dyn_cast<CallInst>(&I)) {
4705 errs() << "ZZZ call " << CI->getCalledFunction()->getName() << "\n";
4706 } else {
4707 errs() << "ZZZ " << I.getOpcodeName() << "\n";
4708 }
4709 errs() << "QQQ " << I << "\n";
4710 }
4711
4712 void visitResumeInst(ResumeInst &I) {
4713 LLVM_DEBUG(dbgs() << "Resume: " << I << "\n");
4714 // Nothing to do here.
4715 }
4716
4717 void visitCleanupReturnInst(CleanupReturnInst &CRI) {
4718 LLVM_DEBUG(dbgs() << "CleanupReturn: " << CRI << "\n");
4719 // Nothing to do here.
4720 }
4721
4722 void visitCatchReturnInst(CatchReturnInst &CRI) {
4723 LLVM_DEBUG(dbgs() << "CatchReturn: " << CRI << "\n");
4724 // Nothing to do here.
4725 }
4726
4727 void instrumentAsmArgument(Value *Operand, Type *ElemTy, Instruction &I,
4728 IRBuilder<> &IRB, const DataLayout &DL,
4729 bool isOutput) {
4730 // For each assembly argument, we check its value for being initialized.
4731 // If the argument is a pointer, we assume it points to a single element
4732 // of the corresponding type (or to a 8-byte word, if the type is unsized).
4733 // Each such pointer is instrumented with a call to the runtime library.
4734 Type *OpType = Operand->getType();
4735 // Check the operand value itself.
4736 insertShadowCheck(Operand, &I);
4737 if (!OpType->isPointerTy() || !isOutput) {
4738 assert(!isOutput);
4739 return;
4740 }
4741 if (!ElemTy->isSized())
4742 return;
4743 auto Size = DL.getTypeStoreSize(ElemTy);
4744 Value *SizeVal = IRB.CreateTypeSize(MS.IntptrTy, Size);
4745 if (MS.CompileKernel) {
4746 IRB.CreateCall(MS.MsanInstrumentAsmStoreFn, {Operand, SizeVal});
4747 } else {
4748 // ElemTy, derived from elementtype(), does not encode the alignment of
4749 // the pointer. Conservatively assume that the shadow memory is unaligned.
4750 // When Size is large, avoid StoreInst as it would expand to many
4751 // instructions.
4752 auto [ShadowPtr, _] =
4753 getShadowOriginPtrUserspace(Operand, IRB, IRB.getInt8Ty(), Align(1));
4754 if (Size <= 32)
4755 IRB.CreateAlignedStore(getCleanShadow(ElemTy), ShadowPtr, Align(1));
4756 else
4757 IRB.CreateMemSet(ShadowPtr, ConstantInt::getNullValue(IRB.getInt8Ty()),
4758 SizeVal, Align(1));
4759 }
4760 }
4761
4762 /// Get the number of output arguments returned by pointers.
4763 int getNumOutputArgs(InlineAsm *IA, CallBase *CB) {
4764 int NumRetOutputs = 0;
4765 int NumOutputs = 0;
4766 Type *RetTy = cast<Value>(CB)->getType();
4767 if (!RetTy->isVoidTy()) {
4768 // Register outputs are returned via the CallInst return value.
4769 auto *ST = dyn_cast<StructType>(RetTy);
4770 if (ST)
4771 NumRetOutputs = ST->getNumElements();
4772 else
4773 NumRetOutputs = 1;
4774 }
4775 InlineAsm::ConstraintInfoVector Constraints = IA->ParseConstraints();
4776 for (const InlineAsm::ConstraintInfo &Info : Constraints) {
4777 switch (Info.Type) {
4779 NumOutputs++;
4780 break;
4781 default:
4782 break;
4783 }
4784 }
4785 return NumOutputs - NumRetOutputs;
4786 }
4787
4788 void visitAsmInstruction(Instruction &I) {
4789 // Conservative inline assembly handling: check for poisoned shadow of
4790 // asm() arguments, then unpoison the result and all the memory locations
4791 // pointed to by those arguments.
4792 // An inline asm() statement in C++ contains lists of input and output
4793 // arguments used by the assembly code. These are mapped to operands of the
4794 // CallInst as follows:
4795 // - nR register outputs ("=r) are returned by value in a single structure
4796 // (SSA value of the CallInst);
4797 // - nO other outputs ("=m" and others) are returned by pointer as first
4798 // nO operands of the CallInst;
4799 // - nI inputs ("r", "m" and others) are passed to CallInst as the
4800 // remaining nI operands.
4801 // The total number of asm() arguments in the source is nR+nO+nI, and the
4802 // corresponding CallInst has nO+nI+1 operands (the last operand is the
4803 // function to be called).
4804 const DataLayout &DL = F.getDataLayout();
4805 CallBase *CB = cast<CallBase>(&I);
4806 IRBuilder<> IRB(&I);
4807 InlineAsm *IA = cast<InlineAsm>(CB->getCalledOperand());
4808 int OutputArgs = getNumOutputArgs(IA, CB);
4809 // The last operand of a CallInst is the function itself.
4810 int NumOperands = CB->getNumOperands() - 1;
4811
4812 // Check input arguments. Doing so before unpoisoning output arguments, so
4813 // that we won't overwrite uninit values before checking them.
4814 for (int i = OutputArgs; i < NumOperands; i++) {
4815 Value *Operand = CB->getOperand(i);
4816 instrumentAsmArgument(Operand, CB->getParamElementType(i), I, IRB, DL,
4817 /*isOutput*/ false);
4818 }
4819 // Unpoison output arguments. This must happen before the actual InlineAsm
4820 // call, so that the shadow for memory published in the asm() statement
4821 // remains valid.
4822 for (int i = 0; i < OutputArgs; i++) {
4823 Value *Operand = CB->getOperand(i);
4824 instrumentAsmArgument(Operand, CB->getParamElementType(i), I, IRB, DL,
4825 /*isOutput*/ true);
4826 }
4827
4828 setShadow(&I, getCleanShadow(&I));
4829 setOrigin(&I, getCleanOrigin());
4830 }
4831
4832 void visitFreezeInst(FreezeInst &I) {
4833 // Freeze always returns a fully defined value.
4834 setShadow(&I, getCleanShadow(&I));
4835 setOrigin(&I, getCleanOrigin());
4836 }
4837
4838 void visitInstruction(Instruction &I) {
4839 // Everything else: stop propagating and check for poisoned shadow.
4841 dumpInst(I);
4842 LLVM_DEBUG(dbgs() << "DEFAULT: " << I << "\n");
4843 for (size_t i = 0, n = I.getNumOperands(); i < n; i++) {
4844 Value *Operand = I.getOperand(i);
4845 if (Operand->getType()->isSized())
4846 insertShadowCheck(Operand, &I);
4847 }
4848 setShadow(&I, getCleanShadow(&I));
4849 setOrigin(&I, getCleanOrigin());
4850 }
4851};
4852
4853struct VarArgHelperBase : public VarArgHelper {
4854 Function &F;
4855 MemorySanitizer &MS;
4856 MemorySanitizerVisitor &MSV;
4857 SmallVector<CallInst *, 16> VAStartInstrumentationList;
4858 const unsigned VAListTagSize;
4859
4860 VarArgHelperBase(Function &F, MemorySanitizer &MS,
4861 MemorySanitizerVisitor &MSV, unsigned VAListTagSize)
4862 : F(F), MS(MS), MSV(MSV), VAListTagSize(VAListTagSize) {}
4863
4864 Value *getShadowAddrForVAArgument(IRBuilder<> &IRB, unsigned ArgOffset) {
4865 Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4866 return IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4867 }
4868
4869 /// Compute the shadow address for a given va_arg.
4870 Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4871 unsigned ArgOffset) {
4872 Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4873 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4874 return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4875 "_msarg_va_s");
4876 }
4877
4878 /// Compute the shadow address for a given va_arg.
4879 Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4880 unsigned ArgOffset, unsigned ArgSize) {
4881 // Make sure we don't overflow __msan_va_arg_tls.
4882 if (ArgOffset + ArgSize > kParamTLSSize)
4883 return nullptr;
4884 return getShadowPtrForVAArgument(Ty, IRB, ArgOffset);
4885 }
4886
4887 /// Compute the origin address for a given va_arg.
4888 Value *getOriginPtrForVAArgument(IRBuilder<> &IRB, int ArgOffset) {
4889 Value *Base = IRB.CreatePointerCast(MS.VAArgOriginTLS, MS.IntptrTy);
4890 // getOriginPtrForVAArgument() is always called after
4891 // getShadowPtrForVAArgument(), so __msan_va_arg_origin_tls can never
4892 // overflow.
4893 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4894 return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0),
4895 "_msarg_va_o");
4896 }
4897
4898 void CleanUnusedTLS(IRBuilder<> &IRB, Value *ShadowBase,
4899 unsigned BaseOffset) {
4900 // The tails of __msan_va_arg_tls is not large enough to fit full
4901 // value shadow, but it will be copied to backup anyway. Make it
4902 // clean.
4903 if (BaseOffset >= kParamTLSSize)
4904 return;
4905 Value *TailSize =
4906 ConstantInt::getSigned(IRB.getInt32Ty(), kParamTLSSize - BaseOffset);
4907 IRB.CreateMemSet(ShadowBase, ConstantInt::getNullValue(IRB.getInt8Ty()),
4908 TailSize, Align(8));
4909 }
4910
4911 void unpoisonVAListTagForInst(IntrinsicInst &I) {
4912 IRBuilder<> IRB(&I);
4913 Value *VAListTag = I.getArgOperand(0);
4914 const Align Alignment = Align(8);
4915 auto [ShadowPtr, OriginPtr] = MSV.getShadowOriginPtr(
4916 VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4917 // Unpoison the whole __va_list_tag.
4918 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4919 VAListTagSize, Alignment, false);
4920 }
4921
4922 void visitVAStartInst(VAStartInst &I) override {
4923 if (F.getCallingConv() == CallingConv::Win64)
4924 return;
4925 VAStartInstrumentationList.push_back(&I);
4926 unpoisonVAListTagForInst(I);
4927 }
4928
4929 void visitVACopyInst(VACopyInst &I) override {
4930 if (