LLVM 18.0.0git
MemorySanitizer.cpp
Go to the documentation of this file.
1//===- MemorySanitizer.cpp - detector of uninitialized reads --------------===//
2//
3// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
4// See https://llvm.org/LICENSE.txt for license information.
5// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
6//
7//===----------------------------------------------------------------------===//
8//
9/// \file
10/// This file is a part of MemorySanitizer, a detector of uninitialized
11/// reads.
12///
13/// The algorithm of the tool is similar to Memcheck
14/// (http://goo.gl/QKbem). We associate a few shadow bits with every
15/// byte of the application memory, poison the shadow of the malloc-ed
16/// or alloca-ed memory, load the shadow bits on every memory read,
17/// propagate the shadow bits through some of the arithmetic
18/// instruction (including MOV), store the shadow bits on every memory
19/// write, report a bug on some other instructions (e.g. JMP) if the
20/// associated shadow is poisoned.
21///
22/// But there are differences too. The first and the major one:
23/// compiler instrumentation instead of binary instrumentation. This
24/// gives us much better register allocation, possible compiler
25/// optimizations and a fast start-up. But this brings the major issue
26/// as well: msan needs to see all program events, including system
27/// calls and reads/writes in system libraries, so we either need to
28/// compile *everything* with msan or use a binary translation
29/// component (e.g. DynamoRIO) to instrument pre-built libraries.
30/// Another difference from Memcheck is that we use 8 shadow bits per
31/// byte of application memory and use a direct shadow mapping. This
32/// greatly simplifies the instrumentation code and avoids races on
33/// shadow updates (Memcheck is single-threaded so races are not a
34/// concern there. Memcheck uses 2 shadow bits per byte with a slow
35/// path storage that uses 8 bits per byte).
36///
37/// The default value of shadow is 0, which means "clean" (not poisoned).
38///
39/// Every module initializer should call __msan_init to ensure that the
40/// shadow memory is ready. On error, __msan_warning is called. Since
41/// parameters and return values may be passed via registers, we have a
42/// specialized thread-local shadow for return values
43/// (__msan_retval_tls) and parameters (__msan_param_tls).
44///
45/// Origin tracking.
46///
47/// MemorySanitizer can track origins (allocation points) of all uninitialized
48/// values. This behavior is controlled with a flag (msan-track-origins) and is
49/// disabled by default.
50///
51/// Origins are 4-byte values created and interpreted by the runtime library.
52/// They are stored in a second shadow mapping, one 4-byte value for 4 bytes
53/// of application memory. Propagation of origins is basically a bunch of
54/// "select" instructions that pick the origin of a dirty argument, if an
55/// instruction has one.
56///
57/// Every 4 aligned, consecutive bytes of application memory have one origin
58/// value associated with them. If these bytes contain uninitialized data
59/// coming from 2 different allocations, the last store wins. Because of this,
60/// MemorySanitizer reports can show unrelated origins, but this is unlikely in
61/// practice.
62///
63/// Origins are meaningless for fully initialized values, so MemorySanitizer
64/// avoids storing origin to memory when a fully initialized value is stored.
65/// This way it avoids needless overwriting origin of the 4-byte region on
66/// a short (i.e. 1 byte) clean store, and it is also good for performance.
67///
68/// Atomic handling.
69///
70/// Ideally, every atomic store of application value should update the
71/// corresponding shadow location in an atomic way. Unfortunately, atomic store
72/// of two disjoint locations can not be done without severe slowdown.
73///
74/// Therefore, we implement an approximation that may err on the safe side.
75/// In this implementation, every atomically accessed location in the program
76/// may only change from (partially) uninitialized to fully initialized, but
77/// not the other way around. We load the shadow _after_ the application load,
78/// and we store the shadow _before_ the app store. Also, we always store clean
79/// shadow (if the application store is atomic). This way, if the store-load
80/// pair constitutes a happens-before arc, shadow store and load are correctly
81/// ordered such that the load will get either the value that was stored, or
82/// some later value (which is always clean).
83///
84/// This does not work very well with Compare-And-Swap (CAS) and
85/// Read-Modify-Write (RMW) operations. To follow the above logic, CAS and RMW
86/// must store the new shadow before the app operation, and load the shadow
87/// after the app operation. Computers don't work this way. Current
88/// implementation ignores the load aspect of CAS/RMW, always returning a clean
89/// value. It implements the store part as a simple atomic store by storing a
90/// clean shadow.
91///
92/// Instrumenting inline assembly.
93///
94/// For inline assembly code LLVM has little idea about which memory locations
95/// become initialized depending on the arguments. It can be possible to figure
96/// out which arguments are meant to point to inputs and outputs, but the
97/// actual semantics can be only visible at runtime. In the Linux kernel it's
98/// also possible that the arguments only indicate the offset for a base taken
99/// from a segment register, so it's dangerous to treat any asm() arguments as
100/// pointers. We take a conservative approach generating calls to
101/// __msan_instrument_asm_store(ptr, size)
102/// , which defer the memory unpoisoning to the runtime library.
103/// The latter can perform more complex address checks to figure out whether
104/// it's safe to touch the shadow memory.
105/// Like with atomic operations, we call __msan_instrument_asm_store() before
106/// the assembly call, so that changes to the shadow memory will be seen by
107/// other threads together with main memory initialization.
108///
109/// KernelMemorySanitizer (KMSAN) implementation.
110///
111/// The major differences between KMSAN and MSan instrumentation are:
112/// - KMSAN always tracks the origins and implies msan-keep-going=true;
113/// - KMSAN allocates shadow and origin memory for each page separately, so
114/// there are no explicit accesses to shadow and origin in the
115/// instrumentation.
116/// Shadow and origin values for a particular X-byte memory location
117/// (X=1,2,4,8) are accessed through pointers obtained via the
118/// __msan_metadata_ptr_for_load_X(ptr)
119/// __msan_metadata_ptr_for_store_X(ptr)
120/// functions. The corresponding functions check that the X-byte accesses
121/// are possible and returns the pointers to shadow and origin memory.
122/// Arbitrary sized accesses are handled with:
123/// __msan_metadata_ptr_for_load_n(ptr, size)
124/// __msan_metadata_ptr_for_store_n(ptr, size);
125/// Note that the sanitizer code has to deal with how shadow/origin pairs
126/// returned by the these functions are represented in different ABIs. In
127/// the X86_64 ABI they are returned in RDX:RAX, and in the SystemZ ABI they
128/// are written to memory pointed to by a hidden parameter.
129/// - TLS variables are stored in a single per-task struct. A call to a
130/// function __msan_get_context_state() returning a pointer to that struct
131/// is inserted into every instrumented function before the entry block;
132/// - __msan_warning() takes a 32-bit origin parameter;
133/// - local variables are poisoned with __msan_poison_alloca() upon function
134/// entry and unpoisoned with __msan_unpoison_alloca() before leaving the
135/// function;
136/// - the pass doesn't declare any global variables or add global constructors
137/// to the translation unit.
138///
139/// Also, KMSAN currently ignores uninitialized memory passed into inline asm
140/// calls, making sure we're on the safe side wrt. possible false positives.
141///
142/// KernelMemorySanitizer only supports X86_64 and SystemZ at the moment.
143///
144//
145// FIXME: This sanitizer does not yet handle scalable vectors
146//
147//===----------------------------------------------------------------------===//
148
150#include "llvm/ADT/APInt.h"
151#include "llvm/ADT/ArrayRef.h"
152#include "llvm/ADT/DenseMap.h"
154#include "llvm/ADT/SetVector.h"
155#include "llvm/ADT/SmallString.h"
156#include "llvm/ADT/SmallVector.h"
158#include "llvm/ADT/StringRef.h"
162#include "llvm/IR/Argument.h"
164#include "llvm/IR/Attributes.h"
165#include "llvm/IR/BasicBlock.h"
166#include "llvm/IR/CallingConv.h"
167#include "llvm/IR/Constant.h"
168#include "llvm/IR/Constants.h"
169#include "llvm/IR/DataLayout.h"
170#include "llvm/IR/DerivedTypes.h"
171#include "llvm/IR/Function.h"
172#include "llvm/IR/GlobalValue.h"
174#include "llvm/IR/IRBuilder.h"
175#include "llvm/IR/InlineAsm.h"
176#include "llvm/IR/InstVisitor.h"
177#include "llvm/IR/InstrTypes.h"
178#include "llvm/IR/Instruction.h"
179#include "llvm/IR/Instructions.h"
181#include "llvm/IR/Intrinsics.h"
182#include "llvm/IR/IntrinsicsX86.h"
183#include "llvm/IR/MDBuilder.h"
184#include "llvm/IR/Module.h"
185#include "llvm/IR/Type.h"
186#include "llvm/IR/Value.h"
187#include "llvm/IR/ValueMap.h"
190#include "llvm/Support/Casting.h"
192#include "llvm/Support/Debug.h"
201#include <algorithm>
202#include <cassert>
203#include <cstddef>
204#include <cstdint>
205#include <memory>
206#include <string>
207#include <tuple>
208
209using namespace llvm;
210
211#define DEBUG_TYPE "msan"
212
213DEBUG_COUNTER(DebugInsertCheck, "msan-insert-check",
214 "Controls which checks to insert");
215
216static const unsigned kOriginSize = 4;
219
220// These constants must be kept in sync with the ones in msan.h.
221static const unsigned kParamTLSSize = 800;
222static const unsigned kRetvalTLSSize = 800;
223
224// Accesses sizes are powers of two: 1, 2, 4, 8.
225static const size_t kNumberOfAccessSizes = 4;
226
227/// Track origins of uninitialized values.
228///
229/// Adds a section to MemorySanitizer report that points to the allocation
230/// (stack or heap) the uninitialized bits came from originally.
232 "msan-track-origins",
233 cl::desc("Track origins (allocation sites) of poisoned memory"), cl::Hidden,
234 cl::init(0));
235
236static cl::opt<bool> ClKeepGoing("msan-keep-going",
237 cl::desc("keep going after reporting a UMR"),
238 cl::Hidden, cl::init(false));
239
240static cl::opt<bool>
241 ClPoisonStack("msan-poison-stack",
242 cl::desc("poison uninitialized stack variables"), cl::Hidden,
243 cl::init(true));
244
246 "msan-poison-stack-with-call",
247 cl::desc("poison uninitialized stack variables with a call"), cl::Hidden,
248 cl::init(false));
249
251 "msan-poison-stack-pattern",
252 cl::desc("poison uninitialized stack variables with the given pattern"),
253 cl::Hidden, cl::init(0xff));
254
255static cl::opt<bool>
256 ClPrintStackNames("msan-print-stack-names",
257 cl::desc("Print name of local stack variable"),
258 cl::Hidden, cl::init(true));
259
260static cl::opt<bool> ClPoisonUndef("msan-poison-undef",
261 cl::desc("poison undef temps"), cl::Hidden,
262 cl::init(true));
263
264static cl::opt<bool>
265 ClHandleICmp("msan-handle-icmp",
266 cl::desc("propagate shadow through ICmpEQ and ICmpNE"),
267 cl::Hidden, cl::init(true));
268
269static cl::opt<bool>
270 ClHandleICmpExact("msan-handle-icmp-exact",
271 cl::desc("exact handling of relational integer ICmp"),
272 cl::Hidden, cl::init(false));
273
275 "msan-handle-lifetime-intrinsics",
276 cl::desc(
277 "when possible, poison scoped variables at the beginning of the scope "
278 "(slower, but more precise)"),
279 cl::Hidden, cl::init(true));
280
281// When compiling the Linux kernel, we sometimes see false positives related to
282// MSan being unable to understand that inline assembly calls may initialize
283// local variables.
284// This flag makes the compiler conservatively unpoison every memory location
285// passed into an assembly call. Note that this may cause false positives.
286// Because it's impossible to figure out the array sizes, we can only unpoison
287// the first sizeof(type) bytes for each type* pointer.
288// The instrumentation is only enabled in KMSAN builds, and only if
289// -msan-handle-asm-conservative is on. This is done because we may want to
290// quickly disable assembly instrumentation when it breaks.
292 "msan-handle-asm-conservative",
293 cl::desc("conservative handling of inline assembly"), cl::Hidden,
294 cl::init(true));
295
296// This flag controls whether we check the shadow of the address
297// operand of load or store. Such bugs are very rare, since load from
298// a garbage address typically results in SEGV, but still happen
299// (e.g. only lower bits of address are garbage, or the access happens
300// early at program startup where malloc-ed memory is more likely to
301// be zeroed. As of 2012-08-28 this flag adds 20% slowdown.
303 "msan-check-access-address",
304 cl::desc("report accesses through a pointer which has poisoned shadow"),
305 cl::Hidden, cl::init(true));
306
308 "msan-eager-checks",
309 cl::desc("check arguments and return values at function call boundaries"),
310 cl::Hidden, cl::init(false));
311
313 "msan-dump-strict-instructions",
314 cl::desc("print out instructions with default strict semantics"),
315 cl::Hidden, cl::init(false));
316
318 "msan-instrumentation-with-call-threshold",
319 cl::desc(
320 "If the function being instrumented requires more than "
321 "this number of checks and origin stores, use callbacks instead of "
322 "inline checks (-1 means never use callbacks)."),
323 cl::Hidden, cl::init(3500));
324
325static cl::opt<bool>
326 ClEnableKmsan("msan-kernel",
327 cl::desc("Enable KernelMemorySanitizer instrumentation"),
328 cl::Hidden, cl::init(false));
329
330static cl::opt<bool>
331 ClDisableChecks("msan-disable-checks",
332 cl::desc("Apply no_sanitize to the whole file"), cl::Hidden,
333 cl::init(false));
334
335static cl::opt<bool>
336 ClCheckConstantShadow("msan-check-constant-shadow",
337 cl::desc("Insert checks for constant shadow values"),
338 cl::Hidden, cl::init(true));
339
340// This is off by default because of a bug in gold:
341// https://sourceware.org/bugzilla/show_bug.cgi?id=19002
342static cl::opt<bool>
343 ClWithComdat("msan-with-comdat",
344 cl::desc("Place MSan constructors in comdat sections"),
345 cl::Hidden, cl::init(false));
346
347// These options allow to specify custom memory map parameters
348// See MemoryMapParams for details.
349static cl::opt<uint64_t> ClAndMask("msan-and-mask",
350 cl::desc("Define custom MSan AndMask"),
351 cl::Hidden, cl::init(0));
352
353static cl::opt<uint64_t> ClXorMask("msan-xor-mask",
354 cl::desc("Define custom MSan XorMask"),
355 cl::Hidden, cl::init(0));
356
357static cl::opt<uint64_t> ClShadowBase("msan-shadow-base",
358 cl::desc("Define custom MSan ShadowBase"),
359 cl::Hidden, cl::init(0));
360
361static cl::opt<uint64_t> ClOriginBase("msan-origin-base",
362 cl::desc("Define custom MSan OriginBase"),
363 cl::Hidden, cl::init(0));
364
365static cl::opt<int>
366 ClDisambiguateWarning("msan-disambiguate-warning-threshold",
367 cl::desc("Define threshold for number of checks per "
368 "debug location to force origin update."),
369 cl::Hidden, cl::init(3));
370
371const char kMsanModuleCtorName[] = "msan.module_ctor";
372const char kMsanInitName[] = "__msan_init";
373
374namespace {
375
376// Memory map parameters used in application-to-shadow address calculation.
377// Offset = (Addr & ~AndMask) ^ XorMask
378// Shadow = ShadowBase + Offset
379// Origin = OriginBase + Offset
380struct MemoryMapParams {
381 uint64_t AndMask;
382 uint64_t XorMask;
383 uint64_t ShadowBase;
384 uint64_t OriginBase;
385};
386
387struct PlatformMemoryMapParams {
388 const MemoryMapParams *bits32;
389 const MemoryMapParams *bits64;
390};
391
392} // end anonymous namespace
393
394// i386 Linux
395static const MemoryMapParams Linux_I386_MemoryMapParams = {
396 0x000080000000, // AndMask
397 0, // XorMask (not used)
398 0, // ShadowBase (not used)
399 0x000040000000, // OriginBase
400};
401
402// x86_64 Linux
403static const MemoryMapParams Linux_X86_64_MemoryMapParams = {
404 0, // AndMask (not used)
405 0x500000000000, // XorMask
406 0, // ShadowBase (not used)
407 0x100000000000, // OriginBase
408};
409
410// mips64 Linux
411static const MemoryMapParams Linux_MIPS64_MemoryMapParams = {
412 0, // AndMask (not used)
413 0x008000000000, // XorMask
414 0, // ShadowBase (not used)
415 0x002000000000, // OriginBase
416};
417
418// ppc64 Linux
419static const MemoryMapParams Linux_PowerPC64_MemoryMapParams = {
420 0xE00000000000, // AndMask
421 0x100000000000, // XorMask
422 0x080000000000, // ShadowBase
423 0x1C0000000000, // OriginBase
424};
425
426// s390x Linux
427static const MemoryMapParams Linux_S390X_MemoryMapParams = {
428 0xC00000000000, // AndMask
429 0, // XorMask (not used)
430 0x080000000000, // ShadowBase
431 0x1C0000000000, // OriginBase
432};
433
434// aarch64 Linux
435static const MemoryMapParams Linux_AArch64_MemoryMapParams = {
436 0, // AndMask (not used)
437 0x0B00000000000, // XorMask
438 0, // ShadowBase (not used)
439 0x0200000000000, // OriginBase
440};
441
442// loongarch64 Linux
443static const MemoryMapParams Linux_LoongArch64_MemoryMapParams = {
444 0, // AndMask (not used)
445 0x500000000000, // XorMask
446 0, // ShadowBase (not used)
447 0x100000000000, // OriginBase
448};
449
450// aarch64 FreeBSD
451static const MemoryMapParams FreeBSD_AArch64_MemoryMapParams = {
452 0x1800000000000, // AndMask
453 0x0400000000000, // XorMask
454 0x0200000000000, // ShadowBase
455 0x0700000000000, // OriginBase
456};
457
458// i386 FreeBSD
459static const MemoryMapParams FreeBSD_I386_MemoryMapParams = {
460 0x000180000000, // AndMask
461 0x000040000000, // XorMask
462 0x000020000000, // ShadowBase
463 0x000700000000, // OriginBase
464};
465
466// x86_64 FreeBSD
467static const MemoryMapParams FreeBSD_X86_64_MemoryMapParams = {
468 0xc00000000000, // AndMask
469 0x200000000000, // XorMask
470 0x100000000000, // ShadowBase
471 0x380000000000, // OriginBase
472};
473
474// x86_64 NetBSD
475static const MemoryMapParams NetBSD_X86_64_MemoryMapParams = {
476 0, // AndMask
477 0x500000000000, // XorMask
478 0, // ShadowBase
479 0x100000000000, // OriginBase
480};
481
482static const PlatformMemoryMapParams Linux_X86_MemoryMapParams = {
485};
486
487static const PlatformMemoryMapParams Linux_MIPS_MemoryMapParams = {
488 nullptr,
490};
491
492static const PlatformMemoryMapParams Linux_PowerPC_MemoryMapParams = {
493 nullptr,
495};
496
497static const PlatformMemoryMapParams Linux_S390_MemoryMapParams = {
498 nullptr,
500};
501
502static const PlatformMemoryMapParams Linux_ARM_MemoryMapParams = {
503 nullptr,
505};
506
507static const PlatformMemoryMapParams Linux_LoongArch_MemoryMapParams = {
508 nullptr,
510};
511
512static const PlatformMemoryMapParams FreeBSD_ARM_MemoryMapParams = {
513 nullptr,
515};
516
517static const PlatformMemoryMapParams FreeBSD_X86_MemoryMapParams = {
520};
521
522static const PlatformMemoryMapParams NetBSD_X86_MemoryMapParams = {
523 nullptr,
525};
526
527namespace {
528
529/// Instrument functions of a module to detect uninitialized reads.
530///
531/// Instantiating MemorySanitizer inserts the msan runtime library API function
532/// declarations into the module if they don't exist already. Instantiating
533/// ensures the __msan_init function is in the list of global constructors for
534/// the module.
535class MemorySanitizer {
536public:
537 MemorySanitizer(Module &M, MemorySanitizerOptions Options)
538 : CompileKernel(Options.Kernel), TrackOrigins(Options.TrackOrigins),
539 Recover(Options.Recover), EagerChecks(Options.EagerChecks) {
540 initializeModule(M);
541 }
542
543 // MSan cannot be moved or copied because of MapParams.
544 MemorySanitizer(MemorySanitizer &&) = delete;
545 MemorySanitizer &operator=(MemorySanitizer &&) = delete;
546 MemorySanitizer(const MemorySanitizer &) = delete;
547 MemorySanitizer &operator=(const MemorySanitizer &) = delete;
548
549 bool sanitizeFunction(Function &F, TargetLibraryInfo &TLI);
550
551private:
552 friend struct MemorySanitizerVisitor;
553 friend struct VarArgAMD64Helper;
554 friend struct VarArgMIPS64Helper;
555 friend struct VarArgAArch64Helper;
556 friend struct VarArgPowerPC64Helper;
557 friend struct VarArgSystemZHelper;
558
559 void initializeModule(Module &M);
560 void initializeCallbacks(Module &M, const TargetLibraryInfo &TLI);
561 void createKernelApi(Module &M, const TargetLibraryInfo &TLI);
562 void createUserspaceApi(Module &M, const TargetLibraryInfo &TLI);
563
564 template <typename... ArgsTy>
565 FunctionCallee getOrInsertMsanMetadataFunction(Module &M, StringRef Name,
566 ArgsTy... Args);
567
568 /// True if we're compiling the Linux kernel.
569 bool CompileKernel;
570 /// Track origins (allocation points) of uninitialized values.
571 int TrackOrigins;
572 bool Recover;
573 bool EagerChecks;
574
575 Triple TargetTriple;
576 LLVMContext *C;
577 Type *IntptrTy; ///< Integer type with the size of a ptr in default AS.
578 Type *OriginTy;
579
580 // XxxTLS variables represent the per-thread state in MSan and per-task state
581 // in KMSAN.
582 // For the userspace these point to thread-local globals. In the kernel land
583 // they point to the members of a per-task struct obtained via a call to
584 // __msan_get_context_state().
585
586 /// Thread-local shadow storage for function parameters.
587 Value *ParamTLS;
588
589 /// Thread-local origin storage for function parameters.
590 Value *ParamOriginTLS;
591
592 /// Thread-local shadow storage for function return value.
593 Value *RetvalTLS;
594
595 /// Thread-local origin storage for function return value.
596 Value *RetvalOriginTLS;
597
598 /// Thread-local shadow storage for in-register va_arg function
599 /// parameters (x86_64-specific).
600 Value *VAArgTLS;
601
602 /// Thread-local shadow storage for in-register va_arg function
603 /// parameters (x86_64-specific).
604 Value *VAArgOriginTLS;
605
606 /// Thread-local shadow storage for va_arg overflow area
607 /// (x86_64-specific).
608 Value *VAArgOverflowSizeTLS;
609
610 /// Are the instrumentation callbacks set up?
611 bool CallbacksInitialized = false;
612
613 /// The run-time callback to print a warning.
614 FunctionCallee WarningFn;
615
616 // These arrays are indexed by log2(AccessSize).
617 FunctionCallee MaybeWarningFn[kNumberOfAccessSizes];
618 FunctionCallee MaybeStoreOriginFn[kNumberOfAccessSizes];
619
620 /// Run-time helper that generates a new origin value for a stack
621 /// allocation.
622 FunctionCallee MsanSetAllocaOriginWithDescriptionFn;
623 // No description version
624 FunctionCallee MsanSetAllocaOriginNoDescriptionFn;
625
626 /// Run-time helper that poisons stack on function entry.
627 FunctionCallee MsanPoisonStackFn;
628
629 /// Run-time helper that records a store (or any event) of an
630 /// uninitialized value and returns an updated origin id encoding this info.
631 FunctionCallee MsanChainOriginFn;
632
633 /// Run-time helper that paints an origin over a region.
634 FunctionCallee MsanSetOriginFn;
635
636 /// MSan runtime replacements for memmove, memcpy and memset.
637 FunctionCallee MemmoveFn, MemcpyFn, MemsetFn;
638
639 /// KMSAN callback for task-local function argument shadow.
640 StructType *MsanContextStateTy;
641 FunctionCallee MsanGetContextStateFn;
642
643 /// Functions for poisoning/unpoisoning local variables
644 FunctionCallee MsanPoisonAllocaFn, MsanUnpoisonAllocaFn;
645
646 /// Pair of shadow/origin pointers.
647 Type *MsanMetadata;
648
649 /// Each of the MsanMetadataPtrXxx functions returns a MsanMetadata.
650 FunctionCallee MsanMetadataPtrForLoadN, MsanMetadataPtrForStoreN;
651 FunctionCallee MsanMetadataPtrForLoad_1_8[4];
652 FunctionCallee MsanMetadataPtrForStore_1_8[4];
653 FunctionCallee MsanInstrumentAsmStoreFn;
654
655 /// Storage for return values of the MsanMetadataPtrXxx functions.
656 Value *MsanMetadataAlloca;
657
658 /// Helper to choose between different MsanMetadataPtrXxx().
659 FunctionCallee getKmsanShadowOriginAccessFn(bool isStore, int size);
660
661 /// Memory map parameters used in application-to-shadow calculation.
662 const MemoryMapParams *MapParams;
663
664 /// Custom memory map parameters used when -msan-shadow-base or
665 // -msan-origin-base is provided.
666 MemoryMapParams CustomMapParams;
667
668 MDNode *ColdCallWeights;
669
670 /// Branch weights for origin store.
671 MDNode *OriginStoreWeights;
672};
673
674void insertModuleCtor(Module &M) {
677 /*InitArgTypes=*/{},
678 /*InitArgs=*/{},
679 // This callback is invoked when the functions are created the first
680 // time. Hook them into the global ctors list in that case:
681 [&](Function *Ctor, FunctionCallee) {
682 if (!ClWithComdat) {
683 appendToGlobalCtors(M, Ctor, 0);
684 return;
685 }
686 Comdat *MsanCtorComdat = M.getOrInsertComdat(kMsanModuleCtorName);
687 Ctor->setComdat(MsanCtorComdat);
688 appendToGlobalCtors(M, Ctor, 0, Ctor);
689 });
690}
691
692template <class T> T getOptOrDefault(const cl::opt<T> &Opt, T Default) {
693 return (Opt.getNumOccurrences() > 0) ? Opt : Default;
694}
695
696} // end anonymous namespace
697
699 bool EagerChecks)
700 : Kernel(getOptOrDefault(ClEnableKmsan, K)),
701 TrackOrigins(getOptOrDefault(ClTrackOrigins, Kernel ? 2 : TO)),
702 Recover(getOptOrDefault(ClKeepGoing, Kernel || R)),
703 EagerChecks(getOptOrDefault(ClEagerChecks, EagerChecks)) {}
704
707 bool Modified = false;
708 if (!Options.Kernel) {
709 insertModuleCtor(M);
710 Modified = true;
711 }
712
713 auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
714 for (Function &F : M) {
715 if (F.empty())
716 continue;
717 MemorySanitizer Msan(*F.getParent(), Options);
718 Modified |=
719 Msan.sanitizeFunction(F, FAM.getResult<TargetLibraryAnalysis>(F));
720 }
721
722 if (!Modified)
723 return PreservedAnalyses::all();
724
726 // GlobalsAA is considered stateless and does not get invalidated unless
727 // explicitly invalidated; PreservedAnalyses::none() is not enough. Sanitizers
728 // make changes that require GlobalsAA to be invalidated.
729 PA.abandon<GlobalsAA>();
730 return PA;
731}
732
734 raw_ostream &OS, function_ref<StringRef(StringRef)> MapClassName2PassName) {
736 OS, MapClassName2PassName);
737 OS << '<';
738 if (Options.Recover)
739 OS << "recover;";
740 if (Options.Kernel)
741 OS << "kernel;";
742 if (Options.EagerChecks)
743 OS << "eager-checks;";
744 OS << "track-origins=" << Options.TrackOrigins;
745 OS << '>';
746}
747
748/// Create a non-const global initialized with the given string.
749///
750/// Creates a writable global for Str so that we can pass it to the
751/// run-time lib. Runtime uses first 4 bytes of the string to store the
752/// frame ID, so the string needs to be mutable.
754 StringRef Str) {
755 Constant *StrConst = ConstantDataArray::getString(M.getContext(), Str);
756 return new GlobalVariable(M, StrConst->getType(), /*isConstant=*/true,
757 GlobalValue::PrivateLinkage, StrConst, "");
758}
759
760template <typename... ArgsTy>
762MemorySanitizer::getOrInsertMsanMetadataFunction(Module &M, StringRef Name,
763 ArgsTy... Args) {
764 if (TargetTriple.getArch() == Triple::systemz) {
765 // SystemZ ABI: shadow/origin pair is returned via a hidden parameter.
766 return M.getOrInsertFunction(Name, Type::getVoidTy(*C),
767 PointerType::get(MsanMetadata, 0),
768 std::forward<ArgsTy>(Args)...);
769 }
770
771 return M.getOrInsertFunction(Name, MsanMetadata,
772 std::forward<ArgsTy>(Args)...);
773}
774
775/// Create KMSAN API callbacks.
776void MemorySanitizer::createKernelApi(Module &M, const TargetLibraryInfo &TLI) {
777 IRBuilder<> IRB(*C);
778
779 // These will be initialized in insertKmsanPrologue().
780 RetvalTLS = nullptr;
781 RetvalOriginTLS = nullptr;
782 ParamTLS = nullptr;
783 ParamOriginTLS = nullptr;
784 VAArgTLS = nullptr;
785 VAArgOriginTLS = nullptr;
786 VAArgOverflowSizeTLS = nullptr;
787
788 WarningFn = M.getOrInsertFunction("__msan_warning",
789 TLI.getAttrList(C, {0}, /*Signed=*/false),
790 IRB.getVoidTy(), IRB.getInt32Ty());
791
792 // Requests the per-task context state (kmsan_context_state*) from the
793 // runtime library.
794 MsanContextStateTy = StructType::get(
795 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
796 ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8),
797 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
798 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8), /* va_arg_origin */
799 IRB.getInt64Ty(), ArrayType::get(OriginTy, kParamTLSSize / 4), OriginTy,
800 OriginTy);
801 MsanGetContextStateFn = M.getOrInsertFunction(
802 "__msan_get_context_state", PointerType::get(MsanContextStateTy, 0));
803
804 MsanMetadata = StructType::get(PointerType::get(IRB.getInt8Ty(), 0),
805 PointerType::get(IRB.getInt32Ty(), 0));
806
807 for (int ind = 0, size = 1; ind < 4; ind++, size <<= 1) {
808 std::string name_load =
809 "__msan_metadata_ptr_for_load_" + std::to_string(size);
810 std::string name_store =
811 "__msan_metadata_ptr_for_store_" + std::to_string(size);
812 MsanMetadataPtrForLoad_1_8[ind] = getOrInsertMsanMetadataFunction(
813 M, name_load, PointerType::get(IRB.getInt8Ty(), 0));
814 MsanMetadataPtrForStore_1_8[ind] = getOrInsertMsanMetadataFunction(
815 M, name_store, PointerType::get(IRB.getInt8Ty(), 0));
816 }
817
818 MsanMetadataPtrForLoadN = getOrInsertMsanMetadataFunction(
819 M, "__msan_metadata_ptr_for_load_n", PointerType::get(IRB.getInt8Ty(), 0),
820 IRB.getInt64Ty());
821 MsanMetadataPtrForStoreN = getOrInsertMsanMetadataFunction(
822 M, "__msan_metadata_ptr_for_store_n",
823 PointerType::get(IRB.getInt8Ty(), 0), IRB.getInt64Ty());
824
825 // Functions for poisoning and unpoisoning memory.
826 MsanPoisonAllocaFn =
827 M.getOrInsertFunction("__msan_poison_alloca", IRB.getVoidTy(),
828 IRB.getInt8PtrTy(), IntptrTy, IRB.getInt8PtrTy());
829 MsanUnpoisonAllocaFn = M.getOrInsertFunction(
830 "__msan_unpoison_alloca", IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy);
831}
832
834 return M.getOrInsertGlobal(Name, Ty, [&] {
835 return new GlobalVariable(M, Ty, false, GlobalVariable::ExternalLinkage,
836 nullptr, Name, nullptr,
838 });
839}
840
841/// Insert declarations for userspace-specific functions and globals.
842void MemorySanitizer::createUserspaceApi(Module &M, const TargetLibraryInfo &TLI) {
843 IRBuilder<> IRB(*C);
844
845 // Create the callback.
846 // FIXME: this function should have "Cold" calling conv,
847 // which is not yet implemented.
848 if (TrackOrigins) {
849 StringRef WarningFnName = Recover ? "__msan_warning_with_origin"
850 : "__msan_warning_with_origin_noreturn";
851 WarningFn = M.getOrInsertFunction(WarningFnName,
852 TLI.getAttrList(C, {0}, /*Signed=*/false),
853 IRB.getVoidTy(), IRB.getInt32Ty());
854 } else {
855 StringRef WarningFnName =
856 Recover ? "__msan_warning" : "__msan_warning_noreturn";
857 WarningFn = M.getOrInsertFunction(WarningFnName, IRB.getVoidTy());
858 }
859
860 // Create the global TLS variables.
861 RetvalTLS =
862 getOrInsertGlobal(M, "__msan_retval_tls",
863 ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8));
864
865 RetvalOriginTLS = getOrInsertGlobal(M, "__msan_retval_origin_tls", OriginTy);
866
867 ParamTLS =
868 getOrInsertGlobal(M, "__msan_param_tls",
869 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
870
871 ParamOriginTLS =
872 getOrInsertGlobal(M, "__msan_param_origin_tls",
873 ArrayType::get(OriginTy, kParamTLSSize / 4));
874
875 VAArgTLS =
876 getOrInsertGlobal(M, "__msan_va_arg_tls",
877 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
878
879 VAArgOriginTLS =
880 getOrInsertGlobal(M, "__msan_va_arg_origin_tls",
881 ArrayType::get(OriginTy, kParamTLSSize / 4));
882
883 VAArgOverflowSizeTLS =
884 getOrInsertGlobal(M, "__msan_va_arg_overflow_size_tls", IRB.getInt64Ty());
885
886 for (size_t AccessSizeIndex = 0; AccessSizeIndex < kNumberOfAccessSizes;
887 AccessSizeIndex++) {
888 unsigned AccessSize = 1 << AccessSizeIndex;
889 std::string FunctionName = "__msan_maybe_warning_" + itostr(AccessSize);
890 MaybeWarningFn[AccessSizeIndex] = M.getOrInsertFunction(
891 FunctionName, TLI.getAttrList(C, {0, 1}, /*Signed=*/false),
892 IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), IRB.getInt32Ty());
893
894 FunctionName = "__msan_maybe_store_origin_" + itostr(AccessSize);
895 MaybeStoreOriginFn[AccessSizeIndex] = M.getOrInsertFunction(
896 FunctionName, TLI.getAttrList(C, {0, 2}, /*Signed=*/false),
897 IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), IRB.getInt8PtrTy(),
898 IRB.getInt32Ty());
899 }
900
901 MsanSetAllocaOriginWithDescriptionFn = M.getOrInsertFunction(
902 "__msan_set_alloca_origin_with_descr", IRB.getVoidTy(),
903 IRB.getInt8PtrTy(), IntptrTy, IRB.getInt8PtrTy(), IRB.getInt8PtrTy());
904 MsanSetAllocaOriginNoDescriptionFn = M.getOrInsertFunction(
905 "__msan_set_alloca_origin_no_descr", IRB.getVoidTy(), IRB.getInt8PtrTy(),
906 IntptrTy, IRB.getInt8PtrTy());
907 MsanPoisonStackFn = M.getOrInsertFunction(
908 "__msan_poison_stack", IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy);
909}
910
911/// Insert extern declaration of runtime-provided functions and globals.
912void MemorySanitizer::initializeCallbacks(Module &M, const TargetLibraryInfo &TLI) {
913 // Only do this once.
914 if (CallbacksInitialized)
915 return;
916
917 IRBuilder<> IRB(*C);
918 // Initialize callbacks that are common for kernel and userspace
919 // instrumentation.
920 MsanChainOriginFn = M.getOrInsertFunction(
921 "__msan_chain_origin",
922 TLI.getAttrList(C, {0}, /*Signed=*/false, /*Ret=*/true), IRB.getInt32Ty(),
923 IRB.getInt32Ty());
924 MsanSetOriginFn = M.getOrInsertFunction(
925 "__msan_set_origin", TLI.getAttrList(C, {2}, /*Signed=*/false),
926 IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy, IRB.getInt32Ty());
927 MemmoveFn =
928 M.getOrInsertFunction("__msan_memmove", IRB.getInt8PtrTy(),
929 IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IntptrTy);
930 MemcpyFn =
931 M.getOrInsertFunction("__msan_memcpy", IRB.getInt8PtrTy(),
932 IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IntptrTy);
933 MemsetFn = M.getOrInsertFunction(
934 "__msan_memset", TLI.getAttrList(C, {1}, /*Signed=*/true),
935 IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt32Ty(), IntptrTy);
936
937 MsanInstrumentAsmStoreFn =
938 M.getOrInsertFunction("__msan_instrument_asm_store", IRB.getVoidTy(),
939 PointerType::get(IRB.getInt8Ty(), 0), IntptrTy);
940
941 if (CompileKernel) {
942 createKernelApi(M, TLI);
943 } else {
944 createUserspaceApi(M, TLI);
945 }
946 CallbacksInitialized = true;
947}
948
949FunctionCallee MemorySanitizer::getKmsanShadowOriginAccessFn(bool isStore,
950 int size) {
951 FunctionCallee *Fns =
952 isStore ? MsanMetadataPtrForStore_1_8 : MsanMetadataPtrForLoad_1_8;
953 switch (size) {
954 case 1:
955 return Fns[0];
956 case 2:
957 return Fns[1];
958 case 4:
959 return Fns[2];
960 case 8:
961 return Fns[3];
962 default:
963 return nullptr;
964 }
965}
966
967/// Module-level initialization.
968///
969/// inserts a call to __msan_init to the module's constructor list.
970void MemorySanitizer::initializeModule(Module &M) {
971 auto &DL = M.getDataLayout();
972
973 TargetTriple = Triple(M.getTargetTriple());
974
975 bool ShadowPassed = ClShadowBase.getNumOccurrences() > 0;
976 bool OriginPassed = ClOriginBase.getNumOccurrences() > 0;
977 // Check the overrides first
978 if (ShadowPassed || OriginPassed) {
979 CustomMapParams.AndMask = ClAndMask;
980 CustomMapParams.XorMask = ClXorMask;
981 CustomMapParams.ShadowBase = ClShadowBase;
982 CustomMapParams.OriginBase = ClOriginBase;
983 MapParams = &CustomMapParams;
984 } else {
985 switch (TargetTriple.getOS()) {
986 case Triple::FreeBSD:
987 switch (TargetTriple.getArch()) {
988 case Triple::aarch64:
989 MapParams = FreeBSD_ARM_MemoryMapParams.bits64;
990 break;
991 case Triple::x86_64:
992 MapParams = FreeBSD_X86_MemoryMapParams.bits64;
993 break;
994 case Triple::x86:
995 MapParams = FreeBSD_X86_MemoryMapParams.bits32;
996 break;
997 default:
998 report_fatal_error("unsupported architecture");
999 }
1000 break;
1001 case Triple::NetBSD:
1002 switch (TargetTriple.getArch()) {
1003 case Triple::x86_64:
1004 MapParams = NetBSD_X86_MemoryMapParams.bits64;
1005 break;
1006 default:
1007 report_fatal_error("unsupported architecture");
1008 }
1009 break;
1010 case Triple::Linux:
1011 switch (TargetTriple.getArch()) {
1012 case Triple::x86_64:
1013 MapParams = Linux_X86_MemoryMapParams.bits64;
1014 break;
1015 case Triple::x86:
1016 MapParams = Linux_X86_MemoryMapParams.bits32;
1017 break;
1018 case Triple::mips64:
1019 case Triple::mips64el:
1020 MapParams = Linux_MIPS_MemoryMapParams.bits64;
1021 break;
1022 case Triple::ppc64:
1023 case Triple::ppc64le:
1024 MapParams = Linux_PowerPC_MemoryMapParams.bits64;
1025 break;
1026 case Triple::systemz:
1027 MapParams = Linux_S390_MemoryMapParams.bits64;
1028 break;
1029 case Triple::aarch64:
1030 case Triple::aarch64_be:
1031 MapParams = Linux_ARM_MemoryMapParams.bits64;
1032 break;
1034 MapParams = Linux_LoongArch_MemoryMapParams.bits64;
1035 break;
1036 default:
1037 report_fatal_error("unsupported architecture");
1038 }
1039 break;
1040 default:
1041 report_fatal_error("unsupported operating system");
1042 }
1043 }
1044
1045 C = &(M.getContext());
1046 IRBuilder<> IRB(*C);
1047 IntptrTy = IRB.getIntPtrTy(DL);
1048 OriginTy = IRB.getInt32Ty();
1049
1050 ColdCallWeights = MDBuilder(*C).createBranchWeights(1, 1000);
1051 OriginStoreWeights = MDBuilder(*C).createBranchWeights(1, 1000);
1052
1053 if (!CompileKernel) {
1054 if (TrackOrigins)
1055 M.getOrInsertGlobal("__msan_track_origins", IRB.getInt32Ty(), [&] {
1056 return new GlobalVariable(
1057 M, IRB.getInt32Ty(), true, GlobalValue::WeakODRLinkage,
1058 IRB.getInt32(TrackOrigins), "__msan_track_origins");
1059 });
1060
1061 if (Recover)
1062 M.getOrInsertGlobal("__msan_keep_going", IRB.getInt32Ty(), [&] {
1063 return new GlobalVariable(M, IRB.getInt32Ty(), true,
1064 GlobalValue::WeakODRLinkage,
1065 IRB.getInt32(Recover), "__msan_keep_going");
1066 });
1067 }
1068}
1069
1070namespace {
1071
1072/// A helper class that handles instrumentation of VarArg
1073/// functions on a particular platform.
1074///
1075/// Implementations are expected to insert the instrumentation
1076/// necessary to propagate argument shadow through VarArg function
1077/// calls. Visit* methods are called during an InstVisitor pass over
1078/// the function, and should avoid creating new basic blocks. A new
1079/// instance of this class is created for each instrumented function.
1080struct VarArgHelper {
1081 virtual ~VarArgHelper() = default;
1082
1083 /// Visit a CallBase.
1084 virtual void visitCallBase(CallBase &CB, IRBuilder<> &IRB) = 0;
1085
1086 /// Visit a va_start call.
1087 virtual void visitVAStartInst(VAStartInst &I) = 0;
1088
1089 /// Visit a va_copy call.
1090 virtual void visitVACopyInst(VACopyInst &I) = 0;
1091
1092 /// Finalize function instrumentation.
1093 ///
1094 /// This method is called after visiting all interesting (see above)
1095 /// instructions in a function.
1096 virtual void finalizeInstrumentation() = 0;
1097};
1098
1099struct MemorySanitizerVisitor;
1100
1101} // end anonymous namespace
1102
1103static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
1104 MemorySanitizerVisitor &Visitor);
1105
1106static unsigned TypeSizeToSizeIndex(TypeSize TS) {
1107 if (TS.isScalable())
1108 // Scalable types unconditionally take slowpaths.
1109 return kNumberOfAccessSizes;
1110 unsigned TypeSizeFixed = TS.getFixedValue();
1111 if (TypeSizeFixed <= 8)
1112 return 0;
1113 return Log2_32_Ceil((TypeSizeFixed + 7) / 8);
1114}
1115
1116namespace {
1117
1118/// Helper class to attach debug information of the given instruction onto new
1119/// instructions inserted after.
1120class NextNodeIRBuilder : public IRBuilder<> {
1121public:
1122 explicit NextNodeIRBuilder(Instruction *IP) : IRBuilder<>(IP->getNextNode()) {
1123 SetCurrentDebugLocation(IP->getDebugLoc());
1124 }
1125};
1126
1127/// This class does all the work for a given function. Store and Load
1128/// instructions store and load corresponding shadow and origin
1129/// values. Most instructions propagate shadow from arguments to their
1130/// return values. Certain instructions (most importantly, BranchInst)
1131/// test their argument shadow and print reports (with a runtime call) if it's
1132/// non-zero.
1133struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
1134 Function &F;
1135 MemorySanitizer &MS;
1136 SmallVector<PHINode *, 16> ShadowPHINodes, OriginPHINodes;
1137 ValueMap<Value *, Value *> ShadowMap, OriginMap;
1138 std::unique_ptr<VarArgHelper> VAHelper;
1139 const TargetLibraryInfo *TLI;
1140 Instruction *FnPrologueEnd;
1141
1142 // The following flags disable parts of MSan instrumentation based on
1143 // exclusion list contents and command-line options.
1144 bool InsertChecks;
1145 bool PropagateShadow;
1146 bool PoisonStack;
1147 bool PoisonUndef;
1148
1149 struct ShadowOriginAndInsertPoint {
1150 Value *Shadow;
1151 Value *Origin;
1152 Instruction *OrigIns;
1153
1154 ShadowOriginAndInsertPoint(Value *S, Value *O, Instruction *I)
1155 : Shadow(S), Origin(O), OrigIns(I) {}
1156 };
1158 DenseMap<const DILocation *, int> LazyWarningDebugLocationCount;
1159 bool InstrumentLifetimeStart = ClHandleLifetimeIntrinsics;
1163 int64_t SplittableBlocksCount = 0;
1164
1165 MemorySanitizerVisitor(Function &F, MemorySanitizer &MS,
1166 const TargetLibraryInfo &TLI)
1167 : F(F), MS(MS), VAHelper(CreateVarArgHelper(F, MS, *this)), TLI(&TLI) {
1168 bool SanitizeFunction =
1169 F.hasFnAttribute(Attribute::SanitizeMemory) && !ClDisableChecks;
1170 InsertChecks = SanitizeFunction;
1171 PropagateShadow = SanitizeFunction;
1172 PoisonStack = SanitizeFunction && ClPoisonStack;
1173 PoisonUndef = SanitizeFunction && ClPoisonUndef;
1174
1175 // In the presence of unreachable blocks, we may see Phi nodes with
1176 // incoming nodes from such blocks. Since InstVisitor skips unreachable
1177 // blocks, such nodes will not have any shadow value associated with them.
1178 // It's easier to remove unreachable blocks than deal with missing shadow.
1180
1181 MS.initializeCallbacks(*F.getParent(), TLI);
1182 FnPrologueEnd = IRBuilder<>(F.getEntryBlock().getFirstNonPHI())
1183 .CreateIntrinsic(Intrinsic::donothing, {}, {});
1184
1185 if (MS.CompileKernel) {
1186 IRBuilder<> IRB(FnPrologueEnd);
1187 insertKmsanPrologue(IRB);
1188 }
1189
1190 LLVM_DEBUG(if (!InsertChecks) dbgs()
1191 << "MemorySanitizer is not inserting checks into '"
1192 << F.getName() << "'\n");
1193 }
1194
1195 bool instrumentWithCalls(Value *V) {
1196 // Constants likely will be eliminated by follow-up passes.
1197 if (isa<Constant>(V))
1198 return false;
1199
1200 ++SplittableBlocksCount;
1202 SplittableBlocksCount > ClInstrumentationWithCallThreshold;
1203 }
1204
1205 bool isInPrologue(Instruction &I) {
1206 return I.getParent() == FnPrologueEnd->getParent() &&
1207 (&I == FnPrologueEnd || I.comesBefore(FnPrologueEnd));
1208 }
1209
1210 // Creates a new origin and records the stack trace. In general we can call
1211 // this function for any origin manipulation we like. However it will cost
1212 // runtime resources. So use this wisely only if it can provide additional
1213 // information helpful to a user.
1214 Value *updateOrigin(Value *V, IRBuilder<> &IRB) {
1215 if (MS.TrackOrigins <= 1)
1216 return V;
1217 return IRB.CreateCall(MS.MsanChainOriginFn, V);
1218 }
1219
1220 Value *originToIntptr(IRBuilder<> &IRB, Value *Origin) {
1221 const DataLayout &DL = F.getParent()->getDataLayout();
1222 unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1223 if (IntptrSize == kOriginSize)
1224 return Origin;
1225 assert(IntptrSize == kOriginSize * 2);
1226 Origin = IRB.CreateIntCast(Origin, MS.IntptrTy, /* isSigned */ false);
1227 return IRB.CreateOr(Origin, IRB.CreateShl(Origin, kOriginSize * 8));
1228 }
1229
1230 /// Fill memory range with the given origin value.
1231 void paintOrigin(IRBuilder<> &IRB, Value *Origin, Value *OriginPtr,
1232 TypeSize TS, Align Alignment) {
1233 const DataLayout &DL = F.getParent()->getDataLayout();
1234 const Align IntptrAlignment = DL.getABITypeAlign(MS.IntptrTy);
1235 unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1236 assert(IntptrAlignment >= kMinOriginAlignment);
1237 assert(IntptrSize >= kOriginSize);
1238
1239 // Note: The loop based formation works for fixed length vectors too,
1240 // however we prefer to unroll and specialize alignment below.
1241 if (TS.isScalable()) {
1242 Value *Size = IRB.CreateTypeSize(IRB.getInt32Ty(), TS);
1243 Value *RoundUp = IRB.CreateAdd(Size, IRB.getInt32(kOriginSize - 1));
1244 Value *End = IRB.CreateUDiv(RoundUp, IRB.getInt32(kOriginSize));
1245 auto [InsertPt, Index] =
1247 IRB.SetInsertPoint(InsertPt);
1248
1249 Value *GEP = IRB.CreateGEP(MS.OriginTy, OriginPtr, Index);
1251 return;
1252 }
1253
1254 unsigned Size = TS.getFixedValue();
1255
1256 unsigned Ofs = 0;
1257 Align CurrentAlignment = Alignment;
1258 if (Alignment >= IntptrAlignment && IntptrSize > kOriginSize) {
1259 Value *IntptrOrigin = originToIntptr(IRB, Origin);
1260 Value *IntptrOriginPtr =
1261 IRB.CreatePointerCast(OriginPtr, PointerType::get(MS.IntptrTy, 0));
1262 for (unsigned i = 0; i < Size / IntptrSize; ++i) {
1263 Value *Ptr = i ? IRB.CreateConstGEP1_32(MS.IntptrTy, IntptrOriginPtr, i)
1264 : IntptrOriginPtr;
1265 IRB.CreateAlignedStore(IntptrOrigin, Ptr, CurrentAlignment);
1266 Ofs += IntptrSize / kOriginSize;
1267 CurrentAlignment = IntptrAlignment;
1268 }
1269 }
1270
1271 for (unsigned i = Ofs; i < (Size + kOriginSize - 1) / kOriginSize; ++i) {
1272 Value *GEP =
1273 i ? IRB.CreateConstGEP1_32(MS.OriginTy, OriginPtr, i) : OriginPtr;
1274 IRB.CreateAlignedStore(Origin, GEP, CurrentAlignment);
1275 CurrentAlignment = kMinOriginAlignment;
1276 }
1277 }
1278
1279 void storeOrigin(IRBuilder<> &IRB, Value *Addr, Value *Shadow, Value *Origin,
1280 Value *OriginPtr, Align Alignment) {
1281 const DataLayout &DL = F.getParent()->getDataLayout();
1282 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1283 TypeSize StoreSize = DL.getTypeStoreSize(Shadow->getType());
1284 Value *ConvertedShadow = convertShadowToScalar(Shadow, IRB);
1285 if (auto *ConstantShadow = dyn_cast<Constant>(ConvertedShadow)) {
1286 if (!ClCheckConstantShadow || ConstantShadow->isZeroValue()) {
1287 // Origin is not needed: value is initialized or const shadow is
1288 // ignored.
1289 return;
1290 }
1291 if (llvm::isKnownNonZero(ConvertedShadow, DL)) {
1292 // Copy origin as the value is definitely uninitialized.
1293 paintOrigin(IRB, updateOrigin(Origin, IRB), OriginPtr, StoreSize,
1294 OriginAlignment);
1295 return;
1296 }
1297 // Fallback to runtime check, which still can be optimized out later.
1298 }
1299
1300 TypeSize TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
1301 unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1302 if (instrumentWithCalls(ConvertedShadow) &&
1303 SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
1304 FunctionCallee Fn = MS.MaybeStoreOriginFn[SizeIndex];
1305 Value *ConvertedShadow2 =
1306 IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1307 CallBase *CB = IRB.CreateCall(
1308 Fn, {ConvertedShadow2,
1309 IRB.CreatePointerCast(Addr, IRB.getInt8PtrTy()), Origin});
1310 CB->addParamAttr(0, Attribute::ZExt);
1311 CB->addParamAttr(2, Attribute::ZExt);
1312 } else {
1313 Value *Cmp = convertToBool(ConvertedShadow, IRB, "_mscmp");
1315 Cmp, &*IRB.GetInsertPoint(), false, MS.OriginStoreWeights);
1316 IRBuilder<> IRBNew(CheckTerm);
1317 paintOrigin(IRBNew, updateOrigin(Origin, IRBNew), OriginPtr, StoreSize,
1318 OriginAlignment);
1319 }
1320 }
1321
1322 void materializeStores() {
1323 for (StoreInst *SI : StoreList) {
1324 IRBuilder<> IRB(SI);
1325 Value *Val = SI->getValueOperand();
1326 Value *Addr = SI->getPointerOperand();
1327 Value *Shadow = SI->isAtomic() ? getCleanShadow(Val) : getShadow(Val);
1328 Value *ShadowPtr, *OriginPtr;
1329 Type *ShadowTy = Shadow->getType();
1330 const Align Alignment = SI->getAlign();
1331 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1332 std::tie(ShadowPtr, OriginPtr) =
1333 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ true);
1334
1335 StoreInst *NewSI = IRB.CreateAlignedStore(Shadow, ShadowPtr, Alignment);
1336 LLVM_DEBUG(dbgs() << " STORE: " << *NewSI << "\n");
1337 (void)NewSI;
1338
1339 if (SI->isAtomic())
1340 SI->setOrdering(addReleaseOrdering(SI->getOrdering()));
1341
1342 if (MS.TrackOrigins && !SI->isAtomic())
1343 storeOrigin(IRB, Addr, Shadow, getOrigin(Val), OriginPtr,
1344 OriginAlignment);
1345 }
1346 }
1347
1348 // Returns true if Debug Location curresponds to multiple warnings.
1349 bool shouldDisambiguateWarningLocation(const DebugLoc &DebugLoc) {
1350 if (MS.TrackOrigins < 2)
1351 return false;
1352
1353 if (LazyWarningDebugLocationCount.empty())
1354 for (const auto &I : InstrumentationList)
1355 ++LazyWarningDebugLocationCount[I.OrigIns->getDebugLoc()];
1356
1357 return LazyWarningDebugLocationCount[DebugLoc] >= ClDisambiguateWarning;
1358 }
1359
1360 /// Helper function to insert a warning at IRB's current insert point.
1361 void insertWarningFn(IRBuilder<> &IRB, Value *Origin) {
1362 if (!Origin)
1363 Origin = (Value *)IRB.getInt32(0);
1364 assert(Origin->getType()->isIntegerTy());
1365
1366 if (shouldDisambiguateWarningLocation(IRB.getCurrentDebugLocation())) {
1367 // Try to create additional origin with debug info of the last origin
1368 // instruction. It may provide additional information to the user.
1369 if (Instruction *OI = dyn_cast_or_null<Instruction>(Origin)) {
1370 assert(MS.TrackOrigins);
1371 auto NewDebugLoc = OI->getDebugLoc();
1372 // Origin update with missing or the same debug location provides no
1373 // additional value.
1374 if (NewDebugLoc && NewDebugLoc != IRB.getCurrentDebugLocation()) {
1375 // Insert update just before the check, so we call runtime only just
1376 // before the report.
1377 IRBuilder<> IRBOrigin(&*IRB.GetInsertPoint());
1378 IRBOrigin.SetCurrentDebugLocation(NewDebugLoc);
1379 Origin = updateOrigin(Origin, IRBOrigin);
1380 }
1381 }
1382 }
1383
1384 if (MS.CompileKernel || MS.TrackOrigins)
1385 IRB.CreateCall(MS.WarningFn, Origin)->setCannotMerge();
1386 else
1387 IRB.CreateCall(MS.WarningFn)->setCannotMerge();
1388 // FIXME: Insert UnreachableInst if !MS.Recover?
1389 // This may invalidate some of the following checks and needs to be done
1390 // at the very end.
1391 }
1392
1393 void materializeOneCheck(IRBuilder<> &IRB, Value *ConvertedShadow,
1394 Value *Origin) {
1395 const DataLayout &DL = F.getParent()->getDataLayout();
1396 TypeSize TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
1397 unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1398 if (instrumentWithCalls(ConvertedShadow) &&
1399 SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
1400 FunctionCallee Fn = MS.MaybeWarningFn[SizeIndex];
1401 Value *ConvertedShadow2 =
1402 IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1403 CallBase *CB = IRB.CreateCall(
1404 Fn, {ConvertedShadow2,
1405 MS.TrackOrigins && Origin ? Origin : (Value *)IRB.getInt32(0)});
1406 CB->addParamAttr(0, Attribute::ZExt);
1407 CB->addParamAttr(1, Attribute::ZExt);
1408 } else {
1409 Value *Cmp = convertToBool(ConvertedShadow, IRB, "_mscmp");
1411 Cmp, &*IRB.GetInsertPoint(),
1412 /* Unreachable */ !MS.Recover, MS.ColdCallWeights);
1413
1414 IRB.SetInsertPoint(CheckTerm);
1415 insertWarningFn(IRB, Origin);
1416 LLVM_DEBUG(dbgs() << " CHECK: " << *Cmp << "\n");
1417 }
1418 }
1419
1420 void materializeInstructionChecks(
1421 ArrayRef<ShadowOriginAndInsertPoint> InstructionChecks) {
1422 const DataLayout &DL = F.getParent()->getDataLayout();
1423 // Disable combining in some cases. TrackOrigins checks each shadow to pick
1424 // correct origin.
1425 bool Combine = !MS.TrackOrigins;
1426 Instruction *Instruction = InstructionChecks.front().OrigIns;
1427 Value *Shadow = nullptr;
1428 for (const auto &ShadowData : InstructionChecks) {
1429 assert(ShadowData.OrigIns == Instruction);
1431
1432 Value *ConvertedShadow = ShadowData.Shadow;
1433
1434 if (auto *ConstantShadow = dyn_cast<Constant>(ConvertedShadow)) {
1435 if (!ClCheckConstantShadow || ConstantShadow->isZeroValue()) {
1436 // Skip, value is initialized or const shadow is ignored.
1437 continue;
1438 }
1439 if (llvm::isKnownNonZero(ConvertedShadow, DL)) {
1440 // Report as the value is definitely uninitialized.
1441 insertWarningFn(IRB, ShadowData.Origin);
1442 if (!MS.Recover)
1443 return; // Always fail and stop here, not need to check the rest.
1444 // Skip entire instruction,
1445 continue;
1446 }
1447 // Fallback to runtime check, which still can be optimized out later.
1448 }
1449
1450 if (!Combine) {
1451 materializeOneCheck(IRB, ConvertedShadow, ShadowData.Origin);
1452 continue;
1453 }
1454
1455 if (!Shadow) {
1456 Shadow = ConvertedShadow;
1457 continue;
1458 }
1459
1460 Shadow = convertToBool(Shadow, IRB, "_mscmp");
1461 ConvertedShadow = convertToBool(ConvertedShadow, IRB, "_mscmp");
1462 Shadow = IRB.CreateOr(Shadow, ConvertedShadow, "_msor");
1463 }
1464
1465 if (Shadow) {
1466 assert(Combine);
1468 materializeOneCheck(IRB, Shadow, nullptr);
1469 }
1470 }
1471
1472 void materializeChecks() {
1473 llvm::stable_sort(InstrumentationList,
1474 [](const ShadowOriginAndInsertPoint &L,
1475 const ShadowOriginAndInsertPoint &R) {
1476 return L.OrigIns < R.OrigIns;
1477 });
1478
1479 for (auto I = InstrumentationList.begin();
1480 I != InstrumentationList.end();) {
1481 auto J =
1482 std::find_if(I + 1, InstrumentationList.end(),
1483 [L = I->OrigIns](const ShadowOriginAndInsertPoint &R) {
1484 return L != R.OrigIns;
1485 });
1486 // Process all checks of instruction at once.
1487 materializeInstructionChecks(ArrayRef<ShadowOriginAndInsertPoint>(I, J));
1488 I = J;
1489 }
1490
1491 LLVM_DEBUG(dbgs() << "DONE:\n" << F);
1492 }
1493
1494 // Returns the last instruction in the new prologue
1495 void insertKmsanPrologue(IRBuilder<> &IRB) {
1496 Value *ContextState = IRB.CreateCall(MS.MsanGetContextStateFn, {});
1497 Constant *Zero = IRB.getInt32(0);
1498 MS.ParamTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1499 {Zero, IRB.getInt32(0)}, "param_shadow");
1500 MS.RetvalTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1501 {Zero, IRB.getInt32(1)}, "retval_shadow");
1502 MS.VAArgTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1503 {Zero, IRB.getInt32(2)}, "va_arg_shadow");
1504 MS.VAArgOriginTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1505 {Zero, IRB.getInt32(3)}, "va_arg_origin");
1506 MS.VAArgOverflowSizeTLS =
1507 IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1508 {Zero, IRB.getInt32(4)}, "va_arg_overflow_size");
1509 MS.ParamOriginTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1510 {Zero, IRB.getInt32(5)}, "param_origin");
1511 MS.RetvalOriginTLS =
1512 IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1513 {Zero, IRB.getInt32(6)}, "retval_origin");
1514 if (MS.TargetTriple.getArch() == Triple::systemz)
1515 MS.MsanMetadataAlloca = IRB.CreateAlloca(MS.MsanMetadata, 0u);
1516 }
1517
1518 /// Add MemorySanitizer instrumentation to a function.
1519 bool runOnFunction() {
1520 // Iterate all BBs in depth-first order and create shadow instructions
1521 // for all instructions (where applicable).
1522 // For PHI nodes we create dummy shadow PHIs which will be finalized later.
1523 for (BasicBlock *BB : depth_first(FnPrologueEnd->getParent()))
1524 visit(*BB);
1525
1526 // Finalize PHI nodes.
1527 for (PHINode *PN : ShadowPHINodes) {
1528 PHINode *PNS = cast<PHINode>(getShadow(PN));
1529 PHINode *PNO = MS.TrackOrigins ? cast<PHINode>(getOrigin(PN)) : nullptr;
1530 size_t NumValues = PN->getNumIncomingValues();
1531 for (size_t v = 0; v < NumValues; v++) {
1532 PNS->addIncoming(getShadow(PN, v), PN->getIncomingBlock(v));
1533 if (PNO)
1534 PNO->addIncoming(getOrigin(PN, v), PN->getIncomingBlock(v));
1535 }
1536 }
1537
1538 VAHelper->finalizeInstrumentation();
1539
1540 // Poison llvm.lifetime.start intrinsics, if we haven't fallen back to
1541 // instrumenting only allocas.
1542 if (InstrumentLifetimeStart) {
1543 for (auto Item : LifetimeStartList) {
1544 instrumentAlloca(*Item.second, Item.first);
1545 AllocaSet.remove(Item.second);
1546 }
1547 }
1548 // Poison the allocas for which we didn't instrument the corresponding
1549 // lifetime intrinsics.
1550 for (AllocaInst *AI : AllocaSet)
1551 instrumentAlloca(*AI);
1552
1553 // Insert shadow value checks.
1554 materializeChecks();
1555
1556 // Delayed instrumentation of StoreInst.
1557 // This may not add new address checks.
1558 materializeStores();
1559
1560 return true;
1561 }
1562
1563 /// Compute the shadow type that corresponds to a given Value.
1564 Type *getShadowTy(Value *V) { return getShadowTy(V->getType()); }
1565
1566 /// Compute the shadow type that corresponds to a given Type.
1567 Type *getShadowTy(Type *OrigTy) {
1568 if (!OrigTy->isSized()) {
1569 return nullptr;
1570 }
1571 // For integer type, shadow is the same as the original type.
1572 // This may return weird-sized types like i1.
1573 if (IntegerType *IT = dyn_cast<IntegerType>(OrigTy))
1574 return IT;
1575 const DataLayout &DL = F.getParent()->getDataLayout();
1576 if (VectorType *VT = dyn_cast<VectorType>(OrigTy)) {
1577 uint32_t EltSize = DL.getTypeSizeInBits(VT->getElementType());
1578 return VectorType::get(IntegerType::get(*MS.C, EltSize),
1579 VT->getElementCount());
1580 }
1581 if (ArrayType *AT = dyn_cast<ArrayType>(OrigTy)) {
1582 return ArrayType::get(getShadowTy(AT->getElementType()),
1583 AT->getNumElements());
1584 }
1585 if (StructType *ST = dyn_cast<StructType>(OrigTy)) {
1587 for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
1588 Elements.push_back(getShadowTy(ST->getElementType(i)));
1589 StructType *Res = StructType::get(*MS.C, Elements, ST->isPacked());
1590 LLVM_DEBUG(dbgs() << "getShadowTy: " << *ST << " ===> " << *Res << "\n");
1591 return Res;
1592 }
1593 uint32_t TypeSize = DL.getTypeSizeInBits(OrigTy);
1594 return IntegerType::get(*MS.C, TypeSize);
1595 }
1596
1597 /// Extract combined shadow of struct elements as a bool
1598 Value *collapseStructShadow(StructType *Struct, Value *Shadow,
1599 IRBuilder<> &IRB) {
1600 Value *FalseVal = IRB.getIntN(/* width */ 1, /* value */ 0);
1601 Value *Aggregator = FalseVal;
1602
1603 for (unsigned Idx = 0; Idx < Struct->getNumElements(); Idx++) {
1604 // Combine by ORing together each element's bool shadow
1605 Value *ShadowItem = IRB.CreateExtractValue(Shadow, Idx);
1606 Value *ShadowBool = convertToBool(ShadowItem, IRB);
1607
1608 if (Aggregator != FalseVal)
1609 Aggregator = IRB.CreateOr(Aggregator, ShadowBool);
1610 else
1611 Aggregator = ShadowBool;
1612 }
1613
1614 return Aggregator;
1615 }
1616
1617 // Extract combined shadow of array elements
1618 Value *collapseArrayShadow(ArrayType *Array, Value *Shadow,
1619 IRBuilder<> &IRB) {
1620 if (!Array->getNumElements())
1621 return IRB.getIntN(/* width */ 1, /* value */ 0);
1622
1623 Value *FirstItem = IRB.CreateExtractValue(Shadow, 0);
1624 Value *Aggregator = convertShadowToScalar(FirstItem, IRB);
1625
1626 for (unsigned Idx = 1; Idx < Array->getNumElements(); Idx++) {
1627 Value *ShadowItem = IRB.CreateExtractValue(Shadow, Idx);
1628 Value *ShadowInner = convertShadowToScalar(ShadowItem, IRB);
1629 Aggregator = IRB.CreateOr(Aggregator, ShadowInner);
1630 }
1631 return Aggregator;
1632 }
1633
1634 /// Convert a shadow value to it's flattened variant. The resulting
1635 /// shadow may not necessarily have the same bit width as the input
1636 /// value, but it will always be comparable to zero.
1637 Value *convertShadowToScalar(Value *V, IRBuilder<> &IRB) {
1638 if (StructType *Struct = dyn_cast<StructType>(V->getType()))
1639 return collapseStructShadow(Struct, V, IRB);
1640 if (ArrayType *Array = dyn_cast<ArrayType>(V->getType()))
1641 return collapseArrayShadow(Array, V, IRB);
1642 if (isa<VectorType>(V->getType())) {
1643 if (isa<ScalableVectorType>(V->getType()))
1644 return convertShadowToScalar(IRB.CreateOrReduce(V), IRB);
1645 unsigned BitWidth =
1646 V->getType()->getPrimitiveSizeInBits().getFixedValue();
1647 return IRB.CreateBitCast(V, IntegerType::get(*MS.C, BitWidth));
1648 }
1649 return V;
1650 }
1651
1652 // Convert a scalar value to an i1 by comparing with 0
1653 Value *convertToBool(Value *V, IRBuilder<> &IRB, const Twine &name = "") {
1654 Type *VTy = V->getType();
1655 if (!VTy->isIntegerTy())
1656 return convertToBool(convertShadowToScalar(V, IRB), IRB, name);
1657 if (VTy->getIntegerBitWidth() == 1)
1658 // Just converting a bool to a bool, so do nothing.
1659 return V;
1660 return IRB.CreateICmpNE(V, ConstantInt::get(VTy, 0), name);
1661 }
1662
1663 Type *ptrToIntPtrType(Type *PtrTy) const {
1664 if (VectorType *VectTy = dyn_cast<VectorType>(PtrTy)) {
1665 return VectorType::get(ptrToIntPtrType(VectTy->getElementType()),
1666 VectTy->getElementCount());
1667 }
1668 assert(PtrTy->isIntOrPtrTy());
1669 return MS.IntptrTy;
1670 }
1671
1672 Type *getPtrToShadowPtrType(Type *IntPtrTy, Type *ShadowTy) const {
1673 if (VectorType *VectTy = dyn_cast<VectorType>(IntPtrTy)) {
1674 return VectorType::get(
1675 getPtrToShadowPtrType(VectTy->getElementType(), ShadowTy),
1676 VectTy->getElementCount());
1677 }
1678 assert(IntPtrTy == MS.IntptrTy);
1679 return PointerType::get(*MS.C, 0);
1680 }
1681
1682 Constant *constToIntPtr(Type *IntPtrTy, uint64_t C) const {
1683 if (VectorType *VectTy = dyn_cast<VectorType>(IntPtrTy)) {
1685 VectTy->getElementCount(), constToIntPtr(VectTy->getElementType(), C));
1686 }
1687 assert(IntPtrTy == MS.IntptrTy);
1688 return ConstantInt::get(MS.IntptrTy, C);
1689 }
1690
1691 /// Compute the integer shadow offset that corresponds to a given
1692 /// application address.
1693 ///
1694 /// Offset = (Addr & ~AndMask) ^ XorMask
1695 /// Addr can be a ptr or <N x ptr>. In both cases ShadowTy the shadow type of
1696 /// a single pointee.
1697 /// Returns <shadow_ptr, origin_ptr> or <<N x shadow_ptr>, <N x origin_ptr>>.
1698 Value *getShadowPtrOffset(Value *Addr, IRBuilder<> &IRB) {
1699 Type *IntptrTy = ptrToIntPtrType(Addr->getType());
1700 Value *OffsetLong = IRB.CreatePointerCast(Addr, IntptrTy);
1701
1702 if (uint64_t AndMask = MS.MapParams->AndMask)
1703 OffsetLong = IRB.CreateAnd(OffsetLong, constToIntPtr(IntptrTy, ~AndMask));
1704
1705 if (uint64_t XorMask = MS.MapParams->XorMask)
1706 OffsetLong = IRB.CreateXor(OffsetLong, constToIntPtr(IntptrTy, XorMask));
1707 return OffsetLong;
1708 }
1709
1710 /// Compute the shadow and origin addresses corresponding to a given
1711 /// application address.
1712 ///
1713 /// Shadow = ShadowBase + Offset
1714 /// Origin = (OriginBase + Offset) & ~3ULL
1715 /// Addr can be a ptr or <N x ptr>. In both cases ShadowTy the shadow type of
1716 /// a single pointee.
1717 /// Returns <shadow_ptr, origin_ptr> or <<N x shadow_ptr>, <N x origin_ptr>>.
1718 std::pair<Value *, Value *>
1719 getShadowOriginPtrUserspace(Value *Addr, IRBuilder<> &IRB, Type *ShadowTy,
1720 MaybeAlign Alignment) {
1721 Type *IntptrTy = ptrToIntPtrType(Addr->getType());
1722 Value *ShadowOffset = getShadowPtrOffset(Addr, IRB);
1723 Value *ShadowLong = ShadowOffset;
1724 if (uint64_t ShadowBase = MS.MapParams->ShadowBase) {
1725 ShadowLong =
1726 IRB.CreateAdd(ShadowLong, constToIntPtr(IntptrTy, ShadowBase));
1727 }
1728 Value *ShadowPtr = IRB.CreateIntToPtr(
1729 ShadowLong, getPtrToShadowPtrType(IntptrTy, ShadowTy));
1730
1731 Value *OriginPtr = nullptr;
1732 if (MS.TrackOrigins) {
1733 Value *OriginLong = ShadowOffset;
1734 uint64_t OriginBase = MS.MapParams->OriginBase;
1735 if (OriginBase != 0)
1736 OriginLong =
1737 IRB.CreateAdd(OriginLong, constToIntPtr(IntptrTy, OriginBase));
1738 if (!Alignment || *Alignment < kMinOriginAlignment) {
1740 OriginLong = IRB.CreateAnd(OriginLong, constToIntPtr(IntptrTy, ~Mask));
1741 }
1742 OriginPtr = IRB.CreateIntToPtr(
1743 OriginLong, getPtrToShadowPtrType(IntptrTy, MS.OriginTy));
1744 }
1745 return std::make_pair(ShadowPtr, OriginPtr);
1746 }
1747
1748 template <typename... ArgsTy>
1749 Value *createMetadataCall(IRBuilder<> &IRB, FunctionCallee Callee,
1750 ArgsTy... Args) {
1751 if (MS.TargetTriple.getArch() == Triple::systemz) {
1752 IRB.CreateCall(Callee,
1753 {MS.MsanMetadataAlloca, std::forward<ArgsTy>(Args)...});
1754 return IRB.CreateLoad(MS.MsanMetadata, MS.MsanMetadataAlloca);
1755 }
1756
1757 return IRB.CreateCall(Callee, {std::forward<ArgsTy>(Args)...});
1758 }
1759
1760 std::pair<Value *, Value *> getShadowOriginPtrKernelNoVec(Value *Addr,
1761 IRBuilder<> &IRB,
1762 Type *ShadowTy,
1763 bool isStore) {
1764 Value *ShadowOriginPtrs;
1765 const DataLayout &DL = F.getParent()->getDataLayout();
1766 TypeSize Size = DL.getTypeStoreSize(ShadowTy);
1767
1768 FunctionCallee Getter = MS.getKmsanShadowOriginAccessFn(isStore, Size);
1769 Value *AddrCast =
1770 IRB.CreatePointerCast(Addr, PointerType::get(IRB.getInt8Ty(), 0));
1771 if (Getter) {
1772 ShadowOriginPtrs = createMetadataCall(IRB, Getter, AddrCast);
1773 } else {
1774 Value *SizeVal = ConstantInt::get(MS.IntptrTy, Size);
1775 ShadowOriginPtrs = createMetadataCall(
1776 IRB,
1777 isStore ? MS.MsanMetadataPtrForStoreN : MS.MsanMetadataPtrForLoadN,
1778 AddrCast, SizeVal);
1779 }
1780 Value *ShadowPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 0);
1781 ShadowPtr = IRB.CreatePointerCast(ShadowPtr, PointerType::get(ShadowTy, 0));
1782 Value *OriginPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 1);
1783
1784 return std::make_pair(ShadowPtr, OriginPtr);
1785 }
1786
1787 /// Addr can be a ptr or <N x ptr>. In both cases ShadowTy the shadow type of
1788 /// a single pointee.
1789 /// Returns <shadow_ptr, origin_ptr> or <<N x shadow_ptr>, <N x origin_ptr>>.
1790 std::pair<Value *, Value *> getShadowOriginPtrKernel(Value *Addr,
1791 IRBuilder<> &IRB,
1792 Type *ShadowTy,
1793 bool isStore) {
1794 VectorType *VectTy = dyn_cast<VectorType>(Addr->getType());
1795 if (!VectTy) {
1796 assert(Addr->getType()->isPointerTy());
1797 return getShadowOriginPtrKernelNoVec(Addr, IRB, ShadowTy, isStore);
1798 }
1799
1800 // TODO: Support callbacs with vectors of addresses.
1801 unsigned NumElements = cast<FixedVectorType>(VectTy)->getNumElements();
1802 Value *ShadowPtrs = ConstantInt::getNullValue(
1803 FixedVectorType::get(IRB.getPtrTy(), NumElements));
1804 Value *OriginPtrs = nullptr;
1805 if (MS.TrackOrigins)
1806 OriginPtrs = ConstantInt::getNullValue(
1807 FixedVectorType::get(IRB.getPtrTy(), NumElements));
1808 for (unsigned i = 0; i < NumElements; ++i) {
1809 Value *OneAddr =
1811 auto [ShadowPtr, OriginPtr] =
1812 getShadowOriginPtrKernelNoVec(OneAddr, IRB, ShadowTy, isStore);
1813
1814 ShadowPtrs = IRB.CreateInsertElement(
1815 ShadowPtrs, ShadowPtr, ConstantInt::get(IRB.getInt32Ty(), i));
1816 if (MS.TrackOrigins)
1817 OriginPtrs = IRB.CreateInsertElement(
1818 OriginPtrs, OriginPtr, ConstantInt::get(IRB.getInt32Ty(), i));
1819 }
1820 return {ShadowPtrs, OriginPtrs};
1821 }
1822
1823 std::pair<Value *, Value *> getShadowOriginPtr(Value *Addr, IRBuilder<> &IRB,
1824 Type *ShadowTy,
1825 MaybeAlign Alignment,
1826 bool isStore) {
1827 if (MS.CompileKernel)
1828 return getShadowOriginPtrKernel(Addr, IRB, ShadowTy, isStore);
1829 return getShadowOriginPtrUserspace(Addr, IRB, ShadowTy, Alignment);
1830 }
1831
1832 /// Compute the shadow address for a given function argument.
1833 ///
1834 /// Shadow = ParamTLS+ArgOffset.
1835 Value *getShadowPtrForArgument(IRBuilder<> &IRB, int ArgOffset) {
1836 Value *Base = IRB.CreatePointerCast(MS.ParamTLS, MS.IntptrTy);
1837 if (ArgOffset)
1838 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1839 return IRB.CreateIntToPtr(Base, IRB.getPtrTy(0), "_msarg");
1840 }
1841
1842 /// Compute the origin address for a given function argument.
1843 Value *getOriginPtrForArgument(IRBuilder<> &IRB, int ArgOffset) {
1844 if (!MS.TrackOrigins)
1845 return nullptr;
1846 Value *Base = IRB.CreatePointerCast(MS.ParamOriginTLS, MS.IntptrTy);
1847 if (ArgOffset)
1848 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1849 return IRB.CreateIntToPtr(Base, IRB.getPtrTy(0), "_msarg_o");
1850 }
1851
1852 /// Compute the shadow address for a retval.
1853 Value *getShadowPtrForRetval(IRBuilder<> &IRB) {
1854 return IRB.CreatePointerCast(MS.RetvalTLS, IRB.getPtrTy(0), "_msret");
1855 }
1856
1857 /// Compute the origin address for a retval.
1858 Value *getOriginPtrForRetval() {
1859 // We keep a single origin for the entire retval. Might be too optimistic.
1860 return MS.RetvalOriginTLS;
1861 }
1862
1863 /// Set SV to be the shadow value for V.
1864 void setShadow(Value *V, Value *SV) {
1865 assert(!ShadowMap.count(V) && "Values may only have one shadow");
1866 ShadowMap[V] = PropagateShadow ? SV : getCleanShadow(V);
1867 }
1868
1869 /// Set Origin to be the origin value for V.
1870 void setOrigin(Value *V, Value *Origin) {
1871 if (!MS.TrackOrigins)
1872 return;
1873 assert(!OriginMap.count(V) && "Values may only have one origin");
1874 LLVM_DEBUG(dbgs() << "ORIGIN: " << *V << " ==> " << *Origin << "\n");
1875 OriginMap[V] = Origin;
1876 }
1877
1878 Constant *getCleanShadow(Type *OrigTy) {
1879 Type *ShadowTy = getShadowTy(OrigTy);
1880 if (!ShadowTy)
1881 return nullptr;
1882 return Constant::getNullValue(ShadowTy);
1883 }
1884
1885 /// Create a clean shadow value for a given value.
1886 ///
1887 /// Clean shadow (all zeroes) means all bits of the value are defined
1888 /// (initialized).
1889 Constant *getCleanShadow(Value *V) { return getCleanShadow(V->getType()); }
1890
1891 /// Create a dirty shadow of a given shadow type.
1892 Constant *getPoisonedShadow(Type *ShadowTy) {
1893 assert(ShadowTy);
1894 if (isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy))
1895 return Constant::getAllOnesValue(ShadowTy);
1896 if (ArrayType *AT = dyn_cast<ArrayType>(ShadowTy)) {
1897 SmallVector<Constant *, 4> Vals(AT->getNumElements(),
1898 getPoisonedShadow(AT->getElementType()));
1899 return ConstantArray::get(AT, Vals);
1900 }
1901 if (StructType *ST = dyn_cast<StructType>(ShadowTy)) {
1903 for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
1904 Vals.push_back(getPoisonedShadow(ST->getElementType(i)));
1905 return ConstantStruct::get(ST, Vals);
1906 }
1907 llvm_unreachable("Unexpected shadow type");
1908 }
1909
1910 /// Create a dirty shadow for a given value.
1911 Constant *getPoisonedShadow(Value *V) {
1912 Type *ShadowTy = getShadowTy(V);
1913 if (!ShadowTy)
1914 return nullptr;
1915 return getPoisonedShadow(ShadowTy);
1916 }
1917
1918 /// Create a clean (zero) origin.
1919 Value *getCleanOrigin() { return Constant::getNullValue(MS.OriginTy); }
1920
1921 /// Get the shadow value for a given Value.
1922 ///
1923 /// This function either returns the value set earlier with setShadow,
1924 /// or extracts if from ParamTLS (for function arguments).
1925 Value *getShadow(Value *V) {
1926 if (Instruction *I = dyn_cast<Instruction>(V)) {
1927 if (!PropagateShadow || I->getMetadata(LLVMContext::MD_nosanitize))
1928 return getCleanShadow(V);
1929 // For instructions the shadow is already stored in the map.
1930 Value *Shadow = ShadowMap[V];
1931 if (!Shadow) {
1932 LLVM_DEBUG(dbgs() << "No shadow: " << *V << "\n" << *(I->getParent()));
1933 (void)I;
1934 assert(Shadow && "No shadow for a value");
1935 }
1936 return Shadow;
1937 }
1938 if (UndefValue *U = dyn_cast<UndefValue>(V)) {
1939 Value *AllOnes = (PropagateShadow && PoisonUndef) ? getPoisonedShadow(V)
1940 : getCleanShadow(V);
1941 LLVM_DEBUG(dbgs() << "Undef: " << *U << " ==> " << *AllOnes << "\n");
1942 (void)U;
1943 return AllOnes;
1944 }
1945 if (Argument *A = dyn_cast<Argument>(V)) {
1946 // For arguments we compute the shadow on demand and store it in the map.
1947 Value *&ShadowPtr = ShadowMap[V];
1948 if (ShadowPtr)
1949 return ShadowPtr;
1950 Function *F = A->getParent();
1951 IRBuilder<> EntryIRB(FnPrologueEnd);
1952 unsigned ArgOffset = 0;
1953 const DataLayout &DL = F->getParent()->getDataLayout();
1954 for (auto &FArg : F->args()) {
1955 if (!FArg.getType()->isSized()) {
1956 LLVM_DEBUG(dbgs() << "Arg is not sized\n");
1957 continue;
1958 }
1959
1960 unsigned Size = FArg.hasByValAttr()
1961 ? DL.getTypeAllocSize(FArg.getParamByValType())
1962 : DL.getTypeAllocSize(FArg.getType());
1963
1964 if (A == &FArg) {
1965 bool Overflow = ArgOffset + Size > kParamTLSSize;
1966 if (FArg.hasByValAttr()) {
1967 // ByVal pointer itself has clean shadow. We copy the actual
1968 // argument shadow to the underlying memory.
1969 // Figure out maximal valid memcpy alignment.
1970 const Align ArgAlign = DL.getValueOrABITypeAlignment(
1971 FArg.getParamAlign(), FArg.getParamByValType());
1972 Value *CpShadowPtr, *CpOriginPtr;
1973 std::tie(CpShadowPtr, CpOriginPtr) =
1974 getShadowOriginPtr(V, EntryIRB, EntryIRB.getInt8Ty(), ArgAlign,
1975 /*isStore*/ true);
1976 if (!PropagateShadow || Overflow) {
1977 // ParamTLS overflow.
1978 EntryIRB.CreateMemSet(
1979 CpShadowPtr, Constant::getNullValue(EntryIRB.getInt8Ty()),
1980 Size, ArgAlign);
1981 } else {
1982 Value *Base = getShadowPtrForArgument(EntryIRB, ArgOffset);
1983 const Align CopyAlign = std::min(ArgAlign, kShadowTLSAlignment);
1984 Value *Cpy = EntryIRB.CreateMemCpy(CpShadowPtr, CopyAlign, Base,
1985 CopyAlign, Size);
1986 LLVM_DEBUG(dbgs() << " ByValCpy: " << *Cpy << "\n");
1987 (void)Cpy;
1988
1989 if (MS.TrackOrigins) {
1990 Value *OriginPtr =
1991 getOriginPtrForArgument(EntryIRB, ArgOffset);
1992 // FIXME: OriginSize should be:
1993 // alignTo(V % kMinOriginAlignment + Size, kMinOriginAlignment)
1994 unsigned OriginSize = alignTo(Size, kMinOriginAlignment);
1995 EntryIRB.CreateMemCpy(
1996 CpOriginPtr,
1997 /* by getShadowOriginPtr */ kMinOriginAlignment, OriginPtr,
1998 /* by origin_tls[ArgOffset] */ kMinOriginAlignment,
1999 OriginSize);
2000 }
2001 }
2002 }
2003
2004 if (!PropagateShadow || Overflow || FArg.hasByValAttr() ||
2005 (MS.EagerChecks && FArg.hasAttribute(Attribute::NoUndef))) {
2006 ShadowPtr = getCleanShadow(V);
2007 setOrigin(A, getCleanOrigin());
2008 } else {
2009 // Shadow over TLS
2010 Value *Base = getShadowPtrForArgument(EntryIRB, ArgOffset);
2011 ShadowPtr = EntryIRB.CreateAlignedLoad(getShadowTy(&FArg), Base,
2013 if (MS.TrackOrigins) {
2014 Value *OriginPtr =
2015 getOriginPtrForArgument(EntryIRB, ArgOffset);
2016 setOrigin(A, EntryIRB.CreateLoad(MS.OriginTy, OriginPtr));
2017 }
2018 }
2020 << " ARG: " << FArg << " ==> " << *ShadowPtr << "\n");
2021 break;
2022 }
2023
2024 ArgOffset += alignTo(Size, kShadowTLSAlignment);
2025 }
2026 assert(ShadowPtr && "Could not find shadow for an argument");
2027 return ShadowPtr;
2028 }
2029 // For everything else the shadow is zero.
2030 return getCleanShadow(V);
2031 }
2032
2033 /// Get the shadow for i-th argument of the instruction I.
2034 Value *getShadow(Instruction *I, int i) {
2035 return getShadow(I->getOperand(i));
2036 }
2037
2038 /// Get the origin for a value.
2039 Value *getOrigin(Value *V) {
2040 if (!MS.TrackOrigins)
2041 return nullptr;
2042 if (!PropagateShadow || isa<Constant>(V) || isa<InlineAsm>(V))
2043 return getCleanOrigin();
2044 assert((isa<Instruction>(V) || isa<Argument>(V)) &&
2045 "Unexpected value type in getOrigin()");
2046 if (Instruction *I = dyn_cast<Instruction>(V)) {
2047 if (I->getMetadata(LLVMContext::MD_nosanitize))
2048 return getCleanOrigin();
2049 }
2050 Value *Origin = OriginMap[V];
2051 assert(Origin && "Missing origin");
2052 return Origin;
2053 }
2054
2055 /// Get the origin for i-th argument of the instruction I.
2056 Value *getOrigin(Instruction *I, int i) {
2057 return getOrigin(I->getOperand(i));
2058 }
2059
2060 /// Remember the place where a shadow check should be inserted.
2061 ///
2062 /// This location will be later instrumented with a check that will print a
2063 /// UMR warning in runtime if the shadow value is not 0.
2064 void insertShadowCheck(Value *Shadow, Value *Origin, Instruction *OrigIns) {
2065 assert(Shadow);
2066 if (!InsertChecks)
2067 return;
2068
2069 if (!DebugCounter::shouldExecute(DebugInsertCheck)) {
2070 LLVM_DEBUG(dbgs() << "Skipping check of " << *Shadow << " before "
2071 << *OrigIns << "\n");
2072 return;
2073 }
2074#ifndef NDEBUG
2075 Type *ShadowTy = Shadow->getType();
2076 assert((isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy) ||
2077 isa<StructType>(ShadowTy) || isa<ArrayType>(ShadowTy)) &&
2078 "Can only insert checks for integer, vector, and aggregate shadow "
2079 "types");
2080#endif
2081 InstrumentationList.push_back(
2082 ShadowOriginAndInsertPoint(Shadow, Origin, OrigIns));
2083 }
2084
2085 /// Remember the place where a shadow check should be inserted.
2086 ///
2087 /// This location will be later instrumented with a check that will print a
2088 /// UMR warning in runtime if the value is not fully defined.
2089 void insertShadowCheck(Value *Val, Instruction *OrigIns) {
2090 assert(Val);
2091 Value *Shadow, *Origin;
2093 Shadow = getShadow(Val);
2094 if (!Shadow)
2095 return;
2096 Origin = getOrigin(Val);
2097 } else {
2098 Shadow = dyn_cast_or_null<Instruction>(getShadow(Val));
2099 if (!Shadow)
2100 return;
2101 Origin = dyn_cast_or_null<Instruction>(getOrigin(Val));
2102 }
2103 insertShadowCheck(Shadow, Origin, OrigIns);
2104 }
2105
2107 switch (a) {
2108 case AtomicOrdering::NotAtomic:
2109 return AtomicOrdering::NotAtomic;
2110 case AtomicOrdering::Unordered:
2111 case AtomicOrdering::Monotonic:
2112 case AtomicOrdering::Release:
2113 return AtomicOrdering::Release;
2114 case AtomicOrdering::Acquire:
2115 case AtomicOrdering::AcquireRelease:
2116 return AtomicOrdering::AcquireRelease;
2117 case AtomicOrdering::SequentiallyConsistent:
2118 return AtomicOrdering::SequentiallyConsistent;
2119 }
2120 llvm_unreachable("Unknown ordering");
2121 }
2122
2123 Value *makeAddReleaseOrderingTable(IRBuilder<> &IRB) {
2124 constexpr int NumOrderings = (int)AtomicOrderingCABI::seq_cst + 1;
2125 uint32_t OrderingTable[NumOrderings] = {};
2126
2127 OrderingTable[(int)AtomicOrderingCABI::relaxed] =
2128 OrderingTable[(int)AtomicOrderingCABI::release] =
2129 (int)AtomicOrderingCABI::release;
2130 OrderingTable[(int)AtomicOrderingCABI::consume] =
2131 OrderingTable[(int)AtomicOrderingCABI::acquire] =
2132 OrderingTable[(int)AtomicOrderingCABI::acq_rel] =
2133 (int)AtomicOrderingCABI::acq_rel;
2134 OrderingTable[(int)AtomicOrderingCABI::seq_cst] =
2135 (int)AtomicOrderingCABI::seq_cst;
2136
2138 ArrayRef(OrderingTable, NumOrderings));
2139 }
2140
2142 switch (a) {
2143 case AtomicOrdering::NotAtomic:
2144 return AtomicOrdering::NotAtomic;
2145 case AtomicOrdering::Unordered:
2146 case AtomicOrdering::Monotonic:
2147 case AtomicOrdering::Acquire:
2148 return AtomicOrdering::Acquire;
2149 case AtomicOrdering::Release:
2150 case AtomicOrdering::AcquireRelease:
2151 return AtomicOrdering::AcquireRelease;
2152 case AtomicOrdering::SequentiallyConsistent:
2153 return AtomicOrdering::SequentiallyConsistent;
2154 }
2155 llvm_unreachable("Unknown ordering");
2156 }
2157
2158 Value *makeAddAcquireOrderingTable(IRBuilder<> &IRB) {
2159 constexpr int NumOrderings = (int)AtomicOrderingCABI::seq_cst + 1;
2160 uint32_t OrderingTable[NumOrderings] = {};
2161
2162 OrderingTable[(int)AtomicOrderingCABI::relaxed] =
2163 OrderingTable[(int)AtomicOrderingCABI::acquire] =
2164 OrderingTable[(int)AtomicOrderingCABI::consume] =
2165 (int)AtomicOrderingCABI::acquire;
2166 OrderingTable[(int)AtomicOrderingCABI::release] =
2167 OrderingTable[(int)AtomicOrderingCABI::acq_rel] =
2168 (int)AtomicOrderingCABI::acq_rel;
2169 OrderingTable[(int)AtomicOrderingCABI::seq_cst] =
2170 (int)AtomicOrderingCABI::seq_cst;
2171
2173 ArrayRef(OrderingTable, NumOrderings));
2174 }
2175
2176 // ------------------- Visitors.
2177 using InstVisitor<MemorySanitizerVisitor>::visit;
2178 void visit(Instruction &I) {
2179 if (I.getMetadata(LLVMContext::MD_nosanitize))
2180 return;
2181 // Don't want to visit if we're in the prologue
2182 if (isInPrologue(I))
2183 return;
2185 }
2186
2187 /// Instrument LoadInst
2188 ///
2189 /// Loads the corresponding shadow and (optionally) origin.
2190 /// Optionally, checks that the load address is fully defined.
2191 void visitLoadInst(LoadInst &I) {
2192 assert(I.getType()->isSized() && "Load type must have size");
2193 assert(!I.getMetadata(LLVMContext::MD_nosanitize));
2194 NextNodeIRBuilder IRB(&I);
2195 Type *ShadowTy = getShadowTy(&I);
2196 Value *Addr = I.getPointerOperand();
2197 Value *ShadowPtr = nullptr, *OriginPtr = nullptr;
2198 const Align Alignment = I.getAlign();
2199 if (PropagateShadow) {
2200 std::tie(ShadowPtr, OriginPtr) =
2201 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
2202 setShadow(&I,
2203 IRB.CreateAlignedLoad(ShadowTy, ShadowPtr, Alignment, "_msld"));
2204 } else {
2205 setShadow(&I, getCleanShadow(&I));
2206 }
2207
2209 insertShadowCheck(I.getPointerOperand(), &I);
2210
2211 if (I.isAtomic())
2212 I.setOrdering(addAcquireOrdering(I.getOrdering()));
2213
2214 if (MS.TrackOrigins) {
2215 if (PropagateShadow) {
2216 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
2217 setOrigin(
2218 &I, IRB.CreateAlignedLoad(MS.OriginTy, OriginPtr, OriginAlignment));
2219 } else {
2220 setOrigin(&I, getCleanOrigin());
2221 }
2222 }
2223 }
2224
2225 /// Instrument StoreInst
2226 ///
2227 /// Stores the corresponding shadow and (optionally) origin.
2228 /// Optionally, checks that the store address is fully defined.
2229 void visitStoreInst(StoreInst &I) {
2230 StoreList.push_back(&I);
2232 insertShadowCheck(I.getPointerOperand(), &I);
2233 }
2234
2235 void handleCASOrRMW(Instruction &I) {
2236 assert(isa<AtomicRMWInst>(I) || isa<AtomicCmpXchgInst>(I));
2237
2238 IRBuilder<> IRB(&I);
2239 Value *Addr = I.getOperand(0);
2240 Value *Val = I.getOperand(1);
2241 Value *ShadowPtr = getShadowOriginPtr(Addr, IRB, getShadowTy(Val), Align(1),
2242 /*isStore*/ true)
2243 .first;
2244
2246 insertShadowCheck(Addr, &I);
2247
2248 // Only test the conditional argument of cmpxchg instruction.
2249 // The other argument can potentially be uninitialized, but we can not
2250 // detect this situation reliably without possible false positives.
2251 if (isa<AtomicCmpXchgInst>(I))
2252 insertShadowCheck(Val, &I);
2253
2254 IRB.CreateStore(getCleanShadow(Val), ShadowPtr);
2255
2256 setShadow(&I, getCleanShadow(&I));
2257 setOrigin(&I, getCleanOrigin());
2258 }
2259
2260 void visitAtomicRMWInst(AtomicRMWInst &I) {
2261 handleCASOrRMW(I);
2262 I.setOrdering(addReleaseOrdering(I.getOrdering()));
2263 }
2264
2265 void visitAtomicCmpXchgInst(AtomicCmpXchgInst &I) {
2266 handleCASOrRMW(I);
2267 I.setSuccessOrdering(addReleaseOrdering(I.getSuccessOrdering()));
2268 }
2269
2270 // Vector manipulation.
2271 void visitExtractElementInst(ExtractElementInst &I) {
2272 insertShadowCheck(I.getOperand(1), &I);
2273 IRBuilder<> IRB(&I);
2274 setShadow(&I, IRB.CreateExtractElement(getShadow(&I, 0), I.getOperand(1),
2275 "_msprop"));
2276 setOrigin(&I, getOrigin(&I, 0));
2277 }
2278
2279 void visitInsertElementInst(InsertElementInst &I) {
2280 insertShadowCheck(I.getOperand(2), &I);
2281 IRBuilder<> IRB(&I);
2282 auto *Shadow0 = getShadow(&I, 0);
2283 auto *Shadow1 = getShadow(&I, 1);
2284 setShadow(&I, IRB.CreateInsertElement(Shadow0, Shadow1, I.getOperand(2),
2285 "_msprop"));
2286 setOriginForNaryOp(I);
2287 }
2288
2289 void visitShuffleVectorInst(ShuffleVectorInst &I) {
2290 IRBuilder<> IRB(&I);
2291 auto *Shadow0 = getShadow(&I, 0);
2292 auto *Shadow1 = getShadow(&I, 1);
2293 setShadow(&I, IRB.CreateShuffleVector(Shadow0, Shadow1, I.getShuffleMask(),
2294 "_msprop"));
2295 setOriginForNaryOp(I);
2296 }
2297
2298 // Casts.
2299 void visitSExtInst(SExtInst &I) {
2300 IRBuilder<> IRB(&I);
2301 setShadow(&I, IRB.CreateSExt(getShadow(&I, 0), I.getType(), "_msprop"));
2302 setOrigin(&I, getOrigin(&I, 0));
2303 }
2304
2305 void visitZExtInst(ZExtInst &I) {
2306 IRBuilder<> IRB(&I);
2307 setShadow(&I, IRB.CreateZExt(getShadow(&I, 0), I.getType(), "_msprop"));
2308 setOrigin(&I, getOrigin(&I, 0));
2309 }
2310
2311 void visitTruncInst(TruncInst &I) {
2312 IRBuilder<> IRB(&I);
2313 setShadow(&I, IRB.CreateTrunc(getShadow(&I, 0), I.getType(), "_msprop"));
2314 setOrigin(&I, getOrigin(&I, 0));
2315 }
2316
2317 void visitBitCastInst(BitCastInst &I) {
2318 // Special case: if this is the bitcast (there is exactly 1 allowed) between
2319 // a musttail call and a ret, don't instrument. New instructions are not
2320 // allowed after a musttail call.
2321 if (auto *CI = dyn_cast<CallInst>(I.getOperand(0)))
2322 if (CI->isMustTailCall())
2323 return;
2324 IRBuilder<> IRB(&I);
2325 setShadow(&I, IRB.CreateBitCast(getShadow(&I, 0), getShadowTy(&I)));
2326 setOrigin(&I, getOrigin(&I, 0));
2327 }
2328
2329 void visitPtrToIntInst(PtrToIntInst &I) {
2330 IRBuilder<> IRB(&I);
2331 setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
2332 "_msprop_ptrtoint"));
2333 setOrigin(&I, getOrigin(&I, 0));
2334 }
2335
2336 void visitIntToPtrInst(IntToPtrInst &I) {
2337 IRBuilder<> IRB(&I);
2338 setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
2339 "_msprop_inttoptr"));
2340 setOrigin(&I, getOrigin(&I, 0));
2341 }
2342
2343 void visitFPToSIInst(CastInst &I) { handleShadowOr(I); }
2344 void visitFPToUIInst(CastInst &I) { handleShadowOr(I); }
2345 void visitSIToFPInst(CastInst &I) { handleShadowOr(I); }
2346 void visitUIToFPInst(CastInst &I) { handleShadowOr(I); }
2347 void visitFPExtInst(CastInst &I) { handleShadowOr(I); }
2348 void visitFPTruncInst(CastInst &I) { handleShadowOr(I); }
2349
2350 /// Propagate shadow for bitwise AND.
2351 ///
2352 /// This code is exact, i.e. if, for example, a bit in the left argument
2353 /// is defined and 0, then neither the value not definedness of the
2354 /// corresponding bit in B don't affect the resulting shadow.
2355 void visitAnd(BinaryOperator &I) {
2356 IRBuilder<> IRB(&I);
2357 // "And" of 0 and a poisoned value results in unpoisoned value.
2358 // 1&1 => 1; 0&1 => 0; p&1 => p;
2359 // 1&0 => 0; 0&0 => 0; p&0 => 0;
2360 // 1&p => p; 0&p => 0; p&p => p;
2361 // S = (S1 & S2) | (V1 & S2) | (S1 & V2)
2362 Value *S1 = getShadow(&I, 0);
2363 Value *S2 = getShadow(&I, 1);
2364 Value *V1 = I.getOperand(0);
2365 Value *V2 = I.getOperand(1);
2366 if (V1->getType() != S1->getType()) {
2367 V1 = IRB.CreateIntCast(V1, S1->getType(), false);
2368 V2 = IRB.CreateIntCast(V2, S2->getType(), false);
2369 }
2370 Value *S1S2 = IRB.CreateAnd(S1, S2);
2371 Value *V1S2 = IRB.CreateAnd(V1, S2);
2372 Value *S1V2 = IRB.CreateAnd(S1, V2);
2373 setShadow(&I, IRB.CreateOr({S1S2, V1S2, S1V2}));
2374 setOriginForNaryOp(I);
2375 }
2376
2377 void visitOr(BinaryOperator &I) {
2378 IRBuilder<> IRB(&I);
2379 // "Or" of 1 and a poisoned value results in unpoisoned value.
2380 // 1|1 => 1; 0|1 => 1; p|1 => 1;
2381 // 1|0 => 1; 0|0 => 0; p|0 => p;
2382 // 1|p => 1; 0|p => p; p|p => p;
2383 // S = (S1 & S2) | (~V1 & S2) | (S1 & ~V2)
2384 Value *S1 = getShadow(&I, 0);
2385 Value *S2 = getShadow(&I, 1);
2386 Value *V1 = IRB.CreateNot(I.getOperand(0));
2387 Value *V2 = IRB.CreateNot(I.getOperand(1));
2388 if (V1->getType() != S1->getType()) {
2389 V1 = IRB.CreateIntCast(V1, S1->getType(), false);
2390 V2 = IRB.CreateIntCast(V2, S2->getType(), false);
2391 }
2392 Value *S1S2 = IRB.CreateAnd(S1, S2);
2393 Value *V1S2 = IRB.CreateAnd(V1, S2);
2394 Value *S1V2 = IRB.CreateAnd(S1, V2);
2395 setShadow(&I, IRB.CreateOr({S1S2, V1S2, S1V2}));
2396 setOriginForNaryOp(I);
2397 }
2398
2399 /// Default propagation of shadow and/or origin.
2400 ///
2401 /// This class implements the general case of shadow propagation, used in all
2402 /// cases where we don't know and/or don't care about what the operation
2403 /// actually does. It converts all input shadow values to a common type
2404 /// (extending or truncating as necessary), and bitwise OR's them.
2405 ///
2406 /// This is much cheaper than inserting checks (i.e. requiring inputs to be
2407 /// fully initialized), and less prone to false positives.
2408 ///
2409 /// This class also implements the general case of origin propagation. For a
2410 /// Nary operation, result origin is set to the origin of an argument that is
2411 /// not entirely initialized. If there is more than one such arguments, the
2412 /// rightmost of them is picked. It does not matter which one is picked if all
2413 /// arguments are initialized.
2414 template <bool CombineShadow> class Combiner {
2415 Value *Shadow = nullptr;
2416 Value *Origin = nullptr;
2417 IRBuilder<> &IRB;
2418 MemorySanitizerVisitor *MSV;
2419
2420 public:
2421 Combiner(MemorySanitizerVisitor *MSV, IRBuilder<> &IRB)
2422 : IRB(IRB), MSV(MSV) {}
2423
2424 /// Add a pair of shadow and origin values to the mix.
2425 Combiner &Add(Value *OpShadow, Value *OpOrigin) {
2426 if (CombineShadow) {
2427 assert(OpShadow);
2428 if (!Shadow)
2429 Shadow = OpShadow;
2430 else {
2431 OpShadow = MSV->CreateShadowCast(IRB, OpShadow, Shadow->getType());
2432 Shadow = IRB.CreateOr(Shadow, OpShadow, "_msprop");
2433 }
2434 }
2435
2436 if (MSV->MS.TrackOrigins) {
2437 assert(OpOrigin);
2438 if (!Origin) {
2439 Origin = OpOrigin;
2440 } else {
2441 Constant *ConstOrigin = dyn_cast<Constant>(OpOrigin);
2442 // No point in adding something that might result in 0 origin value.
2443 if (!ConstOrigin || !ConstOrigin->isNullValue()) {
2444 Value *Cond = MSV->convertToBool(OpShadow, IRB);
2445 Origin = IRB.CreateSelect(Cond, OpOrigin, Origin);
2446 }
2447 }
2448 }
2449 return *this;
2450 }
2451
2452 /// Add an application value to the mix.
2453 Combiner &Add(Value *V) {
2454 Value *OpShadow = MSV->getShadow(V);
2455 Value *OpOrigin = MSV->MS.TrackOrigins ? MSV->getOrigin(V) : nullptr;
2456 return Add(OpShadow, OpOrigin);
2457 }
2458
2459 /// Set the current combined values as the given instruction's shadow
2460 /// and origin.
2461 void Done(Instruction *I) {
2462 if (CombineShadow) {
2463 assert(Shadow);
2464 Shadow = MSV->CreateShadowCast(IRB, Shadow, MSV->getShadowTy(I));
2465 MSV->setShadow(I, Shadow);
2466 }
2467 if (MSV->MS.TrackOrigins) {
2468 assert(Origin);
2469 MSV->setOrigin(I, Origin);
2470 }
2471 }
2472 };
2473
2474 using ShadowAndOriginCombiner = Combiner<true>;
2475 using OriginCombiner = Combiner<false>;
2476
2477 /// Propagate origin for arbitrary operation.
2478 void setOriginForNaryOp(Instruction &I) {
2479 if (!MS.TrackOrigins)
2480 return;
2481 IRBuilder<> IRB(&I);
2482 OriginCombiner OC(this, IRB);
2483 for (Use &Op : I.operands())
2484 OC.Add(Op.get());
2485 OC.Done(&I);
2486 }
2487
2488 size_t VectorOrPrimitiveTypeSizeInBits(Type *Ty) {
2489 assert(!(Ty->isVectorTy() && Ty->getScalarType()->isPointerTy()) &&
2490 "Vector of pointers is not a valid shadow type");
2491 return Ty->isVectorTy() ? cast<FixedVectorType>(Ty)->getNumElements() *
2493 : Ty->getPrimitiveSizeInBits();
2494 }
2495
2496 /// Cast between two shadow types, extending or truncating as
2497 /// necessary.
2498 Value *CreateShadowCast(IRBuilder<> &IRB, Value *V, Type *dstTy,
2499 bool Signed = false) {
2500 Type *srcTy = V->getType();
2501 size_t srcSizeInBits = VectorOrPrimitiveTypeSizeInBits(srcTy);
2502 size_t dstSizeInBits = VectorOrPrimitiveTypeSizeInBits(dstTy);
2503 if (srcSizeInBits > 1 && dstSizeInBits == 1)
2504 return IRB.CreateICmpNE(V, getCleanShadow(V));
2505
2506 if (dstTy->isIntegerTy() && srcTy->isIntegerTy())
2507 return IRB.CreateIntCast(V, dstTy, Signed);
2508 if (dstTy->isVectorTy() && srcTy->isVectorTy() &&
2509 cast<VectorType>(dstTy)->getElementCount() ==
2510 cast<VectorType>(srcTy)->getElementCount())
2511 return IRB.CreateIntCast(V, dstTy, Signed);
2512 Value *V1 = IRB.CreateBitCast(V, Type::getIntNTy(*MS.C, srcSizeInBits));
2513 Value *V2 =
2514 IRB.CreateIntCast(V1, Type::getIntNTy(*MS.C, dstSizeInBits), Signed);
2515 return IRB.CreateBitCast(V2, dstTy);
2516 // TODO: handle struct types.
2517 }
2518
2519 /// Cast an application value to the type of its own shadow.
2520 Value *CreateAppToShadowCast(IRBuilder<> &IRB, Value *V) {
2521 Type *ShadowTy = getShadowTy(V);
2522 if (V->getType() == ShadowTy)
2523 return V;
2524 if (V->getType()->isPtrOrPtrVectorTy())
2525 return IRB.CreatePtrToInt(V, ShadowTy);
2526 else
2527 return IRB.CreateBitCast(V, ShadowTy);
2528 }
2529
2530 /// Propagate shadow for arbitrary operation.
2531 void handleShadowOr(Instruction &I) {
2532 IRBuilder<> IRB(&I);
2533 ShadowAndOriginCombiner SC(this, IRB);
2534 for (Use &Op : I.operands())
2535 SC.Add(Op.get());
2536 SC.Done(&I);
2537 }
2538
2539 void visitFNeg(UnaryOperator &I) { handleShadowOr(I); }
2540
2541 // Handle multiplication by constant.
2542 //
2543 // Handle a special case of multiplication by constant that may have one or
2544 // more zeros in the lower bits. This makes corresponding number of lower bits
2545 // of the result zero as well. We model it by shifting the other operand
2546 // shadow left by the required number of bits. Effectively, we transform
2547 // (X * (A * 2**B)) to ((X << B) * A) and instrument (X << B) as (Sx << B).
2548 // We use multiplication by 2**N instead of shift to cover the case of
2549 // multiplication by 0, which may occur in some elements of a vector operand.
2550 void handleMulByConstant(BinaryOperator &I, Constant *ConstArg,
2551 Value *OtherArg) {
2552 Constant *ShadowMul;
2553 Type *Ty = ConstArg->getType();
2554 if (auto *VTy = dyn_cast<VectorType>(Ty)) {
2555 unsigned NumElements = cast<FixedVectorType>(VTy)->getNumElements();
2556 Type *EltTy = VTy->getElementType();
2558 for (unsigned Idx = 0; Idx < NumElements; ++Idx) {
2559 if (ConstantInt *Elt =
2560 dyn_cast<ConstantInt>(ConstArg->getAggregateElement(Idx))) {
2561 const APInt &V = Elt->getValue();
2562 APInt V2 = APInt(V.getBitWidth(), 1) << V.countr_zero();
2563 Elements.push_back(ConstantInt::get(EltTy, V2));
2564 } else {
2565 Elements.push_back(ConstantInt::get(EltTy, 1));
2566 }
2567 }
2568 ShadowMul = ConstantVector::get(Elements);
2569 } else {
2570 if (ConstantInt *Elt = dyn_cast<ConstantInt>(ConstArg)) {
2571 const APInt &V = Elt->getValue();
2572 APInt V2 = APInt(V.getBitWidth(), 1) << V.countr_zero();
2573 ShadowMul = ConstantInt::get(Ty, V2);
2574 } else {
2575 ShadowMul = ConstantInt::get(Ty, 1);
2576 }
2577 }
2578
2579 IRBuilder<> IRB(&I);
2580 setShadow(&I,
2581 IRB.CreateMul(getShadow(OtherArg), ShadowMul, "msprop_mul_cst"));
2582 setOrigin(&I, getOrigin(OtherArg));
2583 }
2584
2585 void visitMul(BinaryOperator &I) {
2586 Constant *constOp0 = dyn_cast<Constant>(I.getOperand(0));
2587 Constant *constOp1 = dyn_cast<Constant>(I.getOperand(1));
2588 if (constOp0 && !constOp1)
2589 handleMulByConstant(I, constOp0, I.getOperand(1));
2590 else if (constOp1 && !constOp0)
2591 handleMulByConstant(I, constOp1, I.getOperand(0));
2592 else
2593 handleShadowOr(I);
2594 }
2595
2596 void visitFAdd(BinaryOperator &I) { handleShadowOr(I); }
2597 void visitFSub(BinaryOperator &I) { handleShadowOr(I); }
2598 void visitFMul(BinaryOperator &I) { handleShadowOr(I); }
2599 void visitAdd(BinaryOperator &I) { handleShadowOr(I); }
2600 void visitSub(BinaryOperator &I) { handleShadowOr(I); }
2601 void visitXor(BinaryOperator &I) { handleShadowOr(I); }
2602
2603 void handleIntegerDiv(Instruction &I) {
2604 IRBuilder<> IRB(&I);
2605 // Strict on the second argument.
2606 insertShadowCheck(I.getOperand(1), &I);
2607 setShadow(&I, getShadow(&I, 0));
2608 setOrigin(&I, getOrigin(&I, 0));
2609 }
2610
2611 void visitUDiv(BinaryOperator &I) { handleIntegerDiv(I); }
2612 void visitSDiv(BinaryOperator &I) { handleIntegerDiv(I); }
2613 void visitURem(BinaryOperator &I) { handleIntegerDiv(I); }
2614 void visitSRem(BinaryOperator &I) { handleIntegerDiv(I); }
2615
2616 // Floating point division is side-effect free. We can not require that the
2617 // divisor is fully initialized and must propagate shadow. See PR37523.
2618 void visitFDiv(BinaryOperator &I) { handleShadowOr(I); }
2619 void visitFRem(BinaryOperator &I) { handleShadowOr(I); }
2620
2621 /// Instrument == and != comparisons.
2622 ///
2623 /// Sometimes the comparison result is known even if some of the bits of the
2624 /// arguments are not.
2625 void handleEqualityComparison(ICmpInst &I) {
2626 IRBuilder<> IRB(&I);
2627 Value *A = I.getOperand(0);
2628 Value *B = I.getOperand(1);
2629 Value *Sa = getShadow(A);
2630 Value *Sb = getShadow(B);
2631
2632 // Get rid of pointers and vectors of pointers.
2633 // For ints (and vectors of ints), types of A and Sa match,
2634 // and this is a no-op.
2635 A = IRB.CreatePointerCast(A, Sa->getType());
2636 B = IRB.CreatePointerCast(B, Sb->getType());
2637
2638 // A == B <==> (C = A^B) == 0
2639 // A != B <==> (C = A^B) != 0
2640 // Sc = Sa | Sb
2641 Value *C = IRB.CreateXor(A, B);
2642 Value *Sc = IRB.CreateOr(Sa, Sb);
2643 // Now dealing with i = (C == 0) comparison (or C != 0, does not matter now)
2644 // Result is defined if one of the following is true
2645 // * there is a defined 1 bit in C
2646 // * C is fully defined
2647 // Si = !(C & ~Sc) && Sc
2649 Value *MinusOne = Constant::getAllOnesValue(Sc->getType());
2650 Value *LHS = IRB.CreateICmpNE(Sc, Zero);
2651 Value *RHS =
2652 IRB.CreateICmpEQ(IRB.CreateAnd(IRB.CreateXor(Sc, MinusOne), C), Zero);
2653 Value *Si = IRB.CreateAnd(LHS, RHS);
2654 Si->setName("_msprop_icmp");
2655 setShadow(&I, Si);
2656 setOriginForNaryOp(I);
2657 }
2658
2659 /// Build the lowest possible value of V, taking into account V's
2660 /// uninitialized bits.
2661 Value *getLowestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2662 bool isSigned) {
2663 if (isSigned) {
2664 // Split shadow into sign bit and other bits.
2665 Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2666 Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2667 // Maximise the undefined shadow bit, minimize other undefined bits.
2668 return IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaOtherBits)),
2669 SaSignBit);
2670 } else {
2671 // Minimize undefined bits.
2672 return IRB.CreateAnd(A, IRB.CreateNot(Sa));
2673 }
2674 }
2675
2676 /// Build the highest possible value of V, taking into account V's
2677 /// uninitialized bits.
2678 Value *getHighestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2679 bool isSigned) {
2680 if (isSigned) {
2681 // Split shadow into sign bit and other bits.
2682 Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2683 Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2684 // Minimise the undefined shadow bit, maximise other undefined bits.
2685 return IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaSignBit)),
2686 SaOtherBits);
2687 } else {
2688 // Maximize undefined bits.
2689 return IRB.CreateOr(A, Sa);
2690 }
2691 }
2692
2693 /// Instrument relational comparisons.
2694 ///
2695 /// This function does exact shadow propagation for all relational
2696 /// comparisons of integers, pointers and vectors of those.
2697 /// FIXME: output seems suboptimal when one of the operands is a constant
2698 void handleRelationalComparisonExact(ICmpInst &I) {
2699 IRBuilder<> IRB(&I);
2700 Value *A = I.getOperand(0);
2701 Value *B = I.getOperand(1);
2702 Value *Sa = getShadow(A);
2703 Value *Sb = getShadow(B);
2704
2705 // Get rid of pointers and vectors of pointers.
2706 // For ints (and vectors of ints), types of A and Sa match,
2707 // and this is a no-op.
2708 A = IRB.CreatePointerCast(A, Sa->getType());
2709 B = IRB.CreatePointerCast(B, Sb->getType());
2710
2711 // Let [a0, a1] be the interval of possible values of A, taking into account
2712 // its undefined bits. Let [b0, b1] be the interval of possible values of B.
2713 // Then (A cmp B) is defined iff (a0 cmp b1) == (a1 cmp b0).
2714 bool IsSigned = I.isSigned();
2715 Value *S1 = IRB.CreateICmp(I.getPredicate(),
2716 getLowestPossibleValue(IRB, A, Sa, IsSigned),
2717 getHighestPossibleValue(IRB, B, Sb, IsSigned));
2718 Value *S2 = IRB.CreateICmp(I.getPredicate(),
2719 getHighestPossibleValue(IRB, A, Sa, IsSigned),
2720 getLowestPossibleValue(IRB, B, Sb, IsSigned));
2721 Value *Si = IRB.CreateXor(S1, S2);
2722 setShadow(&I, Si);
2723 setOriginForNaryOp(I);
2724 }
2725
2726 /// Instrument signed relational comparisons.
2727 ///
2728 /// Handle sign bit tests: x<0, x>=0, x<=-1, x>-1 by propagating the highest
2729 /// bit of the shadow. Everything else is delegated to handleShadowOr().
2730 void handleSignedRelationalComparison(ICmpInst &I) {
2731 Constant *constOp;
2732 Value *op = nullptr;
2734 if ((constOp = dyn_cast<Constant>(I.getOperand(1)))) {
2735 op = I.getOperand(0);
2736 pre = I.getPredicate();
2737 } else if ((constOp = dyn_cast<Constant>(I.getOperand(0)))) {
2738 op = I.getOperand(1);
2739 pre = I.getSwappedPredicate();
2740 } else {
2741 handleShadowOr(I);
2742 return;
2743 }
2744
2745 if ((constOp->isNullValue() &&
2746 (pre == CmpInst::ICMP_SLT || pre == CmpInst::ICMP_SGE)) ||
2747 (constOp->isAllOnesValue() &&
2748 (pre == CmpInst::ICMP_SGT || pre == CmpInst::ICMP_SLE))) {
2749 IRBuilder<> IRB(&I);
2750 Value *Shadow = IRB.CreateICmpSLT(getShadow(op), getCleanShadow(op),
2751 "_msprop_icmp_s");
2752 setShadow(&I, Shadow);
2753 setOrigin(&I, getOrigin(op));
2754 } else {
2755 handleShadowOr(I);
2756 }
2757 }
2758
2759 void visitICmpInst(ICmpInst &I) {
2760 if (!ClHandleICmp) {
2761 handleShadowOr(I);
2762 return;
2763 }
2764 if (I.isEquality()) {
2765 handleEqualityComparison(I);
2766 return;
2767 }
2768
2769 assert(I.isRelational());
2770 if (ClHandleICmpExact) {
2771 handleRelationalComparisonExact(I);
2772 return;
2773 }
2774 if (I.isSigned()) {
2775 handleSignedRelationalComparison(I);
2776 return;
2777 }
2778
2779 assert(I.isUnsigned());
2780 if ((isa<Constant>(I.getOperand(0)) || isa<Constant>(I.getOperand(1)))) {
2781 handleRelationalComparisonExact(I);
2782 return;
2783 }
2784
2785 handleShadowOr(I);
2786 }
2787
2788 void visitFCmpInst(FCmpInst &I) { handleShadowOr(I); }
2789
2790 void handleShift(BinaryOperator &I) {
2791 IRBuilder<> IRB(&I);
2792 // If any of the S2 bits are poisoned, the whole thing is poisoned.
2793 // Otherwise perform the same shift on S1.
2794 Value *S1 = getShadow(&I, 0);
2795 Value *S2 = getShadow(&I, 1);
2796 Value *S2Conv =
2797 IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)), S2->getType());
2798 Value *V2 = I.getOperand(1);
2799 Value *Shift = IRB.CreateBinOp(I.getOpcode(), S1, V2);
2800 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2801 setOriginForNaryOp(I);
2802 }
2803
2804 void visitShl(BinaryOperator &I) { handleShift(I); }
2805 void visitAShr(BinaryOperator &I) { handleShift(I); }
2806 void visitLShr(BinaryOperator &I) { handleShift(I); }
2807
2808 void handleFunnelShift(IntrinsicInst &I) {
2809 IRBuilder<> IRB(&I);
2810 // If any of the S2 bits are poisoned, the whole thing is poisoned.
2811 // Otherwise perform the same shift on S0 and S1.
2812 Value *S0 = getShadow(&I, 0);
2813 Value *S1 = getShadow(&I, 1);
2814 Value *S2 = getShadow(&I, 2);
2815 Value *S2Conv =
2816 IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)), S2->getType());
2817 Value *V2 = I.getOperand(2);
2819 I.getModule(), I.getIntrinsicID(), S2Conv->getType());
2820 Value *Shift = IRB.CreateCall(Intrin, {S0, S1, V2});
2821 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2822 setOriginForNaryOp(I);
2823 }
2824
2825 /// Instrument llvm.memmove
2826 ///
2827 /// At this point we don't know if llvm.memmove will be inlined or not.
2828 /// If we don't instrument it and it gets inlined,
2829 /// our interceptor will not kick in and we will lose the memmove.
2830 /// If we instrument the call here, but it does not get inlined,
2831 /// we will memove the shadow twice: which is bad in case
2832 /// of overlapping regions. So, we simply lower the intrinsic to a call.
2833 ///
2834 /// Similar situation exists for memcpy and memset.
2835 void visitMemMoveInst(MemMoveInst &I) {
2836 getShadow(I.getArgOperand(1)); // Ensure shadow initialized
2837 IRBuilder<> IRB(&I);
2838 IRB.CreateCall(
2839 MS.MemmoveFn,
2840 {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2841 IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()),
2842 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2843 I.eraseFromParent();
2844 }
2845
2846 /// Instrument memcpy
2847 ///
2848 /// Similar to memmove: avoid copying shadow twice. This is somewhat
2849 /// unfortunate as it may slowdown small constant memcpys.
2850 /// FIXME: consider doing manual inline for small constant sizes and proper
2851 /// alignment.
2852 ///
2853 /// Note: This also handles memcpy.inline, which promises no calls to external
2854 /// functions as an optimization. However, with instrumentation enabled this
2855 /// is difficult to promise; additionally, we know that the MSan runtime
2856 /// exists and provides __msan_memcpy(). Therefore, we assume that with
2857 /// instrumentation it's safe to turn memcpy.inline into a call to
2858 /// __msan_memcpy(). Should this be wrong, such as when implementing memcpy()
2859 /// itself, instrumentation should be disabled with the no_sanitize attribute.
2860 void visitMemCpyInst(MemCpyInst &I) {
2861 getShadow(I.getArgOperand(1)); // Ensure shadow initialized
2862 IRBuilder<> IRB(&I);
2863 IRB.CreateCall(
2864 MS.MemcpyFn,
2865 {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2866 IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()),
2867 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2868 I.eraseFromParent();
2869 }
2870
2871 // Same as memcpy.
2872 void visitMemSetInst(MemSetInst &I) {
2873 IRBuilder<> IRB(&I);
2874 IRB.CreateCall(
2875 MS.MemsetFn,
2876 {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2877 IRB.CreateIntCast(I.getArgOperand(1), IRB.getInt32Ty(), false),
2878 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2879 I.eraseFromParent();
2880 }
2881
2882 void visitVAStartInst(VAStartInst &I) { VAHelper->visitVAStartInst(I); }
2883
2884 void visitVACopyInst(VACopyInst &I) { VAHelper->visitVACopyInst(I); }
2885
2886 /// Handle vector store-like intrinsics.
2887 ///
2888 /// Instrument intrinsics that look like a simple SIMD store: writes memory,
2889 /// has 1 pointer argument and 1 vector argument, returns void.
2890 bool handleVectorStoreIntrinsic(IntrinsicInst &I) {
2891 IRBuilder<> IRB(&I);
2892 Value *Addr = I.getArgOperand(0);
2893 Value *Shadow = getShadow(&I, 1);
2894 Value *ShadowPtr, *OriginPtr;
2895
2896 // We don't know the pointer alignment (could be unaligned SSE store!).
2897 // Have to assume to worst case.
2898 std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
2899 Addr, IRB, Shadow->getType(), Align(1), /*isStore*/ true);
2900 IRB.CreateAlignedStore(Shadow, ShadowPtr, Align(1));
2901
2903 insertShadowCheck(Addr, &I);
2904
2905 // FIXME: factor out common code from materializeStores
2906 if (MS.TrackOrigins)
2907 IRB.CreateStore(getOrigin(&I, 1), OriginPtr);
2908 return true;
2909 }
2910
2911 /// Handle vector load-like intrinsics.
2912 ///
2913 /// Instrument intrinsics that look like a simple SIMD load: reads memory,
2914 /// has 1 pointer argument, returns a vector.
2915 bool handleVectorLoadIntrinsic(IntrinsicInst &I) {
2916 IRBuilder<> IRB(&I);
2917 Value *Addr = I.getArgOperand(0);
2918
2919 Type *ShadowTy = getShadowTy(&I);
2920 Value *ShadowPtr = nullptr, *OriginPtr = nullptr;
2921 if (PropagateShadow) {
2922 // We don't know the pointer alignment (could be unaligned SSE load!).
2923 // Have to assume to worst case.
2924 const Align Alignment = Align(1);
2925 std::tie(ShadowPtr, OriginPtr) =
2926 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
2927 setShadow(&I,
2928 IRB.CreateAlignedLoad(ShadowTy, ShadowPtr, Alignment, "_msld"));
2929 } else {
2930 setShadow(&I, getCleanShadow(&I));
2931 }
2932
2934 insertShadowCheck(Addr, &I);
2935
2936 if (MS.TrackOrigins) {
2937 if (PropagateShadow)
2938 setOrigin(&I, IRB.CreateLoad(MS.OriginTy, OriginPtr));
2939 else
2940 setOrigin(&I, getCleanOrigin());
2941 }
2942 return true;
2943 }
2944
2945 /// Handle (SIMD arithmetic)-like intrinsics.
2946 ///
2947 /// Instrument intrinsics with any number of arguments of the same type,
2948 /// equal to the return type. The type should be simple (no aggregates or
2949 /// pointers; vectors are fine).
2950 /// Caller guarantees that this intrinsic does not access memory.
2951 bool maybeHandleSimpleNomemIntrinsic(IntrinsicInst &I) {
2952 Type *RetTy = I.getType();
2953 if (!(RetTy->isIntOrIntVectorTy() || RetTy->isFPOrFPVectorTy() ||
2954 RetTy->isX86_MMXTy()))
2955 return false;
2956
2957 unsigned NumArgOperands = I.arg_size();
2958 for (unsigned i = 0; i < NumArgOperands; ++i) {
2959 Type *Ty = I.getArgOperand(i)->getType();
2960 if (Ty != RetTy)
2961 return false;
2962 }
2963
2964 IRBuilder<> IRB(&I);
2965 ShadowAndOriginCombiner SC(this, IRB);
2966 for (unsigned i = 0; i < NumArgOperands; ++i)
2967 SC.Add(I.getArgOperand(i));
2968 SC.Done(&I);
2969
2970 return true;
2971 }
2972
2973 /// Heuristically instrument unknown intrinsics.
2974 ///
2975 /// The main purpose of this code is to do something reasonable with all
2976 /// random intrinsics we might encounter, most importantly - SIMD intrinsics.
2977 /// We recognize several classes of intrinsics by their argument types and
2978 /// ModRefBehaviour and apply special instrumentation when we are reasonably
2979 /// sure that we know what the intrinsic does.
2980 ///
2981 /// We special-case intrinsics where this approach fails. See llvm.bswap
2982 /// handling as an example of that.
2983 bool handleUnknownIntrinsic(IntrinsicInst &I) {
2984 unsigned NumArgOperands = I.arg_size();
2985 if (NumArgOperands == 0)
2986 return false;
2987
2988 if (NumArgOperands == 2 && I.getArgOperand(0)->getType()->isPointerTy() &&
2989 I.getArgOperand(1)->getType()->isVectorTy() &&
2990 I.getType()->isVoidTy() && !I.onlyReadsMemory()) {
2991 // This looks like a vector store.
2992 return handleVectorStoreIntrinsic(I);
2993 }
2994
2995 if (NumArgOperands == 1 && I.getArgOperand(0)->getType()->isPointerTy() &&
2996 I.getType()->isVectorTy() && I.onlyReadsMemory()) {
2997 // This looks like a vector load.
2998 return handleVectorLoadIntrinsic(I);
2999 }
3000
3001 if (I.doesNotAccessMemory())
3002 if (maybeHandleSimpleNomemIntrinsic(I))
3003 return true;
3004
3005 // FIXME: detect and handle SSE maskstore/maskload
3006 return false;
3007 }
3008
3009 void handleInvariantGroup(IntrinsicInst &I) {
3010 setShadow(&I, getShadow(&I, 0));
3011 setOrigin(&I, getOrigin(&I, 0));
3012 }
3013
3014 void handleLifetimeStart(IntrinsicInst &I) {
3015 if (!PoisonStack)
3016 return;
3017 AllocaInst *AI = llvm::findAllocaForValue(I.getArgOperand(1));
3018 if (!AI)
3019 InstrumentLifetimeStart = false;
3020 LifetimeStartList.push_back(std::make_pair(&I, AI));
3021 }
3022
3023 void handleBswap(IntrinsicInst &I) {
3024 IRBuilder<> IRB(&I);
3025 Value *Op = I.getArgOperand(0);
3026 Type *OpType = Op->getType();
3028 F.getParent(), Intrinsic::bswap, ArrayRef(&OpType, 1));
3029 setShadow(&I, IRB.CreateCall(BswapFunc, getShadow(Op)));
3030 setOrigin(&I, getOrigin(Op));
3031 }
3032
3033 void handleCountZeroes(IntrinsicInst &I) {
3034 IRBuilder<> IRB(&I);
3035 Value *Src = I.getArgOperand(0);
3036
3037 // Set the Output shadow based on input Shadow
3038 Value *BoolShadow = IRB.CreateIsNotNull(getShadow(Src), "_mscz_bs");
3039
3040 // If zero poison is requested, mix in with the shadow
3041 Constant *IsZeroPoison = cast<Constant>(I.getOperand(1));
3042 if (!IsZeroPoison->isZeroValue()) {
3043 Value *BoolZeroPoison = IRB.CreateIsNull(Src, "_mscz_bzp");
3044 BoolShadow = IRB.CreateOr(BoolShadow, BoolZeroPoison, "_mscz_bs");
3045 }
3046
3047 Value *OutputShadow =
3048 IRB.CreateSExt(BoolShadow, getShadowTy(Src), "_mscz_os");
3049
3050 setShadow(&I, OutputShadow);
3051 setOriginForNaryOp(I);
3052 }
3053
3054 // Instrument vector convert intrinsic.
3055 //
3056 // This function instruments intrinsics like cvtsi2ss:
3057 // %Out = int_xxx_cvtyyy(%ConvertOp)
3058 // or
3059 // %Out = int_xxx_cvtyyy(%CopyOp, %ConvertOp)
3060 // Intrinsic converts \p NumUsedElements elements of \p ConvertOp to the same
3061 // number \p Out elements, and (if has 2 arguments) copies the rest of the
3062 // elements from \p CopyOp.
3063 // In most cases conversion involves floating-point value which may trigger a
3064 // hardware exception when not fully initialized. For this reason we require
3065 // \p ConvertOp[0:NumUsedElements] to be fully initialized and trap otherwise.
3066 // We copy the shadow of \p CopyOp[NumUsedElements:] to \p
3067 // Out[NumUsedElements:]. This means that intrinsics without \p CopyOp always
3068 // return a fully initialized value.
3069 void handleVectorConvertIntrinsic(IntrinsicInst &I, int NumUsedElements,
3070 bool HasRoundingMode = false) {
3071 IRBuilder<> IRB(&I);
3072 Value *CopyOp, *ConvertOp;
3073
3074 assert((!HasRoundingMode ||
3075 isa<ConstantInt>(I.getArgOperand(I.arg_size() - 1))) &&
3076 "Invalid rounding mode");
3077
3078 switch (I.arg_size() - HasRoundingMode) {
3079 case 2:
3080 CopyOp = I.getArgOperand(0);
3081 ConvertOp = I.getArgOperand(1);
3082 break;
3083 case 1:
3084 ConvertOp = I.getArgOperand(0);
3085 CopyOp = nullptr;
3086 break;
3087 default:
3088 llvm_unreachable("Cvt intrinsic with unsupported number of arguments.");
3089 }
3090
3091 // The first *NumUsedElements* elements of ConvertOp are converted to the
3092 // same number of output elements. The rest of the output is copied from
3093 // CopyOp, or (if not available) filled with zeroes.
3094 // Combine shadow for elements of ConvertOp that are used in this operation,
3095 // and insert a check.
3096 // FIXME: consider propagating shadow of ConvertOp, at least in the case of
3097 // int->any conversion.
3098 Value *ConvertShadow = getShadow(ConvertOp);
3099 Value *AggShadow = nullptr;
3100 if (ConvertOp->getType()->isVectorTy()) {
3101 AggShadow = IRB.CreateExtractElement(
3102 ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), 0));
3103 for (int i = 1; i < NumUsedElements; ++i) {
3104 Value *MoreShadow = IRB.CreateExtractElement(
3105 ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), i));
3106 AggShadow = IRB.CreateOr(AggShadow, MoreShadow);
3107 }
3108 } else {
3109 AggShadow = ConvertShadow;
3110 }
3111 assert(AggShadow->getType()->isIntegerTy());
3112 insertShadowCheck(AggShadow, getOrigin(ConvertOp), &I);
3113
3114 // Build result shadow by zero-filling parts of CopyOp shadow that come from
3115 // ConvertOp.
3116 if (CopyOp) {
3117 assert(CopyOp->getType() == I.getType());
3118 assert(CopyOp->getType()->isVectorTy());
3119 Value *ResultShadow = getShadow(CopyOp);
3120 Type *EltTy = cast<VectorType>(ResultShadow->getType())->getElementType();
3121 for (int i = 0; i < NumUsedElements; ++i) {
3122 ResultShadow = IRB.CreateInsertElement(
3123 ResultShadow, ConstantInt::getNullValue(EltTy),
3124 ConstantInt::get(IRB.getInt32Ty(), i));
3125 }
3126 setShadow(&I, ResultShadow);
3127 setOrigin(&I, getOrigin(CopyOp));
3128 } else {
3129 setShadow(&I, getCleanShadow(&I));
3130 setOrigin(&I, getCleanOrigin());
3131 }
3132 }
3133
3134 // Given a scalar or vector, extract lower 64 bits (or less), and return all
3135 // zeroes if it is zero, and all ones otherwise.
3136 Value *Lower64ShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
3137 if (S->getType()->isVectorTy())
3138 S = CreateShadowCast(IRB, S, IRB.getInt64Ty(), /* Signed */ true);
3139 assert(S->getType()->getPrimitiveSizeInBits() <= 64);
3140 Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
3141 return CreateShadowCast(IRB, S2, T, /* Signed */ true);
3142 }
3143
3144 // Given a vector, extract its first element, and return all
3145 // zeroes if it is zero, and all ones otherwise.
3146 Value *LowerElementShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
3147 Value *S1 = IRB.CreateExtractElement(S, (uint64_t)0);
3148 Value *S2 = IRB.CreateICmpNE(S1, getCleanShadow(S1));
3149 return CreateShadowCast(IRB, S2, T, /* Signed */ true);
3150 }
3151
3152 Value *VariableShadowExtend(IRBuilder<> &IRB, Value *S) {
3153 Type *T = S->getType();
3154 assert(T->isVectorTy());
3155 Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
3156 return IRB.CreateSExt(S2, T);
3157 }
3158
3159 // Instrument vector shift intrinsic.
3160 //
3161 // This function instruments intrinsics like int_x86_avx2_psll_w.
3162 // Intrinsic shifts %In by %ShiftSize bits.
3163 // %ShiftSize may be a vector. In that case the lower 64 bits determine shift
3164 // size, and the rest is ignored. Behavior is defined even if shift size is
3165 // greater than register (or field) width.
3166 void handleVectorShiftIntrinsic(IntrinsicInst &I, bool Variable) {
3167 assert(I.arg_size() == 2);
3168 IRBuilder<> IRB(&I);
3169 // If any of the S2 bits are poisoned, the whole thing is poisoned.
3170 // Otherwise perform the same shift on S1.
3171 Value *S1 = getShadow(&I, 0);
3172 Value *S2 = getShadow(&I, 1);
3173 Value *S2Conv = Variable ? VariableShadowExtend(IRB, S2)
3174 : Lower64ShadowExtend(IRB, S2, getShadowTy(&I));
3175 Value *V1 = I.getOperand(0);
3176 Value *V2 = I.getOperand(1);
3177 Value *Shift = IRB.CreateCall(I.getFunctionType(), I.getCalledOperand(),
3178 {IRB.CreateBitCast(S1, V1->getType()), V2});
3179 Shift = IRB.CreateBitCast(Shift, getShadowTy(&I));
3180 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
3181 setOriginForNaryOp(I);
3182 }
3183
3184 // Get an X86_MMX-sized vector type.
3185 Type *getMMXVectorTy(unsigned EltSizeInBits) {
3186 const unsigned X86_MMXSizeInBits = 64;
3187 assert(EltSizeInBits != 0 && (X86_MMXSizeInBits % EltSizeInBits) == 0 &&
3188 "Illegal MMX vector element size");
3189 return FixedVectorType::get(IntegerType::get(*MS.C, EltSizeInBits),
3190 X86_MMXSizeInBits / EltSizeInBits);
3191 }
3192
3193 // Returns a signed counterpart for an (un)signed-saturate-and-pack
3194 // intrinsic.
3195 Intrinsic::ID getSignedPackIntrinsic(Intrinsic::ID id) {
3196 switch (id) {
3197 case Intrinsic::x86_sse2_packsswb_128:
3198 case Intrinsic::x86_sse2_packuswb_128:
3199 return Intrinsic::x86_sse2_packsswb_128;
3200
3201 case Intrinsic::x86_sse2_packssdw_128:
3202 case Intrinsic::x86_sse41_packusdw:
3203 return Intrinsic::x86_sse2_packssdw_128;
3204
3205 case Intrinsic::x86_avx2_packsswb:
3206 case Intrinsic::x86_avx2_packuswb:
3207 return Intrinsic::x86_avx2_packsswb;
3208
3209 case Intrinsic::x86_avx2_packssdw:
3210 case Intrinsic::x86_avx2_packusdw:
3211 return Intrinsic::x86_avx2_packssdw;
3212
3213 case Intrinsic::x86_mmx_packsswb:
3214 case Intrinsic::x86_mmx_packuswb:
3215 return Intrinsic::x86_mmx_packsswb;
3216
3217 case Intrinsic::x86_mmx_packssdw:
3218 return Intrinsic::x86_mmx_packssdw;
3219 default:
3220 llvm_unreachable("unexpected intrinsic id");
3221 }
3222 }
3223
3224 // Instrument vector pack intrinsic.
3225 //
3226 // This function instruments intrinsics like x86_mmx_packsswb, that
3227 // packs elements of 2 input vectors into half as many bits with saturation.
3228 // Shadow is propagated with the signed variant of the same intrinsic applied
3229 // to sext(Sa != zeroinitializer), sext(Sb != zeroinitializer).
3230 // EltSizeInBits is used only for x86mmx arguments.
3231 void handleVectorPackIntrinsic(IntrinsicInst &I, unsigned EltSizeInBits = 0) {
3232 assert(I.arg_size() == 2);
3233 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
3234 IRBuilder<> IRB(&I);
3235 Value *S1 = getShadow(&I, 0);
3236 Value *S2 = getShadow(&I, 1);
3237 assert(isX86_MMX || S1->getType()->isVectorTy());
3238
3239 // SExt and ICmpNE below must apply to individual elements of input vectors.
3240 // In case of x86mmx arguments, cast them to appropriate vector types and
3241 // back.
3242 Type *T = isX86_MMX ? getMMXVectorTy(EltSizeInBits) : S1->getType();
3243 if (isX86_MMX) {
3244 S1 = IRB.CreateBitCast(S1, T);
3245 S2 = IRB.CreateBitCast(S2, T);
3246 }
3247 Value *S1_ext =
3249 Value *S2_ext =
3251 if (isX86_MMX) {
3252 Type *X86_MMXTy = Type::getX86_MMXTy(*MS.C);
3253 S1_ext = IRB.CreateBitCast(S1_ext, X86_MMXTy);
3254 S2_ext = IRB.CreateBitCast(S2_ext, X86_MMXTy);
3255 }
3256
3258 F.getParent(), getSignedPackIntrinsic(I.getIntrinsicID()));
3259
3260 Value *S =
3261 IRB.CreateCall(ShadowFn, {S1_ext, S2_ext}, "_msprop_vector_pack");
3262 if (isX86_MMX)
3263 S = IRB.CreateBitCast(S, getShadowTy(&I));
3264 setShadow(&I, S);
3265 setOriginForNaryOp(I);
3266 }
3267
3268 // Instrument sum-of-absolute-differences intrinsic.
3269 void handleVectorSadIntrinsic(IntrinsicInst &I) {
3270 const unsigned SignificantBitsPerResultElement = 16;
3271 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
3272 Type *ResTy = isX86_MMX ? IntegerType::get(*MS.C, 64) : I.getType();
3273 unsigned ZeroBitsPerResultElement =
3274 ResTy->getScalarSizeInBits() - SignificantBitsPerResultElement;
3275
3276 IRBuilder<> IRB(&I);
3277 auto *Shadow0 = getShadow(&I, 0);
3278 auto *Shadow1 = getShadow(&I, 1);
3279 Value *S = IRB.CreateOr(Shadow0, Shadow1);
3280 S = IRB.CreateBitCast(S, ResTy);
3281 S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
3282 ResTy);
3283 S = IRB.CreateLShr(S, ZeroBitsPerResultElement);
3284 S = IRB.CreateBitCast(S, getShadowTy(&I));
3285 setShadow(&I, S);
3286 setOriginForNaryOp(I);
3287 }
3288
3289 // Instrument multiply-add intrinsic.
3290 void handleVectorPmaddIntrinsic(IntrinsicInst &I,
3291 unsigned EltSizeInBits = 0) {
3292 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
3293 Type *ResTy = isX86_MMX ? getMMXVectorTy(EltSizeInBits * 2) : I.getType();
3294 IRBuilder<> IRB(&I);
3295 auto *Shadow0 = getShadow(&I, 0);
3296 auto *Shadow1 = getShadow(&I, 1);
3297 Value *S = IRB.CreateOr(Shadow0, Shadow1);
3298 S = IRB.CreateBitCast(S, ResTy);
3299 S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
3300 ResTy);
3301 S = IRB.CreateBitCast(S, getShadowTy(&I));
3302 setShadow(&I, S);
3303 setOriginForNaryOp(I);
3304 }
3305
3306 // Instrument compare-packed intrinsic.
3307 // Basically, an or followed by sext(icmp ne 0) to end up with all-zeros or
3308 // all-ones shadow.
3309 void handleVectorComparePackedIntrinsic(IntrinsicInst &I) {
3310 IRBuilder<> IRB(&I);
3311 Type *ResTy = getShadowTy(&I);
3312 auto *Shadow0 = getShadow(&I, 0);
3313 auto *Shadow1 = getShadow(&I, 1);
3314 Value *S0 = IRB.CreateOr(Shadow0, Shadow1);
3315 Value *S = IRB.CreateSExt(
3316 IRB.CreateICmpNE(S0, Constant::getNullValue(ResTy)), ResTy);
3317 setShadow(&I, S);
3318 setOriginForNaryOp(I);
3319 }
3320
3321 // Instrument compare-scalar intrinsic.
3322 // This handles both cmp* intrinsics which return the result in the first
3323 // element of a vector, and comi* which return the result as i32.
3324 void handleVectorCompareScalarIntrinsic(IntrinsicInst &I) {
3325 IRBuilder<> IRB(&I);
3326 auto *Shadow0 = getShadow(&I, 0);
3327 auto *Shadow1 = getShadow(&I, 1);
3328 Value *S0 = IRB.CreateOr(Shadow0, Shadow1);
3329 Value *S = LowerElementShadowExtend(IRB, S0, getShadowTy(&I));
3330 setShadow(&I, S);
3331 setOriginForNaryOp(I);
3332 }
3333
3334 // Instrument generic vector reduction intrinsics
3335 // by ORing together all their fields.
3336 void handleVectorReduceIntrinsic(IntrinsicInst &I) {
3337 IRBuilder<> IRB(&I);
3338 Value *S = IRB.CreateOrReduce(getShadow(&I, 0));
3339 setShadow(&I, S);
3340 setOrigin(&I, getOrigin(&I, 0));
3341 }
3342
3343 // Instrument vector.reduce.or intrinsic.
3344 // Valid (non-poisoned) set bits in the operand pull low the
3345 // corresponding shadow bits.
3346 void handleVectorReduceOrIntrinsic(IntrinsicInst &I) {
3347 IRBuilder<> IRB(&I);
3348 Value *OperandShadow = getShadow(&I, 0);
3349 Value *OperandUnsetBits = IRB.CreateNot(I.getOperand(0));
3350 Value *OperandUnsetOrPoison = IRB.CreateOr(OperandUnsetBits, OperandShadow);
3351 // Bit N is clean if any field's bit N is 1 and unpoison
3352 Value *OutShadowMask = IRB.CreateAndReduce(OperandUnsetOrPoison);
3353 // Otherwise, it is clean if every field's bit N is unpoison
3354 Value *OrShadow = IRB.CreateOrReduce(OperandShadow);
3355 Value *S = IRB.CreateAnd(OutShadowMask, OrShadow);
3356
3357 setShadow(&I, S);
3358 setOrigin(&I, getOrigin(&I, 0));
3359 }
3360
3361 // Instrument vector.reduce.and intrinsic.
3362 // Valid (non-poisoned) unset bits in the operand pull down the
3363 // corresponding shadow bits.
3364 void handleVectorReduceAndIntrinsic(IntrinsicInst &I) {
3365 IRBuilder<> IRB(&I);
3366 Value *OperandShadow = getShadow(&I, 0);
3367 Value *OperandSetOrPoison = IRB.CreateOr(I.getOperand(0), OperandShadow);
3368 // Bit N is clean if any field's bit N is 0 and unpoison
3369 Value *OutShadowMask = IRB.CreateAndReduce(OperandSetOrPoison);
3370 // Otherwise, it is clean if every field's bit N is unpoison
3371 Value *OrShadow = IRB.CreateOrReduce(OperandShadow);
3372 Value *S = IRB.CreateAnd(OutShadowMask, OrShadow);
3373
3374 setShadow(&I, S);
3375 setOrigin(&I, getOrigin(&I, 0));
3376 }
3377
3378 void handleStmxcsr(IntrinsicInst &I) {
3379 IRBuilder<> IRB(&I);
3380 Value *Addr = I.getArgOperand(0);
3381 Type *Ty = IRB.getInt32Ty();
3382 Type *PtrTy = IRB.getPtrTy();
3383 Value *ShadowPtr =
3384 getShadowOriginPtr(Addr, IRB, Ty, Align(1), /*isStore*/ true).first;
3385
3386 IRB.CreateStore(getCleanShadow(Ty),
3387 IRB.CreatePointerCast(ShadowPtr, PtrTy));
3388
3390 insertShadowCheck(Addr, &I);
3391 }
3392
3393 void handleLdmxcsr(IntrinsicInst &I) {
3394 if (!InsertChecks)
3395 return;
3396
3397 IRBuilder<> IRB(&I);
3398 Value *Addr = I.getArgOperand(0);
3399 Type *Ty = IRB.getInt32Ty();
3400 const Align Alignment = Align(1);
3401 Value *ShadowPtr, *OriginPtr;
3402 std::tie(ShadowPtr, OriginPtr) =
3403 getShadowOriginPtr(Addr, IRB, Ty, Alignment, /*isStore*/ false);
3404
3406 insertShadowCheck(Addr, &I);
3407
3408 Value *Shadow = IRB.CreateAlignedLoad(Ty, ShadowPtr, Alignment, "_ldmxcsr");
3409 Value *Origin = MS.TrackOrigins ? IRB.CreateLoad(MS.OriginTy, OriginPtr)
3410 : getCleanOrigin();
3411 insertShadowCheck(Shadow, Origin, &I);
3412 }
3413
3414 void handleMaskedExpandLoad(IntrinsicInst &I) {
3415 IRBuilder<> IRB(&I);
3416 Value *Ptr = I.getArgOperand(0);
3417 Value *Mask = I.getArgOperand(1);
3418 Value *PassThru = I.getArgOperand(2);
3419
3421 insertShadowCheck(Ptr, &I);
3422 insertShadowCheck(Mask, &I);
3423 }
3424
3425 if (!PropagateShadow) {
3426 setShadow(&I, getCleanShadow(&I));
3427 setOrigin(&I, getCleanOrigin());
3428 return;
3429 }
3430
3431 Type *ShadowTy = getShadowTy(&I);
3432 Type *ElementShadowTy = cast<VectorType>(ShadowTy)->getElementType();
3433 auto [ShadowPtr, OriginPtr] =
3434 getShadowOriginPtr(Ptr, IRB, ElementShadowTy, {}, /*isStore*/ false);
3435
3436 Value *Shadow = IRB.CreateMaskedExpandLoad(
3437 ShadowTy, ShadowPtr, Mask, getShadow(PassThru), "_msmaskedexpload");
3438
3439 setShadow(&I, Shadow);
3440
3441 // TODO: Store origins.
3442 setOrigin(&I, getCleanOrigin());
3443 }
3444
3445 void handleMaskedCompressStore(IntrinsicInst &I) {
3446 IRBuilder<> IRB(&I);
3447 Value *Values = I.getArgOperand(0);
3448 Value *Ptr = I.getArgOperand(1);
3449 Value *Mask = I.getArgOperand(2);
3450
3452 insertShadowCheck(Ptr, &I);
3453 insertShadowCheck(Mask, &I);
3454 }
3455
3456 Value *Shadow = getShadow(Values);
3457 Type *ElementShadowTy =
3458 getShadowTy(cast<VectorType>(Values->getType())->getElementType());
3459 auto [ShadowPtr, OriginPtrs] =
3460 getShadowOriginPtr(Ptr, IRB, ElementShadowTy, {}, /*isStore*/ true);
3461
3462 IRB.CreateMaskedCompressStore(Shadow, ShadowPtr, Mask);
3463
3464 // TODO: Store origins.
3465 }
3466
3467 void handleMaskedGather(IntrinsicInst &I) {
3468 IRBuilder<> IRB(&I);
3469 Value *Ptrs = I.getArgOperand(0);
3470 const Align Alignment(
3471 cast<ConstantInt>(I.getArgOperand(1))->getZExtValue());
3472 Value *Mask = I.getArgOperand(2);
3473 Value *PassThru = I.getArgOperand(3);
3474
3475 Type *PtrsShadowTy = getShadowTy(Ptrs);
3477 insertShadowCheck(Mask, &I);
3478 Value *MaskedPtrShadow = IRB.CreateSelect(
3479 Mask, getShadow(Ptrs), Constant::getNullValue((PtrsShadowTy)),
3480 "_msmaskedptrs");
3481 insertShadowCheck(MaskedPtrShadow, getOrigin(Ptrs), &I);
3482 }
3483
3484 if (!PropagateShadow) {
3485 setShadow(&I, getCleanShadow(&I));
3486 setOrigin(&I, getCleanOrigin());
3487 return;
3488 }
3489
3490 Type *ShadowTy = getShadowTy(&I);
3491 Type *ElementShadowTy = cast<VectorType>(ShadowTy)->getElementType();
3492 auto [ShadowPtrs, OriginPtrs] = getShadowOriginPtr(
3493 Ptrs, IRB, ElementShadowTy, Alignment, /*isStore*/ false);
3494
3495 Value *Shadow =
3496 IRB.CreateMaskedGather(ShadowTy, ShadowPtrs, Alignment, Mask,
3497 getShadow(PassThru), "_msmaskedgather");
3498
3499 setShadow(&I, Shadow);
3500
3501 // TODO: Store origins.
3502 setOrigin(&I, getCleanOrigin());
3503 }
3504
3505 void handleMaskedScatter(IntrinsicInst &I) {
3506 IRBuilder<> IRB(&I);
3507 Value *Values = I.getArgOperand(0);
3508 Value *Ptrs = I.getArgOperand(1);
3509 const Align Alignment(
3510 cast<ConstantInt>(I.getArgOperand(2))->getZExtValue());
3511 Value *Mask = I.getArgOperand(3);
3512
3513 Type *PtrsShadowTy = getShadowTy(Ptrs);
3515 insertShadowCheck(Mask, &I);
3516 Value *MaskedPtrShadow = IRB.CreateSelect(
3517 Mask, getShadow(Ptrs), Constant::getNullValue((PtrsShadowTy)),
3518 "_msmaskedptrs");
3519 insertShadowCheck(MaskedPtrShadow, getOrigin(Ptrs), &I);
3520 }
3521
3522 Value *Shadow = getShadow(Values);
3523 Type *ElementShadowTy =
3524 getShadowTy(cast<VectorType>(Values->getType())->getElementType());
3525 auto [ShadowPtrs, OriginPtrs] = getShadowOriginPtr(
3526 Ptrs, IRB, ElementShadowTy, Alignment, /*isStore*/ true);
3527
3528 IRB.CreateMaskedScatter(Shadow, ShadowPtrs, Alignment, Mask);
3529
3530 // TODO: Store origin.
3531 }
3532
3533 void handleMaskedStore(IntrinsicInst &I) {
3534 IRBuilder<> IRB(&I);
3535 Value *V = I.getArgOperand(0);
3536 Value *Ptr = I.getArgOperand(1);
3537 const Align Alignment(
3538 cast<ConstantInt>(I.getArgOperand(2))->getZExtValue());
3539 Value *Mask = I.getArgOperand(3);
3540 Value *Shadow = getShadow(V);
3541
3543 insertShadowCheck(Ptr, &I);
3544 insertShadowCheck(Mask, &I);
3545 }
3546
3547 Value *ShadowPtr;
3548 Value *OriginPtr;
3549 std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
3550 Ptr, IRB, Shadow->getType(), Alignment, /*isStore*/ true);
3551
3552 IRB.CreateMaskedStore(Shadow, ShadowPtr, Alignment, Mask);
3553
3554 if (!MS.TrackOrigins)
3555 return;
3556
3557 auto &DL = F.getParent()->getDataLayout();
3558 paintOrigin(IRB, getOrigin(V), OriginPtr,
3559 DL.getTypeStoreSize(Shadow->getType()),
3560 std::max(Alignment, kMinOriginAlignment));
3561 }
3562
3563 void handleMaskedLoad(IntrinsicInst &I) {
3564 IRBuilder<> IRB(&I);
3565 Value *Ptr = I.getArgOperand(0);
3566 const Align Alignment(
3567 cast<ConstantInt>(I.getArgOperand(1))->getZExtValue());
3568 Value *Mask = I.getArgOperand(2);
3569 Value *PassThru = I.getArgOperand(3);
3570
3572 insertShadowCheck(Ptr, &I);
3573 insertShadowCheck(Mask, &I);
3574 }
3575
3576 if (!PropagateShadow) {
3577 setShadow(&I, getCleanShadow(&I));
3578 setOrigin(&I, getCleanOrigin());
3579 return;
3580 }
3581
3582 Type *ShadowTy = getShadowTy(&I);
3583 Value *ShadowPtr, *OriginPtr;
3584 std::tie(ShadowPtr, OriginPtr) =
3585 getShadowOriginPtr(Ptr, IRB, ShadowTy, Alignment, /*isStore*/ false);
3586 setShadow(&I, IRB.CreateMaskedLoad(ShadowTy, ShadowPtr, Alignment, Mask,
3587 getShadow(PassThru), "_msmaskedld"));
3588
3589 if (!MS.TrackOrigins)
3590 return;
3591
3592 // Choose between PassThru's and the loaded value's origins.
3593 Value *MaskedPassThruShadow = IRB.CreateAnd(
3594 getShadow(PassThru), IRB.CreateSExt(IRB.CreateNeg(Mask), ShadowTy));
3595
3596 Value *NotNull = convertToBool(MaskedPassThruShadow, IRB, "_mscmp");
3597
3598 Value *PtrOrigin = IRB.CreateLoad(MS.OriginTy, OriginPtr);
3599 Value *Origin = IRB.CreateSelect(NotNull, getOrigin(PassThru), PtrOrigin);
3600
3601 setOrigin(&I, Origin);
3602 }
3603
3604 // Instrument BMI / BMI2 intrinsics.
3605 // All of these intrinsics are Z = I(X, Y)
3606 // where the types of all operands and the result match, and are either i32 or
3607 // i64. The following instrumentation happens to work for all of them:
3608 // Sz = I(Sx, Y) | (sext (Sy != 0))
3609 void handleBmiIntrinsic(IntrinsicInst &I) {
3610 IRBuilder<> IRB(&I);
3611 Type *ShadowTy = getShadowTy(&I);
3612
3613 // If any bit of the mask operand is poisoned, then the whole thing is.
3614 Value *SMask = getShadow(&I, 1);
3615 SMask = IRB.CreateSExt(IRB.CreateICmpNE(SMask, getCleanShadow(ShadowTy)),
3616 ShadowTy);
3617 // Apply the same intrinsic to the shadow of the first operand.
3618 Value *S = IRB.CreateCall(I.getCalledFunction(),
3619 {getShadow(&I, 0), I.getOperand(1)});
3620 S = IRB.CreateOr(SMask, S);
3621 setShadow(&I, S);
3622 setOriginForNaryOp(I);
3623 }
3624
3625 SmallVector<int, 8> getPclmulMask(unsigned Width, bool OddElements) {
3627 for (unsigned X = OddElements ? 1 : 0; X < Width; X += 2) {
3628 Mask.append(2, X);
3629 }
3630 return Mask;
3631 }
3632
3633 // Instrument pclmul intrinsics.
3634 // These intrinsics operate either on odd or on even elements of the input
3635 // vectors, depending on the constant in the 3rd argument, ignoring the rest.
3636 // Replace the unused elements with copies of the used ones, ex:
3637 // (0, 1, 2, 3) -> (0, 0, 2, 2) (even case)
3638 // or
3639 // (0, 1, 2, 3) -> (1, 1, 3, 3) (odd case)
3640 // and then apply the usual shadow combining logic.
3641 void handlePclmulIntrinsic(IntrinsicInst &I) {
3642 IRBuilder<> IRB(&I);
3643 unsigned Width =
3644 cast<FixedVectorType>(I.getArgOperand(0)->getType())->getNumElements();
3645 assert(isa<ConstantInt>(I.getArgOperand(2)) &&
3646 "pclmul 3rd operand must be a constant");
3647 unsigned Imm = cast<ConstantInt>(I.getArgOperand(2))->getZExtValue();
3648 Value *Shuf0 = IRB.CreateShuffleVector(getShadow(&I, 0),
3649 getPclmulMask(Width, Imm & 0x01));
3650 Value *Shuf1 = IRB.CreateShuffleVector(getShadow(&I, 1),
3651 getPclmulMask(Width, Imm & 0x10));
3652 ShadowAndOriginCombiner SOC(this, IRB);
3653 SOC.Add(Shuf0, getOrigin(&I, 0));
3654 SOC.Add(Shuf1, getOrigin(&I, 1));
3655 SOC.Done(&I);
3656 }
3657
3658 // Instrument _mm_*_sd|ss intrinsics
3659 void handleUnarySdSsIntrinsic(IntrinsicInst &I) {
3660 IRBuilder<> IRB(&I);
3661 unsigned Width =
3662 cast<FixedVectorType>(I.getArgOperand(0)->getType())->getNumElements();
3663 Value *First = getShadow(&I, 0);
3664 Value *Second = getShadow(&I, 1);
3665 // First element of second operand, remaining elements of first operand
3667 Mask.push_back(Width);
3668 for (unsigned i = 1; i < Width; i++)
3669 Mask.push_back(i);
3670 Value *Shadow = IRB.CreateShuffleVector(First, Second, Mask);
3671
3672 setShadow(&I, Shadow);
3673 setOriginForNaryOp(I);
3674 }
3675
3676 void handleVtestIntrinsic(IntrinsicInst &I) {
3677 IRBuilder<> IRB(&I);
3678 Value *Shadow0 = getShadow(&I, 0);
3679 Value *Shadow1 = getShadow(&I, 1);
3680 Value *Or = IRB.CreateOr(Shadow0, Shadow1);
3681 Value *NZ = IRB.CreateICmpNE(Or, Constant::getNullValue(Or->getType()));
3682 Value *Scalar = convertShadowToScalar(NZ, IRB);
3683 Value *Shadow = IRB.CreateZExt(Scalar, getShadowTy(&I));
3684
3685 setShadow(&I, Shadow);
3686 setOriginForNaryOp(I);
3687 }
3688
3689 void handleBinarySdSsIntrinsic(IntrinsicInst &I) {
3690 IRBuilder<> IRB(&I);
3691 unsigned Width =
3692 cast<FixedVectorType>(I.getArgOperand(0)->getType())->getNumElements();
3693 Value *First = getShadow(&I, 0);
3694 Value *Second = getShadow(&I, 1);
3695 Value *OrShadow = IRB.CreateOr(First, Second);
3696 // First element of both OR'd together, remaining elements of first operand
3698 Mask.push_back(Width);
3699 for (unsigned i = 1; i < Width; i++)
3700 Mask.push_back(i);
3701 Value *Shadow = IRB.CreateShuffleVector(First, OrShadow, Mask);
3702
3703 setShadow(&I, Shadow);
3704 setOriginForNaryOp(I);
3705 }
3706
3707 // Instrument abs intrinsic.
3708 // handleUnknownIntrinsic can't handle it because of the last
3709 // is_int_min_poison argument which does not match the result type.
3710 void handleAbsIntrinsic(IntrinsicInst &I) {
3711 assert(I.getType()->isIntOrIntVectorTy());
3712 assert(I.getArgOperand(0)->getType() == I.getType());
3713
3714 // FIXME: Handle is_int_min_poison.
3715 IRBuilder<> IRB(&I);
3716 setShadow(&I, getShadow(&I, 0));
3717 setOrigin(&I, getOrigin(&I, 0));
3718 }
3719
3720 void handleIsFpClass(IntrinsicInst &I) {
3721 IRBuilder<> IRB(&I);
3722 Value *Shadow = getShadow(&I, 0);
3723 setShadow(&I, IRB.CreateICmpNE(Shadow, getCleanShadow(Shadow)));
3724 setOrigin(&I, getOrigin(&I, 0));
3725 }
3726
3727 void visitIntrinsicInst(IntrinsicInst &I) {
3728 switch (I.getIntrinsicID()) {
3729 case Intrinsic::abs:
3730 handleAbsIntrinsic(I);
3731 break;
3732 case Intrinsic::is_fpclass:
3733 handleIsFpClass(I);
3734 break;
3735 case Intrinsic::lifetime_start:
3736 handleLifetimeStart(I);
3737 break;
3738 case Intrinsic::launder_invariant_group:
3739 case Intrinsic::strip_invariant_group:
3740 handleInvariantGroup(I);
3741 break;
3742 case Intrinsic::bswap:
3743 handleBswap(I);
3744 break;
3745 case Intrinsic::ctlz:
3746 case Intrinsic::cttz:
3747 handleCountZeroes(I);
3748 break;
3749 case Intrinsic::masked_compressstore:
3750 handleMaskedCompressStore(I);
3751 break;
3752 case Intrinsic::masked_expandload:
3753 handleMaskedExpandLoad(I);
3754 break;
3755 case Intrinsic::masked_gather:
3756 handleMaskedGather(I);
3757 break;
3758 case Intrinsic::masked_scatter:
3759 handleMaskedScatter(I);
3760 break;
3761 case Intrinsic::masked_store:
3762 handleMaskedStore(I);
3763 break;
3764 case Intrinsic::masked_load:
3765 handleMaskedLoad(I);
3766 break;
3767 case Intrinsic::vector_reduce_and:
3768 handleVectorReduceAndIntrinsic(I);
3769 break;
3770 case Intrinsic::vector_reduce_or:
3771 handleVectorReduceOrIntrinsic(I);
3772 break;
3773 case Intrinsic::vector_reduce_add:
3774 case Intrinsic::vector_reduce_xor:
3775 case Intrinsic::vector_reduce_mul:
3776 handleVectorReduceIntrinsic(I);
3777 break;
3778 case Intrinsic::x86_sse_stmxcsr:
3779 handleStmxcsr(I);
3780 break;
3781 case Intrinsic::x86_sse_ldmxcsr:
3782 handleLdmxcsr(I);
3783 break;
3784 case Intrinsic::x86_avx512_vcvtsd2usi64:
3785 case Intrinsic::x86_avx512_vcvtsd2usi32:
3786 case Intrinsic::x86_avx512_vcvtss2usi64:
3787 case Intrinsic::x86_avx512_vcvtss2usi32:
3788 case Intrinsic::x86_avx512_cvttss2usi64:
3789 case Intrinsic::x86_avx512_cvttss2usi:
3790 case Intrinsic::x86_avx512_cvttsd2usi64:
3791 case Intrinsic::x86_avx512_cvttsd2usi:
3792 case Intrinsic::x86_avx512_cvtusi2ss:
3793 case Intrinsic::x86_avx512_cvtusi642sd:
3794 case Intrinsic::x86_avx512_cvtusi642ss:
3795 handleVectorConvertIntrinsic(I, 1, true);
3796 break;
3797 case Intrinsic::x86_sse2_cvtsd2si64:
3798 case Intrinsic::x86_sse2_cvtsd2si:
3799 case Intrinsic::x86_sse2_cvtsd2ss:
3800 case Intrinsic::x86_sse2_cvttsd2si64:
3801 case Intrinsic::x86_sse2_cvttsd2si:
3802 case Intrinsic::x86_sse_cvtss2si64:
3803 case Intrinsic::x86_sse_cvtss2si:
3804 case Intrinsic::x86_sse_cvttss2si64:
3805 case Intrinsic::x86_sse_cvttss2si:
3806 handleVectorConvertIntrinsic(I, 1);
3807 break;
3808 case Intrinsic::x86_sse_cvtps2pi:
3809 case Intrinsic::x86_sse_cvttps2pi:
3810 handleVectorConvertIntrinsic(I, 2);
3811 break;
3812
3813 case Intrinsic::x86_avx512_psll_w_512:
3814 case Intrinsic::x86_avx512_psll_d_512:
3815 case Intrinsic::x86_avx512_psll_q_512:
3816 case Intrinsic::x86_avx512_pslli_w_512:
3817 case Intrinsic::x86_avx512_pslli_d_512:
3818 case Intrinsic::x86_avx512_pslli_q_512:
3819 case Intrinsic::x86_avx512_psrl_w_512:
3820 case Intrinsic::x86_avx512_psrl_d_512:
3821 case Intrinsic::x86_avx512_psrl_q_512:
3822 case Intrinsic::x86_avx512_psra_w_512:
3823 case Intrinsic::x86_avx512_psra_d_512:
3824 case Intrinsic::x86_avx512_psra_q_512:
3825 case Intrinsic::x86_avx512_psrli_w_512:
3826 case Intrinsic::x86_avx512_psrli_d_512:
3827 case Intrinsic::x86_avx512_psrli_q_512:
3828 case Intrinsic::x86_avx512_psrai_w_512:
3829 case Intrinsic::x86_avx512_psrai_d_512:
3830 case Intrinsic::x86_avx512_psrai_q_512:
3831 case Intrinsic::x86_avx512_psra_q_256:
3832 case Intrinsic::x86_avx512_psra_q_128:
3833 case Intrinsic::x86_avx512_psrai_q_256:
3834 case Intrinsic::x86_avx512_psrai_q_128:
3835 case Intrinsic::x86_avx2_psll_w:
3836 case Intrinsic::x86_avx2_psll_d:
3837 case Intrinsic::x86_avx2_psll_q:
3838 case Intrinsic::x86_avx2_pslli_w:
3839 case Intrinsic::x86_avx2_pslli_d:
3840 case Intrinsic::x86_avx2_pslli_q:
3841 case Intrinsic::x86_avx2_psrl_w:
3842 case Intrinsic::x86_avx2_psrl_d:
3843 case Intrinsic::x86_avx2_psrl_q:
3844 case Intrinsic::x86_avx2_psra_w:
3845 case Intrinsic::x86_avx2_psra_d:
3846 case Intrinsic::x86_avx2_psrli_w:
3847 case Intrinsic::x86_avx2_psrli_d:
3848 case Intrinsic::x86_avx2_psrli_q:
3849 case Intrinsic::x86_avx2_psrai_w:
3850 case Intrinsic::x86_avx2_psrai_d:
3851 case Intrinsic::x86_sse2_psll_w:
3852 case Intrinsic::x86_sse2_psll_d:
3853 case Intrinsic::x86_sse2_psll_q:
3854 case Intrinsic::x86_sse2_pslli_w:
3855 case Intrinsic::x86_sse2_pslli_d:
3856 case Intrinsic::x86_sse2_pslli_q:
3857 case Intrinsic::x86_sse2_psrl_w:
3858 case Intrinsic::x86_sse2_psrl_d:
3859 case Intrinsic::x86_sse2_psrl_q:
3860 case Intrinsic::x86_sse2_psra_w:
3861 case Intrinsic::x86_sse2_psra_d:
3862 case Intrinsic::x86_sse2_psrli_w:
3863 case Intrinsic::x86_sse2_psrli_d:
3864 case Intrinsic::x86_sse2_psrli_q:
3865 case Intrinsic::x86_sse2_psrai_w:
3866 case Intrinsic::x86_sse2_psrai_d:
3867 case Intrinsic::x86_mmx_psll_w:
3868 case Intrinsic::x86_mmx_psll_d:
3869 case Intrinsic::x86_mmx_psll_q:
3870 case Intrinsic::x86_mmx_pslli_w:
3871 case Intrinsic::x86_mmx_pslli_d:
3872 case Intrinsic::x86_mmx_pslli_q:
3873 case Intrinsic::x86_mmx_psrl_w:
3874 case Intrinsic::x86_mmx_psrl_d:
3875 case Intrinsic::x86_mmx_psrl_q:
3876 case Intrinsic::x86_mmx_psra_w:
3877 case Intrinsic::x86_mmx_psra_d:
3878 case Intrinsic::x86_mmx_psrli_w:
3879 case Intrinsic::x86_mmx_psrli_d:
3880 case Intrinsic::x86_mmx_psrli_q:
3881 case Intrinsic::x86_mmx_psrai_w:
3882 case Intrinsic::x86_mmx_psrai_d:
3883 handleVectorShiftIntrinsic(I, /* Variable */ false);
3884 break;
3885 case Intrinsic::x86_avx2_psllv_d:
3886 case Intrinsic::x86_avx2_psllv_d_256:
3887 case Intrinsic::x86_avx512_psllv_d_512:
3888 case Intrinsic::x86_avx2_psllv_q:
3889 case Intrinsic::x86_avx2_psllv_q_256:
3890 case Intrinsic::x86_avx512_psllv_q_512:
3891 case Intrinsic::x86_avx2_psrlv_d:
3892 case Intrinsic::x86_avx2_psrlv_d_256:
3893 case Intrinsic::x86_avx512_psrlv_d_512:
3894 case Intrinsic::x86_avx2_psrlv_q:
3895 case Intrinsic::x86_avx2_psrlv_q_256:
3896 case Intrinsic::x86_avx512_psrlv_q_512:
3897 case Intrinsic::x86_avx2_psrav_d:
3898 case Intrinsic::x86_avx2_psrav_d_256:
3899 case Intrinsic::x86_avx512_psrav_d_512:
3900 case Intrinsic::x86_avx512_psrav_q_128:
3901 case Intrinsic::x86_avx512_psrav_q_256:
3902 case Intrinsic::x86_avx512_psrav_q_512:
3903 handleVectorShiftIntrinsic(I, /* Variable */ true);
3904 break;
3905
3906 case Intrinsic::x86_sse2_packsswb_128:
3907 case Intrinsic::x86_sse2_packssdw_128:
3908 case Intrinsic::x86_sse2_packuswb_128:
3909 case Intrinsic::x86_sse41_packusdw:
3910 case Intrinsic::x86_avx2_packsswb:
3911 case Intrinsic::x86_avx2_packssdw:
3912 case Intrinsic::x86_avx2_packuswb:
3913 case Intrinsic::x86_avx2_packusdw:
3914 handleVectorPackIntrinsic(I);
3915 break;
3916
3917 case Intrinsic::x86_mmx_packsswb:
3918 case Intrinsic::x86_mmx_packuswb:
3919 handleVectorPackIntrinsic(I, 16);
3920 break;
3921
3922 case Intrinsic::x86_mmx_packssdw:
3923 handleVectorPackIntrinsic(I, 32);
3924 break;
3925
3926 case Intrinsic::x86_mmx_psad_bw:
3927 case Intrinsic::x86_sse2_psad_bw:
3928 case Intrinsic::x86_avx2_psad_bw:
3929 handleVectorSadIntrinsic(I);
3930 break;
3931
3932 case Intrinsic::x86_sse2_pmadd_wd:
3933 case Intrinsic::x86_avx2_pmadd_wd:
3934 case Intrinsic::x86_ssse3_pmadd_ub_sw_128:
3935 case Intrinsic::x86_avx2_pmadd_ub_sw:
3936 handleVectorPmaddIntrinsic(I);
3937 break;
3938
3939 case Intrinsic::x86_ssse3_pmadd_ub_sw:
3940 handleVectorPmaddIntrinsic(I, 8);
3941 break;
3942
3943 case Intrinsic::x86_mmx_pmadd_wd:
3944 handleVectorPmaddIntrinsic(I, 16);
3945 break;
3946
3947 case Intrinsic::x86_sse_cmp_ss:
3948 case Intrinsic::x86_sse2_cmp_sd:
3949 case Intrinsic::x86_sse_comieq_ss:
3950 case Intrinsic::x86_sse_comilt_ss:
3951 case Intrinsic::x86_sse_comile_ss:
3952 case Intrinsic::x86_sse_comigt_ss:
3953 case Intrinsic::x86_sse_comige_ss:
3954 case Intrinsic::x86_sse_comineq_ss:
3955 case Intrinsic::x86_sse_ucomieq_ss:
3956 case Intrinsic::x86_sse_ucomilt_ss:
3957 case Intrinsic::x86_sse_ucomile_ss:
3958 case Intrinsic::x86_sse_ucomigt_ss:
3959 case Intrinsic::x86_sse_ucomige_ss:
3960 case Intrinsic::x86_sse_ucomineq_ss:
3961 case Intrinsic::x86_sse2_comieq_sd:
3962 case Intrinsic::x86_sse2_comilt_sd:
3963 case Intrinsic::x86_sse2_comile_sd:
3964 case Intrinsic::x86_sse2_comigt_sd:
3965 case Intrinsic::x86_sse2_comige_sd:
3966 case Intrinsic::x86_sse2_comineq_sd:
3967 case Intrinsic::x86_sse2_ucomieq_sd:
3968 case Intrinsic::x86_sse2_ucomilt_sd:
3969 case Intrinsic::x86_sse2_ucomile_sd:
3970 case Intrinsic::x86_sse2_ucomigt_sd:
3971 case Intrinsic::x86_sse2_ucomige_sd:
3972 case Intrinsic::x86_sse2_ucomineq_sd:
3973 handleVectorCompareScalarIntrinsic(I);
3974 break;
3975
3976 case Intrinsic::x86_avx_cmp_pd_256:
3977 case Intrinsic::x86_avx_cmp_ps_256:
3978 case Intrinsic::x86_sse2_cmp_pd:
3979 case Intrinsic::x86_sse_cmp_ps:
3980 handleVectorComparePackedIntrinsic(I);
3981 break;
3982
3983 case Intrinsic::x86_bmi_bextr_32:
3984 case Intrinsic::x86_bmi_bextr_64:
3985 case Intrinsic::x86_bmi_bzhi_32:
3986 case Intrinsic::x86_bmi_bzhi_64:
3987 case Intrinsic::x86_bmi_pdep_32:
3988 case Intrinsic::x86_bmi_pdep_64:
3989 case Intrinsic::x86_bmi_pext_32:
3990 case Intrinsic::x86_bmi_pext_64:
3991 handleBmiIntrinsic(I);
3992 break;
3993
3994 case Intrinsic::x86_pclmulqdq:
3995 case Intrinsic::x86_pclmulqdq_256:
3996 case Intrinsic::x86_pclmulqdq_512:
3997 handlePclmulIntrinsic(I);
3998 break;
3999
4000 case Intrinsic::x86_sse41_round_sd:
4001 case Intrinsic::x86_sse41_round_ss:
4002 handleUnarySdSsIntrinsic(I);
4003 break;
4004 case Intrinsic::x86_sse2_max_sd:
4005 case Intrinsic::x86_sse_max_ss:
4006 case Intrinsic::x86_sse2_min_sd:
4007 case Intrinsic::x86_sse_min_ss:
4008 handleBinarySdSsIntrinsic(I);
4009 break;
4010
4011 case Intrinsic::x86_avx_vtestc_pd:
4012 case Intrinsic::x86_avx_vtestc_pd_256:
4013 case Intrinsic::x86_avx_vtestc_ps:
4014 case Intrinsic::x86_avx_vtestc_ps_256:
4015 case Intrinsic::x86_avx_vtestnzc_pd:
4016 case Intrinsic::x86_avx_vtestnzc_pd_256:
4017 case Intrinsic::x86_avx_vtestnzc_ps:
4018 case Intrinsic::x86_avx_vtestnzc_ps_256:
4019 case Intrinsic::x86_avx_vtestz_pd:
4020 case Intrinsic::x86_avx_vtestz_pd_256:
4021 case Intrinsic::x86_avx_vtestz_ps:
4022 case Intrinsic::x86_avx_vtestz_ps_256:
4023 case Intrinsic::x86_avx_ptestc_256:
4024 case Intrinsic::x86_avx_ptestnzc_256:
4025 case Intrinsic::x86_avx_ptestz_256:
4026 case Intrinsic::x86_sse41_ptestc:
4027 case Intrinsic::x86_sse41_ptestnzc:
4028 case Intrinsic::x86_sse41_ptestz:
4029 handleVtestIntrinsic(I);
4030 break;
4031
4032 case Intrinsic::fshl:
4033 case Intrinsic::fshr:
4034 handleFunnelShift(I);
4035 break;
4036
4037 case Intrinsic::is_constant:
4038 // The result of llvm.is.constant() is always defined.
4039 setShadow(&I, getCleanShadow(&I));
4040 setOrigin(&I, getCleanOrigin());
4041 break;
4042
4043 default:
4044 if (!handleUnknownIntrinsic(I))
4045 visitInstruction(I);
4046 break;
4047 }
4048 }
4049
4050 void visitLibAtomicLoad(CallBase &CB) {
4051 // Since we use getNextNode here, we can't have CB terminate the BB.
4052 assert(isa<CallInst>(CB));
4053
4054 IRBuilder<> IRB(&CB);
4055 Value *Size = CB.getArgOperand(0);
4056 Value *SrcPtr = CB.getArgOperand(1);
4057 Value *DstPtr = CB.getArgOperand(2);
4058 Value *Ordering = CB.getArgOperand(3);
4059 // Convert the call to have at least Acquire ordering to make sure
4060 // the shadow operations aren't reordered before it.
4061 Value *NewOrdering =
4062 IRB.CreateExtractElement(makeAddAcquireOrderingTable(IRB), Ordering);
4063 CB.setArgOperand(3, NewOrdering);
4064
4065 NextNodeIRBuilder NextIRB(&CB);
4066 Value *SrcShadowPtr, *SrcOriginPtr;
4067 std::tie(SrcShadowPtr, SrcOriginPtr) =
4068 getShadowOriginPtr(SrcPtr, NextIRB, NextIRB.getInt8Ty(), Align(1),
4069 /*isStore*/ false);
4070 Value *DstShadowPtr =
4071 getShadowOriginPtr(DstPtr, NextIRB, NextIRB.getInt8Ty(), Align(1),
4072 /*isStore*/ true)
4073 .first;
4074
4075 NextIRB.CreateMemCpy(DstShadowPtr, Align(1), SrcShadowPtr, Align(1), Size);
4076 if (MS.TrackOrigins) {
4077 Value *SrcOrigin = NextIRB.CreateAlignedLoad(MS.OriginTy, SrcOriginPtr,
4079 Value *NewOrigin = updateOrigin(SrcOrigin, NextIRB);
4080 NextIRB.CreateCall(MS.MsanSetOriginFn, {DstPtr, Size, NewOrigin});
4081 }
4082 }
4083
4084 void visitLibAtomicStore(CallBase &CB) {
4085 IRBuilder<> IRB(&CB);
4086 Value *Size = CB.getArgOperand(0);
4087 Value *DstPtr = CB.getArgOperand(2);
4088 Value *Ordering = CB.getArgOperand(3);
4089 // Convert the call to have at least Release ordering to make sure
4090 // the shadow operations aren't reordered after it.
4091 Value *NewOrdering =
4092 IRB.CreateExtractElement(makeAddReleaseOrderingTable(IRB), Ordering);
4093 CB.setArgOperand(3, NewOrdering);
4094
4095 Value *DstShadowPtr =
4096 getShadowOriginPtr(DstPtr, IRB, IRB.getInt8Ty(), Align(1),
4097 /*isStore*/ true)
4098 .first;
4099
4100 // Atomic store always paints clean shadow/origin. See file header.
4101 IRB.CreateMemSet(DstShadowPtr, getCleanShadow(IRB.getInt8Ty()), Size,
4102 Align(1));
4103 }
4104
4105 void visitCallBase(CallBase &CB) {
4106 assert(!CB.getMetadata(LLVMContext::MD_nosanitize));
4107 if (CB.isInlineAsm()) {
4108 // For inline asm (either a call to asm function, or callbr instruction),
4109 // do the usual thing: check argument shadow and mark all outputs as
4110 // clean. Note that any side effects of the inline asm that are not
4111 // immediately visible in its constraints are not handled.
4112 if (ClHandleAsmConservative && MS.CompileKernel)
4113 visitAsmInstruction(CB);
4114 else
4115 visitInstruction(CB);
4116 return;
4117 }
4118 LibFunc LF;
4119 if (TLI->getLibFunc(CB, LF)) {
4120 // libatomic.a functions need to have special handling because there isn't
4121 // a good way to intercept them or compile the library with
4122 // instrumentation.
4123 switch (LF) {
4124 case LibFunc_atomic_load:
4125 if (!isa<CallInst>(CB)) {
4126 llvm::errs() << "MSAN -- cannot instrument invoke of libatomic load."
4127 "Ignoring!\n";
4128 break;
4129 }
4130 visitLibAtomicLoad(CB);
4131 return;
4132 case LibFunc_atomic_store:
4133 visitLibAtomicStore(CB);
4134 return;
4135 default:
4136 break;
4137 }
4138 }
4139
4140 if (auto *Call = dyn_cast<CallInst>(&CB)) {
4141 assert(!isa<IntrinsicInst>(Call) && "intrinsics are handled elsewhere");
4142
4143 // We are going to insert code that relies on the fact that the callee
4144 // will become a non-readonly function after it is instrumented by us. To
4145 // prevent this code from being optimized out, mark that function
4146 // non-readonly in advance.
4147 // TODO: We can likely do better than dropping memory() completely here.
4149 B.addAttribute(Attribute::Memory).addAttribute(Attribute::Speculatable);
4150
4151 Call->removeFnAttrs(B);
4152 if (Function *Func = Call->getCalledFunction()) {
4153 Func->removeFnAttrs(B);
4154 }
4155
4157 }
4158 IRBuilder<> IRB(&CB);
4159 bool MayCheckCall = MS.EagerChecks;
4160 if (Function *Func = CB.getCalledFunction()) {
4161 // __sanitizer_unaligned_{load,store} functions may be called by users
4162 // and always expects shadows in the TLS. So don't check them.
4163 MayCheckCall &= !Func->getName().startswith("__sanitizer_unaligned_");
4164 }
4165
4166 unsigned ArgOffset = 0;
4167 LLVM_DEBUG(dbgs() << " CallSite: " << CB << "\n");
4168 for (const auto &[i, A] : llvm::enumerate(CB.args())) {
4169 if (!A->getType()->isSized()) {
4170 LLVM_DEBUG(dbgs() << "Arg " << i << " is not sized: " << CB << "\n");
4171 continue;
4172 }
4173 unsigned Size = 0;
4174 const DataLayout &DL = F.getParent()->getDataLayout();
4175
4176 bool ByVal = CB.paramHasAttr(i, Attribute::ByVal);
4177 bool NoUndef = CB.paramHasAttr(i, Attribute::NoUndef);
4178 bool EagerCheck = MayCheckCall && !ByVal && NoUndef;
4179
4180 if (EagerCheck) {
4181 insertShadowCheck(A, &CB);
4182 Size = DL.getTypeAllocSize(A->getType());
4183 } else {
4184 Value *Store = nullptr;
4185 // Compute the Shadow for arg even if it is ByVal, because
4186 // in that case getShadow() will copy the actual arg shadow to
4187 // __msan_param_tls.
4188 Value *ArgShadow = getShadow(A);
4189 Value *ArgShadowBase = getShadowPtrForArgument(IRB, ArgOffset);
4190 LLVM_DEBUG(dbgs() << " Arg#" << i << ": " << *A
4191 << " Shadow: " << *ArgShadow << "\n");
4192 if (ByVal) {
4193 // ByVal requires some special handling as it's too big for a single
4194 // load
4195 assert(A->getType()->isPointerTy() &&
4196 "ByVal argument is not a pointer!");
4197 Size = DL.getTypeAllocSize(CB.getParamByValType(i));
4198 if (ArgOffset + Size > kParamTLSSize)
4199 break;
4200 const MaybeAlign ParamAlignment(CB.getParamAlign(i));
4201 MaybeAlign Alignment = std::nullopt;
4202 if (ParamAlignment)
4203 Alignment = std::min(*ParamAlignment, kShadowTLSAlignment);
4204 Value *AShadowPtr, *AOriginPtr;
4205 std::tie(AShadowPtr, AOriginPtr) =
4206 getShadowOriginPtr(A, IRB, IRB.getInt8Ty(), Alignment,
4207 /*isStore*/ false);
4208 if (!PropagateShadow) {
4209 Store = IRB.CreateMemSet(ArgShadowBase,
4211 Size, Alignment);
4212 } else {
4213 Store = IRB.CreateMemCpy(ArgShadowBase, Alignment, AShadowPtr,
4214 Alignment, Size);
4215 if (MS.TrackOrigins) {
4216 Value *ArgOriginBase = getOriginPtrForArgument(IRB, ArgOffset);
4217 // FIXME: OriginSize should be:
4218 // alignTo(A % kMinOriginAlignment + Size, kMinOriginAlignment)
4219 unsigned OriginSize = alignTo(Size, kMinOriginAlignment);
4220 IRB.CreateMemCpy(
4221 ArgOriginBase,
4222 /* by origin_tls[ArgOffset] */ kMinOriginAlignment,
4223 AOriginPtr,
4224 /* by getShadowOriginPtr */ kMinOriginAlignment, OriginSize);
4225 }
4226 }
4227 } else {
4228 // Any other parameters mean we need bit-grained tracking of uninit
4229 // data
4230 Size = DL.getTypeAllocSize(A->getType());
4231 if (ArgOffset + Size > kParamTLSSize)
4232 break;
4233 Store = IRB.CreateAlignedStore(ArgShadow, ArgShadowBase,
4235 Constant *Cst = dyn_cast<Constant>(ArgShadow);
4236 if (MS.TrackOrigins && !(Cst && Cst->isNullValue())) {
4237 IRB.CreateStore(getOrigin(A),
4238 getOriginPtrForArgument(IRB, ArgOffset));
4239 }
4240 }
4241 (void)Store;
4242 assert(Store != nullptr);
4243 LLVM_DEBUG(dbgs() << " Param:" << *Store << "\n");
4244 }
4245 assert(Size != 0);
4246 ArgOffset += alignTo(Size, kShadowTLSAlignment);
4247 }
4248 LLVM_DEBUG(dbgs() << " done with call args\n");
4249
4250 FunctionType *FT = CB.getFunctionType();
4251 if (FT->isVarArg()) {
4252 VAHelper->visitCallBase(CB, IRB);
4253 }
4254
4255 // Now, get the shadow for the RetVal.
4256 if (!CB.getType()->isSized())
4257 return;
4258 // Don't emit the epilogue for musttail call returns.
4259 if (isa<CallInst>(CB) && cast<CallInst>(CB).isMustTailCall())
4260 return;
4261
4262 if (MayCheckCall && CB.hasRetAttr(Attribute::NoUndef)) {
4263 setShadow(&CB, getCleanShadow(&CB));
4264 setOrigin(&CB, getCleanOrigin());
4265 return;
4266 }
4267
4268 IRBuilder<> IRBBefore(&CB);
4269 // Until we have full dynamic coverage, make sure the retval shadow is 0.
4270 Value *Base = getShadowPtrForRetval(IRBBefore);
4271 IRBBefore.CreateAlignedStore(getCleanShadow(&CB), Base,
4273 BasicBlock::iterator NextInsn;
4274 if (isa<CallInst>(CB)) {
4275 NextInsn = ++CB.getIterator();
4276 assert(NextInsn != CB.getParent()->end());
4277 } else {
4278 BasicBlock *NormalDest = cast<InvokeInst>(CB).getNormalDest();
4279 if (!NormalDest->getSinglePredecessor()) {
4280 // FIXME: this case is tricky, so we are just conservative here.
4281 // Perhaps we need to split the edge between this BB and NormalDest,
4282 // but a naive attempt to use SplitEdge leads to a crash.
4283 setShadow(&CB, getCleanShadow(&CB));
4284 setOrigin(&CB, getCleanOrigin());
4285 return;
4286 }
4287 // FIXME: NextInsn is likely in a basic block that has not been visited
4288 // yet. Anything inserted there will be instrumented by MSan later!
4289 NextInsn = NormalDest->getFirstInsertionPt();
4290 assert(NextInsn != NormalDest->end() &&
4291 "Could not find insertion point for retval shadow load");
4292 }
4293 IRBuilder<> IRBAfter(&*NextInsn);
4294 Value *RetvalShadow = IRBAfter.CreateAlignedLoad(
4295 getShadowTy(&CB), getShadowPtrForRetval(IRBAfter),
4296 kShadowTLSAlignment, "_msret");
4297 setShadow(&CB, RetvalShadow);
4298 if (MS.TrackOrigins)
4299 setOrigin(&CB, IRBAfter.CreateLoad(MS.OriginTy,
4300 getOriginPtrForRetval()));
4301 }
4302
4303 bool isAMustTailRetVal(Value *RetVal) {
4304 if (auto *I = dyn_cast<BitCastInst>(RetVal)) {
4305 RetVal = I->getOperand(0);
4306 }
4307 if (auto *I = dyn_cast<CallInst>(RetVal)) {
4308 return I->isMustTailCall();
4309 }
4310 return false;
4311 }
4312
4313 void visitReturnInst(ReturnInst &I) {
4314 IRBuilder<> IRB(&I);
4315 Value *RetVal = I.getReturnValue();
4316 if (!RetVal)
4317 return;
4318 // Don't emit the epilogue for musttail call returns.
4319 if (isAMustTailRetVal(RetVal))
4320 return;
4321 Value *ShadowPtr = getShadowPtrForRetval(IRB);
4322 bool HasNoUndef = F.hasRetAttribute(Attribute::NoUndef);
4323 bool StoreShadow = !(MS.EagerChecks && HasNoUndef);
4324 // FIXME: Consider using SpecialCaseList to specify a list of functions that
4325 // must always return fully initialized values. For now, we hardcode "main".
4326 bool EagerCheck = (MS.EagerChecks && HasNoUndef) || (F.getName() == "main");
4327
4328 Value *Shadow = getShadow(RetVal);
4329 bool StoreOrigin = true;
4330 if (EagerCheck) {
4331 insertShadowCheck(RetVal, &I);
4332 Shadow = getCleanShadow(RetVal);
4333 StoreOrigin = false;
4334 }
4335
4336 // The caller may still expect information passed over TLS if we pass our
4337 // check
4338 if (StoreShadow) {
4339 IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment);
4340 if (MS.TrackOrigins && StoreOrigin)
4341 IRB.CreateStore(getOrigin(RetVal), getOriginPtrForRetval());
4342 }
4343 }
4344
4345 void visitPHINode(PHINode &I) {
4346 IRBuilder<> IRB(&I);
4347 if (!PropagateShadow) {
4348 setShadow(&I, getCleanShadow(&I));
4349 setOrigin(&I, getCleanOrigin());
4350 return;
4351 }
4352
4353 ShadowPHINodes.push_back(&I);
4354 setShadow(&I, IRB.CreatePHI(getShadowTy(&I), I.getNumIncomingValues(),
4355 "_msphi_s"));
4356 if (MS.TrackOrigins)
4357 setOrigin(
4358 &I, IRB.CreatePHI(MS.OriginTy, I.getNumIncomingValues(), "_msphi_o"));
4359 }
4360
4361 Value *getLocalVarIdptr(AllocaInst &I) {
4362 ConstantInt *IntConst =
4363 ConstantInt::get(Type::getInt32Ty((*F.getParent()).getContext()), 0);
4364 return new GlobalVariable(*F.getParent(), IntConst->getType(),
4365 /*isConstant=*/false, GlobalValue::PrivateLinkage,
4366 IntConst);
4367 }
4368
4369 Value *getLocalVarDescription(AllocaInst &I) {
4370 return createPrivateConstGlobalForString(*F.getParent(), I.getName());
4371 }
4372
4373 void poisonAllocaUserspace(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
4374 if (PoisonStack && ClPoisonStackWithCall) {
4375 IRB.CreateCall(MS.MsanPoisonStackFn,
4376 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len});
4377 } else {
4378 Value *ShadowBase, *OriginBase;
4379 std::tie(ShadowBase, OriginBase) = getShadowOriginPtr(
4380 &I, IRB, IRB.getInt8Ty(), Align(1), /*isStore*/ true);
4381
4382 Value *PoisonValue = IRB.getInt8(PoisonStack ? ClPoisonStackPattern : 0);
4383 IRB.CreateMemSet(ShadowBase, PoisonValue, Len, I.getAlign());
4384 }
4385
4386 if (PoisonStack && MS.TrackOrigins) {
4387 Value *Idptr = getLocalVarIdptr(I);
4388 if (ClPrintStackNames) {
4389 Value *Descr = getLocalVarDescription(I);
4390 IRB.CreateCall(MS.MsanSetAllocaOriginWithDescriptionFn,
4391 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len,
4392 IRB.CreatePointerCast(Idptr, IRB.getInt8PtrTy()),
4393 IRB.CreatePointerCast(Descr, IRB.getInt8PtrTy())});
4394 } else {
4395 IRB.CreateCall(MS.MsanSetAllocaOriginNoDescriptionFn,
4396 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len,
4397 IRB.CreatePointerCast(Idptr, IRB.getInt8PtrTy())});
4398 }
4399 }
4400 }
4401
4402 void poisonAllocaKmsan(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
4403 Value *Descr = getLocalVarDescription(I);
4404 if (PoisonStack) {
4405 IRB.CreateCall(MS.MsanPoisonAllocaFn,
4406 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len,
4407 IRB.CreatePointerCast(Descr, IRB.getInt8PtrTy())});
4408 } else {
4409 IRB.CreateCall(MS.MsanUnpoisonAllocaFn,
4410 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len});
4411 }
4412 }
4413
4414 void instrumentAlloca(AllocaInst &I, Instruction *InsPoint = nullptr) {
4415 if (!InsPoint)
4416 InsPoint = &I;
4417 NextNodeIRBuilder IRB(InsPoint);
4418 const DataLayout &DL = F.getParent()->getDataLayout();
4419 uint64_t TypeSize = DL.getTypeAllocSize(I.getAllocatedType());
4420 Value *Len = ConstantInt::get(MS.IntptrTy, TypeSize);
4421 if (I.isArrayAllocation())
4422 Len = IRB.CreateMul(Len,
4423 IRB.CreateZExtOrTrunc(I.getArraySize(), MS.IntptrTy));
4424
4425 if (MS.CompileKernel)
4426 poisonAllocaKmsan(I, IRB, Len);
4427 else
4428 poisonAllocaUserspace(I, IRB, Len);
4429 }
4430
4431 void visitAllocaInst(AllocaInst &I) {
4432 setShadow(&I, getCleanShadow(&I));
4433 setOrigin(&I, getCleanOrigin());
4434 // We'll get to this alloca later unless it's poisoned at the corresponding
4435 // llvm.lifetime.start.
4436 AllocaSet.insert(&I);
4437 }
4438
4439 void visitSelectInst(SelectInst &I) {
4440 IRBuilder<> IRB(&I);
4441 // a = select b, c, d
4442 Value *B = I.getCondition();
4443 Value *C = I.getTrueValue();
4444 Value *D = I.getFalseValue();
4445 Value *Sb = getShadow(B);
4446 Value *Sc = getShadow(C);
4447 Value *Sd = getShadow(D);
4448
4449 // Result shadow if condition shadow is 0.
4450 Value *Sa0 = IRB.CreateSelect(B, Sc, Sd);
4451 Value *Sa1;
4452 if (I.getType()->isAggregateType()) {
4453 // To avoid "sign extending" i1 to an arbitrary aggregate type, we just do
4454 // an extra "select". This results in much more compact IR.
4455 // Sa = select Sb, poisoned, (select b, Sc, Sd)
4456 Sa1 = getPoisonedShadow(getShadowTy(I.getType()));
4457 } else {
4458 // Sa = select Sb, [ (c^d) | Sc | Sd ], [ b ? Sc : Sd ]
4459 // If Sb (condition is poisoned), look for bits in c and d that are equal
4460 // and both unpoisoned.
4461 // If !Sb (condition is unpoisoned), simply pick one of Sc and Sd.
4462
4463 // Cast arguments to shadow-compatible type.
4464 C = CreateAppToShadowCast(IRB, C);
4465 D = CreateAppToShadowCast(IRB, D);
4466
4467 // Result shadow if condition shadow is 1.
4468 Sa1 = IRB.CreateOr({IRB.CreateXor(C, D), Sc, Sd});
4469 }
4470 Value *Sa = IRB.CreateSelect(Sb, Sa1, Sa0, "_msprop_select");
4471 setShadow(&I, Sa);
4472 if (MS.TrackOrigins) {
4473 // Origins are always i32, so any vector conditions must be flattened.
4474 // FIXME: consider tracking vector origins for app vectors?
4475 if (B->getType()->isVectorTy()) {
4476 B = convertToBool(B, IRB);
4477 Sb = convertToBool(Sb, IRB);
4478 }
4479 // a = select b, c, d
4480 // Oa = Sb ? Ob : (b ? Oc : Od)
4481 setOrigin(
4482 &I, IRB.CreateSelect(Sb, getOrigin(I.getCondition()),
4483 IRB.CreateSelect(B, getOrigin(I.getTrueValue()),
4484 getOrigin(I.getFalseValue()))));
4485 }
4486 }
4487
4488 void visitLandingPadInst(LandingPadInst &I) {
4489 // Do nothing.
4490 // See https://github.com/google/sanitizers/issues/504
4491 setShadow(&I, getCleanShadow(&I));
4492 setOrigin(&I, getCleanOrigin());
4493 }
4494
4495 void visitCatchSwitchInst(CatchSwitchInst &I) {
4496 setShadow(&I, getCleanShadow(&I));
4497 setOrigin(&I, getCleanOrigin());
4498 }
4499
4500 void visitFuncletPadInst(FuncletPadInst &I) {
4501 setShadow(&I, getCleanShadow(&I));
4502 setOrigin(&I, getCleanOrigin());
4503 }
4504
4505 void visitGetElementPtrInst(GetElementPtrInst &I) { handleShadowOr(I); }
4506
4507 void visitExtractValueInst(ExtractValueInst &I) {
4508 IRBuilder<> IRB(&I);
4509 Value *Agg = I.getAggregateOperand();
4510 LLVM_DEBUG(dbgs() << "ExtractValue: " << I << "\n");
4511 Value *AggShadow = getShadow(Agg);
4512 LLVM_DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
4513 Value *ResShadow = IRB.CreateExtractValue(AggShadow, I.getIndices());
4514 LLVM_DEBUG(dbgs() << " ResShadow: " << *ResShadow << "\n");
4515 setShadow(&I, ResShadow);
4516 setOriginForNaryOp(I);
4517 }
4518
4519 void visitInsertValueInst(InsertValueInst &I) {
4520 IRBuilder<> IRB(&I);
4521 LLVM_DEBUG(dbgs() << "InsertValue: " << I << "\n");
4522 Value *AggShadow = getShadow(I.getAggregateOperand());
4523 Value *InsShadow = getShadow(I.getInsertedValueOperand());
4524 LLVM_DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
4525 LLVM_DEBUG(dbgs() << " InsShadow: " << *InsShadow << "\n");
4526 Value *Res = IRB.CreateInsertValue(AggShadow, InsShadow, I.getIndices());
4527 LLVM_DEBUG(dbgs() << " Res: " << *Res << "\n");
4528 setShadow(&I, Res);
4529 setOriginForNaryOp(I);
4530 }
4531
4532 void dumpInst(Instruction &I) {
4533 if (CallInst *CI = dyn_cast<CallInst>(&I)) {
4534 errs() << "ZZZ call " << CI->getCalledFunction()->getName() << "\n";
4535 } else {
4536 errs() << "ZZZ " << I.getOpcodeName() << "\n";
4537 }
4538 errs() << "QQQ " << I << "\n";
4539 }
4540
4541 void visitResumeInst(ResumeInst &I) {
4542 LLVM_DEBUG(dbgs() << "Resume: " << I << "\n");
4543 // Nothing to do here.
4544 }
4545
4546 void visitCleanupReturnInst(CleanupReturnInst &CRI) {
4547 LLVM_DEBUG(dbgs() << "CleanupReturn: " << CRI << "\n");
4548 // Nothing to do here.
4549 }
4550
4551 void visitCatchReturnInst(CatchReturnInst &CRI) {
4552 LLVM_DEBUG(dbgs() << "CatchReturn: " << CRI << "\n");
4553 // Nothing to do here.
4554 }
4555
4556 void instrumentAsmArgument(Value *Operand, Type *ElemTy, Instruction &I,
4557 IRBuilder<> &IRB, const DataLayout &DL,
4558 bool isOutput) {
4559 // For each assembly argument, we check its value for being initialized.
4560 // If the argument is a pointer, we assume it points to a single element
4561 // of the corresponding type (or to a 8-byte word, if the type is unsized).
4562 // Each such pointer is instrumented with a call to the runtime library.
4563 Type *OpType = Operand->getType();
4564 // Check the operand value itself.
4565 insertShadowCheck(Operand, &I);
4566 if (!OpType->isPointerTy() || !isOutput) {
4567 assert(!isOutput);
4568 return;
4569 }
4570 if (!ElemTy->isSized())
4571 return;
4572 Value *Ptr = IRB.CreatePointerCast(Operand, IRB.getInt8PtrTy());
4573 Value *SizeVal =
4574 IRB.CreateTypeSize(MS.IntptrTy, DL.getTypeStoreSize(ElemTy));
4575 IRB.CreateCall(MS.MsanInstrumentAsmStoreFn, {Ptr, SizeVal});
4576 }
4577
4578 /// Get the number of output arguments returned by pointers.
4579 int getNumOutputArgs(InlineAsm *IA, CallBase *CB) {
4580 int NumRetOutputs = 0;
4581 int NumOutputs = 0;
4582 Type *RetTy = cast<Value>(CB)->getType();
4583 if (!RetTy->isVoidTy()) {
4584 // Register outputs are returned via the CallInst return value.
4585 auto *ST = dyn_cast<StructType>(RetTy);
4586 if (ST)
4587 NumRetOutputs = ST->getNumElements();
4588 else
4589 NumRetOutputs = 1;
4590 }
4591 InlineAsm::ConstraintInfoVector Constraints = IA->ParseConstraints();
4592 for (const InlineAsm::ConstraintInfo &Info : Constraints) {
4593 switch (Info.Type) {
4595 NumOutputs++;
4596 break;
4597 default:
4598 break;
4599 }
4600 }
4601 return NumOutputs - NumRetOutputs;
4602 }
4603
4604 void visitAsmInstruction(Instruction &I) {
4605 // Conservative inline assembly handling: check for poisoned shadow of
4606 // asm() arguments, then unpoison the result and all the memory locations
4607 // pointed to by those arguments.
4608 // An inline asm() statement in C++ contains lists of input and output
4609 // arguments used by the assembly code. These are mapped to operands of the
4610 // CallInst as follows:
4611 // - nR register outputs ("=r) are returned by value in a single structure
4612 // (SSA value of the CallInst);
4613 // - nO other outputs ("=m" and others) are returned by pointer as first
4614 // nO operands of the CallInst;
4615 // - nI inputs ("r", "m" and others) are passed to CallInst as the
4616 // remaining nI operands.
4617 // The total number of asm() arguments in the source is nR+nO+nI, and the
4618 // corresponding CallInst has nO+nI+1 operands (the last operand is the
4619 // function to be called).
4620 const DataLayout &DL = F.getParent()->getDataLayout();
4621 CallBase *CB = cast<CallBase>(&I);
4622 IRBuilder<> IRB(&I);
4623 InlineAsm *IA = cast<InlineAsm>(CB->getCalledOperand());
4624 int OutputArgs = getNumOutputArgs(IA, CB);
4625 // The last operand of a CallInst is the function itself.
4626 int NumOperands = CB->getNumOperands() - 1;
4627
4628 // Check input arguments. Doing so before unpoisoning output arguments, so
4629 // that we won't overwrite uninit values before checking them.
4630 for (int i = OutputArgs; i < NumOperands; i++) {
4631 Value *Operand = CB->getOperand(i);
4632 instrumentAsmArgument(Operand, CB->getParamElementType(i), I, IRB, DL,
4633 /*isOutput*/ false);
4634 }
4635 // Unpoison output arguments. This must happen before the actual InlineAsm
4636 // call, so that the shadow for memory published in the asm() statement
4637 // remains valid.
4638 for (int i = 0; i < OutputArgs; i++) {
4639 Value *Operand = CB->getOperand(i);
4640 instrumentAsmArgument(Operand, CB->getParamElementType(i), I, IRB, DL,
4641 /*isOutput*/ true);
4642 }
4643
4644 setShadow(&I, getCleanShadow(&I));
4645 setOrigin(&I, getCleanOrigin());
4646 }
4647
4648 void visitFreezeInst(FreezeInst &I) {
4649 // Freeze always returns a fully defined value.
4650 setShadow(&I, getCleanShadow(&I));
4651 setOrigin(&I, getCleanOrigin());
4652 }
4653
4654 void visitInstruction(Instruction &I) {
4655 // Everything else: stop propagating and check for poisoned shadow.
4657 dumpInst(I);
4658 LLVM_DEBUG(dbgs() << "DEFAULT: " << I << "\n");
4659 for (size_t i = 0, n = I.getNumOperands(); i < n; i++) {
4660 Value *Operand = I.getOperand(i);
4661 if (Operand->getType()->isSized())
4662 insertShadowCheck(Operand, &I);
4663 }
4664 setShadow(&I, getCleanShadow(&I));
4665 setOrigin(&I, getCleanOrigin());
4666 }
4667};
4668
4669/// AMD64-specific implementation of VarArgHelper.
4670struct VarArgAMD64Helper : public VarArgHelper {
4671 // An unfortunate workaround for asymmetric lowering of va_arg stuff.
4672 // See a comment in visitCallBase for more details.
4673 static const unsigned AMD64GpEndOffset = 48; // AMD64 ABI Draft 0.99.6 p3.5.7
4674 static const unsigned AMD64FpEndOffsetSSE = 176;
4675 // If SSE is disabled, fp_offset in va_list is zero.
4676 static const unsigned AMD64FpEndOffsetNoSSE = AMD64GpEndOffset;
4677
4678 unsigned AMD64FpEndOffset;
4679 Function &F;
4680 MemorySanitizer &MS;
4681 MemorySanitizerVisitor &MSV;
4682 AllocaInst *VAArgTLSCopy = nullptr;
4683 AllocaInst *VAArgTLSOriginCopy = nullptr;
4684 Value *VAArgOverflowSize = nullptr;
4685
4686 SmallVector<CallInst *, 16> VAStartInstrumentationList;
4687
4688 enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory };
4689
4690 VarArgAMD64Helper(Function &F, MemorySanitizer &MS,
4691 MemorySanitizerVisitor &MSV)
4692 : F(F), MS(MS), MSV(MSV) {
4693 AMD64FpEndOffset = AMD64FpEndOffsetSSE;
4694 for (const auto &Attr : F.getAttributes().getFnAttrs()) {
4695 if (Attr.isStringAttribute() &&
4696 (Attr.getKindAsString() == "target-features")) {
4697 if (Attr.getValueAsString().contains("-sse"))
4698 AMD64FpEndOffset = AMD64FpEndOffsetNoSSE;
4699 break;
4700 }
4701 }
4702 }
4703
4704 ArgKind classifyArgument(Value *arg) {
4705 // A very rough approximation of X86_64 argument classification rules.
4706 Type *T = arg->getType();
4707 if (T->isFPOrFPVectorTy() || T->isX86_MMXTy())
4708 return AK_FloatingPoint;
4709 if (T->isIntegerTy() && T->getPrimitiveSizeInBits() <= 64)
4710 return AK_GeneralPurpose;
4711 if (T->isPointerTy())
4712 return AK_GeneralPurpose;
4713 return AK_Memory;
4714 }
4715
4716 // For VarArg functions, store the argument shadow in an ABI-specific format
4717 // that corresponds to va_list layout.
4718 // We do this because Clang lowers va_arg in the frontend, and this pass
4719 // only sees the low level code that deals with va_list internals.
4720 // A much easier alternative (provided that Clang emits va_arg instructions)
4721 // would have been to associate each live instance of va_list with a copy of
4722 // MSanParamTLS, and extract shadow on va_arg() call in the argument list
4723 // order.
4724 void visitCallBase(CallBase &CB, IRBuilder<> &IRB) override {
4725 unsigned GpOffset = 0;
4726 unsigned FpOffset = AMD64GpEndOffset;
4727 unsigned OverflowOffset = AMD64FpEndOffset;
4728 const DataLayout &DL = F.getParent()->getDataLayout();
4729 for (const auto &[ArgNo, A] : llvm::enumerate(CB.args())) {
4730 bool IsFixed = ArgNo < CB.getFunctionType()->getNumParams();
4731 bool IsByVal = CB.paramHasAttr(ArgNo, Attribute::ByVal);
4732 if (IsByVal) {
4733 // ByVal arguments always go to the overflow area.
4734 // Fixed arguments passed through the overflow area will be stepped
4735 // over by va_start, so don't count them towards the offset.
4736 if (IsFixed)
4737 continue;
4738 assert(A->getType()->isPointerTy());
4739 Type *RealTy = CB.getParamByValType(ArgNo);
4740 uint64_t ArgSize = DL.getTypeAllocSize(RealTy);
4741 Value *ShadowBase = getShadowPtrForVAArgument(
4742 RealTy, IRB, OverflowOffset, alignTo(ArgSize, 8));
4743 Value *OriginBase = nullptr;
4744 if (MS.TrackOrigins)
4745 OriginBase = getOriginPtrForVAArgument(RealTy, IRB, OverflowOffset);
4746 OverflowOffset += alignTo(ArgSize, 8);
4747 if (!ShadowBase)
4748 continue;
4749 Value *ShadowPtr, *OriginPtr;
4750 std::tie(ShadowPtr, OriginPtr) =
4751 MSV.getShadowOriginPtr(A, IRB, IRB.getInt8Ty(), kShadowTLSAlignment,
4752 /*isStore*/ false);
4753
4754 IRB.CreateMemCpy(ShadowBase, kShadowTLSAlignment, ShadowPtr,
4755 kShadowTLSAlignment, ArgSize);
4756 if (MS.TrackOrigins)
4757 IRB.CreateMemCpy(OriginBase, kShadowTLSAlignment, OriginPtr,
4758 kShadowTLSAlignment, ArgSize);
4759 } else {
4760 ArgKind AK = classifyArgument(A);
4761 if (AK == AK_GeneralPurpose && GpOffset >= AMD64GpEndOffset)
4762 AK = AK_Memory;
4763 if (AK == AK_FloatingPoint && FpOffset >= AMD64FpEndOffset)
4764 AK = AK_Memory;
4765 Value *ShadowBase, *OriginBase = nullptr;
4766 switch (AK) {
4767 case AK_GeneralPurpose:
4768 ShadowBase =
4769 getShadowPtrForVAArgument(A->getType(), IRB, GpOffset, 8);
4770 if (MS.TrackOrigins)
4771 OriginBase = getOriginPtrForVAArgument(A->getType(), IRB, GpOffset);
4772 GpOffset += 8;
4773 break;
4774 case AK_FloatingPoint:
4775 ShadowBase =
4776 getShadowPtrForVAArgument(A->getType(), IRB, FpOffset, 16);
4777 if (MS.TrackOrigins)
4778 OriginBase = getOriginPtrForVAArgument(A->getType(), IRB, FpOffset);
4779 FpOffset += 16;
4780 break;
4781 case AK_Memory:
4782 if (IsFixed)
4783 continue;
4784 uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
4785 ShadowBase =
4786 getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset, 8);
4787 if (MS.TrackOrigins)
4788 OriginBase =
4789 getOriginPtrForVAArgument(A->getType(), IRB, OverflowOffset);
4790 OverflowOffset += alignTo(ArgSize, 8);
4791 }
4792 // Take fixed arguments into account for GpOffset and FpOffset,
4793 // but don't actually store shadows for them.
4794 // TODO(glider): don't call get*PtrForVAArgument() for them.
4795 if (IsFixed)
4796 continue;
4797 if (!ShadowBase)
4798 continue;
4799 Value *Shadow = MSV.getShadow(A);
4800 IRB.CreateAlignedStore(Shadow, ShadowBase, kShadowTLSAlignment);
4801 if (MS.TrackOrigins) {
4802 Value *Origin = MSV.getOrigin(A);
4803 TypeSize StoreSize = DL.getTypeStoreSize(Shadow->getType());
4804 MSV.paintOrigin(IRB, Origin, OriginBase, StoreSize,
4806 }
4807 }
4808 }
4809 Constant *OverflowSize =
4810 ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AMD64FpEndOffset);
4811 IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
4812 }
4813
4814 /// Compute the shadow address for a given va_arg.
4815 Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4816 unsigned ArgOffset, unsigned ArgSize) {
4817 // Make sure we don't overflow __msan_va_arg_tls.
4818 if (ArgOffset + ArgSize > kParamTLSSize)
4819 return nullptr;
4820 Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4821 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4822 return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4823 "_msarg_va_s");
4824 }
4825
4826 /// Compute the origin address for a given va_arg.
4827 Value *getOriginPtrForVAArgument(Type *Ty, IRBuilder<> &IRB, int ArgOffset) {
4828 Value *Base = IRB.CreatePointerCast(MS.VAArgOriginTLS, MS.IntptrTy);
4829 // getOriginPtrForVAArgument() is always called after
4830 // getShadowPtrForVAArgument(), so __msan_va_arg_origin_tls can never
4831 // overflow.
4832 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4833 return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0),
4834 "_msarg_va_o");
4835 }
4836
4837 void unpoisonVAListTagForInst(IntrinsicInst &I) {
4838 IRBuilder<> IRB(&I);
4839 Value *VAListTag = I.getArgOperand(0);
4840 Value *ShadowPtr, *OriginPtr;
4841 const Align Alignment = Align(8);
4842 std::tie(ShadowPtr, OriginPtr) =
4843 MSV.getShadowOriginPtr(VAListTag, IRB, IRB.getInt8Ty(), Alignment,
4844 /*isStore*/ true);
4845
4846 // Unpoison the whole __va_list_tag.
4847 // FIXME: magic ABI constants.
4848 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4849 /* size */ 24, Alignment, false);
4850 // We shouldn't need to zero out the origins, as they're only checked for
4851 // nonzero shadow.
4852 }
4853
4854 void visitVAStartInst(VAStartInst &I) override {
4855 if (F.getCallingConv() == CallingConv::Win64)
4856 return;
4857 VAStartInstrumentationList.push_back(&I);
4858 unpoisonVAListTagForInst(I);
4859 }
4860
4861 void visitVACopyInst(VACopyInst &I) override {
4862 if (F.getCallingConv() == CallingConv::Win64)
4863 return;
4864 unpoisonVAListTagForInst(I);
4865 }
4866
4867 void finalizeInstrumentation() override {
4868 assert(!VAArgOverflowSize && !VAArgTLSCopy &&
4869 "finalizeInstrumentation called twice");
4870 if (!VAStartInstrumentationList.empty()) {
4871 // If there is a va_start in this function, make a backup copy of
4872 // va_arg_tls somewhere in the function entry block.
4873 IRBuilder<> IRB(MSV.FnPrologueEnd);
4874 VAArgOverflowSize =
4875 IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
4876 Value *CopySize = IRB.CreateAdd(
4877 ConstantInt::get(MS.IntptrTy, AMD64FpEndOffset), VAArgOverflowSize);
4878 VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4879 VAArgTLSCopy->setAlignment(kShadowTLSAlignment);
4880 IRB.CreateMemSet(VAArgTLSCopy, Constant::getNullValue(IRB.getInt8Ty()),
4881 CopySize, kShadowTLSAlignment, false);
4882
4883 Value *SrcSize = IRB.CreateBinaryIntrinsic(
4884 Intrinsic::umin, CopySize,
4885 ConstantInt::get(MS.IntptrTy, kParamTLSSize));
4886 IRB.CreateMemCpy(VAArgTLSCopy, kShadowTLSAlignment, MS.VAArgTLS,
4887 kShadowTLSAlignment, SrcSize);
4888 if (MS.TrackOrigins) {
4889 VAArgTLSOriginCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4890 VAArgTLSOriginCopy->setAlignment(kShadowTLSAlignment);
4891 IRB.CreateMemCpy(VAArgTLSOriginCopy, kShadowTLSAlignment,
4892 MS.VAArgOriginTLS, kShadowTLSAlignment, SrcSize);
4893 }
4894 }
4895
4896 // Instrument va_start.
4897 // Copy va_list shadow from the backup copy of the TLS contents.
4898 for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
4899 CallInst *OrigInst = VAStartInstrumentationList[i];
4900 NextNodeIRBuilder IRB(OrigInst);
4901 Value *VAListTag = OrigInst->getArgOperand(0);
4902
4903 Type *RegSaveAreaPtrTy = PointerType::getUnqual(*MS.C); // i64*
4904 Value *RegSaveAreaPtrPtr = IRB.CreateIntToPtr(
4905 IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4906 ConstantInt::get(MS.IntptrTy, 16)),
4907 PointerType::get(RegSaveAreaPtrTy, 0));
4908 Value *RegSaveAreaPtr =
4909 IRB.CreateLoad(RegSaveAreaPtrTy, RegSaveAreaPtrPtr);
4910 Value *RegSaveAreaShadowPtr, *RegSaveAreaOriginPtr;
4911 const Align Alignment = Align(16);
4912 std::tie(RegSaveAreaShadowPtr, RegSaveAreaOriginPtr) =
4913 MSV.getShadowOriginPtr(RegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4914 Alignment, /*isStore*/ true);
4915 IRB.CreateMemCpy(RegSaveAreaShadowPtr, Alignment, VAArgTLSCopy, Alignment,
4916 AMD64FpEndOffset);
4917 if (MS.TrackOrigins)
4918 IRB.CreateMemCpy(RegSaveAreaOriginPtr, Alignment, VAArgTLSOriginCopy,
4919 Alignment, AMD64FpEndOffset);
4920 Type *OverflowArgAreaPtrTy = PointerType::getUnqual(*MS.C); // i64*
4921 Value *OverflowArgAreaPtrPtr = IRB.CreateIntToPtr(
4922 IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4923 ConstantInt::get(MS.IntptrTy, 8)),
4924 PointerType::get(OverflowArgAreaPtrTy, 0));
4925 Value *OverflowArgAreaPtr =
4926 IRB.CreateLoad(OverflowArgAreaPtrTy, OverflowArgAreaPtrPtr);
4927 Value *OverflowArgAreaShadowPtr, *OverflowArgAreaOriginPtr;
4928 std::tie(OverflowArgAreaShadowPtr, OverflowArgAreaOriginPtr) =
4929 MSV.getShadowOriginPtr(OverflowArgAreaPtr, IRB, IRB.getInt8Ty(),
4930 Alignment, /*isStore*/ true);
4931 Value *SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSCopy,
4932 AMD64FpEndOffset);
4933 IRB.CreateMemCpy(OverflowArgAreaShadowPtr, Alignment, SrcPtr, Alignment,
4934 VAArgOverflowSize);
4935 if (MS.TrackOrigins) {