LLVM 17.0.0git
MemorySanitizer.cpp
Go to the documentation of this file.
1//===- MemorySanitizer.cpp - detector of uninitialized reads --------------===//
2//
3// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
4// See https://llvm.org/LICENSE.txt for license information.
5// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
6//
7//===----------------------------------------------------------------------===//
8//
9/// \file
10/// This file is a part of MemorySanitizer, a detector of uninitialized
11/// reads.
12///
13/// The algorithm of the tool is similar to Memcheck
14/// (http://goo.gl/QKbem). We associate a few shadow bits with every
15/// byte of the application memory, poison the shadow of the malloc-ed
16/// or alloca-ed memory, load the shadow bits on every memory read,
17/// propagate the shadow bits through some of the arithmetic
18/// instruction (including MOV), store the shadow bits on every memory
19/// write, report a bug on some other instructions (e.g. JMP) if the
20/// associated shadow is poisoned.
21///
22/// But there are differences too. The first and the major one:
23/// compiler instrumentation instead of binary instrumentation. This
24/// gives us much better register allocation, possible compiler
25/// optimizations and a fast start-up. But this brings the major issue
26/// as well: msan needs to see all program events, including system
27/// calls and reads/writes in system libraries, so we either need to
28/// compile *everything* with msan or use a binary translation
29/// component (e.g. DynamoRIO) to instrument pre-built libraries.
30/// Another difference from Memcheck is that we use 8 shadow bits per
31/// byte of application memory and use a direct shadow mapping. This
32/// greatly simplifies the instrumentation code and avoids races on
33/// shadow updates (Memcheck is single-threaded so races are not a
34/// concern there. Memcheck uses 2 shadow bits per byte with a slow
35/// path storage that uses 8 bits per byte).
36///
37/// The default value of shadow is 0, which means "clean" (not poisoned).
38///
39/// Every module initializer should call __msan_init to ensure that the
40/// shadow memory is ready. On error, __msan_warning is called. Since
41/// parameters and return values may be passed via registers, we have a
42/// specialized thread-local shadow for return values
43/// (__msan_retval_tls) and parameters (__msan_param_tls).
44///
45/// Origin tracking.
46///
47/// MemorySanitizer can track origins (allocation points) of all uninitialized
48/// values. This behavior is controlled with a flag (msan-track-origins) and is
49/// disabled by default.
50///
51/// Origins are 4-byte values created and interpreted by the runtime library.
52/// They are stored in a second shadow mapping, one 4-byte value for 4 bytes
53/// of application memory. Propagation of origins is basically a bunch of
54/// "select" instructions that pick the origin of a dirty argument, if an
55/// instruction has one.
56///
57/// Every 4 aligned, consecutive bytes of application memory have one origin
58/// value associated with them. If these bytes contain uninitialized data
59/// coming from 2 different allocations, the last store wins. Because of this,
60/// MemorySanitizer reports can show unrelated origins, but this is unlikely in
61/// practice.
62///
63/// Origins are meaningless for fully initialized values, so MemorySanitizer
64/// avoids storing origin to memory when a fully initialized value is stored.
65/// This way it avoids needless overwriting origin of the 4-byte region on
66/// a short (i.e. 1 byte) clean store, and it is also good for performance.
67///
68/// Atomic handling.
69///
70/// Ideally, every atomic store of application value should update the
71/// corresponding shadow location in an atomic way. Unfortunately, atomic store
72/// of two disjoint locations can not be done without severe slowdown.
73///
74/// Therefore, we implement an approximation that may err on the safe side.
75/// In this implementation, every atomically accessed location in the program
76/// may only change from (partially) uninitialized to fully initialized, but
77/// not the other way around. We load the shadow _after_ the application load,
78/// and we store the shadow _before_ the app store. Also, we always store clean
79/// shadow (if the application store is atomic). This way, if the store-load
80/// pair constitutes a happens-before arc, shadow store and load are correctly
81/// ordered such that the load will get either the value that was stored, or
82/// some later value (which is always clean).
83///
84/// This does not work very well with Compare-And-Swap (CAS) and
85/// Read-Modify-Write (RMW) operations. To follow the above logic, CAS and RMW
86/// must store the new shadow before the app operation, and load the shadow
87/// after the app operation. Computers don't work this way. Current
88/// implementation ignores the load aspect of CAS/RMW, always returning a clean
89/// value. It implements the store part as a simple atomic store by storing a
90/// clean shadow.
91///
92/// Instrumenting inline assembly.
93///
94/// For inline assembly code LLVM has little idea about which memory locations
95/// become initialized depending on the arguments. It can be possible to figure
96/// out which arguments are meant to point to inputs and outputs, but the
97/// actual semantics can be only visible at runtime. In the Linux kernel it's
98/// also possible that the arguments only indicate the offset for a base taken
99/// from a segment register, so it's dangerous to treat any asm() arguments as
100/// pointers. We take a conservative approach generating calls to
101/// __msan_instrument_asm_store(ptr, size)
102/// , which defer the memory unpoisoning to the runtime library.
103/// The latter can perform more complex address checks to figure out whether
104/// it's safe to touch the shadow memory.
105/// Like with atomic operations, we call __msan_instrument_asm_store() before
106/// the assembly call, so that changes to the shadow memory will be seen by
107/// other threads together with main memory initialization.
108///
109/// KernelMemorySanitizer (KMSAN) implementation.
110///
111/// The major differences between KMSAN and MSan instrumentation are:
112/// - KMSAN always tracks the origins and implies msan-keep-going=true;
113/// - KMSAN allocates shadow and origin memory for each page separately, so
114/// there are no explicit accesses to shadow and origin in the
115/// instrumentation.
116/// Shadow and origin values for a particular X-byte memory location
117/// (X=1,2,4,8) are accessed through pointers obtained via the
118/// __msan_metadata_ptr_for_load_X(ptr)
119/// __msan_metadata_ptr_for_store_X(ptr)
120/// functions. The corresponding functions check that the X-byte accesses
121/// are possible and returns the pointers to shadow and origin memory.
122/// Arbitrary sized accesses are handled with:
123/// __msan_metadata_ptr_for_load_n(ptr, size)
124/// __msan_metadata_ptr_for_store_n(ptr, size);
125/// - TLS variables are stored in a single per-task struct. A call to a
126/// function __msan_get_context_state() returning a pointer to that struct
127/// is inserted into every instrumented function before the entry block;
128/// - __msan_warning() takes a 32-bit origin parameter;
129/// - local variables are poisoned with __msan_poison_alloca() upon function
130/// entry and unpoisoned with __msan_unpoison_alloca() before leaving the
131/// function;
132/// - the pass doesn't declare any global variables or add global constructors
133/// to the translation unit.
134///
135/// Also, KMSAN currently ignores uninitialized memory passed into inline asm
136/// calls, making sure we're on the safe side wrt. possible false positives.
137///
138/// KernelMemorySanitizer only supports X86_64 at the moment.
139///
140//
141// FIXME: This sanitizer does not yet handle scalable vectors
142//
143//===----------------------------------------------------------------------===//
144
146#include "llvm/ADT/APInt.h"
147#include "llvm/ADT/ArrayRef.h"
148#include "llvm/ADT/DenseMap.h"
150#include "llvm/ADT/SetVector.h"
151#include "llvm/ADT/SmallString.h"
152#include "llvm/ADT/SmallVector.h"
154#include "llvm/ADT/StringRef.h"
158#include "llvm/IR/Argument.h"
159#include "llvm/IR/Attributes.h"
160#include "llvm/IR/BasicBlock.h"
161#include "llvm/IR/CallingConv.h"
162#include "llvm/IR/Constant.h"
163#include "llvm/IR/Constants.h"
164#include "llvm/IR/DataLayout.h"
165#include "llvm/IR/DerivedTypes.h"
166#include "llvm/IR/Function.h"
167#include "llvm/IR/GlobalValue.h"
169#include "llvm/IR/IRBuilder.h"
170#include "llvm/IR/InlineAsm.h"
171#include "llvm/IR/InstVisitor.h"
172#include "llvm/IR/InstrTypes.h"
173#include "llvm/IR/Instruction.h"
174#include "llvm/IR/Instructions.h"
176#include "llvm/IR/Intrinsics.h"
177#include "llvm/IR/IntrinsicsX86.h"
178#include "llvm/IR/MDBuilder.h"
179#include "llvm/IR/Module.h"
180#include "llvm/IR/Type.h"
181#include "llvm/IR/Value.h"
182#include "llvm/IR/ValueMap.h"
185#include "llvm/Support/Casting.h"
187#include "llvm/Support/Debug.h"
196#include <algorithm>
197#include <cassert>
198#include <cstddef>
199#include <cstdint>
200#include <memory>
201#include <string>
202#include <tuple>
203
204using namespace llvm;
205
206#define DEBUG_TYPE "msan"
207
208DEBUG_COUNTER(DebugInsertCheck, "msan-insert-check",
209 "Controls which checks to insert");
210
211static const unsigned kOriginSize = 4;
214
215// These constants must be kept in sync with the ones in msan.h.
216static const unsigned kParamTLSSize = 800;
217static const unsigned kRetvalTLSSize = 800;
218
219// Accesses sizes are powers of two: 1, 2, 4, 8.
220static const size_t kNumberOfAccessSizes = 4;
221
222/// Track origins of uninitialized values.
223///
224/// Adds a section to MemorySanitizer report that points to the allocation
225/// (stack or heap) the uninitialized bits came from originally.
227 "msan-track-origins",
228 cl::desc("Track origins (allocation sites) of poisoned memory"), cl::Hidden,
229 cl::init(0));
230
231static cl::opt<bool> ClKeepGoing("msan-keep-going",
232 cl::desc("keep going after reporting a UMR"),
233 cl::Hidden, cl::init(false));
234
235static cl::opt<bool>
236 ClPoisonStack("msan-poison-stack",
237 cl::desc("poison uninitialized stack variables"), cl::Hidden,
238 cl::init(true));
239
241 "msan-poison-stack-with-call",
242 cl::desc("poison uninitialized stack variables with a call"), cl::Hidden,
243 cl::init(false));
244
246 "msan-poison-stack-pattern",
247 cl::desc("poison uninitialized stack variables with the given pattern"),
248 cl::Hidden, cl::init(0xff));
249
250static cl::opt<bool>
251 ClPrintStackNames("msan-print-stack-names",
252 cl::desc("Print name of local stack variable"),
253 cl::Hidden, cl::init(true));
254
255static cl::opt<bool> ClPoisonUndef("msan-poison-undef",
256 cl::desc("poison undef temps"), cl::Hidden,
257 cl::init(true));
258
259static cl::opt<bool>
260 ClHandleICmp("msan-handle-icmp",
261 cl::desc("propagate shadow through ICmpEQ and ICmpNE"),
262 cl::Hidden, cl::init(true));
263
264static cl::opt<bool>
265 ClHandleICmpExact("msan-handle-icmp-exact",
266 cl::desc("exact handling of relational integer ICmp"),
267 cl::Hidden, cl::init(false));
268
270 "msan-handle-lifetime-intrinsics",
271 cl::desc(
272 "when possible, poison scoped variables at the beginning of the scope "
273 "(slower, but more precise)"),
274 cl::Hidden, cl::init(true));
275
276// When compiling the Linux kernel, we sometimes see false positives related to
277// MSan being unable to understand that inline assembly calls may initialize
278// local variables.
279// This flag makes the compiler conservatively unpoison every memory location
280// passed into an assembly call. Note that this may cause false positives.
281// Because it's impossible to figure out the array sizes, we can only unpoison
282// the first sizeof(type) bytes for each type* pointer.
283// The instrumentation is only enabled in KMSAN builds, and only if
284// -msan-handle-asm-conservative is on. This is done because we may want to
285// quickly disable assembly instrumentation when it breaks.
287 "msan-handle-asm-conservative",
288 cl::desc("conservative handling of inline assembly"), cl::Hidden,
289 cl::init(true));
290
291// This flag controls whether we check the shadow of the address
292// operand of load or store. Such bugs are very rare, since load from
293// a garbage address typically results in SEGV, but still happen
294// (e.g. only lower bits of address are garbage, or the access happens
295// early at program startup where malloc-ed memory is more likely to
296// be zeroed. As of 2012-08-28 this flag adds 20% slowdown.
298 "msan-check-access-address",
299 cl::desc("report accesses through a pointer which has poisoned shadow"),
300 cl::Hidden, cl::init(true));
301
303 "msan-eager-checks",
304 cl::desc("check arguments and return values at function call boundaries"),
305 cl::Hidden, cl::init(false));
306
308 "msan-dump-strict-instructions",
309 cl::desc("print out instructions with default strict semantics"),
310 cl::Hidden, cl::init(false));
311
313 "msan-instrumentation-with-call-threshold",
314 cl::desc(
315 "If the function being instrumented requires more than "
316 "this number of checks and origin stores, use callbacks instead of "
317 "inline checks (-1 means never use callbacks)."),
318 cl::Hidden, cl::init(3500));
319
320static cl::opt<bool>
321 ClEnableKmsan("msan-kernel",
322 cl::desc("Enable KernelMemorySanitizer instrumentation"),
323 cl::Hidden, cl::init(false));
324
325static cl::opt<bool>
326 ClDisableChecks("msan-disable-checks",
327 cl::desc("Apply no_sanitize to the whole file"), cl::Hidden,
328 cl::init(false));
329
330static cl::opt<bool>
331 ClCheckConstantShadow("msan-check-constant-shadow",
332 cl::desc("Insert checks for constant shadow values"),
333 cl::Hidden, cl::init(true));
334
335// This is off by default because of a bug in gold:
336// https://sourceware.org/bugzilla/show_bug.cgi?id=19002
337static cl::opt<bool>
338 ClWithComdat("msan-with-comdat",
339 cl::desc("Place MSan constructors in comdat sections"),
340 cl::Hidden, cl::init(false));
341
342// These options allow to specify custom memory map parameters
343// See MemoryMapParams for details.
344static cl::opt<uint64_t> ClAndMask("msan-and-mask",
345 cl::desc("Define custom MSan AndMask"),
346 cl::Hidden, cl::init(0));
347
348static cl::opt<uint64_t> ClXorMask("msan-xor-mask",
349 cl::desc("Define custom MSan XorMask"),
350 cl::Hidden, cl::init(0));
351
352static cl::opt<uint64_t> ClShadowBase("msan-shadow-base",
353 cl::desc("Define custom MSan ShadowBase"),
354 cl::Hidden, cl::init(0));
355
356static cl::opt<uint64_t> ClOriginBase("msan-origin-base",
357 cl::desc("Define custom MSan OriginBase"),
358 cl::Hidden, cl::init(0));
359
360static cl::opt<int>
361 ClDisambiguateWarning("msan-disambiguate-warning-threshold",
362 cl::desc("Define threshold for number of checks per "
363 "debug location to force origin update."),
364 cl::Hidden, cl::init(3));
365
366const char kMsanModuleCtorName[] = "msan.module_ctor";
367const char kMsanInitName[] = "__msan_init";
368
369namespace {
370
371// Memory map parameters used in application-to-shadow address calculation.
372// Offset = (Addr & ~AndMask) ^ XorMask
373// Shadow = ShadowBase + Offset
374// Origin = OriginBase + Offset
375struct MemoryMapParams {
376 uint64_t AndMask;
377 uint64_t XorMask;
378 uint64_t ShadowBase;
379 uint64_t OriginBase;
380};
381
382struct PlatformMemoryMapParams {
383 const MemoryMapParams *bits32;
384 const MemoryMapParams *bits64;
385};
386
387} // end anonymous namespace
388
389// i386 Linux
390static const MemoryMapParams Linux_I386_MemoryMapParams = {
391 0x000080000000, // AndMask
392 0, // XorMask (not used)
393 0, // ShadowBase (not used)
394 0x000040000000, // OriginBase
395};
396
397// x86_64 Linux
398static const MemoryMapParams Linux_X86_64_MemoryMapParams = {
399 0, // AndMask (not used)
400 0x500000000000, // XorMask
401 0, // ShadowBase (not used)
402 0x100000000000, // OriginBase
403};
404
405// mips64 Linux
406static const MemoryMapParams Linux_MIPS64_MemoryMapParams = {
407 0, // AndMask (not used)
408 0x008000000000, // XorMask
409 0, // ShadowBase (not used)
410 0x002000000000, // OriginBase
411};
412
413// ppc64 Linux
414static const MemoryMapParams Linux_PowerPC64_MemoryMapParams = {
415 0xE00000000000, // AndMask
416 0x100000000000, // XorMask
417 0x080000000000, // ShadowBase
418 0x1C0000000000, // OriginBase
419};
420
421// s390x Linux
422static const MemoryMapParams Linux_S390X_MemoryMapParams = {
423 0xC00000000000, // AndMask
424 0, // XorMask (not used)
425 0x080000000000, // ShadowBase
426 0x1C0000000000, // OriginBase
427};
428
429// aarch64 Linux
430static const MemoryMapParams Linux_AArch64_MemoryMapParams = {
431 0, // AndMask (not used)
432 0x0B00000000000, // XorMask
433 0, // ShadowBase (not used)
434 0x0200000000000, // OriginBase
435};
436
437// aarch64 FreeBSD
438static const MemoryMapParams FreeBSD_AArch64_MemoryMapParams = {
439 0x1800000000000, // AndMask
440 0x0400000000000, // XorMask
441 0x0200000000000, // ShadowBase
442 0x0700000000000, // OriginBase
443};
444
445// i386 FreeBSD
446static const MemoryMapParams FreeBSD_I386_MemoryMapParams = {
447 0x000180000000, // AndMask
448 0x000040000000, // XorMask
449 0x000020000000, // ShadowBase
450 0x000700000000, // OriginBase
451};
452
453// x86_64 FreeBSD
454static const MemoryMapParams FreeBSD_X86_64_MemoryMapParams = {
455 0xc00000000000, // AndMask
456 0x200000000000, // XorMask
457 0x100000000000, // ShadowBase
458 0x380000000000, // OriginBase
459};
460
461// x86_64 NetBSD
462static const MemoryMapParams NetBSD_X86_64_MemoryMapParams = {
463 0, // AndMask
464 0x500000000000, // XorMask
465 0, // ShadowBase
466 0x100000000000, // OriginBase
467};
468
469static const PlatformMemoryMapParams Linux_X86_MemoryMapParams = {
472};
473
474static const PlatformMemoryMapParams Linux_MIPS_MemoryMapParams = {
475 nullptr,
477};
478
479static const PlatformMemoryMapParams Linux_PowerPC_MemoryMapParams = {
480 nullptr,
482};
483
484static const PlatformMemoryMapParams Linux_S390_MemoryMapParams = {
485 nullptr,
487};
488
489static const PlatformMemoryMapParams Linux_ARM_MemoryMapParams = {
490 nullptr,
492};
493
494static const PlatformMemoryMapParams FreeBSD_ARM_MemoryMapParams = {
495 nullptr,
497};
498
499static const PlatformMemoryMapParams FreeBSD_X86_MemoryMapParams = {
502};
503
504static const PlatformMemoryMapParams NetBSD_X86_MemoryMapParams = {
505 nullptr,
507};
508
509namespace {
510
511/// Instrument functions of a module to detect uninitialized reads.
512///
513/// Instantiating MemorySanitizer inserts the msan runtime library API function
514/// declarations into the module if they don't exist already. Instantiating
515/// ensures the __msan_init function is in the list of global constructors for
516/// the module.
517class MemorySanitizer {
518public:
519 MemorySanitizer(Module &M, MemorySanitizerOptions Options)
520 : CompileKernel(Options.Kernel), TrackOrigins(Options.TrackOrigins),
521 Recover(Options.Recover), EagerChecks(Options.EagerChecks) {
522 initializeModule(M);
523 }
524
525 // MSan cannot be moved or copied because of MapParams.
526 MemorySanitizer(MemorySanitizer &&) = delete;
527 MemorySanitizer &operator=(MemorySanitizer &&) = delete;
528 MemorySanitizer(const MemorySanitizer &) = delete;
529 MemorySanitizer &operator=(const MemorySanitizer &) = delete;
530
531 bool sanitizeFunction(Function &F, TargetLibraryInfo &TLI);
532
533private:
534 friend struct MemorySanitizerVisitor;
535 friend struct VarArgAMD64Helper;
536 friend struct VarArgMIPS64Helper;
537 friend struct VarArgAArch64Helper;
538 friend struct VarArgPowerPC64Helper;
539 friend struct VarArgSystemZHelper;
540
541 void initializeModule(Module &M);
542 void initializeCallbacks(Module &M, const TargetLibraryInfo &TLI);
543 void createKernelApi(Module &M, const TargetLibraryInfo &TLI);
544 void createUserspaceApi(Module &M, const TargetLibraryInfo &TLI);
545
546 /// True if we're compiling the Linux kernel.
547 bool CompileKernel;
548 /// Track origins (allocation points) of uninitialized values.
549 int TrackOrigins;
550 bool Recover;
551 bool EagerChecks;
552
553 LLVMContext *C;
554 Type *IntptrTy;
555 Type *OriginTy;
556
557 // XxxTLS variables represent the per-thread state in MSan and per-task state
558 // in KMSAN.
559 // For the userspace these point to thread-local globals. In the kernel land
560 // they point to the members of a per-task struct obtained via a call to
561 // __msan_get_context_state().
562
563 /// Thread-local shadow storage for function parameters.
564 Value *ParamTLS;
565
566 /// Thread-local origin storage for function parameters.
567 Value *ParamOriginTLS;
568
569 /// Thread-local shadow storage for function return value.
570 Value *RetvalTLS;
571
572 /// Thread-local origin storage for function return value.
573 Value *RetvalOriginTLS;
574
575 /// Thread-local shadow storage for in-register va_arg function
576 /// parameters (x86_64-specific).
577 Value *VAArgTLS;
578
579 /// Thread-local shadow storage for in-register va_arg function
580 /// parameters (x86_64-specific).
581 Value *VAArgOriginTLS;
582
583 /// Thread-local shadow storage for va_arg overflow area
584 /// (x86_64-specific).
585 Value *VAArgOverflowSizeTLS;
586
587 /// Are the instrumentation callbacks set up?
588 bool CallbacksInitialized = false;
589
590 /// The run-time callback to print a warning.
591 FunctionCallee WarningFn;
592
593 // These arrays are indexed by log2(AccessSize).
594 FunctionCallee MaybeWarningFn[kNumberOfAccessSizes];
595 FunctionCallee MaybeStoreOriginFn[kNumberOfAccessSizes];
596
597 /// Run-time helper that generates a new origin value for a stack
598 /// allocation.
599 FunctionCallee MsanSetAllocaOriginWithDescriptionFn;
600 // No description version
601 FunctionCallee MsanSetAllocaOriginNoDescriptionFn;
602
603 /// Run-time helper that poisons stack on function entry.
604 FunctionCallee MsanPoisonStackFn;
605
606 /// Run-time helper that records a store (or any event) of an
607 /// uninitialized value and returns an updated origin id encoding this info.
608 FunctionCallee MsanChainOriginFn;
609
610 /// Run-time helper that paints an origin over a region.
611 FunctionCallee MsanSetOriginFn;
612
613 /// MSan runtime replacements for memmove, memcpy and memset.
614 FunctionCallee MemmoveFn, MemcpyFn, MemsetFn;
615
616 /// KMSAN callback for task-local function argument shadow.
617 StructType *MsanContextStateTy;
618 FunctionCallee MsanGetContextStateFn;
619
620 /// Functions for poisoning/unpoisoning local variables
621 FunctionCallee MsanPoisonAllocaFn, MsanUnpoisonAllocaFn;
622
623 /// Each of the MsanMetadataPtrXxx functions returns a pair of shadow/origin
624 /// pointers.
625 FunctionCallee MsanMetadataPtrForLoadN, MsanMetadataPtrForStoreN;
626 FunctionCallee MsanMetadataPtrForLoad_1_8[4];
627 FunctionCallee MsanMetadataPtrForStore_1_8[4];
628 FunctionCallee MsanInstrumentAsmStoreFn;
629
630 /// Helper to choose between different MsanMetadataPtrXxx().
631 FunctionCallee getKmsanShadowOriginAccessFn(bool isStore, int size);
632
633 /// Memory map parameters used in application-to-shadow calculation.
634 const MemoryMapParams *MapParams;
635
636 /// Custom memory map parameters used when -msan-shadow-base or
637 // -msan-origin-base is provided.
638 MemoryMapParams CustomMapParams;
639
640 MDNode *ColdCallWeights;
641
642 /// Branch weights for origin store.
643 MDNode *OriginStoreWeights;
644};
645
646void insertModuleCtor(Module &M) {
649 /*InitArgTypes=*/{},
650 /*InitArgs=*/{},
651 // This callback is invoked when the functions are created the first
652 // time. Hook them into the global ctors list in that case:
653 [&](Function *Ctor, FunctionCallee) {
654 if (!ClWithComdat) {
655 appendToGlobalCtors(M, Ctor, 0);
656 return;
657 }
658 Comdat *MsanCtorComdat = M.getOrInsertComdat(kMsanModuleCtorName);
659 Ctor->setComdat(MsanCtorComdat);
660 appendToGlobalCtors(M, Ctor, 0, Ctor);
661 });
662}
663
664template <class T> T getOptOrDefault(const cl::opt<T> &Opt, T Default) {
665 return (Opt.getNumOccurrences() > 0) ? Opt : Default;
666}
667
668} // end anonymous namespace
669
671 bool EagerChecks)
672 : Kernel(getOptOrDefault(ClEnableKmsan, K)),
673 TrackOrigins(getOptOrDefault(ClTrackOrigins, Kernel ? 2 : TO)),
674 Recover(getOptOrDefault(ClKeepGoing, Kernel || R)),
675 EagerChecks(getOptOrDefault(ClEagerChecks, EagerChecks)) {}
676
679 bool Modified = false;
680 if (!Options.Kernel) {
681 insertModuleCtor(M);
682 Modified = true;
683 }
684
685 auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
686 for (Function &F : M) {
687 if (F.empty())
688 continue;
689 MemorySanitizer Msan(*F.getParent(), Options);
690 Modified |=
691 Msan.sanitizeFunction(F, FAM.getResult<TargetLibraryAnalysis>(F));
692 }
693
694 if (!Modified)
695 return PreservedAnalyses::all();
696
698 // GlobalsAA is considered stateless and does not get invalidated unless
699 // explicitly invalidated; PreservedAnalyses::none() is not enough. Sanitizers
700 // make changes that require GlobalsAA to be invalidated.
701 PA.abandon<GlobalsAA>();
702 return PA;
703}
704
706 raw_ostream &OS, function_ref<StringRef(StringRef)> MapClassName2PassName) {
708 OS, MapClassName2PassName);
709 OS << '<';
710 if (Options.Recover)
711 OS << "recover;";
712 if (Options.Kernel)
713 OS << "kernel;";
714 if (Options.EagerChecks)
715 OS << "eager-checks;";
716 OS << "track-origins=" << Options.TrackOrigins;
717 OS << '>';
718}
719
720/// Create a non-const global initialized with the given string.
721///
722/// Creates a writable global for Str so that we can pass it to the
723/// run-time lib. Runtime uses first 4 bytes of the string to store the
724/// frame ID, so the string needs to be mutable.
726 StringRef Str) {
727 Constant *StrConst = ConstantDataArray::getString(M.getContext(), Str);
728 return new GlobalVariable(M, StrConst->getType(), /*isConstant=*/true,
729 GlobalValue::PrivateLinkage, StrConst, "");
730}
731
732/// Create KMSAN API callbacks.
733void MemorySanitizer::createKernelApi(Module &M, const TargetLibraryInfo &TLI) {
734 IRBuilder<> IRB(*C);
735
736 // These will be initialized in insertKmsanPrologue().
737 RetvalTLS = nullptr;
738 RetvalOriginTLS = nullptr;
739 ParamTLS = nullptr;
740 ParamOriginTLS = nullptr;
741 VAArgTLS = nullptr;
742 VAArgOriginTLS = nullptr;
743 VAArgOverflowSizeTLS = nullptr;
744
745 WarningFn = M.getOrInsertFunction("__msan_warning",
746 TLI.getAttrList(C, {0}, /*Signed=*/false),
747 IRB.getVoidTy(), IRB.getInt32Ty());
748
749 // Requests the per-task context state (kmsan_context_state*) from the
750 // runtime library.
751 MsanContextStateTy = StructType::get(
752 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
753 ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8),
754 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
755 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8), /* va_arg_origin */
756 IRB.getInt64Ty(), ArrayType::get(OriginTy, kParamTLSSize / 4), OriginTy,
757 OriginTy);
758 MsanGetContextStateFn = M.getOrInsertFunction(
759 "__msan_get_context_state", PointerType::get(MsanContextStateTy, 0));
760
761 Type *RetTy = StructType::get(PointerType::get(IRB.getInt8Ty(), 0),
762 PointerType::get(IRB.getInt32Ty(), 0));
763
764 for (int ind = 0, size = 1; ind < 4; ind++, size <<= 1) {
765 std::string name_load =
766 "__msan_metadata_ptr_for_load_" + std::to_string(size);
767 std::string name_store =
768 "__msan_metadata_ptr_for_store_" + std::to_string(size);
769 MsanMetadataPtrForLoad_1_8[ind] = M.getOrInsertFunction(
770 name_load, RetTy, PointerType::get(IRB.getInt8Ty(), 0));
771 MsanMetadataPtrForStore_1_8[ind] = M.getOrInsertFunction(
772 name_store, RetTy, PointerType::get(IRB.getInt8Ty(), 0));
773 }
774
775 MsanMetadataPtrForLoadN = M.getOrInsertFunction(
776 "__msan_metadata_ptr_for_load_n", RetTy,
777 PointerType::get(IRB.getInt8Ty(), 0), IRB.getInt64Ty());
778 MsanMetadataPtrForStoreN = M.getOrInsertFunction(
779 "__msan_metadata_ptr_for_store_n", RetTy,
780 PointerType::get(IRB.getInt8Ty(), 0), IRB.getInt64Ty());
781
782 // Functions for poisoning and unpoisoning memory.
783 MsanPoisonAllocaFn =
784 M.getOrInsertFunction("__msan_poison_alloca", IRB.getVoidTy(),
785 IRB.getInt8PtrTy(), IntptrTy, IRB.getInt8PtrTy());
786 MsanUnpoisonAllocaFn = M.getOrInsertFunction(
787 "__msan_unpoison_alloca", IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy);
788}
789
791 return M.getOrInsertGlobal(Name, Ty, [&] {
792 return new GlobalVariable(M, Ty, false, GlobalVariable::ExternalLinkage,
793 nullptr, Name, nullptr,
795 });
796}
797
798/// Insert declarations for userspace-specific functions and globals.
799void MemorySanitizer::createUserspaceApi(Module &M, const TargetLibraryInfo &TLI) {
800 IRBuilder<> IRB(*C);
801
802 // Create the callback.
803 // FIXME: this function should have "Cold" calling conv,
804 // which is not yet implemented.
805 if (TrackOrigins) {
806 StringRef WarningFnName = Recover ? "__msan_warning_with_origin"
807 : "__msan_warning_with_origin_noreturn";
808 WarningFn = M.getOrInsertFunction(WarningFnName,
809 TLI.getAttrList(C, {0}, /*Signed=*/false),
810 IRB.getVoidTy(), IRB.getInt32Ty());
811 } else {
812 StringRef WarningFnName =
813 Recover ? "__msan_warning" : "__msan_warning_noreturn";
814 WarningFn = M.getOrInsertFunction(WarningFnName, IRB.getVoidTy());
815 }
816
817 // Create the global TLS variables.
818 RetvalTLS =
819 getOrInsertGlobal(M, "__msan_retval_tls",
820 ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8));
821
822 RetvalOriginTLS = getOrInsertGlobal(M, "__msan_retval_origin_tls", OriginTy);
823
824 ParamTLS =
825 getOrInsertGlobal(M, "__msan_param_tls",
826 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
827
828 ParamOriginTLS =
829 getOrInsertGlobal(M, "__msan_param_origin_tls",
830 ArrayType::get(OriginTy, kParamTLSSize / 4));
831
832 VAArgTLS =
833 getOrInsertGlobal(M, "__msan_va_arg_tls",
834 ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
835
836 VAArgOriginTLS =
837 getOrInsertGlobal(M, "__msan_va_arg_origin_tls",
838 ArrayType::get(OriginTy, kParamTLSSize / 4));
839
840 VAArgOverflowSizeTLS =
841 getOrInsertGlobal(M, "__msan_va_arg_overflow_size_tls", IRB.getInt64Ty());
842
843 for (size_t AccessSizeIndex = 0; AccessSizeIndex < kNumberOfAccessSizes;
844 AccessSizeIndex++) {
845 unsigned AccessSize = 1 << AccessSizeIndex;
846 std::string FunctionName = "__msan_maybe_warning_" + itostr(AccessSize);
847 MaybeWarningFn[AccessSizeIndex] = M.getOrInsertFunction(
848 FunctionName, TLI.getAttrList(C, {0, 1}, /*Signed=*/false),
849 IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), IRB.getInt32Ty());
850
851 FunctionName = "__msan_maybe_store_origin_" + itostr(AccessSize);
852 MaybeStoreOriginFn[AccessSizeIndex] = M.getOrInsertFunction(
853 FunctionName, TLI.getAttrList(C, {0, 2}, /*Signed=*/false),
854 IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8), IRB.getInt8PtrTy(),
855 IRB.getInt32Ty());
856 }
857
858 MsanSetAllocaOriginWithDescriptionFn = M.getOrInsertFunction(
859 "__msan_set_alloca_origin_with_descr", IRB.getVoidTy(),
860 IRB.getInt8PtrTy(), IntptrTy, IRB.getInt8PtrTy(), IRB.getInt8PtrTy());
861 MsanSetAllocaOriginNoDescriptionFn = M.getOrInsertFunction(
862 "__msan_set_alloca_origin_no_descr", IRB.getVoidTy(), IRB.getInt8PtrTy(),
863 IntptrTy, IRB.getInt8PtrTy());
864 MsanPoisonStackFn = M.getOrInsertFunction(
865 "__msan_poison_stack", IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy);
866}
867
868/// Insert extern declaration of runtime-provided functions and globals.
869void MemorySanitizer::initializeCallbacks(Module &M, const TargetLibraryInfo &TLI) {
870 // Only do this once.
871 if (CallbacksInitialized)
872 return;
873
874 IRBuilder<> IRB(*C);
875 // Initialize callbacks that are common for kernel and userspace
876 // instrumentation.
877 MsanChainOriginFn = M.getOrInsertFunction(
878 "__msan_chain_origin",
879 TLI.getAttrList(C, {0}, /*Signed=*/false, /*Ret=*/true), IRB.getInt32Ty(),
880 IRB.getInt32Ty());
881 MsanSetOriginFn = M.getOrInsertFunction(
882 "__msan_set_origin", TLI.getAttrList(C, {2}, /*Signed=*/false),
883 IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy, IRB.getInt32Ty());
884 MemmoveFn =
885 M.getOrInsertFunction("__msan_memmove", IRB.getInt8PtrTy(),
886 IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IntptrTy);
887 MemcpyFn =
888 M.getOrInsertFunction("__msan_memcpy", IRB.getInt8PtrTy(),
889 IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IntptrTy);
890 MemsetFn = M.getOrInsertFunction(
891 "__msan_memset", TLI.getAttrList(C, {1}, /*Signed=*/true),
892 IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt32Ty(), IntptrTy);
893
894 MsanInstrumentAsmStoreFn =
895 M.getOrInsertFunction("__msan_instrument_asm_store", IRB.getVoidTy(),
896 PointerType::get(IRB.getInt8Ty(), 0), IntptrTy);
897
898 if (CompileKernel) {
899 createKernelApi(M, TLI);
900 } else {
901 createUserspaceApi(M, TLI);
902 }
903 CallbacksInitialized = true;
904}
905
906FunctionCallee MemorySanitizer::getKmsanShadowOriginAccessFn(bool isStore,
907 int size) {
908 FunctionCallee *Fns =
909 isStore ? MsanMetadataPtrForStore_1_8 : MsanMetadataPtrForLoad_1_8;
910 switch (size) {
911 case 1:
912 return Fns[0];
913 case 2:
914 return Fns[1];
915 case 4:
916 return Fns[2];
917 case 8:
918 return Fns[3];
919 default:
920 return nullptr;
921 }
922}
923
924/// Module-level initialization.
925///
926/// inserts a call to __msan_init to the module's constructor list.
927void MemorySanitizer::initializeModule(Module &M) {
928 auto &DL = M.getDataLayout();
929
930 bool ShadowPassed = ClShadowBase.getNumOccurrences() > 0;
931 bool OriginPassed = ClOriginBase.getNumOccurrences() > 0;
932 // Check the overrides first
933 if (ShadowPassed || OriginPassed) {
934 CustomMapParams.AndMask = ClAndMask;
935 CustomMapParams.XorMask = ClXorMask;
936 CustomMapParams.ShadowBase = ClShadowBase;
937 CustomMapParams.OriginBase = ClOriginBase;
938 MapParams = &CustomMapParams;
939 } else {
940 Triple TargetTriple(M.getTargetTriple());
941 switch (TargetTriple.getOS()) {
942 case Triple::FreeBSD:
943 switch (TargetTriple.getArch()) {
944 case Triple::aarch64:
945 MapParams = FreeBSD_ARM_MemoryMapParams.bits64;
946 break;
947 case Triple::x86_64:
948 MapParams = FreeBSD_X86_MemoryMapParams.bits64;
949 break;
950 case Triple::x86:
951 MapParams = FreeBSD_X86_MemoryMapParams.bits32;
952 break;
953 default:
954 report_fatal_error("unsupported architecture");
955 }
956 break;
957 case Triple::NetBSD:
958 switch (TargetTriple.getArch()) {
959 case Triple::x86_64:
960 MapParams = NetBSD_X86_MemoryMapParams.bits64;
961 break;
962 default:
963 report_fatal_error("unsupported architecture");
964 }
965 break;
966 case Triple::Linux:
967 switch (TargetTriple.getArch()) {
968 case Triple::x86_64:
969 MapParams = Linux_X86_MemoryMapParams.bits64;
970 break;
971 case Triple::x86:
972 MapParams = Linux_X86_MemoryMapParams.bits32;
973 break;
974 case Triple::mips64:
975 case Triple::mips64el:
976 MapParams = Linux_MIPS_MemoryMapParams.bits64;
977 break;
978 case Triple::ppc64:
979 case Triple::ppc64le:
980 MapParams = Linux_PowerPC_MemoryMapParams.bits64;
981 break;
982 case Triple::systemz:
983 MapParams = Linux_S390_MemoryMapParams.bits64;
984 break;
985 case Triple::aarch64:
987 MapParams = Linux_ARM_MemoryMapParams.bits64;
988 break;
989 default:
990 report_fatal_error("unsupported architecture");
991 }
992 break;
993 default:
994 report_fatal_error("unsupported operating system");
995 }
996 }
997
998 C = &(M.getContext());
999 IRBuilder<> IRB(*C);
1000 IntptrTy = IRB.getIntPtrTy(DL);
1001 OriginTy = IRB.getInt32Ty();
1002
1003 ColdCallWeights = MDBuilder(*C).createBranchWeights(1, 1000);
1004 OriginStoreWeights = MDBuilder(*C).createBranchWeights(1, 1000);
1005
1006 if (!CompileKernel) {
1007 if (TrackOrigins)
1008 M.getOrInsertGlobal("__msan_track_origins", IRB.getInt32Ty(), [&] {
1009 return new GlobalVariable(
1010 M, IRB.getInt32Ty(), true, GlobalValue::WeakODRLinkage,
1011 IRB.getInt32(TrackOrigins), "__msan_track_origins");
1012 });
1013
1014 if (Recover)
1015 M.getOrInsertGlobal("__msan_keep_going", IRB.getInt32Ty(), [&] {
1016 return new GlobalVariable(M, IRB.getInt32Ty(), true,
1017 GlobalValue::WeakODRLinkage,
1018 IRB.getInt32(Recover), "__msan_keep_going");
1019 });
1020 }
1021}
1022
1023namespace {
1024
1025/// A helper class that handles instrumentation of VarArg
1026/// functions on a particular platform.
1027///
1028/// Implementations are expected to insert the instrumentation
1029/// necessary to propagate argument shadow through VarArg function
1030/// calls. Visit* methods are called during an InstVisitor pass over
1031/// the function, and should avoid creating new basic blocks. A new
1032/// instance of this class is created for each instrumented function.
1033struct VarArgHelper {
1034 virtual ~VarArgHelper() = default;
1035
1036 /// Visit a CallBase.
1037 virtual void visitCallBase(CallBase &CB, IRBuilder<> &IRB) = 0;
1038
1039 /// Visit a va_start call.
1040 virtual void visitVAStartInst(VAStartInst &I) = 0;
1041
1042 /// Visit a va_copy call.
1043 virtual void visitVACopyInst(VACopyInst &I) = 0;
1044
1045 /// Finalize function instrumentation.
1046 ///
1047 /// This method is called after visiting all interesting (see above)
1048 /// instructions in a function.
1049 virtual void finalizeInstrumentation() = 0;
1050};
1051
1052struct MemorySanitizerVisitor;
1053
1054} // end anonymous namespace
1055
1056static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
1057 MemorySanitizerVisitor &Visitor);
1058
1059static unsigned TypeSizeToSizeIndex(TypeSize TS) {
1060 if (TS.isScalable())
1061 // Scalable types unconditionally take slowpaths.
1062 return kNumberOfAccessSizes;
1063 unsigned TypeSizeFixed = TS.getFixedValue();
1064 if (TypeSizeFixed <= 8)
1065 return 0;
1066 return Log2_32_Ceil((TypeSizeFixed + 7) / 8);
1067}
1068
1069namespace {
1070
1071/// Helper class to attach debug information of the given instruction onto new
1072/// instructions inserted after.
1073class NextNodeIRBuilder : public IRBuilder<> {
1074public:
1075 explicit NextNodeIRBuilder(Instruction *IP) : IRBuilder<>(IP->getNextNode()) {
1076 SetCurrentDebugLocation(IP->getDebugLoc());
1077 }
1078};
1079
1080/// This class does all the work for a given function. Store and Load
1081/// instructions store and load corresponding shadow and origin
1082/// values. Most instructions propagate shadow from arguments to their
1083/// return values. Certain instructions (most importantly, BranchInst)
1084/// test their argument shadow and print reports (with a runtime call) if it's
1085/// non-zero.
1086struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
1087 Function &F;
1088 MemorySanitizer &MS;
1089 SmallVector<PHINode *, 16> ShadowPHINodes, OriginPHINodes;
1090 ValueMap<Value *, Value *> ShadowMap, OriginMap;
1091 std::unique_ptr<VarArgHelper> VAHelper;
1092 const TargetLibraryInfo *TLI;
1093 Instruction *FnPrologueEnd;
1094
1095 // The following flags disable parts of MSan instrumentation based on
1096 // exclusion list contents and command-line options.
1097 bool InsertChecks;
1098 bool PropagateShadow;
1099 bool PoisonStack;
1100 bool PoisonUndef;
1101
1102 struct ShadowOriginAndInsertPoint {
1103 Value *Shadow;
1104 Value *Origin;
1105 Instruction *OrigIns;
1106
1107 ShadowOriginAndInsertPoint(Value *S, Value *O, Instruction *I)
1108 : Shadow(S), Origin(O), OrigIns(I) {}
1109 };
1111 DenseMap<const DILocation *, int> LazyWarningDebugLocationCount;
1112 bool InstrumentLifetimeStart = ClHandleLifetimeIntrinsics;
1116 int64_t SplittableBlocksCount = 0;
1117
1118 MemorySanitizerVisitor(Function &F, MemorySanitizer &MS,
1119 const TargetLibraryInfo &TLI)
1120 : F(F), MS(MS), VAHelper(CreateVarArgHelper(F, MS, *this)), TLI(&TLI) {
1121 bool SanitizeFunction =
1122 F.hasFnAttribute(Attribute::SanitizeMemory) && !ClDisableChecks;
1123 InsertChecks = SanitizeFunction;
1124 PropagateShadow = SanitizeFunction;
1125 PoisonStack = SanitizeFunction && ClPoisonStack;
1126 PoisonUndef = SanitizeFunction && ClPoisonUndef;
1127
1128 // In the presence of unreachable blocks, we may see Phi nodes with
1129 // incoming nodes from such blocks. Since InstVisitor skips unreachable
1130 // blocks, such nodes will not have any shadow value associated with them.
1131 // It's easier to remove unreachable blocks than deal with missing shadow.
1133
1134 MS.initializeCallbacks(*F.getParent(), TLI);
1135 FnPrologueEnd = IRBuilder<>(F.getEntryBlock().getFirstNonPHI())
1136 .CreateIntrinsic(Intrinsic::donothing, {}, {});
1137
1138 if (MS.CompileKernel) {
1139 IRBuilder<> IRB(FnPrologueEnd);
1140 insertKmsanPrologue(IRB);
1141 }
1142
1143 LLVM_DEBUG(if (!InsertChecks) dbgs()
1144 << "MemorySanitizer is not inserting checks into '"
1145 << F.getName() << "'\n");
1146 }
1147
1148 bool instrumentWithCalls(Value *V) {
1149 // Constants likely will be eliminated by follow-up passes.
1150 if (isa<Constant>(V))
1151 return false;
1152
1153 ++SplittableBlocksCount;
1155 SplittableBlocksCount > ClInstrumentationWithCallThreshold;
1156 }
1157
1158 bool isInPrologue(Instruction &I) {
1159 return I.getParent() == FnPrologueEnd->getParent() &&
1160 (&I == FnPrologueEnd || I.comesBefore(FnPrologueEnd));
1161 }
1162
1163 // Creates a new origin and records the stack trace. In general we can call
1164 // this function for any origin manipulation we like. However it will cost
1165 // runtime resources. So use this wisely only if it can provide additional
1166 // information helpful to a user.
1167 Value *updateOrigin(Value *V, IRBuilder<> &IRB) {
1168 if (MS.TrackOrigins <= 1)
1169 return V;
1170 return IRB.CreateCall(MS.MsanChainOriginFn, V);
1171 }
1172
1173 Value *originToIntptr(IRBuilder<> &IRB, Value *Origin) {
1174 const DataLayout &DL = F.getParent()->getDataLayout();
1175 unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1176 if (IntptrSize == kOriginSize)
1177 return Origin;
1178 assert(IntptrSize == kOriginSize * 2);
1179 Origin = IRB.CreateIntCast(Origin, MS.IntptrTy, /* isSigned */ false);
1180 return IRB.CreateOr(Origin, IRB.CreateShl(Origin, kOriginSize * 8));
1181 }
1182
1183 /// Fill memory range with the given origin value.
1184 void paintOrigin(IRBuilder<> &IRB, Value *Origin, Value *OriginPtr,
1185 TypeSize TS, Align Alignment) {
1186 unsigned Size = TS.getFixedValue();
1187 const DataLayout &DL = F.getParent()->getDataLayout();
1188 const Align IntptrAlignment = DL.getABITypeAlign(MS.IntptrTy);
1189 unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1190 assert(IntptrAlignment >= kMinOriginAlignment);
1191 assert(IntptrSize >= kOriginSize);
1192
1193 unsigned Ofs = 0;
1194 Align CurrentAlignment = Alignment;
1195 if (Alignment >= IntptrAlignment && IntptrSize > kOriginSize) {
1196 Value *IntptrOrigin = originToIntptr(IRB, Origin);
1197 Value *IntptrOriginPtr =
1198 IRB.CreatePointerCast(OriginPtr, PointerType::get(MS.IntptrTy, 0));
1199 for (unsigned i = 0; i < Size / IntptrSize; ++i) {
1200 Value *Ptr = i ? IRB.CreateConstGEP1_32(MS.IntptrTy, IntptrOriginPtr, i)
1201 : IntptrOriginPtr;
1202 IRB.CreateAlignedStore(IntptrOrigin, Ptr, CurrentAlignment);
1203 Ofs += IntptrSize / kOriginSize;
1204 CurrentAlignment = IntptrAlignment;
1205 }
1206 }
1207
1208 for (unsigned i = Ofs; i < (Size + kOriginSize - 1) / kOriginSize; ++i) {
1209 Value *GEP =
1210 i ? IRB.CreateConstGEP1_32(MS.OriginTy, OriginPtr, i) : OriginPtr;
1211 IRB.CreateAlignedStore(Origin, GEP, CurrentAlignment);
1212 CurrentAlignment = kMinOriginAlignment;
1213 }
1214 }
1215
1216 void storeOrigin(IRBuilder<> &IRB, Value *Addr, Value *Shadow, Value *Origin,
1217 Value *OriginPtr, Align Alignment) {
1218 const DataLayout &DL = F.getParent()->getDataLayout();
1219 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1220 TypeSize StoreSize = DL.getTypeStoreSize(Shadow->getType());
1221 Value *ConvertedShadow = convertShadowToScalar(Shadow, IRB);
1222 if (auto *ConstantShadow = dyn_cast<Constant>(ConvertedShadow)) {
1223 if (!ClCheckConstantShadow || ConstantShadow->isZeroValue()) {
1224 // Origin is not needed: value is initialized or const shadow is
1225 // ignored.
1226 return;
1227 }
1228 if (llvm::isKnownNonZero(ConvertedShadow, DL)) {
1229 // Copy origin as the value is definitely uninitialized.
1230 paintOrigin(IRB, updateOrigin(Origin, IRB), OriginPtr, StoreSize,
1231 OriginAlignment);
1232 return;
1233 }
1234 // Fallback to runtime check, which still can be optimized out later.
1235 }
1236
1237 TypeSize TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
1238 unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1239 if (instrumentWithCalls(ConvertedShadow) &&
1240 SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
1241 FunctionCallee Fn = MS.MaybeStoreOriginFn[SizeIndex];
1242 Value *ConvertedShadow2 =
1243 IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1244 CallBase *CB = IRB.CreateCall(
1245 Fn, {ConvertedShadow2,
1246 IRB.CreatePointerCast(Addr, IRB.getInt8PtrTy()), Origin});
1247 CB->addParamAttr(0, Attribute::ZExt);
1248 CB->addParamAttr(2, Attribute::ZExt);
1249 } else {
1250 Value *Cmp = convertToBool(ConvertedShadow, IRB, "_mscmp");
1252 Cmp, &*IRB.GetInsertPoint(), false, MS.OriginStoreWeights);
1253 IRBuilder<> IRBNew(CheckTerm);
1254 paintOrigin(IRBNew, updateOrigin(Origin, IRBNew), OriginPtr, StoreSize,
1255 OriginAlignment);
1256 }
1257 }
1258
1259 void materializeStores() {
1260 for (StoreInst *SI : StoreList) {
1261 IRBuilder<> IRB(SI);
1262 Value *Val = SI->getValueOperand();
1263 Value *Addr = SI->getPointerOperand();
1264 Value *Shadow = SI->isAtomic() ? getCleanShadow(Val) : getShadow(Val);
1265 Value *ShadowPtr, *OriginPtr;
1266 Type *ShadowTy = Shadow->getType();
1267 const Align Alignment = SI->getAlign();
1268 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1269 std::tie(ShadowPtr, OriginPtr) =
1270 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ true);
1271
1272 StoreInst *NewSI = IRB.CreateAlignedStore(Shadow, ShadowPtr, Alignment);
1273 LLVM_DEBUG(dbgs() << " STORE: " << *NewSI << "\n");
1274 (void)NewSI;
1275
1276 if (SI->isAtomic())
1277 SI->setOrdering(addReleaseOrdering(SI->getOrdering()));
1278
1279 if (MS.TrackOrigins && !SI->isAtomic())
1280 storeOrigin(IRB, Addr, Shadow, getOrigin(Val), OriginPtr,
1281 OriginAlignment);
1282 }
1283 }
1284
1285 // Returns true if Debug Location curresponds to multiple warnings.
1286 bool shouldDisambiguateWarningLocation(const DebugLoc &DebugLoc) {
1287 if (MS.TrackOrigins < 2)
1288 return false;
1289
1290 if (LazyWarningDebugLocationCount.empty())
1291 for (const auto &I : InstrumentationList)
1292 ++LazyWarningDebugLocationCount[I.OrigIns->getDebugLoc()];
1293
1294 return LazyWarningDebugLocationCount[DebugLoc] >= ClDisambiguateWarning;
1295 }
1296
1297 /// Helper function to insert a warning at IRB's current insert point.
1298 void insertWarningFn(IRBuilder<> &IRB, Value *Origin) {
1299 if (!Origin)
1300 Origin = (Value *)IRB.getInt32(0);
1301 assert(Origin->getType()->isIntegerTy());
1302
1303 if (shouldDisambiguateWarningLocation(IRB.getCurrentDebugLocation())) {
1304 // Try to create additional origin with debug info of the last origin
1305 // instruction. It may provide additional information to the user.
1306 if (Instruction *OI = dyn_cast_or_null<Instruction>(Origin)) {
1307 assert(MS.TrackOrigins);
1308 auto NewDebugLoc = OI->getDebugLoc();
1309 // Origin update with missing or the same debug location provides no
1310 // additional value.
1311 if (NewDebugLoc && NewDebugLoc != IRB.getCurrentDebugLocation()) {
1312 // Insert update just before the check, so we call runtime only just
1313 // before the report.
1314 IRBuilder<> IRBOrigin(&*IRB.GetInsertPoint());
1315 IRBOrigin.SetCurrentDebugLocation(NewDebugLoc);
1316 Origin = updateOrigin(Origin, IRBOrigin);
1317 }
1318 }
1319 }
1320
1321 if (MS.CompileKernel || MS.TrackOrigins)
1322 IRB.CreateCall(MS.WarningFn, Origin)->setCannotMerge();
1323 else
1324 IRB.CreateCall(MS.WarningFn)->setCannotMerge();
1325 // FIXME: Insert UnreachableInst if !MS.Recover?
1326 // This may invalidate some of the following checks and needs to be done
1327 // at the very end.
1328 }
1329
1330 void materializeOneCheck(IRBuilder<> &IRB, Value *ConvertedShadow,
1331 Value *Origin) {
1332 const DataLayout &DL = F.getParent()->getDataLayout();
1333 TypeSize TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
1334 unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1335 if (instrumentWithCalls(ConvertedShadow) &&
1336 SizeIndex < kNumberOfAccessSizes && !MS.CompileKernel) {
1337 FunctionCallee Fn = MS.MaybeWarningFn[SizeIndex];
1338 Value *ConvertedShadow2 =
1339 IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1340 CallBase *CB = IRB.CreateCall(
1341 Fn, {ConvertedShadow2,
1342 MS.TrackOrigins && Origin ? Origin : (Value *)IRB.getInt32(0)});
1343 CB->addParamAttr(0, Attribute::ZExt);
1344 CB->addParamAttr(1, Attribute::ZExt);
1345 } else {
1346 Value *Cmp = convertToBool(ConvertedShadow, IRB, "_mscmp");
1348 Cmp, &*IRB.GetInsertPoint(),
1349 /* Unreachable */ !MS.Recover, MS.ColdCallWeights);
1350
1351 IRB.SetInsertPoint(CheckTerm);
1352 insertWarningFn(IRB, Origin);
1353 LLVM_DEBUG(dbgs() << " CHECK: " << *Cmp << "\n");
1354 }
1355 }
1356
1357 void materializeInstructionChecks(
1358 ArrayRef<ShadowOriginAndInsertPoint> InstructionChecks) {
1359 const DataLayout &DL = F.getParent()->getDataLayout();
1360 // Disable combining in some cases. TrackOrigins checks each shadow to pick
1361 // correct origin.
1362 bool Combine = !MS.TrackOrigins;
1363 Instruction *Instruction = InstructionChecks.front().OrigIns;
1364 Value *Shadow = nullptr;
1365 for (const auto &ShadowData : InstructionChecks) {
1366 assert(ShadowData.OrigIns == Instruction);
1368
1369 Value *ConvertedShadow = ShadowData.Shadow;
1370
1371 if (auto *ConstantShadow = dyn_cast<Constant>(ConvertedShadow)) {
1372 if (!ClCheckConstantShadow || ConstantShadow->isZeroValue()) {
1373 // Skip, value is initialized or const shadow is ignored.
1374 continue;
1375 }
1376 if (llvm::isKnownNonZero(ConvertedShadow, DL)) {
1377 // Report as the value is definitely uninitialized.
1378 insertWarningFn(IRB, ShadowData.Origin);
1379 if (!MS.Recover)
1380 return; // Always fail and stop here, not need to check the rest.
1381 // Skip entire instruction,
1382 continue;
1383 }
1384 // Fallback to runtime check, which still can be optimized out later.
1385 }
1386
1387 if (!Combine) {
1388 materializeOneCheck(IRB, ConvertedShadow, ShadowData.Origin);
1389 continue;
1390 }
1391
1392 if (!Shadow) {
1393 Shadow = ConvertedShadow;
1394 continue;
1395 }
1396
1397 Shadow = convertToBool(Shadow, IRB, "_mscmp");
1398 ConvertedShadow = convertToBool(ConvertedShadow, IRB, "_mscmp");
1399 Shadow = IRB.CreateOr(Shadow, ConvertedShadow, "_msor");
1400 }
1401
1402 if (Shadow) {
1403 assert(Combine);
1405 materializeOneCheck(IRB, Shadow, nullptr);
1406 }
1407 }
1408
1409 void materializeChecks() {
1410 llvm::stable_sort(InstrumentationList,
1411 [](const ShadowOriginAndInsertPoint &L,
1412 const ShadowOriginAndInsertPoint &R) {
1413 return L.OrigIns < R.OrigIns;
1414 });
1415
1416 for (auto I = InstrumentationList.begin();
1417 I != InstrumentationList.end();) {
1418 auto J =
1419 std::find_if(I + 1, InstrumentationList.end(),
1420 [L = I->OrigIns](const ShadowOriginAndInsertPoint &R) {
1421 return L != R.OrigIns;
1422 });
1423 // Process all checks of instruction at once.
1424 materializeInstructionChecks(ArrayRef<ShadowOriginAndInsertPoint>(I, J));
1425 I = J;
1426 }
1427
1428 LLVM_DEBUG(dbgs() << "DONE:\n" << F);
1429 }
1430
1431 // Returns the last instruction in the new prologue
1432 void insertKmsanPrologue(IRBuilder<> &IRB) {
1433 Value *ContextState = IRB.CreateCall(MS.MsanGetContextStateFn, {});
1434 Constant *Zero = IRB.getInt32(0);
1435 MS.ParamTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1436 {Zero, IRB.getInt32(0)}, "param_shadow");
1437 MS.RetvalTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1438 {Zero, IRB.getInt32(1)}, "retval_shadow");
1439 MS.VAArgTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1440 {Zero, IRB.getInt32(2)}, "va_arg_shadow");
1441 MS.VAArgOriginTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1442 {Zero, IRB.getInt32(3)}, "va_arg_origin");
1443 MS.VAArgOverflowSizeTLS =
1444 IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1445 {Zero, IRB.getInt32(4)}, "va_arg_overflow_size");
1446 MS.ParamOriginTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1447 {Zero, IRB.getInt32(5)}, "param_origin");
1448 MS.RetvalOriginTLS =
1449 IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1450 {Zero, IRB.getInt32(6)}, "retval_origin");
1451 }
1452
1453 /// Add MemorySanitizer instrumentation to a function.
1454 bool runOnFunction() {
1455 // Iterate all BBs in depth-first order and create shadow instructions
1456 // for all instructions (where applicable).
1457 // For PHI nodes we create dummy shadow PHIs which will be finalized later.
1458 for (BasicBlock *BB : depth_first(FnPrologueEnd->getParent()))
1459 visit(*BB);
1460
1461 // Finalize PHI nodes.
1462 for (PHINode *PN : ShadowPHINodes) {
1463 PHINode *PNS = cast<PHINode>(getShadow(PN));
1464 PHINode *PNO = MS.TrackOrigins ? cast<PHINode>(getOrigin(PN)) : nullptr;
1465 size_t NumValues = PN->getNumIncomingValues();
1466 for (size_t v = 0; v < NumValues; v++) {
1467 PNS->addIncoming(getShadow(PN, v), PN->getIncomingBlock(v));
1468 if (PNO)
1469 PNO->addIncoming(getOrigin(PN, v), PN->getIncomingBlock(v));
1470 }
1471 }
1472
1473 VAHelper->finalizeInstrumentation();
1474
1475 // Poison llvm.lifetime.start intrinsics, if we haven't fallen back to
1476 // instrumenting only allocas.
1477 if (InstrumentLifetimeStart) {
1478 for (auto Item : LifetimeStartList) {
1479 instrumentAlloca(*Item.second, Item.first);
1480 AllocaSet.remove(Item.second);
1481 }
1482 }
1483 // Poison the allocas for which we didn't instrument the corresponding
1484 // lifetime intrinsics.
1485 for (AllocaInst *AI : AllocaSet)
1486 instrumentAlloca(*AI);
1487
1488 // Insert shadow value checks.
1489 materializeChecks();
1490
1491 // Delayed instrumentation of StoreInst.
1492 // This may not add new address checks.
1493 materializeStores();
1494
1495 return true;
1496 }
1497
1498 /// Compute the shadow type that corresponds to a given Value.
1499 Type *getShadowTy(Value *V) { return getShadowTy(V->getType()); }
1500
1501 /// Compute the shadow type that corresponds to a given Type.
1502 Type *getShadowTy(Type *OrigTy) {
1503 if (!OrigTy->isSized()) {
1504 return nullptr;
1505 }
1506 // For integer type, shadow is the same as the original type.
1507 // This may return weird-sized types like i1.
1508 if (IntegerType *IT = dyn_cast<IntegerType>(OrigTy))
1509 return IT;
1510 const DataLayout &DL = F.getParent()->getDataLayout();
1511 if (VectorType *VT = dyn_cast<VectorType>(OrigTy)) {
1512 uint32_t EltSize = DL.getTypeSizeInBits(VT->getElementType());
1513 return VectorType::get(IntegerType::get(*MS.C, EltSize),
1514 VT->getElementCount());
1515 }
1516 if (ArrayType *AT = dyn_cast<ArrayType>(OrigTy)) {
1517 return ArrayType::get(getShadowTy(AT->getElementType()),
1518 AT->getNumElements());
1519 }
1520 if (StructType *ST = dyn_cast<StructType>(OrigTy)) {
1522 for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
1523 Elements.push_back(getShadowTy(ST->getElementType(i)));
1524 StructType *Res = StructType::get(*MS.C, Elements, ST->isPacked());
1525 LLVM_DEBUG(dbgs() << "getShadowTy: " << *ST << " ===> " << *Res << "\n");
1526 return Res;
1527 }
1528 uint32_t TypeSize = DL.getTypeSizeInBits(OrigTy);
1529 return IntegerType::get(*MS.C, TypeSize);
1530 }
1531
1532 /// Extract combined shadow of struct elements as a bool
1533 Value *collapseStructShadow(StructType *Struct, Value *Shadow,
1534 IRBuilder<> &IRB) {
1535 Value *FalseVal = IRB.getIntN(/* width */ 1, /* value */ 0);
1536 Value *Aggregator = FalseVal;
1537
1538 for (unsigned Idx = 0; Idx < Struct->getNumElements(); Idx++) {
1539 // Combine by ORing together each element's bool shadow
1540 Value *ShadowItem = IRB.CreateExtractValue(Shadow, Idx);
1541 Value *ShadowBool = convertToBool(ShadowItem, IRB);
1542
1543 if (Aggregator != FalseVal)
1544 Aggregator = IRB.CreateOr(Aggregator, ShadowBool);
1545 else
1546 Aggregator = ShadowBool;
1547 }
1548
1549 return Aggregator;
1550 }
1551
1552 // Extract combined shadow of array elements
1553 Value *collapseArrayShadow(ArrayType *Array, Value *Shadow,
1554 IRBuilder<> &IRB) {
1555 if (!Array->getNumElements())
1556 return IRB.getIntN(/* width */ 1, /* value */ 0);
1557
1558 Value *FirstItem = IRB.CreateExtractValue(Shadow, 0);
1559 Value *Aggregator = convertShadowToScalar(FirstItem, IRB);
1560
1561 for (unsigned Idx = 1; Idx < Array->getNumElements(); Idx++) {
1562 Value *ShadowItem = IRB.CreateExtractValue(Shadow, Idx);
1563 Value *ShadowInner = convertShadowToScalar(ShadowItem, IRB);
1564 Aggregator = IRB.CreateOr(Aggregator, ShadowInner);
1565 }
1566 return Aggregator;
1567 }
1568
1569 /// Convert a shadow value to it's flattened variant. The resulting
1570 /// shadow may not necessarily have the same bit width as the input
1571 /// value, but it will always be comparable to zero.
1572 Value *convertShadowToScalar(Value *V, IRBuilder<> &IRB) {
1573 if (StructType *Struct = dyn_cast<StructType>(V->getType()))
1574 return collapseStructShadow(Struct, V, IRB);
1575 if (ArrayType *Array = dyn_cast<ArrayType>(V->getType()))
1576 return collapseArrayShadow(Array, V, IRB);
1577 if (isa<VectorType>(V->getType())) {
1578 unsigned BitWidth =
1579 V->getType()->getPrimitiveSizeInBits().getFixedValue();
1580 return IRB.CreateBitCast(V, IntegerType::get(*MS.C, BitWidth));
1581 }
1582 return V;
1583 }
1584
1585 // Convert a scalar value to an i1 by comparing with 0
1586 Value *convertToBool(Value *V, IRBuilder<> &IRB, const Twine &name = "") {
1587 Type *VTy = V->getType();
1588 if (!VTy->isIntegerTy())
1589 return convertToBool(convertShadowToScalar(V, IRB), IRB, name);
1590 if (VTy->getIntegerBitWidth() == 1)
1591 // Just converting a bool to a bool, so do nothing.
1592 return V;
1593 return IRB.CreateICmpNE(V, ConstantInt::get(VTy, 0), name);
1594 }
1595
1596 Type *ptrToIntPtrType(Type *PtrTy) const {
1597 if (VectorType *VectTy = dyn_cast<VectorType>(PtrTy)) {
1598 return VectorType::get(ptrToIntPtrType(VectTy->getElementType()),
1599 VectTy->getElementCount());
1600 }
1601 assert(PtrTy->isIntOrPtrTy());
1602 return MS.IntptrTy;
1603 }
1604
1605 Type *getPtrToShadowPtrType(Type *IntPtrTy, Type *ShadowTy) const {
1606 if (VectorType *VectTy = dyn_cast<VectorType>(IntPtrTy)) {
1607 return VectorType::get(
1608 getPtrToShadowPtrType(VectTy->getElementType(), ShadowTy),
1609 VectTy->getElementCount());
1610 }
1611 assert(IntPtrTy == MS.IntptrTy);
1612 return ShadowTy->getPointerTo();
1613 }
1614
1615 Constant *constToIntPtr(Type *IntPtrTy, uint64_t C) const {
1616 if (VectorType *VectTy = dyn_cast<VectorType>(IntPtrTy)) {
1618 VectTy->getElementCount(), constToIntPtr(VectTy->getElementType(), C));
1619 }
1620 assert(IntPtrTy == MS.IntptrTy);
1621 return ConstantInt::get(MS.IntptrTy, C);
1622 }
1623
1624 /// Compute the integer shadow offset that corresponds to a given
1625 /// application address.
1626 ///
1627 /// Offset = (Addr & ~AndMask) ^ XorMask
1628 /// Addr can be a ptr or <N x ptr>. In both cases ShadowTy the shadow type of
1629 /// a single pointee.
1630 /// Returns <shadow_ptr, origin_ptr> or <<N x shadow_ptr>, <N x origin_ptr>>.
1631 Value *getShadowPtrOffset(Value *Addr, IRBuilder<> &IRB) {
1632 Type *IntptrTy = ptrToIntPtrType(Addr->getType());
1633 Value *OffsetLong = IRB.CreatePointerCast(Addr, IntptrTy);
1634
1635 if (uint64_t AndMask = MS.MapParams->AndMask)
1636 OffsetLong = IRB.CreateAnd(OffsetLong, constToIntPtr(IntptrTy, ~AndMask));
1637
1638 if (uint64_t XorMask = MS.MapParams->XorMask)
1639 OffsetLong = IRB.CreateXor(OffsetLong, constToIntPtr(IntptrTy, XorMask));
1640 return OffsetLong;
1641 }
1642
1643 /// Compute the shadow and origin addresses corresponding to a given
1644 /// application address.
1645 ///
1646 /// Shadow = ShadowBase + Offset
1647 /// Origin = (OriginBase + Offset) & ~3ULL
1648 /// Addr can be a ptr or <N x ptr>. In both cases ShadowTy the shadow type of
1649 /// a single pointee.
1650 /// Returns <shadow_ptr, origin_ptr> or <<N x shadow_ptr>, <N x origin_ptr>>.
1651 std::pair<Value *, Value *>
1652 getShadowOriginPtrUserspace(Value *Addr, IRBuilder<> &IRB, Type *ShadowTy,
1653 MaybeAlign Alignment) {
1654 Type *IntptrTy = ptrToIntPtrType(Addr->getType());
1655 Value *ShadowOffset = getShadowPtrOffset(Addr, IRB);
1656 Value *ShadowLong = ShadowOffset;
1657 if (uint64_t ShadowBase = MS.MapParams->ShadowBase) {
1658 ShadowLong =
1659 IRB.CreateAdd(ShadowLong, constToIntPtr(IntptrTy, ShadowBase));
1660 }
1661 Value *ShadowPtr = IRB.CreateIntToPtr(
1662 ShadowLong, getPtrToShadowPtrType(IntptrTy, ShadowTy));
1663
1664 Value *OriginPtr = nullptr;
1665 if (MS.TrackOrigins) {
1666 Value *OriginLong = ShadowOffset;
1667 uint64_t OriginBase = MS.MapParams->OriginBase;
1668 if (OriginBase != 0)
1669 OriginLong =
1670 IRB.CreateAdd(OriginLong, constToIntPtr(IntptrTy, OriginBase));
1671 if (!Alignment || *Alignment < kMinOriginAlignment) {
1673 OriginLong = IRB.CreateAnd(OriginLong, constToIntPtr(IntptrTy, ~Mask));
1674 }
1675 OriginPtr = IRB.CreateIntToPtr(
1676 OriginLong, getPtrToShadowPtrType(IntptrTy, MS.OriginTy));
1677 }
1678 return std::make_pair(ShadowPtr, OriginPtr);
1679 }
1680
1681 std::pair<Value *, Value *> getShadowOriginPtrKernelNoVec(Value *Addr,
1682 IRBuilder<> &IRB,
1683 Type *ShadowTy,
1684 bool isStore) {
1685 Value *ShadowOriginPtrs;
1686 const DataLayout &DL = F.getParent()->getDataLayout();
1687 TypeSize Size = DL.getTypeStoreSize(ShadowTy);
1688
1689 FunctionCallee Getter = MS.getKmsanShadowOriginAccessFn(isStore, Size);
1690 Value *AddrCast =
1691 IRB.CreatePointerCast(Addr, PointerType::get(IRB.getInt8Ty(), 0));
1692 if (Getter) {
1693 ShadowOriginPtrs = IRB.CreateCall(Getter, AddrCast);
1694 } else {
1695 Value *SizeVal = ConstantInt::get(MS.IntptrTy, Size);
1696 ShadowOriginPtrs = IRB.CreateCall(isStore ? MS.MsanMetadataPtrForStoreN
1697 : MS.MsanMetadataPtrForLoadN,
1698 {AddrCast, SizeVal});
1699 }
1700 Value *ShadowPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 0);
1701 ShadowPtr = IRB.CreatePointerCast(ShadowPtr, PointerType::get(ShadowTy, 0));
1702 Value *OriginPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 1);
1703
1704 return std::make_pair(ShadowPtr, OriginPtr);
1705 }
1706
1707 /// Addr can be a ptr or <N x ptr>. In both cases ShadowTy the shadow type of
1708 /// a single pointee.
1709 /// Returns <shadow_ptr, origin_ptr> or <<N x shadow_ptr>, <N x origin_ptr>>.
1710 std::pair<Value *, Value *> getShadowOriginPtrKernel(Value *Addr,
1711 IRBuilder<> &IRB,
1712 Type *ShadowTy,
1713 bool isStore) {
1714 VectorType *VectTy = dyn_cast<VectorType>(Addr->getType());
1715 if (!VectTy) {
1716 assert(Addr->getType()->isPointerTy());
1717 return getShadowOriginPtrKernelNoVec(Addr, IRB, ShadowTy, isStore);
1718 }
1719
1720 // TODO: Support callbacs with vectors of addresses.
1721 unsigned NumElements = cast<FixedVectorType>(VectTy)->getNumElements();
1722 Value *ShadowPtrs = ConstantInt::getNullValue(
1723 FixedVectorType::get(ShadowTy->getPointerTo(), NumElements));
1724 Value *OriginPtrs = nullptr;
1725 if (MS.TrackOrigins)
1726 OriginPtrs = ConstantInt::getNullValue(
1727 FixedVectorType::get(MS.OriginTy->getPointerTo(), NumElements));
1728 for (unsigned i = 0; i < NumElements; ++i) {
1729 Value *OneAddr =
1731 auto [ShadowPtr, OriginPtr] =
1732 getShadowOriginPtrKernelNoVec(OneAddr, IRB, ShadowTy, isStore);
1733
1734 ShadowPtrs = IRB.CreateInsertElement(
1735 ShadowPtrs, ShadowPtr, ConstantInt::get(IRB.getInt32Ty(), i));
1736 if (MS.TrackOrigins)
1737 OriginPtrs = IRB.CreateInsertElement(
1738 OriginPtrs, OriginPtr, ConstantInt::get(IRB.getInt32Ty(), i));
1739 }
1740 return {ShadowPtrs, OriginPtrs};
1741 }
1742
1743 std::pair<Value *, Value *> getShadowOriginPtr(Value *Addr, IRBuilder<> &IRB,
1744 Type *ShadowTy,
1745 MaybeAlign Alignment,
1746 bool isStore) {
1747 if (MS.CompileKernel)
1748 return getShadowOriginPtrKernel(Addr, IRB, ShadowTy, isStore);
1749 return getShadowOriginPtrUserspace(Addr, IRB, ShadowTy, Alignment);
1750 }
1751
1752 /// Compute the shadow address for a given function argument.
1753 ///
1754 /// Shadow = ParamTLS+ArgOffset.
1755 Value *getShadowPtrForArgument(Value *A, IRBuilder<> &IRB, int ArgOffset) {
1756 Value *Base = IRB.CreatePointerCast(MS.ParamTLS, MS.IntptrTy);
1757 if (ArgOffset)
1758 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1759 return IRB.CreateIntToPtr(Base, PointerType::get(getShadowTy(A), 0),
1760 "_msarg");
1761 }
1762
1763 /// Compute the origin address for a given function argument.
1764 Value *getOriginPtrForArgument(Value *A, IRBuilder<> &IRB, int ArgOffset) {
1765 if (!MS.TrackOrigins)
1766 return nullptr;
1767 Value *Base = IRB.CreatePointerCast(MS.ParamOriginTLS, MS.IntptrTy);
1768 if (ArgOffset)
1769 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1770 return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0),
1771 "_msarg_o");
1772 }
1773
1774 /// Compute the shadow address for a retval.
1775 Value *getShadowPtrForRetval(Value *A, IRBuilder<> &IRB) {
1776 return IRB.CreatePointerCast(MS.RetvalTLS,
1777 PointerType::get(getShadowTy(A), 0), "_msret");
1778 }
1779
1780 /// Compute the origin address for a retval.
1781 Value *getOriginPtrForRetval(IRBuilder<> &IRB) {
1782 // We keep a single origin for the entire retval. Might be too optimistic.
1783 return MS.RetvalOriginTLS;
1784 }
1785
1786 /// Set SV to be the shadow value for V.
1787 void setShadow(Value *V, Value *SV) {
1788 assert(!ShadowMap.count(V) && "Values may only have one shadow");
1789 ShadowMap[V] = PropagateShadow ? SV : getCleanShadow(V);
1790 }
1791
1792 /// Set Origin to be the origin value for V.
1793 void setOrigin(Value *V, Value *Origin) {
1794 if (!MS.TrackOrigins)
1795 return;
1796 assert(!OriginMap.count(V) && "Values may only have one origin");
1797 LLVM_DEBUG(dbgs() << "ORIGIN: " << *V << " ==> " << *Origin << "\n");
1798 OriginMap[V] = Origin;
1799 }
1800
1801 Constant *getCleanShadow(Type *OrigTy) {
1802 Type *ShadowTy = getShadowTy(OrigTy);
1803 if (!ShadowTy)
1804 return nullptr;
1805 return Constant::getNullValue(ShadowTy);
1806 }
1807
1808 /// Create a clean shadow value for a given value.
1809 ///
1810 /// Clean shadow (all zeroes) means all bits of the value are defined
1811 /// (initialized).
1812 Constant *getCleanShadow(Value *V) { return getCleanShadow(V->getType()); }
1813
1814 /// Create a dirty shadow of a given shadow type.
1815 Constant *getPoisonedShadow(Type *ShadowTy) {
1816 assert(ShadowTy);
1817 if (isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy))
1818 return Constant::getAllOnesValue(ShadowTy);
1819 if (ArrayType *AT = dyn_cast<ArrayType>(ShadowTy)) {
1820 SmallVector<Constant *, 4> Vals(AT->getNumElements(),
1821 getPoisonedShadow(AT->getElementType()));
1822 return ConstantArray::get(AT, Vals);
1823 }
1824 if (StructType *ST = dyn_cast<StructType>(ShadowTy)) {
1826 for (unsigned i = 0, n = ST->getNumElements(); i < n; i++)
1827 Vals.push_back(getPoisonedShadow(ST->getElementType(i)));
1828 return ConstantStruct::get(ST, Vals);
1829 }
1830 llvm_unreachable("Unexpected shadow type");
1831 }
1832
1833 /// Create a dirty shadow for a given value.
1834 Constant *getPoisonedShadow(Value *V) {
1835 Type *ShadowTy = getShadowTy(V);
1836 if (!ShadowTy)
1837 return nullptr;
1838 return getPoisonedShadow(ShadowTy);
1839 }
1840
1841 /// Create a clean (zero) origin.
1842 Value *getCleanOrigin() { return Constant::getNullValue(MS.OriginTy); }
1843
1844 /// Get the shadow value for a given Value.
1845 ///
1846 /// This function either returns the value set earlier with setShadow,
1847 /// or extracts if from ParamTLS (for function arguments).
1848 Value *getShadow(Value *V) {
1849 if (Instruction *I = dyn_cast<Instruction>(V)) {
1850 if (!PropagateShadow || I->getMetadata(LLVMContext::MD_nosanitize))
1851 return getCleanShadow(V);
1852 // For instructions the shadow is already stored in the map.
1853 Value *Shadow = ShadowMap[V];
1854 if (!Shadow) {
1855 LLVM_DEBUG(dbgs() << "No shadow: " << *V << "\n" << *(I->getParent()));
1856 (void)I;
1857 assert(Shadow && "No shadow for a value");
1858 }
1859 return Shadow;
1860 }
1861 if (UndefValue *U = dyn_cast<UndefValue>(V)) {
1862 Value *AllOnes = (PropagateShadow && PoisonUndef) ? getPoisonedShadow(V)
1863 : getCleanShadow(V);
1864 LLVM_DEBUG(dbgs() << "Undef: " << *U << " ==> " << *AllOnes << "\n");
1865 (void)U;
1866 return AllOnes;
1867 }
1868 if (Argument *A = dyn_cast<Argument>(V)) {
1869 // For arguments we compute the shadow on demand and store it in the map.
1870 Value *&ShadowPtr = ShadowMap[V];
1871 if (ShadowPtr)
1872 return ShadowPtr;
1873 Function *F = A->getParent();
1874 IRBuilder<> EntryIRB(FnPrologueEnd);
1875 unsigned ArgOffset = 0;
1876 const DataLayout &DL = F->getParent()->getDataLayout();
1877 for (auto &FArg : F->args()) {
1878 if (!FArg.getType()->isSized()) {
1879 LLVM_DEBUG(dbgs() << "Arg is not sized\n");
1880 continue;
1881 }
1882
1883 unsigned Size = FArg.hasByValAttr()
1884 ? DL.getTypeAllocSize(FArg.getParamByValType())
1885 : DL.getTypeAllocSize(FArg.getType());
1886
1887 if (A == &FArg) {
1888 bool Overflow = ArgOffset + Size > kParamTLSSize;
1889 if (FArg.hasByValAttr()) {
1890 // ByVal pointer itself has clean shadow. We copy the actual
1891 // argument shadow to the underlying memory.
1892 // Figure out maximal valid memcpy alignment.
1893 const Align ArgAlign = DL.getValueOrABITypeAlignment(
1894 FArg.getParamAlign(), FArg.getParamByValType());
1895 Value *CpShadowPtr, *CpOriginPtr;
1896 std::tie(CpShadowPtr, CpOriginPtr) =
1897 getShadowOriginPtr(V, EntryIRB, EntryIRB.getInt8Ty(), ArgAlign,
1898 /*isStore*/ true);
1899 if (!PropagateShadow || Overflow) {
1900 // ParamTLS overflow.
1901 EntryIRB.CreateMemSet(
1902 CpShadowPtr, Constant::getNullValue(EntryIRB.getInt8Ty()),
1903 Size, ArgAlign);
1904 } else {
1905 Value *Base = getShadowPtrForArgument(&FArg, EntryIRB, ArgOffset);
1906 const Align CopyAlign = std::min(ArgAlign, kShadowTLSAlignment);
1907 Value *Cpy = EntryIRB.CreateMemCpy(CpShadowPtr, CopyAlign, Base,
1908 CopyAlign, Size);
1909 LLVM_DEBUG(dbgs() << " ByValCpy: " << *Cpy << "\n");
1910 (void)Cpy;
1911
1912 if (MS.TrackOrigins) {
1913 Value *OriginPtr =
1914 getOriginPtrForArgument(&FArg, EntryIRB, ArgOffset);
1915 // FIXME: OriginSize should be:
1916 // alignTo(V % kMinOriginAlignment + Size, kMinOriginAlignment)
1917 unsigned OriginSize = alignTo(Size, kMinOriginAlignment);
1918 EntryIRB.CreateMemCpy(
1919 CpOriginPtr,
1920 /* by getShadowOriginPtr */ kMinOriginAlignment, OriginPtr,
1921 /* by origin_tls[ArgOffset] */ kMinOriginAlignment,
1922 OriginSize);
1923 }
1924 }
1925 }
1926
1927 if (!PropagateShadow || Overflow || FArg.hasByValAttr() ||
1928 (MS.EagerChecks && FArg.hasAttribute(Attribute::NoUndef))) {
1929 ShadowPtr = getCleanShadow(V);
1930 setOrigin(A, getCleanOrigin());
1931 } else {
1932 // Shadow over TLS
1933 Value *Base = getShadowPtrForArgument(&FArg, EntryIRB, ArgOffset);
1934 ShadowPtr = EntryIRB.CreateAlignedLoad(getShadowTy(&FArg), Base,
1936 if (MS.TrackOrigins) {
1937 Value *OriginPtr =
1938 getOriginPtrForArgument(&FArg, EntryIRB, ArgOffset);
1939 setOrigin(A, EntryIRB.CreateLoad(MS.OriginTy, OriginPtr));
1940 }
1941 }
1943 << " ARG: " << FArg << " ==> " << *ShadowPtr << "\n");
1944 break;
1945 }
1946
1947 ArgOffset += alignTo(Size, kShadowTLSAlignment);
1948 }
1949 assert(ShadowPtr && "Could not find shadow for an argument");
1950 return ShadowPtr;
1951 }
1952 // For everything else the shadow is zero.
1953 return getCleanShadow(V);
1954 }
1955
1956 /// Get the shadow for i-th argument of the instruction I.
1957 Value *getShadow(Instruction *I, int i) {
1958 return getShadow(I->getOperand(i));
1959 }
1960
1961 /// Get the origin for a value.
1962 Value *getOrigin(Value *V) {
1963 if (!MS.TrackOrigins)
1964 return nullptr;
1965 if (!PropagateShadow || isa<Constant>(V) || isa<InlineAsm>(V))
1966 return getCleanOrigin();
1967 assert((isa<Instruction>(V) || isa<Argument>(V)) &&
1968 "Unexpected value type in getOrigin()");
1969 if (Instruction *I = dyn_cast<Instruction>(V)) {
1970 if (I->getMetadata(LLVMContext::MD_nosanitize))
1971 return getCleanOrigin();
1972 }
1973 Value *Origin = OriginMap[V];
1974 assert(Origin && "Missing origin");
1975 return Origin;
1976 }
1977
1978 /// Get the origin for i-th argument of the instruction I.
1979 Value *getOrigin(Instruction *I, int i) {
1980 return getOrigin(I->getOperand(i));
1981 }
1982
1983 /// Remember the place where a shadow check should be inserted.
1984 ///
1985 /// This location will be later instrumented with a check that will print a
1986 /// UMR warning in runtime if the shadow value is not 0.
1987 void insertShadowCheck(Value *Shadow, Value *Origin, Instruction *OrigIns) {
1988 assert(Shadow);
1989 if (!InsertChecks)
1990 return;
1991
1992 if (!DebugCounter::shouldExecute(DebugInsertCheck)) {
1993 LLVM_DEBUG(dbgs() << "Skipping check of " << *Shadow << " before "
1994 << *OrigIns << "\n");
1995 return;
1996 }
1997#ifndef NDEBUG
1998 Type *ShadowTy = Shadow->getType();
1999 assert((isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy) ||
2000 isa<StructType>(ShadowTy) || isa<ArrayType>(ShadowTy)) &&
2001 "Can only insert checks for integer, vector, and aggregate shadow "
2002 "types");
2003#endif
2004 InstrumentationList.push_back(
2005 ShadowOriginAndInsertPoint(Shadow, Origin, OrigIns));
2006 }
2007
2008 /// Remember the place where a shadow check should be inserted.
2009 ///
2010 /// This location will be later instrumented with a check that will print a
2011 /// UMR warning in runtime if the value is not fully defined.
2012 void insertShadowCheck(Value *Val, Instruction *OrigIns) {
2013 assert(Val);
2014 Value *Shadow, *Origin;
2016 Shadow = getShadow(Val);
2017 if (!Shadow)
2018 return;
2019 Origin = getOrigin(Val);
2020 } else {
2021 Shadow = dyn_cast_or_null<Instruction>(getShadow(Val));
2022 if (!Shadow)
2023 return;
2024 Origin = dyn_cast_or_null<Instruction>(getOrigin(Val));
2025 }
2026 insertShadowCheck(Shadow, Origin, OrigIns);
2027 }
2028
2030 switch (a) {
2031 case AtomicOrdering::NotAtomic:
2032 return AtomicOrdering::NotAtomic;
2033 case AtomicOrdering::Unordered:
2034 case AtomicOrdering::Monotonic:
2035 case AtomicOrdering::Release:
2036 return AtomicOrdering::Release;
2037 case AtomicOrdering::Acquire:
2038 case AtomicOrdering::AcquireRelease:
2039 return AtomicOrdering::AcquireRelease;
2040 case AtomicOrdering::SequentiallyConsistent:
2041 return AtomicOrdering::SequentiallyConsistent;
2042 }
2043 llvm_unreachable("Unknown ordering");
2044 }
2045
2046 Value *makeAddReleaseOrderingTable(IRBuilder<> &IRB) {
2047 constexpr int NumOrderings = (int)AtomicOrderingCABI::seq_cst + 1;
2048 uint32_t OrderingTable[NumOrderings] = {};
2049
2050 OrderingTable[(int)AtomicOrderingCABI::relaxed] =
2051 OrderingTable[(int)AtomicOrderingCABI::release] =
2052 (int)AtomicOrderingCABI::release;
2053 OrderingTable[(int)AtomicOrderingCABI::consume] =
2054 OrderingTable[(int)AtomicOrderingCABI::acquire] =
2055 OrderingTable[(int)AtomicOrderingCABI::acq_rel] =
2056 (int)AtomicOrderingCABI::acq_rel;
2057 OrderingTable[(int)AtomicOrderingCABI::seq_cst] =
2058 (int)AtomicOrderingCABI::seq_cst;
2059
2061 ArrayRef(OrderingTable, NumOrderings));
2062 }
2063
2065 switch (a) {
2066 case AtomicOrdering::NotAtomic:
2067 return AtomicOrdering::NotAtomic;
2068 case AtomicOrdering::Unordered:
2069 case AtomicOrdering::Monotonic:
2070 case AtomicOrdering::Acquire:
2071 return AtomicOrdering::Acquire;
2072 case AtomicOrdering::Release:
2073 case AtomicOrdering::AcquireRelease:
2074 return AtomicOrdering::AcquireRelease;
2075 case AtomicOrdering::SequentiallyConsistent:
2076 return AtomicOrdering::SequentiallyConsistent;
2077 }
2078 llvm_unreachable("Unknown ordering");
2079 }
2080
2081 Value *makeAddAcquireOrderingTable(IRBuilder<> &IRB) {
2082 constexpr int NumOrderings = (int)AtomicOrderingCABI::seq_cst + 1;
2083 uint32_t OrderingTable[NumOrderings] = {};
2084
2085 OrderingTable[(int)AtomicOrderingCABI::relaxed] =
2086 OrderingTable[(int)AtomicOrderingCABI::acquire] =
2087 OrderingTable[(int)AtomicOrderingCABI::consume] =
2088 (int)AtomicOrderingCABI::acquire;
2089 OrderingTable[(int)AtomicOrderingCABI::release] =
2090 OrderingTable[(int)AtomicOrderingCABI::acq_rel] =
2091 (int)AtomicOrderingCABI::acq_rel;
2092 OrderingTable[(int)AtomicOrderingCABI::seq_cst] =
2093 (int)AtomicOrderingCABI::seq_cst;
2094
2096 ArrayRef(OrderingTable, NumOrderings));
2097 }
2098
2099 // ------------------- Visitors.
2100 using InstVisitor<MemorySanitizerVisitor>::visit;
2101 void visit(Instruction &I) {
2102 if (I.getMetadata(LLVMContext::MD_nosanitize))
2103 return;
2104 // Don't want to visit if we're in the prologue
2105 if (isInPrologue(I))
2106 return;
2108 }
2109
2110 /// Instrument LoadInst
2111 ///
2112 /// Loads the corresponding shadow and (optionally) origin.
2113 /// Optionally, checks that the load address is fully defined.
2114 void visitLoadInst(LoadInst &I) {
2115 assert(I.getType()->isSized() && "Load type must have size");
2116 assert(!I.getMetadata(LLVMContext::MD_nosanitize));
2117 NextNodeIRBuilder IRB(&I);
2118 Type *ShadowTy = getShadowTy(&I);
2119 Value *Addr = I.getPointerOperand();
2120 Value *ShadowPtr = nullptr, *OriginPtr = nullptr;
2121 const Align Alignment = I.getAlign();
2122 if (PropagateShadow) {
2123 std::tie(ShadowPtr, OriginPtr) =
2124 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
2125 setShadow(&I,
2126 IRB.CreateAlignedLoad(ShadowTy, ShadowPtr, Alignment, "_msld"));
2127 } else {
2128 setShadow(&I, getCleanShadow(&I));
2129 }
2130
2132 insertShadowCheck(I.getPointerOperand(), &I);
2133
2134 if (I.isAtomic())
2135 I.setOrdering(addAcquireOrdering(I.getOrdering()));
2136
2137 if (MS.TrackOrigins) {
2138 if (PropagateShadow) {
2139 const Align OriginAlignment = std::max(kMinOriginAlignment, Alignment);
2140 setOrigin(
2141 &I, IRB.CreateAlignedLoad(MS.OriginTy, OriginPtr, OriginAlignment));
2142 } else {
2143 setOrigin(&I, getCleanOrigin());
2144 }
2145 }
2146 }
2147
2148 /// Instrument StoreInst
2149 ///
2150 /// Stores the corresponding shadow and (optionally) origin.
2151 /// Optionally, checks that the store address is fully defined.
2152 void visitStoreInst(StoreInst &I) {
2153 StoreList.push_back(&I);
2155 insertShadowCheck(I.getPointerOperand(), &I);
2156 }
2157
2158 void handleCASOrRMW(Instruction &I) {
2159 assert(isa<AtomicRMWInst>(I) || isa<AtomicCmpXchgInst>(I));
2160
2161 IRBuilder<> IRB(&I);
2162 Value *Addr = I.getOperand(0);
2163 Value *Val = I.getOperand(1);
2164 Value *ShadowPtr = getShadowOriginPtr(Addr, IRB, getShadowTy(Val), Align(1),
2165 /*isStore*/ true)
2166 .first;
2167
2169 insertShadowCheck(Addr, &I);
2170
2171 // Only test the conditional argument of cmpxchg instruction.
2172 // The other argument can potentially be uninitialized, but we can not
2173 // detect this situation reliably without possible false positives.
2174 if (isa<AtomicCmpXchgInst>(I))
2175 insertShadowCheck(Val, &I);
2176
2177 IRB.CreateStore(getCleanShadow(Val), ShadowPtr);
2178
2179 setShadow(&I, getCleanShadow(&I));
2180 setOrigin(&I, getCleanOrigin());
2181 }
2182
2183 void visitAtomicRMWInst(AtomicRMWInst &I) {
2184 handleCASOrRMW(I);
2185 I.setOrdering(addReleaseOrdering(I.getOrdering()));
2186 }
2187
2188 void visitAtomicCmpXchgInst(AtomicCmpXchgInst &I) {
2189 handleCASOrRMW(I);
2190 I.setSuccessOrdering(addReleaseOrdering(I.getSuccessOrdering()));
2191 }
2192
2193 // Vector manipulation.
2194 void visitExtractElementInst(ExtractElementInst &I) {
2195 insertShadowCheck(I.getOperand(1), &I);
2196 IRBuilder<> IRB(&I);
2197 setShadow(&I, IRB.CreateExtractElement(getShadow(&I, 0), I.getOperand(1),
2198 "_msprop"));
2199 setOrigin(&I, getOrigin(&I, 0));
2200 }
2201
2202 void visitInsertElementInst(InsertElementInst &I) {
2203 insertShadowCheck(I.getOperand(2), &I);
2204 IRBuilder<> IRB(&I);
2205 auto *Shadow0 = getShadow(&I, 0);
2206 auto *Shadow1 = getShadow(&I, 1);
2207 setShadow(&I, IRB.CreateInsertElement(Shadow0, Shadow1, I.getOperand(2),
2208 "_msprop"));
2209 setOriginForNaryOp(I);
2210 }
2211
2212 void visitShuffleVectorInst(ShuffleVectorInst &I) {
2213 IRBuilder<> IRB(&I);
2214 auto *Shadow0 = getShadow(&I, 0);
2215 auto *Shadow1 = getShadow(&I, 1);
2216 setShadow(&I, IRB.CreateShuffleVector(Shadow0, Shadow1, I.getShuffleMask(),
2217 "_msprop"));
2218 setOriginForNaryOp(I);
2219 }
2220
2221 // Casts.
2222 void visitSExtInst(SExtInst &I) {
2223 IRBuilder<> IRB(&I);
2224 setShadow(&I, IRB.CreateSExt(getShadow(&I, 0), I.getType(), "_msprop"));
2225 setOrigin(&I, getOrigin(&I, 0));
2226 }
2227
2228 void visitZExtInst(ZExtInst &I) {
2229 IRBuilder<> IRB(&I);
2230 setShadow(&I, IRB.CreateZExt(getShadow(&I, 0), I.getType(), "_msprop"));
2231 setOrigin(&I, getOrigin(&I, 0));
2232 }
2233
2234 void visitTruncInst(TruncInst &I) {
2235 IRBuilder<> IRB(&I);
2236 setShadow(&I, IRB.CreateTrunc(getShadow(&I, 0), I.getType(), "_msprop"));
2237 setOrigin(&I, getOrigin(&I, 0));
2238 }
2239
2240 void visitBitCastInst(BitCastInst &I) {
2241 // Special case: if this is the bitcast (there is exactly 1 allowed) between
2242 // a musttail call and a ret, don't instrument. New instructions are not
2243 // allowed after a musttail call.
2244 if (auto *CI = dyn_cast<CallInst>(I.getOperand(0)))
2245 if (CI->isMustTailCall())
2246 return;
2247 IRBuilder<> IRB(&I);
2248 setShadow(&I, IRB.CreateBitCast(getShadow(&I, 0), getShadowTy(&I)));
2249 setOrigin(&I, getOrigin(&I, 0));
2250 }
2251
2252 void visitPtrToIntInst(PtrToIntInst &I) {
2253 IRBuilder<> IRB(&I);
2254 setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
2255 "_msprop_ptrtoint"));
2256 setOrigin(&I, getOrigin(&I, 0));
2257 }
2258
2259 void visitIntToPtrInst(IntToPtrInst &I) {
2260 IRBuilder<> IRB(&I);
2261 setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
2262 "_msprop_inttoptr"));
2263 setOrigin(&I, getOrigin(&I, 0));
2264 }
2265
2266 void visitFPToSIInst(CastInst &I) { handleShadowOr(I); }
2267 void visitFPToUIInst(CastInst &I) { handleShadowOr(I); }
2268 void visitSIToFPInst(CastInst &I) { handleShadowOr(I); }
2269 void visitUIToFPInst(CastInst &I) { handleShadowOr(I); }
2270 void visitFPExtInst(CastInst &I) { handleShadowOr(I); }
2271 void visitFPTruncInst(CastInst &I) { handleShadowOr(I); }
2272
2273 /// Propagate shadow for bitwise AND.
2274 ///
2275 /// This code is exact, i.e. if, for example, a bit in the left argument
2276 /// is defined and 0, then neither the value not definedness of the
2277 /// corresponding bit in B don't affect the resulting shadow.
2278 void visitAnd(BinaryOperator &I) {
2279 IRBuilder<> IRB(&I);
2280 // "And" of 0 and a poisoned value results in unpoisoned value.
2281 // 1&1 => 1; 0&1 => 0; p&1 => p;
2282 // 1&0 => 0; 0&0 => 0; p&0 => 0;
2283 // 1&p => p; 0&p => 0; p&p => p;
2284 // S = (S1 & S2) | (V1 & S2) | (S1 & V2)
2285 Value *S1 = getShadow(&I, 0);
2286 Value *S2 = getShadow(&I, 1);
2287 Value *V1 = I.getOperand(0);
2288 Value *V2 = I.getOperand(1);
2289 if (V1->getType() != S1->getType()) {
2290 V1 = IRB.CreateIntCast(V1, S1->getType(), false);
2291 V2 = IRB.CreateIntCast(V2, S2->getType(), false);
2292 }
2293 Value *S1S2 = IRB.CreateAnd(S1, S2);
2294 Value *V1S2 = IRB.CreateAnd(V1, S2);
2295 Value *S1V2 = IRB.CreateAnd(S1, V2);
2296 setShadow(&I, IRB.CreateOr({S1S2, V1S2, S1V2}));
2297 setOriginForNaryOp(I);
2298 }
2299
2300 void visitOr(BinaryOperator &I) {
2301 IRBuilder<> IRB(&I);
2302 // "Or" of 1 and a poisoned value results in unpoisoned value.
2303 // 1|1 => 1; 0|1 => 1; p|1 => 1;
2304 // 1|0 => 1; 0|0 => 0; p|0 => p;
2305 // 1|p => 1; 0|p => p; p|p => p;
2306 // S = (S1 & S2) | (~V1 & S2) | (S1 & ~V2)
2307 Value *S1 = getShadow(&I, 0);
2308 Value *S2 = getShadow(&I, 1);
2309 Value *V1 = IRB.CreateNot(I.getOperand(0));
2310 Value *V2 = IRB.CreateNot(I.getOperand(1));
2311 if (V1->getType() != S1->getType()) {
2312 V1 = IRB.CreateIntCast(V1, S1->getType(), false);
2313 V2 = IRB.CreateIntCast(V2, S2->getType(), false);
2314 }
2315 Value *S1S2 = IRB.CreateAnd(S1, S2);
2316 Value *V1S2 = IRB.CreateAnd(V1, S2);
2317 Value *S1V2 = IRB.CreateAnd(S1, V2);
2318 setShadow(&I, IRB.CreateOr({S1S2, V1S2, S1V2}));
2319 setOriginForNaryOp(I);
2320 }
2321
2322 /// Default propagation of shadow and/or origin.
2323 ///
2324 /// This class implements the general case of shadow propagation, used in all
2325 /// cases where we don't know and/or don't care about what the operation
2326 /// actually does. It converts all input shadow values to a common type
2327 /// (extending or truncating as necessary), and bitwise OR's them.
2328 ///
2329 /// This is much cheaper than inserting checks (i.e. requiring inputs to be
2330 /// fully initialized), and less prone to false positives.
2331 ///
2332 /// This class also implements the general case of origin propagation. For a
2333 /// Nary operation, result origin is set to the origin of an argument that is
2334 /// not entirely initialized. If there is more than one such arguments, the
2335 /// rightmost of them is picked. It does not matter which one is picked if all
2336 /// arguments are initialized.
2337 template <bool CombineShadow> class Combiner {
2338 Value *Shadow = nullptr;
2339 Value *Origin = nullptr;
2340 IRBuilder<> &IRB;
2341 MemorySanitizerVisitor *MSV;
2342
2343 public:
2344 Combiner(MemorySanitizerVisitor *MSV, IRBuilder<> &IRB)
2345 : IRB(IRB), MSV(MSV) {}
2346
2347 /// Add a pair of shadow and origin values to the mix.
2348 Combiner &Add(Value *OpShadow, Value *OpOrigin) {
2349 if (CombineShadow) {
2350 assert(OpShadow);
2351 if (!Shadow)
2352 Shadow = OpShadow;
2353 else {
2354 OpShadow = MSV->CreateShadowCast(IRB, OpShadow, Shadow->getType());
2355 Shadow = IRB.CreateOr(Shadow, OpShadow, "_msprop");
2356 }
2357 }
2358
2359 if (MSV->MS.TrackOrigins) {
2360 assert(OpOrigin);
2361 if (!Origin) {
2362 Origin = OpOrigin;
2363 } else {
2364 Constant *ConstOrigin = dyn_cast<Constant>(OpOrigin);
2365 // No point in adding something that might result in 0 origin value.
2366 if (!ConstOrigin || !ConstOrigin->isNullValue()) {
2367 Value *Cond = MSV->convertToBool(OpShadow, IRB);
2368 Origin = IRB.CreateSelect(Cond, OpOrigin, Origin);
2369 }
2370 }
2371 }
2372 return *this;
2373 }
2374
2375 /// Add an application value to the mix.
2376 Combiner &Add(Value *V) {
2377 Value *OpShadow = MSV->getShadow(V);
2378 Value *OpOrigin = MSV->MS.TrackOrigins ? MSV->getOrigin(V) : nullptr;
2379 return Add(OpShadow, OpOrigin);
2380 }
2381
2382 /// Set the current combined values as the given instruction's shadow
2383 /// and origin.
2384 void Done(Instruction *I) {
2385 if (CombineShadow) {
2386 assert(Shadow);
2387 Shadow = MSV->CreateShadowCast(IRB, Shadow, MSV->getShadowTy(I));
2388 MSV->setShadow(I, Shadow);
2389 }
2390 if (MSV->MS.TrackOrigins) {
2391 assert(Origin);
2392 MSV->setOrigin(I, Origin);
2393 }
2394 }
2395 };
2396
2397 using ShadowAndOriginCombiner = Combiner<true>;
2398 using OriginCombiner = Combiner<false>;
2399
2400 /// Propagate origin for arbitrary operation.
2401 void setOriginForNaryOp(Instruction &I) {
2402 if (!MS.TrackOrigins)
2403 return;
2404 IRBuilder<> IRB(&I);
2405 OriginCombiner OC(this, IRB);
2406 for (Use &Op : I.operands())
2407 OC.Add(Op.get());
2408 OC.Done(&I);
2409 }
2410
2411 size_t VectorOrPrimitiveTypeSizeInBits(Type *Ty) {
2412 assert(!(Ty->isVectorTy() && Ty->getScalarType()->isPointerTy()) &&
2413 "Vector of pointers is not a valid shadow type");
2414 return Ty->isVectorTy() ? cast<FixedVectorType>(Ty)->getNumElements() *
2416 : Ty->getPrimitiveSizeInBits();
2417 }
2418
2419 /// Cast between two shadow types, extending or truncating as
2420 /// necessary.
2421 Value *CreateShadowCast(IRBuilder<> &IRB, Value *V, Type *dstTy,
2422 bool Signed = false) {
2423 Type *srcTy = V->getType();
2424 size_t srcSizeInBits = VectorOrPrimitiveTypeSizeInBits(srcTy);
2425 size_t dstSizeInBits = VectorOrPrimitiveTypeSizeInBits(dstTy);
2426 if (srcSizeInBits > 1 && dstSizeInBits == 1)
2427 return IRB.CreateICmpNE(V, getCleanShadow(V));
2428
2429 if (dstTy->isIntegerTy() && srcTy->isIntegerTy())
2430 return IRB.CreateIntCast(V, dstTy, Signed);
2431 if (dstTy->isVectorTy() && srcTy->isVectorTy() &&
2432 cast<VectorType>(dstTy)->getElementCount() ==
2433 cast<VectorType>(srcTy)->getElementCount())
2434 return IRB.CreateIntCast(V, dstTy, Signed);
2435 Value *V1 = IRB.CreateBitCast(V, Type::getIntNTy(*MS.C, srcSizeInBits));
2436 Value *V2 =
2437 IRB.CreateIntCast(V1, Type::getIntNTy(*MS.C, dstSizeInBits), Signed);
2438 return IRB.CreateBitCast(V2, dstTy);
2439 // TODO: handle struct types.
2440 }
2441
2442 /// Cast an application value to the type of its own shadow.
2443 Value *CreateAppToShadowCast(IRBuilder<> &IRB, Value *V) {
2444 Type *ShadowTy = getShadowTy(V);
2445 if (V->getType() == ShadowTy)
2446 return V;
2447 if (V->getType()->isPtrOrPtrVectorTy())
2448 return IRB.CreatePtrToInt(V, ShadowTy);
2449 else
2450 return IRB.CreateBitCast(V, ShadowTy);
2451 }
2452
2453 /// Propagate shadow for arbitrary operation.
2454 void handleShadowOr(Instruction &I) {
2455 IRBuilder<> IRB(&I);
2456 ShadowAndOriginCombiner SC(this, IRB);
2457 for (Use &Op : I.operands())
2458 SC.Add(Op.get());
2459 SC.Done(&I);
2460 }
2461
2462 void visitFNeg(UnaryOperator &I) { handleShadowOr(I); }
2463
2464 // Handle multiplication by constant.
2465 //
2466 // Handle a special case of multiplication by constant that may have one or
2467 // more zeros in the lower bits. This makes corresponding number of lower bits
2468 // of the result zero as well. We model it by shifting the other operand
2469 // shadow left by the required number of bits. Effectively, we transform
2470 // (X * (A * 2**B)) to ((X << B) * A) and instrument (X << B) as (Sx << B).
2471 // We use multiplication by 2**N instead of shift to cover the case of
2472 // multiplication by 0, which may occur in some elements of a vector operand.
2473 void handleMulByConstant(BinaryOperator &I, Constant *ConstArg,
2474 Value *OtherArg) {
2475 Constant *ShadowMul;
2476 Type *Ty = ConstArg->getType();
2477 if (auto *VTy = dyn_cast<VectorType>(Ty)) {
2478 unsigned NumElements = cast<FixedVectorType>(VTy)->getNumElements();
2479 Type *EltTy = VTy->getElementType();
2481 for (unsigned Idx = 0; Idx < NumElements; ++Idx) {
2482 if (ConstantInt *Elt =
2483 dyn_cast<ConstantInt>(ConstArg->getAggregateElement(Idx))) {
2484 const APInt &V = Elt->getValue();
2485 APInt V2 = APInt(V.getBitWidth(), 1) << V.countr_zero();
2486 Elements.push_back(ConstantInt::get(EltTy, V2));
2487 } else {
2488 Elements.push_back(ConstantInt::get(EltTy, 1));
2489 }
2490 }
2491 ShadowMul = ConstantVector::get(Elements);
2492 } else {
2493 if (ConstantInt *Elt = dyn_cast<ConstantInt>(ConstArg)) {
2494 const APInt &V = Elt->getValue();
2495 APInt V2 = APInt(V.getBitWidth(), 1) << V.countr_zero();
2496 ShadowMul = ConstantInt::get(Ty, V2);
2497 } else {
2498 ShadowMul = ConstantInt::get(Ty, 1);
2499 }
2500 }
2501
2502 IRBuilder<> IRB(&I);
2503 setShadow(&I,
2504 IRB.CreateMul(getShadow(OtherArg), ShadowMul, "msprop_mul_cst"));
2505 setOrigin(&I, getOrigin(OtherArg));
2506 }
2507
2508 void visitMul(BinaryOperator &I) {
2509 Constant *constOp0 = dyn_cast<Constant>(I.getOperand(0));
2510 Constant *constOp1 = dyn_cast<Constant>(I.getOperand(1));
2511 if (constOp0 && !constOp1)
2512 handleMulByConstant(I, constOp0, I.getOperand(1));
2513 else if (constOp1 && !constOp0)
2514 handleMulByConstant(I, constOp1, I.getOperand(0));
2515 else
2516 handleShadowOr(I);
2517 }
2518
2519 void visitFAdd(BinaryOperator &I) { handleShadowOr(I); }
2520 void visitFSub(BinaryOperator &I) { handleShadowOr(I); }
2521 void visitFMul(BinaryOperator &I) { handleShadowOr(I); }
2522 void visitAdd(BinaryOperator &I) { handleShadowOr(I); }
2523 void visitSub(BinaryOperator &I) { handleShadowOr(I); }
2524 void visitXor(BinaryOperator &I) { handleShadowOr(I); }
2525
2526 void handleIntegerDiv(Instruction &I) {
2527 IRBuilder<> IRB(&I);
2528 // Strict on the second argument.
2529 insertShadowCheck(I.getOperand(1), &I);
2530 setShadow(&I, getShadow(&I, 0));
2531 setOrigin(&I, getOrigin(&I, 0));
2532 }
2533
2534 void visitUDiv(BinaryOperator &I) { handleIntegerDiv(I); }
2535 void visitSDiv(BinaryOperator &I) { handleIntegerDiv(I); }
2536 void visitURem(BinaryOperator &I) { handleIntegerDiv(I); }
2537 void visitSRem(BinaryOperator &I) { handleIntegerDiv(I); }
2538
2539 // Floating point division is side-effect free. We can not require that the
2540 // divisor is fully initialized and must propagate shadow. See PR37523.
2541 void visitFDiv(BinaryOperator &I) { handleShadowOr(I); }
2542 void visitFRem(BinaryOperator &I) { handleShadowOr(I); }
2543
2544 /// Instrument == and != comparisons.
2545 ///
2546 /// Sometimes the comparison result is known even if some of the bits of the
2547 /// arguments are not.
2548 void handleEqualityComparison(ICmpInst &I) {
2549 IRBuilder<> IRB(&I);
2550 Value *A = I.getOperand(0);
2551 Value *B = I.getOperand(1);
2552 Value *Sa = getShadow(A);
2553 Value *Sb = getShadow(B);
2554
2555 // Get rid of pointers and vectors of pointers.
2556 // For ints (and vectors of ints), types of A and Sa match,
2557 // and this is a no-op.
2558 A = IRB.CreatePointerCast(A, Sa->getType());
2559 B = IRB.CreatePointerCast(B, Sb->getType());
2560
2561 // A == B <==> (C = A^B) == 0
2562 // A != B <==> (C = A^B) != 0
2563 // Sc = Sa | Sb
2564 Value *C = IRB.CreateXor(A, B);
2565 Value *Sc = IRB.CreateOr(Sa, Sb);
2566 // Now dealing with i = (C == 0) comparison (or C != 0, does not matter now)
2567 // Result is defined if one of the following is true
2568 // * there is a defined 1 bit in C
2569 // * C is fully defined
2570 // Si = !(C & ~Sc) && Sc
2572 Value *MinusOne = Constant::getAllOnesValue(Sc->getType());
2573 Value *LHS = IRB.CreateICmpNE(Sc, Zero);
2574 Value *RHS =
2575 IRB.CreateICmpEQ(IRB.CreateAnd(IRB.CreateXor(Sc, MinusOne), C), Zero);
2576 Value *Si = IRB.CreateAnd(LHS, RHS);
2577 Si->setName("_msprop_icmp");
2578 setShadow(&I, Si);
2579 setOriginForNaryOp(I);
2580 }
2581
2582 /// Build the lowest possible value of V, taking into account V's
2583 /// uninitialized bits.
2584 Value *getLowestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2585 bool isSigned) {
2586 if (isSigned) {
2587 // Split shadow into sign bit and other bits.
2588 Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2589 Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2590 // Maximise the undefined shadow bit, minimize other undefined bits.
2591 return IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaOtherBits)),
2592 SaSignBit);
2593 } else {
2594 // Minimize undefined bits.
2595 return IRB.CreateAnd(A, IRB.CreateNot(Sa));
2596 }
2597 }
2598
2599 /// Build the highest possible value of V, taking into account V's
2600 /// uninitialized bits.
2601 Value *getHighestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2602 bool isSigned) {
2603 if (isSigned) {
2604 // Split shadow into sign bit and other bits.
2605 Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2606 Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2607 // Minimise the undefined shadow bit, maximise other undefined bits.
2608 return IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaSignBit)),
2609 SaOtherBits);
2610 } else {
2611 // Maximize undefined bits.
2612 return IRB.CreateOr(A, Sa);
2613 }
2614 }
2615
2616 /// Instrument relational comparisons.
2617 ///
2618 /// This function does exact shadow propagation for all relational
2619 /// comparisons of integers, pointers and vectors of those.
2620 /// FIXME: output seems suboptimal when one of the operands is a constant
2621 void handleRelationalComparisonExact(ICmpInst &I) {
2622 IRBuilder<> IRB(&I);
2623 Value *A = I.getOperand(0);
2624 Value *B = I.getOperand(1);
2625 Value *Sa = getShadow(A);
2626 Value *Sb = getShadow(B);
2627
2628 // Get rid of pointers and vectors of pointers.
2629 // For ints (and vectors of ints), types of A and Sa match,
2630 // and this is a no-op.
2631 A = IRB.CreatePointerCast(A, Sa->getType());
2632 B = IRB.CreatePointerCast(B, Sb->getType());
2633
2634 // Let [a0, a1] be the interval of possible values of A, taking into account
2635 // its undefined bits. Let [b0, b1] be the interval of possible values of B.
2636 // Then (A cmp B) is defined iff (a0 cmp b1) == (a1 cmp b0).
2637 bool IsSigned = I.isSigned();
2638 Value *S1 = IRB.CreateICmp(I.getPredicate(),
2639 getLowestPossibleValue(IRB, A, Sa, IsSigned),
2640 getHighestPossibleValue(IRB, B, Sb, IsSigned));
2641 Value *S2 = IRB.CreateICmp(I.getPredicate(),
2642 getHighestPossibleValue(IRB, A, Sa, IsSigned),
2643 getLowestPossibleValue(IRB, B, Sb, IsSigned));
2644 Value *Si = IRB.CreateXor(S1, S2);
2645 setShadow(&I, Si);
2646 setOriginForNaryOp(I);
2647 }
2648
2649 /// Instrument signed relational comparisons.
2650 ///
2651 /// Handle sign bit tests: x<0, x>=0, x<=-1, x>-1 by propagating the highest
2652 /// bit of the shadow. Everything else is delegated to handleShadowOr().
2653 void handleSignedRelationalComparison(ICmpInst &I) {
2654 Constant *constOp;
2655 Value *op = nullptr;
2657 if ((constOp = dyn_cast<Constant>(I.getOperand(1)))) {
2658 op = I.getOperand(0);
2659 pre = I.getPredicate();
2660 } else if ((constOp = dyn_cast<Constant>(I.getOperand(0)))) {
2661 op = I.getOperand(1);
2662 pre = I.getSwappedPredicate();
2663 } else {
2664 handleShadowOr(I);
2665 return;
2666 }
2667
2668 if ((constOp->isNullValue() &&
2669 (pre == CmpInst::ICMP_SLT || pre == CmpInst::ICMP_SGE)) ||
2670 (constOp->isAllOnesValue() &&
2671 (pre == CmpInst::ICMP_SGT || pre == CmpInst::ICMP_SLE))) {
2672 IRBuilder<> IRB(&I);
2673 Value *Shadow = IRB.CreateICmpSLT(getShadow(op), getCleanShadow(op),
2674 "_msprop_icmp_s");
2675 setShadow(&I, Shadow);
2676 setOrigin(&I, getOrigin(op));
2677 } else {
2678 handleShadowOr(I);
2679 }
2680 }
2681
2682 void visitICmpInst(ICmpInst &I) {
2683 if (!ClHandleICmp) {
2684 handleShadowOr(I);
2685 return;
2686 }
2687 if (I.isEquality()) {
2688 handleEqualityComparison(I);
2689 return;
2690 }
2691
2692 assert(I.isRelational());
2693 if (ClHandleICmpExact) {
2694 handleRelationalComparisonExact(I);
2695 return;
2696 }
2697 if (I.isSigned()) {
2698 handleSignedRelationalComparison(I);
2699 return;
2700 }
2701
2702 assert(I.isUnsigned());
2703 if ((isa<Constant>(I.getOperand(0)) || isa<Constant>(I.getOperand(1)))) {
2704 handleRelationalComparisonExact(I);
2705 return;
2706 }
2707
2708 handleShadowOr(I);
2709 }
2710
2711 void visitFCmpInst(FCmpInst &I) { handleShadowOr(I); }
2712
2713 void handleShift(BinaryOperator &I) {
2714 IRBuilder<> IRB(&I);
2715 // If any of the S2 bits are poisoned, the whole thing is poisoned.
2716 // Otherwise perform the same shift on S1.
2717 Value *S1 = getShadow(&I, 0);
2718 Value *S2 = getShadow(&I, 1);
2719 Value *S2Conv =
2720 IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)), S2->getType());
2721 Value *V2 = I.getOperand(1);
2722 Value *Shift = IRB.CreateBinOp(I.getOpcode(), S1, V2);
2723 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2724 setOriginForNaryOp(I);
2725 }
2726
2727 void visitShl(BinaryOperator &I) { handleShift(I); }
2728 void visitAShr(BinaryOperator &I) { handleShift(I); }
2729 void visitLShr(BinaryOperator &I) { handleShift(I); }
2730
2731 void handleFunnelShift(IntrinsicInst &I) {
2732 IRBuilder<> IRB(&I);
2733 // If any of the S2 bits are poisoned, the whole thing is poisoned.
2734 // Otherwise perform the same shift on S0 and S1.
2735 Value *S0 = getShadow(&I, 0);
2736 Value *S1 = getShadow(&I, 1);
2737 Value *S2 = getShadow(&I, 2);
2738 Value *S2Conv =
2739 IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)), S2->getType());
2740 Value *V2 = I.getOperand(2);
2742 I.getModule(), I.getIntrinsicID(), S2Conv->getType());
2743 Value *Shift = IRB.CreateCall(Intrin, {S0, S1, V2});
2744 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2745 setOriginForNaryOp(I);
2746 }
2747
2748 /// Instrument llvm.memmove
2749 ///
2750 /// At this point we don't know if llvm.memmove will be inlined or not.
2751 /// If we don't instrument it and it gets inlined,
2752 /// our interceptor will not kick in and we will lose the memmove.
2753 /// If we instrument the call here, but it does not get inlined,
2754 /// we will memove the shadow twice: which is bad in case
2755 /// of overlapping regions. So, we simply lower the intrinsic to a call.
2756 ///
2757 /// Similar situation exists for memcpy and memset.
2758 void visitMemMoveInst(MemMoveInst &I) {
2759 getShadow(I.getArgOperand(1)); // Ensure shadow initialized
2760 IRBuilder<> IRB(&I);
2761 IRB.CreateCall(
2762 MS.MemmoveFn,
2763 {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2764 IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()),
2765 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2766 I.eraseFromParent();
2767 }
2768
2769 /// Instrument memcpy
2770 ///
2771 /// Similar to memmove: avoid copying shadow twice. This is somewhat
2772 /// unfortunate as it may slowdown small constant memcpys.
2773 /// FIXME: consider doing manual inline for small constant sizes and proper
2774 /// alignment.
2775 ///
2776 /// Note: This also handles memcpy.inline, which promises no calls to external
2777 /// functions as an optimization. However, with instrumentation enabled this
2778 /// is difficult to promise; additionally, we know that the MSan runtime
2779 /// exists and provides __msan_memcpy(). Therefore, we assume that with
2780 /// instrumentation it's safe to turn memcpy.inline into a call to
2781 /// __msan_memcpy(). Should this be wrong, such as when implementing memcpy()
2782 /// itself, instrumentation should be disabled with the no_sanitize attribute.
2783 void visitMemCpyInst(MemCpyInst &I) {
2784 getShadow(I.getArgOperand(1)); // Ensure shadow initialized
2785 IRBuilder<> IRB(&I);
2786 IRB.CreateCall(
2787 MS.MemcpyFn,
2788 {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2789 IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()),
2790 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2791 I.eraseFromParent();
2792 }
2793
2794 // Same as memcpy.
2795 void visitMemSetInst(MemSetInst &I) {
2796 IRBuilder<> IRB(&I);
2797 IRB.CreateCall(
2798 MS.MemsetFn,
2799 {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2800 IRB.CreateIntCast(I.getArgOperand(1), IRB.getInt32Ty(), false),
2801 IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2802 I.eraseFromParent();
2803 }
2804
2805 void visitVAStartInst(VAStartInst &I) { VAHelper->visitVAStartInst(I); }
2806
2807 void visitVACopyInst(VACopyInst &I) { VAHelper->visitVACopyInst(I); }
2808
2809 /// Handle vector store-like intrinsics.
2810 ///
2811 /// Instrument intrinsics that look like a simple SIMD store: writes memory,
2812 /// has 1 pointer argument and 1 vector argument, returns void.
2813 bool handleVectorStoreIntrinsic(IntrinsicInst &I) {
2814 IRBuilder<> IRB(&I);
2815 Value *Addr = I.getArgOperand(0);
2816 Value *Shadow = getShadow(&I, 1);
2817 Value *ShadowPtr, *OriginPtr;
2818
2819 // We don't know the pointer alignment (could be unaligned SSE store!).
2820 // Have to assume to worst case.
2821 std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
2822 Addr, IRB, Shadow->getType(), Align(1), /*isStore*/ true);
2823 IRB.CreateAlignedStore(Shadow, ShadowPtr, Align(1));
2824
2826 insertShadowCheck(Addr, &I);
2827
2828 // FIXME: factor out common code from materializeStores
2829 if (MS.TrackOrigins)
2830 IRB.CreateStore(getOrigin(&I, 1), OriginPtr);
2831 return true;
2832 }
2833
2834 /// Handle vector load-like intrinsics.
2835 ///
2836 /// Instrument intrinsics that look like a simple SIMD load: reads memory,
2837 /// has 1 pointer argument, returns a vector.
2838 bool handleVectorLoadIntrinsic(IntrinsicInst &I) {
2839 IRBuilder<> IRB(&I);
2840 Value *Addr = I.getArgOperand(0);
2841
2842 Type *ShadowTy = getShadowTy(&I);
2843 Value *ShadowPtr = nullptr, *OriginPtr = nullptr;
2844 if (PropagateShadow) {
2845 // We don't know the pointer alignment (could be unaligned SSE load!).
2846 // Have to assume to worst case.
2847 const Align Alignment = Align(1);
2848 std::tie(ShadowPtr, OriginPtr) =
2849 getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
2850 setShadow(&I,
2851 IRB.CreateAlignedLoad(ShadowTy, ShadowPtr, Alignment, "_msld"));
2852 } else {
2853 setShadow(&I, getCleanShadow(&I));
2854 }
2855
2857 insertShadowCheck(Addr, &I);
2858
2859 if (MS.TrackOrigins) {
2860 if (PropagateShadow)
2861 setOrigin(&I, IRB.CreateLoad(MS.OriginTy, OriginPtr));
2862 else
2863 setOrigin(&I, getCleanOrigin());
2864 }
2865 return true;
2866 }
2867
2868 /// Handle (SIMD arithmetic)-like intrinsics.
2869 ///
2870 /// Instrument intrinsics with any number of arguments of the same type,
2871 /// equal to the return type. The type should be simple (no aggregates or
2872 /// pointers; vectors are fine).
2873 /// Caller guarantees that this intrinsic does not access memory.
2874 bool maybeHandleSimpleNomemIntrinsic(IntrinsicInst &I) {
2875 Type *RetTy = I.getType();
2876 if (!(RetTy->isIntOrIntVectorTy() || RetTy->isFPOrFPVectorTy() ||
2877 RetTy->isX86_MMXTy()))
2878 return false;
2879
2880 unsigned NumArgOperands = I.arg_size();
2881 for (unsigned i = 0; i < NumArgOperands; ++i) {
2882 Type *Ty = I.getArgOperand(i)->getType();
2883 if (Ty != RetTy)
2884 return false;
2885 }
2886
2887 IRBuilder<> IRB(&I);
2888 ShadowAndOriginCombiner SC(this, IRB);
2889 for (unsigned i = 0; i < NumArgOperands; ++i)
2890 SC.Add(I.getArgOperand(i));
2891 SC.Done(&I);
2892
2893 return true;
2894 }
2895
2896 /// Heuristically instrument unknown intrinsics.
2897 ///
2898 /// The main purpose of this code is to do something reasonable with all
2899 /// random intrinsics we might encounter, most importantly - SIMD intrinsics.
2900 /// We recognize several classes of intrinsics by their argument types and
2901 /// ModRefBehaviour and apply special instrumentation when we are reasonably
2902 /// sure that we know what the intrinsic does.
2903 ///
2904 /// We special-case intrinsics where this approach fails. See llvm.bswap
2905 /// handling as an example of that.
2906 bool handleUnknownIntrinsic(IntrinsicInst &I) {
2907 unsigned NumArgOperands = I.arg_size();
2908 if (NumArgOperands == 0)
2909 return false;
2910
2911 if (NumArgOperands == 2 && I.getArgOperand(0)->getType()->isPointerTy() &&
2912 I.getArgOperand(1)->getType()->isVectorTy() &&
2913 I.getType()->isVoidTy() && !I.onlyReadsMemory()) {
2914 // This looks like a vector store.
2915 return handleVectorStoreIntrinsic(I);
2916 }
2917
2918 if (NumArgOperands == 1 && I.getArgOperand(0)->getType()->isPointerTy() &&
2919 I.getType()->isVectorTy() && I.onlyReadsMemory()) {
2920 // This looks like a vector load.
2921 return handleVectorLoadIntrinsic(I);
2922 }
2923
2924 if (I.doesNotAccessMemory())
2925 if (maybeHandleSimpleNomemIntrinsic(I))
2926 return true;
2927
2928 // FIXME: detect and handle SSE maskstore/maskload
2929 return false;
2930 }
2931
2932 void handleInvariantGroup(IntrinsicInst &I) {
2933 setShadow(&I, getShadow(&I, 0));
2934 setOrigin(&I, getOrigin(&I, 0));
2935 }
2936
2937 void handleLifetimeStart(IntrinsicInst &I) {
2938 if (!PoisonStack)
2939 return;
2940 AllocaInst *AI = llvm::findAllocaForValue(I.getArgOperand(1));
2941 if (!AI)
2942 InstrumentLifetimeStart = false;
2943 LifetimeStartList.push_back(std::make_pair(&I, AI));
2944 }
2945
2946 void handleBswap(IntrinsicInst &I) {
2947 IRBuilder<> IRB(&I);
2948 Value *Op = I.getArgOperand(0);
2949 Type *OpType = Op->getType();
2951 F.getParent(), Intrinsic::bswap, ArrayRef(&OpType, 1));
2952 setShadow(&I, IRB.CreateCall(BswapFunc, getShadow(Op)));
2953 setOrigin(&I, getOrigin(Op));
2954 }
2955
2956 void handleCountZeroes(IntrinsicInst &I) {
2957 IRBuilder<> IRB(&I);
2958 Value *Src = I.getArgOperand(0);
2959
2960 // Set the Output shadow based on input Shadow
2961 Value *BoolShadow = IRB.CreateIsNotNull(getShadow(Src), "_mscz_bs");
2962
2963 // If zero poison is requested, mix in with the shadow
2964 Constant *IsZeroPoison = cast<Constant>(I.getOperand(1));
2965 if (!IsZeroPoison->isZeroValue()) {
2966 Value *BoolZeroPoison = IRB.CreateIsNull(Src, "_mscz_bzp");
2967 BoolShadow = IRB.CreateOr(BoolShadow, BoolZeroPoison, "_mscz_bs");
2968 }
2969
2970 Value *OutputShadow =
2971 IRB.CreateSExt(BoolShadow, getShadowTy(Src), "_mscz_os");
2972
2973 setShadow(&I, OutputShadow);
2974 setOriginForNaryOp(I);
2975 }
2976
2977 // Instrument vector convert intrinsic.
2978 //
2979 // This function instruments intrinsics like cvtsi2ss:
2980 // %Out = int_xxx_cvtyyy(%ConvertOp)
2981 // or
2982 // %Out = int_xxx_cvtyyy(%CopyOp, %ConvertOp)
2983 // Intrinsic converts \p NumUsedElements elements of \p ConvertOp to the same
2984 // number \p Out elements, and (if has 2 arguments) copies the rest of the
2985 // elements from \p CopyOp.
2986 // In most cases conversion involves floating-point value which may trigger a
2987 // hardware exception when not fully initialized. For this reason we require
2988 // \p ConvertOp[0:NumUsedElements] to be fully initialized and trap otherwise.
2989 // We copy the shadow of \p CopyOp[NumUsedElements:] to \p
2990 // Out[NumUsedElements:]. This means that intrinsics without \p CopyOp always
2991 // return a fully initialized value.
2992 void handleVectorConvertIntrinsic(IntrinsicInst &I, int NumUsedElements,
2993 bool HasRoundingMode = false) {
2994 IRBuilder<> IRB(&I);
2995 Value *CopyOp, *ConvertOp;
2996
2997 assert((!HasRoundingMode ||
2998 isa<ConstantInt>(I.getArgOperand(I.arg_size() - 1))) &&
2999 "Invalid rounding mode");
3000
3001 switch (I.arg_size() - HasRoundingMode) {
3002 case 2:
3003 CopyOp = I.getArgOperand(0);
3004 ConvertOp = I.getArgOperand(1);
3005 break;
3006 case 1:
3007 ConvertOp = I.getArgOperand(0);
3008 CopyOp = nullptr;
3009 break;
3010 default:
3011 llvm_unreachable("Cvt intrinsic with unsupported number of arguments.");
3012 }
3013
3014 // The first *NumUsedElements* elements of ConvertOp are converted to the
3015 // same number of output elements. The rest of the output is copied from
3016 // CopyOp, or (if not available) filled with zeroes.
3017 // Combine shadow for elements of ConvertOp that are used in this operation,
3018 // and insert a check.
3019 // FIXME: consider propagating shadow of ConvertOp, at least in the case of
3020 // int->any conversion.
3021 Value *ConvertShadow = getShadow(ConvertOp);
3022 Value *AggShadow = nullptr;
3023 if (ConvertOp->getType()->isVectorTy()) {
3024 AggShadow = IRB.CreateExtractElement(
3025 ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), 0));
3026 for (int i = 1; i < NumUsedElements; ++i) {
3027 Value *MoreShadow = IRB.CreateExtractElement(
3028 ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), i));
3029 AggShadow = IRB.CreateOr(AggShadow, MoreShadow);
3030 }
3031 } else {
3032 AggShadow = ConvertShadow;
3033 }
3034 assert(AggShadow->getType()->isIntegerTy());
3035 insertShadowCheck(AggShadow, getOrigin(ConvertOp), &I);
3036
3037 // Build result shadow by zero-filling parts of CopyOp shadow that come from
3038 // ConvertOp.
3039 if (CopyOp) {
3040 assert(CopyOp->getType() == I.getType());
3041 assert(CopyOp->getType()->isVectorTy());
3042 Value *ResultShadow = getShadow(CopyOp);
3043 Type *EltTy = cast<VectorType>(ResultShadow->getType())->getElementType();
3044 for (int i = 0; i < NumUsedElements; ++i) {
3045 ResultShadow = IRB.CreateInsertElement(
3046 ResultShadow, ConstantInt::getNullValue(EltTy),
3047 ConstantInt::get(IRB.getInt32Ty(), i));
3048 }
3049 setShadow(&I, ResultShadow);
3050 setOrigin(&I, getOrigin(CopyOp));
3051 } else {
3052 setShadow(&I, getCleanShadow(&I));
3053 setOrigin(&I, getCleanOrigin());
3054 }
3055 }
3056
3057 // Given a scalar or vector, extract lower 64 bits (or less), and return all
3058 // zeroes if it is zero, and all ones otherwise.
3059 Value *Lower64ShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
3060 if (S->getType()->isVectorTy())
3061 S = CreateShadowCast(IRB, S, IRB.getInt64Ty(), /* Signed */ true);
3062 assert(S->getType()->getPrimitiveSizeInBits() <= 64);
3063 Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
3064 return CreateShadowCast(IRB, S2, T, /* Signed */ true);
3065 }
3066
3067 // Given a vector, extract its first element, and return all
3068 // zeroes if it is zero, and all ones otherwise.
3069 Value *LowerElementShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
3070 Value *S1 = IRB.CreateExtractElement(S, (uint64_t)0);
3071 Value *S2 = IRB.CreateICmpNE(S1, getCleanShadow(S1));
3072 return CreateShadowCast(IRB, S2, T, /* Signed */ true);
3073 }
3074
3075 Value *VariableShadowExtend(IRBuilder<> &IRB, Value *S) {
3076 Type *T = S->getType();
3077 assert(T->isVectorTy());
3078 Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
3079 return IRB.CreateSExt(S2, T);
3080 }
3081
3082 // Instrument vector shift intrinsic.
3083 //
3084 // This function instruments intrinsics like int_x86_avx2_psll_w.
3085 // Intrinsic shifts %In by %ShiftSize bits.
3086 // %ShiftSize may be a vector. In that case the lower 64 bits determine shift
3087 // size, and the rest is ignored. Behavior is defined even if shift size is
3088 // greater than register (or field) width.
3089 void handleVectorShiftIntrinsic(IntrinsicInst &I, bool Variable) {
3090 assert(I.arg_size() == 2);
3091 IRBuilder<> IRB(&I);
3092 // If any of the S2 bits are poisoned, the whole thing is poisoned.
3093 // Otherwise perform the same shift on S1.
3094 Value *S1 = getShadow(&I, 0);
3095 Value *S2 = getShadow(&I, 1);
3096 Value *S2Conv = Variable ? VariableShadowExtend(IRB, S2)
3097 : Lower64ShadowExtend(IRB, S2, getShadowTy(&I));
3098 Value *V1 = I.getOperand(0);
3099 Value *V2 = I.getOperand(1);
3100 Value *Shift = IRB.CreateCall(I.getFunctionType(), I.getCalledOperand(),
3101 {IRB.CreateBitCast(S1, V1->getType()), V2});
3102 Shift = IRB.CreateBitCast(Shift, getShadowTy(&I));
3103 setShadow(&I, IRB.CreateOr(Shift, S2Conv));
3104 setOriginForNaryOp(I);
3105 }
3106
3107 // Get an X86_MMX-sized vector type.
3108 Type *getMMXVectorTy(unsigned EltSizeInBits) {
3109 const unsigned X86_MMXSizeInBits = 64;
3110 assert(EltSizeInBits != 0 && (X86_MMXSizeInBits % EltSizeInBits) == 0 &&
3111 "Illegal MMX vector element size");
3112 return FixedVectorType::get(IntegerType::get(*MS.C, EltSizeInBits),
3113 X86_MMXSizeInBits / EltSizeInBits);
3114 }
3115
3116 // Returns a signed counterpart for an (un)signed-saturate-and-pack
3117 // intrinsic.
3118 Intrinsic::ID getSignedPackIntrinsic(Intrinsic::ID id) {
3119 switch (id) {
3120 case Intrinsic::x86_sse2_packsswb_128:
3121 case Intrinsic::x86_sse2_packuswb_128:
3122 return Intrinsic::x86_sse2_packsswb_128;
3123
3124 case Intrinsic::x86_sse2_packssdw_128:
3125 case Intrinsic::x86_sse41_packusdw:
3126 return Intrinsic::x86_sse2_packssdw_128;
3127
3128 case Intrinsic::x86_avx2_packsswb:
3129 case Intrinsic::x86_avx2_packuswb:
3130 return Intrinsic::x86_avx2_packsswb;
3131
3132 case Intrinsic::x86_avx2_packssdw:
3133 case Intrinsic::x86_avx2_packusdw:
3134 return Intrinsic::x86_avx2_packssdw;
3135
3136 case Intrinsic::x86_mmx_packsswb:
3137 case Intrinsic::x86_mmx_packuswb:
3138 return Intrinsic::x86_mmx_packsswb;
3139
3140 case Intrinsic::x86_mmx_packssdw:
3141 return Intrinsic::x86_mmx_packssdw;
3142 default:
3143 llvm_unreachable("unexpected intrinsic id");
3144 }
3145 }
3146
3147 // Instrument vector pack intrinsic.
3148 //
3149 // This function instruments intrinsics like x86_mmx_packsswb, that
3150 // packs elements of 2 input vectors into half as many bits with saturation.
3151 // Shadow is propagated with the signed variant of the same intrinsic applied
3152 // to sext(Sa != zeroinitializer), sext(Sb != zeroinitializer).
3153 // EltSizeInBits is used only for x86mmx arguments.
3154 void handleVectorPackIntrinsic(IntrinsicInst &I, unsigned EltSizeInBits = 0) {
3155 assert(I.arg_size() == 2);
3156 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
3157 IRBuilder<> IRB(&I);
3158 Value *S1 = getShadow(&I, 0);
3159 Value *S2 = getShadow(&I, 1);
3160 assert(isX86_MMX || S1->getType()->isVectorTy());
3161
3162 // SExt and ICmpNE below must apply to individual elements of input vectors.
3163 // In case of x86mmx arguments, cast them to appropriate vector types and
3164 // back.
3165 Type *T = isX86_MMX ? getMMXVectorTy(EltSizeInBits) : S1->getType();
3166 if (isX86_MMX) {
3167 S1 = IRB.CreateBitCast(S1, T);
3168 S2 = IRB.CreateBitCast(S2, T);
3169 }
3170 Value *S1_ext =
3172 Value *S2_ext =
3174 if (isX86_MMX) {
3175 Type *X86_MMXTy = Type::getX86_MMXTy(*MS.C);
3176 S1_ext = IRB.CreateBitCast(S1_ext, X86_MMXTy);
3177 S2_ext = IRB.CreateBitCast(S2_ext, X86_MMXTy);
3178 }
3179
3181 F.getParent(), getSignedPackIntrinsic(I.getIntrinsicID()));
3182
3183 Value *S =
3184 IRB.CreateCall(ShadowFn, {S1_ext, S2_ext}, "_msprop_vector_pack");
3185 if (isX86_MMX)
3186 S = IRB.CreateBitCast(S, getShadowTy(&I));
3187 setShadow(&I, S);
3188 setOriginForNaryOp(I);
3189 }
3190
3191 // Instrument sum-of-absolute-differences intrinsic.
3192 void handleVectorSadIntrinsic(IntrinsicInst &I) {
3193 const unsigned SignificantBitsPerResultElement = 16;
3194 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
3195 Type *ResTy = isX86_MMX ? IntegerType::get(*MS.C, 64) : I.getType();
3196 unsigned ZeroBitsPerResultElement =
3197 ResTy->getScalarSizeInBits() - SignificantBitsPerResultElement;
3198
3199 IRBuilder<> IRB(&I);
3200 auto *Shadow0 = getShadow(&I, 0);
3201 auto *Shadow1 = getShadow(&I, 1);
3202 Value *S = IRB.CreateOr(Shadow0, Shadow1);
3203 S = IRB.CreateBitCast(S, ResTy);
3204 S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
3205 ResTy);
3206 S = IRB.CreateLShr(S, ZeroBitsPerResultElement);
3207 S = IRB.CreateBitCast(S, getShadowTy(&I));
3208 setShadow(&I, S);
3209 setOriginForNaryOp(I);
3210 }
3211
3212 // Instrument multiply-add intrinsic.
3213 void handleVectorPmaddIntrinsic(IntrinsicInst &I,
3214 unsigned EltSizeInBits = 0) {
3215 bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
3216 Type *ResTy = isX86_MMX ? getMMXVectorTy(EltSizeInBits * 2) : I.getType();
3217 IRBuilder<> IRB(&I);
3218 auto *Shadow0 = getShadow(&I, 0);
3219 auto *Shadow1 = getShadow(&I, 1);
3220 Value *S = IRB.CreateOr(Shadow0, Shadow1);
3221 S = IRB.CreateBitCast(S, ResTy);
3222 S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
3223 ResTy);
3224 S = IRB.CreateBitCast(S, getShadowTy(&I));
3225 setShadow(&I, S);
3226 setOriginForNaryOp(I);
3227 }
3228
3229 // Instrument compare-packed intrinsic.
3230 // Basically, an or followed by sext(icmp ne 0) to end up with all-zeros or
3231 // all-ones shadow.
3232 void handleVectorComparePackedIntrinsic(IntrinsicInst &I) {
3233 IRBuilder<> IRB(&I);
3234 Type *ResTy = getShadowTy(&I);
3235 auto *Shadow0 = getShadow(&I, 0);
3236 auto *Shadow1 = getShadow(&I, 1);
3237 Value *S0 = IRB.CreateOr(Shadow0, Shadow1);
3238 Value *S = IRB.CreateSExt(
3239 IRB.CreateICmpNE(S0, Constant::getNullValue(ResTy)), ResTy);
3240 setShadow(&I, S);
3241 setOriginForNaryOp(I);
3242 }
3243
3244 // Instrument compare-scalar intrinsic.
3245 // This handles both cmp* intrinsics which return the result in the first
3246 // element of a vector, and comi* which return the result as i32.
3247 void handleVectorCompareScalarIntrinsic(IntrinsicInst &I) {
3248 IRBuilder<> IRB(&I);
3249 auto *Shadow0 = getShadow(&I, 0);
3250 auto *Shadow1 = getShadow(&I, 1);
3251 Value *S0 = IRB.CreateOr(Shadow0, Shadow1);
3252 Value *S = LowerElementShadowExtend(IRB, S0, getShadowTy(&I));
3253 setShadow(&I, S);
3254 setOriginForNaryOp(I);
3255 }
3256
3257 // Instrument generic vector reduction intrinsics
3258 // by ORing together all their fields.
3259 void handleVectorReduceIntrinsic(IntrinsicInst &I) {
3260 IRBuilder<> IRB(&I);
3261 Value *S = IRB.CreateOrReduce(getShadow(&I, 0));
3262 setShadow(&I, S);
3263 setOrigin(&I, getOrigin(&I, 0));
3264 }
3265
3266 // Instrument vector.reduce.or intrinsic.
3267 // Valid (non-poisoned) set bits in the operand pull low the
3268 // corresponding shadow bits.
3269 void handleVectorReduceOrIntrinsic(IntrinsicInst &I) {
3270 IRBuilder<> IRB(&I);
3271 Value *OperandShadow = getShadow(&I, 0);
3272 Value *OperandUnsetBits = IRB.CreateNot(I.getOperand(0));
3273 Value *OperandUnsetOrPoison = IRB.CreateOr(OperandUnsetBits, OperandShadow);
3274 // Bit N is clean if any field's bit N is 1 and unpoison
3275 Value *OutShadowMask = IRB.CreateAndReduce(OperandUnsetOrPoison);
3276 // Otherwise, it is clean if every field's bit N is unpoison
3277 Value *OrShadow = IRB.CreateOrReduce(OperandShadow);
3278 Value *S = IRB.CreateAnd(OutShadowMask, OrShadow);
3279
3280 setShadow(&I, S);
3281 setOrigin(&I, getOrigin(&I, 0));
3282 }
3283
3284 // Instrument vector.reduce.and intrinsic.
3285 // Valid (non-poisoned) unset bits in the operand pull down the
3286 // corresponding shadow bits.
3287 void handleVectorReduceAndIntrinsic(IntrinsicInst &I) {
3288 IRBuilder<> IRB(&I);
3289 Value *OperandShadow = getShadow(&I, 0);
3290 Value *OperandSetOrPoison = IRB.CreateOr(I.getOperand(0), OperandShadow);
3291 // Bit N is clean if any field's bit N is 0 and unpoison
3292 Value *OutShadowMask = IRB.CreateAndReduce(OperandSetOrPoison);
3293 // Otherwise, it is clean if every field's bit N is unpoison
3294 Value *OrShadow = IRB.CreateOrReduce(OperandShadow);
3295 Value *S = IRB.CreateAnd(OutShadowMask, OrShadow);
3296
3297 setShadow(&I, S);
3298 setOrigin(&I, getOrigin(&I, 0));
3299 }
3300
3301 void handleStmxcsr(IntrinsicInst &I) {
3302 IRBuilder<> IRB(&I);
3303 Value *Addr = I.getArgOperand(0);
3304 Type *Ty = IRB.getInt32Ty();
3305 Value *ShadowPtr =
3306 getShadowOriginPtr(Addr, IRB, Ty, Align(1), /*isStore*/ true).first;
3307
3308 IRB.CreateStore(getCleanShadow(Ty),
3309 IRB.CreatePointerCast(ShadowPtr, Ty->getPointerTo()));
3310
3312 insertShadowCheck(Addr, &I);
3313 }
3314
3315 void handleLdmxcsr(IntrinsicInst &I) {
3316 if (!InsertChecks)
3317 return;
3318
3319 IRBuilder<> IRB(&I);
3320 Value *Addr = I.getArgOperand(0);
3321 Type *Ty = IRB.getInt32Ty();
3322 const Align Alignment = Align(1);
3323 Value *ShadowPtr, *OriginPtr;
3324 std::tie(ShadowPtr, OriginPtr) =
3325 getShadowOriginPtr(Addr, IRB, Ty, Alignment, /*isStore*/ false);
3326
3328 insertShadowCheck(Addr, &I);
3329
3330 Value *Shadow = IRB.CreateAlignedLoad(Ty, ShadowPtr, Alignment, "_ldmxcsr");
3331 Value *Origin = MS.TrackOrigins ? IRB.CreateLoad(MS.OriginTy, OriginPtr)
3332 : getCleanOrigin();
3333 insertShadowCheck(Shadow, Origin, &I);
3334 }
3335
3336 void handleMaskedExpandLoad(IntrinsicInst &I) {
3337 IRBuilder<> IRB(&I);
3338 Value *Ptr = I.getArgOperand(0);
3339 Value *Mask = I.getArgOperand(1);
3340 Value *PassThru = I.getArgOperand(2);
3341
3343 insertShadowCheck(Ptr, &I);
3344 insertShadowCheck(Mask, &I);
3345 }
3346
3347 if (!PropagateShadow) {
3348 setShadow(&I, getCleanShadow(&I));
3349 setOrigin(&I, getCleanOrigin());
3350 return;
3351 }
3352
3353 Type *ShadowTy = getShadowTy(&I);
3354 Type *ElementShadowTy = cast<VectorType>(ShadowTy)->getElementType();
3355 auto [ShadowPtr, OriginPtr] =
3356 getShadowOriginPtr(Ptr, IRB, ElementShadowTy, {}, /*isStore*/ false);
3357
3358 Value *Shadow = IRB.CreateMaskedExpandLoad(
3359 ShadowTy, ShadowPtr, Mask, getShadow(PassThru), "_msmaskedexpload");
3360
3361 setShadow(&I, Shadow);
3362
3363 // TODO: Store origins.
3364 setOrigin(&I, getCleanOrigin());
3365 }
3366
3367 void handleMaskedCompressStore(IntrinsicInst &I) {
3368 IRBuilder<> IRB(&I);
3369 Value *Values = I.getArgOperand(0);
3370 Value *Ptr = I.getArgOperand(1);
3371 Value *Mask = I.getArgOperand(2);
3372
3374 insertShadowCheck(Ptr, &I);
3375 insertShadowCheck(Mask, &I);
3376 }
3377
3378 Value *Shadow = getShadow(Values);
3379 Type *ElementShadowTy =
3380 getShadowTy(cast<VectorType>(Values->getType())->getElementType());
3381 auto [ShadowPtr, OriginPtrs] =
3382 getShadowOriginPtr(Ptr, IRB, ElementShadowTy, {}, /*isStore*/ true);
3383
3384 IRB.CreateMaskedCompressStore(Shadow, ShadowPtr, Mask);
3385
3386 // TODO: Store origins.
3387 }
3388
3389 void handleMaskedGather(IntrinsicInst &I) {
3390 IRBuilder<> IRB(&I);
3391 Value *Ptrs = I.getArgOperand(0);
3392 const Align Alignment(
3393 cast<ConstantInt>(I.getArgOperand(1))->getZExtValue());
3394 Value *Mask = I.getArgOperand(2);
3395 Value *PassThru = I.getArgOperand(3);
3396
3397 Type *PtrsShadowTy = getShadowTy(Ptrs);
3399 insertShadowCheck(Mask, &I);
3400 Value *MaskedPtrShadow = IRB.CreateSelect(
3401 Mask, getShadow(Ptrs), Constant::getNullValue((PtrsShadowTy)),
3402 "_msmaskedptrs");
3403 insertShadowCheck(MaskedPtrShadow, getOrigin(Ptrs), &I);
3404 }
3405
3406 if (!PropagateShadow) {
3407 setShadow(&I, getCleanShadow(&I));
3408 setOrigin(&I, getCleanOrigin());
3409 return;
3410 }
3411
3412 Type *ShadowTy = getShadowTy(&I);
3413 Type *ElementShadowTy = cast<VectorType>(ShadowTy)->getElementType();
3414 auto [ShadowPtrs, OriginPtrs] = getShadowOriginPtr(
3415 Ptrs, IRB, ElementShadowTy, Alignment, /*isStore*/ false);
3416
3417 Value *Shadow =
3418 IRB.CreateMaskedGather(ShadowTy, ShadowPtrs, Alignment, Mask,
3419 getShadow(PassThru), "_msmaskedgather");
3420
3421 setShadow(&I, Shadow);
3422
3423 // TODO: Store origins.
3424 setOrigin(&I, getCleanOrigin());
3425 }
3426
3427 void handleMaskedScatter(IntrinsicInst &I) {
3428 IRBuilder<> IRB(&I);
3429 Value *Values = I.getArgOperand(0);
3430 Value *Ptrs = I.getArgOperand(1);
3431 const Align Alignment(
3432 cast<ConstantInt>(I.getArgOperand(2))->getZExtValue());
3433 Value *Mask = I.getArgOperand(3);
3434
3435 Type *PtrsShadowTy = getShadowTy(Ptrs);
3437 insertShadowCheck(Mask, &I);
3438 Value *MaskedPtrShadow = IRB.CreateSelect(
3439 Mask, getShadow(Ptrs), Constant::getNullValue((PtrsShadowTy)),
3440 "_msmaskedptrs");
3441 insertShadowCheck(MaskedPtrShadow, getOrigin(Ptrs), &I);
3442 }
3443
3444 Value *Shadow = getShadow(Values);
3445 Type *ElementShadowTy =
3446 getShadowTy(cast<VectorType>(Values->getType())->getElementType());
3447 auto [ShadowPtrs, OriginPtrs] = getShadowOriginPtr(
3448 Ptrs, IRB, ElementShadowTy, Alignment, /*isStore*/ true);
3449
3450 IRB.CreateMaskedScatter(Shadow, ShadowPtrs, Alignment, Mask);
3451
3452 // TODO: Store origin.
3453 }
3454
3455 void handleMaskedStore(IntrinsicInst &I) {
3456 IRBuilder<> IRB(&I);
3457 Value *V = I.getArgOperand(0);
3458 Value *Ptr = I.getArgOperand(1);
3459 const Align Alignment(
3460 cast<ConstantInt>(I.getArgOperand(2))->getZExtValue());
3461 Value *Mask = I.getArgOperand(3);
3462 Value *Shadow = getShadow(V);
3463
3465 insertShadowCheck(Ptr, &I);
3466 insertShadowCheck(Mask, &I);
3467 }
3468
3469 Value *ShadowPtr;
3470 Value *OriginPtr;
3471 std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
3472 Ptr, IRB, Shadow->getType(), Alignment, /*isStore*/ true);
3473
3474 IRB.CreateMaskedStore(Shadow, ShadowPtr, Alignment, Mask);
3475
3476 if (!MS.TrackOrigins)
3477 return;
3478
3479 auto &DL = F.getParent()->getDataLayout();
3480 paintOrigin(IRB, getOrigin(V), OriginPtr,
3481 DL.getTypeStoreSize(Shadow->getType()),
3482 std::max(Alignment, kMinOriginAlignment));
3483 }
3484
3485 void handleMaskedLoad(IntrinsicInst &I) {
3486 IRBuilder<> IRB(&I);
3487 Value *Ptr = I.getArgOperand(0);
3488 const Align Alignment(
3489 cast<ConstantInt>(I.getArgOperand(1))->getZExtValue());
3490 Value *Mask = I.getArgOperand(2);
3491 Value *PassThru = I.getArgOperand(3);
3492
3494 insertShadowCheck(Ptr, &I);
3495 insertShadowCheck(Mask, &I);
3496 }
3497
3498 if (!PropagateShadow) {
3499 setShadow(&I, getCleanShadow(&I));
3500 setOrigin(&I, getCleanOrigin());
3501 return;
3502 }
3503
3504 Type *ShadowTy = getShadowTy(&I);
3505 Value *ShadowPtr, *OriginPtr;
3506 std::tie(ShadowPtr, OriginPtr) =
3507 getShadowOriginPtr(Ptr, IRB, ShadowTy, Alignment, /*isStore*/ false);
3508 setShadow(&I, IRB.CreateMaskedLoad(ShadowTy, ShadowPtr, Alignment, Mask,
3509 getShadow(PassThru), "_msmaskedld"));
3510
3511 if (!MS.TrackOrigins)
3512 return;
3513
3514 // Choose between PassThru's and the loaded value's origins.
3515 Value *MaskedPassThruShadow = IRB.CreateAnd(
3516 getShadow(PassThru), IRB.CreateSExt(IRB.CreateNeg(Mask), ShadowTy));
3517
3518 Value *NotNull = convertToBool(MaskedPassThruShadow, IRB, "_mscmp");
3519
3520 Value *PtrOrigin = IRB.CreateLoad(MS.OriginTy, OriginPtr);
3521 Value *Origin = IRB.CreateSelect(NotNull, getOrigin(PassThru), PtrOrigin);
3522
3523 setOrigin(&I, Origin);
3524 }
3525
3526 // Instrument BMI / BMI2 intrinsics.
3527 // All of these intrinsics are Z = I(X, Y)
3528 // where the types of all operands and the result match, and are either i32 or
3529 // i64. The following instrumentation happens to work for all of them:
3530 // Sz = I(Sx, Y) | (sext (Sy != 0))
3531 void handleBmiIntrinsic(IntrinsicInst &I) {
3532 IRBuilder<> IRB(&I);
3533 Type *ShadowTy = getShadowTy(&I);
3534
3535 // If any bit of the mask operand is poisoned, then the whole thing is.
3536 Value *SMask = getShadow(&I, 1);
3537 SMask = IRB.CreateSExt(IRB.CreateICmpNE(SMask, getCleanShadow(ShadowTy)),
3538 ShadowTy);
3539 // Apply the same intrinsic to the shadow of the first operand.
3540 Value *S = IRB.CreateCall(I.getCalledFunction(),
3541 {getShadow(&I, 0), I.getOperand(1)});
3542 S = IRB.CreateOr(SMask, S);
3543 setShadow(&I, S);
3544 setOriginForNaryOp(I);
3545 }
3546
3547 SmallVector<int, 8> getPclmulMask(unsigned Width, bool OddElements) {
3549 for (unsigned X = OddElements ? 1 : 0; X < Width; X += 2) {
3550 Mask.append(2, X);
3551 }
3552 return Mask;
3553 }
3554
3555 // Instrument pclmul intrinsics.
3556 // These intrinsics operate either on odd or on even elements of the input
3557 // vectors, depending on the constant in the 3rd argument, ignoring the rest.
3558 // Replace the unused elements with copies of the used ones, ex:
3559 // (0, 1, 2, 3) -> (0, 0, 2, 2) (even case)
3560 // or
3561 // (0, 1, 2, 3) -> (1, 1, 3, 3) (odd case)
3562 // and then apply the usual shadow combining logic.
3563 void handlePclmulIntrinsic(IntrinsicInst &I) {
3564 IRBuilder<> IRB(&I);
3565 unsigned Width =
3566 cast<FixedVectorType>(I.getArgOperand(0)->getType())->getNumElements();
3567 assert(isa<ConstantInt>(I.getArgOperand(2)) &&
3568 "pclmul 3rd operand must be a constant");
3569 unsigned Imm = cast<ConstantInt>(I.getArgOperand(2))->getZExtValue();
3570 Value *Shuf0 = IRB.CreateShuffleVector(getShadow(&I, 0),
3571 getPclmulMask(Width, Imm & 0x01));
3572 Value *Shuf1 = IRB.CreateShuffleVector(getShadow(&I, 1),
3573 getPclmulMask(Width, Imm & 0x10));
3574 ShadowAndOriginCombiner SOC(this, IRB);
3575 SOC.Add(Shuf0, getOrigin(&I, 0));
3576 SOC.Add(Shuf1, getOrigin(&I, 1));
3577 SOC.Done(&I);
3578 }
3579
3580 // Instrument _mm_*_sd|ss intrinsics
3581 void handleUnarySdSsIntrinsic(IntrinsicInst &I) {
3582 IRBuilder<> IRB(&I);
3583 unsigned Width =
3584 cast<FixedVectorType>(I.getArgOperand(0)->getType())->getNumElements();
3585 Value *First = getShadow(&I, 0);
3586 Value *Second = getShadow(&I, 1);
3587 // First element of second operand, remaining elements of first operand
3589 Mask.push_back(Width);
3590 for (unsigned i = 1; i < Width; i++)
3591 Mask.push_back(i);
3592 Value *Shadow = IRB.CreateShuffleVector(First, Second, Mask);
3593
3594 setShadow(&I, Shadow);
3595 setOriginForNaryOp(I);
3596 }
3597
3598 void handleVtestIntrinsic(IntrinsicInst &I) {
3599 IRBuilder<> IRB(&I);
3600 Value *Shadow0 = getShadow(&I, 0);
3601 Value *Shadow1 = getShadow(&I, 1);
3602 Value *Or = IRB.CreateOr(Shadow0, Shadow1);
3603 Value *NZ = IRB.CreateICmpNE(Or, Constant::getNullValue(Or->getType()));
3604 Value *Scalar = convertShadowToScalar(NZ, IRB);
3605 Value *Shadow = IRB.CreateZExt(Scalar, getShadowTy(&I));
3606
3607 setShadow(&I, Shadow);
3608 setOriginForNaryOp(I);
3609 }
3610
3611 void handleBinarySdSsIntrinsic(IntrinsicInst &I) {
3612 IRBuilder<> IRB(&I);
3613 unsigned Width =
3614 cast<FixedVectorType>(I.getArgOperand(0)->getType())->getNumElements();
3615 Value *First = getShadow(&I, 0);
3616 Value *Second = getShadow(&I, 1);
3617 Value *OrShadow = IRB.CreateOr(First, Second);
3618 // First element of both OR'd together, remaining elements of first operand
3620 Mask.push_back(Width);
3621 for (unsigned i = 1; i < Width; i++)
3622 Mask.push_back(i);
3623 Value *Shadow = IRB.CreateShuffleVector(First, OrShadow, Mask);
3624
3625 setShadow(&I, Shadow);
3626 setOriginForNaryOp(I);
3627 }
3628
3629 // Instrument abs intrinsic.
3630 // handleUnknownIntrinsic can't handle it because of the last
3631 // is_int_min_poison argument which does not match the result type.
3632 void handleAbsIntrinsic(IntrinsicInst &I) {
3633 assert(I.getType()->isIntOrIntVectorTy());
3634 assert(I.getArgOperand(0)->getType() == I.getType());
3635
3636 // FIXME: Handle is_int_min_poison.
3637 IRBuilder<> IRB(&I);
3638 setShadow(&I, getShadow(&I, 0));
3639 setOrigin(&I, getOrigin(&I, 0));
3640 }
3641
3642 void visitIntrinsicInst(IntrinsicInst &I) {
3643 switch (I.getIntrinsicID()) {
3644 case Intrinsic::abs:
3645 handleAbsIntrinsic(I);
3646 break;
3647 case Intrinsic::lifetime_start:
3648 handleLifetimeStart(I);
3649 break;
3650 case Intrinsic::launder_invariant_group:
3651 case Intrinsic::strip_invariant_group:
3652 handleInvariantGroup(I);
3653 break;
3654 case Intrinsic::bswap:
3655 handleBswap(I);
3656 break;
3657 case Intrinsic::ctlz:
3658 case Intrinsic::cttz:
3659 handleCountZeroes(I);
3660 break;
3661 case Intrinsic::masked_compressstore:
3662 handleMaskedCompressStore(I);
3663 break;
3664 case Intrinsic::masked_expandload:
3665 handleMaskedExpandLoad(I);
3666 break;
3667 case Intrinsic::masked_gather:
3668 handleMaskedGather(I);
3669 break;
3670 case Intrinsic::masked_scatter:
3671 handleMaskedScatter(I);
3672 break;
3673 case Intrinsic::masked_store:
3674 handleMaskedStore(I);
3675 break;
3676 case Intrinsic::masked_load:
3677 handleMaskedLoad(I);
3678 break;
3679 case Intrinsic::vector_reduce_and:
3680 handleVectorReduceAndIntrinsic(I);
3681 break;
3682 case Intrinsic::vector_reduce_or:
3683 handleVectorReduceOrIntrinsic(I);
3684 break;
3685 case Intrinsic::vector_reduce_add:
3686 case Intrinsic::vector_reduce_xor:
3687 case Intrinsic::vector_reduce_mul:
3688 handleVectorReduceIntrinsic(I);
3689 break;
3690 case Intrinsic::x86_sse_stmxcsr:
3691 handleStmxcsr(I);
3692 break;
3693 case Intrinsic::x86_sse_ldmxcsr:
3694 handleLdmxcsr(I);
3695 break;
3696 case Intrinsic::x86_avx512_vcvtsd2usi64:
3697 case Intrinsic::x86_avx512_vcvtsd2usi32:
3698 case Intrinsic::x86_avx512_vcvtss2usi64:
3699 case Intrinsic::x86_avx512_vcvtss2usi32:
3700 case Intrinsic::x86_avx512_cvttss2usi64:
3701 case Intrinsic::x86_avx512_cvttss2usi:
3702 case Intrinsic::x86_avx512_cvttsd2usi64:
3703 case Intrinsic::x86_avx512_cvttsd2usi:
3704 case Intrinsic::x86_avx512_cvtusi2ss:
3705 case Intrinsic::x86_avx512_cvtusi642sd:
3706 case Intrinsic::x86_avx512_cvtusi642ss:
3707 handleVectorConvertIntrinsic(I, 1, true);
3708 break;
3709 case Intrinsic::x86_sse2_cvtsd2si64:
3710 case Intrinsic::x86_sse2_cvtsd2si:
3711 case Intrinsic::x86_sse2_cvtsd2ss:
3712 case Intrinsic::x86_sse2_cvttsd2si64:
3713 case Intrinsic::x86_sse2_cvttsd2si:
3714 case Intrinsic::x86_sse_cvtss2si64:
3715 case Intrinsic::x86_sse_cvtss2si:
3716 case Intrinsic::x86_sse_cvttss2si64:
3717 case Intrinsic::x86_sse_cvttss2si:
3718 handleVectorConvertIntrinsic(I, 1);
3719 break;
3720 case Intrinsic::x86_sse_cvtps2pi:
3721 case Intrinsic::x86_sse_cvttps2pi:
3722 handleVectorConvertIntrinsic(I, 2);
3723 break;
3724
3725 case Intrinsic::x86_avx512_psll_w_512:
3726 case Intrinsic::x86_avx512_psll_d_512:
3727 case Intrinsic::x86_avx512_psll_q_512:
3728 case Intrinsic::x86_avx512_pslli_w_512:
3729 case Intrinsic::x86_avx512_pslli_d_512:
3730 case Intrinsic::x86_avx512_pslli_q_512:
3731 case Intrinsic::x86_avx512_psrl_w_512:
3732 case Intrinsic::x86_avx512_psrl_d_512:
3733 case Intrinsic::x86_avx512_psrl_q_512:
3734 case Intrinsic::x86_avx512_psra_w_512:
3735 case Intrinsic::x86_avx512_psra_d_512:
3736 case Intrinsic::x86_avx512_psra_q_512:
3737 case Intrinsic::x86_avx512_psrli_w_512:
3738 case Intrinsic::x86_avx512_psrli_d_512:
3739 case Intrinsic::x86_avx512_psrli_q_512:
3740 case Intrinsic::x86_avx512_psrai_w_512:
3741 case Intrinsic::x86_avx512_psrai_d_512:
3742 case Intrinsic::x86_avx512_psrai_q_512:
3743 case Intrinsic::x86_avx512_psra_q_256:
3744 case Intrinsic::x86_avx512_psra_q_128:
3745 case Intrinsic::x86_avx512_psrai_q_256:
3746 case Intrinsic::x86_avx512_psrai_q_128:
3747 case Intrinsic::x86_avx2_psll_w:
3748 case Intrinsic::x86_avx2_psll_d:
3749 case Intrinsic::x86_avx2_psll_q:
3750 case Intrinsic::x86_avx2_pslli_w:
3751 case Intrinsic::x86_avx2_pslli_d:
3752 case Intrinsic::x86_avx2_pslli_q:
3753 case Intrinsic::x86_avx2_psrl_w:
3754 case Intrinsic::x86_avx2_psrl_d:
3755 case Intrinsic::x86_avx2_psrl_q:
3756 case Intrinsic::x86_avx2_psra_w:
3757 case Intrinsic::x86_avx2_psra_d:
3758 case Intrinsic::x86_avx2_psrli_w:
3759 case Intrinsic::x86_avx2_psrli_d:
3760 case Intrinsic::x86_avx2_psrli_q:
3761 case Intrinsic::x86_avx2_psrai_w:
3762 case Intrinsic::x86_avx2_psrai_d:
3763 case Intrinsic::x86_sse2_psll_w:
3764 case Intrinsic::x86_sse2_psll_d:
3765 case Intrinsic::x86_sse2_psll_q:
3766 case Intrinsic::x86_sse2_pslli_w:
3767 case Intrinsic::x86_sse2_pslli_d:
3768 case Intrinsic::x86_sse2_pslli_q:
3769 case Intrinsic::x86_sse2_psrl_w:
3770 case Intrinsic::x86_sse2_psrl_d:
3771 case Intrinsic::x86_sse2_psrl_q:
3772 case Intrinsic::x86_sse2_psra_w:
3773 case Intrinsic::x86_sse2_psra_d:
3774 case Intrinsic::x86_sse2_psrli_w:
3775 case Intrinsic::x86_sse2_psrli_d:
3776 case Intrinsic::x86_sse2_psrli_q:
3777 case Intrinsic::x86_sse2_psrai_w:
3778 case Intrinsic::x86_sse2_psrai_d:
3779 case Intrinsic::x86_mmx_psll_w:
3780 case Intrinsic::x86_mmx_psll_d:
3781 case Intrinsic::x86_mmx_psll_q:
3782 case Intrinsic::x86_mmx_pslli_w:
3783 case Intrinsic::x86_mmx_pslli_d:
3784 case Intrinsic::x86_mmx_pslli_q:
3785 case Intrinsic::x86_mmx_psrl_w:
3786 case Intrinsic::x86_mmx_psrl_d:
3787 case Intrinsic::x86_mmx_psrl_q:
3788 case Intrinsic::x86_mmx_psra_w:
3789 case Intrinsic::x86_mmx_psra_d:
3790 case Intrinsic::x86_mmx_psrli_w:
3791 case Intrinsic::x86_mmx_psrli_d:
3792 case Intrinsic::x86_mmx_psrli_q:
3793 case Intrinsic::x86_mmx_psrai_w:
3794 case Intrinsic::x86_mmx_psrai_d:
3795 handleVectorShiftIntrinsic(I, /* Variable */ false);
3796 break;
3797 case Intrinsic::x86_avx2_psllv_d:
3798 case Intrinsic::x86_avx2_psllv_d_256:
3799 case Intrinsic::x86_avx512_psllv_d_512:
3800 case Intrinsic::x86_avx2_psllv_q:
3801 case Intrinsic::x86_avx2_psllv_q_256:
3802 case Intrinsic::x86_avx512_psllv_q_512:
3803 case Intrinsic::x86_avx2_psrlv_d:
3804 case Intrinsic::x86_avx2_psrlv_d_256:
3805 case Intrinsic::x86_avx512_psrlv_d_512:
3806 case Intrinsic::x86_avx2_psrlv_q:
3807 case Intrinsic::x86_avx2_psrlv_q_256:
3808 case Intrinsic::x86_avx512_psrlv_q_512:
3809 case Intrinsic::x86_avx2_psrav_d:
3810 case Intrinsic::x86_avx2_psrav_d_256:
3811 case Intrinsic::x86_avx512_psrav_d_512:
3812 case Intrinsic::x86_avx512_psrav_q_128:
3813 case Intrinsic::x86_avx512_psrav_q_256:
3814 case Intrinsic::x86_avx512_psrav_q_512:
3815 handleVectorShiftIntrinsic(I, /* Variable */ true);
3816 break;
3817
3818 case Intrinsic::x86_sse2_packsswb_128:
3819 case Intrinsic::x86_sse2_packssdw_128:
3820 case Intrinsic::x86_sse2_packuswb_128:
3821 case Intrinsic::x86_sse41_packusdw:
3822 case Intrinsic::x86_avx2_packsswb:
3823 case Intrinsic::x86_avx2_packssdw:
3824 case Intrinsic::x86_avx2_packuswb:
3825 case Intrinsic::x86_avx2_packusdw:
3826 handleVectorPackIntrinsic(I);
3827 break;
3828
3829 case Intrinsic::x86_mmx_packsswb:
3830 case Intrinsic::x86_mmx_packuswb:
3831 handleVectorPackIntrinsic(I, 16);
3832 break;
3833
3834 case Intrinsic::x86_mmx_packssdw:
3835 handleVectorPackIntrinsic(I, 32);
3836 break;
3837
3838 case Intrinsic::x86_mmx_psad_bw:
3839 case Intrinsic::x86_sse2_psad_bw:
3840 case Intrinsic::x86_avx2_psad_bw:
3841 handleVectorSadIntrinsic(I);
3842 break;
3843
3844 case Intrinsic::x86_sse2_pmadd_wd:
3845 case Intrinsic::x86_avx2_pmadd_wd:
3846 case Intrinsic::x86_ssse3_pmadd_ub_sw_128:
3847 case Intrinsic::x86_avx2_pmadd_ub_sw:
3848 handleVectorPmaddIntrinsic(I);
3849 break;
3850
3851 case Intrinsic::x86_ssse3_pmadd_ub_sw:
3852 handleVectorPmaddIntrinsic(I, 8);
3853 break;
3854
3855 case Intrinsic::x86_mmx_pmadd_wd:
3856 handleVectorPmaddIntrinsic(I, 16);
3857 break;
3858
3859 case Intrinsic::x86_sse_cmp_ss:
3860 case Intrinsic::x86_sse2_cmp_sd:
3861 case Intrinsic::x86_sse_comieq_ss:
3862 case Intrinsic::x86_sse_comilt_ss:
3863 case Intrinsic::x86_sse_comile_ss:
3864 case Intrinsic::x86_sse_comigt_ss:
3865 case Intrinsic::x86_sse_comige_ss:
3866 case Intrinsic::x86_sse_comineq_ss:
3867 case Intrinsic::x86_sse_ucomieq_ss:
3868 case Intrinsic::x86_sse_ucomilt_ss:
3869 case Intrinsic::x86_sse_ucomile_ss:
3870 case Intrinsic::x86_sse_ucomigt_ss:
3871 case Intrinsic::x86_sse_ucomige_ss:
3872 case Intrinsic::x86_sse_ucomineq_ss:
3873 case Intrinsic::x86_sse2_comieq_sd:
3874 case Intrinsic::x86_sse2_comilt_sd:
3875 case Intrinsic::x86_sse2_comile_sd:
3876 case Intrinsic::x86_sse2_comigt_sd:
3877 case Intrinsic::x86_sse2_comige_sd:
3878 case Intrinsic::x86_sse2_comineq_sd:
3879 case Intrinsic::x86_sse2_ucomieq_sd:
3880 case Intrinsic::x86_sse2_ucomilt_sd:
3881 case Intrinsic::x86_sse2_ucomile_sd:
3882 case Intrinsic::x86_sse2_ucomigt_sd:
3883 case Intrinsic::x86_sse2_ucomige_sd:
3884 case Intrinsic::x86_sse2_ucomineq_sd:
3885 handleVectorCompareScalarIntrinsic(I);
3886 break;
3887
3888 case Intrinsic::x86_avx_cmp_pd_256:
3889 case Intrinsic::x86_avx_cmp_ps_256:
3890 case Intrinsic::x86_sse2_cmp_pd:
3891 case Intrinsic::x86_sse_cmp_ps:
3892 handleVectorComparePackedIntrinsic(I);
3893 break;
3894
3895 case Intrinsic::x86_bmi_bextr_32:
3896 case Intrinsic::x86_bmi_bextr_64:
3897 case Intrinsic::x86_bmi_bzhi_32:
3898 case Intrinsic::x86_bmi_bzhi_64:
3899 case Intrinsic::x86_bmi_pdep_32:
3900 case Intrinsic::x86_bmi_pdep_64:
3901 case Intrinsic::x86_bmi_pext_32:
3902 case Intrinsic::x86_bmi_pext_64:
3903 handleBmiIntrinsic(I);
3904 break;
3905
3906 case Intrinsic::x86_pclmulqdq:
3907 case Intrinsic::x86_pclmulqdq_256:
3908 case Intrinsic::x86_pclmulqdq_512:
3909 handlePclmulIntrinsic(I);
3910 break;
3911
3912 case Intrinsic::x86_sse41_round_sd:
3913 case Intrinsic::x86_sse41_round_ss:
3914 handleUnarySdSsIntrinsic(I);
3915 break;
3916 case Intrinsic::x86_sse2_max_sd:
3917 case Intrinsic::x86_sse_max_ss:
3918 case Intrinsic::x86_sse2_min_sd:
3919 case Intrinsic::x86_sse_min_ss:
3920 handleBinarySdSsIntrinsic(I);
3921 break;
3922
3923 case Intrinsic::x86_avx_vtestc_pd:
3924 case Intrinsic::x86_avx_vtestc_pd_256:
3925 case Intrinsic::x86_avx_vtestc_ps:
3926 case Intrinsic::x86_avx_vtestc_ps_256:
3927 case Intrinsic::x86_avx_vtestnzc_pd:
3928 case Intrinsic::x86_avx_vtestnzc_pd_256:
3929 case Intrinsic::x86_avx_vtestnzc_ps:
3930 case Intrinsic::x86_avx_vtestnzc_ps_256:
3931 case Intrinsic::x86_avx_vtestz_pd:
3932 case Intrinsic::x86_avx_vtestz_pd_256:
3933 case Intrinsic::x86_avx_vtestz_ps:
3934 case Intrinsic::x86_avx_vtestz_ps_256:
3935 case Intrinsic::x86_avx_ptestc_256:
3936 case Intrinsic::x86_avx_ptestnzc_256:
3937 case Intrinsic::x86_avx_ptestz_256:
3938 case Intrinsic::x86_sse41_ptestc:
3939 case Intrinsic::x86_sse41_ptestnzc:
3940 case Intrinsic::x86_sse41_ptestz:
3941 handleVtestIntrinsic(I);
3942 break;
3943
3944 case Intrinsic::fshl:
3945 case Intrinsic::fshr:
3946 handleFunnelShift(I);
3947 break;
3948
3949 case Intrinsic::is_constant:
3950 // The result of llvm.is.constant() is always defined.
3951 setShadow(&I, getCleanShadow(&I));
3952 setOrigin(&I, getCleanOrigin());
3953 break;
3954
3955 default:
3956 if (!handleUnknownIntrinsic(I))
3957 visitInstruction(I);
3958 break;
3959 }
3960 }
3961
3962 void visitLibAtomicLoad(CallBase &CB) {
3963 // Since we use getNextNode here, we can't have CB terminate the BB.
3964 assert(isa<CallInst>(CB));
3965
3966 IRBuilder<> IRB(&CB);
3967 Value *Size = CB.getArgOperand(0);
3968 Value *SrcPtr = CB.getArgOperand(1);
3969 Value *DstPtr = CB.getArgOperand(2);
3970 Value *Ordering = CB.getArgOperand(3);
3971 // Convert the call to have at least Acquire ordering to make sure
3972 // the shadow operations aren't reordered before it.
3973 Value *NewOrdering =
3974 IRB.CreateExtractElement(makeAddAcquireOrderingTable(IRB), Ordering);
3975 CB.setArgOperand(3, NewOrdering);
3976
3977 NextNodeIRBuilder NextIRB(&CB);
3978 Value *SrcShadowPtr, *SrcOriginPtr;
3979 std::tie(SrcShadowPtr, SrcOriginPtr) =
3980 getShadowOriginPtr(SrcPtr, NextIRB, NextIRB.getInt8Ty(), Align(1),
3981 /*isStore*/ false);
3982 Value *DstShadowPtr =
3983 getShadowOriginPtr(DstPtr, NextIRB, NextIRB.getInt8Ty(), Align(1),
3984 /*isStore*/ true)
3985 .first;
3986
3987 NextIRB.CreateMemCpy(DstShadowPtr, Align(1), SrcShadowPtr, Align(1), Size);
3988 if (MS.TrackOrigins) {
3989 Value *SrcOrigin = NextIRB.CreateAlignedLoad(MS.OriginTy, SrcOriginPtr,
3991 Value *NewOrigin = updateOrigin(SrcOrigin, NextIRB);
3992 NextIRB.CreateCall(MS.MsanSetOriginFn, {DstPtr, Size, NewOrigin});
3993 }
3994 }
3995
3996 void visitLibAtomicStore(CallBase &CB) {
3997 IRBuilder<> IRB(&CB);
3998 Value *Size = CB.getArgOperand(0);
3999 Value *DstPtr = CB.getArgOperand(2);
4000 Value *Ordering = CB.getArgOperand(3);
4001 // Convert the call to have at least Release ordering to make sure
4002 // the shadow operations aren't reordered after it.
4003 Value *NewOrdering =
4004 IRB.CreateExtractElement(makeAddReleaseOrderingTable(IRB), Ordering);
4005 CB.setArgOperand(3, NewOrdering);
4006
4007 Value *DstShadowPtr =
4008 getShadowOriginPtr(DstPtr, IRB, IRB.getInt8Ty(), Align(1),
4009 /*isStore*/ true)
4010 .first;
4011
4012 // Atomic store always paints clean shadow/origin. See file header.
4013 IRB.CreateMemSet(DstShadowPtr, getCleanShadow(IRB.getInt8Ty()), Size,
4014 Align(1));
4015 }
4016
4017 void visitCallBase(CallBase &CB) {
4018 assert(!CB.getMetadata(LLVMContext::MD_nosanitize));
4019 if (CB.isInlineAsm()) {
4020 // For inline asm (either a call to asm function, or callbr instruction),
4021 // do the usual thing: check argument shadow and mark all outputs as
4022 // clean. Note that any side effects of the inline asm that are not
4023 // immediately visible in its constraints are not handled.
4024 if (ClHandleAsmConservative && MS.CompileKernel)
4025 visitAsmInstruction(CB);
4026 else
4027 visitInstruction(CB);
4028 return;
4029 }
4030 LibFunc LF;
4031 if (TLI->getLibFunc(CB, LF)) {
4032 // libatomic.a functions need to have special handling because there isn't
4033 // a good way to intercept them or compile the library with
4034 // instrumentation.
4035 switch (LF) {
4036 case LibFunc_atomic_load:
4037 if (!isa<CallInst>(CB)) {
4038 llvm::errs() << "MSAN -- cannot instrument invoke of libatomic load."
4039 "Ignoring!\n";
4040 break;
4041 }
4042 visitLibAtomicLoad(CB);
4043 return;
4044 case LibFunc_atomic_store:
4045 visitLibAtomicStore(CB);
4046 return;
4047 default:
4048 break;
4049 }
4050 }
4051
4052 if (auto *Call = dyn_cast<CallInst>(&CB)) {
4053 assert(!isa<IntrinsicInst>(Call) && "intrinsics are handled elsewhere");
4054
4055 // We are going to insert code that relies on the fact that the callee
4056 // will become a non-readonly function after it is instrumented by us. To
4057 // prevent this code from being optimized out, mark that function
4058 // non-readonly in advance.
4059 // TODO: We can likely do better than dropping memory() completely here.
4061 B.addAttribute(Attribute::Memory).addAttribute(Attribute::Speculatable);
4062
4063 Call->removeFnAttrs(B);
4064 if (Function *Func = Call->getCalledFunction()) {
4065 Func->removeFnAttrs(B);
4066 }
4067
4069 }
4070 IRBuilder<> IRB(&CB);
4071 bool MayCheckCall = MS.EagerChecks;
4072 if (Function *Func = CB.getCalledFunction()) {
4073 // __sanitizer_unaligned_{load,store} functions may be called by users
4074 // and always expects shadows in the TLS. So don't check them.
4075 MayCheckCall &= !Func->getName().startswith("__sanitizer_unaligned_");
4076 }
4077
4078 unsigned ArgOffset = 0;
4079 LLVM_DEBUG(dbgs() << " CallSite: " << CB << "\n");
4080 for (const auto &[i, A] : llvm::enumerate(CB.args())) {
4081 if (!A->getType()->isSized()) {
4082 LLVM_DEBUG(dbgs() << "Arg " << i << " is not sized: " << CB << "\n");
4083 continue;
4084 }
4085 unsigned Size = 0;
4086 const DataLayout &DL = F.getParent()->getDataLayout();
4087
4088 bool ByVal = CB.paramHasAttr(i, Attribute::ByVal);
4089 bool NoUndef = CB.paramHasAttr(i, Attribute::NoUndef);
4090 bool EagerCheck = MayCheckCall && !ByVal && NoUndef;
4091
4092 if (EagerCheck) {
4093 insertShadowCheck(A, &CB);
4094 Size = DL.getTypeAllocSize(A->getType());
4095 } else {
4096 Value *Store = nullptr;
4097 // Compute the Shadow for arg even if it is ByVal, because
4098 // in that case getShadow() will copy the actual arg shadow to
4099 // __msan_param_tls.
4100 Value *ArgShadow = getShadow(A);
4101 Value *ArgShadowBase = getShadowPtrForArgument(A, IRB, ArgOffset);
4102 LLVM_DEBUG(dbgs() << " Arg#" << i << ": " << *A
4103 << " Shadow: " << *ArgShadow << "\n");
4104 if (ByVal) {
4105 // ByVal requires some special handling as it's too big for a single
4106 // load
4107 assert(A->getType()->isPointerTy() &&
4108 "ByVal argument is not a pointer!");
4109 Size = DL.getTypeAllocSize(CB.getParamByValType(i));
4110 if (ArgOffset + Size > kParamTLSSize)
4111 break;
4112 const MaybeAlign ParamAlignment(CB.getParamAlign(i));
4113 MaybeAlign Alignment = std::nullopt;
4114 if (ParamAlignment)
4115 Alignment = std::min(*ParamAlignment, kShadowTLSAlignment);
4116 Value *AShadowPtr, *AOriginPtr;
4117 std::tie(AShadowPtr, AOriginPtr) =
4118 getShadowOriginPtr(A, IRB, IRB.getInt8Ty(), Alignment,
4119 /*isStore*/ false);
4120 if (!PropagateShadow) {
4121 Store = IRB.CreateMemSet(ArgShadowBase,
4123 Size, Alignment);
4124 } else {
4125 Store = IRB.CreateMemCpy(ArgShadowBase, Alignment, AShadowPtr,
4126 Alignment, Size);
4127 if (MS.TrackOrigins) {
4128 Value *ArgOriginBase = getOriginPtrForArgument(A, IRB, ArgOffset);
4129 // FIXME: OriginSize should be:
4130 // alignTo(A % kMinOriginAlignment + Size, kMinOriginAlignment)
4131 unsigned OriginSize = alignTo(Size, kMinOriginAlignment);
4132 IRB.CreateMemCpy(
4133 ArgOriginBase,
4134 /* by origin_tls[ArgOffset] */ kMinOriginAlignment,
4135 AOriginPtr,
4136 /* by getShadowOriginPtr */ kMinOriginAlignment, OriginSize);
4137 }
4138 }
4139 } else {
4140 // Any other parameters mean we need bit-grained tracking of uninit
4141 // data
4142 Size = DL.getTypeAllocSize(A->getType());
4143 if (ArgOffset + Size > kParamTLSSize)
4144 break;
4145 Store = IRB.CreateAlignedStore(ArgShadow, ArgShadowBase,
4147 Constant *Cst = dyn_cast<Constant>(ArgShadow);
4148 if (MS.TrackOrigins && !(Cst && Cst->isNullValue())) {
4149 IRB.CreateStore(getOrigin(A),
4150 getOriginPtrForArgument(A, IRB, ArgOffset));
4151 }
4152 }
4153 (void)Store;
4154 assert(Store != nullptr);
4155 LLVM_DEBUG(dbgs() << " Param:" << *Store << "\n");
4156 }
4157 assert(Size != 0);
4158 ArgOffset += alignTo(Size, kShadowTLSAlignment);
4159 }
4160 LLVM_DEBUG(dbgs() << " done with call args\n");
4161
4162 FunctionType *FT = CB.getFunctionType();
4163 if (FT->isVarArg()) {
4164 VAHelper->visitCallBase(CB, IRB);
4165 }
4166
4167 // Now, get the shadow for the RetVal.
4168 if (!CB.getType()->isSized())
4169 return;
4170 // Don't emit the epilogue for musttail call returns.
4171 if (isa<CallInst>(CB) && cast<CallInst>(CB).isMustTailCall())
4172 return;
4173
4174 if (MayCheckCall && CB.hasRetAttr(Attribute::NoUndef)) {
4175 setShadow(&CB, getCleanShadow(&CB));
4176 setOrigin(&CB, getCleanOrigin());
4177 return;
4178 }
4179
4180 IRBuilder<> IRBBefore(&CB);
4181 // Until we have full dynamic coverage, make sure the retval shadow is 0.
4182 Value *Base = getShadowPtrForRetval(&CB, IRBBefore);
4183 IRBBefore.CreateAlignedStore(getCleanShadow(&CB), Base,
4185 BasicBlock::iterator NextInsn;
4186 if (isa<CallInst>(CB)) {
4187 NextInsn = ++CB.getIterator();
4188 assert(NextInsn != CB.getParent()->end());
4189 } else {
4190 BasicBlock *NormalDest = cast<InvokeInst>(CB).getNormalDest();
4191 if (!NormalDest->getSinglePredecessor()) {
4192 // FIXME: this case is tricky, so we are just conservative here.
4193 // Perhaps we need to split the edge between this BB and NormalDest,
4194 // but a naive attempt to use SplitEdge leads to a crash.
4195 setShadow(&CB, getCleanShadow(&CB));
4196 setOrigin(&CB, getCleanOrigin());
4197 return;
4198 }
4199 // FIXME: NextInsn is likely in a basic block that has not been visited
4200 // yet. Anything inserted there will be instrumented by MSan later!
4201 NextInsn = NormalDest->getFirstInsertionPt();
4202 assert(NextInsn != NormalDest->end() &&
4203 "Could not find insertion point for retval shadow load");
4204 }
4205 IRBuilder<> IRBAfter(&*NextInsn);
4206 Value *RetvalShadow = IRBAfter.CreateAlignedLoad(
4207 getShadowTy(&CB), getShadowPtrForRetval(&CB, IRBAfter),
4208 kShadowTLSAlignment, "_msret");
4209 setShadow(&CB, RetvalShadow);
4210 if (MS.TrackOrigins)
4211 setOrigin(&CB, IRBAfter.CreateLoad(MS.OriginTy,
4212 getOriginPtrForRetval(IRBAfter)));
4213 }
4214
4215 bool isAMustTailRetVal(Value *RetVal) {
4216 if (auto *I = dyn_cast<BitCastInst>(RetVal)) {
4217 RetVal = I->getOperand(0);
4218 }
4219 if (auto *I = dyn_cast<CallInst>(RetVal)) {
4220 return I->isMustTailCall();
4221 }
4222 return false;
4223 }
4224
4225 void visitReturnInst(ReturnInst &I) {
4226 IRBuilder<> IRB(&I);
4227 Value *RetVal = I.getReturnValue();
4228 if (!RetVal)
4229 return;
4230 // Don't emit the epilogue for musttail call returns.
4231 if (isAMustTailRetVal(RetVal))
4232 return;
4233 Value *ShadowPtr = getShadowPtrForRetval(RetVal, IRB);
4234 bool HasNoUndef = F.hasRetAttribute(Attribute::NoUndef);
4235 bool StoreShadow = !(MS.EagerChecks && HasNoUndef);
4236 // FIXME: Consider using SpecialCaseList to specify a list of functions that
4237 // must always return fully initialized values. For now, we hardcode "main".
4238 bool EagerCheck = (MS.EagerChecks && HasNoUndef) || (F.getName() == "main");
4239
4240 Value *Shadow = getShadow(RetVal);
4241 bool StoreOrigin = true;
4242 if (EagerCheck) {
4243 insertShadowCheck(RetVal, &I);
4244 Shadow = getCleanShadow(RetVal);
4245 StoreOrigin = false;
4246 }
4247
4248 // The caller may still expect information passed over TLS if we pass our
4249 // check
4250 if (StoreShadow) {
4251 IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment);
4252 if (MS.TrackOrigins && StoreOrigin)
4253 IRB.CreateStore(getOrigin(RetVal), getOriginPtrForRetval(IRB));
4254 }
4255 }
4256
4257 void visitPHINode(PHINode &I) {
4258 IRBuilder<> IRB(&I);
4259 if (!PropagateShadow) {
4260 setShadow(&I, getCleanShadow(&I));
4261 setOrigin(&I, getCleanOrigin());
4262 return;
4263 }
4264
4265 ShadowPHINodes.push_back(&I);
4266 setShadow(&I, IRB.CreatePHI(getShadowTy(&I), I.getNumIncomingValues(),
4267 "_msphi_s"));
4268 if (MS.TrackOrigins)
4269 setOrigin(
4270 &I, IRB.CreatePHI(MS.OriginTy, I.getNumIncomingValues(), "_msphi_o"));
4271 }
4272
4273 Value *getLocalVarIdptr(AllocaInst &I) {
4274 ConstantInt *IntConst =
4275 ConstantInt::get(Type::getInt32Ty((*F.getParent()).getContext()), 0);
4276 return new GlobalVariable(*F.getParent(), IntConst->getType(),
4277 /*isConstant=*/false, GlobalValue::PrivateLinkage,
4278 IntConst);
4279 }
4280
4281 Value *getLocalVarDescription(AllocaInst &I) {
4282 return createPrivateConstGlobalForString(*F.getParent(), I.getName());
4283 }
4284
4285 void poisonAllocaUserspace(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
4286 if (PoisonStack && ClPoisonStackWithCall) {
4287 IRB.CreateCall(MS.MsanPoisonStackFn,
4288 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len});
4289 } else {
4290 Value *ShadowBase, *OriginBase;
4291 std::tie(ShadowBase, OriginBase) = getShadowOriginPtr(
4292 &I, IRB, IRB.getInt8Ty(), Align(1), /*isStore*/ true);
4293
4294 Value *PoisonValue = IRB.getInt8(PoisonStack ? ClPoisonStackPattern : 0);
4295 IRB.CreateMemSet(ShadowBase, PoisonValue, Len, I.getAlign());
4296 }
4297
4298 if (PoisonStack && MS.TrackOrigins) {
4299 Value *Idptr = getLocalVarIdptr(I);
4300 if (ClPrintStackNames) {
4301 Value *Descr = getLocalVarDescription(I);
4302 IRB.CreateCall(MS.MsanSetAllocaOriginWithDescriptionFn,
4303 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len,
4304 IRB.CreatePointerCast(Idptr, IRB.getInt8PtrTy()),
4305 IRB.CreatePointerCast(Descr, IRB.getInt8PtrTy())});
4306 } else {
4307 IRB.CreateCall(MS.MsanSetAllocaOriginNoDescriptionFn,
4308 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len,
4309 IRB.CreatePointerCast(Idptr, IRB.getInt8PtrTy())});
4310 }
4311 }
4312 }
4313
4314 void poisonAllocaKmsan(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
4315 Value *Descr = getLocalVarDescription(I);
4316 if (PoisonStack) {
4317 IRB.CreateCall(MS.MsanPoisonAllocaFn,
4318 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len,
4319 IRB.CreatePointerCast(Descr, IRB.getInt8PtrTy())});
4320 } else {
4321 IRB.CreateCall(MS.MsanUnpoisonAllocaFn,
4322 {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len});
4323 }
4324 }
4325
4326 void instrumentAlloca(AllocaInst &I, Instruction *InsPoint = nullptr) {
4327 if (!InsPoint)
4328 InsPoint = &I;
4329 NextNodeIRBuilder IRB(InsPoint);
4330 const DataLayout &DL = F.getParent()->getDataLayout();
4331 uint64_t TypeSize = DL.getTypeAllocSize(I.getAllocatedType());
4332 Value *Len = ConstantInt::get(MS.IntptrTy, TypeSize);
4333 if (I.isArrayAllocation())
4334 Len = IRB.CreateMul(Len,
4335 IRB.CreateZExtOrTrunc(I.getArraySize(), MS.IntptrTy));
4336
4337 if (MS.CompileKernel)
4338 poisonAllocaKmsan(I, IRB, Len);
4339 else
4340 poisonAllocaUserspace(I, IRB, Len);
4341 }
4342
4343 void visitAllocaInst(AllocaInst &I) {
4344 setShadow(&I, getCleanShadow(&I));
4345 setOrigin(&I, getCleanOrigin());
4346 // We'll get to this alloca later unless it's poisoned at the corresponding
4347 // llvm.lifetime.start.
4348 AllocaSet.insert(&I);
4349 }
4350
4351 void visitSelectInst(SelectInst &I) {
4352 IRBuilder<> IRB(&I);
4353 // a = select b, c, d
4354 Value *B = I.getCondition();
4355 Value *C = I.getTrueValue();
4356 Value *D = I.getFalseValue();
4357 Value *Sb = getShadow(B);
4358 Value *Sc = getShadow(C);
4359 Value *Sd = getShadow(D);
4360
4361 // Result shadow if condition shadow is 0.
4362 Value *Sa0 = IRB.CreateSelect(B, Sc, Sd);
4363 Value *Sa1;
4364 if (I.getType()->isAggregateType()) {
4365 // To avoid "sign extending" i1 to an arbitrary aggregate type, we just do
4366 // an extra "select". This results in much more compact IR.
4367 // Sa = select Sb, poisoned, (select b, Sc, Sd)
4368 Sa1 = getPoisonedShadow(getShadowTy(I.getType()));
4369 } else {
4370 // Sa = select Sb, [ (c^d) | Sc | Sd ], [ b ? Sc : Sd ]
4371 // If Sb (condition is poisoned), look for bits in c and d that are equal
4372 // and both unpoisoned.
4373 // If !Sb (condition is unpoisoned), simply pick one of Sc and Sd.
4374
4375 // Cast arguments to shadow-compatible type.
4376 C = CreateAppToShadowCast(IRB, C);
4377 D = CreateAppToShadowCast(IRB, D);
4378
4379 // Result shadow if condition shadow is 1.
4380 Sa1 = IRB.CreateOr({IRB.CreateXor(C, D), Sc, Sd});
4381 }
4382 Value *Sa = IRB.CreateSelect(Sb, Sa1, Sa0, "_msprop_select");
4383 setShadow(&I, Sa);
4384 if (MS.TrackOrigins) {
4385 // Origins are always i32, so any vector conditions must be flattened.
4386 // FIXME: consider tracking vector origins for app vectors?
4387 if (B->getType()->isVectorTy()) {
4388 B = convertToBool(B, IRB);
4389 Sb = convertToBool(Sb, IRB);
4390 }
4391 // a = select b, c, d
4392 // Oa = Sb ? Ob : (b ? Oc : Od)
4393 setOrigin(
4394 &I, IRB.CreateSelect(Sb, getOrigin(I.getCondition()),
4395 IRB.CreateSelect(B, getOrigin(I.getTrueValue()),
4396 getOrigin(I.getFalseValue()))));
4397 }
4398 }
4399
4400 void visitLandingPadInst(LandingPadInst &I) {
4401 // Do nothing.
4402 // See https://github.com/google/sanitizers/issues/504
4403 setShadow(&I, getCleanShadow(&I));
4404 setOrigin(&I, getCleanOrigin());
4405 }
4406
4407 void visitCatchSwitchInst(CatchSwitchInst &I) {
4408 setShadow(&I, getCleanShadow(&I));
4409 setOrigin(&I, getCleanOrigin());
4410 }
4411
4412 void visitFuncletPadInst(FuncletPadInst &I) {
4413 setShadow(&I, getCleanShadow(&I));
4414 setOrigin(&I, getCleanOrigin());
4415 }
4416
4417 void visitGetElementPtrInst(GetElementPtrInst &I) { handleShadowOr(I); }
4418
4419 void visitExtractValueInst(ExtractValueInst &I) {
4420 IRBuilder<> IRB(&I);
4421 Value *Agg = I.getAggregateOperand();
4422 LLVM_DEBUG(dbgs() << "ExtractValue: " << I << "\n");
4423 Value *AggShadow = getShadow(Agg);
4424 LLVM_DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
4425 Value *ResShadow = IRB.CreateExtractValue(AggShadow, I.getIndices());
4426 LLVM_DEBUG(dbgs() << " ResShadow: " << *ResShadow << "\n");
4427 setShadow(&I, ResShadow);
4428 setOriginForNaryOp(I);
4429 }
4430
4431 void visitInsertValueInst(InsertValueInst &I) {
4432 IRBuilder<> IRB(&I);
4433 LLVM_DEBUG(dbgs() << "InsertValue: " << I << "\n");
4434 Value *AggShadow = getShadow(I.getAggregateOperand());
4435 Value *InsShadow = getShadow(I.getInsertedValueOperand());
4436 LLVM_DEBUG(dbgs() << " AggShadow: " << *AggShadow << "\n");
4437 LLVM_DEBUG(dbgs() << " InsShadow: " << *InsShadow << "\n");
4438 Value *Res = IRB.CreateInsertValue(AggShadow, InsShadow, I.getIndices());
4439 LLVM_DEBUG(dbgs() << " Res: " << *Res << "\n");
4440 setShadow(&I, Res);
4441 setOriginForNaryOp(I);
4442 }
4443
4444 void dumpInst(Instruction &I) {
4445 if (CallInst *CI = dyn_cast<CallInst>(&I)) {
4446 errs() << "ZZZ call " << CI->getCalledFunction()->getName() << "\n";
4447 } else {
4448 errs() << "ZZZ " << I.getOpcodeName() << "\n";
4449 }
4450 errs() << "QQQ " << I << "\n";
4451 }
4452
4453 void visitResumeInst(ResumeInst &I) {
4454 LLVM_DEBUG(dbgs() << "Resume: " << I << "\n");
4455 // Nothing to do here.
4456 }
4457
4458 void visitCleanupReturnInst(CleanupReturnInst &CRI) {
4459 LLVM_DEBUG(dbgs() << "CleanupReturn: " << CRI << "\n");
4460 // Nothing to do here.
4461 }
4462
4463 void visitCatchReturnInst(CatchReturnInst &CRI) {
4464 LLVM_DEBUG(dbgs() << "CatchReturn: " << CRI << "\n");
4465 // Nothing to do here.
4466 }
4467
4468 void instrumentAsmArgument(Value *Operand, Type *ElemTy, Instruction &I,
4469 IRBuilder<> &IRB, const DataLayout &DL,
4470 bool isOutput) {
4471 // For each assembly argument, we check its value for being initialized.
4472 // If the argument is a pointer, we assume it points to a single element
4473 // of the corresponding type (or to a 8-byte word, if the type is unsized).
4474 // Each such pointer is instrumented with a call to the runtime library.
4475 Type *OpType = Operand->getType();
4476 // Check the operand value itself.
4477 insertShadowCheck(Operand, &I);
4478 if (!OpType->isPointerTy() || !isOutput) {
4479 assert(!isOutput);
4480 return;
4481 }
4482 if (!ElemTy->isSized())
4483 return;
4484 Value *Ptr = IRB.CreatePointerCast(Operand, IRB.getInt8PtrTy());
4485 Value *SizeVal =
4486 IRB.CreateTypeSize(MS.IntptrTy, DL.getTypeStoreSize(ElemTy));
4487 IRB.CreateCall(MS.MsanInstrumentAsmStoreFn, {Ptr, SizeVal});
4488 }
4489
4490 /// Get the number of output arguments returned by pointers.
4491 int getNumOutputArgs(InlineAsm *IA, CallBase *CB) {
4492 int NumRetOutputs = 0;
4493 int NumOutputs = 0;
4494 Type *RetTy = cast<Value>(CB)->getType();
4495 if (!RetTy->isVoidTy()) {
4496 // Register outputs are returned via the CallInst return value.
4497 auto *ST = dyn_cast<StructType>(RetTy);
4498 if (ST)
4499 NumRetOutputs = ST->getNumElements();
4500 else
4501 NumRetOutputs = 1;
4502 }
4503 InlineAsm::ConstraintInfoVector Constraints = IA->ParseConstraints();
4504 for (const InlineAsm::ConstraintInfo &Info : Constraints) {
4505 switch (Info.Type) {
4507 NumOutputs++;
4508 break;
4509 default:
4510 break;
4511 }
4512 }
4513 return NumOutputs - NumRetOutputs;
4514 }
4515
4516 void visitAsmInstruction(Instruction &I) {
4517 // Conservative inline assembly handling: check for poisoned shadow of
4518 // asm() arguments, then unpoison the result and all the memory locations
4519 // pointed to by those arguments.
4520 // An inline asm() statement in C++ contains lists of input and output
4521 // arguments used by the assembly code. These are mapped to operands of the
4522 // CallInst as follows:
4523 // - nR register outputs ("=r) are returned by value in a single structure
4524 // (SSA value of the CallInst);
4525 // - nO other outputs ("=m" and others) are returned by pointer as first
4526 // nO operands of the CallInst;
4527 // - nI inputs ("r", "m" and others) are passed to CallInst as the
4528 // remaining nI operands.
4529 // The total number of asm() arguments in the source is nR+nO+nI, and the
4530 // corresponding CallInst has nO+nI+1 operands (the last operand is the
4531 // function to be called).
4532 const DataLayout &DL = F.getParent()->getDataLayout();
4533 CallBase *CB = cast<CallBase>(&I);
4534 IRBuilder<> IRB(&I);
4535 InlineAsm *IA = cast<InlineAsm>(CB->getCalledOperand());
4536 int OutputArgs = getNumOutputArgs(IA, CB);
4537 // The last operand of a CallInst is the function itself.
4538 int NumOperands = CB->getNumOperands() - 1;
4539
4540 // Check input arguments. Doing so before unpoisoning output arguments, so
4541 // that we won't overwrite uninit values before checking them.
4542 for (int i = OutputArgs; i < NumOperands; i++) {
4543 Value *Operand = CB->getOperand(i);
4544 instrumentAsmArgument(Operand, CB->getParamElementType(i), I, IRB, DL,
4545 /*isOutput*/ false);
4546 }
4547 // Unpoison output arguments. This must happen before the actual InlineAsm
4548 // call, so that the shadow for memory published in the asm() statement
4549 // remains valid.
4550 for (int i = 0; i < OutputArgs; i++) {
4551 Value *Operand = CB->getOperand(i);
4552 instrumentAsmArgument(Operand, CB->getParamElementType(i), I, IRB, DL,
4553 /*isOutput*/ true);
4554 }
4555
4556 setShadow(&I, getCleanShadow(&I));
4557 setOrigin(&I, getCleanOrigin());
4558 }
4559
4560 void visitFreezeInst(FreezeInst &I) {
4561 // Freeze always returns a fully defined value.
4562 setShadow(&I, getCleanShadow(&I));
4563 setOrigin(&I, getCleanOrigin());
4564 }
4565
4566 void visitInstruction(Instruction &I) {
4567 // Everything else: stop propagating and check for poisoned shadow.
4569 dumpInst(I);
4570 LLVM_DEBUG(dbgs() << "DEFAULT: " << I << "\n");
4571 for (size_t i = 0, n = I.getNumOperands(); i < n; i++) {
4572 Value *Operand = I.getOperand(i);
4573 if (Operand->getType()->isSized())
4574 insertShadowCheck(Operand, &I);
4575 }
4576 setShadow(&I, getCleanShadow(&I));
4577 setOrigin(&I, getCleanOrigin());
4578 }
4579};
4580
4581/// AMD64-specific implementation of VarArgHelper.
4582struct VarArgAMD64Helper : public VarArgHelper {
4583 // An unfortunate workaround for asymmetric lowering of va_arg stuff.
4584 // See a comment in visitCallBase for more details.
4585 static const unsigned AMD64GpEndOffset = 48; // AMD64 ABI Draft 0.99.6 p3.5.7
4586 static const unsigned AMD64FpEndOffsetSSE = 176;
4587 // If SSE is disabled, fp_offset in va_list is zero.
4588 static const unsigned AMD64FpEndOffsetNoSSE = AMD64GpEndOffset;
4589
4590 unsigned AMD64FpEndOffset;
4591 Function &F;
4592 MemorySanitizer &MS;
4593 MemorySanitizerVisitor &MSV;
4594 Value *VAArgTLSCopy = nullptr;
4595 Value *VAArgTLSOriginCopy = nullptr;
4596 Value *VAArgOverflowSize = nullptr;
4597
4598 SmallVector<CallInst *, 16> VAStartInstrumentationList;
4599
4600 enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory };
4601
4602 VarArgAMD64Helper(Function &F, MemorySanitizer &MS,
4603 MemorySanitizerVisitor &MSV)
4604 : F(F), MS(MS), MSV(MSV) {
4605 AMD64FpEndOffset = AMD64FpEndOffsetSSE;
4606 for (const auto &Attr : F.getAttributes().getFnAttrs()) {
4607 if (Attr.isStringAttribute() &&
4608 (Attr.getKindAsString() == "target-features")) {
4609 if (Attr.getValueAsString().contains("-sse"))
4610 AMD64FpEndOffset = AMD64FpEndOffsetNoSSE;
4611 break;
4612 }
4613 }
4614 }
4615
4616 ArgKind classifyArgument(Value *arg) {
4617 // A very rough approximation of X86_64 argument classification rules.
4618 Type *T = arg->getType();
4619 if (T->isFPOrFPVectorTy() || T->isX86_MMXTy())
4620 return AK_FloatingPoint;
4621 if (T->isIntegerTy() && T->getPrimitiveSizeInBits() <= 64)
4622 return AK_GeneralPurpose;
4623 if (T->isPointerTy())
4624 return AK_GeneralPurpose;
4625 return AK_Memory;
4626 }
4627
4628 // For VarArg functions, store the argument shadow in an ABI-specific format
4629 // that corresponds to va_list layout.
4630 // We do this because Clang lowers va_arg in the frontend, and this pass
4631 // only sees the low level code that deals with va_list internals.
4632 // A much easier alternative (provided that Clang emits va_arg instructions)
4633 // would have been to associate each live instance of va_list with a copy of
4634 // MSanParamTLS, and extract shadow on va_arg() call in the argument list
4635 // order.
4636 void visitCallBase(CallBase &CB, IRBuilder<> &IRB) override {
4637 unsigned GpOffset = 0;
4638 unsigned FpOffset = AMD64GpEndOffset;
4639 unsigned OverflowOffset = AMD64FpEndOffset;
4640 const DataLayout &DL = F.getParent()->getDataLayout();
4641 for (const auto &[ArgNo, A] : llvm::enumerate(CB.args())) {
4642 bool IsFixed = ArgNo < CB.getFunctionType()->getNumParams();
4643 bool IsByVal = CB.paramHasAttr(ArgNo, Attribute::ByVal);
4644 if (IsByVal) {
4645 // ByVal arguments always go to the overflow area.
4646 // Fixed arguments passed through the overflow area will be stepped
4647 // over by va_start, so don't count them towards the offset.
4648 if (IsFixed)
4649 continue;
4650 assert(A->getType()->isPointerTy());
4651 Type *RealTy = CB.getParamByValType(ArgNo);
4652 uint64_t ArgSize = DL.getTypeAllocSize(RealTy);
4653 Value *ShadowBase = getShadowPtrForVAArgument(
4654 RealTy, IRB, OverflowOffset, alignTo(ArgSize, 8));
4655 Value *OriginBase = nullptr;
4656 if (MS.TrackOrigins)
4657 OriginBase = getOriginPtrForVAArgument(RealTy, IRB, OverflowOffset);
4658 OverflowOffset += alignTo(ArgSize, 8);
4659 if (!ShadowBase)
4660 continue;
4661 Value *ShadowPtr, *OriginPtr;
4662 std::tie(ShadowPtr, OriginPtr) =
4663 MSV.getShadowOriginPtr(A, IRB, IRB.getInt8Ty(), kShadowTLSAlignment,
4664 /*isStore*/ false);
4665
4666 IRB.CreateMemCpy(ShadowBase, kShadowTLSAlignment, ShadowPtr,
4667 kShadowTLSAlignment, ArgSize);
4668 if (MS.TrackOrigins)
4669 IRB.CreateMemCpy(OriginBase, kShadowTLSAlignment, OriginPtr,
4670 kShadowTLSAlignment, ArgSize);
4671 } else {
4672 ArgKind AK = classifyArgument(A);
4673 if (AK == AK_GeneralPurpose && GpOffset >= AMD64GpEndOffset)
4674 AK = AK_Memory;
4675 if (AK == AK_FloatingPoint && FpOffset >= AMD64FpEndOffset)
4676 AK = AK_Memory;
4677 Value *ShadowBase, *OriginBase = nullptr;
4678 switch (AK) {
4679 case AK_GeneralPurpose:
4680 ShadowBase =
4681 getShadowPtrForVAArgument(A->getType(), IRB, GpOffset, 8);
4682 if (MS.TrackOrigins)
4683 OriginBase = getOriginPtrForVAArgument(A->getType(), IRB, GpOffset);
4684 GpOffset += 8;
4685 break;
4686 case AK_FloatingPoint:
4687 ShadowBase =
4688 getShadowPtrForVAArgument(A->getType(), IRB, FpOffset, 16);
4689 if (MS.TrackOrigins)
4690 OriginBase = getOriginPtrForVAArgument(A->getType(), IRB, FpOffset);
4691 FpOffset += 16;
4692 break;
4693 case AK_Memory:
4694 if (IsFixed)
4695 continue;
4696 uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
4697 ShadowBase =
4698 getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset, 8);
4699 if (MS.TrackOrigins)
4700 OriginBase =
4701 getOriginPtrForVAArgument(A->getType(), IRB, OverflowOffset);
4702 OverflowOffset += alignTo(ArgSize, 8);
4703 }
4704 // Take fixed arguments into account for GpOffset and FpOffset,
4705 // but don't actually store shadows for them.
4706 // TODO(glider): don't call get*PtrForVAArgument() for them.
4707 if (IsFixed)
4708 continue;
4709 if (!ShadowBase)
4710 continue;
4711 Value *Shadow = MSV.getShadow(A);
4712 IRB.CreateAlignedStore(Shadow, ShadowBase, kShadowTLSAlignment);
4713 if (MS.TrackOrigins) {
4714 Value *Origin = MSV.getOrigin(A);
4715 TypeSize StoreSize = DL.getTypeStoreSize(Shadow->getType());
4716 MSV.paintOrigin(IRB, Origin, OriginBase, StoreSize,
4718 }
4719 }
4720 }
4721 Constant *OverflowSize =
4722 ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AMD64FpEndOffset);
4723 IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
4724 }
4725
4726 /// Compute the shadow address for a given va_arg.
4727 Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4728 unsigned ArgOffset, unsigned ArgSize) {
4729 // Make sure we don't overflow __msan_va_arg_tls.
4730 if (ArgOffset + ArgSize > kParamTLSSize)
4731 return nullptr;
4732 Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4733 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4734 return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4735 "_msarg_va_s");
4736 }
4737
4738 /// Compute the origin address for a given va_arg.
4739 Value *getOriginPtrForVAArgument(Type *Ty, IRBuilder<> &IRB, int ArgOffset) {
4740 Value *Base = IRB.CreatePointerCast(MS.VAArgOriginTLS, MS.IntptrTy);
4741 // getOriginPtrForVAArgument() is always called after
4742 // getShadowPtrForVAArgument(), so __msan_va_arg_origin_tls can never
4743 // overflow.
4744 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4745 return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0),
4746 "_msarg_va_o");
4747 }
4748
4749 void unpoisonVAListTagForInst(IntrinsicInst &I) {
4750 IRBuilder<> IRB(&I);
4751 Value *VAListTag = I.getArgOperand(0);
4752 Value *ShadowPtr, *OriginPtr;
4753 const Align Alignment = Align(8);
4754 std::tie(ShadowPtr, OriginPtr) =
4755 MSV.getShadowOriginPtr(VAListTag, IRB, IRB.getInt8Ty(), Alignment,
4756 /*isStore*/ true);
4757
4758 // Unpoison the whole __va_list_tag.
4759 // FIXME: magic ABI constants.
4760 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4761 /* size */ 24, Alignment, false);
4762 // We shouldn't need to zero out the origins, as they're only checked for
4763 // nonzero shadow.
4764 }
4765
4766 void visitVAStartInst(VAStartInst &I) override {
4767 if (F.getCallingConv() == CallingConv::Win64)
4768 return;
4769 VAStartInstrumentationList.push_back(&I);
4770 unpoisonVAListTagForInst(I);
4771 }
4772
4773 void visitVACopyInst(VACopyInst &I) override {
4774 if (F.getCallingConv() == CallingConv::Win64)
4775 return;
4776 unpoisonVAListTagForInst(I);
4777 }
4778
4779 void finalizeInstrumentation() override {
4780 assert(!VAArgOverflowSize && !VAArgTLSCopy &&
4781 "finalizeInstrumentation called twice");
4782 if (!VAStartInstrumentationList.empty()) {
4783 // If there is a va_start in this function, make a backup copy of
4784 // va_arg_tls somewhere in the function entry block.
4785 IRBuilder<> IRB(MSV.FnPrologueEnd);
4786 VAArgOverflowSize =
4787 IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
4788 Value *CopySize = IRB.CreateAdd(
4789 ConstantInt::get(MS.IntptrTy, AMD64FpEndOffset), VAArgOverflowSize);
4790 VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4791 IRB.CreateMemCpy(VAArgTLSCopy, Align(8), MS.VAArgTLS, Align(8), CopySize);
4792 if (MS.TrackOrigins) {
4793 VAArgTLSOriginCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4794 IRB.CreateMemCpy(VAArgTLSOriginCopy, Align(8), MS.VAArgOriginTLS,
4795 Align(8), CopySize);
4796 }
4797 }
4798
4799 // Instrument va_start.
4800 // Copy va_list shadow from the backup copy of the TLS contents.
4801 for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
4802 CallInst *OrigInst = VAStartInstrumentationList[i];
4803 NextNodeIRBuilder IRB(OrigInst);
4804 Value *VAListTag = OrigInst->getArgOperand(0);
4805
4806 Type *RegSaveAreaPtrTy = Type::getInt64PtrTy(*MS.C);
4807 Value *RegSaveAreaPtrPtr = IRB.CreateIntToPtr(
4808 IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4809 ConstantInt::get(MS.IntptrTy, 16)),
4810 PointerType::get(RegSaveAreaPtrTy, 0));
4811 Value *RegSaveAreaPtr =
4812 IRB.CreateLoad(RegSaveAreaPtrTy, RegSaveAreaPtrPtr);
4813 Value *RegSaveAreaShadowPtr, *RegSaveAreaOriginPtr;
4814 const Align Alignment = Align(16);
4815 std::tie(RegSaveAreaShadowPtr, RegSaveAreaOriginPtr) =
4816 MSV.getShadowOriginPtr(RegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4817 Alignment, /*isStore*/ true);
4818 IRB.CreateMemCpy(RegSaveAreaShadowPtr, Alignment, VAArgTLSCopy, Alignment,
4819 AMD64FpEndOffset);
4820 if (MS.TrackOrigins)
4821 IRB.CreateMemCpy(RegSaveAreaOriginPtr, Alignment, VAArgTLSOriginCopy,
4822 Alignment, AMD64FpEndOffset);
4823 Type *OverflowArgAreaPtrTy = Type::getInt64PtrTy(*MS.C);
4824 Value *OverflowArgAreaPtrPtr = IRB.CreateIntToPtr(
4825 IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4826 ConstantInt::get(MS.IntptrTy, 8)),
4827 PointerType::get(OverflowArgAreaPtrTy, 0));
4828 Value *OverflowArgAreaPtr =
4829 IRB.CreateLoad(OverflowArgAreaPtrTy, OverflowArgAreaPtrPtr);
4830 Value *OverflowArgAreaShadowPtr, *OverflowArgAreaOriginPtr;
4831 std::tie(OverflowArgAreaShadowPtr, OverflowArgAreaOriginPtr) =
4832 MSV.getShadowOriginPtr(OverflowArgAreaPtr, IRB, IRB.getInt8Ty(),
4833 Alignment, /*isStore*/ true);
4834 Value *SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSCopy,
4835 AMD64FpEndOffset);
4836 IRB.CreateMemCpy(OverflowArgAreaShadowPtr, Alignment, SrcPtr, Alignment,
4837 VAArgOverflowSize);
4838 if (MS.TrackOrigins) {
4839 SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSOriginCopy,
4840 AMD64FpEndOffset);
4841 IRB.CreateMemCpy(OverflowArgAreaOriginPtr, Alignment, SrcPtr, Alignment,
4842 VAArgOverflowSize);
4843 }
4844 }
4845 }
4846};
4847
4848/// MIPS64-specific implementation of VarArgHelper.
4849struct VarArgMIPS64Helper : public VarArgHelper {
4850 Function &F;
4851 MemorySanitizer &MS;
4852 MemorySanitizerVisitor &MSV;
4853 Value *VAArgTLSCopy = nullptr;
4854 Value *VAArgSize = nullptr;
4855
4856 SmallVector<CallInst *, 16> VAStartInstrumentationList;
4857
4858 VarArgMIPS64Helper(Function &F, MemorySanitizer &MS,
4859 MemorySanitizerVisitor &MSV)
4860 : F(F), MS(MS), MSV(MSV) {}
4861
4862 void visitCallBase(CallBase &CB, IRBuilder<> &IRB) override {
4863 unsigned VAArgOffset = 0;
4864 const DataLayout &DL = F.getParent()->getDataLayout();
4865 for (Value *A :
4867 Triple TargetTriple(F.getParent()->getTargetTriple());
4868 Value *Base;
4869 uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
4870 if (TargetTriple.getArch() == Triple::mips64) {
4871 // Adjusting the shadow for argument with size < 8 to match the
4872 // placement of bits in big endian system
4873 if (ArgSize < 8)
4874 VAArgOffset += (8 - ArgSize);
4875 }
4876 Base = getShadowPtrForVAArgument(A->getType(), IRB, VAArgOffset, ArgSize);
4877 VAArgOffset += ArgSize;
4878 VAArgOffset = alignTo(VAArgOffset, 8);
4879 if (!Base)
4880 continue;
4881 IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
4882 }
4883
4884 Constant *TotalVAArgSize = ConstantInt::get(IRB.getInt64Ty(), VAArgOffset);
4885 // Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of
4886 // a new class member i.e. it is the total size of all VarArgs.
4887 IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS);
4888 }
4889
4890 /// Compute the shadow address for a given va_arg.
4891 Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4892 unsigned ArgOffset, unsigned ArgSize) {
4893 // Make sure we don't overflow __msan_va_arg_tls.
4894 if (ArgOffset + ArgSize > kParamTLSSize)
4895 return nullptr;
4896 Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4897 Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4898 return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4899 "_msarg");
4900 }
4901
4902 void visitVAStartInst(VAStartInst &I) override {
4903 IRBuilder<> IRB(&I);
4904 VAStartInstrumentationList.push_back(&I);
4905 Value *VAListTag = I.getArgOperand(0);
4906 Value *ShadowPtr, *OriginPtr;
4907 const Align Alignment = Align(8);
4908 std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4909 VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4910 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4911 /* size */ 8, Alignment, false);
4912 }
4913
4914 void visitVACopyInst(VACopyInst &I) override {
4915 IRBuilder<> IRB(&I);
4916 VAStartInstrumentationList.push_back(&I);
4917 Value *VAListTag = I.getArgOperand(0);
4918 Value *ShadowPtr, *OriginPtr;
4919 const Align Alignment = Align(8);
4920 std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4921 VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4922 IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4923 /* size */ 8, Alignment, false);
4924 }
4925
4926 void finalizeInstrumentation() override {
4927