X86FenceNonTemporalStores.cpp File Reference

This pass performs forward dataflow analysis to insert llvm.x86.sse.sfence intrinsics between non-temporal stores and subsequent potential synchronization points (such as atomic operations, function calls, exception unwinding, and function returns), guaranteeing that weakly-ordered non-temporal stores are serialized before synchronization, while still allowing the performance benefits of non-temporal stores in a loop. More...

#include "X86.h"
#include "X86Subtarget.h"
#include "X86TargetMachine.h"
#include "llvm/ADT/DenseMap.h"
#include "llvm/ADT/STLExtras.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/CodeGen/TargetPassConfig.h"
#include "llvm/IR/BasicBlock.h"
#include "llvm/IR/CFG.h"
#include "llvm/IR/Function.h"
#include "llvm/IR/IRBuilder.h"
#include "llvm/IR/Instructions.h"
#include "llvm/IR/IntrinsicInst.h"
#include "llvm/IR/IntrinsicsX86.h"
#include "llvm/IR/PassManager.h"
#include "llvm/Pass.h"
#include "llvm/Support/Debug.h"

Go to the source code of this file.

Macros
#define	DEBUG_TYPE   "x86-fence-nontemporal-stores"

Functions
static bool	isNonTemporalStore (const Instruction &I)
static bool	isNonTemporalFence (const Instruction &I)
	Returns true if I is a hardware fence that serializes write-combining stores:
static bool	isSyncOrReturnPoint (const Instruction &I)
	Returns true if I is a potential cross-thread synchronization point, or which might return to a caller (either via return or throwing).
static bool	containsNonTemporalStores (Function &F)
static bool	runImpl (Function &F, const X86Subtarget *ST)
	INITIALIZE_PASS (X86FenceNonTemporalStoresLegacy, DEBUG_TYPE, "X86 Fence Non-Temporal Stores", false, false) FunctionPass *llvm

Detailed Description

This pass performs forward dataflow analysis to insert llvm.x86.sse.sfence intrinsics between non-temporal stores and subsequent potential synchronization points (such as atomic operations, function calls, exception unwinding, and function returns), guaranteeing that weakly-ordered non-temporal stores are serialized before synchronization, while still allowing the performance benefits of non-temporal stores in a loop.

Non-temporal store instructions in LLVM IR (!nontemporal metadata) are documented as optimization hints. However, on X86, regular stores follow the TSO (Total Store Order) memory model, whereas non-temporal stores (MOVNT*) write to weakly-ordered write-combining (WC) memory buffers that bypass cache hierarchies.

Without explicit memory fences, write-combining stores are not ordered with respect to other WC stores or normal WB stores. As a result, cross-thread synchronization that follows a non-temporal store may allow other threads to observe out-of-order writes, unless an SFENCE is executed.

Definition in file X86FenceNonTemporalStores.cpp.

Macro Definition Documentation

DEBUG_TYPE

#define DEBUG_TYPE   "x86-fence-nontemporal-stores"

Definition at line 52 of file X86FenceNonTemporalStores.cpp.

Function Documentation

containsNonTemporalStores()

bool containsNonTemporalStores ( Function & F )

static

Definition at line 86 of file X86FenceNonTemporalStores.cpp.

References F, I, and isNonTemporalStore().

Referenced by runImpl().

INITIALIZE_PASS()

INITIALIZE_PASS	(	X86FenceNonTemporalStoresLegacy	,
		DEBUG_TYPE	,
		"X86 Fence Non-Temporal Stores"	,
		false	,
		false	)

Definition at line 224 of file X86FenceNonTemporalStores.cpp.

References llvm::createX86FenceNonTemporalStoresLegacyPass(), and DEBUG_TYPE.

isNonTemporalFence()

bool isNonTemporalFence ( const Instruction & I )

static

Returns true if I is a hardware fence that serializes write-combining stores:

• SFENCE (llvm.x86.sse.sfence) flushes and serializes WC stores.
• MFENCE (llvm.x86.sse2.mfence) serializes all memory operations.

Note: Atomic fences (such as fence seq_cst or fence release) do NOT necessarily serialize write-combining stores.

Definition at line 65 of file X86FenceNonTemporalStores.cpp.

References llvm::dyn_cast(), and I.

Referenced by runImpl().

isNonTemporalStore()

bool isNonTemporalStore ( const Instruction & I )

static

Definition at line 54 of file X86FenceNonTemporalStores.cpp.

References I.

Referenced by containsNonTemporalStores(), and runImpl().

isSyncOrReturnPoint()

bool isSyncOrReturnPoint ( const Instruction & I )

static

Returns true if I is a potential cross-thread synchronization point, or which might return to a caller (either via return or throwing).

InvokeInst and CleanupRetInst are handled as a special case, because I.mayThrow() may return false, yet return true for their CatchSwitchInst unwind destination, which cannot have non-PHI instructions preceeding it.

Definition at line 81 of file X86FenceNonTemporalStores.cpp.

References I, and llvm::isa().

Referenced by runImpl().

runImpl()

bool runImpl ( Function & F, const X86Subtarget * ST )

static

Definition at line 96 of file X86FenceNonTemporalStores.cpp.

References Changed, containsNonTemporalStores(), llvm::SetVector< T, Vector, Set, N >::empty(), F, I, llvm::SetVector< T, Vector, Set, N >::insert(), isNonTemporalFence(), isNonTemporalStore(), isSyncOrReturnPoint(), llvm::make_early_inc_range(), llvm::SetVector< T, Vector, Set, N >::pop_back_val(), llvm::predecessors(), and llvm::successors().