Helper struct to manage generating runtime checks for vectorization. More...

Public Member Functions
	GeneratedRTChecks (ScalarEvolution &SE, DominatorTree DT, LoopInfo LI, TargetTransformInfo *TTI, const DataLayout &DL)

void	Create (Loop *L, const LoopAccessInfo &LAI, const SCEVPredicate &UnionPred, ElementCount VF, unsigned IC)
	Generate runtime checks in SCEVCheckBlock and MemCheckBlock, so we can accurately estimate the cost of the runtime checks. More...

InstructionCost	getCost ()

	~GeneratedRTChecks ()
	Remove the created SCEV & memory runtime check blocks & instructions, if unused. More...

BasicBlock *	emitSCEVChecks (BasicBlock Bypass, BasicBlock LoopVectorPreHeader, BasicBlock *LoopExitBlock)
	Adds the generated SCEVCheckBlock before `LoopVectorPreHeader` and adjusts the branches to branch to the vector preheader or `Bypass`, depending on the generated condition. More...

BasicBlock *	emitMemRuntimeChecks (BasicBlock Bypass, BasicBlock LoopVectorPreHeader)
	Adds the generated MemCheckBlock before `LoopVectorPreHeader` and adjusts the branches to branch to the vector preheader or `Bypass`, depending on the generated condition. More...

Detailed Description

Helper struct to manage generating runtime checks for vectorization.

The runtime checks are created up-front in temporary blocks to allow better estimating the cost and un-linked from the existing IR. After deciding to vectorize, the checks are moved back. If deciding not to vectorize, the temporary blocks are completely removed.

Definition at line 1860 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ GeneratedRTChecks()

GeneratedRTChecks::GeneratedRTChecks	(	ScalarEvolution &	SE,
		DominatorTree *	DT,
		LoopInfo *	LI,
		TargetTransformInfo *	TTI,
		const DataLayout &	DL
	)

inline

Definition at line 1886 of file LoopVectorize.cpp.

◆ ~GeneratedRTChecks()

GeneratedRTChecks::~GeneratedRTChecks ( )

inline

Remove the created SCEV & memory runtime check blocks & instructions, if unused.

Definition at line 2022 of file LoopVectorize.cpp.

References llvm::SCEVExpanderCleaner::cleanup(), llvm::BasicBlock::eraseFromParent(), llvm::ScalarEvolution::forgetValue(), llvm::SCEVExpander::getSE(), I, llvm::SCEVExpander::isInsertedInstruction(), llvm::make_early_inc_range(), llvm::SCEVExpanderCleaner::markResultUsed(), and llvm::reverse().

Member Function Documentation

◆ Create()

void GeneratedRTChecks::Create	(	Loop *	L,
		const LoopAccessInfo &	LAI,
		const SCEVPredicate &	UnionPred,
		ElementCount	VF,
		unsigned	IC
	)

inline

Generate runtime checks in SCEVCheckBlock and MemCheckBlock, so we can accurately estimate the cost of the runtime checks.

The blocks are un-linked from the IR and is added back during vector code generation. If there is no vector code generation, the check blocks are removed completely.

Definition at line 1896 of file LoopVectorize.cpp.

References llvm::addDiffRuntimeChecks(), llvm::addRuntimeChecks(), assert(), B, llvm::tgtok::Bits, llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), llvm::InnerLoopVectorizer::DT, llvm::Instruction::eraseFromParent(), llvm::DominatorTreeBase< NodeT, IsPostDom >::eraseNode(), llvm::SCEVExpander::expandCodeForPredicate(), llvm::BasicBlock::getContext(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::LoopAccessInfo::getNumRuntimePointerChecks(), llvm::LoopAccessInfo::getRuntimePointerChecking(), llvm::BasicBlock::getTerminator(), llvm::SCEVPredicate::isAlwaysTrue(), llvm::InnerLoopVectorizer::LI, llvm::Instruction::moveBefore(), llvm::LoopInfoBase< BlockT, LoopT >::removeBlock(), llvm::Value::replaceAllUsesWith(), llvm::SplitBlock(), VectorizeMemoryCheckThreshold, and llvm::InnerLoopVectorizer::VF.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ emitMemRuntimeChecks()

BasicBlock* GeneratedRTChecks::emitMemRuntimeChecks	(	BasicBlock *	Bypass,
		BasicBlock *	LoopVectorPreHeader
	)

inline

Adds the generated MemCheckBlock before LoopVectorPreHeader and adjusts the branches to branch to the vector preheader or Bypass, depending on the generated condition.

Definition at line 2090 of file LoopVectorize.cpp.

References llvm::DominatorTreeBase< NodeT, IsPostDom >::addNewBlock(), llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), llvm::BranchInst::Create(), llvm::InnerLoopVectorizer::DT, llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::BasicBlock::getSinglePredecessor(), llvm::BasicBlock::getTerminator(), llvm::InnerLoopVectorizer::LI, llvm::InnerLoopVectorizer::LoopVectorPreHeader, llvm::BasicBlock::moveBefore(), llvm::AArch64CC::PL, llvm::ReplaceInstWithInst(), llvm::Instruction::replaceSuccessorWith(), and llvm::Instruction::setDebugLoc().

Referenced by llvm::InnerLoopVectorizer::emitMemRuntimeChecks().

◆ emitSCEVChecks()

BasicBlock* GeneratedRTChecks::emitSCEVChecks	(	BasicBlock *	Bypass,
		BasicBlock *	LoopVectorPreHeader,
		BasicBlock *	LoopExitBlock
	)

inline

Adds the generated SCEVCheckBlock before LoopVectorPreHeader and adjusts the branches to branch to the vector preheader or Bypass, depending on the generated condition.

Definition at line 2054 of file LoopVectorize.cpp.

References llvm::DominatorTreeBase< NodeT, IsPostDom >::addNewBlock(), llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), Cond, llvm::BranchInst::Create(), llvm::InnerLoopVectorizer::DT, llvm::Instruction::eraseFromParent(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::BasicBlock::getSinglePredecessor(), llvm::BasicBlock::getTerminator(), llvm::InnerLoopVectorizer::LI, llvm::InnerLoopVectorizer::LoopVectorPreHeader, llvm::BasicBlock::moveBefore(), llvm::AArch64CC::PL, and llvm::ReplaceInstWithInst().