InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF). More...

Inheritance diagram for llvm::InnerLoopVectorizer:

[legend]

Collaboration diagram for llvm::InnerLoopVectorizer:

[legend]

Public Types
using	VectorParts = SmallVector< Value *, 2 >
	A type for vectorized values in the new loop. More...

Public Member Functions
	InnerLoopVectorizer (Loop OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo LI, DominatorTree DT, const TargetLibraryInfo TLI, const TargetTransformInfo TTI, AssumptionCache AC, OptimizationRemarkEmitter ORE, ElementCount VecWidth, unsigned UnrollFactor, LoopVectorizationLegality LVL, LoopVectorizationCostModel CM, BlockFrequencyInfo BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks)

virtual	~InnerLoopVectorizer ()=default

virtual std::pair< BasicBlock , Value >	createVectorizedLoopSkeleton ()
	Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder. More...

void	widenCallInstruction (CallInst &I, VPValue *Def, VPUser &ArgOperands, VPTransformState &State)
	Widen a single call instruction within the innermost loop. More...

void	fixVectorizedLoop (VPTransformState &State, VPlan &Plan)
	Fix the vectorized code, taking care of header phi's, live-outs, and more. More...

bool	areSafetyChecksAdded ()

void	scalarizeInstruction (Instruction Instr, VPReplicateRecipe RepRecipe, const VPIteration &Instance, bool IfPredicateInstr, VPTransformState &State)
	A helper function to scalarize a single Instruction in the innermost loop. More...

void	packScalarIntoVectorValue (VPValue *Def, const VPIteration &Instance, VPTransformState &State)
	Construct the vector value of a scalarized value `V` one lane at a time. More...

void	vectorizeInterleaveGroup (const InterleaveGroup< Instruction > Group, ArrayRef< VPValue > VPDefs, VPTransformState &State, VPValue Addr, ArrayRef< VPValue > StoredValues, VPValue *BlockInMask=nullptr)
	Try to vectorize interleaved access group `Group` with the base address given in `Addr`, optionally masking the vector operations if `BlockInMask` is non-null. More...

void	setDebugLocFromInst (const Value *V)
	Set the debug location in the builder `Ptr` using the debug location in `V`. More...

void	fixNonInductionPHIs (VPlan &Plan, VPTransformState &State)
	Fix the non-induction PHIs in `Plan`. More...

bool	useOrderedReductions (const RecurrenceDescriptor &RdxDesc)
	Returns true if the reordering of FP operations is not allowed, but we are able to vectorize with strict in-order reductions for the given RdxDesc. More...

virtual Value *	getBroadcastInstrs (Value *V)
	Create a broadcast instruction. More...

void	addMetadata (Instruction To, Instruction From)
	Add metadata from one instruction to another. More...

void	addMetadata (ArrayRef< Value * > To, Instruction *From)
	Similar to the previous function but it adds the metadata to a vector of instructions. More...

PHINode *	getReductionResumeValue (const RecurrenceDescriptor &RdxDesc)

Protected Types
using	PhiVector = SmallVector< PHINode *, 4 >
	A small list of PHINodes. More...

using	ScalarParts = SmallVector< SmallVector< Value *, 4 >, 2 >
	A type for scalarized values in the new loop. More...

Protected Member Functions
void	fixupIVUsers (PHINode OrigPhi, const InductionDescriptor &II, Value VectorTripCount, Value EndValue, BasicBlock MiddleBlock, BasicBlock *VectorHeader, VPlan &Plan)
	Set up the values of the IVs correctly when exiting the vector loop. More...

void	fixCrossIterationPHIs (VPTransformState &State)
	Handle all cross-iteration phis in the header. More...

void	fixFirstOrderRecurrence (VPFirstOrderRecurrencePHIRecipe *PhiR, VPTransformState &State)
	Create the exit value of first order recurrences in the middle block and update their users. More...

void	fixReduction (VPReductionPHIRecipe *Phi, VPTransformState &State)
	Create code for the loop exit value of the reduction. More...

void	clearReductionWrapFlags (VPReductionPHIRecipe *PhiR, VPTransformState &State)
	Clear NSW/NUW flags from reduction instructions if necessary. More...

void	sinkScalarOperands (Instruction *PredInst)
	Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it. More...

void	truncateToMinimalBitwidths (VPTransformState &State)
	Shrinks vector element sizes to the smallest bitwidth they can be legally represented as. More...

Value *	getOrCreateTripCount (BasicBlock *InsertBlock)
	Returns (and creates if needed) the original loop trip count. More...

Value *	getOrCreateVectorTripCount (BasicBlock *InsertBlock)
	Returns (and creates if needed) the trip count of the widened loop. More...

Value *	createBitOrPointerCast (Value V, VectorType DstVTy, const DataLayout &DL)
	Returns a bitcasted value to the requested vector type. More...

void	emitIterationCountCheck (BasicBlock *Bypass)
	Emit a bypass check to see if the vector trip count is zero, including if it overflows. More...

BasicBlock *	emitSCEVChecks (BasicBlock *Bypass)
	Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct. More...

BasicBlock *	emitMemRuntimeChecks (BasicBlock *Bypass)
	Emit bypass checks to check any memory assumptions we may have made. More...

void	createVectorLoopSkeleton (StringRef Prefix)
	Emit basic blocks (prefixed with `Prefix`) for the iteration check, vector loop preheader, middle block and scalar preheader. More...

void	createInductionResumeValues (std::pair< BasicBlock , Value > AdditionalBypass={nullptr, nullptr})
	Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off. More...

BasicBlock *	completeLoopSkeleton (MDNode *OrigLoopID)
	Complete the loop skeleton by adding debug MDs, creating appropriate conditional branches in the middle block, preparing the builder and running the verifier. More...

void	addNewMetadata (Instruction To, const Instruction Orig)
	Add additional metadata to `To` that was not present on `Orig`. More...

void	collectPoisonGeneratingRecipes (VPTransformState &State)
	Collect poison-generating recipes that may generate a poison value that is used after vectorization, even when their operands are not poison. More...

virtual void	printDebugTracesAtStart ()
	Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested. More...

virtual void	printDebugTracesAtEnd ()

Protected Attributes
Loop *	OrigLoop
	The original loop. More...

PredicatedScalarEvolution &	PSE
	A wrapper around ScalarEvolution used to add runtime SCEV checks. More...

LoopInfo *	LI
	Loop Info. More...

DominatorTree *	DT
	Dominator Tree. More...

AAResults *	AA
	Alias Analysis. More...

const TargetLibraryInfo *	TLI
	Target Library Info. More...

const TargetTransformInfo *	TTI
	Target Transform Info. More...

AssumptionCache *	AC
	Assumption Cache. More...

OptimizationRemarkEmitter *	ORE
	Interface to emit optimization remarks. More...

std::unique_ptr< LoopVersioning >	LVer
	LoopVersioning. More...

ElementCount	VF
	The vectorization SIMD factor to use. More...

unsigned	UF
	The vectorization unroll factor to use. More...

IRBuilder	Builder
	The builder that we use. More...

BasicBlock *	LoopVectorPreHeader
	The vector-loop preheader. More...

BasicBlock *	LoopScalarPreHeader
	The scalar-loop preheader. More...

BasicBlock *	LoopMiddleBlock
	Middle Block between the vector and the scalar. More...

BasicBlock *	LoopExitBlock
	The unique ExitBlock of the scalar loop if one exists. More...

BasicBlock *	LoopScalarBody
	The scalar loop body. More...

SmallVector< BasicBlock *, 4 >	LoopBypassBlocks
	A list of all bypass blocks. The first block is the entry of the loop. More...

SmallVector< Instruction *, 4 >	PredicatedInstructions
	Store instructions that were predicated. More...

Value *	TripCount = nullptr
	Trip count of the original loop. More...

Value *	VectorTripCount = nullptr
	Trip count of the widened loop (TripCount - TripCount % (VF*UF)) More...

LoopVectorizationLegality *	Legal
	The legality analysis. More...

LoopVectorizationCostModel *	Cost
	The profitablity analysis. More...

bool	AddedSafetyChecks = false

DenseMap< PHINode , Value >	IVEndValues

BlockFrequencyInfo *	BFI
	BFI and PSI are used to check for profile guided size optimizations. More...

ProfileSummaryInfo *	PSI

bool	OptForSizeBasedOnProfile

GeneratedRTChecks &	RTChecks
	Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable. More...

SmallMapVector< const RecurrenceDescriptor , PHINode , 4 >	ReductionResumeValues

Friends
class	LoopVectorizationPlanner

Detailed Description

InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF).

This class performs the widening of scalars into vectors, or multiple scalars. This class also implements the following features:

It inserts an epilogue loop for handling loops that don't have iteration counts that are known to be a multiple of the vectorization factor.
It handles the code generation for reduction variables.
Scalarization (implementation using scalars) of un-vectorizable instructions. InnerLoopVectorizer does not perform any vectorization-legality checks, and relies on the caller to check for the different legality aspects. The InnerLoopVectorizer relies on the LoopVectorizationLegality class to provide information about the induction and reduction variables that were found to a given vectorization factor.

Definition at line 438 of file LoopVectorize.cpp.

Member Typedef Documentation

◆ PhiVector

using llvm::InnerLoopVectorizer::PhiVector = SmallVector<PHINode *, 4>

protected

A small list of PHINodes.

Definition at line 545 of file LoopVectorize.cpp.

◆ ScalarParts

using llvm::InnerLoopVectorizer::ScalarParts = SmallVector<SmallVector<Value *, 4>, 2>

protected

A type for scalarized values in the new loop.

Each value from the original loop, when scalarized, is represented by UF x VF scalar values in the new unrolled loop, where UF is the unroll factor and VF is the vectorization factor.

Definition at line 551 of file LoopVectorize.cpp.

◆ VectorParts

using llvm::InnerLoopVectorizer::VectorParts = SmallVector<Value *, 2>

A type for vectorized values in the new loop.

Each value from the original loop, when vectorized, is represented by UF vector values in the new unrolled loop, where UF is the unroll factor.

Definition at line 483 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ InnerLoopVectorizer()

llvm::InnerLoopVectorizer::InnerLoopVectorizer	(	Loop *	OrigLoop,
		PredicatedScalarEvolution &	PSE,
		LoopInfo *	LI,
		DominatorTree *	DT,
		const TargetLibraryInfo *	TLI,
		const TargetTransformInfo *	TTI,
		AssumptionCache *	AC,
		OptimizationRemarkEmitter *	ORE,
		ElementCount	VecWidth,
		unsigned	UnrollFactor,
		LoopVectorizationLegality *	LVL,
		LoopVectorizationCostModel *	CM,
		BlockFrequencyInfo *	BFI,
		ProfileSummaryInfo *	PSI,
		GeneratedRTChecks &	RTChecks
	)

inline

Definition at line 440 of file LoopVectorize.cpp.

References BFI, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::IRPass, OptForSizeBasedOnProfile, OrigLoop, PSI, and llvm::shouldOptimizeForSize().

◆ ~InnerLoopVectorizer()

virtual llvm::InnerLoopVectorizer::~InnerLoopVectorizer ( )

virtualdefault

Member Function Documentation

◆ addMetadata() [1/2]

void InnerLoopVectorizer::addMetadata	(	ArrayRef< Value * >	To,
		Instruction *	From
	)

Similar to the previous function but it adds the metadata to a vector of instructions.

Definition at line 1143 of file LoopVectorize.cpp.

References addMetadata(), From, and I.

◆ addMetadata() [2/2]

void InnerLoopVectorizer::addMetadata	(	Instruction *	To,
		Instruction *	From
	)

Add metadata from one instruction to another.

This includes both the original MDs from From and additional ones (

See also: addNewMetadata). Use this for newly created instructions in the vector loop.

Definition at line 1137 of file LoopVectorize.cpp.

References addNewMetadata(), From, and llvm::propagateMetadata().

Referenced by addMetadata(), llvm::VPWidenRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenMemoryInstructionRecipe::execute(), and widenCallInstruction().

◆ addNewMetadata()

void InnerLoopVectorizer::addNewMetadata	(	Instruction *	To,
		const Instruction *	Orig
	)

protected

Add additional metadata to To that was not present on Orig.

Currently this is used to add the noalias annotations based on the inserted memchecks. Use this for instructions that are cloned into the vector loop.

Definition at line 1049 of file LoopVectorize.cpp.

References LVer.

Referenced by addMetadata(), and scalarizeInstruction().

◆ areSafetyChecksAdded()

bool llvm::InnerLoopVectorizer::areSafetyChecksAdded ( )

inline

Definition at line 478 of file LoopVectorize.cpp.

References AddedSafetyChecks.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ clearReductionWrapFlags()

void InnerLoopVectorizer::clearReductionWrapFlags	(	VPReductionPHIRecipe *	PhiR,
		VPTransformState &	State
	)

protected

Clear NSW/NUW flags from reduction instructions if necessary.

Definition at line 4051 of file LoopVectorize.cpp.

References llvm::Add, llvm::VPTransformState::get(), llvm::VPReductionPHIRecipe::getRecurrenceDescriptor(), llvm::RecurrenceDescriptor::getRecurrenceKind(), llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::Mul, llvm::SmallVectorImpl< T >::pop_back_val(), UF, and llvm::VPValue::users().

Referenced by fixReduction().

◆ collectPoisonGeneratingRecipes()

void InnerLoopVectorizer::collectPoisonGeneratingRecipes ( VPTransformState & State )

protected

Collect poison-generating recipes that may generate a poison value that is used after vectorization, even when their operands are not poison.

Those recipes meet the following conditions:

Contribute to the address computation of a recipe generating a widen memory load/store (VPWidenMemoryInstructionRecipe or VPInterleaveRecipe).
Such a widen memory load/store has at least one underlying Instruction that is in a basic block that needs predication and after vectorization the generated instruction won't be predicated.

Definition at line 1057 of file LoopVectorize.cpp.

References llvm::LoopVectorizationLegality::blockNeedsPredication(), llvm::depth_first(), llvm::VPlan::getEntry(), llvm::InterleaveGroup< InstTy >::getMember(), llvm::InterleaveGroup< InstTy >::getNumMembers(), llvm::Instruction::getParent(), I, Legal, and llvm::VPTransformState::Plan.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ completeLoopSkeleton()

BasicBlock * InnerLoopVectorizer::completeLoopSkeleton ( MDNode * OrigLoopID )

protected

Complete the loop skeleton by adding debug MDs, creating appropriate conditional branches in the middle block, preparing the builder and running the verifier.

Return the preheader of the completed vector loop.

Definition at line 3195 of file LoopVectorize.cpp.

References assert(), Cost, llvm::CmpInst::Create(), DT, llvm::Sched::Fast, llvm::LoopVectorizationCostModel::foldTailByMasking(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), getOrCreateTripCount(), getOrCreateVectorTripCount(), llvm::BasicBlock::getTerminator(), llvm::CmpInst::ICMP_EQ, LoopMiddleBlock, LoopVectorPreHeader, OrigLoop, llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::Instruction::setDebugLoc(), VectorTripCount, llvm::DominatorTreeBase< NodeT, IsPostDom >::verify(), and VF.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().

◆ createBitOrPointerCast()

Value * InnerLoopVectorizer::createBitOrPointerCast	(	Value *	V,
		VectorType *	DstVTy,
		const DataLayout &	DL
	)

protected

Returns a bitcasted value to the requested vector type.

Also handles bitcasts of vector<float> <-> vector<pointer> types.

Definition at line 2909 of file LoopVectorize.cpp.

References assert(), Builder, llvm::IRBuilderBase::CreateBitOrPointerCast(), DL, llvm::FixedVectorType::get(), llvm::Value::getContext(), llvm::Type::getIntNTy(), llvm::Value::getType(), llvm::CastInst::isBitOrNoopPointerCastable(), llvm::Type::isFloatingPointTy(), llvm::Type::isPointerTy(), and VF.

Referenced by vectorizeInterleaveGroup().

◆ createInductionResumeValues()

void InnerLoopVectorizer::createInductionResumeValues ( std::pair< BasicBlock *, Value * > AdditionalBypass = {nullptr, nullptr} )

protected

Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off.

In cases where the loop skeleton is more complicated (eg. epilogue vectorization) and the resume values can come from an additional bypass block, the AdditionalBypass pair provides information about the bypass block and the end value on the edge from bypass to this loop.

Definition at line 3114 of file LoopVectorize.cpp.

References llvm::PHINode::addIncoming(), assert(), B, BB, llvm::PHINode::Create(), CreateStepValue(), emitTransformedIndex(), llvm::CastInst::getCastOpcode(), llvm::Instruction::getDebugLoc(), llvm::Instruction::getFastMathFlags(), llvm::InductionDescriptor::getInductionBinOp(), llvm::LoopVectorizationLegality::getInductionVars(), getOrCreateVectorTripCount(), llvm::LoopVectorizationLegality::getPrimaryInduction(), llvm::PredicatedScalarEvolution::getSE(), llvm::InductionDescriptor::getStartValue(), llvm::InductionDescriptor::getStep(), llvm::BasicBlock::getTerminator(), llvm::SCEV::getType(), llvm::Value::getType(), IVEndValues, Legal, LoopBypassBlocks, LoopMiddleBlock, LoopScalarPreHeader, LoopVectorPreHeader, PSE, llvm::Instruction::setDebugLoc(), llvm::PHINode::setIncomingValueForBlock(), llvm::Value::setName(), and VectorTripCount.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().

◆ createVectorizedLoopSkeleton()

std::pair< BasicBlock *, Value * > InnerLoopVectorizer::createVectorizedLoopSkeleton ( )

virtual

Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.

Control flow is generated around the vectorized (and scalar epilogue) loops consisting of various checks and bypasses. Return the pre-header block of the new loop and the start value for the canonical induction, if it is != 0. The latter is the case when vectorizing the epilogue loop. In the case of epilogue vectorization, this function is overriden to handle the more complex control flow around the loops.

Reimplemented in llvm::InnerLoopAndEpilogueVectorizer.

Definition at line 3231 of file LoopVectorize.cpp.

References completeLoopSkeleton(), createInductionResumeValues(), createVectorLoopSkeleton(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), emitSCEVChecks(), llvm::Loop::getLoopID(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), getOrCreateTripCount(), LoopScalarPreHeader, and OrigLoop.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ createVectorLoopSkeleton()

void InnerLoopVectorizer::createVectorLoopSkeleton ( StringRef Prefix )

protected

Emit basic blocks (prefixed with Prefix) for the iteration check, vector loop preheader, middle block and scalar preheader.

Definition at line 3072 of file LoopVectorize.cpp.

References assert(), Builder, llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), Cost, llvm::BranchInst::Create(), DT, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::BasicBlock::getTerminator(), llvm::IRBuilderBase::getTrue(), llvm::LoopBase< BlockT, LoopT >::getUniqueExitBlock(), LI, LoopExitBlock, LoopMiddleBlock, LoopScalarBody, LoopScalarPreHeader, LoopVectorPreHeader, OrigLoop, llvm::cl::Prefix, llvm::ReplaceInstWithInst(), llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::Instruction::setDebugLoc(), llvm::SplitBlock(), and VF.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().

◆ emitIterationCountCheck()

void InnerLoopVectorizer::emitIterationCountCheck ( BasicBlock * Bypass )

protected

Emit a bypass check to see if the vector trip count is zero, including if it overflows.

Definition at line 2940 of file LoopVectorize.cpp.

References assert(), Builder, llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), Cost, llvm::BranchInst::Create(), llvm::IRBuilderBase::CreateICmp(), llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), DT, llvm::LoopVectorizationCostModel::foldTailByMasking(), llvm::ConstantInt::get(), llvm::IRBuilderBase::getFalse(), llvm::DomTreeNodeBase< NodeT >::getIDom(), llvm::DominatorTreeBase< NodeT, IsPostDom >::getNode(), getOrCreateTripCount(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::CmpInst::ICMP_ULE, llvm::CmpInst::ICMP_ULT, llvm::LinearPolySize< LeafTy >::isScalable(), LHS, LI, LoopBypassBlocks, LoopExitBlock, LoopVectorPreHeader, P, llvm::DominatorTreeBase< NodeT, IsPostDom >::properlyDominates(), llvm::ReplaceInstWithInst(), llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::SplitBlock(), UF, and VF.

◆ emitMemRuntimeChecks()

BasicBlock * InnerLoopVectorizer::emitMemRuntimeChecks ( BasicBlock * Bypass )

protected

Emit bypass checks to check any memory assumptions we may have made.

Returns the block containing the checks or nullptr if no checks have been added.

Definition at line 3025 of file LoopVectorize.cpp.

References AddedSafetyChecks, assert(), Cost, DEBUG_TYPE, DT, llvm::OptimizationRemarkEmitter::emit(), GeneratedRTChecks::emitMemRuntimeChecks(), EnableVPlanNativePath, llvm::LoopVectorizeHints::FK_Enabled, llvm::RuntimePointerChecking::getChecks(), llvm::RuntimePointerChecking::getDiffChecks(), llvm::LoopVectorizeHints::getForce(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::LoopVectorizationLegality::getLAI(), llvm::BasicBlock::getParent(), llvm::LoopAccessInfo::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getSE(), llvm::Loop::getStartLoc(), llvm::Function::hasOptSize(), llvm::LoopVectorizationCostModel::Hints, Legal, LI, LoopBypassBlocks, LoopVectorPreHeader, LVer, OptForSizeBasedOnProfile, ORE, OrigLoop, PSE, and RTChecks.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().

◆ emitSCEVChecks()

BasicBlock * InnerLoopVectorizer::emitSCEVChecks ( BasicBlock * Bypass )

protected

Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct.

Returns the block containing the checks or nullptr if no checks have been added.

Definition at line 2997 of file LoopVectorize.cpp.

References AddedSafetyChecks, assert(), llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), Cost, DT, GeneratedRTChecks::emitSCEVChecks(), llvm::LoopVectorizeHints::FK_Enabled, llvm::LoopVectorizeHints::getForce(), llvm::BasicBlock::getParent(), llvm::Function::hasOptSize(), llvm::LoopVectorizationCostModel::Hints, LoopBypassBlocks, LoopExitBlock, LoopVectorPreHeader, OptForSizeBasedOnProfile, llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), RTChecks, and VF.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().

◆ fixCrossIterationPHIs()

void InnerLoopVectorizer::fixCrossIterationPHIs ( VPTransformState & State )

protected

Handle all cross-iteration phis in the header.

Definition at line 3722 of file LoopVectorize.cpp.

References fixFirstOrderRecurrence(), fixReduction(), llvm::X86ISD::FOR, llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPlan::getVectorLoopRegion(), llvm::VPBasicBlock::phis(), and llvm::VPTransformState::Plan.

Referenced by fixVectorizedLoop().

◆ fixFirstOrderRecurrence()

void InnerLoopVectorizer::fixFirstOrderRecurrence	(	VPFirstOrderRecurrencePHIRecipe *	PhiR,
		VPTransformState &	State
	)

protected

Create the exit value of first order recurrences in the middle block and update their users.

Definition at line 3739 of file LoopVectorize.cpp.

References BB, llvm::BasicBlock::begin(), Builder, Cost, llvm::IRBuilderBase::CreateExtractElement(), llvm::IRBuilderBase::CreatePHI(), llvm::IRBuilderBase::CreateSub(), llvm::ConstantInt::get(), llvm::VPTransformState::get(), llvm::VPHeaderPHIRecipe::getBackedgeValue(), llvm::IRBuilderBase::getInt32Ty(), llvm::VPValue::getLiveInIRValue(), llvm::getRuntimeVF(), llvm::VPHeaderPHIRecipe::getStartValue(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::VPValue::getUnderlyingValue(), llvm::is_contained(), llvm::ElementCount::isVector(), LoopExitBlock, LoopMiddleBlock, LoopScalarPreHeader, llvm::BasicBlock::phis(), llvm::VPTransformState::Plan, llvm::predecessors(), llvm::VPlan::removeLiveOut(), llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::PHINode::setIncomingValueForBlock(), llvm::IRBuilderBase::SetInsertPoint(), llvm::Value::setName(), UF, and VF.

Referenced by fixCrossIterationPHIs().

◆ fixNonInductionPHIs()

void InnerLoopVectorizer::fixNonInductionPHIs	(	VPlan &	Plan,
		VPTransformState &	State
	)

Fix the non-induction PHIs in Plan.

Definition at line 4157 of file LoopVectorize.cpp.

References llvm::PHINode::addIncoming(), Builder, llvm::VPTransformState::CFG, llvm::depth_first(), llvm::VPTransformState::get(), llvm::VPlan::getEntry(), llvm::VPWidenPHIRecipe::getIncomingBlock(), llvm::VPWidenPHIRecipe::getIncomingValue(), llvm::VPUser::getNumOperands(), i, P, llvm::IRBuilderBase::SetInsertPoint(), and llvm::VPTransformState::CFGState::VPBB2IRBB.

Referenced by fixVectorizedLoop().

◆ fixReduction()

void InnerLoopVectorizer::fixReduction	(	VPReductionPHIRecipe *	Phi,
		VPTransformState &	State
	)

protected

Create code for the loop exit value of the reduction.

Definition at line 3852 of file LoopVectorize.cpp.

Referenced by fixCrossIterationPHIs().

◆ fixupIVUsers()

void InnerLoopVectorizer::fixupIVUsers	(	PHINode *	OrigPhi,
		const InductionDescriptor &	II,
		Value *	VectorTripCount,
		Value *	EndValue,
		BasicBlock *	MiddleBlock,
		BasicBlock *	VectorHeader,
		VPlan &	Plan
	)

protected

Set up the values of the IVs correctly when exiting the vector loop.

Definition at line 3306 of file LoopVectorize.cpp.

References llvm::PHINode::addIncoming(), assert(), B, llvm::LoopBase< BlockT, LoopT >::contains(), CreateStepValue(), emitTransformedIndex(), llvm::ConstantInt::get(), llvm::PHINode::getBasicBlockIndex(), llvm::Instruction::getFastMathFlags(), llvm::PHINode::getIncomingValueForBlock(), llvm::InductionDescriptor::getInductionBinOp(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::PredicatedScalarEvolution::getSE(), llvm::InductionDescriptor::getStartValue(), llvm::InductionDescriptor::getStep(), llvm::BasicBlock::getTerminator(), llvm::SCEV::getType(), llvm::Value::getType(), llvm::LoopBase< BlockT, LoopT >::getUniqueExitBlock(), I, llvm::Type::isIntegerTy(), OrigLoop, PostInc, PSE, llvm::VPlan::removeLiveOut(), llvm::Value::setName(), llvm::Value::users(), and VectorTripCount.

◆ fixVectorizedLoop()

void InnerLoopVectorizer::fixVectorizedLoop	(	VPTransformState &	State,
		VPlan &	Plan
	)

Fix the vectorized code, taking care of header phi's, live-outs, and more.

Definition at line 3652 of file LoopVectorize.cpp.

References llvm::VPTransformState::CFG, llvm::VPlan::clearLiveOuts(), Cost, EnableVPlanNativePath, fixCrossIterationPHIs(), fixNonInductionPHIs(), llvm::ScalarEvolution::forgetLoop(), llvm::VPBlockBase::getExitingBasicBlock(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::PredicatedScalarEvolution::getSE(), llvm::VPlan::getVectorLoopRegion(), llvm::ElementCount::isVector(), LI, OrigLoop, PSE, llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), truncateToMinimalBitwidths(), VF, and llvm::VPTransformState::CFGState::VPBB2IRBB.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ getBroadcastInstrs()

Value * InnerLoopVectorizer::getBroadcastInstrs ( Value * V )

virtual

Create a broadcast instruction.

This method generates a broadcast instruction (shuffle) for loop invariant values and for the induction value. If this is the induction variable then we extend it to N, N+1, ... this is needed because each iteration in the loop corresponds to a SIMD element.

Definition at line 2258 of file LoopVectorize.cpp.

References Builder, llvm::IRBuilderBase::CreateVectorSplat(), llvm::DominatorTree::dominates(), DT, llvm::Instruction::getParent(), llvm::BasicBlock::getTerminator(), llvm::Loop::isLoopInvariant(), LoopVectorPreHeader, OrigLoop, llvm::IRBuilderBase::SetInsertPoint(), and VF.

◆ getOrCreateTripCount()

Value * InnerLoopVectorizer::getOrCreateTripCount ( BasicBlock * InsertBlock )

protected

Returns (and creates if needed) the original loop trip count.

Definition at line 2811 of file LoopVectorize.cpp.

References assert(), Builder, llvm::CastInst::CreatePointerCast(), DL, llvm::ScalarEvolution::getAddExpr(), llvm::PredicatedScalarEvolution::getBackedgeTakenCount(), llvm::Module::getDataLayout(), llvm::BasicBlock::getModule(), llvm::ScalarEvolution::getNoopOrZeroExtend(), llvm::ScalarEvolution::getOne(), llvm::Type::getPrimitiveSizeInBits(), llvm::PredicatedScalarEvolution::getSE(), llvm::BasicBlock::getTerminator(), llvm::ScalarEvolution::getTruncateOrNoop(), llvm::SCEV::getType(), llvm::Value::getType(), llvm::ScalarEvolution::getTypeSizeInBits(), llvm::LoopVectorizationLegality::getWidestInductionType(), llvm::Type::isPointerTy(), Legal, PSE, and TripCount.

Referenced by completeLoopSkeleton(), llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), createVectorizedLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::LoopVectorizationPlanner::executePlan(), and getOrCreateVectorTripCount().

◆ getOrCreateVectorTripCount()

Value * InnerLoopVectorizer::getOrCreateVectorTripCount ( BasicBlock * InsertBlock )

protected

Returns (and creates if needed) the trip count of the widened loop.

Definition at line 2859 of file LoopVectorize.cpp.

References assert(), Builder, Cost, llvm::IRBuilderBase::CreateAdd(), llvm::IRBuilderBase::CreateICmpEQ(), llvm::IRBuilderBase::CreateSelect(), llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::IRBuilderBase::CreateURem(), llvm::LoopVectorizationCostModel::foldTailByMasking(), llvm::ConstantInt::get(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), getOrCreateTripCount(), llvm::getRuntimeVF(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::isPowerOf2_32(), llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), UF, VectorTripCount, and VF.

Referenced by completeLoopSkeleton(), llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), createInductionResumeValues(), and llvm::LoopVectorizationPlanner::executePlan().

◆ getReductionResumeValue()

PHINode * InnerLoopVectorizer::getReductionResumeValue ( const RecurrenceDescriptor & RdxDesc )

Definition at line 1151 of file LoopVectorize.cpp.

References assert(), and ReductionResumeValues.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ packScalarIntoVectorValue()

void InnerLoopVectorizer::packScalarIntoVectorValue	(	VPValue *	Def,
		const VPIteration &	Instance,
		VPTransformState &	State
	)

Construct the vector value of a scalarized value V one lane at a time.

Definition at line 2503 of file LoopVectorize.cpp.

References llvm::VPTransformState::Builder, Builder, llvm::IRBuilderBase::CreateInsertElement(), llvm::tgtok::Def, llvm::VPTransformState::get(), llvm::VPLane::getAsRuntimeExpr(), llvm::VPIteration::Lane, llvm::VPIteration::Part, llvm::VPTransformState::set(), and VF.

Referenced by llvm::VPReplicateRecipe::execute().

◆ printDebugTracesAtEnd()

virtual void llvm::InnerLoopVectorizer::printDebugTracesAtEnd ( )

inlineprotectedvirtual

Reimplemented in llvm::EpilogueVectorizerEpilogueLoop, and llvm::EpilogueVectorizerMainLoop.

Definition at line 646 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ printDebugTracesAtStart()

virtual void llvm::InnerLoopVectorizer::printDebugTracesAtStart ( )

inlineprotectedvirtual

Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.

Reimplemented in llvm::EpilogueVectorizerEpilogueLoop, and llvm::EpilogueVectorizerMainLoop.

Definition at line 645 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ scalarizeInstruction()

void InnerLoopVectorizer::scalarizeInstruction	(	Instruction *	Instr,
		VPReplicateRecipe *	RepRecipe,
		const VPIteration &	Instance,
		bool	IfPredicateInstr,
		VPTransformState &	State
	)

A helper function to scalarize a single Instruction in the innermost loop.

Generates a sequence of scalar instances for each lane between MinLane and MaxLane, times each part between MinPart and MaxPart, inclusive. Uses the VPValue operands from RepRecipe instead of Instr's operands.

Definition at line 2753 of file LoopVectorize.cpp.

References AC, addNewMetadata(), assert(), llvm::VPTransformState::Builder, llvm::Instruction::clone(), llvm::Instruction::dropPoisonGeneratingFlags(), llvm::enumerate(), llvm::VPTransformState::get(), llvm::Instruction::getDebugLoc(), llvm::VPLane::getFirstLane(), llvm::Value::getName(), llvm::Value::getType(), I, llvm::IRBuilderBase::Insert(), llvm::Type::isAggregateType(), llvm::VPIteration::isFirstIteration(), llvm::VPReplicateRecipe::isUniform(), llvm::Type::isVoidTy(), llvm::VPTransformState::MayGeneratePoisonRecipes, llvm::VPUser::operands(), PredicatedInstructions, llvm::AssumptionCache::registerAssumption(), llvm::VPTransformState::set(), setDebugLocFromInst(), llvm::Value::setName(), and llvm::User::setOperand().

Referenced by llvm::VPReplicateRecipe::execute().

◆ setDebugLocFromInst()

void InnerLoopVectorizer::setDebugLocFromInst ( const Value * V )

Set the debug location in the builder Ptr using the debug location in V.

If Ptr is None then it uses the class member's Builder.

Definition at line 923 of file LoopVectorize.cpp.

References Builder, llvm::DILocation::cloneByMultiplyingDuplicationFactor(), llvm::dbgs(), llvm::EnableFSDiscriminator, llvm::LinearPolySize< LeafTy >::getKnownMinValue(), LLVM_DEBUG, llvm::IRBuilderBase::SetCurrentDebugLocation(), UF, and VF.

Referenced by llvm::VPWidenRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPWidenMemoryInstructionRecipe::execute(), fixReduction(), scalarizeInstruction(), vectorizeInterleaveGroup(), and widenCallInstruction().

◆ sinkScalarOperands()

void InnerLoopVectorizer::sinkScalarOperands ( Instruction * PredInst )

protected

Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it.

Definition at line 4083 of file LoopVectorize.cpp.

References llvm::all_of(), BB, llvm::SmallVectorImpl< T >::clear(), llvm::SetVector< T, Vector, Set >::empty(), llvm::PHINode::getIncomingValueNumForOperand(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::Instruction::getParent(), I, llvm::SetVector< T, Vector, Set >::insert(), LI, llvm::User::op_begin(), llvm::User::op_end(), and llvm::SetVector< T, Vector, Set >::pop_back_val().

◆ truncateToMinimalBitwidths()

void InnerLoopVectorizer::truncateToMinimalBitwidths ( VPTransformState & State )

protected

Shrinks vector element sizes to the smallest bitwidth they can be legally represented as.

Definition at line 3519 of file LoopVectorize.cpp.

References B, Cost, llvm::SmallPtrSetImpl< PtrType >::count(), llvm::tgtok::Def, llvm::Instruction::eraseFromParent(), llvm::IntegerType::get(), llvm::VPTransformState::get(), llvm::VectorType::get(), llvm::Type::getContext(), llvm::LoopVectorizationCostModel::getMinimalBitwidths(), llvm::User::getOperand(), llvm::VPlan::getVPValue(), llvm::VPTransformState::hasAnyVectorValue(), I, llvm::ARM_PROC::IE, llvm::SmallPtrSetImpl< PtrType >::insert(), llvm_unreachable, llvm::VPTransformState::Plan, llvm::VPTransformState::reset(), SI, smallestIntegerVectorType(), llvm::Value::takeName(), UF, and llvm::Value::use_empty().

Referenced by fixVectorizedLoop().

◆ useOrderedReductions()

bool InnerLoopVectorizer::useOrderedReductions ( const RecurrenceDescriptor & RdxDesc )

Returns true if the reordering of FP operations is not allowed, but we are able to vectorize with strict in-order reductions for the given RdxDesc.

Definition at line 4178 of file LoopVectorize.cpp.

References Cost, and llvm::LoopVectorizationCostModel::useOrderedReductions().

Referenced by llvm::LoopVectorizationCostModel::collectElementTypesForWidening(), llvm::LoopVectorizationCostModel::collectInLoopReductions(), and llvm::VPReductionRecipe::execute().

◆ vectorizeInterleaveGroup()

void InnerLoopVectorizer::vectorizeInterleaveGroup	(	const InterleaveGroup< Instruction > *	Group,
		ArrayRef< VPValue * >	VPDefs,
		VPTransformState &	State,
		VPValue *	Addr,
		ArrayRef< VPValue * >	StoredValues,
		VPValue *	BlockInMask = `nullptr`
	)

Try to vectorize interleaved access group Group with the base address given in Addr, optionally masking the vector operations if BlockInMask is non-null.

Use State to translate given VPValues to IR values in the vectorized loop.

Definition at line 2553 of file LoopVectorize.cpp.

Referenced by llvm::VPInterleaveRecipe::execute().

◆ widenCallInstruction()

void InnerLoopVectorizer::widenCallInstruction	(	CallInst &	I,
		VPValue *	Def,
		VPUser &	ArgOperands,
		VPTransformState &	State
	)

Widen a single call instruction within the innermost loop.

Definition at line 4201 of file LoopVectorize.cpp.

References addMetadata(), Arg, llvm::AMDGPU::HSAMD::Kernel::Key::Args, assert(), Builder, llvm::Instruction::copyFastMathFlags(), Cost, llvm::IRBuilderBase::CreateCall(), llvm::tgtok::Def, llvm::enumerate(), llvm::VFShape::get(), llvm::VPTransformState::get(), llvm::VectorType::get(), llvm::Intrinsic::getDeclaration(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::LoopVectorizationCostModel::getVectorCallCost(), llvm::LoopVectorizationCostModel::getVectorIntrinsicCost(), llvm::getVectorIntrinsicIDForCall(), llvm::VFDatabase::getVectorizedFunction(), I, llvm::InstructionCost::isValid(), llvm::ElementCount::isVector(), llvm::isVectorIntrinsicWithOverloadTypeAtArg(), llvm::isVectorIntrinsicWithScalarOpAtArg(), M, llvm::VPUser::operands(), llvm::VPTransformState::set(), setDebugLocFromInst(), TLI, llvm::ToVectorTy(), UF, and VF.

Referenced by llvm::VPWidenCallRecipe::execute().

Friends And Related Function Documentation

◆ LoopVectorizationPlanner

friend class LoopVectorizationPlanner

friend

Definition at line 542 of file LoopVectorize.cpp.

Member Data Documentation

◆ AA

AAResults* llvm::InnerLoopVectorizer::AA

protected

Alias Analysis.

Definition at line 663 of file LoopVectorize.cpp.

◆ AC

AssumptionCache* llvm::InnerLoopVectorizer::AC

protected

Assumption Cache.

Definition at line 672 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationCostModel::collectValuesToIgnore(), getScalarEpilogueLowering(), processLoopInVPlanNativePath(), and scalarizeInstruction().

◆ AddedSafetyChecks

bool llvm::InnerLoopVectorizer::AddedSafetyChecks = false

protected

Definition at line 732 of file LoopVectorize.cpp.

Referenced by areSafetyChecksAdded(), emitMemRuntimeChecks(), and emitSCEVChecks().

◆ BFI

BlockFrequencyInfo* llvm::InnerLoopVectorizer::BFI

protected

BFI and PSI are used to check for profile guided size optimizations.

Definition at line 739 of file LoopVectorize.cpp.

Referenced by getScalarEpilogueLowering(), InnerLoopVectorizer(), and processLoopInVPlanNativePath().

◆ Builder

IRBuilder llvm::InnerLoopVectorizer::Builder

protected

The builder that we use.

Definition at line 693 of file LoopVectorize.cpp.

Referenced by buildScalarSteps(), createBitOrPointerCast(), llvm::VPRecipeBuilder::createBlockInMask(), llvm::VPRecipeBuilder::createEdgeMask(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), llvm::VPWidenRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenMemoryInstructionRecipe::execute(), llvm::LoopVectorizationPlanner::executePlan(), fixFirstOrderRecurrence(), fixNonInductionPHIs(), fixReduction(), llvm::VPTransformState::get(), getBroadcastInstrs(), getOrCreateTripCount(), getOrCreateVectorTripCount(), getStepVector(), packScalarIntoVectorValue(), setDebugLocFromInst(), vectorizeInterleaveGroup(), and widenCallInstruction().

◆ Cost

LoopVectorizationCostModel* llvm::InnerLoopVectorizer::Cost

protected

◆ DT

DominatorTree* llvm::InnerLoopVectorizer::DT

protected

Dominator Tree.

Definition at line 660 of file LoopVectorize.cpp.

Referenced by completeLoopSkeleton(), GeneratedRTChecks::Create(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), GeneratedRTChecks::emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), emitSCEVChecks(), GeneratedRTChecks::emitSCEVChecks(), llvm::LoopVectorizationPlanner::executePlan(), getBroadcastInstrs(), getScalarEpilogueLowering(), and processLoopInVPlanNativePath().

◆ IVEndValues

DenseMap<PHINode *, Value *> llvm::InnerLoopVectorizer::IVEndValues

protected

Definition at line 736 of file LoopVectorize.cpp.

Referenced by createInductionResumeValues().

◆ Legal

LoopVectorizationLegality* llvm::InnerLoopVectorizer::Legal

protected

The legality analysis.

Definition at line 726 of file LoopVectorize.cpp.

◆ LI

LoopInfo* llvm::InnerLoopVectorizer::LI

protected

◆ LoopBypassBlocks

SmallVector<BasicBlock *, 4> llvm::InnerLoopVectorizer::LoopBypassBlocks

protected

A list of all bypass blocks. The first block is the entry of the loop.

Definition at line 714 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createInductionResumeValues(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), and emitSCEVChecks().

◆ LoopExitBlock

BasicBlock* llvm::InnerLoopVectorizer::LoopExitBlock

protected

The unique ExitBlock of the scalar loop if one exists.

Note that there can be multiple exiting edges reaching this block.

Definition at line 708 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitSCEVChecks(), fixFirstOrderRecurrence(), and fixReduction().

◆ LoopMiddleBlock

BasicBlock* llvm::InnerLoopVectorizer::LoopMiddleBlock

protected

Middle Block between the vector and the scalar.

Definition at line 704 of file LoopVectorize.cpp.

Referenced by completeLoopSkeleton(), createInductionResumeValues(), createVectorLoopSkeleton(), fixFirstOrderRecurrence(), and fixReduction().

◆ LoopScalarBody

BasicBlock* llvm::InnerLoopVectorizer::LoopScalarBody

protected

The scalar loop body.

Definition at line 711 of file LoopVectorize.cpp.

Referenced by createVectorLoopSkeleton().

◆ LoopScalarPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopScalarPreHeader

protected

The scalar-loop preheader.

Definition at line 701 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createInductionResumeValues(), createVectorizedLoopSkeleton(), createVectorLoopSkeleton(), fixFirstOrderRecurrence(), and fixReduction().

◆ LoopVectorPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopVectorPreHeader

protected

The vector-loop preheader.

Definition at line 698 of file LoopVectorize.cpp.

Referenced by completeLoopSkeleton(), llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createInductionResumeValues(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), GeneratedRTChecks::emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), emitSCEVChecks(), GeneratedRTChecks::emitSCEVChecks(), and getBroadcastInstrs().

◆ LVer

std::unique_ptr<LoopVersioning> llvm::InnerLoopVectorizer::LVer

protected

LoopVersioning.

It's only set up (non-null) if memchecks were used.

This is currently only used to add no-alias metadata based on the memchecks. The actually versioning is performed manually.

Definition at line 682 of file LoopVectorize.cpp.

Referenced by addNewMetadata(), and emitMemRuntimeChecks().

◆ OptForSizeBasedOnProfile

bool llvm::InnerLoopVectorizer::OptForSizeBasedOnProfile

protected

Definition at line 744 of file LoopVectorize.cpp.

Referenced by emitMemRuntimeChecks(), emitSCEVChecks(), and InnerLoopVectorizer().

◆ ORE

OptimizationRemarkEmitter* llvm::InnerLoopVectorizer::ORE

protected

Interface to emit optimization remarks.

Definition at line 675 of file LoopVectorize.cpp.

Referenced by checkMixedPrecision(), collectSupportedLoops(), llvm::LoopVectorizationCostModel::computeMaxVF(), emitMemRuntimeChecks(), llvm::LoopVectorizationPlanner::executePlan(), isExplicitVecOuterLoop(), llvm::LoopVectorizationPlanner::plan(), processLoopInVPlanNativePath(), llvm::reportVectorizationFailure(), llvm::reportVectorizationInfo(), llvm::LoopVectorizationCostModel::runtimeChecksRequired(), and llvm::LoopVectorizationCostModel::selectVectorizationFactor().

◆ OrigLoop

Loop* llvm::InnerLoopVectorizer::OrigLoop

protected

The original loop.

Definition at line 646 of file LoopVectorize.cpp.

Referenced by addUsersInExitBlock(), llvm::LoopVectorizationPlanner::collectTriviallyDeadInstructions(), completeLoopSkeleton(), llvm::VPRecipeBuilder::createBlockInMask(), llvm::VPRecipeBuilder::createEdgeMask(), llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), CreateStepValue(), createVectorizedLoopSkeleton(), createVectorLoopSkeleton(), createWidenInductionRecipes(), emitMemRuntimeChecks(), llvm::LoopVectorizationPlanner::executePlan(), llvm::VPRecipeBuilder::fixHeaderPhis(), fixReduction(), fixupIVUsers(), fixVectorizedLoop(), getBroadcastInstrs(), InnerLoopVectorizer(), llvm::LoopVectorizationPlanner::plan(), llvm::LoopVectorizationPlanner::planInVPlanNativePath(), llvm::EpilogueVectorizerMainLoop::printDebugTracesAtEnd(), llvm::EpilogueVectorizerEpilogueLoop::printDebugTracesAtEnd(), and llvm::VPRecipeBuilder::tryToCreateWidenRecipe().

◆ PredicatedInstructions

SmallVector<Instruction *, 4> llvm::InnerLoopVectorizer::PredicatedInstructions

protected

Store instructions that were predicated.

Definition at line 717 of file LoopVectorize.cpp.

Referenced by scalarizeInstruction().

◆ PSE

PredicatedScalarEvolution& llvm::InnerLoopVectorizer::PSE

protected

A wrapper around ScalarEvolution used to add runtime SCEV checks.

Applies dynamic knowledge to simplify SCEV expressions and converts them to a more usable form.

Definition at line 654 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationCostModel::computeMaxVF(), createInductionResumeValues(), emitMemRuntimeChecks(), fixupIVUsers(), fixVectorizedLoop(), getAddressAccessSCEV(), getOrCreateTripCount(), llvm::LoopVectorizationCostModel::isMoreProfitable(), processLoopInVPlanNativePath(), llvm::LoopVectorizationCostModel::runtimeChecksRequired(), llvm::LoopVectorizationCostModel::selectInterleaveCount(), and llvm::VPRecipeBuilder::tryToCreateWidenRecipe().

◆ PSI

ProfileSummaryInfo* llvm::InnerLoopVectorizer::PSI

protected

Definition at line 740 of file LoopVectorize.cpp.

Referenced by getScalarEpilogueLowering(), InnerLoopVectorizer(), and processLoopInVPlanNativePath().

◆ ReductionResumeValues

SmallMapVector<const RecurrenceDescriptor *, PHINode *, 4> llvm::InnerLoopVectorizer::ReductionResumeValues

protected

Definition at line 753 of file LoopVectorize.cpp.

Referenced by fixReduction(), and getReductionResumeValue().

◆ RTChecks

GeneratedRTChecks& llvm::InnerLoopVectorizer::RTChecks

protected

Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.

Definition at line 748 of file LoopVectorize.cpp.

Referenced by emitMemRuntimeChecks(), and emitSCEVChecks().

◆ TLI

const TargetLibraryInfo* llvm::InnerLoopVectorizer::TLI

protected

Target Library Info.

Definition at line 666 of file LoopVectorize.cpp.

Referenced by getScalarEpilogueLowering(), llvm::LoopVectorizationCostModel::getVectorCallCost(), llvm::LoopVectorizationCostModel::getVectorIntrinsicCost(), processLoopInVPlanNativePath(), and widenCallInstruction().

◆ TripCount

Value* llvm::InnerLoopVectorizer::TripCount = nullptr

protected

Trip count of the original loop.

Definition at line 720 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop(), and getOrCreateTripCount().

◆ TTI

const TargetTransformInfo* llvm::InnerLoopVectorizer::TTI

protected

Target Transform Info.

Definition at line 669 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationCostModel::calculateRegisterUsage().

◆ UF

unsigned llvm::InnerLoopVectorizer::UF

protected

The vectorization unroll factor to use.

Each scalar is vectorized to this many different vector instructions.

Definition at line 690 of file LoopVectorize.cpp.

Referenced by clearReductionWrapFlags(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), fixFirstOrderRecurrence(), fixReduction(), getOrCreateVectorTripCount(), setDebugLocFromInst(), truncateToMinimalBitwidths(), vectorizeInterleaveGroup(), and widenCallInstruction().

◆ VectorTripCount

Value* llvm::InnerLoopVectorizer::VectorTripCount = nullptr

protected

Trip count of the widened loop (TripCount - TripCount % (VF*UF))

Definition at line 723 of file LoopVectorize.cpp.

Referenced by completeLoopSkeleton(), createInductionResumeValues(), fixupIVUsers(), and getOrCreateVectorTripCount().

◆ VF

ElementCount llvm::InnerLoopVectorizer::VF

protected

The vectorization SIMD factor to use.

Each vector will have this many vector elements.

Definition at line 686 of file LoopVectorize.cpp.

The documentation for this class was generated from the following file:

lib/Transforms/Vectorize/LoopVectorize.cpp

Public Types

Public Member Functions

Protected Types

Protected Member Functions

Protected Attributes

Friends

Detailed Description

Member Typedef Documentation

◆ PhiVector

◆ ScalarParts

◆ VectorParts

Constructor & Destructor Documentation

◆ InnerLoopVectorizer()

◆ ~InnerLoopVectorizer()

Member Function Documentation

◆ addMetadata() [1/2]

◆ addMetadata() [2/2]

◆ addNewMetadata()

◆ areSafetyChecksAdded()

◆ clearReductionWrapFlags()

◆ collectPoisonGeneratingRecipes()

◆ completeLoopSkeleton()

◆ createBitOrPointerCast()

◆ createInductionResumeValues()

◆ createVectorizedLoopSkeleton()

◆ createVectorLoopSkeleton()

◆ emitIterationCountCheck()

◆ emitMemRuntimeChecks()

◆ emitSCEVChecks()

◆ fixCrossIterationPHIs()

◆ fixFirstOrderRecurrence()

◆ fixNonInductionPHIs()

◆ fixReduction()

◆ fixupIVUsers()

◆ fixVectorizedLoop()

◆ getBroadcastInstrs()

◆ getOrCreateTripCount()

◆ getOrCreateVectorTripCount()

◆ getReductionResumeValue()

◆ packScalarIntoVectorValue()

◆ printDebugTracesAtEnd()

◆ printDebugTracesAtStart()

◆ scalarizeInstruction()

◆ setDebugLocFromInst()

◆ sinkScalarOperands()

◆ truncateToMinimalBitwidths()

◆ useOrderedReductions()

◆ vectorizeInterleaveGroup()

◆ widenCallInstruction()

Friends And Related Function Documentation

◆ LoopVectorizationPlanner

Member Data Documentation

◆ AA

◆ AC

◆ AddedSafetyChecks

◆ BFI

◆ Builder

◆ Cost

◆ DT

◆ IVEndValues

◆ Legal

◆ LI

◆ LoopBypassBlocks

◆ LoopExitBlock

◆ LoopMiddleBlock

◆ LoopScalarBody

◆ LoopScalarPreHeader

◆ LoopVectorPreHeader

◆ LVer

◆ OptForSizeBasedOnProfile

◆ ORE

◆ OrigLoop

◆ PredicatedInstructions

◆ PSE

◆ PSI

◆ ReductionResumeValues

◆ RTChecks

◆ TLI

◆ TripCount

◆ TTI