LoopVectorizationCostModel - estimates the expected speedups due to vectorization. More...

Collaboration diagram for llvm::LoopVectorizationCostModel:

[legend]

Classes
struct	RegisterUsage
	A struct that represents some properties of the register usage of a loop. More...

Public Types
enum	InstWidening { CM_Unknown, CM_Widen, CM_Widen_Reverse, CM_Interleave, CM_GatherScatter, CM_Scalarize }
	Decision that was taken during cost calculation for memory instruction. More...

using	ReductionChainMap = SmallMapVector< PHINode , SmallVector< Instruction , 4 >, 4 >
	A SmallMapVector to store the InLoop reduction op chains, mapping phi nodes to the chain of instructions representing the reductions. More...

Public Member Functions
	LoopVectorizationCostModel (ScalarEpilogueLowering SEL, Loop L, PredicatedScalarEvolution &PSE, LoopInfo LI, LoopVectorizationLegality Legal, const TargetTransformInfo &TTI, const TargetLibraryInfo TLI, DemandedBits DB, AssumptionCache AC, OptimizationRemarkEmitter ORE, const Function F, const LoopVectorizeHints *Hints, InterleavedAccessInfo &IAI)

FixedScalableVFPair	computeMaxVF (ElementCount UserVF, unsigned UserIC)

bool	runtimeChecksRequired ()

VectorizationFactor	selectVectorizationFactor (const ElementCountSet &CandidateVFs)

VectorizationFactor	selectEpilogueVectorizationFactor (const ElementCount MaxVF, const LoopVectorizationPlanner &LVP)

bool	selectUserVectorizationFactor (ElementCount UserVF)
	Setup cost-based decisions for user vectorization factor. More...

std::pair< unsigned, unsigned >	getSmallestAndWidestTypes ()

unsigned	selectInterleaveCount (ElementCount VF, unsigned LoopCost)

void	setCostBasedWideningDecision (ElementCount VF)
	Memory access instruction may be vectorized in more than one way. More...

SmallVector< RegisterUsage, 8 >	calculateRegisterUsage (ArrayRef< ElementCount > VFs)

void	collectValuesToIgnore ()
	Collect values we want to ignore in the cost model. More...

void	collectElementTypesForWidening ()
	Collect all element types in the loop for which widening is needed. More...

void	collectInLoopReductions ()
	Split reductions into those that happen in the loop, and those that happen outside. More...

bool	useOrderedReductions (const RecurrenceDescriptor &RdxDesc)
	Returns true if we should use strict in-order reductions for the given RdxDesc. More...

const MapVector< Instruction *, uint64_t > &	getMinimalBitwidths () const

bool	isProfitableToScalarize (Instruction *I, ElementCount VF) const

bool	isUniformAfterVectorization (Instruction *I, ElementCount VF) const
	Returns true if `I` is known to be uniform after vectorization. More...

bool	isScalarAfterVectorization (Instruction *I, ElementCount VF) const
	Returns true if `I` is known to be scalar after vectorization. More...

bool	canTruncateToMinimalBitwidth (Instruction *I, ElementCount VF) const

void	setWideningDecision (Instruction *I, ElementCount VF, InstWidening W, InstructionCost Cost)
	Save vectorization decision `W` and `Cost` taken by the cost model for instruction `I` and vector width `VF`. More...

void	setWideningDecision (const InterleaveGroup< Instruction > *Grp, ElementCount VF, InstWidening W, InstructionCost Cost)
	Save vectorization decision `W` and `Cost` taken by the cost model for interleaving group `Grp` and vector width `VF`. More...

InstWidening	getWideningDecision (Instruction *I, ElementCount VF) const
	Return the cost model decision for the given instruction `I` and vector width `VF`. More...

InstructionCost	getWideningCost (Instruction *I, ElementCount VF)
	Return the vectorization cost for the given instruction `I` and vector width `VF`. More...

bool	isOptimizableIVTruncate (Instruction *I, ElementCount VF)
	Return True if instruction `I` is an optimizable truncate whose operand is an induction variable. More...

void	collectInstsToScalarize (ElementCount VF)
	Collects the instructions to scalarize for each predicated instruction in the loop. More...

void	collectUniformsAndScalars (ElementCount VF)
	Collect Uniform and Scalar values for the given `VF`. More...

bool	isLegalMaskedStore (Type DataType, Value Ptr, Align Alignment) const
	Returns true if the target machine supports masked store operation for the given `DataType` and kind of access to `Ptr`. More...

bool	isLegalMaskedLoad (Type DataType, Value Ptr, Align Alignment) const
	Returns true if the target machine supports masked load operation for the given `DataType` and kind of access to `Ptr`. More...

bool	isLegalGatherOrScatter (Value *V)
	Returns true if the target machine can represent `V` as a masked gather or scatter operation. More...

bool	canVectorizeReductions (ElementCount VF) const
	Returns true if the target machine supports all of the reduction variables found for the given VF. More...

bool	isScalarWithPredication (Instruction *I) const
	Returns true if `I` is an instruction that will be scalarized with predication. More...

bool	isPredicatedInst (Instruction *I)

bool	memoryInstructionCanBeWidened (Instruction *I, ElementCount VF=ElementCount::getFixed(1))
	Returns true if `I` is a memory instruction with consecutive memory access that can be widened. More...

bool	interleavedAccessCanBeWidened (Instruction *I, ElementCount VF=ElementCount::getFixed(1))
	Returns true if `I` is a memory instruction in an interleaved-group of memory accesses that can be vectorized with wide vector loads/stores and shuffles. More...

bool	isAccessInterleaved (Instruction *Instr)
	Check if `Instr` belongs to any interleaved access group. More...

const InterleaveGroup< Instruction > *	getInterleavedAccessGroup (Instruction *Instr)
	Get the interleaved access group that `Instr` belongs to. More...

bool	requiresScalarEpilogue (ElementCount VF) const
	Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop. More...

bool	isScalarEpilogueAllowed () const
	Returns true if a scalar epilogue is not allowed due to optsize or a loop hint annotation. More...

bool	foldTailByMasking () const
	Returns true if all loop blocks should be masked to fold tail loop. More...

bool	blockNeedsPredication (BasicBlock *BB) const

const ReductionChainMap &	getInLoopReductionChains () const
	Return the chain of instructions representing an inloop reduction. More...

bool	isInLoopReduction (PHINode *Phi) const
	Returns true if the Phi is part of an inloop reduction. More...

InstructionCost	getVectorIntrinsicCost (CallInst *CI, ElementCount VF) const
	Estimate cost of an intrinsic call instruction CI if it were vectorized with factor VF. More...

InstructionCost	getVectorCallCost (CallInst *CI, ElementCount VF, bool &NeedToScalarize) const
	Estimate cost of a call instruction CI if it were vectorized with factor VF. More...

bool	isMoreProfitable (const VectorizationFactor &A, const VectorizationFactor &B) const
	Returns true if the per-lane cost of VectorizationFactor A is lower than that of B. More...

void	invalidateCostModelingDecisions ()
	Invalidates decisions already taken by the cost model. More...

Public Attributes
Loop *	TheLoop
	The loop that we evaluate. More...

PredicatedScalarEvolution &	PSE
	Predicated scalar evolution analysis. More...

LoopInfo *	LI
	Loop Info analysis. More...

LoopVectorizationLegality *	Legal
	Vectorization legality. More...

const TargetTransformInfo &	TTI
	Vector target information. More...

const TargetLibraryInfo *	TLI
	Target Library Info. More...

DemandedBits *	DB
	Demanded bits analysis. More...

AssumptionCache *	AC
	Assumption cache. More...

OptimizationRemarkEmitter *	ORE
	Interface to emit optimization remarks. More...

const Function *	TheFunction

const LoopVectorizeHints *	Hints
	Loop Vectorize Hint. More...

InterleavedAccessInfo &	InterleaveInfo
	The interleave access information contains groups of interleaved accesses with the same stride and close to each other. More...

SmallPtrSet< const Value *, 16 >	ValuesToIgnore
	Values to ignore in the cost model. More...

SmallPtrSet< const Value *, 16 >	VecValuesToIgnore
	Values to ignore in the cost model when VF > 1. More...

SmallPtrSet< Type *, 16 >	ElementTypesInLoop
	All element types found in the loop. More...

SmallVector< VectorizationFactor, 8 >	ProfitableVFs
	Profitable vector factors. More...

Detailed Description

LoopVectorizationCostModel - estimates the expected speedups due to vectorization.

In many cases vectorization is not profitable. This can happen because of a number of reasons. In this class we mainly attempt to predict the expected speedup/slowdowns due to the supported instruction set. We use the TargetTransformInfo to query the different backends for the cost of different operations.

Definition at line 1226 of file LoopVectorize.cpp.

Member Typedef Documentation

◆ ReductionChainMap

using llvm::LoopVectorizationCostModel::ReductionChainMap = SmallMapVector<PHINode *, SmallVector<Instruction *, 4>, 4>

A SmallMapVector to store the InLoop reduction op chains, mapping phi nodes to the chain of instructions representing the reductions.

Uses a MapVector to ensure deterministic iteration order.

Definition at line 1601 of file LoopVectorize.cpp.

Member Enumeration Documentation

◆ InstWidening

enum llvm::LoopVectorizationCostModel::InstWidening

Decision that was taken during cost calculation for memory instruction.

Enumerator
CM_Unknown
CM_Widen
CM_Widen_Reverse
CM_Interleave
CM_GatherScatter
CM_Scalarize

Definition at line 1388 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ LoopVectorizationCostModel()

llvm::LoopVectorizationCostModel::LoopVectorizationCostModel	(	ScalarEpilogueLowering	SEL,
		Loop *	L,
		PredicatedScalarEvolution &	PSE,
		LoopInfo *	LI,
		LoopVectorizationLegality *	Legal,
		const TargetTransformInfo &	TTI,
		const TargetLibraryInfo *	TLI,
		DemandedBits *	DB,
		AssumptionCache *	AC,
		OptimizationRemarkEmitter *	ORE,
		const Function *	F,
		const LoopVectorizeHints *	Hints,
		InterleavedAccessInfo &	IAI
	)

inline

Definition at line 1228 of file LoopVectorize.cpp.

Member Function Documentation

◆ blockNeedsPredication()

bool llvm::LoopVectorizationCostModel::blockNeedsPredication ( BasicBlock * BB ) const

inline

Definition at line 1593 of file LoopVectorize.cpp.

References BB, llvm::LoopVectorizationLegality::blockNeedsPredication(), and llvm::InnerLoopVectorizer::Legal.

◆ calculateRegisterUsage()

SmallVector< LoopVectorizationCostModel::RegisterUsage, 8 > LoopVectorizationCostModel::calculateRegisterUsage ( ArrayRef< ElementCount > VFs )

Returns: Returns information about the register usages of the loop for the given vectorization factors.

Definition at line 6584 of file LoopVectorize.cpp.

References assert(), BB, llvm::LoopBlocksDFS::beginRPO(), llvm::SmallPtrSetImpl< PtrType >::count(), llvm::dbgs(), llvm::numbers::e, llvm::MapVector< KeyT, ValueT, MapType, VectorType >::end(), llvm::LoopBlocksDFS::endRPO(), llvm::SmallPtrSetImpl< PtrType >::erase(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::find(), llvm::find(), llvm::VectorType::get(), llvm::TargetTransformInfo::getRegisterClassForType(), llvm::TargetTransformInfo::getRegisterClassName(), i, I, llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::LegalityPredicates::isScalar(), llvm::Type::isTokenTy(), llvm::VectorType::isValidElementType(), llvm::LegalityPredicates::isVector(), j(), llvm::InnerLoopVectorizer::LI, List, LLVM_DEBUG, llvm::LoopVectorizationCostModel::RegisterUsage::LoopInvariantRegs, llvm::make_range(), llvm::max(), llvm::LoopVectorizationCostModel::RegisterUsage::MaxLocalUsers, llvm::LoopBlocksDFS::perform(), llvm::RegUsage, s, llvm::MapVector< KeyT, ValueT, MapType, VectorType >::size(), llvm::SmallPtrSetImplBase::size(), llvm::ArrayRef< T >::size(), ToRemove, llvm::InnerLoopVectorizer::TTI, and llvm::InnerLoopVectorizer::VF.

◆ canTruncateToMinimalBitwidth()

bool llvm::LoopVectorizationCostModel::canTruncateToMinimalBitwidth	(	Instruction *	I,
		ElementCount	VF
	)		const

inline

Returns: True if instruction I can be truncated to a smaller bitwidth for vectorization factor VF.

Definition at line 1381 of file LoopVectorize.cpp.

References I, llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.

◆ canVectorizeReductions()

bool llvm::LoopVectorizationCostModel::canVectorizeReductions ( ElementCount VF ) const

inline

Returns true if the target machine supports all of the reduction variables found for the given VF.

Definition at line 1521 of file LoopVectorize.cpp.

References llvm::all_of(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::InnerLoopVectorizer::Legal, and Reduction.

◆ collectElementTypesForWidening()

void LoopVectorizationCostModel::collectElementTypesForWidening ( )

Collect all element types in the loop for which widening is needed.

Definition at line 6292 of file LoopVectorize.cpp.

References BB, llvm::RecurrenceDescriptor::getOpcode(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::LoopVectorizationLegality::getReductionVars(), I, llvm::LoopVectorizationLegality::isReductionVariable(), llvm::InnerLoopVectorizer::Legal, llvm::TargetTransformInfo::preferInLoopReduction(), PreferInLoopReductions, llvm::ARM_MB::ST, and llvm::InnerLoopVectorizer::useOrderedReductions().

Referenced by llvm::LoopVectorizePass::processLoop(), and processLoopInVPlanNativePath().

◆ collectInLoopReductions()

void LoopVectorizationCostModel::collectInLoopReductions ( )

Split reductions into those that happen in the loop, and those that happen outside.

In loop reductions are collected into InLoopReductionChains.

Definition at line 8004 of file LoopVectorize.cpp.

References llvm::dbgs(), llvm::RecurrenceDescriptor::getOpcode(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::RecurrenceDescriptor::getReductionOpChain(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::Value::getType(), I, llvm::InnerLoopVectorizer::Legal, LLVM_DEBUG, llvm::TargetTransformInfo::preferInLoopReduction(), PreferInLoopReductions, Reduction, and llvm::InnerLoopVectorizer::useOrderedReductions().

◆ collectInstsToScalarize()

void LoopVectorizationCostModel::collectInstsToScalarize ( ElementCount VF )

Collects the instructions to scalarize for each predicated instruction in the loop.

Definition at line 6795 of file LoopVectorize.cpp.

References BB, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::begin(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::DenseMapBase< DenseMap< KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >, KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >::end(), llvm::DenseMapBase< DenseMap< KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >, KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >::find(), I, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::insert(), llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::LinearPolySize< LeafTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::UnivariateLinearPolyBase< LeafTy >::isZero(), and llvm::InnerLoopVectorizer::VF.

◆ collectUniformsAndScalars()

void llvm::LoopVectorizationCostModel::collectUniformsAndScalars ( ElementCount VF )

inline

Collect Uniform and Scalar values for the given VF.

The sets depend on CM decision for Load/Store instructions that may be vectorized as interleave, gather-scatter or scalarized.

Definition at line 1483 of file LoopVectorize.cpp.

References llvm::ElementCount::isScalar(), and llvm::InnerLoopVectorizer::VF.

◆ collectValuesToIgnore()

void LoopVectorizationCostModel::collectValuesToIgnore ( )

Collect values we want to ignore in the cost model.

Definition at line 7984 of file LoopVectorize.cpp.

References llvm::InnerLoopVectorizer::AC, llvm::SmallPtrSetImpl< PtrType >::begin(), llvm::CodeMetrics::collectEphemeralValues(), llvm::SmallPtrSetImpl< PtrType >::end(), llvm::RecurrenceDescriptor::getCastInsts(), llvm::InductionDescriptor::getCastInsts(), llvm::LoopVectorizationLegality::getInductionVars(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::InnerLoopVectorizer::Induction, llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::SmallVectorImpl< T >::insert(), llvm::InnerLoopVectorizer::Legal, and Reduction.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ computeMaxVF()

FixedScalableVFPair LoopVectorizationCostModel::computeMaxVF	(	ElementCount	UserVF,
		unsigned	UserIC
	)

Returns: An upper bound for the vectorization factors (both fixed and scalable). If the factors are 0, vectorization and interleaving should be avoided up front.

Definition at line 5756 of file LoopVectorize.cpp.

References llvm::ScalarEvolution::applyLoopGuards(), assert(), llvm::CM_ScalarEpilogueAllowed, llvm::CM_ScalarEpilogueNotAllowedLowTripLoop, llvm::CM_ScalarEpilogueNotAllowedOptSize, llvm::CM_ScalarEpilogueNotAllowedUsePredicate, llvm::CM_ScalarEpilogueNotNeededUsePredicate, llvm::dbgs(), llvm::DenseMapBase< DenseMap< KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >, KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >::empty(), llvm::FixedScalableVFPair::FixedVF, llvm::ScalarEvolution::getAddExpr(), llvm::PredicatedScalarEvolution::getBackedgeTakenCount(), llvm::ScalarEvolution::getConstant(), llvm::LinearPolySize< LeafTy >::getFixedValue(), llvm::FixedScalableVFPair::getNone(), llvm::ScalarEvolution::getOne(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::LinearPolySize< ElementCount >::getScalable(), llvm::PredicatedScalarEvolution::getSE(), llvm::ScalarEvolution::getSmallConstantTripCount(), llvm::SCEV::getType(), llvm::ScalarEvolution::getURemExpr(), llvm::TargetTransformInfo::hasBranchDivergence(), llvm::UnivariateLinearPolyBase< LeafTy >::isNonZero(), llvm::isPowerOf2_32(), llvm::ElementCount::isVector(), llvm::SCEV::isZero(), llvm::InnerLoopVectorizer::Legal, LLVM_DEBUG, LLVM_FALLTHROUGH, llvm::RuntimePointerChecking::Need, llvm::InnerLoopVectorizer::ORE, llvm::LoopVectorizationLegality::prepareToFoldTailByMasking(), llvm::InnerLoopVectorizer::PSE, llvm::reportVectorizationFailure(), llvm::FixedScalableVFPair::ScalableVF, and useMaskedInterleavedAccesses().

◆ foldTailByMasking()

bool llvm::LoopVectorizationCostModel::foldTailByMasking ( ) const

inline

Returns true if all loop blocks should be masked to fold tail loop.

Definition at line 1591 of file LoopVectorize.cpp.

Referenced by llvm::InnerLoopVectorizer::completeLoopSkeleton(), llvm::InnerLoopVectorizer::emitMinimumIterationCountCheck(), llvm::InnerLoopVectorizer::fixReduction(), and llvm::InnerLoopVectorizer::getOrCreateVectorTripCount().

◆ getInLoopReductionChains()

const ReductionChainMap& llvm::LoopVectorizationCostModel::getInLoopReductionChains ( ) const

inline

Return the chain of instructions representing an inloop reduction.

Definition at line 1604 of file LoopVectorize.cpp.

◆ getInterleavedAccessGroup()

const InterleaveGroup<Instruction>* llvm::LoopVectorizationCostModel::getInterleavedAccessGroup ( Instruction * Instr )

inline

Get the interleaved access group that Instr belongs to.

Definition at line 1568 of file LoopVectorize.cpp.

◆ getMinimalBitwidths()

const MapVector<Instruction *, uint64_t>& llvm::LoopVectorizationCostModel::getMinimalBitwidths ( ) const

inline

Returns: The smallest bitwidth each instruction can be represented with. The vector equivalents of these instructions should be truncated to this type.

Definition at line 1326 of file LoopVectorize.cpp.

Referenced by llvm::InnerLoopVectorizer::truncateToMinimalBitwidths().

◆ getSmallestAndWidestTypes()

std::pair< unsigned, unsigned > LoopVectorizationCostModel::getSmallestAndWidestTypes ( )

Returns: The size (in bits) of the smallest and widest types in the code that needs to be vectorized. We ignore values that remain scalar such as 64 bit loop indices.

Definition at line 6279 of file LoopVectorize.cpp.

References DL, llvm::MachineFunction::getDataLayout(), and llvm::MachineBasicBlock::getParent().

Referenced by determineVPlanVF().

◆ getVectorCallCost()

InstructionCost LoopVectorizationCostModel::getVectorCallCost	(	CallInst *	CI,
		ElementCount	VF,
		bool &	NeedToScalarize
	)		const

Estimate cost of a call instruction CI if it were vectorized with factor VF.

Return the cost of the instruction, including scalarization overhead if it's needed. The flag NeedToScalarize shows if the call needs to be scalarized - i.e. either vector version isn't available, or is too expensive.

Definition at line 3873 of file LoopVectorize.cpp.

References llvm::CallBase::args(), llvm::InnerLoopVectorizer::Cost, F, llvm::VFShape::get(), llvm::CallBase::getCalledFunction(), llvm::TargetTransformInfo::getCallInstrCost(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::Value::getType(), llvm::VFDatabase::getVectorizedFunction(), llvm::CallBase::isNoBuiltin(), llvm::ElementCount::isScalar(), llvm::TargetTransformInfo::TCK_RecipThroughput, llvm::InnerLoopVectorizer::TLI, llvm::ToVectorTy(), and llvm::InnerLoopVectorizer::VF.

Referenced by llvm::InnerLoopVectorizer::widenCallInstruction().

◆ getVectorIntrinsicCost()

InstructionCost LoopVectorizationCostModel::getVectorIntrinsicCost	(	CallInst *	CI,
		ElementCount	VF
	)		const

Estimate cost of an intrinsic call instruction CI if it were vectorized with factor VF.

Return the cost of the instruction, including scalarization overhead if it's needed.

Definition at line 3928 of file LoopVectorize.cpp.

References llvm::CallBase::args(), Arguments, assert(), llvm::CallBase::getCalledFunction(), llvm::Function::getFunctionType(), llvm::TargetTransformInfo::getIntrinsicInstrCost(), llvm::Value::getType(), llvm::getVectorIntrinsicIDForCall(), MaybeVectorizeType(), llvm::FunctionType::param_begin(), llvm::FunctionType::param_end(), llvm::TargetTransformInfo::TCK_RecipThroughput, llvm::InnerLoopVectorizer::TLI, transform, and llvm::InnerLoopVectorizer::VF.

Referenced by llvm::InnerLoopVectorizer::widenCallInstruction().

◆ getWideningCost()

InstructionCost llvm::LoopVectorizationCostModel::getWideningCost	(	Instruction *	I,
		ElementCount	VF
	)

inline

Return the vectorization cost for the given instruction I and vector width VF.

Definition at line 1442 of file LoopVectorize.cpp.

References assert(), I, llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.

◆ getWideningDecision()

InstWidening llvm::LoopVectorizationCostModel::getWideningDecision	(	Instruction *	I,
		ElementCount	VF
	)		const

inline

Return the cost model decision for the given instruction I and vector width VF.

Return CM_Unknown if this instruction did not pass through the cost modeling.

Definition at line 1426 of file LoopVectorize.cpp.

References assert(), EnableVPlanNativePath, I, llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.

◆ interleavedAccessCanBeWidened()

bool LoopVectorizationCostModel::interleavedAccessCanBeWidened	(	Instruction *	I,
		ElementCount	VF = `ElementCount::getFixed(1)`
	)

Returns true if I is a memory instruction in an interleaved-group of memory accesses that can be vectorized with wide vector loads/stores and shuffles.

Definition at line 5270 of file LoopVectorize.cpp.

References assert(), DL, llvm::getLoadStoreAlignment(), llvm::getLoadStoreType(), hasIrregularType(), I, llvm::TargetTransformInfo::isLegalMaskedLoad(), llvm::TargetTransformInfo::isLegalMaskedStore(), llvm::LoopVectorizationLegality::isMaskRequired(), llvm::InnerLoopVectorizer::Legal, useMaskedInterleavedAccesses(), and llvm::InnerLoopVectorizer::VF.

◆ invalidateCostModelingDecisions()

void llvm::LoopVectorizationCostModel::invalidateCostModelingDecisions ( )

inline

Invalidates decisions already taken by the cost model.

Definition at line 1632 of file LoopVectorize.cpp.

◆ isAccessInterleaved()

bool llvm::LoopVectorizationCostModel::isAccessInterleaved ( Instruction * Instr )

inline

Check if Instr belongs to any interleaved access group.

Definition at line 1562 of file LoopVectorize.cpp.

◆ isInLoopReduction()

bool llvm::LoopVectorizationCostModel::isInLoopReduction ( PHINode * Phi ) const

inline

Returns true if the Phi is part of an inloop reduction.

Definition at line 1609 of file LoopVectorize.cpp.

◆ isLegalGatherOrScatter()

bool llvm::LoopVectorizationCostModel::isLegalGatherOrScatter ( Value * V )

inline

Returns true if the target machine can represent V as a masked gather or scatter operation.

Definition at line 1508 of file LoopVectorize.cpp.

References llvm::getLoadStoreAlignment(), llvm::getLoadStoreType(), llvm::TargetTransformInfo::isLegalMaskedGather(), llvm::TargetTransformInfo::isLegalMaskedScatter(), llvm::InnerLoopVectorizer::LI, and SI.

◆ isLegalMaskedLoad()

bool llvm::LoopVectorizationCostModel::isLegalMaskedLoad	(	Type *	DataType,
		Value *	Ptr,
		Align	Alignment
	)		const

inline

Returns true if the target machine supports masked load operation for the given DataType and kind of access to Ptr.

Definition at line 1501 of file LoopVectorize.cpp.

References llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::TargetTransformInfo::isLegalMaskedLoad(), and llvm::InnerLoopVectorizer::Legal.

◆ isLegalMaskedStore()

bool llvm::LoopVectorizationCostModel::isLegalMaskedStore	(	Type *	DataType,
		Value *	Ptr,
		Align	Alignment
	)		const

inline

Returns true if the target machine supports masked store operation for the given DataType and kind of access to Ptr.

Definition at line 1494 of file LoopVectorize.cpp.

References llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::TargetTransformInfo::isLegalMaskedStore(), and llvm::InnerLoopVectorizer::Legal.

◆ isMoreProfitable()

bool LoopVectorizationCostModel::isMoreProfitable	(	const VectorizationFactor &	A,
		const VectorizationFactor &	B
	)		const

Returns true if the per-lane cost of VectorizationFactor A is lower than that of B.

Definition at line 5992 of file LoopVectorize.cpp.

References B, llvm::divideCeil(), llvm::PredicatedScalarEvolution::getSE(), llvm::ScalarEvolution::getSmallConstantMaxTripCount(), and llvm::InnerLoopVectorizer::PSE.

◆ isOptimizableIVTruncate()

bool llvm::LoopVectorizationCostModel::isOptimizableIVTruncate	(	Instruction *	I,
		ElementCount	VF
	)

inline

Return True if instruction I is an optimizable truncate whose operand is an induction variable.

Such a truncate will be removed by adding a new induction variable with the destination type.

Definition at line 1453 of file LoopVectorize.cpp.

References llvm::LoopVectorizationLegality::getPrimaryInduction(), I, llvm::LoopVectorizationLegality::isInductionPhi(), llvm::TargetTransformInfo::isTruncateFree(), llvm::InnerLoopVectorizer::Legal, llvm::ToVectorTy(), and llvm::InnerLoopVectorizer::VF.

◆ isPredicatedInst()

bool llvm::LoopVectorizationCostModel::isPredicatedInst ( Instruction * I )

inline

Definition at line 1538 of file LoopVectorize.cpp.

References I, llvm::LoopVectorizationLegality::isMaskRequired(), and llvm::InnerLoopVectorizer::Legal.

◆ isProfitableToScalarize()

bool llvm::LoopVectorizationCostModel::isProfitableToScalarize	(	Instruction *	I,
		ElementCount	VF
	)		const

inline

Returns: True if it is more profitable to scalarize instruction I for vectorization factor VF.

Definition at line 1332 of file LoopVectorize.cpp.

References assert(), EnableVPlanNativePath, I, llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.

Referenced by llvm::InnerLoopVectorizer::shouldScalarizeInstruction().

◆ isScalarAfterVectorization()

bool llvm::LoopVectorizationCostModel::isScalarAfterVectorization	(	Instruction *	I,
		ElementCount	VF
	)		const

inline

Returns true if I is known to be scalar after vectorization.

Definition at line 1364 of file LoopVectorize.cpp.

References assert(), EnableVPlanNativePath, I, llvm::ElementCount::isScalar(), and llvm::InnerLoopVectorizer::VF.

Referenced by llvm::InnerLoopVectorizer::shouldScalarizeInstruction(), and llvm::InnerLoopVectorizer::widenPHIInstruction().

◆ isScalarEpilogueAllowed()

bool llvm::LoopVectorizationCostModel::isScalarEpilogueAllowed ( ) const

inline

Returns true if a scalar epilogue is not allowed due to optsize or a loop hint annotation.

Definition at line 1586 of file LoopVectorize.cpp.

References llvm::CM_ScalarEpilogueAllowed.

Referenced by llvm::InnerLoopVectorizer::vectorizeInterleaveGroup().

◆ isScalarWithPredication()

bool LoopVectorizationCostModel::isScalarWithPredication ( Instruction * I ) const

Returns true if I is an instruction that will be scalarized with predication.

Such instructions include conditional stores and instructions that may divide by zero. If a non-zero VF has been calculated, we check if I will be scalarized predication for that VF.

Definition at line 5243 of file LoopVectorize.cpp.

References llvm::getLoadStoreAlignment(), llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), I, llvm::TargetTransformInfo::isLegalMaskedGather(), llvm::TargetTransformInfo::isLegalMaskedScatter(), llvm::LoopVectorizationLegality::isMaskRequired(), llvm::InnerLoopVectorizer::Legal, llvm::SPII::Load, mayDivideByZero(), and llvm::SPII::Store.

◆ isUniformAfterVectorization()

bool llvm::LoopVectorizationCostModel::isUniformAfterVectorization	(	Instruction *	I,
		ElementCount	VF
	)		const

inline

Returns true if I is known to be uniform after vectorization.

Definition at line 1348 of file LoopVectorize.cpp.

References assert(), EnableVPlanNativePath, I, llvm::ElementCount::isScalar(), and llvm::InnerLoopVectorizer::VF.

Referenced by llvm::InnerLoopVectorizer::buildScalarSteps(), llvm::InnerLoopVectorizer::fixLCSSAPHIs(), llvm::InnerLoopVectorizer::scalarizeInstruction(), and llvm::InnerLoopVectorizer::widenPHIInstruction().

◆ memoryInstructionCanBeWidened()

bool LoopVectorizationCostModel::memoryInstructionCanBeWidened	(	Instruction *	I,
		ElementCount	VF = `ElementCount::getFixed(1)`
	)

Returns true if I is a memory instruction with consecutive memory access that can be widened.

Definition at line 5317 of file LoopVectorize.cpp.

References assert(), DL, llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), hasIrregularType(), I, llvm::LoopVectorizationLegality::isConsecutivePtr(), and llvm::InnerLoopVectorizer::Legal.

◆ requiresScalarEpilogue()

bool llvm::LoopVectorizationCostModel::requiresScalarEpilogue ( ElementCount VF ) const

inline

Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop.

Definition at line 1574 of file LoopVectorize.cpp.

References llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.

Referenced by llvm::InnerLoopVectorizer::completeLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), llvm::InnerLoopVectorizer::createVectorLoopSkeleton(), llvm::InnerLoopVectorizer::emitMinimumIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitMinimumIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), llvm::InnerLoopVectorizer::emitSCEVChecks(), llvm::InnerLoopVectorizer::fixReduction(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), and llvm::InnerLoopVectorizer::getOrCreateVectorTripCount().

◆ runtimeChecksRequired()

bool LoopVectorizationCostModel::runtimeChecksRequired ( )

Returns: True if runtime checks are required for vectorization, and false otherwise.

Definition at line 5552 of file LoopVectorize.cpp.

References llvm::dbgs(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::empty(), llvm::LoopVectorizationLegality::getLAI(), llvm::SCEVUnionPredicate::getPredicates(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::LoopAccessInfo::getSymbolicStrides(), llvm::PredicatedScalarEvolution::getUnionPredicate(), llvm::InnerLoopVectorizer::Legal, LLVM_DEBUG, llvm::RuntimePointerChecking::Need, llvm::InnerLoopVectorizer::ORE, llvm::InnerLoopVectorizer::PSE, and llvm::reportVectorizationFailure().

◆ selectEpilogueVectorizationFactor()

VectorizationFactor LoopVectorizationCostModel::selectEpilogueVectorizationFactor	(	const ElementCount	MaxVF,
		const LoopVectorizationPlanner &	LVP
	)

Definition at line 6208 of file LoopVectorize.cpp.

References llvm::dbgs(), llvm::VectorizationFactor::Disabled(), EnableEpilogueVectorization, EpilogueVectorizationForceVF, llvm::LinearPolySize< ElementCount >::getFixed(), llvm::LoopVectorizationPlanner::hasPlanWithVFs(), llvm::LinearPolySize< ElementCount >::isKnownLT(), llvm::LinearPolySize< LeafTy >::isScalable(), and LLVM_DEBUG.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ selectInterleaveCount()

unsigned LoopVectorizationCostModel::selectInterleaveCount	(	ElementCount	VF,
		unsigned	LoopCost
	)

Returns: The desired interleave count. If interleave count has been specified by metadata it will be returned. Otherwise, the interleave count is computed and returned. VF and LoopCost are the selected vectorization factor and the cost of the selected VF.

Definition at line 6344 of file LoopVectorize.cpp.

References llvm::any_of(), assert(), llvm::dbgs(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::empty(), llvm::TargetTransformInfo::enableAggressiveInterleaving(), EnableIndVarRegisterHeur, EnableLoadStoreRuntimeInterleave, F, ForceTargetMaxScalarInterleaveFactor, ForceTargetMaxVectorInterleaveFactor, ForceTargetNumScalarRegs, ForceTargetNumVectorRegs, llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::TargetTransformInfo::getMaxInterleaveFactor(), llvm::LoopVectorizationLegality::getMaxSafeDepDistBytes(), llvm::TargetTransformInfo::getNumberOfRegisters(), llvm::LoopVectorizationLegality::getNumLoads(), llvm::LoopVectorizationLegality::getNumStores(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::TargetTransformInfo::getRegisterClassName(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getSE(), getSmallBestKnownTC(), InterleaveSmallLoopScalarReduction, llvm::ElementCount::isScalar(), llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::Legal, LLVM_DEBUG, llvm::max(), MaxNestedScalarReductionIC, llvm::min(), llvm::RuntimePointerChecking::Need, llvm::PowerOf2Floor(), llvm::InnerLoopVectorizer::PSE, Reduction, SmallLoopCost, TinyTripCountInterleaveThreshold, and llvm::InnerLoopVectorizer::VF.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ selectUserVectorizationFactor()

bool llvm::LoopVectorizationCostModel::selectUserVectorizationFactor ( ElementCount UserVF )

inline

Setup cost-based decisions for user vectorization factor.

Returns: true if the UserVF is a feasible VF to be chosen.

Definition at line 1263 of file LoopVectorize.cpp.

◆ selectVectorizationFactor()

VectorizationFactor LoopVectorizationCostModel::selectVectorizationFactor ( const ElementCountSet & CandidateVFs )

Returns: The most profitable vectorization factor and the cost of that VF. This method checks every VF in CandidateVFs. If UserVF is not ZERO then this vectorization factor will be selected if vectorization is possible.

Definition at line 6028 of file LoopVectorize.cpp.

References assert(), B, llvm::VectorizationFactor::Cost, llvm::SmallSet< T, N, C >::count(), llvm::dbgs(), EnableCondStoresVectorization, llvm::LoopVectorizeHints::FK_Enabled, llvm::raw_ostream::flush(), llvm::LinearPolySize< ElementCount >::getFixed(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::InstructionCost::getMax(), i, I, if(), llvm::ElementCount::isScalar(), llvm::InstructionCost::isValid(), LLVM_DEBUG, Numbering, llvm::InnerLoopVectorizer::ORE, llvm::reportVectorizationFailure(), llvm::reportVectorizationInfo(), llvm::SmallSet< T, N, C >::size(), llvm::sort(), llvm::CallingConv::Tail, and llvm::VectorizationFactor::Width.

◆ setCostBasedWideningDecision()

void LoopVectorizationCostModel::setCostBasedWideningDecision ( ElementCount VF )

Memory access instruction may be vectorized in more than one way.

Form of instruction after vectorization depends on cost. This function takes cost-based decisions for Load/Store instructions and collects them in a map. This decisions map is used for building the lists of loop-uniform and loop-scalar instructions. The calculated cost is saved with widening decision in order to avoid redundant calculations.

Definition at line 7439 of file LoopVectorize.cpp.

References llvm::append_range(), assert(), BB, llvm::InnerLoopVectorizer::Cost, llvm::LinearPolySize< ElementCount >::getFixed(), llvm::InstructionCost::getInvalid(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), I, llvm::DenseMapBase< DenseMap< KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >, KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >::insert(), llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::LinearPolySize< LeafTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::LoopVectorizationLegality::isUniformMemOp(), llvm::InnerLoopVectorizer::Legal, llvm::SmallVectorImpl< T >::pop_back_val(), llvm::TargetTransformInfo::prefersVectorizedAddressing(), and llvm::InnerLoopVectorizer::VF.

◆ setWideningDecision() [1/2]

void llvm::LoopVectorizationCostModel::setWideningDecision	(	const InterleaveGroup< Instruction > *	Grp,
		ElementCount	VF,
		InstWidening	W,
		InstructionCost	Cost
	)

inline

Save vectorization decision W and Cost taken by the cost model for interleaving group Grp and vector width VF.

Broadcast this decicion to all instructions inside the group. But the cost will be assigned to one instruction only.

Definition at line 1407 of file LoopVectorize.cpp.

References assert(), llvm::InnerLoopVectorizer::Cost, llvm::InterleaveGroup< InstTy >::getFactor(), llvm::InterleaveGroup< InstTy >::getInsertPos(), llvm::InterleaveGroup< InstTy >::getMember(), i, I, llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::VF, and llvm::RISCVFenceField::W.

◆ setWideningDecision() [2/2]

void llvm::LoopVectorizationCostModel::setWideningDecision	(	Instruction *	I,
		ElementCount	VF,
		InstWidening	W,
		InstructionCost	Cost
	)

inline

Save vectorization decision W and Cost taken by the cost model for instruction I and vector width VF.

Definition at line 1399 of file LoopVectorize.cpp.

References assert(), llvm::InnerLoopVectorizer::Cost, I, llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::VF, and llvm::RISCVFenceField::W.

◆ useOrderedReductions()

bool llvm::LoopVectorizationCostModel::useOrderedReductions ( const RecurrenceDescriptor & RdxDesc )

inline

Returns true if we should use strict in-order reductions for the given RdxDesc.

This is true if the -enable-strict-reductions flag is passed, the IsOrdered flag of RdxDesc is set and we do not allow reordering of FP operations.

Definition at line 1319 of file LoopVectorize.cpp.

References llvm::RecurrenceDescriptor::isOrdered().

Referenced by llvm::InnerLoopVectorizer::useOrderedReductions().

Member Data Documentation

◆ AC

AssumptionCache* llvm::LoopVectorizationCostModel::AC

Assumption cache.

Definition at line 1880 of file LoopVectorize.cpp.

◆ DB

DemandedBits* llvm::LoopVectorizationCostModel::DB

Demanded bits analysis.

Definition at line 1877 of file LoopVectorize.cpp.

◆ ElementTypesInLoop

SmallPtrSet<Type *, 16> llvm::LoopVectorizationCostModel::ElementTypesInLoop

All element types found in the loop.

Definition at line 1901 of file LoopVectorize.cpp.

◆ Hints

const LoopVectorizeHints* llvm::LoopVectorizationCostModel::Hints

Loop Vectorize Hint.

Definition at line 1888 of file LoopVectorize.cpp.

Referenced by llvm::InnerLoopVectorizer::emitMemRuntimeChecks(), and llvm::InnerLoopVectorizer::emitSCEVChecks().

◆ InterleaveInfo

InterleavedAccessInfo& llvm::LoopVectorizationCostModel::InterleaveInfo

The interleave access information contains groups of interleaved accesses with the same stride and close to each other.

Definition at line 1892 of file LoopVectorize.cpp.

◆ Legal

LoopVectorizationLegality* llvm::LoopVectorizationCostModel::Legal

Vectorization legality.

Definition at line 1868 of file LoopVectorize.cpp.

◆ LI

LoopInfo* llvm::LoopVectorizationCostModel::LI

Loop Info analysis.

Definition at line 1865 of file LoopVectorize.cpp.

◆ ORE

OptimizationRemarkEmitter* llvm::LoopVectorizationCostModel::ORE

Interface to emit optimization remarks.

Definition at line 1883 of file LoopVectorize.cpp.

◆ ProfitableVFs

SmallVector<VectorizationFactor, 8> llvm::LoopVectorizationCostModel::ProfitableVFs

Profitable vector factors.

Definition at line 1904 of file LoopVectorize.cpp.

◆ PSE

PredicatedScalarEvolution& llvm::LoopVectorizationCostModel::PSE

Predicated scalar evolution analysis.

Definition at line 1862 of file LoopVectorize.cpp.

◆ TheFunction

const Function* llvm::LoopVectorizationCostModel::TheFunction

Definition at line 1885 of file LoopVectorize.cpp.

◆ TheLoop

Loop* llvm::LoopVectorizationCostModel::TheLoop

The loop that we evaluate.

Definition at line 1859 of file LoopVectorize.cpp.

◆ TLI

const TargetLibraryInfo* llvm::LoopVectorizationCostModel::TLI

Target Library Info.

Definition at line 1874 of file LoopVectorize.cpp.

◆ TTI

const TargetTransformInfo& llvm::LoopVectorizationCostModel::TTI

Vector target information.

Definition at line 1871 of file LoopVectorize.cpp.

◆ ValuesToIgnore

SmallPtrSet<const Value *, 16> llvm::LoopVectorizationCostModel::ValuesToIgnore

Values to ignore in the cost model.

Definition at line 1895 of file LoopVectorize.cpp.

◆ VecValuesToIgnore

SmallPtrSet<const Value *, 16> llvm::LoopVectorizationCostModel::VecValuesToIgnore

Values to ignore in the cost model when VF > 1.

Definition at line 1898 of file LoopVectorize.cpp.

The documentation for this class was generated from the following file:

lib/Transforms/Vectorize/LoopVectorize.cpp

Classes

Public Types

Public Member Functions

Public Attributes

Detailed Description

Member Typedef Documentation

◆ ReductionChainMap

Member Enumeration Documentation

◆ InstWidening

Constructor & Destructor Documentation

◆ LoopVectorizationCostModel()

Member Function Documentation

◆ blockNeedsPredication()

◆ calculateRegisterUsage()

◆ canTruncateToMinimalBitwidth()

◆ canVectorizeReductions()

◆ collectElementTypesForWidening()

◆ collectInLoopReductions()

◆ collectInstsToScalarize()

◆ collectUniformsAndScalars()

◆ collectValuesToIgnore()

◆ computeMaxVF()

◆ foldTailByMasking()

◆ getInLoopReductionChains()

◆ getInterleavedAccessGroup()

◆ getMinimalBitwidths()

◆ getSmallestAndWidestTypes()

◆ getVectorCallCost()

◆ getVectorIntrinsicCost()

◆ getWideningCost()

◆ getWideningDecision()

◆ interleavedAccessCanBeWidened()

◆ invalidateCostModelingDecisions()

◆ isAccessInterleaved()

◆ isInLoopReduction()

◆ isLegalGatherOrScatter()

◆ isLegalMaskedLoad()

◆ isLegalMaskedStore()

◆ isMoreProfitable()

◆ isOptimizableIVTruncate()

◆ isPredicatedInst()

◆ isProfitableToScalarize()

◆ isScalarAfterVectorization()

◆ isScalarEpilogueAllowed()

◆ isScalarWithPredication()

◆ isUniformAfterVectorization()

◆ memoryInstructionCanBeWidened()

◆ requiresScalarEpilogue()

◆ runtimeChecksRequired()

◆ selectEpilogueVectorizationFactor()

◆ selectInterleaveCount()

◆ selectUserVectorizationFactor()

◆ selectVectorizationFactor()

◆ setCostBasedWideningDecision()

◆ setWideningDecision() [1/2]

◆ setWideningDecision() [2/2]

◆ useOrderedReductions()

Member Data Documentation

◆ AC

◆ DB

◆ ElementTypesInLoop

◆ Hints

◆ InterleaveInfo

◆ Legal

◆ LI

◆ ORE

◆ ProfitableVFs

◆ PSE

◆ TheFunction

◆ TheLoop

◆ TLI

◆ TTI

◆ ValuesToIgnore

◆ VecValuesToIgnore