LLVM 19.0.0git
Classes | Public Types | Public Member Functions | Public Attributes | List of all members
llvm::LoopVectorizationCostModel Class Reference

LoopVectorizationCostModel - estimates the expected speedups due to vectorization. More...

Classes

struct  CallWideningDecision
 
struct  RegisterUsage
 A struct that represents some properties of the register usage of a loop. More...
 

Public Types

enum  InstWidening {
  CM_Unknown , CM_Widen , CM_Widen_Reverse , CM_Interleave ,
  CM_GatherScatter , CM_Scalarize , CM_VectorCall , CM_IntrinsicCall
}
 Decision that was taken during cost calculation for memory instruction. More...
 
using VectorizationCostTy = std::pair< InstructionCost, bool >
 The vectorization cost is a combination of the cost itself and a boolean indicating whether any of the contributing operations will actually operate on vector values after type legalization in the backend.
 

Public Member Functions

 LoopVectorizationCostModel (ScalarEpilogueLowering SEL, Loop *L, PredicatedScalarEvolution &PSE, LoopInfo *LI, LoopVectorizationLegality *Legal, const TargetTransformInfo &TTI, const TargetLibraryInfo *TLI, DemandedBits *DB, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, const Function *F, const LoopVectorizeHints *Hints, InterleavedAccessInfo &IAI)
 
FixedScalableVFPair computeMaxVF (ElementCount UserVF, unsigned UserIC)
 
bool runtimeChecksRequired ()
 
bool selectUserVectorizationFactor (ElementCount UserVF)
 Setup cost-based decisions for user vectorization factor.
 
std::pair< unsigned, unsignedgetSmallestAndWidestTypes ()
 
unsigned selectInterleaveCount (ElementCount VF, InstructionCost LoopCost)
 
void setCostBasedWideningDecision (ElementCount VF)
 Memory access instruction may be vectorized in more than one way.
 
void setVectorizedCallDecision (ElementCount VF)
 A call may be vectorized in different ways depending on whether we have vectorized variants available and whether the target supports masking.
 
SmallVector< RegisterUsage, 8 > calculateRegisterUsage (ArrayRef< ElementCount > VFs)
 
void collectValuesToIgnore ()
 Collect values we want to ignore in the cost model.
 
void collectElementTypesForWidening ()
 Collect all element types in the loop for which widening is needed.
 
void collectInLoopReductions ()
 Split reductions into those that happen in the loop, and those that happen outside.
 
bool useOrderedReductions (const RecurrenceDescriptor &RdxDesc) const
 Returns true if we should use strict in-order reductions for the given RdxDesc.
 
const MapVector< Instruction *, uint64_t > & getMinimalBitwidths () const
 
bool isProfitableToScalarize (Instruction *I, ElementCount VF) const
 
bool isUniformAfterVectorization (Instruction *I, ElementCount VF) const
 Returns true if I is known to be uniform after vectorization.
 
bool isScalarAfterVectorization (Instruction *I, ElementCount VF) const
 Returns true if I is known to be scalar after vectorization.
 
bool canTruncateToMinimalBitwidth (Instruction *I, ElementCount VF) const
 
void setWideningDecision (Instruction *I, ElementCount VF, InstWidening W, InstructionCost Cost)
 Save vectorization decision W and Cost taken by the cost model for instruction I and vector width VF.
 
void setWideningDecision (const InterleaveGroup< Instruction > *Grp, ElementCount VF, InstWidening W, InstructionCost Cost)
 Save vectorization decision W and Cost taken by the cost model for interleaving group Grp and vector width VF.
 
InstWidening getWideningDecision (Instruction *I, ElementCount VF) const
 Return the cost model decision for the given instruction I and vector width VF.
 
InstructionCost getWideningCost (Instruction *I, ElementCount VF)
 Return the vectorization cost for the given instruction I and vector width VF.
 
void setCallWideningDecision (CallInst *CI, ElementCount VF, InstWidening Kind, Function *Variant, Intrinsic::ID IID, std::optional< unsigned > MaskPos, InstructionCost Cost)
 
CallWideningDecision getCallWideningDecision (CallInst *CI, ElementCount VF) const
 
bool isOptimizableIVTruncate (Instruction *I, ElementCount VF)
 Return True if instruction I is an optimizable truncate whose operand is an induction variable.
 
void collectInstsToScalarize (ElementCount VF)
 Collects the instructions to scalarize for each predicated instruction in the loop.
 
void collectUniformsAndScalars (ElementCount VF)
 Collect Uniform and Scalar values for the given VF.
 
bool isLegalMaskedStore (Type *DataType, Value *Ptr, Align Alignment) const
 Returns true if the target machine supports masked store operation for the given DataType and kind of access to Ptr.
 
bool isLegalMaskedLoad (Type *DataType, Value *Ptr, Align Alignment) const
 Returns true if the target machine supports masked load operation for the given DataType and kind of access to Ptr.
 
bool isLegalGatherOrScatter (Value *V, ElementCount VF)
 Returns true if the target machine can represent V as a masked gather or scatter operation.
 
bool canVectorizeReductions (ElementCount VF) const
 Returns true if the target machine supports all of the reduction variables found for the given VF.
 
bool isDivRemScalarWithPredication (InstructionCost ScalarCost, InstructionCost SafeDivisorCost) const
 Given costs for both strategies, return true if the scalar predication lowering should be used for div/rem.
 
bool isScalarWithPredication (Instruction *I, ElementCount VF) const
 Returns true if I is an instruction which requires predication and for which our chosen predication strategy is scalarization (i.e.
 
bool isPredicatedInst (Instruction *I) const
 Returns true if I is an instruction that needs to be predicated at runtime.
 
std::pair< InstructionCost, InstructionCostgetDivRemSpeculationCost (Instruction *I, ElementCount VF) const
 Return the costs for our two available strategies for lowering a div/rem operation which requires speculating at least one lane.
 
bool memoryInstructionCanBeWidened (Instruction *I, ElementCount VF)
 Returns true if I is a memory instruction with consecutive memory access that can be widened.
 
bool interleavedAccessCanBeWidened (Instruction *I, ElementCount VF)
 Returns true if I is a memory instruction in an interleaved-group of memory accesses that can be vectorized with wide vector loads/stores and shuffles.
 
bool isAccessInterleaved (Instruction *Instr)
 Check if Instr belongs to any interleaved access group.
 
const InterleaveGroup< Instruction > * getInterleavedAccessGroup (Instruction *Instr)
 Get the interleaved access group that Instr belongs to.
 
bool requiresScalarEpilogue (bool IsVectorizing) const
 Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop.
 
bool requiresScalarEpilogue (VFRange Range) const
 Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop for all VFs in Range.
 
bool isScalarEpilogueAllowed () const
 Returns true if a scalar epilogue is not allowed due to optsize or a loop hint annotation.
 
TailFoldingStyle getTailFoldingStyle (bool IVUpdateMayOverflow=true) const
 Returns the TailFoldingStyle that is best for the current loop.
 
void setTailFoldingStyles (bool IsScalableVF, unsigned UserIC)
 Selects and saves TailFoldingStyle for 2 options - if IV update may overflow or not.
 
bool foldTailByMasking () const
 Returns true if all loop blocks should be masked to fold tail loop.
 
bool blockNeedsPredicationForAnyReason (BasicBlock *BB) const
 Returns true if the instructions in this block requires predication for any reason, e.g.
 
bool foldTailWithEVL () const
 Returns true if VP intrinsics with explicit vector length support should be generated in the tail folded loop.
 
bool isInLoopReduction (PHINode *Phi) const
 Returns true if the Phi is part of an inloop reduction.
 
InstructionCost getVectorIntrinsicCost (CallInst *CI, ElementCount VF) const
 Estimate cost of an intrinsic call instruction CI if it were vectorized with factor VF.
 
InstructionCost getVectorCallCost (CallInst *CI, ElementCount VF) const
 Estimate cost of a call instruction CI if it were vectorized with factor VF.
 
void invalidateCostModelingDecisions ()
 Invalidates decisions already taken by the cost model.
 
VectorizationCostTy expectedCost (ElementCount VF, SmallVectorImpl< InstructionVFPair > *Invalid=nullptr)
 Returns the expected execution cost.
 
bool hasPredStores () const
 
bool isEpilogueVectorizationProfitable (const ElementCount VF) const
 Returns true if epilogue vectorization is considered profitable, and false otherwise.
 

Public Attributes

LoopTheLoop
 The loop that we evaluate.
 
PredicatedScalarEvolutionPSE
 Predicated scalar evolution analysis.
 
LoopInfoLI
 Loop Info analysis.
 
LoopVectorizationLegalityLegal
 Vectorization legality.
 
const TargetTransformInfoTTI
 Vector target information.
 
const TargetLibraryInfoTLI
 Target Library Info.
 
DemandedBitsDB
 Demanded bits analysis.
 
AssumptionCacheAC
 Assumption cache.
 
OptimizationRemarkEmitterORE
 Interface to emit optimization remarks.
 
const FunctionTheFunction
 
const LoopVectorizeHintsHints
 Loop Vectorize Hint.
 
InterleavedAccessInfoInterleaveInfo
 The interleave access information contains groups of interleaved accesses with the same stride and close to each other.
 
SmallPtrSet< const Value *, 16 > ValuesToIgnore
 Values to ignore in the cost model.
 
SmallPtrSet< const Value *, 16 > VecValuesToIgnore
 Values to ignore in the cost model when VF > 1.
 
SmallPtrSet< Type *, 16 > ElementTypesInLoop
 All element types found in the loop.
 

Detailed Description

LoopVectorizationCostModel - estimates the expected speedups due to vectorization.

In many cases vectorization is not profitable. This can happen because of a number of reasons. In this class we mainly attempt to predict the expected speedup/slowdowns due to the supported instruction set. We use the TargetTransformInfo to query the different backends for the cost of different operations.

Definition at line 1081 of file LoopVectorize.cpp.

Member Typedef Documentation

◆ VectorizationCostTy

The vectorization cost is a combination of the cost itself and a boolean indicating whether any of the contributing operations will actually operate on vector values after type legalization in the backend.

If this latter value is false, then all operations will be scalarized (i.e. no vectorization has actually taken place).

Definition at line 1605 of file LoopVectorize.cpp.

Member Enumeration Documentation

◆ InstWidening

Decision that was taken during cost calculation for memory instruction.

Enumerator
CM_Unknown 
CM_Widen 
CM_Widen_Reverse 
CM_Interleave 
CM_GatherScatter 
CM_Scalarize 
CM_VectorCall 
CM_IntrinsicCall 

Definition at line 1239 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ LoopVectorizationCostModel()

llvm::LoopVectorizationCostModel::LoopVectorizationCostModel ( ScalarEpilogueLowering  SEL,
Loop L,
PredicatedScalarEvolution PSE,
LoopInfo LI,
LoopVectorizationLegality Legal,
const TargetTransformInfo TTI,
const TargetLibraryInfo TLI,
DemandedBits DB,
AssumptionCache AC,
OptimizationRemarkEmitter ORE,
const Function F,
const LoopVectorizeHints Hints,
InterleavedAccessInfo IAI 
)
inline

Definition at line 1083 of file LoopVectorize.cpp.

Member Function Documentation

◆ blockNeedsPredicationForAnyReason()

bool llvm::LoopVectorizationCostModel::blockNeedsPredicationForAnyReason ( BasicBlock BB) const
inline

Returns true if the instructions in this block requires predication for any reason, e.g.

because tail folding now requires a predicate or because the block in the original loop was predicated.

Definition at line 1567 of file LoopVectorize.cpp.

References llvm::LoopVectorizationLegality::blockNeedsPredication(), foldTailByMasking(), and Legal.

Referenced by collectInstsToScalarize(), interleavedAccessCanBeWidened(), isPredicatedInst(), and llvm::LoopVectorizationPlanner::plan().

◆ calculateRegisterUsage()

SmallVector< LoopVectorizationCostModel::RegisterUsage, 8 > LoopVectorizationCostModel::calculateRegisterUsage ( ArrayRef< ElementCount VFs)

◆ canTruncateToMinimalBitwidth()

bool llvm::LoopVectorizationCostModel::canTruncateToMinimalBitwidth ( Instruction I,
ElementCount  VF 
) const
inline
Returns
True if instruction I can be truncated to a smaller bitwidth for vectorization factor VF.

Definition at line 1232 of file LoopVectorize.cpp.

References I, isProfitableToScalarize(), isScalarAfterVectorization(), and llvm::ElementCount::isVector().

◆ canVectorizeReductions()

bool llvm::LoopVectorizationCostModel::canVectorizeReductions ( ElementCount  VF) const
inline

Returns true if the target machine supports all of the reduction variables found for the given VF.

Definition at line 1402 of file LoopVectorize.cpp.

References llvm::all_of(), llvm::LoopVectorizationLegality::getReductionVars(), Legal, and Reduction.

◆ collectElementTypesForWidening()

void LoopVectorizationCostModel::collectElementTypesForWidening ( )

◆ collectInLoopReductions()

void LoopVectorizationCostModel::collectInLoopReductions ( )

◆ collectInstsToScalarize()

void LoopVectorizationCostModel::collectInstsToScalarize ( ElementCount  VF)

◆ collectUniformsAndScalars()

void llvm::LoopVectorizationCostModel::collectUniformsAndScalars ( ElementCount  VF)
inline

Collect Uniform and Scalar values for the given VF.

The sets depend on CM decision for Load/Store instructions that may be vectorized as interleave, gather-scatter or scalarized. Also make a decision on what to do about call instructions in the loop at that VF – scalarize, call a known vector routine, or call a vector intrinsic.

Definition at line 1361 of file LoopVectorize.cpp.

References llvm::ElementCount::isScalar(), setCostBasedWideningDecision(), and setVectorizedCallDecision().

Referenced by calculateRegisterUsage(), llvm::LoopVectorizationPlanner::plan(), and selectUserVectorizationFactor().

◆ collectValuesToIgnore()

void LoopVectorizationCostModel::collectValuesToIgnore ( )

◆ computeMaxVF()

FixedScalableVFPair LoopVectorizationCostModel::computeMaxVF ( ElementCount  UserVF,
unsigned  UserIC 
)
Returns
An upper bound for the vectorization factors (both fixed and scalable). If the factors are 0, vectorization and interleaving should be avoided up front.

Definition at line 4567 of file LoopVectorize.cpp.

References llvm::ScalarEvolution::applyLoopGuards(), assert(), llvm::CM_ScalarEpilogueAllowed, llvm::CM_ScalarEpilogueNotAllowedLowTripLoop, llvm::CM_ScalarEpilogueNotAllowedOptSize, llvm::CM_ScalarEpilogueNotAllowedUsePredicate, llvm::CM_ScalarEpilogueNotNeededUsePredicate, llvm::DataWithEVL, llvm::dbgs(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::empty(), llvm::FixedScalableVFPair::FixedVF, foldTailByMasking(), llvm::ScalarEvolution::getAddExpr(), llvm::PredicatedScalarEvolution::getBackedgeTakenCount(), llvm::ScalarEvolution::getConstant(), llvm::LoopBase< BlockT, LoopT >::getExitingBlock(), llvm::ElementCount::getFixed(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getFixedValue(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), getMaxVScale(), llvm::FixedScalableVFPair::getNone(), llvm::ScalarEvolution::getOne(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getSE(), llvm::ScalarEvolution::getSmallConstantMaxTripCount(), llvm::ScalarEvolution::getSmallConstantTripCount(), getTailFoldingStyle(), llvm::SCEV::getType(), llvm::ScalarEvolution::getURemExpr(), llvm::TargetTransformInfo::hasBranchDivergence(), InterleaveInfo, llvm::InterleavedAccessInfo::invalidateGroupsRequiringScalarEpilogue(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isNonZero(), llvm::isPowerOf2_32(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::TargetTransformInfo::isVScaleKnownToBeAPowerOfTwo(), llvm::SCEV::isZero(), Legal, LLVM_DEBUG, llvm::RuntimePointerChecking::Need, ORE, PSE, llvm::reportVectorizationFailure(), runtimeChecksRequired(), llvm::FixedScalableVFPair::ScalableVF, setTailFoldingStyles(), TheFunction, TheLoop, and useMaskedInterleavedAccesses().

Referenced by llvm::LoopVectorizationPlanner::plan().

◆ expectedCost()

LoopVectorizationCostModel::VectorizationCostTy LoopVectorizationCostModel::expectedCost ( ElementCount  VF,
SmallVectorImpl< InstructionVFPair > *  Invalid = nullptr 
)

Returns the expected execution cost.

The unit of the cost does not matter because we use the 'cost' units to compare different vector widths. The cost that is returned is not normalized by the factor width. If Invalid is not nullptr, this function will add a pair(Instruction*, ElementCount) to Invalid for each instruction that has an Invalid cost for the given VF.

Definition at line 5950 of file LoopVectorize.cpp.

References llvm::LoopVectorizationLegality::blockNeedsPredication(), llvm::LoopBase< BlockT, LoopT >::blocks(), llvm::CallingConv::C, llvm::dbgs(), ForceTargetInstructionCost, getReciprocalPredBlockProb(), I, llvm::Invalid, llvm::ElementCount::isScalar(), llvm::ElementCount::isVector(), Legal, LLVM_DEBUG, TheLoop, ValuesToIgnore, and VecValuesToIgnore.

Referenced by selectInterleaveCount(), and selectUserVectorizationFactor().

◆ foldTailByMasking()

bool llvm::LoopVectorizationCostModel::foldTailByMasking ( ) const
inline

Returns true if all loop blocks should be masked to fold tail loop.

Definition at line 1558 of file LoopVectorize.cpp.

References getTailFoldingStyle(), and llvm::None.

Referenced by blockNeedsPredicationForAnyReason(), computeMaxVF(), llvm::VPRecipeBuilder::createHeaderMask(), and setCostBasedWideningDecision().

◆ foldTailWithEVL()

bool llvm::LoopVectorizationCostModel::foldTailWithEVL ( ) const
inline

Returns true if VP intrinsics with explicit vector length support should be generated in the tail folded loop.

Definition at line 1573 of file LoopVectorize.cpp.

References llvm::DataWithEVL, and getTailFoldingStyle().

Referenced by selectInterleaveCount().

◆ getCallWideningDecision()

CallWideningDecision llvm::LoopVectorizationCostModel::getCallWideningDecision ( CallInst CI,
ElementCount  VF 
) const
inline

◆ getDivRemSpeculationCost()

std::pair< InstructionCost, InstructionCost > LoopVectorizationCostModel::getDivRemSpeculationCost ( Instruction I,
ElementCount  VF 
) const

◆ getInterleavedAccessGroup()

const InterleaveGroup< Instruction > * llvm::LoopVectorizationCostModel::getInterleavedAccessGroup ( Instruction Instr)
inline

Get the interleaved access group that Instr belongs to.

Definition at line 1460 of file LoopVectorize.cpp.

References llvm::InterleavedAccessInfo::getInterleaveGroup(), and InterleaveInfo.

Referenced by interleavedAccessCanBeWidened(), and setCostBasedWideningDecision().

◆ getMinimalBitwidths()

const MapVector< Instruction *, uint64_t > & llvm::LoopVectorizationCostModel::getMinimalBitwidths ( ) const
inline
Returns
The smallest bitwidth each instruction can be represented with. The vector equivalents of these instructions should be truncated to this type.

Definition at line 1177 of file LoopVectorize.cpp.

◆ getSmallestAndWidestTypes()

std::pair< unsigned, unsigned > LoopVectorizationCostModel::getSmallestAndWidestTypes ( )

◆ getTailFoldingStyle()

TailFoldingStyle llvm::LoopVectorizationCostModel::getTailFoldingStyle ( bool  IVUpdateMayOverflow = true) const
inline

Returns the TailFoldingStyle that is best for the current loop.

Definition at line 1498 of file LoopVectorize.cpp.

References llvm::None.

Referenced by computeMaxVF(), foldTailByMasking(), and foldTailWithEVL().

◆ getVectorCallCost()

InstructionCost LoopVectorizationCostModel::getVectorCallCost ( CallInst CI,
ElementCount  VF 
) const

◆ getVectorIntrinsicCost()

InstructionCost LoopVectorizationCostModel::getVectorIntrinsicCost ( CallInst CI,
ElementCount  VF 
) const

◆ getWideningCost()

InstructionCost llvm::LoopVectorizationCostModel::getWideningCost ( Instruction I,
ElementCount  VF 
)
inline

Return the vectorization cost for the given instruction I and vector width VF.

Definition at line 1294 of file LoopVectorize.cpp.

References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains(), I, and llvm::ElementCount::isVector().

◆ getWideningDecision()

InstWidening llvm::LoopVectorizationCostModel::getWideningDecision ( Instruction I,
ElementCount  VF 
) const
inline

Return the cost model decision for the given instruction I and vector width VF.

Return CM_Unknown if this instruction did not pass through the cost modeling.

Definition at line 1279 of file LoopVectorize.cpp.

References assert(), CM_Unknown, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::find(), I, llvm::LoopBase< BlockT, LoopT >::isInnermost(), llvm::ElementCount::isVector(), and TheLoop.

Referenced by interleavedAccessCanBeWidened(), and setCostBasedWideningDecision().

◆ hasPredStores()

bool llvm::LoopVectorizationCostModel::hasPredStores ( ) const
inline

Definition at line 1617 of file LoopVectorize.cpp.

◆ interleavedAccessCanBeWidened()

bool LoopVectorizationCostModel::interleavedAccessCanBeWidened ( Instruction I,
ElementCount  VF 
)

◆ invalidateCostModelingDecisions()

void llvm::LoopVectorizationCostModel::invalidateCostModelingDecisions ( )
inline

Invalidates decisions already taken by the cost model.

Definition at line 1593 of file LoopVectorize.cpp.

References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::clear().

Referenced by llvm::LoopVectorizationPlanner::plan().

◆ isAccessInterleaved()

bool llvm::LoopVectorizationCostModel::isAccessInterleaved ( Instruction Instr)
inline

Check if Instr belongs to any interleaved access group.

Definition at line 1454 of file LoopVectorize.cpp.

References InterleaveInfo, and llvm::InterleavedAccessInfo::isInterleaved().

Referenced by interleavedAccessCanBeWidened(), and setCostBasedWideningDecision().

◆ isDivRemScalarWithPredication()

bool llvm::LoopVectorizationCostModel::isDivRemScalarWithPredication ( InstructionCost  ScalarCost,
InstructionCost  SafeDivisorCost 
) const
inline

Given costs for both strategies, return true if the scalar predication lowering should be used for div/rem.

This incorporates an override option so it is not simply a cost comparison.

Definition at line 1412 of file LoopVectorize.cpp.

References llvm::cl::BOU_FALSE, llvm::cl::BOU_TRUE, llvm::cl::BOU_UNSET, ForceSafeDivisor, and llvm_unreachable.

Referenced by isScalarWithPredication().

◆ isEpilogueVectorizationProfitable()

bool LoopVectorizationCostModel::isEpilogueVectorizationProfitable ( const ElementCount  VF) const

◆ isInLoopReduction()

bool llvm::LoopVectorizationCostModel::isInLoopReduction ( PHINode Phi) const
inline

Returns true if the Phi is part of an inloop reduction.

Definition at line 1578 of file LoopVectorize.cpp.

Referenced by llvm::VPRecipeBuilder::tryToCreateWidenRecipe().

◆ isLegalGatherOrScatter()

bool llvm::LoopVectorizationCostModel::isLegalGatherOrScatter ( Value V,
ElementCount  VF 
)
inline

◆ isLegalMaskedLoad()

bool llvm::LoopVectorizationCostModel::isLegalMaskedLoad ( Type DataType,
Value Ptr,
Align  Alignment 
) const
inline

Returns true if the target machine supports masked load operation for the given DataType and kind of access to Ptr.

Definition at line 1380 of file LoopVectorize.cpp.

References llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::TargetTransformInfo::isLegalMaskedLoad(), Legal, and Ptr.

Referenced by isScalarWithPredication().

◆ isLegalMaskedStore()

bool llvm::LoopVectorizationCostModel::isLegalMaskedStore ( Type DataType,
Value Ptr,
Align  Alignment 
) const
inline

Returns true if the target machine supports masked store operation for the given DataType and kind of access to Ptr.

Definition at line 1373 of file LoopVectorize.cpp.

References llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::TargetTransformInfo::isLegalMaskedStore(), Legal, and Ptr.

Referenced by isScalarWithPredication().

◆ isOptimizableIVTruncate()

bool llvm::LoopVectorizationCostModel::isOptimizableIVTruncate ( Instruction I,
ElementCount  VF 
)
inline

Return True if instruction I is an optimizable truncate whose operand is an induction variable.

Such a truncate will be removed by adding a new induction variable with the destination type.

Definition at line 1328 of file LoopVectorize.cpp.

References llvm::LoopVectorizationLegality::getPrimaryInduction(), I, llvm::LoopVectorizationLegality::isInductionPhi(), llvm::TargetTransformInfo::isTruncateFree(), Legal, and llvm::ToVectorTy().

◆ isPredicatedInst()

bool LoopVectorizationCostModel::isPredicatedInst ( Instruction I) const

Returns true if I is an instruction that needs to be predicated at runtime.

The result is independent of the predication mechanism. Superset of instructions that return true for isScalarWithPredication.

Definition at line 3933 of file LoopVectorize.cpp.

References llvm::LoopVectorizationLegality::blockNeedsPredication(), blockNeedsPredicationForAnyReason(), llvm::getLoadStorePointerOperand(), I, llvm::LoopVectorizationLegality::isInvariant(), llvm::Loop::isLoopInvariant(), llvm::LoopVectorizationLegality::isMaskRequired(), llvm::isSafeToSpeculativelyExecute(), Legal, and TheLoop.

Referenced by llvm::VPRecipeBuilder::handleReplication(), and isScalarWithPredication().

◆ isProfitableToScalarize()

bool llvm::LoopVectorizationCostModel::isProfitableToScalarize ( Instruction I,
ElementCount  VF 
) const
inline
Returns
True if it is more profitable to scalarize instruction I for vectorization factor VF.

Definition at line 1183 of file LoopVectorize.cpp.

References assert(), I, llvm::LoopBase< BlockT, LoopT >::isInnermost(), llvm::ElementCount::isVector(), and TheLoop.

Referenced by canTruncateToMinimalBitwidth().

◆ isScalarAfterVectorization()

bool llvm::LoopVectorizationCostModel::isScalarAfterVectorization ( Instruction I,
ElementCount  VF 
) const
inline

Returns true if I is known to be scalar after vectorization.

Definition at line 1217 of file LoopVectorize.cpp.

References assert(), I, llvm::LoopBase< BlockT, LoopT >::isInnermost(), llvm::ElementCount::isScalar(), and TheLoop.

Referenced by calculateRegisterUsage(), canTruncateToMinimalBitwidth(), and collectInstsToScalarize().

◆ isScalarEpilogueAllowed()

bool llvm::LoopVectorizationCostModel::isScalarEpilogueAllowed ( ) const
inline

Returns true if a scalar epilogue is not allowed due to optsize or a loop hint annotation.

Definition at line 1493 of file LoopVectorize.cpp.

References llvm::CM_ScalarEpilogueAllowed.

Referenced by interleavedAccessCanBeWidened(), requiresScalarEpilogue(), llvm::LoopVectorizationPlanner::selectEpilogueVectorizationFactor(), and selectInterleaveCount().

◆ isScalarWithPredication()

bool LoopVectorizationCostModel::isScalarWithPredication ( Instruction I,
ElementCount  VF 
) const

Returns true if I is an instruction which requires predication and for which our chosen predication strategy is scalarization (i.e.

we don't have an alternate strategy such as masking available). VF is the vectorization factor that will be used to vectorize I.

Definition at line 3892 of file LoopVectorize.cpp.

References CM_Scalarize, llvm::VectorType::get(), getDivRemSpeculationCost(), llvm::getLoadStoreAlignment(), llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), I, isDivRemScalarWithPredication(), llvm::TargetTransformInfo::isLegalMaskedGather(), isLegalMaskedLoad(), llvm::TargetTransformInfo::isLegalMaskedScatter(), isLegalMaskedStore(), isPredicatedInst(), llvm::ElementCount::isScalar(), llvm::ElementCount::isVector(), and Ptr.

Referenced by collectInstsToScalarize(), memoryInstructionCanBeWidened(), and setCostBasedWideningDecision().

◆ isUniformAfterVectorization()

bool llvm::LoopVectorizationCostModel::isUniformAfterVectorization ( Instruction I,
ElementCount  VF 
) const
inline

Returns true if I is known to be uniform after vectorization.

Definition at line 1197 of file LoopVectorize.cpp.

References assert(), I, llvm::LoopBase< BlockT, LoopT >::isInnermost(), llvm::ElementCount::isScalar(), and TheLoop.

Referenced by llvm::VPRecipeBuilder::handleReplication().

◆ memoryInstructionCanBeWidened()

bool LoopVectorizationCostModel::memoryInstructionCanBeWidened ( Instruction I,
ElementCount  VF 
)

Returns true if I is a memory instruction with consecutive memory access that can be widened.

Definition at line 4110 of file LoopVectorize.cpp.

References assert(), DL, llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), hasIrregularType(), I, llvm::LoopVectorizationLegality::isConsecutivePtr(), isScalarWithPredication(), Legal, and Ptr.

Referenced by setCostBasedWideningDecision().

◆ requiresScalarEpilogue() [1/2]

bool llvm::LoopVectorizationCostModel::requiresScalarEpilogue ( bool  IsVectorizing) const
inline

Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop.

Definition at line 1466 of file LoopVectorize.cpp.

References llvm::LoopBase< BlockT, LoopT >::getExitingBlock(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), InterleaveInfo, isScalarEpilogueAllowed(), llvm::InterleavedAccessInfo::requiresScalarEpilogue(), and TheLoop.

Referenced by requiresScalarEpilogue(), and selectInterleaveCount().

◆ requiresScalarEpilogue() [2/2]

bool llvm::LoopVectorizationCostModel::requiresScalarEpilogue ( VFRange  Range) const
inline

Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop for all VFs in Range.

A scalar epilogue must either be required for all VFs in Range or for none.

Definition at line 1480 of file LoopVectorize.cpp.

References llvm::all_of(), assert(), llvm::none_of(), and requiresScalarEpilogue().

◆ runtimeChecksRequired()

bool LoopVectorizationCostModel::runtimeChecksRequired ( )

◆ selectInterleaveCount()

unsigned LoopVectorizationCostModel::selectInterleaveCount ( ElementCount  VF,
InstructionCost  LoopCost 
)
Returns
The desired interleave count. If interleave count has been specified by metadata it will be returned. Otherwise, the interleave count is computed and returned. VF and LoopCost are the selected vectorization factor and the cost of the selected VF.

Definition at line 5290 of file LoopVectorize.cpp.

References llvm::any_of(), assert(), llvm::bit_floor(), llvm::LoopBase< BlockT, LoopT >::blocks(), calculateRegisterUsage(), llvm::dbgs(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::empty(), llvm::TargetTransformInfo::enableAggressiveInterleaving(), EnableIndVarRegisterHeur, EnableLoadStoreRuntimeInterleave, expectedCost(), F, foldTailWithEVL(), ForceTargetMaxScalarInterleaveFactor, ForceTargetMaxVectorInterleaveFactor, ForceTargetNumScalarRegs, ForceTargetNumVectorRegs, llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::LoopBase< BlockT, LoopT >::getLoopDepth(), llvm::TargetTransformInfo::getMaxInterleaveFactor(), llvm::TargetTransformInfo::getNumberOfRegisters(), llvm::LoopVectorizationLegality::getNumLoads(), llvm::LoopVectorizationLegality::getNumStores(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::TargetTransformInfo::getRegisterClassName(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getSE(), getSmallBestKnownTC(), llvm::ScalarEvolution::getSmallConstantTripCount(), llvm::InstructionCost::getValue(), getVScaleForTuning(), llvm::LoopVectorizationLegality::isSafeForAnyVectorWidth(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), isScalarEpilogueAllowed(), llvm::InstructionCost::isValid(), llvm::ElementCount::isVector(), Legal, LLVM_DEBUG, MaxNestedScalarReductionIC, llvm::RuntimePointerChecking::Need, PSE, Reduction, requiresScalarEpilogue(), SmallLoopCost, and TheLoop.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ selectUserVectorizationFactor()

bool llvm::LoopVectorizationCostModel::selectUserVectorizationFactor ( ElementCount  UserVF)
inline

Setup cost-based decisions for user vectorization factor.

Returns
true if the UserVF is a feasible VF to be chosen.

Definition at line 1107 of file LoopVectorize.cpp.

References collectInstsToScalarize(), collectUniformsAndScalars(), and expectedCost().

Referenced by llvm::LoopVectorizationPlanner::plan().

◆ setCallWideningDecision()

void llvm::LoopVectorizationCostModel::setCallWideningDecision ( CallInst CI,
ElementCount  VF,
InstWidening  Kind,
Function Variant,
Intrinsic::ID  IID,
std::optional< unsigned MaskPos,
InstructionCost  Cost 
)
inline

Definition at line 1310 of file LoopVectorize.cpp.

References assert(), and llvm::ElementCount::isScalar().

Referenced by setVectorizedCallDecision().

◆ setCostBasedWideningDecision()

void LoopVectorizationCostModel::setCostBasedWideningDecision ( ElementCount  VF)

Memory access instruction may be vectorized in more than one way.

Form of instruction after vectorization depends on cost. This function takes cost-based decisions for Load/Store instructions and collects them in a map. This decisions map is used for building the lists of loop-uniform and loop-scalar instructions. The calculated cost is saved with widening decision in order to avoid redundant calculations.

Definition at line 6477 of file LoopVectorize.cpp.

References llvm::append_range(), assert(), llvm::LoopBase< BlockT, LoopT >::blocks(), CM_GatherScatter, CM_Interleave, CM_Scalarize, CM_Unknown, CM_Widen, CM_Widen_Reverse, llvm::LoopBase< BlockT, LoopT >::contains(), llvm::SmallVectorBase< Size_T >::empty(), foldTailByMasking(), llvm::ElementCount::getFixed(), getInterleavedAccessGroup(), llvm::InstructionCost::getInvalid(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), getWideningDecision(), I, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::insert(), llvm::SmallPtrSetImpl< PtrType >::insert(), interleavedAccessCanBeWidened(), isAccessInterleaved(), llvm::LoopVectorizationLegality::isConsecutivePtr(), isLegalGatherOrScatter(), llvm::Loop::isLoopInvariant(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), isScalarWithPredication(), llvm::LoopVectorizationLegality::isUniformMemOp(), Legal, memoryInstructionCanBeWidened(), llvm::SmallVectorImpl< T >::pop_back_val(), llvm::TargetTransformInfo::prefersVectorizedAddressing(), Ptr, llvm::SmallVectorTemplateBase< T, bool >::push_back(), setWideningDecision(), and TheLoop.

Referenced by collectUniformsAndScalars().

◆ setTailFoldingStyles()

void llvm::LoopVectorizationCostModel::setTailFoldingStyles ( bool  IsScalableVF,
unsigned  UserIC 
)
inline

◆ setVectorizedCallDecision()

void LoopVectorizationCostModel::setVectorizedCallDecision ( ElementCount  VF)

A call may be vectorized in different ways depending on whether we have vectorized variants available and whether the target supports masking.

This function analyzes all calls in the function at the supplied VF, makes a decision based on the costs of available options, and stores that decision in a map for use in planning and plan execution.

Definition at line 6663 of file LoopVectorize.cpp.

References llvm::CallBase::args(), assert(), llvm::LoopBase< BlockT, LoopT >::blocks(), CM_IntrinsicCall, CM_Scalarize, CM_VectorCall, CostKind, llvm::VectorType::get(), llvm::SCEVConstant::getAPInt(), llvm::CallBase::getArgOperand(), llvm::CallBase::getCalledFunction(), llvm::TargetTransformInfo::getCallInstrCost(), llvm::Type::getContext(), llvm::Module::getFunction(), llvm::Function::getFunctionType(), llvm::Type::getInt1Ty(), llvm::InstructionCost::getInvalid(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::VFDatabase::getMappings(), llvm::Instruction::getModule(), llvm::VFInfo::getParamIndexForOptionalMask(), llvm::ScalarEvolution::getSCEV(), llvm::PredicatedScalarEvolution::getSCEV(), llvm::PredicatedScalarEvolution::getSE(), llvm::APInt::getSExtValue(), llvm::TargetTransformInfo::getShuffleCost(), llvm::Value::getType(), getVectorIntrinsicCost(), llvm::getVectorIntrinsicIDForCall(), llvm::GlobalPredicate, I, Info, llvm::RecurrenceDescriptor::isFMulAddIntrinsic(), llvm::ScalarEvolution::isLoopInvariant(), llvm::LoopVectorizationLegality::isMaskRequired(), llvm::CallBase::isNoBuiltin(), llvm::ElementCount::isScalar(), Legal, llvm::Intrinsic::not_intrinsic, llvm::OMP_Linear, llvm::OMP_Uniform, PSE, llvm::SmallVectorTemplateBase< T, bool >::push_back(), RetTy, setCallWideningDecision(), llvm::TargetTransformInfo::SK_Broadcast, llvm::TargetTransformInfo::TCK_RecipThroughput, TheLoop, TLI, llvm::ToVectorTy(), and llvm::Vector.

Referenced by collectUniformsAndScalars().

◆ setWideningDecision() [1/2]

void llvm::LoopVectorizationCostModel::setWideningDecision ( const InterleaveGroup< Instruction > *  Grp,
ElementCount  VF,
InstWidening  W,
InstructionCost  Cost 
)
inline

Save vectorization decision W and Cost taken by the cost model for interleaving group Grp and vector width VF.

Broadcast this decicion to all instructions inside the group. But the cost will be assigned to one instruction only.

Definition at line 1260 of file LoopVectorize.cpp.

References assert(), llvm::InterleaveGroup< InstTy >::getFactor(), llvm::InterleaveGroup< InstTy >::getInsertPos(), llvm::InterleaveGroup< InstTy >::getMember(), I, and llvm::ElementCount::isVector().

◆ setWideningDecision() [2/2]

void llvm::LoopVectorizationCostModel::setWideningDecision ( Instruction I,
ElementCount  VF,
InstWidening  W,
InstructionCost  Cost 
)
inline

Save vectorization decision W and Cost taken by the cost model for instruction I and vector width VF.

Definition at line 1252 of file LoopVectorize.cpp.

References assert(), I, and llvm::ElementCount::isVector().

Referenced by setCostBasedWideningDecision().

◆ useOrderedReductions()

bool llvm::LoopVectorizationCostModel::useOrderedReductions ( const RecurrenceDescriptor RdxDesc) const
inline

Returns true if we should use strict in-order reductions for the given RdxDesc.

This is true if the -enable-strict-reductions flag is passed, the IsOrdered flag of RdxDesc is set and we do not allow reordering of FP operations.

Definition at line 1170 of file LoopVectorize.cpp.

References llvm::LoopVectorizeHints::allowReordering(), Hints, and llvm::RecurrenceDescriptor::isOrdered().

Referenced by collectElementTypesForWidening(), collectInLoopReductions(), and llvm::VPRecipeBuilder::tryToCreateWidenRecipe().

Member Data Documentation

◆ AC

AssumptionCache* llvm::LoopVectorizationCostModel::AC

Assumption cache.

Definition at line 1838 of file LoopVectorize.cpp.

Referenced by collectValuesToIgnore().

◆ DB

DemandedBits* llvm::LoopVectorizationCostModel::DB

Demanded bits analysis.

Definition at line 1835 of file LoopVectorize.cpp.

◆ ElementTypesInLoop

SmallPtrSet<Type *, 16> llvm::LoopVectorizationCostModel::ElementTypesInLoop

All element types found in the loop.

Definition at line 1859 of file LoopVectorize.cpp.

Referenced by collectElementTypesForWidening(), and getSmallestAndWidestTypes().

◆ Hints

const LoopVectorizeHints* llvm::LoopVectorizationCostModel::Hints

Loop Vectorize Hint.

Definition at line 1846 of file LoopVectorize.cpp.

Referenced by useOrderedReductions().

◆ InterleaveInfo

InterleavedAccessInfo& llvm::LoopVectorizationCostModel::InterleaveInfo

The interleave access information contains groups of interleaved accesses with the same stride and close to each other.

Definition at line 1850 of file LoopVectorize.cpp.

Referenced by computeMaxVF(), getInterleavedAccessGroup(), isAccessInterleaved(), llvm::LoopVectorizationPlanner::plan(), and requiresScalarEpilogue().

◆ Legal

LoopVectorizationLegality* llvm::LoopVectorizationCostModel::Legal

◆ LI

LoopInfo* llvm::LoopVectorizationCostModel::LI

Loop Info analysis.

Definition at line 1823 of file LoopVectorize.cpp.

Referenced by calculateRegisterUsage(), and isLegalGatherOrScatter().

◆ ORE

OptimizationRemarkEmitter* llvm::LoopVectorizationCostModel::ORE

Interface to emit optimization remarks.

Definition at line 1841 of file LoopVectorize.cpp.

Referenced by computeMaxVF(), and runtimeChecksRequired().

◆ PSE

PredicatedScalarEvolution& llvm::LoopVectorizationCostModel::PSE

Predicated scalar evolution analysis.

Definition at line 1820 of file LoopVectorize.cpp.

Referenced by computeMaxVF(), runtimeChecksRequired(), selectInterleaveCount(), and setVectorizedCallDecision().

◆ TheFunction

const Function* llvm::LoopVectorizationCostModel::TheFunction

Definition at line 1843 of file LoopVectorize.cpp.

Referenced by computeMaxVF(), and getSmallestAndWidestTypes().

◆ TheLoop

Loop* llvm::LoopVectorizationCostModel::TheLoop

◆ TLI

const TargetLibraryInfo* llvm::LoopVectorizationCostModel::TLI

Target Library Info.

Definition at line 1832 of file LoopVectorize.cpp.

Referenced by getVectorCallCost(), getVectorIntrinsicCost(), and setVectorizedCallDecision().

◆ TTI

const TargetTransformInfo& llvm::LoopVectorizationCostModel::TTI

Vector target information.

Definition at line 1829 of file LoopVectorize.cpp.

Referenced by calculateRegisterUsage().

◆ ValuesToIgnore

SmallPtrSet<const Value *, 16> llvm::LoopVectorizationCostModel::ValuesToIgnore

Values to ignore in the cost model.

Definition at line 1853 of file LoopVectorize.cpp.

Referenced by calculateRegisterUsage(), collectElementTypesForWidening(), collectValuesToIgnore(), and expectedCost().

◆ VecValuesToIgnore

SmallPtrSet<const Value *, 16> llvm::LoopVectorizationCostModel::VecValuesToIgnore

Values to ignore in the cost model when VF > 1.

Definition at line 1856 of file LoopVectorize.cpp.

Referenced by calculateRegisterUsage(), collectValuesToIgnore(), and expectedCost().


The documentation for this class was generated from the following file: