LLVM  13.0.0git
Classes | Public Types | Public Member Functions | Public Attributes | List of all members
llvm::LoopVectorizationCostModel Class Reference

LoopVectorizationCostModel - estimates the expected speedups due to vectorization. More...

Collaboration diagram for llvm::LoopVectorizationCostModel:
Collaboration graph
[legend]

Classes

struct  RegisterUsage
 A struct that represents some properties of the register usage of a loop. More...
 

Public Types

enum  InstWidening {
  CM_Unknown, CM_Widen, CM_Widen_Reverse, CM_Interleave,
  CM_GatherScatter, CM_Scalarize
}
 Decision that was taken during cost calculation for memory instruction. More...
 
using ReductionChainMap = SmallMapVector< PHINode *, SmallVector< Instruction *, 4 >, 4 >
 A SmallMapVector to store the InLoop reduction op chains, mapping phi nodes to the chain of instructions representing the reductions. More...
 

Public Member Functions

 LoopVectorizationCostModel (ScalarEpilogueLowering SEL, Loop *L, PredicatedScalarEvolution &PSE, LoopInfo *LI, LoopVectorizationLegality *Legal, const TargetTransformInfo &TTI, const TargetLibraryInfo *TLI, DemandedBits *DB, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, const Function *F, const LoopVectorizeHints *Hints, InterleavedAccessInfo &IAI)
 
Optional< ElementCountcomputeMaxVF (ElementCount UserVF, unsigned UserIC)
 
bool runtimeChecksRequired ()
 
VectorizationFactor selectVectorizationFactor (ElementCount MaxVF)
 
VectorizationFactor selectEpilogueVectorizationFactor (const ElementCount MaxVF, const LoopVectorizationPlanner &LVP)
 
void selectUserVectorizationFactor (ElementCount UserVF)
 Setup cost-based decisions for user vectorization factor. More...
 
std::pair< unsigned, unsigned > getSmallestAndWidestTypes ()
 
unsigned selectInterleaveCount (ElementCount VF, unsigned LoopCost)
 
void setCostBasedWideningDecision (ElementCount VF)
 Memory access instruction may be vectorized in more than one way. More...
 
SmallVector< RegisterUsage, 8 > calculateRegisterUsage (ArrayRef< ElementCount > VFs)
 
void collectValuesToIgnore ()
 Collect values we want to ignore in the cost model. More...
 
void collectInLoopReductions ()
 Split reductions into those that happen in the loop, and those that happen outside. More...
 
const MapVector< Instruction *, uint64_t > & getMinimalBitwidths () const
 
bool isProfitableToScalarize (Instruction *I, ElementCount VF) const
 
bool isUniformAfterVectorization (Instruction *I, ElementCount VF) const
 Returns true if I is known to be uniform after vectorization. More...
 
bool isScalarAfterVectorization (Instruction *I, ElementCount VF) const
 Returns true if I is known to be scalar after vectorization. More...
 
bool canTruncateToMinimalBitwidth (Instruction *I, ElementCount VF) const
 
void setWideningDecision (Instruction *I, ElementCount VF, InstWidening W, InstructionCost Cost)
 Save vectorization decision W and Cost taken by the cost model for instruction I and vector width VF. More...
 
void setWideningDecision (const InterleaveGroup< Instruction > *Grp, ElementCount VF, InstWidening W, InstructionCost Cost)
 Save vectorization decision W and Cost taken by the cost model for interleaving group Grp and vector width VF. More...
 
InstWidening getWideningDecision (Instruction *I, ElementCount VF) const
 Return the cost model decision for the given instruction I and vector width VF. More...
 
InstructionCost getWideningCost (Instruction *I, ElementCount VF)
 Return the vectorization cost for the given instruction I and vector width VF. More...
 
bool isOptimizableIVTruncate (Instruction *I, ElementCount VF)
 Return True if instruction I is an optimizable truncate whose operand is an induction variable. More...
 
void collectInstsToScalarize (ElementCount VF)
 Collects the instructions to scalarize for each predicated instruction in the loop. More...
 
void collectUniformsAndScalars (ElementCount VF)
 Collect Uniform and Scalar values for the given VF. More...
 
bool isLegalMaskedStore (Type *DataType, Value *Ptr, Align Alignment) const
 Returns true if the target machine supports masked store operation for the given DataType and kind of access to Ptr. More...
 
bool isLegalMaskedLoad (Type *DataType, Value *Ptr, Align Alignment) const
 Returns true if the target machine supports masked load operation for the given DataType and kind of access to Ptr. More...
 
bool isLegalMaskedScatter (Type *DataType, Align Alignment) const
 Returns true if the target machine supports masked scatter operation for the given DataType. More...
 
bool isLegalMaskedGather (Type *DataType, Align Alignment) const
 Returns true if the target machine supports masked gather operation for the given DataType. More...
 
bool isLegalGatherOrScatter (Value *V)
 Returns true if the target machine can represent V as a masked gather or scatter operation. More...
 
bool canVectorizeReductions (ElementCount VF)
 Returns true if the target machine supports all of the reduction variables found for the given VF. More...
 
bool isScalarWithPredication (Instruction *I, ElementCount VF=ElementCount::getFixed(1)) const
 Returns true if I is an instruction that will be scalarized with predication. More...
 
bool isPredicatedInst (Instruction *I)
 
bool memoryInstructionCanBeWidened (Instruction *I, ElementCount VF=ElementCount::getFixed(1))
 Returns true if I is a memory instruction with consecutive memory access that can be widened. More...
 
bool interleavedAccessCanBeWidened (Instruction *I, ElementCount VF=ElementCount::getFixed(1))
 Returns true if I is a memory instruction in an interleaved-group of memory accesses that can be vectorized with wide vector loads/stores and shuffles. More...
 
bool isAccessInterleaved (Instruction *Instr)
 Check if Instr belongs to any interleaved access group. More...
 
const InterleaveGroup< Instruction > * getInterleavedAccessGroup (Instruction *Instr)
 Get the interleaved access group that Instr belongs to. More...
 
bool requiresScalarEpilogue () const
 Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop. More...
 
bool isScalarEpilogueAllowed () const
 Returns true if a scalar epilogue is not allowed due to optsize or a loop hint annotation. More...
 
bool foldTailByMasking () const
 Returns true if all loop blocks should be masked to fold tail loop. More...
 
bool blockNeedsPredication (BasicBlock *BB) const
 
const ReductionChainMapgetInLoopReductionChains () const
 Return the chain of instructions representing an inloop reduction. More...
 
bool isInLoopReduction (PHINode *Phi) const
 Returns true if the Phi is part of an inloop reduction. More...
 
InstructionCost getVectorIntrinsicCost (CallInst *CI, ElementCount VF) const
 Estimate cost of an intrinsic call instruction CI if it were vectorized with factor VF. More...
 
InstructionCost getVectorCallCost (CallInst *CI, ElementCount VF, bool &NeedToScalarize) const
 Estimate cost of a call instruction CI if it were vectorized with factor VF. More...
 
bool isMoreProfitable (const VectorizationFactor &A, const VectorizationFactor &B) const
 Returns true if the per-lane cost of VectorizationFactor A is lower than that of B. More...
 
void invalidateCostModelingDecisions ()
 Invalidates decisions already taken by the cost model. More...
 

Public Attributes

LoopTheLoop
 The loop that we evaluate. More...
 
PredicatedScalarEvolutionPSE
 Predicated scalar evolution analysis. More...
 
LoopInfoLI
 Loop Info analysis. More...
 
LoopVectorizationLegalityLegal
 Vectorization legality. More...
 
const TargetTransformInfoTTI
 Vector target information. More...
 
const TargetLibraryInfoTLI
 Target Library Info. More...
 
DemandedBitsDB
 Demanded bits analysis. More...
 
AssumptionCacheAC
 Assumption cache. More...
 
OptimizationRemarkEmitterORE
 Interface to emit optimization remarks. More...
 
const FunctionTheFunction
 
const LoopVectorizeHintsHints
 Loop Vectorize Hint. More...
 
InterleavedAccessInfoInterleaveInfo
 The interleave access information contains groups of interleaved accesses with the same stride and close to each other. More...
 
SmallPtrSet< const Value *, 16 > ValuesToIgnore
 Values to ignore in the cost model. More...
 
SmallPtrSet< const Value *, 16 > VecValuesToIgnore
 Values to ignore in the cost model when VF > 1. More...
 
SmallVector< VectorizationFactor, 8 > ProfitableVFs
 Profitable vector factors. More...
 

Detailed Description

LoopVectorizationCostModel - estimates the expected speedups due to vectorization.

In many cases vectorization is not profitable. This can happen because of a number of reasons. In this class we mainly attempt to predict the expected speedup/slowdowns due to the supported instruction set. We use the TargetTransformInfo to query the different backends for the cost of different operations.

Definition at line 1207 of file LoopVectorize.cpp.

Member Typedef Documentation

◆ ReductionChainMap

A SmallMapVector to store the InLoop reduction op chains, mapping phi nodes to the chain of instructions representing the reductions.

Uses a MapVector to ensure deterministic iteration order.

Definition at line 1580 of file LoopVectorize.cpp.

Member Enumeration Documentation

◆ InstWidening

Decision that was taken during cost calculation for memory instruction.

Enumerator
CM_Unknown 
CM_Widen 
CM_Widen_Reverse 
CM_Interleave 
CM_GatherScatter 
CM_Scalarize 

Definition at line 1353 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ LoopVectorizationCostModel()

llvm::LoopVectorizationCostModel::LoopVectorizationCostModel ( ScalarEpilogueLowering  SEL,
Loop L,
PredicatedScalarEvolution PSE,
LoopInfo LI,
LoopVectorizationLegality Legal,
const TargetTransformInfo TTI,
const TargetLibraryInfo TLI,
DemandedBits DB,
AssumptionCache AC,
OptimizationRemarkEmitter ORE,
const Function F,
const LoopVectorizeHints Hints,
InterleavedAccessInfo IAI 
)
inline

Definition at line 1209 of file LoopVectorize.cpp.

Member Function Documentation

◆ blockNeedsPredication()

bool llvm::LoopVectorizationCostModel::blockNeedsPredication ( BasicBlock BB) const
inline

◆ calculateRegisterUsage()

SmallVector< LoopVectorizationCostModel::RegisterUsage, 8 > LoopVectorizationCostModel::calculateRegisterUsage ( ArrayRef< ElementCount VFs)

◆ canTruncateToMinimalBitwidth()

bool llvm::LoopVectorizationCostModel::canTruncateToMinimalBitwidth ( Instruction I,
ElementCount  VF 
) const
inline
Returns
True if instruction I can be truncated to a smaller bitwidth for vectorization factor VF.

Definition at line 1346 of file LoopVectorize.cpp.

References I, llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.

◆ canVectorizeReductions()

bool llvm::LoopVectorizationCostModel::canVectorizeReductions ( ElementCount  VF)
inline

Returns true if the target machine supports all of the reduction variables found for the given VF.

Definition at line 1498 of file LoopVectorize.cpp.

References llvm::all_of(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::InnerLoopVectorizer::Legal, and Reduction.

◆ collectInLoopReductions()

void LoopVectorizationCostModel::collectInLoopReductions ( )

◆ collectInstsToScalarize()

void LoopVectorizationCostModel::collectInstsToScalarize ( ElementCount  VF)

◆ collectUniformsAndScalars()

void llvm::LoopVectorizationCostModel::collectUniformsAndScalars ( ElementCount  VF)
inline

Collect Uniform and Scalar values for the given VF.

The sets depend on CM decision for Load/Store instructions that may be vectorized as interleave, gather-scatter or scalarized.

Definition at line 1448 of file LoopVectorize.cpp.

References llvm::ElementCount::isScalar(), and llvm::InnerLoopVectorizer::VF.

◆ collectValuesToIgnore()

void LoopVectorizationCostModel::collectValuesToIgnore ( )

◆ computeMaxVF()

Optional< ElementCount > LoopVectorizationCostModel::computeMaxVF ( ElementCount  UserVF,
unsigned  UserIC 
)
Returns
An upper bound for the vectorization factor, or None if vectorization and interleaving should be avoided up front.

Definition at line 5571 of file LoopVectorize.cpp.

References llvm::ScalarEvolution::applyLoopGuards(), assert(), llvm::CM_ScalarEpilogueAllowed, llvm::CM_ScalarEpilogueNotAllowedLowTripLoop, llvm::CM_ScalarEpilogueNotAllowedOptSize, llvm::CM_ScalarEpilogueNotAllowedUsePredicate, llvm::CM_ScalarEpilogueNotNeededUsePredicate, llvm::dbgs(), llvm::DenseMapBase< DenseMap< KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >, KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >::empty(), llvm::ScalarEvolution::getAddExpr(), llvm::PredicatedScalarEvolution::getBackedgeTakenCount(), llvm::ScalarEvolution::getConstant(), llvm::LinearPolySize< LeafTy >::getFixedValue(), llvm::ScalarEvolution::getOne(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getSE(), llvm::ScalarEvolution::getSmallConstantTripCount(), llvm::SCEV::getType(), llvm::ScalarEvolution::getURemExpr(), llvm::TargetTransformInfo::hasBranchDivergence(), llvm::UnivariateLinearPolyBase< LeafTy >::isNonZero(), llvm::isPowerOf2_32(), llvm::LinearPolySize< LeafTy >::isScalable(), llvm::SCEV::isZero(), llvm::InnerLoopVectorizer::Legal, LLVM_DEBUG, LLVM_FALLTHROUGH, llvm::RuntimePointerChecking::Need, llvm::None, llvm::InnerLoopVectorizer::ORE, llvm::LoopVectorizationLegality::prepareToFoldTailByMasking(), llvm::InnerLoopVectorizer::PSE, llvm::reportVectorizationFailure(), and useMaskedInterleavedAccesses().

◆ foldTailByMasking()

bool llvm::LoopVectorizationCostModel::foldTailByMasking ( ) const
inline

◆ getInLoopReductionChains()

const ReductionChainMap& llvm::LoopVectorizationCostModel::getInLoopReductionChains ( ) const
inline

Return the chain of instructions representing an inloop reduction.

Definition at line 1583 of file LoopVectorize.cpp.

◆ getInterleavedAccessGroup()

const InterleaveGroup<Instruction>* llvm::LoopVectorizationCostModel::getInterleavedAccessGroup ( Instruction Instr)
inline

Get the interleaved access group that Instr belongs to.

Definition at line 1547 of file LoopVectorize.cpp.

◆ getMinimalBitwidths()

const MapVector<Instruction *, uint64_t>& llvm::LoopVectorizationCostModel::getMinimalBitwidths ( ) const
inline
Returns
The smallest bitwidth each instruction can be represented with. The vector equivalents of these instructions should be truncated to this type.

Definition at line 1291 of file LoopVectorize.cpp.

Referenced by llvm::InnerLoopVectorizer::truncateToMinimalBitwidths().

◆ getSmallestAndWidestTypes()

std::pair< unsigned, unsigned > LoopVectorizationCostModel::getSmallestAndWidestTypes ( )

◆ getVectorCallCost()

InstructionCost LoopVectorizationCostModel::getVectorCallCost ( CallInst CI,
ElementCount  VF,
bool &  NeedToScalarize 
) const

Estimate cost of a call instruction CI if it were vectorized with factor VF.

Return the cost of the instruction, including scalarization overhead if it's needed. The flag NeedToScalarize shows if the call needs to be scalarized - i.e. either vector version isn't available, or is too expensive.

Definition at line 3783 of file LoopVectorize.cpp.

References llvm::CallBase::arg_operands(), llvm::InnerLoopVectorizer::Cost, F, llvm::VFShape::get(), llvm::CallBase::getCalledFunction(), llvm::TargetTransformInfo::getCallInstrCost(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::Value::getType(), llvm::VFDatabase::getVectorizedFunction(), llvm::CallBase::isNoBuiltin(), llvm::ElementCount::isScalar(), llvm::TargetTransformInfo::TCK_RecipThroughput, llvm::InnerLoopVectorizer::TLI, llvm::ToVectorTy(), and llvm::InnerLoopVectorizer::VF.

Referenced by llvm::InnerLoopVectorizer::widenCallInstruction().

◆ getVectorIntrinsicCost()

InstructionCost LoopVectorizationCostModel::getVectorIntrinsicCost ( CallInst CI,
ElementCount  VF 
) const

◆ getWideningCost()

InstructionCost llvm::LoopVectorizationCostModel::getWideningCost ( Instruction I,
ElementCount  VF 
)
inline

Return the vectorization cost for the given instruction I and vector width VF.

Definition at line 1407 of file LoopVectorize.cpp.

References assert(), I, llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.

◆ getWideningDecision()

InstWidening llvm::LoopVectorizationCostModel::getWideningDecision ( Instruction I,
ElementCount  VF 
) const
inline

Return the cost model decision for the given instruction I and vector width VF.

Return CM_Unknown if this instruction did not pass through the cost modeling.

Definition at line 1391 of file LoopVectorize.cpp.

References assert(), EnableVPlanNativePath, I, llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.

Referenced by llvm::InnerLoopVectorizer::vectorizeMemoryInstruction().

◆ interleavedAccessCanBeWidened()

bool LoopVectorizationCostModel::interleavedAccessCanBeWidened ( Instruction I,
ElementCount  VF = ElementCount::getFixed(1) 
)

◆ invalidateCostModelingDecisions()

void llvm::LoopVectorizationCostModel::invalidateCostModelingDecisions ( )
inline

Invalidates decisions already taken by the cost model.

Definition at line 1611 of file LoopVectorize.cpp.

◆ isAccessInterleaved()

bool llvm::LoopVectorizationCostModel::isAccessInterleaved ( Instruction Instr)
inline

Check if Instr belongs to any interleaved access group.

Definition at line 1541 of file LoopVectorize.cpp.

◆ isInLoopReduction()

bool llvm::LoopVectorizationCostModel::isInLoopReduction ( PHINode Phi) const
inline

Returns true if the Phi is part of an inloop reduction.

Definition at line 1588 of file LoopVectorize.cpp.

Referenced by llvm::InnerLoopVectorizer::fixReduction(), and llvm::InnerLoopVectorizer::widenPHIInstruction().

◆ isLegalGatherOrScatter()

bool llvm::LoopVectorizationCostModel::isLegalGatherOrScatter ( Value V)
inline

Returns true if the target machine can represent V as a masked gather or scatter operation.

Definition at line 1485 of file LoopVectorize.cpp.

References llvm::getLoadStoreAlignment(), getMemInstValueType(), llvm::InnerLoopVectorizer::LI, and SI.

◆ isLegalMaskedGather()

bool llvm::LoopVectorizationCostModel::isLegalMaskedGather ( Type DataType,
Align  Alignment 
) const
inline

Returns true if the target machine supports masked gather operation for the given DataType.

Definition at line 1479 of file LoopVectorize.cpp.

References llvm::TargetTransformInfo::isLegalMaskedGather().

◆ isLegalMaskedLoad()

bool llvm::LoopVectorizationCostModel::isLegalMaskedLoad ( Type DataType,
Value Ptr,
Align  Alignment 
) const
inline

Returns true if the target machine supports masked load operation for the given DataType and kind of access to Ptr.

Definition at line 1466 of file LoopVectorize.cpp.

References llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::TargetTransformInfo::isLegalMaskedLoad(), and llvm::InnerLoopVectorizer::Legal.

◆ isLegalMaskedScatter()

bool llvm::LoopVectorizationCostModel::isLegalMaskedScatter ( Type DataType,
Align  Alignment 
) const
inline

Returns true if the target machine supports masked scatter operation for the given DataType.

Definition at line 1473 of file LoopVectorize.cpp.

References llvm::TargetTransformInfo::isLegalMaskedScatter().

◆ isLegalMaskedStore()

bool llvm::LoopVectorizationCostModel::isLegalMaskedStore ( Type DataType,
Value Ptr,
Align  Alignment 
) const
inline

Returns true if the target machine supports masked store operation for the given DataType and kind of access to Ptr.

Definition at line 1459 of file LoopVectorize.cpp.

References llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::TargetTransformInfo::isLegalMaskedStore(), and llvm::InnerLoopVectorizer::Legal.

◆ isMoreProfitable()

bool LoopVectorizationCostModel::isMoreProfitable ( const VectorizationFactor A,
const VectorizationFactor B 
) const

Returns true if the per-lane cost of VectorizationFactor A is lower than that of B.

Definition at line 5884 of file LoopVectorize.cpp.

References B.

◆ isOptimizableIVTruncate()

bool llvm::LoopVectorizationCostModel::isOptimizableIVTruncate ( Instruction I,
ElementCount  VF 
)
inline

Return True if instruction I is an optimizable truncate whose operand is an induction variable.

Such a truncate will be removed by adding a new induction variable with the destination type.

Definition at line 1418 of file LoopVectorize.cpp.

References llvm::LoopVectorizationLegality::getPrimaryInduction(), I, llvm::LoopVectorizationLegality::isInductionPhi(), llvm::TargetTransformInfo::isTruncateFree(), llvm::InnerLoopVectorizer::Legal, llvm::ToVectorTy(), and llvm::InnerLoopVectorizer::VF.

◆ isPredicatedInst()

bool llvm::LoopVectorizationCostModel::isPredicatedInst ( Instruction I)
inline

◆ isProfitableToScalarize()

bool llvm::LoopVectorizationCostModel::isProfitableToScalarize ( Instruction I,
ElementCount  VF 
) const
inline
Returns
True if it is more profitable to scalarize instruction I for vectorization factor VF.

Definition at line 1297 of file LoopVectorize.cpp.

References assert(), EnableVPlanNativePath, I, llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.

Referenced by llvm::InnerLoopVectorizer::shouldScalarizeInstruction().

◆ isScalarAfterVectorization()

bool llvm::LoopVectorizationCostModel::isScalarAfterVectorization ( Instruction I,
ElementCount  VF 
) const
inline

◆ isScalarEpilogueAllowed()

bool llvm::LoopVectorizationCostModel::isScalarEpilogueAllowed ( ) const
inline

Returns true if a scalar epilogue is not allowed due to optsize or a loop hint annotation.

Definition at line 1565 of file LoopVectorize.cpp.

References llvm::CM_ScalarEpilogueAllowed.

Referenced by llvm::InnerLoopVectorizer::vectorizeInterleaveGroup().

◆ isScalarWithPredication()

bool LoopVectorizationCostModel::isScalarWithPredication ( Instruction I,
ElementCount  VF = ElementCount::getFixed(1) 
) const

Returns true if I is an instruction that will be scalarized with predication.

Such instructions include conditional stores and instructions that may divide by zero. If a non-zero VF has been calculated, we check if I will be scalarized predication for that VF.

Definition at line 5251 of file LoopVectorize.cpp.

References assert(), llvm::getLoadStoreAlignment(), llvm::getLoadStorePointerOperand(), getMemInstValueType(), I, llvm::LoopVectorizationLegality::isMaskRequired(), llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::Legal, llvm::SPII::Load, mayDivideByZero(), llvm::SPII::Store, and llvm::InnerLoopVectorizer::VF.

◆ isUniformAfterVectorization()

bool llvm::LoopVectorizationCostModel::isUniformAfterVectorization ( Instruction I,
ElementCount  VF 
) const
inline

◆ memoryInstructionCanBeWidened()

bool LoopVectorizationCostModel::memoryInstructionCanBeWidened ( Instruction I,
ElementCount  VF = ElementCount::getFixed(1) 
)

Returns true if I is a memory instruction with consecutive memory access that can be widened.

Definition at line 5324 of file LoopVectorize.cpp.

References assert(), DL, llvm::getLoadStorePointerOperand(), hasIrregularType(), I, llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::InnerLoopVectorizer::Legal, llvm::InnerLoopVectorizer::LI, and SI.

◆ requiresScalarEpilogue()

bool llvm::LoopVectorizationCostModel::requiresScalarEpilogue ( ) const
inline

◆ runtimeChecksRequired()

bool LoopVectorizationCostModel::runtimeChecksRequired ( )

◆ selectEpilogueVectorizationFactor()

VectorizationFactor LoopVectorizationCostModel::selectEpilogueVectorizationFactor ( const ElementCount  MaxVF,
const LoopVectorizationPlanner LVP 
)

◆ selectInterleaveCount()

unsigned LoopVectorizationCostModel::selectInterleaveCount ( ElementCount  VF,
unsigned  LoopCost 
)
Returns
The desired interleave count. If interleave count has been specified by metadata it will be returned. Otherwise, the interleave count is computed and returned. VF and LoopCost are the selected vectorization factor and the cost of the selected VF.

Definition at line 6145 of file LoopVectorize.cpp.

References assert(), llvm::dbgs(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::empty(), llvm::TargetTransformInfo::enableAggressiveInterleaving(), EnableIndVarRegisterHeur, EnableLoadStoreRuntimeInterleave, F, ForceTargetMaxScalarInterleaveFactor, ForceTargetMaxVectorInterleaveFactor, ForceTargetNumScalarRegs, ForceTargetNumVectorRegs, llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::TargetTransformInfo::getMaxInterleaveFactor(), llvm::LoopVectorizationLegality::getMaxSafeDepDistBytes(), llvm::TargetTransformInfo::getNumberOfRegisters(), llvm::LoopVectorizationLegality::getNumLoads(), llvm::LoopVectorizationLegality::getNumStores(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::TargetTransformInfo::getRegisterClassName(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getSE(), getSmallBestKnownTC(), InterleaveSmallLoopScalarReduction, llvm::ElementCount::isScalar(), llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::Legal, LLVM_DEBUG, llvm::max(), MaxNestedScalarReductionIC, llvm::min(), llvm::RuntimePointerChecking::Need, llvm::PowerOf2Floor(), llvm::InnerLoopVectorizer::PSE, SmallLoopCost, TinyTripCountInterleaveThreshold, and llvm::InnerLoopVectorizer::VF.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ selectUserVectorizationFactor()

void llvm::LoopVectorizationCostModel::selectUserVectorizationFactor ( ElementCount  UserVF)
inline

Setup cost-based decisions for user vectorization factor.

Definition at line 1240 of file LoopVectorize.cpp.

◆ selectVectorizationFactor()

VectorizationFactor LoopVectorizationCostModel::selectVectorizationFactor ( ElementCount  MaxVF)

◆ setCostBasedWideningDecision()

void LoopVectorizationCostModel::setCostBasedWideningDecision ( ElementCount  VF)

◆ setWideningDecision() [1/2]

void llvm::LoopVectorizationCostModel::setWideningDecision ( const InterleaveGroup< Instruction > *  Grp,
ElementCount  VF,
InstWidening  W,
InstructionCost  Cost 
)
inline

Save vectorization decision W and Cost taken by the cost model for interleaving group Grp and vector width VF.

Broadcast this decicion to all instructions inside the group. But the cost will be assigned to one instruction only.

Definition at line 1372 of file LoopVectorize.cpp.

References assert(), llvm::InnerLoopVectorizer::Cost, llvm::InterleaveGroup< InstTy >::getFactor(), llvm::InterleaveGroup< InstTy >::getInsertPos(), llvm::InterleaveGroup< InstTy >::getMember(), i, I, llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::VF, and llvm::RISCVFenceField::W.

◆ setWideningDecision() [2/2]

void llvm::LoopVectorizationCostModel::setWideningDecision ( Instruction I,
ElementCount  VF,
InstWidening  W,
InstructionCost  Cost 
)
inline

Save vectorization decision W and Cost taken by the cost model for instruction I and vector width VF.

Definition at line 1364 of file LoopVectorize.cpp.

References assert(), llvm::InnerLoopVectorizer::Cost, I, llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::VF, and llvm::RISCVFenceField::W.

Member Data Documentation

◆ AC

AssumptionCache* llvm::LoopVectorizationCostModel::AC

Assumption cache.

Definition at line 1837 of file LoopVectorize.cpp.

◆ DB

DemandedBits* llvm::LoopVectorizationCostModel::DB

Demanded bits analysis.

Definition at line 1834 of file LoopVectorize.cpp.

◆ Hints

const LoopVectorizeHints* llvm::LoopVectorizationCostModel::Hints

◆ InterleaveInfo

InterleavedAccessInfo& llvm::LoopVectorizationCostModel::InterleaveInfo

The interleave access information contains groups of interleaved accesses with the same stride and close to each other.

Definition at line 1849 of file LoopVectorize.cpp.

◆ Legal

LoopVectorizationLegality* llvm::LoopVectorizationCostModel::Legal

Vectorization legality.

Definition at line 1825 of file LoopVectorize.cpp.

◆ LI

LoopInfo* llvm::LoopVectorizationCostModel::LI

Loop Info analysis.

Definition at line 1822 of file LoopVectorize.cpp.

◆ ORE

OptimizationRemarkEmitter* llvm::LoopVectorizationCostModel::ORE

Interface to emit optimization remarks.

Definition at line 1840 of file LoopVectorize.cpp.

◆ ProfitableVFs

SmallVector<VectorizationFactor, 8> llvm::LoopVectorizationCostModel::ProfitableVFs

Profitable vector factors.

Definition at line 1858 of file LoopVectorize.cpp.

◆ PSE

PredicatedScalarEvolution& llvm::LoopVectorizationCostModel::PSE

Predicated scalar evolution analysis.

Definition at line 1819 of file LoopVectorize.cpp.

◆ TheFunction

const Function* llvm::LoopVectorizationCostModel::TheFunction

Definition at line 1842 of file LoopVectorize.cpp.

◆ TheLoop

Loop* llvm::LoopVectorizationCostModel::TheLoop

The loop that we evaluate.

Definition at line 1816 of file LoopVectorize.cpp.

◆ TLI

const TargetLibraryInfo* llvm::LoopVectorizationCostModel::TLI

Target Library Info.

Definition at line 1831 of file LoopVectorize.cpp.

◆ TTI

const TargetTransformInfo& llvm::LoopVectorizationCostModel::TTI

Vector target information.

Definition at line 1828 of file LoopVectorize.cpp.

◆ ValuesToIgnore

SmallPtrSet<const Value *, 16> llvm::LoopVectorizationCostModel::ValuesToIgnore

Values to ignore in the cost model.

Definition at line 1852 of file LoopVectorize.cpp.

◆ VecValuesToIgnore

SmallPtrSet<const Value *, 16> llvm::LoopVectorizationCostModel::VecValuesToIgnore

Values to ignore in the cost model when VF > 1.

Definition at line 1855 of file LoopVectorize.cpp.


The documentation for this class was generated from the following file: