LLVM
15.0.0git
|
LoopVectorizationCostModel - estimates the expected speedups due to vectorization. More...
Classes | |
struct | RegisterUsage |
A struct that represents some properties of the register usage of a loop. More... | |
Public Types | |
enum | InstWidening { CM_Unknown, CM_Widen, CM_Widen_Reverse, CM_Interleave, CM_GatherScatter, CM_Scalarize } |
Decision that was taken during cost calculation for memory instruction. More... | |
using | ReductionChainMap = SmallMapVector< PHINode *, SmallVector< Instruction *, 4 >, 4 > |
A SmallMapVector to store the InLoop reduction op chains, mapping phi nodes to the chain of instructions representing the reductions. More... | |
Public Member Functions | |
LoopVectorizationCostModel (ScalarEpilogueLowering SEL, Loop *L, PredicatedScalarEvolution &PSE, LoopInfo *LI, LoopVectorizationLegality *Legal, const TargetTransformInfo &TTI, const TargetLibraryInfo *TLI, DemandedBits *DB, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, const Function *F, const LoopVectorizeHints *Hints, InterleavedAccessInfo &IAI) | |
FixedScalableVFPair | computeMaxVF (ElementCount UserVF, unsigned UserIC) |
bool | runtimeChecksRequired () |
VectorizationFactor | selectVectorizationFactor (const ElementCountSet &CandidateVFs) |
VectorizationFactor | selectEpilogueVectorizationFactor (const ElementCount MaxVF, const LoopVectorizationPlanner &LVP) |
bool | selectUserVectorizationFactor (ElementCount UserVF) |
Setup cost-based decisions for user vectorization factor. More... | |
std::pair< unsigned, unsigned > | getSmallestAndWidestTypes () |
unsigned | selectInterleaveCount (ElementCount VF, unsigned LoopCost) |
void | setCostBasedWideningDecision (ElementCount VF) |
Memory access instruction may be vectorized in more than one way. More... | |
SmallVector< RegisterUsage, 8 > | calculateRegisterUsage (ArrayRef< ElementCount > VFs) |
void | collectValuesToIgnore () |
Collect values we want to ignore in the cost model. More... | |
void | collectElementTypesForWidening () |
Collect all element types in the loop for which widening is needed. More... | |
void | collectInLoopReductions () |
Split reductions into those that happen in the loop, and those that happen outside. More... | |
bool | useOrderedReductions (const RecurrenceDescriptor &RdxDesc) const |
Returns true if we should use strict in-order reductions for the given RdxDesc. More... | |
const MapVector< Instruction *, uint64_t > & | getMinimalBitwidths () const |
bool | isProfitableToScalarize (Instruction *I, ElementCount VF) const |
bool | isUniformAfterVectorization (Instruction *I, ElementCount VF) const |
Returns true if I is known to be uniform after vectorization. More... | |
bool | isScalarAfterVectorization (Instruction *I, ElementCount VF) const |
Returns true if I is known to be scalar after vectorization. More... | |
bool | canTruncateToMinimalBitwidth (Instruction *I, ElementCount VF) const |
void | setWideningDecision (Instruction *I, ElementCount VF, InstWidening W, InstructionCost Cost) |
Save vectorization decision W and Cost taken by the cost model for instruction I and vector width VF . More... | |
void | setWideningDecision (const InterleaveGroup< Instruction > *Grp, ElementCount VF, InstWidening W, InstructionCost Cost) |
Save vectorization decision W and Cost taken by the cost model for interleaving group Grp and vector width VF . More... | |
InstWidening | getWideningDecision (Instruction *I, ElementCount VF) const |
Return the cost model decision for the given instruction I and vector width VF . More... | |
InstructionCost | getWideningCost (Instruction *I, ElementCount VF) |
Return the vectorization cost for the given instruction I and vector width VF . More... | |
bool | isOptimizableIVTruncate (Instruction *I, ElementCount VF) |
Return True if instruction I is an optimizable truncate whose operand is an induction variable. More... | |
void | collectInstsToScalarize (ElementCount VF) |
Collects the instructions to scalarize for each predicated instruction in the loop. More... | |
void | collectUniformsAndScalars (ElementCount VF) |
Collect Uniform and Scalar values for the given VF . More... | |
bool | isLegalMaskedStore (Type *DataType, Value *Ptr, Align Alignment) const |
Returns true if the target machine supports masked store operation for the given DataType and kind of access to Ptr . More... | |
bool | isLegalMaskedLoad (Type *DataType, Value *Ptr, Align Alignment) const |
Returns true if the target machine supports masked load operation for the given DataType and kind of access to Ptr . More... | |
bool | isLegalGatherOrScatter (Value *V, ElementCount VF=ElementCount::getFixed(1)) |
Returns true if the target machine can represent V as a masked gather or scatter operation. More... | |
bool | canVectorizeReductions (ElementCount VF) const |
Returns true if the target machine supports all of the reduction variables found for the given VF. More... | |
bool | isScalarWithPredication (Instruction *I, ElementCount VF) const |
Returns true if I is an instruction that will be scalarized with predication when vectorizing I with vectorization factor VF . More... | |
bool | isPredicatedInst (Instruction *I, ElementCount VF, bool IsKnownUniform=false) |
bool | memoryInstructionCanBeWidened (Instruction *I, ElementCount VF=ElementCount::getFixed(1)) |
Returns true if I is a memory instruction with consecutive memory access that can be widened. More... | |
bool | interleavedAccessCanBeWidened (Instruction *I, ElementCount VF=ElementCount::getFixed(1)) |
Returns true if I is a memory instruction in an interleaved-group of memory accesses that can be vectorized with wide vector loads/stores and shuffles. More... | |
bool | isAccessInterleaved (Instruction *Instr) |
Check if Instr belongs to any interleaved access group. More... | |
const InterleaveGroup< Instruction > * | getInterleavedAccessGroup (Instruction *Instr) |
Get the interleaved access group that Instr belongs to. More... | |
bool | requiresScalarEpilogue (ElementCount VF) const |
Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop. More... | |
bool | isScalarEpilogueAllowed () const |
Returns true if a scalar epilogue is not allowed due to optsize or a loop hint annotation. More... | |
bool | foldTailByMasking () const |
Returns true if all loop blocks should be masked to fold tail loop. More... | |
bool | blockNeedsPredicationForAnyReason (BasicBlock *BB) const |
Returns true if the instructions in this block requires predication for any reason, e.g. More... | |
const ReductionChainMap & | getInLoopReductionChains () const |
Return the chain of instructions representing an inloop reduction. More... | |
bool | isInLoopReduction (PHINode *Phi) const |
Returns true if the Phi is part of an inloop reduction. More... | |
InstructionCost | getVectorIntrinsicCost (CallInst *CI, ElementCount VF) const |
Estimate cost of an intrinsic call instruction CI if it were vectorized with factor VF. More... | |
InstructionCost | getVectorCallCost (CallInst *CI, ElementCount VF, bool &NeedToScalarize) const |
Estimate cost of a call instruction CI if it were vectorized with factor VF. More... | |
bool | isMoreProfitable (const VectorizationFactor &A, const VectorizationFactor &B) const |
Returns true if the per-lane cost of VectorizationFactor A is lower than that of B. More... | |
void | invalidateCostModelingDecisions () |
Invalidates decisions already taken by the cost model. More... | |
Public Attributes | |
Loop * | TheLoop |
The loop that we evaluate. More... | |
PredicatedScalarEvolution & | PSE |
Predicated scalar evolution analysis. More... | |
LoopInfo * | LI |
Loop Info analysis. More... | |
LoopVectorizationLegality * | Legal |
Vectorization legality. More... | |
const TargetTransformInfo & | TTI |
Vector target information. More... | |
const TargetLibraryInfo * | TLI |
Target Library Info. More... | |
DemandedBits * | DB |
Demanded bits analysis. More... | |
AssumptionCache * | AC |
Assumption cache. More... | |
OptimizationRemarkEmitter * | ORE |
Interface to emit optimization remarks. More... | |
const Function * | TheFunction |
const LoopVectorizeHints * | Hints |
Loop Vectorize Hint. More... | |
InterleavedAccessInfo & | InterleaveInfo |
The interleave access information contains groups of interleaved accesses with the same stride and close to each other. More... | |
SmallPtrSet< const Value *, 16 > | ValuesToIgnore |
Values to ignore in the cost model. More... | |
SmallPtrSet< const Value *, 16 > | VecValuesToIgnore |
Values to ignore in the cost model when VF > 1. More... | |
SmallPtrSet< Type *, 16 > | ElementTypesInLoop |
All element types found in the loop. More... | |
SmallVector< VectorizationFactor, 8 > | ProfitableVFs |
Profitable vector factors. More... | |
LoopVectorizationCostModel - estimates the expected speedups due to vectorization.
In many cases vectorization is not profitable. This can happen because of a number of reasons. In this class we mainly attempt to predict the expected speedup/slowdowns due to the supported instruction set. We use the TargetTransformInfo to query the different backends for the cost of different operations.
Definition at line 1201 of file LoopVectorize.cpp.
using llvm::LoopVectorizationCostModel::ReductionChainMap = SmallMapVector<PHINode *, SmallVector<Instruction *, 4>, 4> |
A SmallMapVector to store the InLoop reduction op chains, mapping phi nodes to the chain of instructions representing the reductions.
Uses a MapVector to ensure deterministic iteration order.
Definition at line 1592 of file LoopVectorize.cpp.
Decision that was taken during cost calculation for memory instruction.
Enumerator | |
---|---|
CM_Unknown | |
CM_Widen | |
CM_Widen_Reverse | |
CM_Interleave | |
CM_GatherScatter | |
CM_Scalarize |
Definition at line 1363 of file LoopVectorize.cpp.
|
inline |
Definition at line 1203 of file LoopVectorize.cpp.
|
inline |
Returns true if the instructions in this block requires predication for any reason, e.g.
because tail folding now requires a predicate or because the block in the original loop was predicated.
Definition at line 1584 of file LoopVectorize.cpp.
References BB, llvm::LoopVectorizationLegality::blockNeedsPredication(), and llvm::InnerLoopVectorizer::Legal.
SmallVector< LoopVectorizationCostModel::RegisterUsage, 8 > LoopVectorizationCostModel::calculateRegisterUsage | ( | ArrayRef< ElementCount > | VFs | ) |
Definition at line 5884 of file LoopVectorize.cpp.
References BB, llvm::LoopBlocksDFS::beginRPO(), llvm::SmallPtrSetImpl< PtrType >::count(), llvm::dbgs(), llvm::numbers::e, llvm::LoopBlocksDFS::endRPO(), llvm::SmallPtrSetImpl< PtrType >::erase(), llvm::find(), llvm::VectorType::get(), llvm::TargetTransformInfo::getRegisterClassForType(), llvm::TargetTransformInfo::getRegisterClassName(), llvm::TargetTransformInfo::getRegUsageForType(), i, I, llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::LegalityPredicates::isScalar(), llvm::VectorType::isValidElementType(), llvm::LegalityPredicates::isVector(), j(), llvm::InnerLoopVectorizer::LI, List, LLVM_DEBUG, llvm::LoopVectorizationCostModel::RegisterUsage::LoopInvariantRegs, llvm::make_range(), llvm::max(), llvm::LoopVectorizationCostModel::RegisterUsage::MaxLocalUsers, llvm::LoopBlocksDFS::perform(), llvm::RegUsage, s, llvm::SmallPtrSetImplBase::size(), llvm::ArrayRef< T >::size(), ToRemove, llvm::InnerLoopVectorizer::TTI, and llvm::InnerLoopVectorizer::VF.
|
inline |
I
can be truncated to a smaller bitwidth for vectorization factor VF
. Definition at line 1356 of file LoopVectorize.cpp.
References I, llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.
|
inline |
Returns true if the target machine supports all of the reduction variables found for the given VF.
Definition at line 1499 of file LoopVectorize.cpp.
References llvm::all_of(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::InnerLoopVectorizer::Legal, and Reduction.
void LoopVectorizationCostModel::collectElementTypesForWidening | ( | ) |
Collect all element types in the loop for which widening is needed.
Definition at line 5592 of file LoopVectorize.cpp.
References assert(), BB, llvm::MapVector< KeyT, ValueT, MapType, VectorType >::find(), llvm::RecurrenceDescriptor::getOpcode(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::LoopVectorizationLegality::getReductionVars(), I, llvm::LoopVectorizationLegality::isReductionVariable(), llvm::InnerLoopVectorizer::Legal, llvm::TargetTransformInfo::preferInLoopReduction(), PreferInLoopReductions, llvm::ARM_MB::ST, T, and llvm::InnerLoopVectorizer::useOrderedReductions().
Referenced by llvm::LoopVectorizePass::processLoop(), and processLoopInVPlanNativePath().
void LoopVectorizationCostModel::collectInLoopReductions | ( | ) |
Split reductions into those that happen in the loop, and those that happen outside.
In loop reductions are collected into InLoopReductionChains.
Definition at line 7351 of file LoopVectorize.cpp.
References llvm::dbgs(), llvm::RecurrenceDescriptor::getOpcode(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::RecurrenceDescriptor::getReductionOpChain(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::Value::getType(), I, llvm::InnerLoopVectorizer::Legal, LLVM_DEBUG, llvm::TargetTransformInfo::preferInLoopReduction(), PreferInLoopReductions, Reduction, and llvm::InnerLoopVectorizer::useOrderedReductions().
void LoopVectorizationCostModel::collectInstsToScalarize | ( | ElementCount | VF | ) |
Collects the instructions to scalarize for each predicated instruction in the loop.
Definition at line 6089 of file LoopVectorize.cpp.
References BB, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::begin(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::DenseMapBase< DenseMap< KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >, KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >::end(), llvm::DenseMapBase< DenseMap< KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >, KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >::find(), I, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::insert(), llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::LinearPolySize< LeafTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::UnivariateLinearPolyBase< LeafTy >::isZero(), and llvm::InnerLoopVectorizer::VF.
|
inline |
Collect Uniform and Scalar values for the given VF
.
The sets depend on CM decision for Load/Store instructions that may be vectorized as interleave, gather-scatter or scalarized.
Definition at line 1458 of file LoopVectorize.cpp.
References llvm::ElementCount::isScalar(), and llvm::InnerLoopVectorizer::VF.
void LoopVectorizationCostModel::collectValuesToIgnore | ( | ) |
Collect values we want to ignore in the cost model.
Definition at line 7321 of file LoopVectorize.cpp.
References llvm::InnerLoopVectorizer::AC, BB, llvm::SmallPtrSetImpl< PtrType >::begin(), llvm::CodeMetrics::collectEphemeralValues(), llvm::SmallPtrSetImpl< PtrType >::end(), llvm::RecurrenceDescriptor::getCastInsts(), llvm::InductionDescriptor::getCastInsts(), llvm::LoopVectorizationLegality::getInductionVars(), llvm::LoopVectorizationLegality::getReductionVars(), I, llvm::SmallVectorImpl< T >::insert(), llvm::LoopVectorizationLegality::isInvariantAddressOfReduction(), llvm::InnerLoopVectorizer::Legal, Reduction, and SI.
Referenced by llvm::LoopVectorizePass::processLoop().
FixedScalableVFPair LoopVectorizationCostModel::computeMaxVF | ( | ElementCount | UserVF, |
unsigned | UserIC | ||
) |
Definition at line 4997 of file LoopVectorize.cpp.
References llvm::ScalarEvolution::applyLoopGuards(), assert(), llvm::CM_ScalarEpilogueAllowed, llvm::CM_ScalarEpilogueNotAllowedLowTripLoop, llvm::CM_ScalarEpilogueNotAllowedOptSize, llvm::CM_ScalarEpilogueNotAllowedUsePredicate, llvm::CM_ScalarEpilogueNotNeededUsePredicate, llvm::dbgs(), llvm::DenseMapBase< DenseMap< KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >, KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >::empty(), llvm::FixedScalableVFPair::FixedVF, llvm::ScalarEvolution::getAddExpr(), llvm::PredicatedScalarEvolution::getBackedgeTakenCount(), llvm::ScalarEvolution::getConstant(), llvm::LinearPolySize< LeafTy >::getFixedValue(), llvm::FixedScalableVFPair::getNone(), llvm::ScalarEvolution::getOne(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getSE(), llvm::ScalarEvolution::getSmallConstantTripCount(), llvm::SCEV::getType(), llvm::ScalarEvolution::getURemExpr(), llvm::TargetTransformInfo::hasBranchDivergence(), llvm::UnivariateLinearPolyBase< LeafTy >::isNonZero(), llvm::isPowerOf2_32(), llvm::ElementCount::isVector(), llvm::SCEV::isZero(), llvm::InnerLoopVectorizer::Legal, LLVM_DEBUG, LLVM_FALLTHROUGH, llvm::RuntimePointerChecking::Need, llvm::InnerLoopVectorizer::ORE, llvm::LoopVectorizationLegality::prepareToFoldTailByMasking(), llvm::InnerLoopVectorizer::PSE, llvm::reportVectorizationFailure(), llvm::FixedScalableVFPair::ScalableVF, and useMaskedInterleavedAccesses().
|
inline |
Returns true if all loop blocks should be masked to fold tail loop.
Definition at line 1579 of file LoopVectorize.cpp.
Referenced by llvm::InnerLoopVectorizer::completeLoopSkeleton(), llvm::InnerLoopVectorizer::emitIterationCountCheck(), llvm::InnerLoopVectorizer::fixReduction(), and llvm::InnerLoopVectorizer::getOrCreateVectorTripCount().
|
inline |
Return the chain of instructions representing an inloop reduction.
Definition at line 1595 of file LoopVectorize.cpp.
|
inline |
Get the interleaved access group that Instr
belongs to.
Definition at line 1556 of file LoopVectorize.cpp.
|
inline |
Definition at line 1301 of file LoopVectorize.cpp.
Referenced by llvm::InnerLoopVectorizer::truncateToMinimalBitwidths().
std::pair< unsigned, unsigned > LoopVectorizationCostModel::getSmallestAndWidestTypes | ( | ) |
Definition at line 5561 of file LoopVectorize.cpp.
References DL, llvm::MapVector< KeyT, ValueT, MapType, VectorType >::empty(), llvm::MachineFunction::getDataLayout(), llvm::RecurrenceDescriptor::getMinWidthCastToRecurrenceTypeInBits(), llvm::MachineBasicBlock::getParent(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::Type::getScalarSizeInBits(), llvm::InnerLoopVectorizer::Legal, and T.
Referenced by determineVPlanVF().
InstructionCost LoopVectorizationCostModel::getVectorCallCost | ( | CallInst * | CI, |
ElementCount | VF, | ||
bool & | NeedToScalarize | ||
) | const |
Estimate cost of a call instruction CI if it were vectorized with factor VF.
Return the cost of the instruction, including scalarization overhead if it's needed. The flag NeedToScalarize shows if the call needs to be scalarized - i.e. either vector version isn't available, or is too expensive.
Definition at line 3430 of file LoopVectorize.cpp.
References llvm::CallBase::args(), llvm::InnerLoopVectorizer::Cost, F, llvm::VFShape::get(), llvm::CallBase::getCalledFunction(), llvm::TargetTransformInfo::getCallInstrCost(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::Value::getType(), llvm::VFDatabase::getVectorizedFunction(), llvm::CallBase::isNoBuiltin(), llvm::ElementCount::isScalar(), llvm::TargetTransformInfo::TCK_RecipThroughput, llvm::InnerLoopVectorizer::TLI, llvm::ToVectorTy(), and llvm::InnerLoopVectorizer::VF.
Referenced by llvm::InnerLoopVectorizer::widenCallInstruction().
InstructionCost LoopVectorizationCostModel::getVectorIntrinsicCost | ( | CallInst * | CI, |
ElementCount | VF | ||
) | const |
Estimate cost of an intrinsic call instruction CI if it were vectorized with factor VF.
Return the cost of the instruction, including scalarization overhead if it's needed.
Definition at line 3485 of file LoopVectorize.cpp.
References llvm::CallBase::args(), Arguments, assert(), llvm::CallBase::getCalledFunction(), llvm::Function::getFunctionType(), llvm::TargetTransformInfo::getIntrinsicInstrCost(), llvm::Value::getType(), llvm::getVectorIntrinsicIDForCall(), MaybeVectorizeType(), llvm::FunctionType::param_begin(), llvm::FunctionType::param_end(), llvm::TargetTransformInfo::TCK_RecipThroughput, llvm::InnerLoopVectorizer::TLI, transform, and llvm::InnerLoopVectorizer::VF.
Referenced by llvm::InnerLoopVectorizer::widenCallInstruction().
|
inline |
Return the vectorization cost for the given instruction I
and vector width VF
.
Definition at line 1417 of file LoopVectorize.cpp.
References assert(), I, llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.
|
inline |
Return the cost model decision for the given instruction I
and vector width VF
.
Return CM_Unknown if this instruction did not pass through the cost modeling.
Definition at line 1401 of file LoopVectorize.cpp.
References assert(), EnableVPlanNativePath, I, llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.
bool LoopVectorizationCostModel::interleavedAccessCanBeWidened | ( | Instruction * | I, |
ElementCount | VF = ElementCount::getFixed(1) |
||
) |
Returns true if I
is a memory instruction in an interleaved-group of memory accesses that can be vectorized with wide vector loads/stores and shuffles.
Definition at line 4490 of file LoopVectorize.cpp.
References assert(), DL, llvm::getLoadStoreAlignment(), llvm::getLoadStoreType(), hasIrregularType(), i, I, llvm::TargetTransformInfo::isLegalMaskedLoad(), llvm::TargetTransformInfo::isLegalMaskedStore(), llvm::LoopVectorizationLegality::isMaskRequired(), llvm::InnerLoopVectorizer::Legal, useMaskedInterleavedAccesses(), and llvm::InnerLoopVectorizer::VF.
|
inline |
Invalidates decisions already taken by the cost model.
Definition at line 1623 of file LoopVectorize.cpp.
|
inline |
Check if Instr
belongs to any interleaved access group.
Definition at line 1550 of file LoopVectorize.cpp.
|
inline |
Returns true if the Phi is part of an inloop reduction.
Definition at line 1600 of file LoopVectorize.cpp.
|
inline |
Returns true if the target machine can represent V
as a masked gather or scatter operation.
Definition at line 1483 of file LoopVectorize.cpp.
References llvm::VectorType::get(), llvm::getLoadStoreAlignment(), llvm::getLoadStoreType(), llvm::TargetTransformInfo::isLegalMaskedGather(), llvm::TargetTransformInfo::isLegalMaskedScatter(), llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::LI, SI, and llvm::InnerLoopVectorizer::VF.
|
inline |
Returns true if the target machine supports masked load operation for the given DataType
and kind of access to Ptr
.
Definition at line 1476 of file LoopVectorize.cpp.
References llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::TargetTransformInfo::isLegalMaskedLoad(), and llvm::InnerLoopVectorizer::Legal.
|
inline |
Returns true if the target machine supports masked store operation for the given DataType
and kind of access to Ptr
.
Definition at line 1469 of file LoopVectorize.cpp.
References llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::TargetTransformInfo::isLegalMaskedStore(), and llvm::InnerLoopVectorizer::Legal.
bool LoopVectorizationCostModel::isMoreProfitable | ( | const VectorizationFactor & | A, |
const VectorizationFactor & | B | ||
) | const |
Returns true if the per-lane cost of VectorizationFactor A is lower than that of B.
Definition at line 5250 of file LoopVectorize.cpp.
References B, llvm::divideCeil(), llvm::PredicatedScalarEvolution::getSE(), llvm::ScalarEvolution::getSmallConstantMaxTripCount(), and llvm::InnerLoopVectorizer::PSE.
|
inline |
Return True if instruction I
is an optimizable truncate whose operand is an induction variable.
Such a truncate will be removed by adding a new induction variable with the destination type.
Definition at line 1428 of file LoopVectorize.cpp.
References llvm::LoopVectorizationLegality::getPrimaryInduction(), I, llvm::LoopVectorizationLegality::isInductionPhi(), llvm::TargetTransformInfo::isTruncateFree(), llvm::InnerLoopVectorizer::Legal, llvm::ToVectorTy(), and llvm::InnerLoopVectorizer::VF.
|
inline |
Definition at line 1516 of file LoopVectorize.cpp.
References llvm::LoopVectorizationLegality::blockNeedsPredication(), I, llvm::LoopVectorizationLegality::isMaskRequired(), llvm::InnerLoopVectorizer::Legal, and llvm::InnerLoopVectorizer::VF.
|
inline |
I
for vectorization factor VF
. Definition at line 1307 of file LoopVectorize.cpp.
References assert(), EnableVPlanNativePath, I, llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.
Referenced by createWidenInductionRecipes().
|
inline |
Returns true if I
is known to be scalar after vectorization.
Definition at line 1339 of file LoopVectorize.cpp.
References assert(), EnableVPlanNativePath, I, llvm::ElementCount::isScalar(), and llvm::InnerLoopVectorizer::VF.
Referenced by createWidenInductionRecipes().
|
inline |
Returns true if a scalar epilogue is not allowed due to optsize or a loop hint annotation.
Definition at line 1574 of file LoopVectorize.cpp.
References llvm::CM_ScalarEpilogueAllowed.
Referenced by llvm::InnerLoopVectorizer::vectorizeInterleaveGroup().
bool LoopVectorizationCostModel::isScalarWithPredication | ( | Instruction * | I, |
ElementCount | VF | ||
) | const |
Returns true if I
is an instruction that will be scalarized with predication when vectorizing I
with vectorization factor VF
.
Such instructions include conditional stores and instructions that may divide by zero.
Definition at line 4459 of file LoopVectorize.cpp.
References llvm::VectorType::get(), llvm::getLoadStoreAlignment(), llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), I, llvm::TargetTransformInfo::isLegalMaskedGather(), llvm::TargetTransformInfo::isLegalMaskedScatter(), llvm::LoopVectorizationLegality::isMaskRequired(), llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::Legal, llvm::SPII::Load, mayDivideByZero(), llvm::SPII::Store, and llvm::InnerLoopVectorizer::VF.
|
inline |
Returns true if I
is known to be uniform after vectorization.
Definition at line 1323 of file LoopVectorize.cpp.
References assert(), EnableVPlanNativePath, I, llvm::ElementCount::isScalar(), and llvm::InnerLoopVectorizer::VF.
bool LoopVectorizationCostModel::memoryInstructionCanBeWidened | ( | Instruction * | I, |
ElementCount | VF = ElementCount::getFixed(1) |
||
) |
Returns true if I
is a memory instruction with consecutive memory access that can be widened.
Definition at line 4559 of file LoopVectorize.cpp.
References assert(), DL, llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), hasIrregularType(), I, llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::InnerLoopVectorizer::Legal, and llvm::InnerLoopVectorizer::VF.
|
inline |
Returns true if we're required to use a scalar epilogue for at least the final iteration of the original loop.
Definition at line 1562 of file LoopVectorize.cpp.
References llvm::ElementCount::isVector(), and llvm::InnerLoopVectorizer::VF.
Referenced by llvm::InnerLoopVectorizer::completeLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), llvm::InnerLoopVectorizer::createVectorLoopSkeleton(), llvm::InnerLoopVectorizer::emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), llvm::InnerLoopVectorizer::emitSCEVChecks(), llvm::InnerLoopVectorizer::fixFirstOrderRecurrence(), llvm::InnerLoopVectorizer::fixReduction(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), and llvm::InnerLoopVectorizer::getOrCreateVectorTripCount().
bool LoopVectorizationCostModel::runtimeChecksRequired | ( | ) |
Definition at line 4796 of file LoopVectorize.cpp.
References llvm::dbgs(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::empty(), llvm::LoopVectorizationLegality::getLAI(), llvm::PredicatedScalarEvolution::getPredicate(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::LoopAccessInfo::getSymbolicStrides(), llvm::SCEVPredicate::isAlwaysTrue(), llvm::InnerLoopVectorizer::Legal, LLVM_DEBUG, llvm::RuntimePointerChecking::Need, llvm::InnerLoopVectorizer::ORE, llvm::InnerLoopVectorizer::PSE, and llvm::reportVectorizationFailure().
VectorizationFactor LoopVectorizationCostModel::selectEpilogueVectorizationFactor | ( | const ElementCount | MaxVF, |
const LoopVectorizationPlanner & | LVP | ||
) |
Definition at line 5485 of file LoopVectorize.cpp.
References llvm::dbgs(), llvm::VectorizationFactor::Disabled(), EnableEpilogueVectorization, EpilogueVectorizationForceVF, llvm::LinearPolySize< ElementCount >::getFixed(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::LoopVectorizationPlanner::hasPlanWithVF(), llvm::LinearPolySize< ElementCount >::isKnownLT(), llvm::LinearPolySize< LeafTy >::isScalable(), and LLVM_DEBUG.
Referenced by llvm::LoopVectorizePass::processLoop().
unsigned LoopVectorizationCostModel::selectInterleaveCount | ( | ElementCount | VF, |
unsigned | LoopCost | ||
) |
Definition at line 5635 of file LoopVectorize.cpp.
References llvm::any_of(), assert(), BB, llvm::dbgs(), llvm::MapVector< KeyT, ValueT, MapType, VectorType >::empty(), llvm::TargetTransformInfo::enableAggressiveInterleaving(), EnableIndVarRegisterHeur, EnableLoadStoreRuntimeInterleave, F, ForceTargetMaxScalarInterleaveFactor, ForceTargetMaxVectorInterleaveFactor, ForceTargetNumScalarRegs, ForceTargetNumVectorRegs, llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::TargetTransformInfo::getMaxInterleaveFactor(), llvm::LoopVectorizationLegality::getMaxSafeDepDistBytes(), llvm::TargetTransformInfo::getNumberOfRegisters(), llvm::LoopVectorizationLegality::getNumLoads(), llvm::LoopVectorizationLegality::getNumStores(), llvm::LoopVectorizationLegality::getReductionVars(), llvm::TargetTransformInfo::getRegisterClassName(), llvm::LoopVectorizationLegality::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getSE(), getSmallBestKnownTC(), InterleaveSmallLoopScalarReduction, llvm::ElementCount::isScalar(), llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::Legal, LLVM_DEBUG, llvm::max(), MaxNestedScalarReductionIC, llvm::min(), llvm::RuntimePointerChecking::Need, llvm::PowerOf2Floor(), llvm::InnerLoopVectorizer::PSE, Reduction, SmallLoopCost, TinyTripCountInterleaveThreshold, and llvm::InnerLoopVectorizer::VF.
Referenced by llvm::LoopVectorizePass::processLoop().
|
inline |
Setup cost-based decisions for user vectorization factor.
Definition at line 1238 of file LoopVectorize.cpp.
VectorizationFactor LoopVectorizationCostModel::selectVectorizationFactor | ( | const ElementCountSet & | CandidateVFs | ) |
CandidateVFs
. If UserVF is not ZERO then this vectorization factor will be selected if vectorization is possible. Definition at line 5293 of file LoopVectorize.cpp.
References assert(), B, llvm::VectorizationFactor::Cost, llvm::SmallSet< T, N, C >::count(), llvm::dbgs(), EnableCondStoresVectorization, llvm::LoopVectorizeHints::FK_Enabled, llvm::raw_ostream::flush(), llvm::LinearPolySize< ElementCount >::getFixed(), llvm::LinearPolySize< LeafTy >::getFixedValue(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::InstructionCost::getMax(), i, I, if(), llvm::LinearPolySize< LeafTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::InstructionCost::isValid(), LLVM_DEBUG, Numbering, llvm::InnerLoopVectorizer::ORE, llvm::reportVectorizationFailure(), llvm::reportVectorizationInfo(), llvm::VectorizationFactor::ScalarCost, llvm::SmallSet< T, N, C >::size(), llvm::sort(), llvm::CallingConv::Tail, and llvm::VectorizationFactor::Width.
void LoopVectorizationCostModel::setCostBasedWideningDecision | ( | ElementCount | VF | ) |
Memory access instruction may be vectorized in more than one way.
Form of instruction after vectorization depends on cost. This function takes cost-based decisions for Load/Store instructions and collects them in a map. This decisions map is used for building the lists of loop-uniform and loop-scalar instructions. The calculated cost is saved with widening decision in order to avoid redundant calculations.
Definition at line 6765 of file LoopVectorize.cpp.
References llvm::append_range(), assert(), BB, llvm::InnerLoopVectorizer::Cost, llvm::LinearPolySize< ElementCount >::getFixed(), llvm::InstructionCost::getInvalid(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), I, llvm::DenseMapBase< DenseMap< KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >, KeyT, ValueT, DenseMapInfo< KeyT >, llvm::detail::DenseMapPair< KeyT, ValueT > >::insert(), llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::LoopVectorizationLegality::isConsecutivePtr(), llvm::LinearPolySize< LeafTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::LoopVectorizationLegality::isUniformMemOp(), llvm::InnerLoopVectorizer::Legal, llvm::SmallVectorImpl< T >::pop_back_val(), llvm::TargetTransformInfo::prefersVectorizedAddressing(), and llvm::InnerLoopVectorizer::VF.
|
inline |
Save vectorization decision W
and Cost
taken by the cost model for interleaving group Grp
and vector width VF
.
Broadcast this decicion to all instructions inside the group. But the cost will be assigned to one instruction only.
Definition at line 1382 of file LoopVectorize.cpp.
References assert(), llvm::InnerLoopVectorizer::Cost, llvm::InterleaveGroup< InstTy >::getFactor(), llvm::InterleaveGroup< InstTy >::getInsertPos(), llvm::InterleaveGroup< InstTy >::getMember(), i, I, llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::VF, and llvm::RISCVFenceField::W.
|
inline |
Save vectorization decision W
and Cost
taken by the cost model for instruction I
and vector width VF
.
Definition at line 1374 of file LoopVectorize.cpp.
References assert(), llvm::InnerLoopVectorizer::Cost, I, llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::VF, and llvm::RISCVFenceField::W.
|
inline |
Returns true if we should use strict in-order reductions for the given RdxDesc.
This is true if the -enable-strict-reductions flag is passed, the IsOrdered flag of RdxDesc is set and we do not allow reordering of FP operations.
Definition at line 1294 of file LoopVectorize.cpp.
References llvm::RecurrenceDescriptor::isOrdered().
Referenced by llvm::InnerLoopVectorizer::useOrderedReductions().
AssumptionCache* llvm::LoopVectorizationCostModel::AC |
Assumption cache.
Definition at line 1875 of file LoopVectorize.cpp.
DemandedBits* llvm::LoopVectorizationCostModel::DB |
Demanded bits analysis.
Definition at line 1872 of file LoopVectorize.cpp.
SmallPtrSet<Type *, 16> llvm::LoopVectorizationCostModel::ElementTypesInLoop |
All element types found in the loop.
Definition at line 1896 of file LoopVectorize.cpp.
const LoopVectorizeHints* llvm::LoopVectorizationCostModel::Hints |
Loop Vectorize Hint.
Definition at line 1883 of file LoopVectorize.cpp.
Referenced by llvm::InnerLoopVectorizer::emitMemRuntimeChecks(), and llvm::InnerLoopVectorizer::emitSCEVChecks().
InterleavedAccessInfo& llvm::LoopVectorizationCostModel::InterleaveInfo |
The interleave access information contains groups of interleaved accesses with the same stride and close to each other.
Definition at line 1887 of file LoopVectorize.cpp.
LoopVectorizationLegality* llvm::LoopVectorizationCostModel::Legal |
Vectorization legality.
Definition at line 1863 of file LoopVectorize.cpp.
LoopInfo* llvm::LoopVectorizationCostModel::LI |
Loop Info analysis.
Definition at line 1860 of file LoopVectorize.cpp.
OptimizationRemarkEmitter* llvm::LoopVectorizationCostModel::ORE |
Interface to emit optimization remarks.
Definition at line 1878 of file LoopVectorize.cpp.
SmallVector<VectorizationFactor, 8> llvm::LoopVectorizationCostModel::ProfitableVFs |
Profitable vector factors.
Definition at line 1899 of file LoopVectorize.cpp.
PredicatedScalarEvolution& llvm::LoopVectorizationCostModel::PSE |
Predicated scalar evolution analysis.
Definition at line 1857 of file LoopVectorize.cpp.
Definition at line 1880 of file LoopVectorize.cpp.
Loop* llvm::LoopVectorizationCostModel::TheLoop |
The loop that we evaluate.
Definition at line 1854 of file LoopVectorize.cpp.
const TargetLibraryInfo* llvm::LoopVectorizationCostModel::TLI |
Target Library Info.
Definition at line 1869 of file LoopVectorize.cpp.
const TargetTransformInfo& llvm::LoopVectorizationCostModel::TTI |
Vector target information.
Definition at line 1866 of file LoopVectorize.cpp.
SmallPtrSet<const Value *, 16> llvm::LoopVectorizationCostModel::ValuesToIgnore |
Values to ignore in the cost model.
Definition at line 1890 of file LoopVectorize.cpp.
SmallPtrSet<const Value *, 16> llvm::LoopVectorizationCostModel::VecValuesToIgnore |
Values to ignore in the cost model when VF > 1.
Definition at line 1893 of file LoopVectorize.cpp.