LLVM
13.0.0git
|
LoopVectorizationLegality checks if it is legal to vectorize a loop, and to what vectorization factor. More...
#include "llvm/Transforms/Vectorize/LoopVectorizationLegality.h"
Public Types | |
using | ReductionList = MapVector< PHINode *, RecurrenceDescriptor > |
ReductionList contains the reduction descriptors for all of the reductions that were found in the loop. More... | |
using | InductionList = MapVector< PHINode *, InductionDescriptor > |
InductionList saves induction variables and maps them to the induction descriptor. More... | |
using | RecurrenceSet = SmallPtrSet< const PHINode *, 8 > |
RecurrenceSet contains the phi nodes that are recurrences other than inductions and reductions. More... | |
Public Member Functions | |
LoopVectorizationLegality (Loop *L, PredicatedScalarEvolution &PSE, DominatorTree *DT, TargetTransformInfo *TTI, TargetLibraryInfo *TLI, AAResults *AA, Function *F, std::function< const LoopAccessInfo &(Loop &)> *GetLAA, LoopInfo *LI, OptimizationRemarkEmitter *ORE, LoopVectorizationRequirements *R, LoopVectorizeHints *H, DemandedBits *DB, AssumptionCache *AC, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI) | |
bool | canVectorize (bool UseVPlanNativePath) |
Returns true if it is legal to vectorize this loop. More... | |
bool | prepareToFoldTailByMasking () |
Return true if we can vectorize this loop while folding its tail by masking, and mark all respective loads/stores for masking. More... | |
PHINode * | getPrimaryInduction () |
Returns the primary induction variable. More... | |
ReductionList & | getReductionVars () |
Returns the reduction variables found in the loop. More... | |
InductionList & | getInductionVars () |
Returns the induction variables found in the loop. More... | |
RecurrenceSet & | getFirstOrderRecurrences () |
Return the first-order recurrences found in the loop. More... | |
DenseMap< Instruction *, Instruction * > & | getSinkAfter () |
Return the set of instructions to sink to handle first-order recurrences. More... | |
Type * | getWidestInductionType () |
Returns the widest induction type. More... | |
bool | isInductionPhi (const Value *V) |
Returns True if V is a Phi node of an induction variable in this loop. More... | |
bool | isCastedInductionVariable (const Value *V) |
Returns True if V is a cast that is part of an induction def-use chain, and had been proven to be redundant under a runtime guard (in other words, the cast has the same SCEV expression as the induction phi). More... | |
bool | isInductionVariable (const Value *V) |
Returns True if V can be considered as an induction variable in this loop. More... | |
bool | isReductionVariable (PHINode *PN) |
Returns True if PN is a reduction variable in this loop. More... | |
bool | isFirstOrderRecurrence (const PHINode *Phi) |
Returns True if Phi is a first-order recurrence in this loop. More... | |
bool | blockNeedsPredication (BasicBlock *BB) |
Return true if the block BB needs to be predicated in order for the loop to be vectorized. More... | |
int | isConsecutivePtr (Value *Ptr) |
Check if this pointer is consecutive when vectorizing. More... | |
bool | isUniform (Value *V) |
Returns true if the value V is uniform within the loop. More... | |
bool | isUniformMemOp (Instruction &I) |
A uniform memory op is a load or store which accesses the same memory location on all lanes. More... | |
const RuntimePointerChecking * | getRuntimePointerChecking () const |
Returns the information that we collected about runtime memory check. More... | |
const LoopAccessInfo * | getLAI () const |
bool | isSafeForAnyVectorWidth () const |
unsigned | getMaxSafeDepDistBytes () |
uint64_t | getMaxSafeVectorWidthInBits () const |
bool | hasStride (Value *V) |
bool | isMaskRequired (const Instruction *I) |
Returns true if vector representation of the instruction I requires mask. More... | |
unsigned | getNumStores () const |
unsigned | getNumLoads () const |
const SmallPtrSetImpl< Instruction * > & | getConditionalAssumes () const |
Returns all assume calls in predicated blocks. More... | |
LoopVectorizationLegality checks if it is legal to vectorize a loop, and to what vectorization factor.
This class does not look at the profitability of vectorization, only the legality. This class has two main kinds of checks:
Definition at line 216 of file LoopVectorizationLegality.h.
InductionList saves induction variables and maps them to the induction descriptor.
Definition at line 235 of file LoopVectorizationLegality.h.
RecurrenceSet contains the phi nodes that are recurrences other than inductions and reductions.
Definition at line 239 of file LoopVectorizationLegality.h.
ReductionList contains the reduction descriptors for all of the reductions that were found in the loop.
Definition at line 231 of file LoopVectorizationLegality.h.
|
inline |
Definition at line 218 of file LoopVectorizationLegality.h.
bool llvm::LoopVectorizationLegality::blockNeedsPredication | ( | BasicBlock * | BB | ) |
Return true if the block BB needs to be predicated in order for the loop to be vectorized.
Definition at line 882 of file LoopVectorizationLegality.cpp.
References BB, and llvm::LoopAccessInfo::blockNeedsPredication().
Referenced by llvm::LoopVectorizationCostModel::blockNeedsPredication(), llvm::LoopVectorizationCostModel::interleavedAccessCanBeWidened(), and isUniformMemOp().
bool llvm::LoopVectorizationLegality::canVectorize | ( | bool | UseVPlanNativePath | ) |
Returns true if it is legal to vectorize this loop.
This does not mean that it is profitable to vectorize this loop, only that it is legal to do so. Temporarily taking UseVPlanNativePath parameter. If true, take the new code path being implemented for outer loop vectorization (should be functional for inner loop vectorization) based on VPlan. If false, good old LV code.
Definition at line 1109 of file LoopVectorizationLegality.cpp.
References llvm::OptimizationRemarkEmitter::allowExtraAnalysis(), assert(), llvm::dbgs(), DEBUG_TYPE, llvm::LoopVectorizeHints::FK_Enabled, llvm::SCEVUnionPredicate::getComplexity(), llvm::LoopAccessInfo::getRuntimePointerChecking(), llvm::PredicatedScalarEvolution::getUnionPredicate(), LLVM_DEBUG, llvm::RuntimePointerChecking::Need, PragmaVectorizeSCEVCheckThreshold, llvm::reportVectorizationFailure(), and VectorizeSCEVCheckThreshold.
Referenced by llvm::LoopVectorizePass::processLoop().
|
inline |
Returns all assume calls in predicated blocks.
They need to be dropped when flattening the CFG.
Definition at line 352 of file LoopVectorizationLegality.h.
|
inline |
Return the first-order recurrences found in the loop.
Definition at line 265 of file LoopVectorizationLegality.h.
|
inline |
Returns the induction variables found in the loop.
Definition at line 262 of file LoopVectorizationLegality.h.
Referenced by llvm::LoopVectorizationPlanner::collectTriviallyDeadInstructions(), llvm::LoopVectorizationCostModel::collectValuesToIgnore(), llvm::InnerLoopVectorizer::createInductionResumeValues(), and llvm::InnerLoopVectorizer::widenPHIInstruction().
|
inline |
|
inline |
Definition at line 335 of file LoopVectorizationLegality.h.
References llvm::LoopAccessInfo::getMaxSafeDepDistBytes().
Referenced by llvm::LoopVectorizationCostModel::selectInterleaveCount().
|
inline |
Definition at line 337 of file LoopVectorizationLegality.h.
References llvm::LoopAccessInfo::getDepChecker(), and llvm::MemoryDepChecker::getMaxSafeVectorWidthInBits().
|
inline |
Definition at line 348 of file LoopVectorizationLegality.h.
References llvm::LoopAccessInfo::getNumLoads().
Referenced by llvm::LoopVectorizationCostModel::selectInterleaveCount().
|
inline |
Definition at line 347 of file LoopVectorizationLegality.h.
References llvm::LoopAccessInfo::getNumStores().
Referenced by llvm::LoopVectorizationCostModel::selectInterleaveCount().
|
inline |
Returns the primary induction variable.
Definition at line 256 of file LoopVectorizationLegality.h.
Referenced by llvm::LoopVectorizationPlanner::collectTriviallyDeadInstructions(), llvm::VPRecipeBuilder::createBlockInMask(), llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), llvm::InnerLoopVectorizer::createVectorizedLoopSkeleton(), and llvm::LoopVectorizationCostModel::isOptimizableIVTruncate().
|
inline |
Returns the reduction variables found in the loop.
Definition at line 259 of file LoopVectorizationLegality.h.
Referenced by llvm::LoopVectorizationCostModel::canVectorizeReductions(), llvm::LoopVectorizationCostModel::collectInLoopReductions(), llvm::LoopVectorizationCostModel::collectValuesToIgnore(), llvm::InnerLoopVectorizer::fixReduction(), llvm::LoopVectorizationCostModel::getSmallestAndWidestTypes(), prepareToFoldTailByMasking(), llvm::LoopVectorizationCostModel::selectInterleaveCount(), and llvm::VPRecipeBuilder::tryToCreateWidenRecipe().
|
inline |
Returns the information that we collected about runtime memory check.
Definition at line 325 of file LoopVectorizationLegality.h.
References llvm::LoopAccessInfo::getRuntimePointerChecking().
Referenced by llvm::LoopVectorizationCostModel::computeMaxVF(), llvm::LoopVectorizationCostModel::runtimeChecksRequired(), and llvm::LoopVectorizationCostModel::selectInterleaveCount().
|
inline |
Return the set of instructions to sink to handle first-order recurrences.
Definition at line 268 of file LoopVectorizationLegality.h.
|
inline |
Returns the widest induction type.
Definition at line 271 of file LoopVectorizationLegality.h.
Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), llvm::InnerLoopVectorizer::createVectorizedLoopSkeleton(), and llvm::InnerLoopVectorizer::getOrCreateTripCount().
|
inline |
Definition at line 341 of file LoopVectorizationLegality.h.
References llvm::LoopAccessInfo::hasStride().
Referenced by isStrideMul().
Returns True if V is a cast that is part of an induction def-use chain, and had been proven to be redundant under a runtime guard (in other words, the cast has the same SCEV expression as the induction phi).
Definition at line 869 of file LoopVectorizationLegality.cpp.
References llvm::SmallPtrSetImpl< PtrType >::count().
Referenced by isInductionVariable().
Check if this pointer is consecutive when vectorizing.
This happens when the last index of the GEP is the induction variable, or that the pointer itself is an induction variable. This check allows us to vectorize A[idx] into a wide load/store. Returns: 0 - Stride is unknown or non-consecutive. 1 - Address is consecutive. -1 - Address is consecutive, and decreasing. NOTE: This method must only be used before modifying the original scalar loop. Do not use after invoking 'createVectorizedLoopSkeleton' (PR34965).
Definition at line 379 of file LoopVectorizationLegality.cpp.
References F, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::BasicBlock::getParent(), llvm::getPtrStride(), llvm::IRPass, and llvm::shouldOptimizeForSize().
Referenced by llvm::LoopVectorizationCostModel::isLegalMaskedLoad(), llvm::LoopVectorizationCostModel::isLegalMaskedStore(), llvm::LoopVectorizationCostModel::memoryInstructionCanBeWidened(), and llvm::LoopVectorizationCostModel::setCostBasedWideningDecision().
Returns True if Phi is a first-order recurrence in this loop.
Definition at line 878 of file LoopVectorizationLegality.cpp.
References llvm::SmallPtrSetImpl< PtrType >::count().
Referenced by llvm::InnerLoopVectorizer::fixCrossIterationPHIs(), and llvm::InnerLoopVectorizer::widenPHIInstruction().
Returns True if V is a Phi node of an induction variable in this loop.
Definition at line 860 of file LoopVectorizationLegality.cpp.
References llvm::MapVector< KeyT, ValueT, MapType, VectorType >::count().
Referenced by isInductionVariable(), and llvm::LoopVectorizationCostModel::isOptimizableIVTruncate().
Returns True if V can be considered as an induction variable in this loop.
V can be the induction phi, or some redundant cast in the def-use chain of the inducion phi.
Definition at line 874 of file LoopVectorizationLegality.cpp.
References isCastedInductionVariable(), and isInductionPhi().
Referenced by getAddressAccessSCEV().
|
inline |
Returns true if vector representation of the instruction I
requires mask.
Definition at line 345 of file LoopVectorizationLegality.h.
References I.
Referenced by llvm::LoopVectorizationCostModel::interleavedAccessCanBeWidened(), llvm::LoopVectorizationCostModel::isPredicatedInst(), and llvm::LoopVectorizationCostModel::isScalarWithPredication().
|
inline |
Returns True if PN is a reduction variable in this loop.
Definition at line 287 of file LoopVectorizationLegality.h.
References llvm::MapVector< KeyT, ValueT, MapType, VectorType >::count().
Referenced by llvm::InnerLoopVectorizer::fixCrossIterationPHIs(), llvm::InnerLoopVectorizer::fixReduction(), llvm::LoopVectorizationCostModel::getSmallestAndWidestTypes(), llvm::VPRecipeBuilder::tryToCreateWidenRecipe(), and llvm::InnerLoopVectorizer::widenPHIInstruction().
|
inline |
Definition at line 331 of file LoopVectorizationLegality.h.
References llvm::LoopAccessInfo::getDepChecker(), and llvm::MemoryDepChecker::isSafeForAnyVectorWidth().
bool llvm::LoopVectorizationLegality::isUniform | ( | Value * | V | ) |
Returns true if the value V is uniform within the loop.
Definition at line 394 of file LoopVectorizationLegality.cpp.
References llvm::LoopAccessInfo::isUniform().
Referenced by isUniformMemOp().
|
inline |
A uniform memory op is a load or store which accesses the same memory location on all lanes.
Definition at line 313 of file LoopVectorizationLegality.h.
References blockNeedsPredication(), llvm::getLoadStorePointerOperand(), I, and isUniform().
Referenced by llvm::LoopVectorizationCostModel::setCostBasedWideningDecision().
bool llvm::LoopVectorizationLegality::prepareToFoldTailByMasking | ( | ) |
Return true if we can vectorize this loop while folding its tail by masking, and mark all respective loads/stores for masking.
This object's state is only modified iff this function returns true.
Definition at line 1203 of file LoopVectorizationLegality.cpp.
References BB, llvm::SmallPtrSetImpl< PtrType >::begin(), llvm::SmallPtrSetImpl< PtrType >::count(), llvm::dbgs(), llvm::SmallPtrSetImpl< PtrType >::end(), getReductionVars(), llvm::SmallPtrSetImpl< PtrType >::insert(), LLVM_DEBUG, Reduction, and llvm::Value::users().
Referenced by llvm::LoopVectorizationCostModel::computeMaxVF().