LLVM 19.0.0git
Public Types | Public Member Functions | List of all members
llvm::LoopVectorizationLegality Class Reference

LoopVectorizationLegality checks if it is legal to vectorize a loop, and to what vectorization factor. More...

#include "llvm/Transforms/Vectorize/LoopVectorizationLegality.h"

Public Types

using ReductionList = MapVector< PHINode *, RecurrenceDescriptor >
 ReductionList contains the reduction descriptors for all of the reductions that were found in the loop.
 
using InductionList = MapVector< PHINode *, InductionDescriptor >
 InductionList saves induction variables and maps them to the induction descriptor.
 
using RecurrenceSet = SmallPtrSet< const PHINode *, 8 >
 RecurrenceSet contains the phi nodes that are recurrences other than inductions and reductions.
 

Public Member Functions

 LoopVectorizationLegality (Loop *L, PredicatedScalarEvolution &PSE, DominatorTree *DT, TargetTransformInfo *TTI, TargetLibraryInfo *TLI, Function *F, LoopAccessInfoManager &LAIs, LoopInfo *LI, OptimizationRemarkEmitter *ORE, LoopVectorizationRequirements *R, LoopVectorizeHints *H, DemandedBits *DB, AssumptionCache *AC, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI)
 
bool canVectorize (bool UseVPlanNativePath)
 Returns true if it is legal to vectorize this loop.
 
bool canVectorizeFPMath (bool EnableStrictReductions)
 Returns true if it is legal to vectorize the FP math operations in this loop.
 
bool prepareToFoldTailByMasking ()
 Return true if we can vectorize this loop while folding its tail by masking, and mark all respective loads/stores for masking.
 
PHINodegetPrimaryInduction ()
 Returns the primary induction variable.
 
const ReductionListgetReductionVars () const
 Returns the reduction variables found in the loop.
 
const InductionListgetInductionVars () const
 Returns the induction variables found in the loop.
 
RecurrenceSetgetFixedOrderRecurrences ()
 Return the fixed-order recurrences found in the loop.
 
TypegetWidestInductionType ()
 Returns the widest induction type.
 
bool isInvariantStoreOfReduction (StoreInst *SI)
 Returns True if given store is a final invariant store of one of the reductions found in the loop.
 
bool isInvariantAddressOfReduction (Value *V)
 Returns True if given address is invariant and is used to store recurrent expression.
 
bool isInductionPhi (const Value *V) const
 Returns True if V is a Phi node of an induction variable in this loop.
 
const InductionDescriptorgetIntOrFpInductionDescriptor (PHINode *Phi) const
 Returns a pointer to the induction descriptor, if Phi is an integer or floating point induction.
 
const InductionDescriptorgetPointerInductionDescriptor (PHINode *Phi) const
 Returns a pointer to the induction descriptor, if Phi is pointer induction.
 
bool isCastedInductionVariable (const Value *V) const
 Returns True if V is a cast that is part of an induction def-use chain, and had been proven to be redundant under a runtime guard (in other words, the cast has the same SCEV expression as the induction phi).
 
bool isInductionVariable (const Value *V) const
 Returns True if V can be considered as an induction variable in this loop.
 
bool isReductionVariable (PHINode *PN) const
 Returns True if PN is a reduction variable in this loop.
 
bool isFixedOrderRecurrence (const PHINode *Phi) const
 Returns True if Phi is a fixed-order recurrence in this loop.
 
bool blockNeedsPredication (BasicBlock *BB) const
 Return true if the block BB needs to be predicated in order for the loop to be vectorized.
 
int isConsecutivePtr (Type *AccessTy, Value *Ptr) const
 Check if this pointer is consecutive when vectorizing.
 
bool isInvariant (Value *V) const
 Returns true if value V is uniform across VF lanes, when VF is provided, and otherwise if V is invariant across all loop iterations.
 
bool isUniform (Value *V, ElementCount VF) const
 Returns true if value V is uniform across VF lanes, when VF is provided, and otherwise if V is invariant across all loop iterations.
 
bool isUniformMemOp (Instruction &I, ElementCount VF) const
 A uniform memory op is a load or store which accesses the same memory location on all VF lanes, if VF is provided and otherwise if the memory location is invariant.
 
const RuntimePointerCheckinggetRuntimePointerChecking () const
 Returns the information that we collected about runtime memory check.
 
const LoopAccessInfogetLAI () const
 
bool isSafeForAnyVectorWidth () const
 
uint64_t getMaxSafeVectorWidthInBits () const
 
bool isMaskRequired (const Instruction *I) const
 Returns true if vector representation of the instruction I requires mask.
 
bool hasVectorCallVariants () const
 Returns true if there is at least one function call in the loop which has a vectorized variant available.
 
unsigned getNumStores () const
 
unsigned getNumLoads () const
 
PredicatedScalarEvolutiongetPredicatedScalarEvolution () const
 
LoopgetLoop () const
 
LoopInfogetLoopInfo () const
 
AssumptionCachegetAssumptionCache () const
 
ScalarEvolutiongetScalarEvolution () const
 
DominatorTreegetDominatorTree () const
 

Detailed Description

LoopVectorizationLegality checks if it is legal to vectorize a loop, and to what vectorization factor.

This class does not look at the profitability of vectorization, only the legality. This class has two main kinds of checks:

Definition at line 240 of file LoopVectorizationLegality.h.

Member Typedef Documentation

◆ InductionList

InductionList saves induction variables and maps them to the induction descriptor.

Definition at line 258 of file LoopVectorizationLegality.h.

◆ RecurrenceSet

RecurrenceSet contains the phi nodes that are recurrences other than inductions and reductions.

Definition at line 262 of file LoopVectorizationLegality.h.

◆ ReductionList

ReductionList contains the reduction descriptors for all of the reductions that were found in the loop.

Definition at line 254 of file LoopVectorizationLegality.h.

Constructor & Destructor Documentation

◆ LoopVectorizationLegality()

llvm::LoopVectorizationLegality::LoopVectorizationLegality ( Loop L,
PredicatedScalarEvolution PSE,
DominatorTree DT,
TargetTransformInfo TTI,
TargetLibraryInfo TLI,
Function F,
LoopAccessInfoManager LAIs,
LoopInfo LI,
OptimizationRemarkEmitter ORE,
LoopVectorizationRequirements R,
LoopVectorizeHints H,
DemandedBits DB,
AssumptionCache AC,
BlockFrequencyInfo BFI,
ProfileSummaryInfo PSI 
)
inline

Definition at line 242 of file LoopVectorizationLegality.h.

Member Function Documentation

◆ blockNeedsPredication()

bool llvm::LoopVectorizationLegality::blockNeedsPredication ( BasicBlock BB) const

◆ canVectorize()

bool llvm::LoopVectorizationLegality::canVectorize ( bool  UseVPlanNativePath)

Returns true if it is legal to vectorize this loop.

This does not mean that it is profitable to vectorize this loop, only that it is legal to do so. Temporarily taking UseVPlanNativePath parameter. If true, take the new code path being implemented for outer loop vectorization (should be functional for inner loop vectorization) based on VPlan. If false, good old LV code.

Definition at line 1442 of file LoopVectorizationLegality.cpp.

References llvm::OptimizationRemarkEmitter::allowExtraAnalysis(), assert(), llvm::dbgs(), DEBUG_TYPE, llvm::LoopVectorizeHints::FK_Enabled, llvm::PredicatedScalarEvolution::getBackedgeTakenCount(), llvm::SCEVPredicate::getComplexity(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::Value::getName(), llvm::LoopBase< BlockT, LoopT >::getNumBlocks(), llvm::PredicatedScalarEvolution::getPredicate(), llvm::LoopAccessInfo::getRuntimePointerChecking(), llvm::LoopBase< BlockT, LoopT >::isInnermost(), LLVM_DEBUG, llvm::RuntimePointerChecking::Need, PragmaVectorizeSCEVCheckThreshold, llvm::reportVectorizationFailure(), and VectorizeSCEVCheckThreshold.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ canVectorizeFPMath()

bool llvm::LoopVectorizationLegality::canVectorizeFPMath ( bool  EnableStrictReductions)

Returns true if it is legal to vectorize the FP math operations in this loop.

Vectorizing is legal if we allow reordering of FP operations, or if we can use in-order reductions.

Definition at line 1161 of file LoopVectorizationLegality.cpp.

References llvm::all_of(), llvm::any_of(), llvm::LoopVectorizationRequirements::getExactFPInst(), llvm::InductionDescriptor::getExactFPMathInst(), getInductionVars(), getReductionVars(), llvm::RecurrenceDescriptor::hasExactFPMath(), llvm::RecurrenceDescriptor::isOrdered(), and Reduction.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ getAssumptionCache()

AssumptionCache * llvm::LoopVectorizationLegality::getAssumptionCache ( ) const
inline

◆ getDominatorTree()

DominatorTree * llvm::LoopVectorizationLegality::getDominatorTree ( ) const
inline

◆ getFixedOrderRecurrences()

RecurrenceSet & llvm::LoopVectorizationLegality::getFixedOrderRecurrences ( )
inline

Return the fixed-order recurrences found in the loop.

Definition at line 293 of file LoopVectorizationLegality.h.

Referenced by llvm::AArch64TTIImpl::preferPredicateOverEpilogue().

◆ getInductionVars()

const InductionList & llvm::LoopVectorizationLegality::getInductionVars ( ) const
inline

◆ getIntOrFpInductionDescriptor()

const InductionDescriptor * llvm::LoopVectorizationLegality::getIntOrFpInductionDescriptor ( PHINode Phi) const

Returns a pointer to the induction descriptor, if Phi is an integer or floating point induction.

Definition at line 1217 of file LoopVectorizationLegality.cpp.

References llvm::MapVector< KeyT, ValueT, MapType, VectorType >::find(), getInductionVars(), llvm::InductionDescriptor::IK_FpInduction, llvm::InductionDescriptor::IK_IntInduction, and isInductionPhi().

◆ getLAI()

const LoopAccessInfo * llvm::LoopVectorizationLegality::getLAI ( ) const
inline

◆ getLoop()

Loop * llvm::LoopVectorizationLegality::getLoop ( ) const
inline

◆ getLoopInfo()

LoopInfo * llvm::LoopVectorizationLegality::getLoopInfo ( ) const
inline

◆ getMaxSafeVectorWidthInBits()

uint64_t llvm::LoopVectorizationLegality::getMaxSafeVectorWidthInBits ( ) const
inline

◆ getNumLoads()

unsigned llvm::LoopVectorizationLegality::getNumLoads ( ) const
inline

◆ getNumStores()

unsigned llvm::LoopVectorizationLegality::getNumStores ( ) const
inline

◆ getPointerInductionDescriptor()

const InductionDescriptor * llvm::LoopVectorizationLegality::getPointerInductionDescriptor ( PHINode Phi) const

Returns a pointer to the induction descriptor, if Phi is pointer induction.

Definition at line 1228 of file LoopVectorizationLegality.cpp.

References llvm::MapVector< KeyT, ValueT, MapType, VectorType >::find(), getInductionVars(), llvm::InductionDescriptor::IK_PtrInduction, and isInductionPhi().

◆ getPredicatedScalarEvolution()

PredicatedScalarEvolution * llvm::LoopVectorizationLegality::getPredicatedScalarEvolution ( ) const
inline

◆ getPrimaryInduction()

PHINode * llvm::LoopVectorizationLegality::getPrimaryInduction ( )
inline

◆ getReductionVars()

const ReductionList & llvm::LoopVectorizationLegality::getReductionVars ( ) const
inline

◆ getRuntimePointerChecking()

const RuntimePointerChecking * llvm::LoopVectorizationLegality::getRuntimePointerChecking ( ) const
inline

◆ getScalarEvolution()

ScalarEvolution * llvm::LoopVectorizationLegality::getScalarEvolution ( ) const
inline

◆ getWidestInductionType()

Type * llvm::LoopVectorizationLegality::getWidestInductionType ( )
inline

Returns the widest induction type.

Definition at line 296 of file LoopVectorizationLegality.h.

Referenced by llvm::LoopVectorizationPlanner::selectEpilogueVectorizationFactor().

◆ hasVectorCallVariants()

bool llvm::LoopVectorizationLegality::hasVectorCallVariants ( ) const
inline

Returns true if there is at least one function call in the loop which has a vectorized variant available.

Definition at line 385 of file LoopVectorizationLegality.h.

◆ isCastedInductionVariable()

bool llvm::LoopVectorizationLegality::isCastedInductionVariable ( const Value V) const

Returns True if V is a cast that is part of an induction def-use chain, and had been proven to be redundant under a runtime guard (in other words, the cast has the same SCEV expression as the induction phi).

Definition at line 1237 of file LoopVectorizationLegality.cpp.

References llvm::SmallPtrSetImpl< PtrType >::count().

Referenced by isInductionVariable().

◆ isConsecutivePtr()

int llvm::LoopVectorizationLegality::isConsecutivePtr ( Type AccessTy,
Value Ptr 
) const

Check if this pointer is consecutive when vectorizing.

This happens when the last index of the GEP is the induction variable, or that the pointer itself is an induction variable. This check allows us to vectorize A[idx] into a wide load/store. Returns: 0 - Stride is unknown or non-consecutive. 1 - Address is consecutive. -1 - Address is consecutive, and decreasing. NOTE: This method must only be used before modifying the original scalar loop. Do not use after invoking 'createVectorizedLoopSkeleton' (PR34965).

Definition at line 453 of file LoopVectorizationLegality.cpp.

References F, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::BasicBlock::getParent(), llvm::getPtrStride(), llvm::LoopAccessInfo::getSymbolicStrides(), llvm::IRPass, Ptr, and llvm::shouldOptimizeForSize().

Referenced by llvm::LoopVectorizationCostModel::isLegalMaskedLoad(), llvm::LoopVectorizationCostModel::isLegalMaskedStore(), llvm::LoopVectorizationCostModel::memoryInstructionCanBeWidened(), and llvm::LoopVectorizationCostModel::setCostBasedWideningDecision().

◆ isFixedOrderRecurrence()

bool llvm::LoopVectorizationLegality::isFixedOrderRecurrence ( const PHINode Phi) const

Returns True if Phi is a fixed-order recurrence in this loop.

Definition at line 1247 of file LoopVectorizationLegality.cpp.

References llvm::SmallPtrSetImpl< PtrType >::count().

Referenced by llvm::InnerLoopVectorizer::fixVectorizedLoop(), and llvm::VPRecipeBuilder::tryToCreateWidenRecipe().

◆ isInductionPhi()

bool llvm::LoopVectorizationLegality::isInductionPhi ( const Value V) const

◆ isInductionVariable()

bool llvm::LoopVectorizationLegality::isInductionVariable ( const Value V) const

Returns True if V can be considered as an induction variable in this loop.

V can be the induction phi, or some redundant cast in the def-use chain of the inducion phi.

Definition at line 1243 of file LoopVectorizationLegality.cpp.

References isCastedInductionVariable(), and isInductionPhi().

◆ isInvariant()

bool llvm::LoopVectorizationLegality::isInvariant ( Value V) const

Returns true if value V is uniform across VF lanes, when VF is provided, and otherwise if V is invariant across all loop iterations.

Definition at line 474 of file LoopVectorizationLegality.cpp.

References llvm::LoopAccessInfo::isInvariant().

Referenced by llvm::LoopVectorizationCostModel::getDivRemSpeculationCost(), llvm::LoopVectorizationCostModel::isPredicatedInst(), and isUniform().

◆ isInvariantAddressOfReduction()

bool llvm::LoopVectorizationLegality::isInvariantAddressOfReduction ( Value V)

◆ isInvariantStoreOfReduction()

bool llvm::LoopVectorizationLegality::isInvariantStoreOfReduction ( StoreInst SI)

Returns True if given store is a final invariant store of one of the reductions found in the loop.

Definition at line 1187 of file LoopVectorizationLegality.cpp.

References llvm::any_of(), getReductionVars(), llvm::RecurrenceDescriptor::IntermediateStore, and Reduction.

◆ isMaskRequired()

bool llvm::LoopVectorizationLegality::isMaskRequired ( const Instruction I) const
inline

Returns true if vector representation of the instruction I requires mask.

Definition at line 379 of file LoopVectorizationLegality.h.

References I.

Referenced by llvm::LoopVectorizationCostModel::interleavedAccessCanBeWidened(), llvm::LoopVectorizationCostModel::isPredicatedInst(), and llvm::LoopVectorizationCostModel::setVectorizedCallDecision().

◆ isReductionVariable()

bool llvm::LoopVectorizationLegality::isReductionVariable ( PHINode PN) const
inline

◆ isSafeForAnyVectorWidth()

bool llvm::LoopVectorizationLegality::isSafeForAnyVectorWidth ( ) const
inline

◆ isUniform()

bool llvm::LoopVectorizationLegality::isUniform ( Value V,
ElementCount  VF 
) const

◆ isUniformMemOp()

bool llvm::LoopVectorizationLegality::isUniformMemOp ( Instruction I,
ElementCount  VF 
) const

A uniform memory op is a load or store which accesses the same memory location on all VF lanes, if VF is provided and otherwise if the memory location is invariant.

Definition at line 601 of file LoopVectorizationLegality.cpp.

References blockNeedsPredication(), llvm::getLoadStorePointerOperand(), I, isUniform(), and Ptr.

Referenced by llvm::LoopVectorizationCostModel::setCostBasedWideningDecision().

◆ prepareToFoldTailByMasking()

bool llvm::LoopVectorizationLegality::prepareToFoldTailByMasking ( )

Return true if we can vectorize this loop while folding its tail by masking, and mark all respective loads/stores for masking.

This object's state is only modified iff this function returns true.

Definition at line 1546 of file LoopVectorizationLegality.cpp.

References llvm::SmallPtrSetImpl< PtrType >::begin(), llvm::LoopBase< BlockT, LoopT >::blocks(), llvm::LoopBase< BlockT, LoopT >::contains(), llvm::SmallPtrSetImpl< PtrType >::count(), llvm::dbgs(), llvm::SmallPtrSetImpl< PtrType >::end(), getInductionVars(), getReductionVars(), llvm::SmallPtrSetImpl< PtrType >::insert(), LLVM_DEBUG, Reduction, and llvm::Value::users().

Referenced by llvm::LoopVectorizationCostModel::setTailFoldingStyles().


The documentation for this class was generated from the following files: