LLVM 18.0.0git
|
A specialized derived class of inner loop vectorizer that performs vectorization of main loops in the process of vectorizing loops and their epilogues. More...
Public Member Functions | |
EpilogueVectorizerMainLoop (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationLegality *LVL, llvm::LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &Check) | |
std::pair< BasicBlock *, Value * > | createEpilogueVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs) final |
Implements the interface for creating a vectorized skeleton using the main loop strategy (ie the first pass of vplan execution). | |
![]() | |
InnerLoopAndEpilogueVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationLegality *LVL, llvm::LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &Checks) | |
std::pair< BasicBlock *, Value * > | createVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs) final |
Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder. | |
virtual std::pair< BasicBlock *, Value * > | createEpilogueVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs)=0 |
The interface for creating a vectorized skeleton using one of two different strategies, each corresponding to one execution of the vplan as described above. | |
![]() | |
InnerLoopVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, ElementCount VecWidth, ElementCount MinProfitableTripCount, unsigned UnrollFactor, LoopVectorizationLegality *LVL, LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks) | |
virtual | ~InnerLoopVectorizer ()=default |
virtual std::pair< BasicBlock *, Value * > | createVectorizedLoopSkeleton (const SCEV2ValueTy &ExpandedSCEVs) |
Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder. | |
void | fixVectorizedLoop (VPTransformState &State, VPlan &Plan) |
Fix the vectorized code, taking care of header phi's, live-outs, and more. | |
bool | areSafetyChecksAdded () |
void | scalarizeInstruction (const Instruction *Instr, VPReplicateRecipe *RepRecipe, const VPIteration &Instance, VPTransformState &State) |
A helper function to scalarize a single Instruction in the innermost loop. | |
void | vectorizeInterleaveGroup (const InterleaveGroup< Instruction > *Group, ArrayRef< VPValue * > VPDefs, VPTransformState &State, VPValue *Addr, ArrayRef< VPValue * > StoredValues, VPValue *BlockInMask, bool NeedsMaskForGaps) |
Try to vectorize interleaved access group Group with the base address given in Addr , optionally masking the vector operations if BlockInMask is non-null. | |
void | fixNonInductionPHIs (VPlan &Plan, VPTransformState &State) |
Fix the non-induction PHIs in Plan . | |
bool | useOrderedReductions (const RecurrenceDescriptor &RdxDesc) |
Returns true if the reordering of FP operations is not allowed, but we are able to vectorize with strict in-order reductions for the given RdxDesc. | |
PHINode * | getReductionResumeValue (const RecurrenceDescriptor &RdxDesc) |
PHINode * | createInductionResumeValue (PHINode *OrigPhi, const InductionDescriptor &ID, Value *Step, ArrayRef< BasicBlock * > BypassBlocks, std::pair< BasicBlock *, Value * > AdditionalBypass={nullptr, nullptr}) |
Create a new phi node for the induction variable OrigPhi to resume iteration count in the scalar epilogue, from where the vectorized loop left off. | |
Value * | getTripCount () const |
Returns the original loop trip count. | |
void | setTripCount (Value *TC) |
Used to set the trip count after ILV's construction and after the preheader block has been executed. | |
Protected Member Functions | |
BasicBlock * | emitIterationCountCheck (BasicBlock *Bypass, bool ForEpilogue) |
Emits an iteration count bypass check once for the main loop (when ForEpilogue is false) and once for the epilogue loop (when ForEpilogue is true). | |
void | printDebugTracesAtStart () override |
Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested. | |
void | printDebugTracesAtEnd () override |
![]() | |
void | fixupIVUsers (PHINode *OrigPhi, const InductionDescriptor &II, Value *VectorTripCount, Value *EndValue, BasicBlock *MiddleBlock, BasicBlock *VectorHeader, VPlan &Plan, VPTransformState &State) |
Set up the values of the IVs correctly when exiting the vector loop. | |
void | fixCrossIterationPHIs (VPTransformState &State) |
Handle all cross-iteration phis in the header. | |
void | fixFixedOrderRecurrence (VPFirstOrderRecurrencePHIRecipe *PhiR, VPTransformState &State) |
Create the exit value of first order recurrences in the middle block and update their users. | |
void | fixReduction (VPReductionPHIRecipe *Phi, VPTransformState &State) |
Create code for the loop exit value of the reduction. | |
void | sinkScalarOperands (Instruction *PredInst) |
Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it. | |
void | truncateToMinimalBitwidths (VPTransformState &State) |
Shrinks vector element sizes to the smallest bitwidth they can be legally represented as. | |
Value * | getOrCreateVectorTripCount (BasicBlock *InsertBlock) |
Returns (and creates if needed) the trip count of the widened loop. | |
Value * | createBitOrPointerCast (Value *V, VectorType *DstVTy, const DataLayout &DL) |
Returns a bitcasted value to the requested vector type. | |
void | emitIterationCountCheck (BasicBlock *Bypass) |
Emit a bypass check to see if the vector trip count is zero, including if it overflows. | |
BasicBlock * | emitSCEVChecks (BasicBlock *Bypass) |
Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct. | |
BasicBlock * | emitMemRuntimeChecks (BasicBlock *Bypass) |
Emit bypass checks to check any memory assumptions we may have made. | |
void | createVectorLoopSkeleton (StringRef Prefix) |
Emit basic blocks (prefixed with Prefix ) for the iteration check, vector loop preheader, middle block and scalar preheader. | |
void | createInductionResumeValues (const SCEV2ValueTy &ExpandedSCEVs, std::pair< BasicBlock *, Value * > AdditionalBypass={nullptr, nullptr}) |
Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off. | |
BasicBlock * | completeLoopSkeleton () |
Complete the loop skeleton by adding debug MDs, creating appropriate conditional branches in the middle block, preparing the builder and running the verifier. | |
void | collectPoisonGeneratingRecipes (VPTransformState &State) |
Collect poison-generating recipes that may generate a poison value that is used after vectorization, even when their operands are not poison. | |
virtual void | printDebugTracesAtStart () |
Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested. | |
virtual void | printDebugTracesAtEnd () |
Additional Inherited Members | |
![]() | |
using | VectorParts = SmallVector< Value *, 2 > |
A type for vectorized values in the new loop. | |
![]() | |
EpilogueLoopVectorizationInfo & | EPI |
Holds and updates state information required to vectorize the main loop and its epilogue in two separate passes. | |
![]() | |
using | PhiVector = SmallVector< PHINode *, 4 > |
A small list of PHINodes. | |
using | ScalarParts = SmallVector< SmallVector< Value *, 4 >, 2 > |
A type for scalarized values in the new loop. | |
![]() | |
Loop * | OrigLoop |
The original loop. | |
PredicatedScalarEvolution & | PSE |
A wrapper around ScalarEvolution used to add runtime SCEV checks. | |
LoopInfo * | LI |
Loop Info. | |
DominatorTree * | DT |
Dominator Tree. | |
const TargetLibraryInfo * | TLI |
Target Library Info. | |
const TargetTransformInfo * | TTI |
Target Transform Info. | |
AssumptionCache * | AC |
Assumption Cache. | |
OptimizationRemarkEmitter * | ORE |
Interface to emit optimization remarks. | |
ElementCount | VF |
The vectorization SIMD factor to use. | |
ElementCount | MinProfitableTripCount |
unsigned | UF |
The vectorization unroll factor to use. | |
IRBuilder | Builder |
The builder that we use. | |
BasicBlock * | LoopVectorPreHeader |
The vector-loop preheader. | |
BasicBlock * | LoopScalarPreHeader |
The scalar-loop preheader. | |
BasicBlock * | LoopMiddleBlock |
Middle Block between the vector and the scalar. | |
BasicBlock * | LoopExitBlock |
The unique ExitBlock of the scalar loop if one exists. | |
BasicBlock * | LoopScalarBody |
The scalar loop body. | |
SmallVector< BasicBlock *, 4 > | LoopBypassBlocks |
A list of all bypass blocks. The first block is the entry of the loop. | |
SmallVector< Instruction *, 4 > | PredicatedInstructions |
Store instructions that were predicated. | |
Value * | TripCount = nullptr |
Trip count of the original loop. | |
Value * | VectorTripCount = nullptr |
Trip count of the widened loop (TripCount - TripCount % (VF*UF)) | |
LoopVectorizationLegality * | Legal |
The legality analysis. | |
LoopVectorizationCostModel * | Cost |
The profitablity analysis. | |
bool | AddedSafetyChecks = false |
DenseMap< PHINode *, Value * > | IVEndValues |
BlockFrequencyInfo * | BFI |
BFI and PSI are used to check for profile guided size optimizations. | |
ProfileSummaryInfo * | PSI |
bool | OptForSizeBasedOnProfile |
GeneratedRTChecks & | RTChecks |
Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable. | |
SmallMapVector< const RecurrenceDescriptor *, PHINode *, 4 > | ReductionResumeValues |
A specialized derived class of inner loop vectorizer that performs vectorization of main loops in the process of vectorizing loops and their epilogues.
Definition at line 891 of file LoopVectorize.cpp.
|
inline |
Definition at line 893 of file LoopVectorize.cpp.
References Check.
|
finalvirtual |
Implements the interface for creating a vectorized skeleton using the main loop strategy (ie the first pass of vplan execution).
This function is partially responsible for generating the control flow depicted in https://llvm.org/docs/Vectorizers.html#epilogue-vectorization.
Implements llvm::InnerLoopAndEpilogueVectorizer.
Definition at line 7843 of file LoopVectorize.cpp.
References llvm::InnerLoopVectorizer::completeLoopSkeleton(), llvm::InnerLoopVectorizer::createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::InnerLoopVectorizer::emitMemRuntimeChecks(), llvm::InnerLoopVectorizer::emitSCEVChecks(), llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueIterationCountCheck, llvm::InnerLoopVectorizer::getOrCreateVectorTripCount(), llvm::InnerLoopVectorizer::LoopScalarPreHeader, llvm::InnerLoopVectorizer::LoopVectorPreHeader, llvm::EpilogueLoopVectorizationInfo::MainLoopIterationCountCheck, llvm::EpilogueLoopVectorizationInfo::MemSafetyCheck, llvm::EpilogueLoopVectorizationInfo::SCEVSafetyCheck, llvm::Value::setName(), and llvm::EpilogueLoopVectorizationInfo::VectorTripCount.
|
protected |
Emits an iteration count bypass check once for the main loop (when ForEpilogue
is false) and once for the epilogue loop (when ForEpilogue
is true).
Definition at line 7900 of file LoopVectorize.cpp.
References assert(), llvm::InnerLoopVectorizer::Builder, llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), llvm::BranchInst::Create(), llvm::IRBuilderBase::CreateICmp(), llvm::createStepForVF(), llvm::InnerLoopVectorizer::DT, llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueUF, llvm::EpilogueLoopVectorizationInfo::EpilogueVF, llvm::DomTreeNodeBase< NodeT >::getIDom(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::DominatorTreeBase< NodeT, IsPostDom >::getNode(), llvm::BasicBlock::getTerminator(), llvm::InnerLoopVectorizer::getTripCount(), llvm::Value::getType(), llvm::hasBranchWeightMD(), llvm::CmpInst::ICMP_ULE, llvm::CmpInst::ICMP_ULT, llvm::ElementCount::isVector(), llvm::InnerLoopVectorizer::LI, llvm::InnerLoopVectorizer::LoopBypassBlocks, llvm::InnerLoopVectorizer::LoopExitBlock, llvm::InnerLoopVectorizer::LoopVectorPreHeader, MinItersBypassWeights, llvm::InnerLoopVectorizer::OrigLoop, P, llvm::DominatorTreeBase< NodeT, IsPostDom >::properlyDominates(), llvm::ReplaceInstWithInst(), llvm::setBranchWeights(), llvm::Value::setName(), llvm::SplitBlock(), llvm::EpilogueLoopVectorizationInfo::TripCount, llvm::InnerLoopVectorizer::UF, and llvm::InnerLoopVectorizer::VF.
Referenced by createEpilogueVectorizedLoopSkeleton().
|
overrideprotectedvirtual |
Reimplemented from llvm::InnerLoopVectorizer.
Definition at line 7892 of file LoopVectorize.cpp.
References llvm::dbgs(), DEBUG_WITH_TYPE, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::BasicBlock::getParent(), llvm::InnerLoopVectorizer::OrigLoop, and VerboseDebug.
|
overrideprotectedvirtual |
Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.
Reimplemented from llvm::InnerLoopVectorizer.
Definition at line 7882 of file LoopVectorize.cpp.
References llvm::dbgs(), llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueUF, llvm::EpilogueLoopVectorizationInfo::EpilogueVF, LLVM_DEBUG, llvm::EpilogueLoopVectorizationInfo::MainLoopUF, and llvm::EpilogueLoopVectorizationInfo::MainLoopVF.