A specialized derived class of inner loop vectorizer that performs vectorization of main loops in the process of vectorizing loops and their epilogues. More...
Public Member Functions | |
| EpilogueVectorizerMainLoop (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationLegality *LVL, llvm::LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &Check) | |
| BasicBlock * | createEpilogueVectorizedLoopSkeleton () final override |
| Implements the interface for creating a vectorized skeleton using the main loop strategy (ie the first pass of vplan execution). More... | |
 Public Member Functions inherited from llvm::InnerLoopAndEpilogueVectorizer | |
| InnerLoopAndEpilogueVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, EpilogueLoopVectorizationInfo &EPI, LoopVectorizationLegality *LVL, llvm::LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &Checks) | |
| BasicBlock * | createVectorizedLoopSkeleton () final override |
| Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder. More... | |
| Public Member Functions inherited from llvm::InnerLoopVectorizer | |
| InnerLoopVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, ElementCount VecWidth, unsigned UnrollFactor, LoopVectorizationLegality *LVL, LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks) | |
| virtual | ~InnerLoopVectorizer ()=default |
| void | widenInstruction (Instruction &I, VPValue *Def, VPUser &Operands, VPTransformState &State) |
| Widen a single instruction within the innermost loop. More... | |
| void | widenCallInstruction (CallInst &I, VPValue *Def, VPUser &ArgOperands, VPTransformState &State) |
| Widen a single call instruction within the innermost loop. More... | |
| void | widenSelectInstruction (SelectInst &I, VPValue *VPDef, VPUser &Operands, bool InvariantCond, VPTransformState &State) |
| Widen a single select instruction within the innermost loop. More... | |
| void | fixVectorizedLoop (VPTransformState &State) |
| Fix the vectorized code, taking care of header phi's, live-outs, and more. More... | |
| bool | areSafetyChecksAdded () |
| void | widenGEP (GetElementPtrInst *GEP, VPValue *VPDef, VPUser &Indices, unsigned UF, ElementCount VF, bool IsPtrLoopInvariant, SmallBitVector &IsIndexLoopInvariant, VPTransformState &State) |
| Vectorize a single GetElementPtrInst based on information gathered and decisions taken during planning. More... | |
| void | widenPHIInstruction (Instruction *PN, RecurrenceDescriptor *RdxDesc, VPWidenPHIRecipe *PhiR, VPTransformState &State) |
| Vectorize a single PHINode in a block. More... | |
| void | scalarizeInstruction (Instruction *Instr, VPValue *Def, VPUser &Operands, const VPIteration &Instance, bool IfPredicateInstr, VPTransformState &State) |
| A helper function to scalarize a single Instruction in the innermost loop. More... | |
| void | widenIntOrFpInduction (PHINode *IV, Value *Start, TruncInst *Trunc, VPValue *Def, VPValue *CastDef, VPTransformState &State) |
Widen an integer or floating-point induction variable IV. More... | |
| void | packScalarIntoVectorValue (VPValue *Def, const VPIteration &Instance, VPTransformState &State) |
Construct the vector value of a scalarized value V one lane at a time. More... | |
| void | vectorizeInterleaveGroup (const InterleaveGroup< Instruction > *Group, ArrayRef< VPValue * > VPDefs, VPTransformState &State, VPValue *Addr, ArrayRef< VPValue * > StoredValues, VPValue *BlockInMask=nullptr) |
Try to vectorize interleaved access group Group with the base address given in Addr, optionally masking the vector operations if BlockInMask is non-null. More... | |
| void | vectorizeMemoryInstruction (Instruction *Instr, VPTransformState &State, VPValue *Def, VPValue *Addr, VPValue *StoredValue, VPValue *BlockInMask) |
Vectorize Load and Store instructions with the base address given in Addr, optionally masking the vector operations if BlockInMask is non-null. More... | |
| void | setDebugLocFromInst (IRBuilder<> &B, const Value *Ptr) |
| Set the debug location in the builder using the debug location in the instruction. More... | |
| void | fixNonInductionPHIs (VPTransformState &State) |
| Fix the non-induction PHIs in the OrigPHIsToFix vector. More... | |
| virtual Value * | getBroadcastInstrs (Value *V) |
| Create a broadcast instruction. More... | |
Protected Member Functions | |
| BasicBlock * | emitMinimumIterationCountCheck (Loop *L, BasicBlock *Bypass, bool ForEpilogue) |
Emits an iteration count bypass check once for the main loop (when ForEpilogue is false) and once for the epilogue loop (when ForEpilogue is true). More... | |
| void | printDebugTracesAtStart () override |
| Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested. More... | |
| void | printDebugTracesAtEnd () override |
| Protected Member Functions inherited from llvm::InnerLoopVectorizer | |
| void | fixupIVUsers (PHINode *OrigPhi, const InductionDescriptor &II, Value *CountRoundDown, Value *EndValue, BasicBlock *MiddleBlock) |
| Set up the values of the IVs correctly when exiting the vector loop. More... | |
| PHINode * | createInductionVariable (Loop *L, Value *Start, Value *End, Value *Step, Instruction *DL) |
| Create a new induction variable inside L. More... | |
| void | fixCrossIterationPHIs (VPTransformState &State) |
| Handle all cross-iteration phis in the header. More... | |
| void | fixFirstOrderRecurrence (PHINode *Phi, VPTransformState &State) |
| Fix a first-order recurrence. More... | |
| void | fixReduction (PHINode *Phi, VPTransformState &State) |
| Fix a reduction cross-iteration phi. More... | |
| void | clearReductionWrapFlags (RecurrenceDescriptor &RdxDesc, VPTransformState &State) |
| Clear NSW/NUW flags from reduction instructions if necessary. More... | |
| void | fixLCSSAPHIs (VPTransformState &State) |
| Fixup the LCSSA phi nodes in the unique exit block. More... | |
| void | sinkScalarOperands (Instruction *PredInst) |
| Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it. More... | |
| void | truncateToMinimalBitwidths (VPTransformState &State) |
| Shrinks vector element sizes to the smallest bitwidth they can be legally represented as. More... | |
| virtual Value * | getStepVector (Value *Val, int StartIdx, Value *Step, Instruction::BinaryOps Opcode=Instruction::BinaryOpsEnd) |
| This function adds (StartIdx * Step, (StartIdx + 1) * Step, (StartIdx + 2) * Step, ...) to each vector element of Val. More... | |
| void | buildScalarSteps (Value *ScalarIV, Value *Step, Instruction *EntryVal, const InductionDescriptor &ID, VPValue *Def, VPValue *CastDef, VPTransformState &State) |
| Compute scalar induction steps. More... | |
| void | createVectorIntOrFpInductionPHI (const InductionDescriptor &II, Value *Step, Value *Start, Instruction *EntryVal, VPValue *Def, VPValue *CastDef, VPTransformState &State) |
| Create a vector induction phi node based on an existing scalar one. More... | |
| bool | shouldScalarizeInstruction (Instruction *I) const |
Returns true if an instruction I should be scalarized instead of vectorized for the chosen vectorization factor. More... | |
| bool | needsScalarInduction (Instruction *IV) const |
Returns true if we should generate a scalar version of IV. More... | |
| void | recordVectorLoopValueForInductionCast (const InductionDescriptor &ID, const Instruction *EntryVal, Value *VectorLoopValue, VPValue *CastDef, VPTransformState &State, unsigned Part, unsigned Lane=UINT_MAX) |
If there is a cast involved in the induction variable ID, which should be ignored in the vectorized loop body, this function records the VectorLoopValue of the respective Phi also as the VectorLoopValue of the cast. More... | |
| virtual Value * | reverseVector (Value *Vec) |
| Generate a shuffle sequence that will reverse the vector Vec. More... | |
| Value * | getOrCreateTripCount (Loop *NewLoop) |
| Returns (and creates if needed) the original loop trip count. More... | |
| Value * | getOrCreateVectorTripCount (Loop *NewLoop) |
| Returns (and creates if needed) the trip count of the widened loop. More... | |
| Value * | createBitOrPointerCast (Value *V, VectorType *DstVTy, const DataLayout &DL) |
| Returns a bitcasted value to the requested vector type. More... | |
| void | emitMinimumIterationCountCheck (Loop *L, BasicBlock *Bypass) |
| Emit a bypass check to see if the vector trip count is zero, including if it overflows. More... | |
| BasicBlock * | emitSCEVChecks (Loop *L, BasicBlock *Bypass) |
| Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct. More... | |
| BasicBlock * | emitMemRuntimeChecks (Loop *L, BasicBlock *Bypass) |
| Emit bypass checks to check any memory assumptions we may have made. More... | |
| Value * | emitTransformedIndex (IRBuilder<> &B, Value *Index, ScalarEvolution *SE, const DataLayout &DL, const InductionDescriptor &ID) const |
| Compute the transformed value of Index at offset StartValue using step StepValue. More... | |
| Loop * | createVectorLoopSkeleton (StringRef Prefix) |
Emit basic blocks (prefixed with Prefix) for the iteration check, vector loop preheader, middle block and scalar preheader. More... | |
| void | createInductionResumeValues (Loop *L, Value *VectorTripCount, std::pair< BasicBlock *, Value * > AdditionalBypass={nullptr, nullptr}) |
Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off (given by VectorTripCount). More... | |
| BasicBlock * | completeLoopSkeleton (Loop *L, MDNode *OrigLoopID) |
| Complete the loop skeleton by adding debug MDs, creating appropriate conditional branches in the middle block, preparing the builder and running the verifier. More... | |
| void | addNewMetadata (Instruction *To, const Instruction *Orig) |
Add additional metadata to To that was not present on Orig. More... | |
| void | addMetadata (Instruction *To, Instruction *From) |
| Add metadata from one instruction to another. More... | |
| void | addMetadata (ArrayRef< Value * > To, Instruction *From) |
| Similar to the previous function but it adds the metadata to a vector of instructions. More... | |
Additional Inherited Members | |
| Public Types inherited from llvm::InnerLoopVectorizer | |
| using | VectorParts = SmallVector< Value *, 2 > |
| A type for vectorized values in the new loop. More... | |
| Public Attributes inherited from llvm::InnerLoopAndEpilogueVectorizer | |
| EpilogueLoopVectorizationInfo & | EPI |
| Holds and updates state information required to vectorize the main loop and its epilogue in two separate passes. More... | |
| Protected Types inherited from llvm::InnerLoopVectorizer | |
| using | PhiVector = SmallVector< PHINode *, 4 > |
| A small list of PHINodes. More... | |
| using | ScalarParts = SmallVector< SmallVector< Value *, 4 >, 2 > |
| A type for scalarized values in the new loop. More... | |
| Protected Attributes inherited from llvm::InnerLoopVectorizer | |
| Loop * | OrigLoop |
| The original loop. More... | |
| PredicatedScalarEvolution & | PSE |
| A wrapper around ScalarEvolution used to add runtime SCEV checks. More... | |
| LoopInfo * | LI |
| Loop Info. More... | |
| DominatorTree * | DT |
| Dominator Tree. More... | |
| AAResults * | AA |
| Alias Analysis. More... | |
| const TargetLibraryInfo * | TLI |
| Target Library Info. More... | |
| const TargetTransformInfo * | TTI |
| Target Transform Info. More... | |
| AssumptionCache * | AC |
| Assumption Cache. More... | |
| OptimizationRemarkEmitter * | ORE |
| Interface to emit optimization remarks. More... | |
| std::unique_ptr< LoopVersioning > | LVer |
| LoopVersioning. More... | |
| ElementCount | VF |
| The vectorization SIMD factor to use. More... | |
| unsigned | UF |
| The vectorization unroll factor to use. More... | |
| IRBuilder | Builder |
| The builder that we use. More... | |
| BasicBlock * | LoopVectorPreHeader |
| The vector-loop preheader. More... | |
| BasicBlock * | LoopScalarPreHeader |
| The scalar-loop preheader. More... | |
| BasicBlock * | LoopMiddleBlock |
| Middle Block between the vector and the scalar. More... | |
| BasicBlock * | LoopExitBlock |
| The (unique) ExitBlock of the scalar loop. More... | |
| BasicBlock * | LoopVectorBody |
| The vector loop body. More... | |
| BasicBlock * | LoopScalarBody |
| The scalar loop body. More... | |
| SmallVector< BasicBlock *, 4 > | LoopBypassBlocks |
| A list of all bypass blocks. The first block is the entry of the loop. More... | |
| PHINode * | Induction = nullptr |
| The new Induction variable which was added to the new block. More... | |
| PHINode * | OldInduction = nullptr |
| The induction variable of the old basic block. More... | |
| SmallVector< Instruction *, 4 > | PredicatedInstructions |
| Store instructions that were predicated. More... | |
| Value * | TripCount = nullptr |
| Trip count of the original loop. More... | |
| Value * | VectorTripCount = nullptr |
| Trip count of the widened loop (TripCount - TripCount % (VF*UF)) More... | |
| LoopVectorizationLegality * | Legal |
| The legality analysis. More... | |
| LoopVectorizationCostModel * | Cost |
| The profitablity analysis. More... | |
| bool | AddedSafetyChecks = false |
| DenseMap< PHINode *, Value * > | IVEndValues |
| SmallVector< PHINode *, 8 > | OrigPHIsToFix |
| BlockFrequencyInfo * | BFI |
| BFI and PSI are used to check for profile guided size optimizations. More... | |
| ProfileSummaryInfo * | PSI |
| bool | OptForSizeBasedOnProfile |
| GeneratedRTChecks & | RTChecks |
| Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable. More... | |
Detailed Description
A specialized derived class of inner loop vectorizer that performs vectorization of main loops in the process of vectorizing loops and their epilogues.
Definition at line 971 of file LoopVectorize.cpp.
Constructor & Destructor Documentation
◆ EpilogueVectorizerMainLoop()
|
inline |
Definition at line 973 of file LoopVectorize.cpp.
Member Function Documentation
◆ createEpilogueVectorizedLoopSkeleton()
|
finaloverridevirtual |
Implements the interface for creating a vectorized skeleton using the main loop strategy (ie the first pass of vplan execution).
This function is partially responsible for generating the control flow depicted in https://llvm.org/docs/Vectorizers.html#epilogue-vectorization.
Implements llvm::InnerLoopAndEpilogueVectorizer.
Definition at line 8042 of file LoopVectorize.cpp.
References llvm::InnerLoopVectorizer::completeLoopSkeleton(), llvm::InnerLoopVectorizer::createInductionVariable(), llvm::InnerLoopVectorizer::createVectorLoopSkeleton(), llvm::InnerLoopVectorizer::emitMemRuntimeChecks(), emitMinimumIterationCountCheck(), llvm::InnerLoopVectorizer::emitSCEVChecks(), llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueIterationCountCheck, llvm::ConstantInt::get(), getDebugLocFromInstOrOperands(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::Loop::getLoopID(), llvm::InnerLoopVectorizer::getOrCreateVectorTripCount(), llvm::LoopVectorizationLegality::getPrimaryInduction(), llvm::LoopVectorizationLegality::getWidestInductionType(), llvm::InnerLoopVectorizer::Induction, llvm::InnerLoopVectorizer::Legal, llvm::InnerLoopVectorizer::LoopScalarPreHeader, llvm::EpilogueLoopVectorizationInfo::MainLoopIterationCountCheck, llvm::EpilogueLoopVectorizationInfo::MemSafetyCheck, llvm::InnerLoopVectorizer::OldInduction, llvm::InnerLoopVectorizer::OrigLoop, llvm::EpilogueLoopVectorizationInfo::SCEVSafetyCheck, llvm::Value::setName(), llvm::InnerLoopVectorizer::UF, llvm::EpilogueLoopVectorizationInfo::VectorTripCount, and llvm::InnerLoopVectorizer::VF.
◆ emitMinimumIterationCountCheck()
|
protected |
Emits an iteration count bypass check once for the main loop (when ForEpilogue is false) and once for the epilogue loop (when ForEpilogue is true).
Definition at line 8105 of file LoopVectorize.cpp.
References assert(), llvm::InnerLoopVectorizer::Builder, llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), llvm::InnerLoopVectorizer::Cost, llvm::BranchInst::Create(), llvm::IRBuilderBase::CreateICmp(), llvm::InnerLoopVectorizer::DT, llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueUF, llvm::EpilogueLoopVectorizationInfo::EpilogueVF, llvm::ConstantInt::get(), llvm::DomTreeNodeBase< NodeT >::getIDom(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::DominatorTreeBase< NodeT, IsPostDom >::getNode(), llvm::InnerLoopVectorizer::getOrCreateTripCount(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::CmpInst::ICMP_ULE, llvm::CmpInst::ICMP_ULT, llvm::InnerLoopVectorizer::LI, llvm::InnerLoopVectorizer::LoopBypassBlocks, llvm::InnerLoopVectorizer::LoopExitBlock, llvm::InnerLoopVectorizer::LoopVectorPreHeader, P, llvm::DominatorTreeBase< NodeT, IsPostDom >::properlyDominates(), llvm::ReplaceInstWithInst(), llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::Value::setName(), llvm::SplitBlock(), llvm::EpilogueLoopVectorizationInfo::TripCount, llvm::InnerLoopVectorizer::UF, and llvm::InnerLoopVectorizer::VF.
Referenced by createEpilogueVectorizedLoopSkeleton(), and llvm::InnerLoopVectorizer::createVectorizedLoopSkeleton().
◆ printDebugTracesAtEnd()
|
overrideprotectedvirtual |
Reimplemented from llvm::InnerLoopVectorizer.
Definition at line 8099 of file LoopVectorize.cpp.
References llvm::dbgs(), DEBUG_WITH_TYPE, llvm::Instruction::getFunction(), llvm::InnerLoopVectorizer::Induction, and VerboseDebug.
◆ printDebugTracesAtStart()
|
overrideprotectedvirtual |
Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.
Reimplemented from llvm::InnerLoopVectorizer.
Definition at line 8089 of file LoopVectorize.cpp.
References llvm::dbgs(), llvm::InnerLoopAndEpilogueVectorizer::EPI, llvm::EpilogueLoopVectorizationInfo::EpilogueUF, llvm::EpilogueLoopVectorizationInfo::EpilogueVF, llvm::LinearPolySize< LeafTy >::getKnownMinValue(), LLVM_DEBUG, llvm::EpilogueLoopVectorizationInfo::MainLoopUF, and llvm::EpilogueLoopVectorizationInfo::MainLoopVF.
The documentation for this class was generated from the following file:
- lib/Transforms/Vectorize/LoopVectorize.cpp
