LLVM 17.0.0git
|
InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF). More...
Public Types | |
using | VectorParts = SmallVector< Value *, 2 > |
A type for vectorized values in the new loop. | |
Public Member Functions | |
InnerLoopVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, ElementCount VecWidth, ElementCount MinProfitableTripCount, unsigned UnrollFactor, LoopVectorizationLegality *LVL, LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks) | |
virtual | ~InnerLoopVectorizer ()=default |
virtual std::pair< BasicBlock *, Value * > | createVectorizedLoopSkeleton () |
Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder. | |
void | fixVectorizedLoop (VPTransformState &State, VPlan &Plan) |
Fix the vectorized code, taking care of header phi's, live-outs, and more. | |
bool | areSafetyChecksAdded () |
void | scalarizeInstruction (const Instruction *Instr, VPReplicateRecipe *RepRecipe, const VPIteration &Instance, VPTransformState &State) |
A helper function to scalarize a single Instruction in the innermost loop. | |
void | packScalarIntoVectorValue (VPValue *Def, const VPIteration &Instance, VPTransformState &State) |
Construct the vector value of a scalarized value V one lane at a time. | |
void | vectorizeInterleaveGroup (const InterleaveGroup< Instruction > *Group, ArrayRef< VPValue * > VPDefs, VPTransformState &State, VPValue *Addr, ArrayRef< VPValue * > StoredValues, VPValue *BlockInMask=nullptr) |
Try to vectorize interleaved access group Group with the base address given in Addr , optionally masking the vector operations if BlockInMask is non-null. | |
void | fixNonInductionPHIs (VPlan &Plan, VPTransformState &State) |
Fix the non-induction PHIs in Plan . | |
bool | useOrderedReductions (const RecurrenceDescriptor &RdxDesc) |
Returns true if the reordering of FP operations is not allowed, but we are able to vectorize with strict in-order reductions for the given RdxDesc. | |
virtual Value * | getBroadcastInstrs (Value *V) |
Create a broadcast instruction. | |
PHINode * | getReductionResumeValue (const RecurrenceDescriptor &RdxDesc) |
PHINode * | createInductionResumeValue (PHINode *OrigPhi, const InductionDescriptor &ID, ArrayRef< BasicBlock * > BypassBlocks, std::pair< BasicBlock *, Value * > AdditionalBypass={nullptr, nullptr}) |
Create a new phi node for the induction variable OrigPhi to resume iteration count in the scalar epilogue, from where the vectorized loop left off. | |
Protected Types | |
using | PhiVector = SmallVector< PHINode *, 4 > |
A small list of PHINodes. | |
using | ScalarParts = SmallVector< SmallVector< Value *, 4 >, 2 > |
A type for scalarized values in the new loop. | |
Protected Member Functions | |
void | fixupIVUsers (PHINode *OrigPhi, const InductionDescriptor &II, Value *VectorTripCount, Value *EndValue, BasicBlock *MiddleBlock, BasicBlock *VectorHeader, VPlan &Plan) |
Set up the values of the IVs correctly when exiting the vector loop. | |
void | fixCrossIterationPHIs (VPTransformState &State) |
Handle all cross-iteration phis in the header. | |
void | fixFixedOrderRecurrence (VPFirstOrderRecurrencePHIRecipe *PhiR, VPTransformState &State) |
Create the exit value of first order recurrences in the middle block and update their users. | |
void | fixReduction (VPReductionPHIRecipe *Phi, VPTransformState &State) |
Create code for the loop exit value of the reduction. | |
void | clearReductionWrapFlags (VPReductionPHIRecipe *PhiR, VPTransformState &State) |
Clear NSW/NUW flags from reduction instructions if necessary. | |
void | sinkScalarOperands (Instruction *PredInst) |
Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it. | |
void | truncateToMinimalBitwidths (VPTransformState &State) |
Shrinks vector element sizes to the smallest bitwidth they can be legally represented as. | |
Value * | getOrCreateTripCount (BasicBlock *InsertBlock) |
Returns (and creates if needed) the original loop trip count. | |
Value * | getOrCreateVectorTripCount (BasicBlock *InsertBlock) |
Returns (and creates if needed) the trip count of the widened loop. | |
Value * | createBitOrPointerCast (Value *V, VectorType *DstVTy, const DataLayout &DL) |
Returns a bitcasted value to the requested vector type. | |
void | emitIterationCountCheck (BasicBlock *Bypass) |
Emit a bypass check to see if the vector trip count is zero, including if it overflows. | |
BasicBlock * | emitSCEVChecks (BasicBlock *Bypass) |
Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct. | |
BasicBlock * | emitMemRuntimeChecks (BasicBlock *Bypass) |
Emit bypass checks to check any memory assumptions we may have made. | |
void | createVectorLoopSkeleton (StringRef Prefix) |
Emit basic blocks (prefixed with Prefix ) for the iteration check, vector loop preheader, middle block and scalar preheader. | |
void | createInductionResumeValues (std::pair< BasicBlock *, Value * > AdditionalBypass={nullptr, nullptr}) |
Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off. | |
BasicBlock * | completeLoopSkeleton () |
Complete the loop skeleton by adding debug MDs, creating appropriate conditional branches in the middle block, preparing the builder and running the verifier. | |
void | collectPoisonGeneratingRecipes (VPTransformState &State) |
Collect poison-generating recipes that may generate a poison value that is used after vectorization, even when their operands are not poison. | |
virtual void | printDebugTracesAtStart () |
Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested. | |
virtual void | printDebugTracesAtEnd () |
Protected Attributes | |
Loop * | OrigLoop |
The original loop. | |
PredicatedScalarEvolution & | PSE |
A wrapper around ScalarEvolution used to add runtime SCEV checks. | |
LoopInfo * | LI |
Loop Info. | |
DominatorTree * | DT |
Dominator Tree. | |
const TargetLibraryInfo * | TLI |
Target Library Info. | |
const TargetTransformInfo * | TTI |
Target Transform Info. | |
AssumptionCache * | AC |
Assumption Cache. | |
OptimizationRemarkEmitter * | ORE |
Interface to emit optimization remarks. | |
ElementCount | VF |
The vectorization SIMD factor to use. | |
ElementCount | MinProfitableTripCount |
unsigned | UF |
The vectorization unroll factor to use. | |
IRBuilder | Builder |
The builder that we use. | |
BasicBlock * | LoopVectorPreHeader |
The vector-loop preheader. | |
BasicBlock * | LoopScalarPreHeader |
The scalar-loop preheader. | |
BasicBlock * | LoopMiddleBlock |
Middle Block between the vector and the scalar. | |
BasicBlock * | LoopExitBlock |
The unique ExitBlock of the scalar loop if one exists. | |
BasicBlock * | LoopScalarBody |
The scalar loop body. | |
SmallVector< BasicBlock *, 4 > | LoopBypassBlocks |
A list of all bypass blocks. The first block is the entry of the loop. | |
SmallVector< Instruction *, 4 > | PredicatedInstructions |
Store instructions that were predicated. | |
Value * | TripCount = nullptr |
Trip count of the original loop. | |
Value * | VectorTripCount = nullptr |
Trip count of the widened loop (TripCount - TripCount % (VF*UF)) | |
LoopVectorizationLegality * | Legal |
The legality analysis. | |
LoopVectorizationCostModel * | Cost |
The profitablity analysis. | |
bool | AddedSafetyChecks = false |
DenseMap< PHINode *, Value * > | IVEndValues |
BlockFrequencyInfo * | BFI |
BFI and PSI are used to check for profile guided size optimizations. | |
ProfileSummaryInfo * | PSI |
bool | OptForSizeBasedOnProfile |
GeneratedRTChecks & | RTChecks |
Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable. | |
SmallMapVector< const RecurrenceDescriptor *, PHINode *, 4 > | ReductionResumeValues |
Friends | |
class | LoopVectorizationPlanner |
InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF).
This class performs the widening of scalars into vectors, or multiple scalars. This class also implements the following features:
Definition at line 467 of file LoopVectorize.cpp.
|
protected |
A small list of PHINodes.
Definition at line 573 of file LoopVectorize.cpp.
|
protected |
A type for scalarized values in the new loop.
Each value from the original loop, when scalarized, is represented by UF x VF scalar values in the new unrolled loop, where UF is the unroll factor and VF is the vectorization factor.
Definition at line 579 of file LoopVectorize.cpp.
using llvm::InnerLoopVectorizer::VectorParts = SmallVector<Value *, 2> |
A type for vectorized values in the new loop.
Each value from the original loop, when vectorized, is represented by UF vector values in the new unrolled loop, where UF is the unroll factor.
Definition at line 514 of file LoopVectorize.cpp.
|
inline |
Definition at line 469 of file LoopVectorize.cpp.
References BFI, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::IRPass, llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isZero(), MinProfitableTripCount, OptForSizeBasedOnProfile, OrigLoop, PSI, and llvm::shouldOptimizeForSize().
|
virtualdefault |
|
inline |
Definition at line 509 of file LoopVectorize.cpp.
References AddedSafetyChecks.
Referenced by llvm::LoopVectorizePass::processLoop().
|
protected |
Clear NSW/NUW flags from reduction instructions if necessary.
Definition at line 4100 of file LoopVectorize.cpp.
References llvm::Add, llvm::SmallVectorBase< Size_T >::empty(), llvm::VPTransformState::get(), llvm::VPReductionPHIRecipe::getRecurrenceDescriptor(), llvm::RecurrenceDescriptor::getRecurrenceKind(), llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::Mul, llvm::SmallVectorImpl< T >::pop_back_val(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), UF, and llvm::VPValue::users().
Referenced by fixReduction().
|
protected |
Collect poison-generating recipes that may generate a poison value that is used after vectorization, even when their operands are not poison.
Those recipes meet the following conditions:
Definition at line 1055 of file LoopVectorize.cpp.
References llvm::SmallVectorBase< Size_T >::empty(), llvm::VPlan::getEntry(), llvm::InterleaveGroup< InstTy >::getMember(), llvm::InterleaveGroup< InstTy >::getNumMembers(), llvm::Instruction::getParent(), I, llvm::IRSimilarity::Legal, llvm::VPTransformState::Plan, llvm::SmallVectorTemplateBase< T, bool >::push_back(), and llvm::vp_depth_first_deep().
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
protected |
Complete the loop skeleton by adding debug MDs, creating appropriate conditional branches in the middle block, preparing the builder and running the verifier.
Return the preheader of the completed vector loop.
Definition at line 3242 of file LoopVectorize.cpp.
References assert(), Cost, llvm::CmpInst::Create(), DT, llvm::LoopVectorizationCostModel::foldTailByMasking(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), getOrCreateTripCount(), getOrCreateVectorTripCount(), llvm::BasicBlock::getTerminator(), llvm::CmpInst::ICMP_EQ, LoopMiddleBlock, LoopVectorPreHeader, OrigLoop, llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::Instruction::setDebugLoc(), VectorTripCount, llvm::DominatorTreeBase< NodeT, IsPostDom >::verify(), and VF.
Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().
|
protected |
Returns a bitcasted value to the requested vector type.
Also handles bitcasts of vector<float> <-> vector<pointer> types.
Definition at line 2952 of file LoopVectorize.cpp.
References assert(), Builder, llvm::IRBuilderBase::CreateBitOrPointerCast(), DL, llvm::VectorType::get(), llvm::Type::getIntNTy(), llvm::CastInst::isBitOrNoopPointerCastable(), llvm::Type::isFloatingPointTy(), llvm::Type::isPointerTy(), and VF.
Referenced by vectorizeInterleaveGroup().
PHINode * InnerLoopVectorizer::createInductionResumeValue | ( | PHINode * | OrigPhi, |
const InductionDescriptor & | ID, | ||
ArrayRef< BasicBlock * > | BypassBlocks, | ||
std::pair< BasicBlock *, Value * > | AdditionalBypass = {nullptr, nullptr} |
||
) |
Create a new phi node for the induction variable OrigPhi
to resume iteration count in the scalar epilogue, from where the vectorized loop left off.
In cases where the loop skeleton is more complicated (eg. epilogue vectorization) and the resume values can come from an additional bypass block, the AdditionalBypass
pair provides information about the bypass block and the end value on the edge from bypass to this loop.
Definition at line 3162 of file LoopVectorize.cpp.
References llvm::PHINode::addIncoming(), assert(), B, llvm::PHINode::Create(), CreateStepValue(), emitTransformedIndex(), llvm::Instruction::getDebugLoc(), llvm::Instruction::getFastMathFlags(), llvm::InductionDescriptor::getInductionBinOp(), getOrCreateVectorTripCount(), llvm::PredicatedScalarEvolution::getSE(), llvm::InductionDescriptor::getStartValue(), llvm::InductionDescriptor::getStep(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), IVEndValues, llvm::IRSimilarity::Legal, LoopMiddleBlock, LoopScalarPreHeader, LoopVectorPreHeader, PSE, llvm::Instruction::setDebugLoc(), llvm::PHINode::setIncomingValueForBlock(), llvm::Value::setName(), and VectorTripCount.
Referenced by createInductionResumeValues(), and llvm::LoopVectorizePass::processLoop().
|
protected |
Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off.
In cases where the loop skeleton is more complicated (eg. epilogue vectorization) and the resume values can come from an additional bypass block, the AdditionalBypass
pair provides information about the bypass block and the end value on the edge from bypass to this loop.
Definition at line 3221 of file LoopVectorize.cpp.
References assert(), createInductionResumeValue(), llvm::IRSimilarity::Legal, LoopBypassBlocks, LoopScalarPreHeader, and llvm::PHINode::setIncomingValueForBlock().
Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().
|
virtual |
Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.
Control flow is generated around the vectorized (and scalar epilogue) loops consisting of various checks and bypasses. Return the pre-header block of the new loop and the start value for the canonical induction, if it is != 0. The latter is the case when vectorizing the epilogue loop. In the case of epilogue vectorization, this function is overriden to handle the more complex control flow around the loops.
Reimplemented in llvm::InnerLoopAndEpilogueVectorizer.
Definition at line 3278 of file LoopVectorize.cpp.
References completeLoopSkeleton(), createInductionResumeValues(), createVectorLoopSkeleton(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), emitSCEVChecks(), and LoopScalarPreHeader.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
protected |
Emit basic blocks (prefixed with Prefix
) for the iteration check, vector loop preheader, middle block and scalar preheader.
Definition at line 3120 of file LoopVectorize.cpp.
References assert(), Builder, llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), Cost, llvm::BranchInst::Create(), DT, llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::BasicBlock::getTerminator(), llvm::IRBuilderBase::getTrue(), llvm::LoopBase< BlockT, LoopT >::getUniqueExitBlock(), LI, LoopExitBlock, LoopMiddleBlock, LoopScalarBody, LoopScalarPreHeader, LoopVectorPreHeader, OrigLoop, llvm::ReplaceInstWithInst(), llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::Instruction::setDebugLoc(), llvm::SplitBlock(), and VF.
Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().
|
protected |
Emit a bypass check to see if the vector trip count is zero, including if it overflows.
Definition at line 2983 of file LoopVectorize.cpp.
References assert(), Builder, llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), Cost, llvm::BranchInst::Create(), llvm::IRBuilderBase::CreateBinaryIntrinsic(), llvm::IRBuilderBase::CreateICmp(), llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::DataAndControlFlowWithoutRuntimeCheck, DT, llvm::ConstantInt::get(), llvm::IRBuilderBase::getFalse(), llvm::DomTreeNodeBase< NodeT >::getIDom(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::DominatorTreeBase< NodeT, IsPostDom >::getNode(), getOrCreateTripCount(), llvm::LoopVectorizationCostModel::getTailFoldingStyle(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::CmpInst::ICMP_ULE, llvm::CmpInst::ICMP_ULT, isIndvarOverflowCheckKnownFalse(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), LHS, LI, LoopBypassBlocks, LoopExitBlock, LoopVectorPreHeader, MinProfitableTripCount, llvm::None, P, llvm::DominatorTreeBase< NodeT, IsPostDom >::properlyDominates(), llvm::ReplaceInstWithInst(), llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::SplitBlock(), UF, and VF.
|
protected |
Emit bypass checks to check any memory assumptions we may have made.
Returns the block containing the checks or nullptr if no checks have been added.
Definition at line 3084 of file LoopVectorize.cpp.
References AddedSafetyChecks, assert(), Cost, DEBUG_TYPE, llvm::OptimizationRemarkEmitter::emit(), llvm::EnableVPlanNativePath, llvm::LoopVectorizeHints::FK_Enabled, llvm::LoopVectorizeHints::getForce(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::BasicBlock::getParent(), llvm::Loop::getStartLoc(), llvm::Function::hasOptSize(), llvm::LoopVectorizationCostModel::Hints, LoopBypassBlocks, LoopVectorPreHeader, OptForSizeBasedOnProfile, ORE, OrigLoop, and RTChecks.
Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().
|
protected |
Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct.
Returns the block containing the checks or nullptr if no checks have been added.
Definition at line 3057 of file LoopVectorize.cpp.
References AddedSafetyChecks, assert(), llvm::DominatorTreeBase< NodeT, IsPostDom >::changeImmediateDominator(), Cost, DT, llvm::LoopVectorizeHints::FK_Enabled, llvm::LoopVectorizeHints::getForce(), llvm::BasicBlock::getParent(), llvm::Function::hasOptSize(), llvm::LoopVectorizationCostModel::Hints, LoopBypassBlocks, LoopExitBlock, LoopVectorPreHeader, OptForSizeBasedOnProfile, llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), RTChecks, and VF.
Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().
|
protected |
Handle all cross-iteration phis in the header.
Definition at line 3771 of file LoopVectorize.cpp.
References fixFixedOrderRecurrence(), fixReduction(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPlan::getVectorLoopRegion(), and llvm::VPTransformState::Plan.
Referenced by fixVectorizedLoop().
|
protected |
Create the exit value of first order recurrences in the middle block and update their users.
Definition at line 3788 of file LoopVectorize.cpp.
References llvm::BasicBlock::begin(), Builder, Cost, llvm::IRBuilderBase::CreateExtractElement(), llvm::IRBuilderBase::CreatePHI(), llvm::IRBuilderBase::CreateSub(), llvm::ConstantInt::get(), llvm::VPTransformState::get(), llvm::VPHeaderPHIRecipe::getBackedgeValue(), llvm::IRBuilderBase::getInt32Ty(), llvm::VPValue::getLiveInIRValue(), llvm::getRuntimeVF(), llvm::VPHeaderPHIRecipe::getStartValue(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::VPValue::getUnderlyingValue(), Idx, llvm::is_contained(), llvm::ElementCount::isVector(), LoopExitBlock, LoopMiddleBlock, LoopScalarPreHeader, llvm::BasicBlock::phis(), llvm::VPTransformState::Plan, llvm::predecessors(), llvm::VPlan::removeLiveOut(), llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::PHINode::setIncomingValueForBlock(), llvm::IRBuilderBase::SetInsertPoint(), llvm::Value::setName(), UF, and VF.
Referenced by fixCrossIterationPHIs().
void InnerLoopVectorizer::fixNonInductionPHIs | ( | VPlan & | Plan, |
VPTransformState & | State | ||
) |
Fix the non-induction PHIs in Plan
.
Definition at line 4206 of file LoopVectorize.cpp.
References llvm::PHINode::addIncoming(), Builder, llvm::VPTransformState::CFG, llvm::VPTransformState::get(), llvm::VPlan::getEntry(), llvm::VPWidenPHIRecipe::getIncomingBlock(), llvm::VPWidenPHIRecipe::getIncomingValue(), llvm::VPUser::getNumOperands(), P, llvm::IRBuilderBase::SetInsertPoint(), llvm::vp_depth_first_deep(), and llvm::VPTransformState::CFGState::VPBB2IRBB.
Referenced by fixVectorizedLoop().
|
protected |
Create code for the loop exit value of the reduction.
Definition at line 3901 of file LoopVectorize.cpp.
References llvm::PHINode::addIncoming(), assert(), llvm::PHINode::blocks(), Builder, llvm::VPTransformState::CFG, clearReductionWrapFlags(), Cost, llvm::PHINode::Create(), llvm::IRBuilderBase::CreateBinOp(), llvm::createMinMaxOp(), llvm::createSelectCmpOp(), llvm::IRBuilderBase::CreateSExt(), llvm::IRBuilderBase::CreateStore(), llvm::createTargetReduction(), llvm::IRBuilderBase::CreateTrunc(), llvm::IRBuilderBase::CreateZExt(), llvm::LoopVectorizationCostModel::foldTailByMasking(), llvm::VectorType::get(), llvm::VPTransformState::get(), llvm::VPHeaderPHIRecipe::getBackedgeValue(), llvm::PHINode::getBasicBlockIndex(), llvm::VPBasicBlock::getEnclosingLoopRegion(), llvm::VPBlockBase::getExitingBasicBlock(), llvm::RecurrenceDescriptor::getFastMathFlags(), llvm::BasicBlock::getFirstInsertionPt(), llvm::PHINode::getIncomingValueForBlock(), llvm::RecurrenceDescriptor::getLoopExitInstr(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::RecurrenceDescriptor::getOpcode(), llvm::VPRecipeBase::getParent(), llvm::VPReductionPHIRecipe::getRecurrenceDescriptor(), llvm::RecurrenceDescriptor::getRecurrenceKind(), llvm::RecurrenceDescriptor::getRecurrenceStartValue(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::VPHeaderPHIRecipe::getStartValue(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::VPValue::getUnderlyingValue(), llvm::RecurrenceDescriptor::IntermediateStore, llvm::is_contained(), llvm::VPReductionPHIRecipe::isInLoop(), llvm::VPReductionPHIRecipe::isOrdered(), llvm::RecurrenceDescriptor::isSelectCmpRecurrenceKind(), llvm::RecurrenceDescriptor::isSigned(), llvm::ElementCount::isVector(), llvm::IRSimilarity::Legal, LoopExitBlock, LoopMiddleBlock, LoopScalarPreHeader, llvm::make_early_inc_range(), OrigLoop, llvm::BasicBlock::phis(), llvm::VPTransformState::Plan, llvm::predecessors(), PreferPredicatedReductionSelect, llvm::TargetTransformInfo::preferPredicatedReductionSelect(), llvm::propagateMetadata(), ReductionResumeValues, llvm::VPlan::removeLiveOut(), llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::VPTransformState::reset(), llvm::VPTransformState::setDebugLocFromInst(), llvm::Instruction::setFastMathFlags(), llvm::IRBuilderBase::setFastMathFlags(), llvm::PHINode::setIncomingValue(), llvm::IRBuilderBase::SetInsertPoint(), SI, UF, llvm::Value::users(), users, VF, and llvm::VPTransformState::CFGState::VPBB2IRBB.
Referenced by fixCrossIterationPHIs().
|
protected |
Set up the values of the IVs correctly when exiting the vector loop.
Definition at line 3341 of file LoopVectorize.cpp.
References assert(), B, llvm::LoopBase< BlockT, LoopT >::contains(), CreateStepValue(), emitTransformedIndex(), llvm::ConstantInt::get(), llvm::Instruction::getFastMathFlags(), llvm::PHINode::getIncomingValueForBlock(), llvm::InductionDescriptor::getInductionBinOp(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::PredicatedScalarEvolution::getSE(), llvm::InductionDescriptor::getStartValue(), llvm::InductionDescriptor::getStep(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::LoopBase< BlockT, LoopT >::getUniqueExitBlock(), I, OrigLoop, PHI, PostInc, PSE, llvm::VPlan::removeLiveOut(), llvm::Value::setName(), llvm::Value::users(), and VectorTripCount.
Referenced by fixVectorizedLoop().
void InnerLoopVectorizer::fixVectorizedLoop | ( | VPTransformState & | State, |
VPlan & | Plan | ||
) |
Fix the vectorized code, taking care of header phi's, live-outs, and more.
Definition at line 3701 of file LoopVectorize.cpp.
References llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::VPlan::clearLiveOuts(), Cost, cse(), llvm::EnableVPlanNativePath, llvm::VPTransformState::CFGState::ExitBB, fixCrossIterationPHIs(), fixNonInductionPHIs(), fixupIVUsers(), llvm::ScalarEvolution::forgetLoop(), llvm::VPBlockBase::getExitingBasicBlock(), llvm::BasicBlock::getFirstNonPHI(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::VPlan::getLiveOuts(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), getOrCreateVectorTripCount(), llvm::PredicatedScalarEvolution::getSE(), llvm::VPlan::getVectorLoopRegion(), llvm::ElementCount::isVector(), IVEndValues, llvm::IRSimilarity::Legal, LI, LoopMiddleBlock, LoopScalarBody, OrigLoop, PredicatedInstructions, PSE, llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::IRBuilderBase::SetInsertPoint(), llvm::setProfileInfoAfterUnrolling(), sinkScalarOperands(), truncateToMinimalBitwidths(), UF, VF, and llvm::VPTransformState::CFGState::VPBB2IRBB.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
Create a broadcast instruction.
This method generates a broadcast instruction (shuffle) for loop invariant values and for the induction value. If this is the induction variable then we extend it to N, N+1, ... this is needed because each iteration in the loop corresponds to a SIMD element.
Definition at line 2255 of file LoopVectorize.cpp.
References Builder, llvm::IRBuilderBase::CreateVectorSplat(), llvm::DominatorTree::dominates(), DT, llvm::Instruction::getParent(), llvm::BasicBlock::getTerminator(), llvm::Loop::isLoopInvariant(), LoopVectorPreHeader, OrigLoop, llvm::IRBuilderBase::SetInsertPoint(), and VF.
|
protected |
Returns (and creates if needed) the original loop trip count.
Definition at line 2872 of file LoopVectorize.cpp.
References assert(), Builder, llvm::CastInst::CreatePointerCast(), llvm::createTripCountSCEV(), DL, llvm::Module::getDataLayout(), llvm::BasicBlock::getModule(), llvm::PredicatedScalarEvolution::getSE(), llvm::BasicBlock::getTerminator(), llvm::SCEV::getType(), llvm::Value::getType(), llvm::Type::isPointerTy(), llvm::IRSimilarity::Legal, PSE, and TripCount.
Referenced by completeLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::LoopVectorizationPlanner::executePlan(), and getOrCreateVectorTripCount().
|
protected |
Returns (and creates if needed) the trip count of the widened loop.
Definition at line 2902 of file LoopVectorize.cpp.
References assert(), Builder, Cost, llvm::IRBuilderBase::CreateAdd(), llvm::IRBuilderBase::CreateICmpEQ(), llvm::IRBuilderBase::CreateSelect(), llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::IRBuilderBase::CreateURem(), llvm::LoopVectorizationCostModel::foldTailByMasking(), llvm::ConstantInt::get(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), getOrCreateTripCount(), llvm::getRuntimeVF(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::isPowerOf2_32(), llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), UF, VectorTripCount, and VF.
Referenced by completeLoopSkeleton(), llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), createInductionResumeValue(), llvm::LoopVectorizationPlanner::executePlan(), and fixVectorizedLoop().
PHINode * InnerLoopVectorizer::getReductionResumeValue | ( | const RecurrenceDescriptor & | RdxDesc | ) |
Definition at line 1133 of file LoopVectorize.cpp.
References assert(), and ReductionResumeValues.
Referenced by llvm::LoopVectorizePass::processLoop().
void InnerLoopVectorizer::packScalarIntoVectorValue | ( | VPValue * | Def, |
const VPIteration & | Instance, | ||
VPTransformState & | State | ||
) |
Construct the vector value of a scalarized value V
one lane at a time.
Definition at line 2563 of file LoopVectorize.cpp.
References Builder, llvm::VPTransformState::Builder, llvm::IRBuilderBase::CreateInsertElement(), llvm::VPTransformState::get(), llvm::VPLane::getAsRuntimeExpr(), llvm::VPIteration::Lane, llvm::VPIteration::Part, llvm::VPTransformState::set(), and VF.
Referenced by llvm::VPReplicateRecipe::execute().
|
inlineprotectedvirtual |
Reimplemented in llvm::EpilogueVectorizerMainLoop, and llvm::EpilogueVectorizerEpilogueLoop.
Definition at line 667 of file LoopVectorize.cpp.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
inlineprotectedvirtual |
Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.
Reimplemented in llvm::EpilogueVectorizerMainLoop, and llvm::EpilogueVectorizerEpilogueLoop.
Definition at line 666 of file LoopVectorize.cpp.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
void InnerLoopVectorizer::scalarizeInstruction | ( | const Instruction * | Instr, |
VPReplicateRecipe * | RepRecipe, | ||
const VPIteration & | Instance, | ||
VPTransformState & | State | ||
) |
A helper function to scalarize a single Instruction in the innermost loop.
Generates a sequence of scalar instances for each lane between MinLane
and MaxLane
, times each part between MinPart
and MaxPart
, inclusive. Uses the VPValue operands from RepRecipe
instead of Instr's
operands.
Definition at line 2815 of file LoopVectorize.cpp.
References AC, llvm::VPTransformState::addNewMetadata(), assert(), llvm::VPTransformState::Builder, llvm::Instruction::clone(), llvm::Instruction::dropPoisonGeneratingFlags(), llvm::enumerate(), llvm::VPTransformState::get(), llvm::Instruction::getDebugLoc(), llvm::VPLane::getFirstLane(), llvm::Value::getName(), llvm::VPBlockBase::getParent(), llvm::VPRecipeBase::getParent(), llvm::Value::getType(), I, llvm::IRBuilderBase::Insert(), llvm::Type::isAggregateType(), llvm::VPIteration::isFirstIteration(), llvm::VPRegionBlock::isReplicator(), llvm::vputils::isUniformAfterVectorization(), llvm::Type::isVoidTy(), llvm::VPTransformState::MayGeneratePoisonRecipes, llvm::VPUser::operands(), PredicatedInstructions, llvm::AssumptionCache::registerAssumption(), llvm::VPTransformState::set(), llvm::VPTransformState::setDebugLocFromInst(), llvm::Value::setName(), and llvm::User::setOperand().
Referenced by llvm::VPReplicateRecipe::execute().
|
protected |
Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it.
Definition at line 4132 of file LoopVectorize.cpp.
References llvm::all_of(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), llvm::SmallVectorImpl< T >::clear(), llvm::SetVector< T, Vector, Set >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::end(), llvm::PHINode::getIncomingValueNumForOperand(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::Instruction::getParent(), I, llvm::SetVector< T, Vector, Set >::insert(), LI, llvm::User::op_begin(), llvm::User::op_end(), llvm::SetVector< T, Vector, Set >::pop_back_val(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().
Referenced by fixVectorizedLoop().
|
protected |
Shrinks vector element sizes to the smallest bitwidth they can be legally represented as.
Definition at line 3568 of file LoopVectorize.cpp.
References B, Cost, llvm::SmallPtrSetImpl< PtrType >::count(), llvm::Instruction::eraseFromParent(), llvm::IntegerType::get(), llvm::VectorType::get(), llvm::VPTransformState::get(), llvm::Type::getContext(), llvm::LoopVectorizationCostModel::getMinimalBitwidths(), llvm::User::getOperand(), llvm::VPlan::getVPValue(), llvm::VPTransformState::hasAnyVectorValue(), I, llvm::SmallPtrSetImpl< PtrType >::insert(), llvm_unreachable, llvm::VPTransformState::Plan, llvm::VPTransformState::reset(), SI, smallestIntegerVectorType(), llvm::Value::takeName(), UF, and llvm::Value::use_empty().
Referenced by fixVectorizedLoop().
bool InnerLoopVectorizer::useOrderedReductions | ( | const RecurrenceDescriptor & | RdxDesc | ) |
Returns true if the reordering of FP operations is not allowed, but we are able to vectorize with strict in-order reductions for the given RdxDesc.
Definition at line 4226 of file LoopVectorize.cpp.
References Cost, and llvm::LoopVectorizationCostModel::useOrderedReductions().
Referenced by llvm::LoopVectorizationCostModel::collectElementTypesForWidening(), llvm::LoopVectorizationCostModel::collectInLoopReductions(), and llvm::VPReductionRecipe::execute().
void InnerLoopVectorizer::vectorizeInterleaveGroup | ( | const InterleaveGroup< Instruction > * | Group, |
ArrayRef< VPValue * > | VPDefs, | ||
VPTransformState & | State, | ||
VPValue * | Addr, | ||
ArrayRef< VPValue * > | StoredValues, | ||
VPValue * | BlockInMask = nullptr |
||
) |
Try to vectorize interleaved access group Group
with the base address given in Addr
, optionally masking the vector operations if BlockInMask
is non-null.
Use State
to translate given VPValues to IR values in the vectorized loop.
Definition at line 2613 of file LoopVectorize.cpp.
References llvm::InterleaveGroup< InstTy >::addMetadata(), Addr, assert(), Builder, llvm::concatenateVectors(), Cost, llvm::IRBuilderBase::CreateAlignedLoad(), llvm::IRBuilderBase::CreateAlignedStore(), llvm::IRBuilderBase::CreateBinOp(), llvm::IRBuilderBase::CreateBitCast(), llvm::createBitMaskForGaps(), createBitOrPointerCast(), llvm::IRBuilderBase::CreateGEP(), llvm::createInterleaveMask(), llvm::IRBuilderBase::CreateMaskedLoad(), llvm::IRBuilderBase::CreateMaskedStore(), llvm::createReplicatedMask(), llvm::IRBuilderBase::CreateShuffleVector(), llvm::createStrideMask(), llvm::IRBuilderBase::CreateVectorReverse(), DL, gep, llvm::VectorType::get(), llvm::PoisonValue::get(), llvm::VPTransformState::get(), llvm::InterleaveGroup< InstTy >::getAlign(), llvm::Module::getDataLayout(), llvm::InterleaveGroup< InstTy >::getFactor(), llvm::InterleaveGroup< InstTy >::getIndex(), llvm::InterleaveGroup< InstTy >::getInsertPos(), llvm::IRBuilderBase::getInt32(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::getLoadStoreType(), llvm::InterleaveGroup< InstTy >::getMember(), llvm::Instruction::getModule(), llvm::Type::getPointerAddressSpace(), llvm::Type::getPointerTo(), llvm::Value::getType(), I, llvm::InterleaveGroup< InstTy >::isReverse(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::LoopVectorizationCostModel::isScalarEpilogueAllowed(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::InterleaveGroup< InstTy >::requiresScalarEpilogue(), llvm::VPTransformState::set(), llvm::VPTransformState::setDebugLocFromInst(), llvm::Value::stripPointerCasts(), UF, useMaskedInterleavedAccesses(), and VF.
Referenced by llvm::VPInterleaveRecipe::execute().
|
friend |
Definition at line 570 of file LoopVectorize.cpp.
|
protected |
Assumption Cache.
Definition at line 690 of file LoopVectorize.cpp.
Referenced by llvm::LoopVectorizationCostModel::collectValuesToIgnore(), getScalarEpilogueLowering(), processLoopInVPlanNativePath(), and scalarizeInstruction().
|
protected |
Definition at line 745 of file LoopVectorize.cpp.
Referenced by areSafetyChecksAdded(), emitMemRuntimeChecks(), and emitSCEVChecks().
|
protected |
BFI and PSI are used to check for profile guided size optimizations.
Definition at line 752 of file LoopVectorize.cpp.
Referenced by getScalarEpilogueLowering(), InnerLoopVectorizer(), and processLoopInVPlanNativePath().
|
protected |
The builder that we use.
Definition at line 706 of file LoopVectorize.cpp.
Referenced by buildScalarSteps(), createBitOrPointerCast(), llvm::VPRecipeBuilder::createBlockInMask(), llvm::VPRecipeBuilder::createEdgeMask(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenMemoryInstructionRecipe::execute(), llvm::LoopVectorizationPlanner::executePlan(), fixFixedOrderRecurrence(), fixNonInductionPHIs(), fixReduction(), llvm::VPTransformState::get(), getBroadcastInstrs(), getOrCreateTripCount(), getOrCreateVectorTripCount(), getStepVector(), packScalarIntoVectorValue(), and vectorizeInterleaveGroup().
|
protected |
The profitablity analysis.
Definition at line 742 of file LoopVectorize.cpp.
Referenced by completeLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), emitSCEVChecks(), fixFixedOrderRecurrence(), fixReduction(), fixVectorizedLoop(), getOrCreateVectorTripCount(), llvm::LoopVectorizationCostModel::getVectorCallCost(), isIndvarOverflowCheckKnownFalse(), llvm::LoopVectorizationCostModel::setCostBasedWideningDecision(), truncateToMinimalBitwidths(), useOrderedReductions(), and vectorizeInterleaveGroup().
|
protected |
Dominator Tree.
Definition at line 681 of file LoopVectorize.cpp.
Referenced by completeLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), emitSCEVChecks(), llvm::LoopVectorizationPlanner::executePlan(), getBroadcastInstrs(), getScalarEpilogueLowering(), and processLoopInVPlanNativePath().
Definition at line 749 of file LoopVectorize.cpp.
Referenced by createInductionResumeValue(), and fixVectorizedLoop().
|
protected |
The legality analysis.
Definition at line 739 of file LoopVectorize.cpp.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
protected |
Loop Info.
Definition at line 678 of file LoopVectorize.cpp.
Referenced by llvm::LoopVectorizationCostModel::calculateRegisterUsage(), collectSupportedLoops(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), llvm::VPWidenMemoryInstructionRecipe::execute(), llvm::LoopVectorizationPlanner::executePlan(), fixVectorizedLoop(), getScalarEpilogueLowering(), processLoopInVPlanNativePath(), and sinkScalarOperands().
|
protected |
A list of all bypass blocks. The first block is the entry of the loop.
Definition at line 727 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createInductionResumeValues(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), and emitSCEVChecks().
|
protected |
The unique ExitBlock of the scalar loop if one exists.
Note that there can be multiple exiting edges reaching this block.
Definition at line 721 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitSCEVChecks(), fixFixedOrderRecurrence(), and fixReduction().
|
protected |
Middle Block between the vector and the scalar.
Definition at line 717 of file LoopVectorize.cpp.
Referenced by completeLoopSkeleton(), createInductionResumeValue(), createVectorLoopSkeleton(), fixFixedOrderRecurrence(), fixReduction(), and fixVectorizedLoop().
|
protected |
The scalar loop body.
Definition at line 724 of file LoopVectorize.cpp.
Referenced by createVectorLoopSkeleton(), and fixVectorizedLoop().
|
protected |
The scalar-loop preheader.
Definition at line 714 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createInductionResumeValue(), createInductionResumeValues(), createVectorizedLoopSkeleton(), createVectorLoopSkeleton(), fixFixedOrderRecurrence(), and fixReduction().
|
protected |
The vector-loop preheader.
Definition at line 711 of file LoopVectorize.cpp.
Referenced by completeLoopSkeleton(), llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createInductionResumeValue(), createVectorLoopSkeleton(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerEpilogueLoop::emitMinimumVectorEpilogueIterCountCheck(), emitSCEVChecks(), and getBroadcastInstrs().
|
protected |
Definition at line 699 of file LoopVectorize.cpp.
Referenced by emitIterationCountCheck(), and InnerLoopVectorizer().
|
protected |
Definition at line 757 of file LoopVectorize.cpp.
Referenced by emitMemRuntimeChecks(), emitSCEVChecks(), and InnerLoopVectorizer().
|
protected |
Interface to emit optimization remarks.
Definition at line 693 of file LoopVectorize.cpp.
Referenced by checkMixedPrecision(), collectSupportedLoops(), llvm::LoopVectorizationCostModel::computeMaxVF(), emitInvalidCostRemarks(), emitMemRuntimeChecks(), llvm::LoopVectorizationPlanner::executePlan(), isExplicitVecOuterLoop(), llvm::LoopVectorizationPlanner::plan(), processLoopInVPlanNativePath(), llvm::LoopVectorizationCostModel::runtimeChecksRequired(), and llvm::LoopVectorizationCostModel::selectVectorizationFactor().
|
protected |
The original loop.
Definition at line 670 of file LoopVectorize.cpp.
Referenced by addUsersInExitBlock(), completeLoopSkeleton(), llvm::VPRecipeBuilder::createBlockInMask(), llvm::VPRecipeBuilder::createEdgeMask(), CreateStepValue(), createVectorLoopSkeleton(), createWidenInductionRecipes(), emitMemRuntimeChecks(), llvm::LoopVectorizationPlanner::executePlan(), llvm::VPRecipeBuilder::fixHeaderPhis(), fixReduction(), fixupIVUsers(), fixVectorizedLoop(), getBroadcastInstrs(), InnerLoopVectorizer(), llvm::LoopVectorizationPlanner::plan(), llvm::LoopVectorizationPlanner::planInVPlanNativePath(), llvm::EpilogueVectorizerMainLoop::printDebugTracesAtEnd(), llvm::EpilogueVectorizerEpilogueLoop::printDebugTracesAtEnd(), and llvm::VPRecipeBuilder::tryToCreateWidenRecipe().
|
protected |
Store instructions that were predicated.
Definition at line 730 of file LoopVectorize.cpp.
Referenced by fixVectorizedLoop(), and scalarizeInstruction().
|
protected |
A wrapper around ScalarEvolution used to add runtime SCEV checks.
Applies dynamic knowledge to simplify SCEV expressions and converts them to a more usable form.
Definition at line 675 of file LoopVectorize.cpp.
Referenced by llvm::LoopVectorizationCostModel::computeMaxVF(), createInductionResumeValue(), llvm::LoopVectorizationPlanner::executePlan(), fixupIVUsers(), fixVectorizedLoop(), getAddressAccessSCEV(), getOrCreateTripCount(), llvm::LoopVectorizationCostModel::isMoreProfitable(), processLoopInVPlanNativePath(), llvm::LoopVectorizationCostModel::runtimeChecksRequired(), and llvm::LoopVectorizationCostModel::selectInterleaveCount().
|
protected |
Definition at line 753 of file LoopVectorize.cpp.
Referenced by getScalarEpilogueLowering(), InnerLoopVectorizer(), and processLoopInVPlanNativePath().
|
protected |
Definition at line 766 of file LoopVectorize.cpp.
Referenced by fixReduction(), and getReductionResumeValue().
|
protected |
Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.
Definition at line 761 of file LoopVectorize.cpp.
Referenced by emitMemRuntimeChecks(), and emitSCEVChecks().
|
protected |
Target Library Info.
Definition at line 684 of file LoopVectorize.cpp.
Referenced by getScalarEpilogueLowering(), llvm::LoopVectorizationCostModel::getVectorCallCost(), llvm::LoopVectorizationCostModel::getVectorIntrinsicCost(), and processLoopInVPlanNativePath().
|
protected |
Trip count of the original loop.
Definition at line 733 of file LoopVectorize.cpp.
Referenced by addCanonicalIVRecipes(), llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop(), and getOrCreateTripCount().
|
protected |
Target Transform Info.
Definition at line 687 of file LoopVectorize.cpp.
|
protected |
The vectorization unroll factor to use.
Each scalar is vectorized to this many different vector instructions.
Definition at line 703 of file LoopVectorize.cpp.
Referenced by clearReductionWrapFlags(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), fixFixedOrderRecurrence(), fixReduction(), fixVectorizedLoop(), getOrCreateVectorTripCount(), isIndvarOverflowCheckKnownFalse(), truncateToMinimalBitwidths(), and vectorizeInterleaveGroup().
|
protected |
Trip count of the widened loop (TripCount - TripCount % (VF*UF))
Definition at line 736 of file LoopVectorize.cpp.
Referenced by completeLoopSkeleton(), createInductionResumeValue(), fixupIVUsers(), and getOrCreateVectorTripCount().
|
protected |
The vectorization SIMD factor to use.
Each vector will have this many vector elements.
Definition at line 697 of file LoopVectorize.cpp.
Referenced by areRuntimeChecksProfitable(), llvm::LoopVectorizationPlanner::buildVPlans(), llvm::LoopVectorizationCostModel::calculateRegisterUsage(), llvm::LoopVectorizationCostModel::collectInstsToScalarize(), completeLoopSkeleton(), createBitOrPointerCast(), createVectorLoopSkeleton(), createWidenInductionRecipes(), emitIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitIterationCountCheck(), emitSCEVChecks(), fixFixedOrderRecurrence(), fixReduction(), fixVectorizedLoop(), llvm::VPTransformState::get(), llvm::LoopVectorizationPlanner::getBestPlanFor(), getBroadcastInstrs(), llvm::LoopVectorizationCostModel::getDivRemSpeculationCost(), getOrCreateVectorTripCount(), getStepVector(), llvm::LoopVectorizationCostModel::getVectorCallCost(), llvm::LoopVectorizationCostModel::getVectorIntrinsicCost(), llvm::VPRecipeBuilder::handleReplication(), llvm::LoopVectorizationCostModel::interleavedAccessCanBeWidened(), isIndvarOverflowCheckKnownFalse(), llvm::LoopVectorizationCostModel::isScalarWithPredication(), MaybeVectorizeType(), llvm::LoopVectorizationCostModel::memoryInstructionCanBeWidened(), packScalarIntoVectorValue(), llvm::LoopVectorizationPlanner::plan(), llvm::LoopVectorizationPlanner::planInVPlanNativePath(), processLoopInVPlanNativePath(), llvm::LoopVectorizationCostModel::selectInterleaveCount(), llvm::LoopVectorizationCostModel::setCostBasedWideningDecision(), llvm::VPRecipeBuilder::tryToCreateWidenRecipe(), and vectorizeInterleaveGroup().