LLVM  14.0.0git
Public Types | Public Member Functions | Protected Types | Protected Member Functions | Protected Attributes | Friends | List of all members
llvm::InnerLoopVectorizer Class Reference

InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF). More...

Inheritance diagram for llvm::InnerLoopVectorizer:
Inheritance graph
[legend]
Collaboration diagram for llvm::InnerLoopVectorizer:
Collaboration graph
[legend]

Public Types

using VectorParts = SmallVector< Value *, 2 >
 A type for vectorized values in the new loop. More...
 

Public Member Functions

 InnerLoopVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo *TTI, AssumptionCache *AC, OptimizationRemarkEmitter *ORE, ElementCount VecWidth, unsigned UnrollFactor, LoopVectorizationLegality *LVL, LoopVectorizationCostModel *CM, BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, GeneratedRTChecks &RTChecks)
 
virtual ~InnerLoopVectorizer ()=default
 
virtual BasicBlockcreateVectorizedLoopSkeleton ()
 Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder. More...
 
void widenInstruction (Instruction &I, VPValue *Def, VPUser &Operands, VPTransformState &State)
 Widen a single instruction within the innermost loop. More...
 
void widenCallInstruction (CallInst &I, VPValue *Def, VPUser &ArgOperands, VPTransformState &State)
 Widen a single call instruction within the innermost loop. More...
 
void widenSelectInstruction (SelectInst &I, VPValue *VPDef, VPUser &Operands, bool InvariantCond, VPTransformState &State)
 Widen a single select instruction within the innermost loop. More...
 
void fixVectorizedLoop (VPTransformState &State)
 Fix the vectorized code, taking care of header phi's, live-outs, and more. More...
 
bool areSafetyChecksAdded ()
 
void widenGEP (GetElementPtrInst *GEP, VPValue *VPDef, VPUser &Indices, unsigned UF, ElementCount VF, bool IsPtrLoopInvariant, SmallBitVector &IsIndexLoopInvariant, VPTransformState &State)
 Vectorize a single GetElementPtrInst based on information gathered and decisions taken during planning. More...
 
void widenPHIInstruction (Instruction *PN, VPWidenPHIRecipe *PhiR, VPTransformState &State)
 Vectorize a single first-order recurrence or pointer induction PHINode in a block. More...
 
void scalarizeInstruction (Instruction *Instr, VPValue *Def, VPUser &Operands, const VPIteration &Instance, bool IfPredicateInstr, VPTransformState &State)
 A helper function to scalarize a single Instruction in the innermost loop. More...
 
void widenIntOrFpInduction (PHINode *IV, Value *Start, TruncInst *Trunc, VPValue *Def, VPValue *CastDef, VPTransformState &State)
 Widen an integer or floating-point induction variable IV. More...
 
void packScalarIntoVectorValue (VPValue *Def, const VPIteration &Instance, VPTransformState &State)
 Construct the vector value of a scalarized value V one lane at a time. More...
 
void vectorizeInterleaveGroup (const InterleaveGroup< Instruction > *Group, ArrayRef< VPValue * > VPDefs, VPTransformState &State, VPValue *Addr, ArrayRef< VPValue * > StoredValues, VPValue *BlockInMask=nullptr)
 Try to vectorize interleaved access group Group with the base address given in Addr, optionally masking the vector operations if BlockInMask is non-null. More...
 
void vectorizeMemoryInstruction (Instruction *Instr, VPTransformState &State, VPValue *Def, VPValue *Addr, VPValue *StoredValue, VPValue *BlockInMask, bool ConsecutiveStride, bool Reverse)
 Vectorize Load and Store instructions with the base address given in Addr, optionally masking the vector operations if BlockInMask is non-null. More...
 
void setDebugLocFromInst (const Value *V, Optional< IRBuilder<> * > CustomBuilder=None)
 Set the debug location in the builder Ptr using the debug location in V. More...
 
void fixNonInductionPHIs (VPTransformState &State)
 Fix the non-induction PHIs in the OrigPHIsToFix vector. More...
 
bool useOrderedReductions (RecurrenceDescriptor &RdxDesc)
 Returns true if the reordering of FP operations is not allowed, but we are able to vectorize with strict in-order reductions for the given RdxDesc. More...
 
virtual ValuegetBroadcastInstrs (Value *V)
 Create a broadcast instruction. More...
 

Protected Types

using PhiVector = SmallVector< PHINode *, 4 >
 A small list of PHINodes. More...
 
using ScalarParts = SmallVector< SmallVector< Value *, 4 >, 2 >
 A type for scalarized values in the new loop. More...
 

Protected Member Functions

void fixupIVUsers (PHINode *OrigPhi, const InductionDescriptor &II, Value *CountRoundDown, Value *EndValue, BasicBlock *MiddleBlock)
 Set up the values of the IVs correctly when exiting the vector loop. More...
 
PHINodecreateInductionVariable (Loop *L, Value *Start, Value *End, Value *Step, Instruction *DL)
 Create a new induction variable inside L. More...
 
void fixCrossIterationPHIs (VPTransformState &State)
 Handle all cross-iteration phis in the header. More...
 
void fixFirstOrderRecurrence (VPWidenPHIRecipe *PhiR, VPTransformState &State)
 Create the exit value of first order recurrences in the middle block and update their users. More...
 
void fixReduction (VPReductionPHIRecipe *Phi, VPTransformState &State)
 Create code for the loop exit value of the reduction. More...
 
void clearReductionWrapFlags (const RecurrenceDescriptor &RdxDesc, VPTransformState &State)
 Clear NSW/NUW flags from reduction instructions if necessary. More...
 
void fixLCSSAPHIs (VPTransformState &State)
 Fixup the LCSSA phi nodes in the unique exit block. More...
 
void sinkScalarOperands (Instruction *PredInst)
 Iteratively sink the scalarized operands of a predicated instruction into the block that was created for it. More...
 
void truncateToMinimalBitwidths (VPTransformState &State)
 Shrinks vector element sizes to the smallest bitwidth they can be legally represented as. More...
 
virtual ValuegetStepVector (Value *Val, int StartIdx, Value *Step, Instruction::BinaryOps Opcode=Instruction::BinaryOpsEnd)
 This function adds (StartIdx * Step, (StartIdx + 1) * Step, (StartIdx + 2) * Step, ...) to each vector element of Val. More...
 
void buildScalarSteps (Value *ScalarIV, Value *Step, Instruction *EntryVal, const InductionDescriptor &ID, VPValue *Def, VPValue *CastDef, VPTransformState &State)
 Compute scalar induction steps. More...
 
void createVectorIntOrFpInductionPHI (const InductionDescriptor &II, Value *Step, Value *Start, Instruction *EntryVal, VPValue *Def, VPValue *CastDef, VPTransformState &State)
 Create a vector induction phi node based on an existing scalar one. More...
 
bool shouldScalarizeInstruction (Instruction *I) const
 Returns true if an instruction I should be scalarized instead of vectorized for the chosen vectorization factor. More...
 
bool needsScalarInduction (Instruction *IV) const
 Returns true if we should generate a scalar version of IV. More...
 
void recordVectorLoopValueForInductionCast (const InductionDescriptor &ID, const Instruction *EntryVal, Value *VectorLoopValue, VPValue *CastDef, VPTransformState &State, unsigned Part, unsigned Lane=UINT_MAX)
 If there is a cast involved in the induction variable ID, which should be ignored in the vectorized loop body, this function records the VectorLoopValue of the respective Phi also as the VectorLoopValue of the cast. More...
 
virtual ValuereverseVector (Value *Vec)
 Generate a shuffle sequence that will reverse the vector Vec. More...
 
ValuegetOrCreateTripCount (Loop *NewLoop)
 Returns (and creates if needed) the original loop trip count. More...
 
ValuegetOrCreateVectorTripCount (Loop *NewLoop)
 Returns (and creates if needed) the trip count of the widened loop. More...
 
ValuecreateBitOrPointerCast (Value *V, VectorType *DstVTy, const DataLayout &DL)
 Returns a bitcasted value to the requested vector type. More...
 
void emitMinimumIterationCountCheck (Loop *L, BasicBlock *Bypass)
 Emit a bypass check to see if the vector trip count is zero, including if it overflows. More...
 
BasicBlockemitSCEVChecks (Loop *L, BasicBlock *Bypass)
 Emit a bypass check to see if all of the SCEV assumptions we've had to make are correct. More...
 
BasicBlockemitMemRuntimeChecks (Loop *L, BasicBlock *Bypass)
 Emit bypass checks to check any memory assumptions we may have made. More...
 
ValueemitTransformedIndex (IRBuilder<> &B, Value *Index, ScalarEvolution *SE, const DataLayout &DL, const InductionDescriptor &ID) const
 Compute the transformed value of Index at offset StartValue using step StepValue. More...
 
LoopcreateVectorLoopSkeleton (StringRef Prefix)
 Emit basic blocks (prefixed with Prefix) for the iteration check, vector loop preheader, middle block and scalar preheader. More...
 
void createInductionResumeValues (Loop *L, Value *VectorTripCount, std::pair< BasicBlock *, Value * > AdditionalBypass={nullptr, nullptr})
 Create new phi nodes for the induction variables to resume iteration count in the scalar epilogue, from where the vectorized loop left off (given by VectorTripCount). More...
 
BasicBlockcompleteLoopSkeleton (Loop *L, MDNode *OrigLoopID)
 Complete the loop skeleton by adding debug MDs, creating appropriate conditional branches in the middle block, preparing the builder and running the verifier. More...
 
void addNewMetadata (Instruction *To, const Instruction *Orig)
 Add additional metadata to To that was not present on Orig. More...
 
void addMetadata (Instruction *To, Instruction *From)
 Add metadata from one instruction to another. More...
 
void addMetadata (ArrayRef< Value * > To, Instruction *From)
 Similar to the previous function but it adds the metadata to a vector of instructions. More...
 
virtual void printDebugTracesAtStart ()
 Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested. More...
 
virtual void printDebugTracesAtEnd ()
 

Protected Attributes

LoopOrigLoop
 The original loop. More...
 
PredicatedScalarEvolutionPSE
 A wrapper around ScalarEvolution used to add runtime SCEV checks. More...
 
LoopInfoLI
 Loop Info. More...
 
DominatorTreeDT
 Dominator Tree. More...
 
AAResultsAA
 Alias Analysis. More...
 
const TargetLibraryInfoTLI
 Target Library Info. More...
 
const TargetTransformInfoTTI
 Target Transform Info. More...
 
AssumptionCacheAC
 Assumption Cache. More...
 
OptimizationRemarkEmitterORE
 Interface to emit optimization remarks. More...
 
std::unique_ptr< LoopVersioningLVer
 LoopVersioning. More...
 
ElementCount VF
 The vectorization SIMD factor to use. More...
 
unsigned UF
 The vectorization unroll factor to use. More...
 
IRBuilder Builder
 The builder that we use. More...
 
BasicBlockLoopVectorPreHeader
 The vector-loop preheader. More...
 
BasicBlockLoopScalarPreHeader
 The scalar-loop preheader. More...
 
BasicBlockLoopMiddleBlock
 Middle Block between the vector and the scalar. More...
 
BasicBlockLoopExitBlock
 The unique ExitBlock of the scalar loop if one exists. More...
 
BasicBlockLoopVectorBody
 The vector loop body. More...
 
BasicBlockLoopScalarBody
 The scalar loop body. More...
 
SmallVector< BasicBlock *, 4 > LoopBypassBlocks
 A list of all bypass blocks. The first block is the entry of the loop. More...
 
PHINodeInduction = nullptr
 The new Induction variable which was added to the new block. More...
 
PHINodeOldInduction = nullptr
 The induction variable of the old basic block. More...
 
SmallVector< Instruction *, 4 > PredicatedInstructions
 Store instructions that were predicated. More...
 
ValueTripCount = nullptr
 Trip count of the original loop. More...
 
ValueVectorTripCount = nullptr
 Trip count of the widened loop (TripCount - TripCount % (VF*UF)) More...
 
LoopVectorizationLegalityLegal
 The legality analysis. More...
 
LoopVectorizationCostModelCost
 The profitablity analysis. More...
 
bool AddedSafetyChecks = false
 
DenseMap< PHINode *, Value * > IVEndValues
 
SmallVector< PHINode *, 8 > OrigPHIsToFix
 
BlockFrequencyInfoBFI
 BFI and PSI are used to check for profile guided size optimizations. More...
 
ProfileSummaryInfoPSI
 
bool OptForSizeBasedOnProfile
 
GeneratedRTChecksRTChecks
 Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable. More...
 

Friends

class LoopVectorizationPlanner
 

Detailed Description

InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF).

This class performs the widening of scalars into vectors, or multiple scalars. This class also implements the following features:

Definition at line 445 of file LoopVectorize.cpp.

Member Typedef Documentation

◆ PhiVector

A small list of PHINodes.

Definition at line 573 of file LoopVectorize.cpp.

◆ ScalarParts

A type for scalarized values in the new loop.

Each value from the original loop, when scalarized, is represented by UF x VF scalar values in the new unrolled loop, where UF is the unroll factor and VF is the vectorization factor.

Definition at line 579 of file LoopVectorize.cpp.

◆ VectorParts

A type for vectorized values in the new loop.

Each value from the original loop, when vectorized, is represented by UF vector values in the new unrolled loop, where UF is the unroll factor.

Definition at line 497 of file LoopVectorize.cpp.

Constructor & Destructor Documentation

◆ InnerLoopVectorizer()

llvm::InnerLoopVectorizer::InnerLoopVectorizer ( Loop OrigLoop,
PredicatedScalarEvolution PSE,
LoopInfo LI,
DominatorTree DT,
const TargetLibraryInfo TLI,
const TargetTransformInfo TTI,
AssumptionCache AC,
OptimizationRemarkEmitter ORE,
ElementCount  VecWidth,
unsigned  UnrollFactor,
LoopVectorizationLegality LVL,
LoopVectorizationCostModel CM,
BlockFrequencyInfo BFI,
ProfileSummaryInfo PSI,
GeneratedRTChecks RTChecks 
)
inline

◆ ~InnerLoopVectorizer()

virtual llvm::InnerLoopVectorizer::~InnerLoopVectorizer ( )
virtualdefault

Member Function Documentation

◆ addMetadata() [1/2]

void InnerLoopVectorizer::addMetadata ( ArrayRef< Value * >  To,
Instruction From 
)
protected

Similar to the previous function but it adds the metadata to a vector of instructions.

Definition at line 1176 of file LoopVectorize.cpp.

References addMetadata(), From, and I.

◆ addMetadata() [2/2]

void InnerLoopVectorizer::addMetadata ( Instruction To,
Instruction From 
)
protected

Add metadata from one instruction to another.

This includes both the original MDs from From and additional ones (

See also
addNewMetadata). Use this for newly created instructions in the vector loop.

Definition at line 1170 of file LoopVectorize.cpp.

References addNewMetadata(), From, and llvm::propagateMetadata().

Referenced by addMetadata(), vectorizeMemoryInstruction(), widenCallInstruction(), widenGEP(), widenInstruction(), and widenSelectInstruction().

◆ addNewMetadata()

void InnerLoopVectorizer::addNewMetadata ( Instruction To,
const Instruction Orig 
)
protected

Add additional metadata to To that was not present on Orig.

Currently this is used to add the noalias annotations based on the inserted memchecks. Use this for instructions that are cloned into the vector loop.

Definition at line 1162 of file LoopVectorize.cpp.

References LVer.

Referenced by addMetadata(), and scalarizeInstruction().

◆ areSafetyChecksAdded()

bool llvm::InnerLoopVectorizer::areSafetyChecksAdded ( )
inline

Definition at line 492 of file LoopVectorize.cpp.

References AddedSafetyChecks.

Referenced by llvm::LoopVectorizePass::processLoop().

◆ buildScalarSteps()

void InnerLoopVectorizer::buildScalarSteps ( Value ScalarIV,
Value Step,
Instruction EntryVal,
const InductionDescriptor ID,
VPValue Def,
VPValue CastDef,
VPTransformState State 
)
protected

◆ clearReductionWrapFlags()

void InnerLoopVectorizer::clearReductionWrapFlags ( const RecurrenceDescriptor RdxDesc,
VPTransformState State 
)
protected

◆ completeLoopSkeleton()

BasicBlock * InnerLoopVectorizer::completeLoopSkeleton ( Loop L,
MDNode OrigLoopID 
)
protected

◆ createBitOrPointerCast()

Value * InnerLoopVectorizer::createBitOrPointerCast ( Value V,
VectorType DstVTy,
const DataLayout DL 
)
protected

Returns a bitcasted value to the requested vector type.

Also handles bitcasts of vector<float> <-> vector<pointer> types.

Definition at line 3211 of file LoopVectorize.cpp.

References assert(), Builder, llvm::IRBuilderBase::CreateBitOrPointerCast(), DL, llvm::FixedVectorType::get(), llvm::Value::getContext(), llvm::Type::getIntNTy(), llvm::Value::getType(), llvm::CastInst::isBitOrNoopPointerCastable(), llvm::Type::isFloatingPointTy(), llvm::Type::isPointerTy(), and VF.

Referenced by vectorizeInterleaveGroup().

◆ createInductionResumeValues()

void InnerLoopVectorizer::createInductionResumeValues ( Loop L,
Value VectorTripCount,
std::pair< BasicBlock *, Value * >  AdditionalBypass = {nullptr, nullptr} 
)
protected

◆ createInductionVariable()

PHINode * InnerLoopVectorizer::createInductionVariable ( Loop L,
Value Start,
Value End,
Value Step,
Instruction DL 
)
protected

◆ createVectorIntOrFpInductionPHI()

void InnerLoopVectorizer::createVectorIntOrFpInductionPHI ( const InductionDescriptor II,
Value Step,
Value Start,
Instruction EntryVal,
VPValue Def,
VPValue CastDef,
VPTransformState State 
)
protected

Create a vector induction phi node based on an existing scalar one.

EntryVal is the value from the original loop that maps to the vector phi node, and Step is the loop-invariant step. If EntryVal is a truncate instruction, instead of widening the original IV, we widen a version of the IV truncated to EntryVal's type.

Definition at line 2272 of file LoopVectorize.cpp.

◆ createVectorizedLoopSkeleton()

BasicBlock * InnerLoopVectorizer::createVectorizedLoopSkeleton ( )
virtual

Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder.

Control flow is generated around the vectorized (and scalar epilogue) loops consisting of various checks and bypasses. Return the pre-header block of the new loop. In the case of epilogue vectorization, this function is overriden to handle the more complex control flow around the loops.

Reimplemented in llvm::InnerLoopAndEpilogueVectorizer.

Definition at line 3656 of file LoopVectorize.cpp.

References Builder, completeLoopSkeleton(), createInductionResumeValues(), createInductionVariable(), createStepForVF(), createVectorLoopSkeleton(), emitMemRuntimeChecks(), llvm::EpilogueVectorizerMainLoop::emitMinimumIterationCountCheck(), emitSCEVChecks(), llvm::ConstantInt::get(), getDebugLocFromInstOrOperands(), llvm::BasicBlock::getFirstInsertionPt(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::Loop::getLoopID(), getOrCreateTripCount(), getOrCreateVectorTripCount(), llvm::LoopVectorizationLegality::getPrimaryInduction(), llvm::LoopVectorizationLegality::getWidestInductionType(), Induction, Legal, LoopScalarPreHeader, OldInduction, OrigLoop, llvm::IRBuilderBase::SetInsertPoint(), UF, and VF.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ createVectorLoopSkeleton()

Loop * InnerLoopVectorizer::createVectorLoopSkeleton ( StringRef  Prefix)
protected

◆ emitMemRuntimeChecks()

BasicBlock * InnerLoopVectorizer::emitMemRuntimeChecks ( Loop L,
BasicBlock Bypass 
)
protected

◆ emitMinimumIterationCountCheck()

void InnerLoopVectorizer::emitMinimumIterationCountCheck ( Loop L,
BasicBlock Bypass 
)
protected

◆ emitSCEVChecks()

BasicBlock * InnerLoopVectorizer::emitSCEVChecks ( Loop L,
BasicBlock Bypass 
)
protected

◆ emitTransformedIndex()

Value * InnerLoopVectorizer::emitTransformedIndex ( IRBuilder<> &  B,
Value Index,
ScalarEvolution SE,
const DataLayout DL,
const InductionDescriptor ID 
) const
protected

Compute the transformed value of Index at offset StartValue using step StepValue.

For integer induction, returns StartValue + Index * StepValue. For pointer induction, returns StartValue[Index * StepValue]. FIXME: The newly created binary instructions should contain nsw/nuw flags, which can be found from the original scalar operations.

Definition at line 3359 of file LoopVectorize.cpp.

References assert(), B, CreateAdd(), CreateMul(), DL, llvm::VectorType::getElementCount(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::BasicBlock::getTerminator(), llvm::InductionDescriptor::IK_FpInduction, llvm::InductionDescriptor::IK_IntInduction, llvm::InductionDescriptor::IK_NoInduction, llvm::InductionDescriptor::IK_PtrInduction, Index, LI, llvm_unreachable, LoopVectorBody, Offset, X, and Y.

Referenced by createInductionResumeValues(), fixupIVUsers(), and widenPHIInstruction().

◆ fixCrossIterationPHIs()

void InnerLoopVectorizer::fixCrossIterationPHIs ( VPTransformState State)
protected

◆ fixFirstOrderRecurrence()

void InnerLoopVectorizer::fixFirstOrderRecurrence ( VPWidenPHIRecipe PhiR,
VPTransformState State 
)
protected

◆ fixLCSSAPHIs()

void InnerLoopVectorizer::fixLCSSAPHIs ( VPTransformState State)
protected

Fixup the LCSSA phi nodes in the unique exit block.

This simply means we need to add the appropriate incoming value from the middle block as exiting edges from the scalar epilogue loop (if present) are already in place, and we exit the vector loop exclusively to the middle block.

Definition at line 4480 of file LoopVectorize.cpp.

References Builder, Cost, llvm::VPTransformState::get(), llvm::VPLane::getFirstLane(), llvm::VPLane::getLastLaneForVF(), llvm::BasicBlock::getTerminator(), llvm::VPlan::getVPValue(), llvm::Loop::isLoopInvariant(), llvm::LoopVectorizationCostModel::isUniformAfterVectorization(), LoopExitBlock, LoopMiddleBlock, OrigLoop, llvm::BasicBlock::phis(), llvm::VPTransformState::Plan, llvm::IRBuilderBase::SetInsertPoint(), UF, and VF.

◆ fixNonInductionPHIs()

void InnerLoopVectorizer::fixNonInductionPHIs ( VPTransformState State)

◆ fixReduction()

void InnerLoopVectorizer::fixReduction ( VPReductionPHIRecipe Phi,
VPTransformState State 
)
protected

Create code for the loop exit value of the reduction.

Definition at line 4281 of file LoopVectorize.cpp.

References llvm::PHINode::addIncoming(), assert(), Builder, clearReductionWrapFlags(), Cost, llvm::PHINode::Create(), llvm::IRBuilderBase::CreateBinOp(), llvm::createMinMaxOp(), llvm::createSelectCmpOp(), llvm::IRBuilderBase::CreateSExt(), llvm::createTargetReduction(), llvm::IRBuilderBase::CreateTrunc(), llvm::IRBuilderBase::CreateZExt(), E, llvm::LoopVectorizationCostModel::foldTailByMasking(), llvm::VPTransformState::get(), llvm::VectorType::get(), llvm::VPWidenPHIRecipe::getBackedgeValue(), llvm::PHINode::getBasicBlockIndex(), llvm::RecurrenceDescriptor::getFastMathFlags(), llvm::BasicBlock::getFirstInsertionPt(), llvm::RecurrenceDescriptor::getLoopExitInstr(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::RecurrenceDescriptor::getOpcode(), llvm::VPReductionPHIRecipe::getRecurrenceDescriptor(), llvm::RecurrenceDescriptor::getRecurrenceKind(), llvm::RecurrenceDescriptor::getRecurrenceStartValue(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::VPValue::getUnderlyingValue(), llvm::VPDef::getVPSingleValue(), I, llvm::is_contained(), llvm::VPReductionPHIRecipe::isInLoop(), llvm::VPReductionPHIRecipe::isOrdered(), llvm::LoopVectorizationLegality::isReductionVariable(), llvm::RecurrenceDescriptor::isSelectCmpRecurrenceKind(), llvm::RecurrenceDescriptor::isSigned(), llvm::ElementCount::isVector(), Legal, LI, LoopBypassBlocks, LoopExitBlock, LoopMiddleBlock, LoopScalarPreHeader, LoopVectorBody, OrigLoop, llvm::BasicBlock::phis(), PreferPredicatedReductionSelect, llvm::TargetTransformInfo::preferPredicatedReductionSelect(), replaceUsesOfWith(), llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::VPTransformState::reset(), setDebugLocFromInst(), llvm::IRBuilderBase::setFastMathFlags(), llvm::PHINode::setIncomingValue(), llvm::IRBuilderBase::SetInsertPoint(), UF, llvm::Value::users(), and VF.

Referenced by fixCrossIterationPHIs().

◆ fixupIVUsers()

void InnerLoopVectorizer::fixupIVUsers ( PHINode OrigPhi,
const InductionDescriptor II,
Value CountRoundDown,
Value EndValue,
BasicBlock MiddleBlock 
)
protected

◆ fixVectorizedLoop()

void InnerLoopVectorizer::fixVectorizedLoop ( VPTransformState State)

◆ getBroadcastInstrs()

Value * InnerLoopVectorizer::getBroadcastInstrs ( Value V)
virtual

Create a broadcast instruction.

This method generates a broadcast instruction (shuffle) for loop invariant values and for the induction value. If this is the induction variable then we extend it to N, N+1, ... this is needed because each iteration in the loop corresponds to a SIMD element.

Definition at line 2253 of file LoopVectorize.cpp.

References Builder, llvm::IRBuilderBase::CreateVectorSplat(), llvm::DominatorTree::dominates(), DT, llvm::Instruction::getParent(), llvm::BasicBlock::getTerminator(), llvm::Loop::isLoopInvariant(), LoopVectorPreHeader, OrigLoop, llvm::IRBuilderBase::SetInsertPoint(), and VF.

◆ getOrCreateTripCount()

Value * InnerLoopVectorizer::getOrCreateTripCount ( Loop NewLoop)
protected

◆ getOrCreateVectorTripCount()

Value * InnerLoopVectorizer::getOrCreateVectorTripCount ( Loop NewLoop)
protected

◆ getStepVector()

Value * InnerLoopVectorizer::getStepVector ( Value Val,
int  StartIdx,
Value Step,
Instruction::BinaryOps  Opcode = Instruction::BinaryOpsEnd 
)
protectedvirtual

◆ needsScalarInduction()

bool InnerLoopVectorizer::needsScalarInduction ( Instruction IV) const
protected

Returns true if we should generate a scalar version of IV.

Definition at line 2362 of file LoopVectorize.cpp.

References llvm::any_of(), llvm::LoopBase< BlockT, LoopT >::contains(), I, OrigLoop, shouldScalarizeInstruction(), and llvm::Value::users().

◆ packScalarIntoVectorValue()

void InnerLoopVectorizer::packScalarIntoVectorValue ( VPValue Def,
const VPIteration Instance,
VPTransformState State 
)

◆ printDebugTracesAtEnd()

virtual void llvm::InnerLoopVectorizer::printDebugTracesAtEnd ( )
inlineprotectedvirtual

◆ printDebugTracesAtStart()

virtual void llvm::InnerLoopVectorizer::printDebugTracesAtStart ( )
inlineprotectedvirtual

Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.

Reimplemented in llvm::EpilogueVectorizerEpilogueLoop, and llvm::EpilogueVectorizerMainLoop.

Definition at line 757 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationPlanner::executePlan().

◆ recordVectorLoopValueForInductionCast()

void InnerLoopVectorizer::recordVectorLoopValueForInductionCast ( const InductionDescriptor ID,
const Instruction EntryVal,
Value VectorLoopValue,
VPValue CastDef,
VPTransformState State,
unsigned  Part,
unsigned  Lane = UINT_MAX 
)
protected

If there is a cast involved in the induction variable ID, which should be ignored in the vectorized loop body, this function records the VectorLoopValue of the respective Phi also as the VectorLoopValue of the cast.

We had already proved that the casted Phi is equal to the uncasted Phi in the vectorized loop (under a runtime guard), and therefore there is no need to vectorize the cast - the same value can be used in the vector loop for both the Phi and the cast. If VectorLoopValue is a scalarized value, Lane is also specified, Otherwise, VectorLoopValue is a widened/vectorized value.

EntryVal is the value from the original loop that maps to the vector phi node and is used to distinguish what is the IV currently being processed - original one (if EntryVal is a phi corresponding to the original IV) or the "newly-created" one based on the proof mentioned above (see also buildScalarSteps() and createVectorIntOrFPInductionPHI()). In the latter case EntryVal is a TruncInst and we must not record anything for that IV, but it's error-prone to expect callers of this routine to care about that, hence this explicit parameter.

Definition at line 2372 of file LoopVectorize.cpp.

References assert(), and llvm::VPTransformState::set().

Referenced by buildScalarSteps().

◆ reverseVector()

Value * InnerLoopVectorizer::reverseVector ( Value Vec)
protectedvirtual

Generate a shuffle sequence that will reverse the vector Vec.

Definition at line 2658 of file LoopVectorize.cpp.

References assert(), Builder, llvm::IRBuilderBase::CreateVectorReverse(), llvm::Value::getType(), and llvm::Type::isVectorTy().

Referenced by vectorizeInterleaveGroup(), and vectorizeMemoryInstruction().

◆ scalarizeInstruction()

void InnerLoopVectorizer::scalarizeInstruction ( Instruction Instr,
VPValue Def,
VPUser Operands,
const VPIteration Instance,
bool  IfPredicateInstr,
VPTransformState State 
)

◆ setDebugLocFromInst()

void InnerLoopVectorizer::setDebugLocFromInst ( const Value V,
Optional< IRBuilder<> * >  CustomBuilder = None 
)

◆ shouldScalarizeInstruction()

bool InnerLoopVectorizer::shouldScalarizeInstruction ( Instruction I) const
protected

Returns true if an instruction I should be scalarized instead of vectorized for the chosen vectorization factor.

Definition at line 2357 of file LoopVectorize.cpp.

References Cost, I, llvm::LoopVectorizationCostModel::isProfitableToScalarize(), llvm::LoopVectorizationCostModel::isScalarAfterVectorization(), and VF.

Referenced by needsScalarInduction().

◆ sinkScalarOperands()

void InnerLoopVectorizer::sinkScalarOperands ( Instruction PredInst)
protected

◆ truncateToMinimalBitwidths()

void InnerLoopVectorizer::truncateToMinimalBitwidths ( VPTransformState State)
protected

◆ useOrderedReductions()

bool InnerLoopVectorizer::useOrderedReductions ( RecurrenceDescriptor RdxDesc)

Returns true if the reordering of FP operations is not allowed, but we are able to vectorize with strict in-order reductions for the given RdxDesc.

Definition at line 4598 of file LoopVectorize.cpp.

References Cost, and llvm::LoopVectorizationCostModel::useOrderedReductions().

Referenced by llvm::LoopVectorizationCostModel::collectElementTypesForWidening(), llvm::LoopVectorizationCostModel::collectInLoopReductions(), and llvm::VPReductionRecipe::execute().

◆ vectorizeInterleaveGroup()

void InnerLoopVectorizer::vectorizeInterleaveGroup ( const InterleaveGroup< Instruction > *  Group,
ArrayRef< VPValue * >  VPDefs,
VPTransformState State,
VPValue Addr,
ArrayRef< VPValue * >  StoredValues,
VPValue BlockInMask = nullptr 
)

Try to vectorize interleaved access group Group with the base address given in Addr, optionally masking the vector operations if BlockInMask is non-null.

Use State to translate given VPValues to IR values in the vectorized loop.

Definition at line 2702 of file LoopVectorize.cpp.

References llvm::InterleaveGroup< InstTy >::addMetadata(), Addr, assert(), Builder, llvm::concatenateVectors(), Cost, llvm::IRBuilderBase::CreateAlignedLoad(), llvm::IRBuilderBase::CreateAlignedStore(), llvm::IRBuilderBase::CreateBinOp(), llvm::IRBuilderBase::CreateBitCast(), llvm::createBitMaskForGaps(), createBitOrPointerCast(), llvm::IRBuilderBase::CreateGEP(), llvm::createInterleaveMask(), llvm::IRBuilderBase::CreateMaskedLoad(), llvm::IRBuilderBase::CreateMaskedStore(), llvm::createReplicatedMask(), llvm::IRBuilderBase::CreateShuffleVector(), llvm::createStrideMask(), DL, gep, llvm::VPTransformState::get(), llvm::VectorType::get(), llvm::PoisonValue::get(), llvm::InterleaveGroup< InstTy >::getAlign(), llvm::Module::getDataLayout(), llvm::InterleaveGroup< InstTy >::getFactor(), llvm::InterleaveGroup< InstTy >::getIndex(), llvm::InterleaveGroup< InstTy >::getInsertPos(), llvm::IRBuilderBase::getInt32(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::getLoadStoreType(), llvm::InterleaveGroup< InstTy >::getMember(), llvm::Instruction::getModule(), llvm::Type::getPointerAddressSpace(), llvm::Type::getPointerTo(), llvm::Value::getType(), i, I, Index, llvm::InterleaveGroup< InstTy >::isReverse(), llvm::LinearPolySize< LeafTy >::isScalable(), llvm::LoopVectorizationCostModel::isScalarEpilogueAllowed(), llvm::InterleaveGroup< InstTy >::requiresScalarEpilogue(), reverseVector(), llvm::VPTransformState::set(), setDebugLocFromInst(), llvm::Value::stripPointerCasts(), UF, llvm::RegState::Undef, useMaskedInterleavedAccesses(), and VF.

Referenced by llvm::VPInterleaveRecipe::execute().

◆ vectorizeMemoryInstruction()

void InnerLoopVectorizer::vectorizeMemoryInstruction ( Instruction Instr,
VPTransformState State,
VPValue Def,
VPValue Addr,
VPValue StoredValue,
VPValue BlockInMask,
bool  ConsecutiveStride,
bool  Reverse 
)

◆ widenCallInstruction()

void InnerLoopVectorizer::widenCallInstruction ( CallInst I,
VPValue Def,
VPUser ArgOperands,
VPTransformState State 
)

◆ widenGEP()

void InnerLoopVectorizer::widenGEP ( GetElementPtrInst GEP,
VPValue VPDef,
VPUser Indices,
unsigned  UF,
ElementCount  VF,
bool  IsPtrLoopInvariant,
SmallBitVector IsIndexLoopInvariant,
VPTransformState State 
)

◆ widenInstruction()

void InnerLoopVectorizer::widenInstruction ( Instruction I,
VPValue Def,
VPUser Operands,
VPTransformState State 
)

◆ widenIntOrFpInduction()

void InnerLoopVectorizer::widenIntOrFpInduction ( PHINode IV,
Value Start,
TruncInst Trunc,
VPValue Def,
VPValue CastDef,
VPTransformState State 
)

Widen an integer or floating-point induction variable IV.

If Trunc is provided, the integer induction variable will first be truncated to the corresponding type.

Definition at line 2400 of file LoopVectorize.cpp.

Referenced by llvm::VPWidenIntOrFpInductionRecipe::execute().

◆ widenPHIInstruction()

void InnerLoopVectorizer::widenPHIInstruction ( Instruction PN,
VPWidenPHIRecipe PhiR,
VPTransformState State 
)

Vectorize a single first-order recurrence or pointer induction PHINode in a block.

This method handles the induction variable canonicalization. It supports both VF = 1 for unrolled loops and arbitrary length vectors.

Definition at line 4673 of file LoopVectorize.cpp.

References llvm::PHINode::addIncoming(), assert(), Builder, Cost, llvm::MapVector< KeyT, ValueT, MapType, VectorType >::count(), llvm::GetElementPtrInst::Create(), llvm::PHINode::Create(), llvm::IRBuilderBase::CreateAdd(), llvm::IRBuilderBase::CreateGEP(), llvm::IRBuilderBase::CreateMul(), llvm::IRBuilderBase::CreatePHI(), llvm::IRBuilderBase::CreateSExtOrTrunc(), createStepForVF(), llvm::IRBuilderBase::CreateStepVector(), llvm::IRBuilderBase::CreateVectorSplat(), DL, emitTransformedIndex(), EnableVPlanNativePath, GEP, llvm::ConstantInt::get(), llvm::VectorType::get(), llvm::Module::getDataLayout(), llvm::InductionDescriptor::getElementType(), llvm::LoopBase< BlockT, LoopT >::getHeader(), llvm::LoopVectorizationLegality::getInductionVars(), llvm::InductionDescriptor::getKind(), llvm::LinearPolySize< LeafTy >::getKnownMinValue(), llvm::LoopInfoBase< BlockT, LoopT >::getLoopFor(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::BasicBlock::getModule(), llvm::User::getNumOperands(), llvm::Instruction::getParent(), llvm::getRuntimeVF(), llvm::PredicatedScalarEvolution::getSE(), llvm::InductionDescriptor::getStartValue(), llvm::InductionDescriptor::getStep(), llvm::BasicBlock::getTerminator(), llvm::SCEV::getType(), llvm::Value::getType(), llvm::InductionDescriptor::IK_FpInduction, llvm::InductionDescriptor::IK_IntInduction, llvm::InductionDescriptor::IK_NoInduction, llvm::InductionDescriptor::IK_PtrInduction, Induction, llvm::LoopVectorizationLegality::isReductionVariable(), llvm::LinearPolySize< LeafTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::LoopVectorizationCostModel::isScalarAfterVectorization(), llvm::LoopVectorizationCostModel::isUniformAfterVectorization(), Legal, LI, llvm_unreachable, llvm::MapVector< KeyT, ValueT, MapType, VectorType >::lookup(), LoopVectorBody, LoopVectorPreHeader, OrigLoop, OrigPHIsToFix, P, PSE, llvm::VPTransformState::set(), setDebugLocFromInst(), llvm::Value::setName(), llvm::VPTransformState::UF, UF, llvm::VPTransformState::VF, and VF.

Referenced by llvm::VPWidenPHIRecipe::execute().

◆ widenSelectInstruction()

void InnerLoopVectorizer::widenSelectInstruction ( SelectInst I,
VPValue VPDef,
VPUser Operands,
bool  InvariantCond,
VPTransformState State 
)

Widen a single select instruction within the innermost loop.

Definition at line 5026 of file LoopVectorize.cpp.

References addMetadata(), Builder, Cond, llvm::IRBuilderBase::CreateSelect(), llvm::VPTransformState::get(), I, Operands, llvm::VPTransformState::set(), setDebugLocFromInst(), and UF.

Referenced by llvm::VPWidenSelectRecipe::execute().

Friends And Related Function Documentation

◆ LoopVectorizationPlanner

friend class LoopVectorizationPlanner
friend

Definition at line 570 of file LoopVectorize.cpp.

Member Data Documentation

◆ AA

AAResults* llvm::InnerLoopVectorizer::AA
protected

Alias Analysis.

Definition at line 775 of file LoopVectorize.cpp.

◆ AC

AssumptionCache* llvm::InnerLoopVectorizer::AC
protected

◆ AddedSafetyChecks

bool llvm::InnerLoopVectorizer::AddedSafetyChecks = false
protected

Definition at line 853 of file LoopVectorize.cpp.

Referenced by areSafetyChecksAdded(), emitMemRuntimeChecks(), and emitSCEVChecks().

◆ BFI

BlockFrequencyInfo* llvm::InnerLoopVectorizer::BFI
protected

BFI and PSI are used to check for profile guided size optimizations.

Definition at line 864 of file LoopVectorize.cpp.

Referenced by getScalarEpilogueLowering(), InnerLoopVectorizer(), and processLoopInVPlanNativePath().

◆ Builder

IRBuilder llvm::InnerLoopVectorizer::Builder
protected

◆ Cost

LoopVectorizationCostModel* llvm::InnerLoopVectorizer::Cost
protected

◆ DT

DominatorTree* llvm::InnerLoopVectorizer::DT
protected

◆ Induction

PHINode* llvm::InnerLoopVectorizer::Induction = nullptr
protected

◆ IVEndValues

DenseMap<PHINode *, Value *> llvm::InnerLoopVectorizer::IVEndValues
protected

Definition at line 857 of file LoopVectorize.cpp.

Referenced by createInductionResumeValues().

◆ Legal

LoopVectorizationLegality* llvm::InnerLoopVectorizer::Legal
protected

The legality analysis.

Definition at line 847 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationCostModel::blockNeedsPredication(), llvm::LoopVectorizationCostModel::canVectorizeReductions(), llvm::LoopVectorizationCostModel::collectElementTypesForWidening(), llvm::LoopVectorizationCostModel::collectInLoopReductions(), llvm::LoopVectorizationPlanner::collectTriviallyDeadInstructions(), llvm::LoopVectorizationCostModel::collectValuesToIgnore(), llvm::LoopVectorizationCostModel::computeMaxVF(), llvm::VPRecipeBuilder::createBlockInMask(), llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createInductionResumeValues(), createVectorizedLoopSkeleton(), emitMemRuntimeChecks(), fixReduction(), getAddressAccessSCEV(), getOrCreateTripCount(), llvm::LoopVectorizationCostModel::interleavedAccessCanBeWidened(), llvm::LoopVectorizationCostModel::isLegalMaskedLoad(), llvm::LoopVectorizationCostModel::isLegalMaskedStore(), llvm::LoopVectorizationCostModel::isOptimizableIVTruncate(), llvm::LoopVectorizationCostModel::isPredicatedInst(), llvm::LoopVectorizationCostModel::isScalarWithPredication(), isStrideMul(), llvm::LoopVectorizationCostModel::memoryInstructionCanBeWidened(), llvm::LoopVectorizationCostModel::runtimeChecksRequired(), llvm::LoopVectorizationCostModel::selectInterleaveCount(), llvm::LoopVectorizationCostModel::setCostBasedWideningDecision(), llvm::VPRecipeBuilder::tryToCreateWidenRecipe(), and widenPHIInstruction().

◆ LI

LoopInfo* llvm::InnerLoopVectorizer::LI
protected

◆ LoopBypassBlocks

SmallVector<BasicBlock *, 4> llvm::InnerLoopVectorizer::LoopBypassBlocks
protected

◆ LoopExitBlock

BasicBlock* llvm::InnerLoopVectorizer::LoopExitBlock
protected

The unique ExitBlock of the scalar loop if one exists.

Note that there can be multiple exiting edges reaching this block.

Definition at line 820 of file LoopVectorize.cpp.

Referenced by llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitMinimumIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitMinimumIterationCountCheck(), emitSCEVChecks(), fixLCSSAPHIs(), and fixReduction().

◆ LoopMiddleBlock

BasicBlock* llvm::InnerLoopVectorizer::LoopMiddleBlock
protected

Middle Block between the vector and the scalar.

Definition at line 816 of file LoopVectorize.cpp.

Referenced by completeLoopSkeleton(), createInductionResumeValues(), createVectorLoopSkeleton(), fixFirstOrderRecurrence(), fixLCSSAPHIs(), and fixReduction().

◆ LoopScalarBody

BasicBlock* llvm::InnerLoopVectorizer::LoopScalarBody
protected

The scalar loop body.

Definition at line 826 of file LoopVectorize.cpp.

Referenced by createInductionResumeValues(), and createVectorLoopSkeleton().

◆ LoopScalarPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopScalarPreHeader
protected

◆ LoopVectorBody

BasicBlock* llvm::InnerLoopVectorizer::LoopVectorBody
protected

◆ LoopVectorPreHeader

BasicBlock* llvm::InnerLoopVectorizer::LoopVectorPreHeader
protected

◆ LVer

std::unique_ptr<LoopVersioning> llvm::InnerLoopVectorizer::LVer
protected

LoopVersioning.

It's only set up (non-null) if memchecks were used.

This is currently only used to add no-alias metadata based on the memchecks. The actually versioning is performed manually.

Definition at line 794 of file LoopVectorize.cpp.

Referenced by addNewMetadata(), and emitMemRuntimeChecks().

◆ OldInduction

PHINode* llvm::InnerLoopVectorizer::OldInduction = nullptr
protected

◆ OptForSizeBasedOnProfile

bool llvm::InnerLoopVectorizer::OptForSizeBasedOnProfile
protected

Definition at line 869 of file LoopVectorize.cpp.

Referenced by emitMemRuntimeChecks(), emitSCEVChecks(), and InnerLoopVectorizer().

◆ ORE

OptimizationRemarkEmitter* llvm::InnerLoopVectorizer::ORE
protected

◆ OrigLoop

Loop* llvm::InnerLoopVectorizer::OrigLoop
protected

◆ OrigPHIsToFix

SmallVector<PHINode *, 8> llvm::InnerLoopVectorizer::OrigPHIsToFix
protected

Definition at line 861 of file LoopVectorize.cpp.

Referenced by fixNonInductionPHIs(), fixVectorizedLoop(), and widenPHIInstruction().

◆ PredicatedInstructions

SmallVector<Instruction *, 4> llvm::InnerLoopVectorizer::PredicatedInstructions
protected

Store instructions that were predicated.

Definition at line 838 of file LoopVectorize.cpp.

Referenced by scalarizeInstruction().

◆ PSE

PredicatedScalarEvolution& llvm::InnerLoopVectorizer::PSE
protected

◆ PSI

ProfileSummaryInfo* llvm::InnerLoopVectorizer::PSI
protected

◆ RTChecks

GeneratedRTChecks& llvm::InnerLoopVectorizer::RTChecks
protected

Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.

Definition at line 873 of file LoopVectorize.cpp.

Referenced by emitMemRuntimeChecks(), and emitSCEVChecks().

◆ TLI

const TargetLibraryInfo* llvm::InnerLoopVectorizer::TLI
protected

◆ TripCount

Value* llvm::InnerLoopVectorizer::TripCount = nullptr
protected

Trip count of the original loop.

Definition at line 841 of file LoopVectorize.cpp.

Referenced by getOrCreateTripCount().

◆ TTI

const TargetTransformInfo* llvm::InnerLoopVectorizer::TTI
protected

Target Transform Info.

Definition at line 781 of file LoopVectorize.cpp.

Referenced by llvm::LoopVectorizationCostModel::calculateRegisterUsage().

◆ UF

unsigned llvm::InnerLoopVectorizer::UF
protected

◆ VectorTripCount

Value* llvm::InnerLoopVectorizer::VectorTripCount = nullptr
protected

Trip count of the widened loop (TripCount - TripCount % (VF*UF))

Definition at line 844 of file LoopVectorize.cpp.

Referenced by completeLoopSkeleton(), createInductionResumeValues(), and getOrCreateVectorTripCount().

◆ VF

ElementCount llvm::InnerLoopVectorizer::VF
protected

The vectorization SIMD factor to use.

Each vector will have this many vector elements.

Definition at line 798 of file LoopVectorize.cpp.

Referenced by buildScalarSteps(), llvm::LoopVectorizationPlanner::buildVPlans(), llvm::LoopVectorizationCostModel::calculateRegisterUsage(), llvm::LoopVectorizationCostModel::canTruncateToMinimalBitwidth(), llvm::LoopVectorizationCostModel::collectInstsToScalarize(), llvm::LoopVectorizationCostModel::collectUniformsAndScalars(), completeLoopSkeleton(), createBitOrPointerCast(), llvm::EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton(), createStepForVF(), createVectorizedLoopSkeleton(), createVectorLoopSkeleton(), emitMinimumIterationCountCheck(), llvm::EpilogueVectorizerMainLoop::emitMinimumIterationCountCheck(), emitSCEVChecks(), fixFirstOrderRecurrence(), fixLCSSAPHIs(), fixReduction(), fixVectorizedLoop(), llvm::VPTransformState::get(), getBroadcastInstrs(), getOrCreateVectorTripCount(), llvm::getRuntimeVF(), llvm::LoopVectorizationCostModel::getVectorCallCost(), llvm::LoopVectorizationCostModel::getVectorIntrinsicCost(), llvm::LoopVectorizationCostModel::getWideningCost(), llvm::LoopVectorizationCostModel::getWideningDecision(), llvm::VPRecipeBuilder::handleReplication(), llvm::LoopVectorizationCostModel::interleavedAccessCanBeWidened(), llvm::LoopVectorizationCostModel::isOptimizableIVTruncate(), llvm::LoopVectorizationCostModel::isProfitableToScalarize(), llvm::LoopVectorizationCostModel::isScalarAfterVectorization(), llvm::LoopVectorizationCostModel::isUniformAfterVectorization(), MaybeVectorizeType(), packScalarIntoVectorValue(), llvm::LoopVectorizationPlanner::plan(), llvm::LoopVectorizationPlanner::planInVPlanNativePath(), processLoopInVPlanNativePath(), llvm::LoopVectorizationCostModel::requiresScalarEpilogue(), llvm::LoopVectorizationCostModel::selectInterleaveCount(), llvm::LoopVectorizationPlanner::setBestPlan(), llvm::LoopVectorizationCostModel::setCostBasedWideningDecision(), setDebugLocFromInst(), llvm::LoopVectorizationCostModel::setWideningDecision(), shouldScalarizeInstruction(), vectorizeInterleaveGroup(), vectorizeMemoryInstruction(), widenCallInstruction(), widenGEP(), widenInstruction(), and widenPHIInstruction().


The documentation for this class was generated from the following file: