LLVM 20.0.0git
|
VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost. More...
#include "Transforms/Vectorize/VPlan.h"
Public Member Functions | |
VPlan (VPBasicBlock *Preheader, VPValue *TC, VPBasicBlock *Entry) | |
Construct a VPlan with original preheader Preheader , trip count TC and Entry to the plan. | |
VPlan (VPBasicBlock *Preheader, VPBasicBlock *Entry) | |
Construct a VPlan with original preheader Preheader and Entry to the plan. | |
~VPlan () | |
void | prepareToExecute (Value *TripCount, Value *VectorTripCount, Value *CanonicalIVStartValue, VPTransformState &State) |
Prepare the plan for execution, setting up the required live-in values. | |
void | execute (VPTransformState *State) |
Generate the IR code for this VPlan. | |
InstructionCost | cost (ElementCount VF, VPCostContext &Ctx) |
Return the cost of this plan. | |
VPBasicBlock * | getEntry () |
const VPBasicBlock * | getEntry () const |
VPValue * | getTripCount () const |
The trip count of the original loop. | |
void | resetTripCount (VPValue *NewTripCount) |
Resets the trip count for the VPlan. | |
VPValue * | getOrCreateBackedgeTakenCount () |
The backedge taken count of the original loop. | |
VPValue & | getVectorTripCount () |
The vector trip count. | |
VPValue & | getVFxUF () |
Returns VF * UF of the vector loop region. | |
void | addVF (ElementCount VF) |
void | setVF (ElementCount VF) |
bool | hasVF (ElementCount VF) |
bool | hasScalableVF () |
iterator_range< SmallSetVector< ElementCount, 2 >::iterator > | vectorFactors () const |
Returns an iterator range over all VFs of the plan. | |
bool | hasScalarVFOnly () const |
bool | hasUF (unsigned UF) const |
void | setUF (unsigned UF) |
std::string | getName () const |
Return a string with the name of the plan and the applicable VFs and UFs. | |
void | setName (const Twine &newName) |
VPValue * | getOrAddLiveIn (Value *V) |
Gets the live-in VPValue for V or adds a new live-in (if none exists yet) for V . | |
VPValue * | getLiveIn (Value *V) const |
Return the live-in VPValue for V , if there is one or nullptr otherwise. | |
void | printLiveIns (raw_ostream &O) const |
Print the live-ins of this VPlan to O . | |
void | print (raw_ostream &O) const |
Print this VPlan to O . | |
void | printDOT (raw_ostream &O) const |
Print this VPlan in DOT format to O . | |
LLVM_DUMP_METHOD void | dump () const |
Dump the plan to stderr (for debugging). | |
VPRegionBlock * | getVectorLoopRegion () |
Returns the VPRegionBlock of the vector loop. | |
const VPRegionBlock * | getVectorLoopRegion () const |
VPCanonicalIVPHIRecipe * | getCanonicalIV () |
Returns the canonical induction recipe of the vector loop. | |
void | addLiveOut (PHINode *PN, VPValue *V) |
const MapVector< PHINode *, VPLiveOut * > & | getLiveOuts () const |
VPValue * | getSCEVExpansion (const SCEV *S) const |
void | addSCEVExpansion (const SCEV *S, VPValue *V) |
VPBasicBlock * | getPreheader () |
const VPBasicBlock * | getPreheader () const |
VPlan * | duplicate () |
Clone the current VPlan, update all VPValues of the new VPlan and cloned recipes to refer to the clones, and return it. | |
Static Public Member Functions | |
static VPlanPtr | createInitialVPlan (const SCEV *TripCount, ScalarEvolution &PSE, bool RequiresScalarEpilogueCheck, bool TailFolded, Loop *TheLoop) |
Create initial VPlan, having an "entry" VPBasicBlock (wrapping original scalar pre-header ) which contains SCEV expansions that need to happen before the CFG is modified; a VPBasicBlock for the vector pre-header, followed by a region for the vector loop, followed by the middle VPBasicBlock. | |
Friends | |
class | VPlanPrinter |
class | VPSlotTracker |
VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost.
VPlan holds a Hierarchical-CFG of VPBasicBlocks and VPRegionBlocks rooted at an Entry VPBasicBlock.
|
inline |
|
inline |
Construct a VPlan with original preheader Preheader
and Entry
to the plan.
At the moment, Preheader
and Entry
need to be disconnected, as the bypass blocks between them are not yet modeled in VPlan.
Definition at line 3329 of file VPlan.h.
References assert(), llvm::VPBlockBase::getNumPredecessors(), llvm::VPBlockBase::getNumSuccessors(), and llvm::VPBlockBase::setPlan().
VPlan::~VPlan | ( | ) |
Definition at line 837 of file VPlan.cpp.
References llvm::Block, llvm::VPBlockBase::deleteCFG(), and llvm::vp_depth_first_shallow().
Definition at line 1182 of file VPlan.cpp.
References assert().
Referenced by addLiveOutsForFirstOrderRecurrences(), and addUsersInExitBlock().
Definition at line 3489 of file VPlan.h.
References assert(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains().
Referenced by llvm::vputils::getOrCreateVPValueForSCEVExpr().
|
inline |
Definition at line 3392 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::insert().
InstructionCost VPlan::cost | ( | ElementCount | VF, |
VPCostContext & | Ctx | ||
) |
Return the cost of this plan.
Definition at line 1086 of file VPlan.cpp.
References llvm::VPTransformState::VF.
|
static |
Create initial VPlan, having an "entry" VPBasicBlock (wrapping original scalar pre-header ) which contains SCEV expansions that need to happen before the CFG is modified; a VPBasicBlock for the vector pre-header, followed by a region for the vector loop, followed by the middle VPBasicBlock.
If a check is needed to guard executing the scalar epilogue loop, it will be added to the middle block, together with VPBasicBlocks for the scalar preheader and exit blocks.
Definition at line 858 of file VPlan.cpp.
References llvm::VPInstruction::BranchOnCond, llvm::VPTransformState::Builder, llvm::VPBlockUtils::connectBlocks(), llvm::Type::getContext(), llvm::Type::getInt1Ty(), llvm::LoopBase< BlockT, LoopT >::getLoopLatch(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), getOrAddLiveIn(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::BasicBlock::getTerminator(), getTripCount(), llvm::ConstantInt::getTrue(), llvm::SCEV::getType(), llvm::LoopBase< BlockT, LoopT >::getUniqueExitBlock(), getVectorTripCount(), llvm::CmpInst::ICMP_EQ, llvm::VPBlockUtils::insertBlockAfter(), and llvm::VPTransformState::Plan.
LLVM_DUMP_METHOD void VPlan::dump | ( | ) | const |
Dump the plan to stderr (for debugging).
Definition at line 1179 of file VPlan.cpp.
References llvm::dbgs(), and llvm::print().
Referenced by llvm::LoopVectorizationPlanner::executePlan().
VPlan * VPlan::duplicate | ( | ) |
Clone the current VPlan, update all VPValues of the new VPlan and cloned recipes to refer to the clones, and return it.
Definition at line 1225 of file VPlan.cpp.
References _, assert(), llvm::VPBasicBlock::clone(), cloneFrom(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains(), Name, and remapOperands().
Referenced by llvm::LoopVectorizePass::processLoop().
void VPlan::execute | ( | VPTransformState * | State | ) |
Generate the IR code for this VPlan.
Generate the code inside the preheader and body of the vectorized loop.
Assumes a single pre-header basic-block was created for this. Introduce additional basic-blocks as needed, and fill them all.
Definition at line 976 of file VPlan.cpp.
References llvm::GenericDomTreeUpdater< DerivedT, DomTreeT, PostDomTreeT >::applyUpdates(), assert(), llvm::Block, llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::DominatorTreeBase< BasicBlock, false >::Delete, llvm::VPTransformState::CFGState::DTU, llvm::Instruction::eraseFromParent(), llvm::VPTransformState::CFGState::ExitBB, llvm::GenericDomTreeUpdater< DerivedT, DomTreeT, PostDomTreeT >::flush(), GEP, llvm::VPTransformState::get(), llvm::BasicBlock::getContext(), llvm::GenericDomTreeUpdater< DerivedT, DomTreeT, PostDomTreeT >::getDomTree(), llvm::VPBlockBase::getExitingBasicBlock(), llvm::ilist_node_with_parent< NodeTy, ParentTy, Options >::getPrevNode(), llvm::BasicBlock::getSingleSuccessor(), llvm::VPBlockBase::getSuccessors(), llvm::BasicBlock::getTerminator(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::Instruction::moveBefore(), llvm::VPTransformState::CFGState::PrevBB, llvm::VPTransformState::CFGState::PrevVPBB, replaceVPBBWithIRVPBB(), llvm::IRBuilderBase::SetInsertPoint(), llvm::VPTransformState::UF, llvm::DominatorTreeBase< NodeT, IsPostDom >::verify(), llvm::VPTransformState::VF, llvm::vp_depth_first_shallow(), and llvm::VPTransformState::CFGState::VPBB2IRBB.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
inline |
Returns the canonical induction recipe of the vector loop.
Definition at line 3470 of file VPlan.h.
References llvm::VPBasicBlock::begin(), llvm::VPBasicBlock::empty(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPBlockBase::getSingleSuccessor(), and getVectorLoopRegion().
Referenced by llvm::VPlanTransforms::addActiveLaneMask(), addLiveOutsForFirstOrderRecurrences(), addVPLaneMaskPhiAndUpdateExitBranch(), llvm::VPlanTransforms::adjustFixedOrderRecurrences(), collectAllHeaderMasks(), llvm::VPRecipeBuilder::createHeaderMask(), createScalarIVSteps(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPlanTransforms::optimizeForVFAndUF(), removeRedundantCanonicalIVs(), simplifyRecipes(), llvm::VPlanTransforms::truncateToMinimalBitwidths(), llvm::VPlanTransforms::tryAddExplicitVectorLength(), and willGenerateVectors().
|
inline |
Definition at line 3362 of file VPlan.h.
Referenced by addReplicateRegions(), llvm::VPlanPrinter::dump(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::DomTreeNodeTraits< VPBlockBase >::getEntryNode(), getVectorLoopRegion(), mergeBlocksIntoPredecessors(), mergeReplicateRegionsIntoSuccessors(), removeDeadRecipes(), removeRedundantExpandSCEVRecipes(), llvm::VPBlockBase::setPlan(), simplifyRecipes(), sinkScalarOperands(), and llvm::VPlanTransforms::truncateToMinimalBitwidths().
|
inline |
Return the live-in VPValue for V
, if there is one or nullptr otherwise.
Definition at line 3445 of file VPlan.h.
References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::lookup().
Definition at line 3481 of file VPlan.h.
Referenced by llvm::InnerLoopVectorizer::fixVectorizedLoop().
std::string VPlan::getName | ( | ) | const |
Return a string with the name of the plan and the applicable VFs and UFs.
Definition at line 1149 of file VPlan.cpp.
References llvm::drop_begin(), Name, llvm::VPTransformState::UF, and llvm::VPTransformState::VF.
Referenced by llvm::VPlanPrinter::dump().
Gets the live-in VPValue for V
or adds a new live-in (if none exists yet) for V
.
Definition at line 3428 of file VPlan.h.
References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), llvm::VPValue::isLiveIn(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().
Referenced by addCanonicalIVRecipes(), addLiveOutsForFirstOrderRecurrences(), createInitialVPlan(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::VPRecipeBuilder::getVPValueOrAddLiveIn(), legalizeAndOptimizeInductions(), llvm::VPRecipeBuilder::mapToVPValues(), llvm::VPlanTransforms::optimizeForVFAndUF(), and llvm::LoopVectorizePass::processLoop().
|
inline |
The backedge taken count of the original loop.
Definition at line 3380 of file VPlan.h.
Referenced by llvm::VPRecipeBuilder::createHeaderMask(), and llvm::vputils::isHeaderMask().
|
inline |
Definition at line 3495 of file VPlan.h.
Referenced by llvm::VPlanPrinter::dump(), llvm::LoopVectorizationPlanner::executePlan(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::LoopVectorizePass::processLoop(), and llvm::VPBlockBase::setPlan().
|
inline |
Definition at line 3485 of file VPlan.h.
References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::lookup().
Referenced by llvm::InnerLoopVectorizer::fixupIVUsers(), and llvm::vputils::getOrCreateVPValueForSCEVExpr().
|
inline |
The trip count of the original loop.
Definition at line 3366 of file VPlan.h.
References assert().
Referenced by llvm::VPlanTransforms::addActiveLaneMask(), addVPLaneMaskPhiAndUpdateExitBranch(), createInitialVPlan(), llvm::LoopVectorizationPlanner::executePlan(), llvm::vputils::isHeaderMask(), llvm::LoopVectorizePass::processLoop(), and llvm::VPlanTransforms::tryAddExplicitVectorLength().
|
inline |
Returns the VPRegionBlock of the vector loop.
Definition at line 3462 of file VPlan.h.
References getEntry().
Referenced by addCanonicalIVRecipes(), addLiveOutsForFirstOrderRecurrences(), addUsersInExitBlock(), addVPLaneMaskPhiAndUpdateExitBranch(), llvm::VPlanTransforms::adjustFixedOrderRecurrences(), llvm::VPlanTransforms::clearReductionWrapFlags(), collectAllHeaderMasks(), llvm::collectEphemeralRecipesForVPlan(), llvm::VPRecipeBuilder::createHeaderMask(), createScalarIVSteps(), llvm::LoopVectorizationPlanner::executePlan(), llvm::VPLiveOut::fixPhi(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), llvm::VPTransformState::get(), getCanonicalIV(), legalizeAndOptimizeInductions(), llvm::VPlanTransforms::optimizeForVFAndUF(), llvm::LoopVectorizePass::processLoop(), removeRedundantCanonicalIVs(), removeRedundantInductionCasts(), llvm::VPlanTransforms::truncateToMinimalBitwidths(), llvm::VPlanTransforms::tryAddExplicitVectorLength(), llvm::VPInterleavedAccessInfo::VPInterleavedAccessInfo(), and willGenerateVectors().
|
inline |
Definition at line 3465 of file VPlan.h.
References getEntry().
|
inline |
The vector trip count.
Definition at line 3387 of file VPlan.h.
Referenced by addCanonicalIVRecipes(), and createInitialVPlan().
|
inline |
Returns VF * UF of the vector loop region.
Definition at line 3390 of file VPlan.h.
Referenced by addCanonicalIVRecipes().
|
inline |
Definition at line 3401 of file VPlan.h.
References llvm::any_of(), and llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable().
Referenced by legalizeAndOptimizeInductions().
|
inline |
Definition at line 3411 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::size().
Referenced by sinkScalarOperands().
Definition at line 3413 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::contains(), and llvm::SetVector< T, Vector, Set, N >::empty().
Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::VPlanTransforms::optimizeForVFAndUF(), and setUF().
|
inline |
Definition at line 3400 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::count().
Referenced by llvm::LoopVectorizationPlanner::executePlan(), legalizeAndOptimizeInductions(), llvm::VPlanTransforms::optimizeForVFAndUF(), and setVF().
void VPlan::prepareToExecute | ( | Value * | TripCount, |
Value * | VectorTripCount, | ||
Value * | CanonicalIVStartValue, | ||
VPTransformState & | State | ||
) |
Prepare the plan for execution, setting up the required live-in values.
Definition at line 916 of file VPlan.cpp.
References llvm::all_of(), assert(), llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::createStepForVF(), llvm::IRBuilderBase::CreateSub(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), IV, llvm::VPTransformState::CFGState::PrevBB, llvm::VPTransformState::UF, and llvm::VPTransformState::VF.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
LLVM_DUMP_METHOD void VPlan::print | ( | raw_ostream & | O | ) | const |
Print this VPlan to O
.
Definition at line 1123 of file VPlan.cpp.
References llvm::Block, getName(), and llvm::vp_depth_first_shallow().
Referenced by llvm::operator<<(), and llvm::LoopVectorizationPlanner::printPlans().
LLVM_DUMP_METHOD void VPlan::printDOT | ( | raw_ostream & | O | ) | const |
Print this VPlan in DOT format to O
.
Definition at line 1173 of file VPlan.cpp.
References llvm::Metadata::dump(), and Printer.
Referenced by llvm::LoopVectorizationPlanner::printPlans().
void VPlan::printLiveIns | ( | raw_ostream & | O | ) | const |
Print the live-ins of this VPlan to O
.
Definition at line 1093 of file VPlan.cpp.
Referenced by llvm::VPlanPrinter::dump().
|
inline |
Resets the trip count for the VPlan.
The caller must make sure all uses of the original trip count have been replaced.
Definition at line 3373 of file VPlan.h.
References assert(), and llvm::VPValue::getNumUsers().
Referenced by llvm::LoopVectorizePass::processLoop().
Definition at line 3424 of file VPlan.h.
References Name, and llvm::Twine::str().
Referenced by llvm::VPlanHCFGBuilder::buildHierarchicalCFG(), and llvm::LoopVectorizationPlanner::executePlan().
|
inline |
Definition at line 3415 of file VPlan.h.
References assert(), llvm::SetVector< T, Vector, Set, N >::clear(), hasUF(), and llvm::SetVector< T, Vector, Set, N >::insert().
Referenced by llvm::VPlanTransforms::optimizeForVFAndUF(), and llvm::VPlanTransforms::tryAddExplicitVectorLength().
|
inline |
Definition at line 3394 of file VPlan.h.
References assert(), llvm::SetVector< T, Vector, Set, N >::clear(), hasVF(), and llvm::SetVector< T, Vector, Set, N >::insert().
Referenced by llvm::VPlanTransforms::optimizeForVFAndUF().
|
inline |
Returns an iterator range over all VFs of the plan.
Definition at line 3407 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::begin(), and llvm::SetVector< T, Vector, Set, N >::end().
Referenced by llvm::LoopVectorizationPlanner::getBestVF().
|
friend |
|
friend |