LLVM 18.0.0git
|
VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost. More...
#include "Transforms/Vectorize/VPlan.h"
Public Member Functions | |
VPlan (VPBasicBlock *Preheader, VPValue *TC, VPBasicBlock *Entry) | |
Construct a VPlan with original preheader Preheader , trip count TC and Entry to the plan. | |
VPlan (VPBasicBlock *Preheader, VPBasicBlock *Entry) | |
Construct a VPlan with original preheader Preheader and Entry to the plan. | |
~VPlan () | |
void | prepareToExecute (Value *TripCount, Value *VectorTripCount, Value *CanonicalIVStartValue, VPTransformState &State) |
Prepare the plan for execution, setting up the required live-in values. | |
void | execute (VPTransformState *State) |
Generate the IR code for this VPlan. | |
VPBasicBlock * | getEntry () |
const VPBasicBlock * | getEntry () const |
VPValue * | getTripCount () const |
The trip count of the original loop. | |
VPValue * | getOrCreateBackedgeTakenCount () |
The backedge taken count of the original loop. | |
VPValue & | getVectorTripCount () |
The vector trip count. | |
void | disableValue2VPValue () |
Mark the plan to indicate that using Value2VPValue is not safe any longer, because it may be stale. | |
void | addVF (ElementCount VF) |
void | setVF (ElementCount VF) |
bool | hasVF (ElementCount VF) |
bool | hasScalarVFOnly () const |
bool | hasUF (unsigned UF) const |
void | setUF (unsigned UF) |
std::string | getName () const |
Return a string with the name of the plan and the applicable VFs and UFs. | |
void | setName (const Twine &newName) |
void | addVPValue (Value *V, VPValue *VPV) |
VPValue * | getVPValue (Value *V, bool OverrideAllowed=false) |
Returns the VPValue for V . | |
VPValue * | getVPValueOrAddLiveIn (Value *V) |
Gets the VPValue for V or adds a new live-in (if none exists yet) for V . | |
void | print (raw_ostream &O) const |
Print this VPlan to O . | |
void | printDOT (raw_ostream &O) const |
Print this VPlan in DOT format to O . | |
LLVM_DUMP_METHOD void | dump () const |
Dump the plan to stderr (for debugging). | |
iterator_range< mapped_iterator< Use *, std::function< VPValue *(Value *)> > > | mapToVPValues (User::op_range Operands) |
Returns a range mapping the values the range Operands to their corresponding VPValues. | |
VPRegionBlock * | getVectorLoopRegion () |
Returns the VPRegionBlock of the vector loop. | |
const VPRegionBlock * | getVectorLoopRegion () const |
VPCanonicalIVPHIRecipe * | getCanonicalIV () |
Returns the canonical induction recipe of the vector loop. | |
void | addLiveOut (PHINode *PN, VPValue *V) |
void | removeLiveOut (PHINode *PN) |
const MapVector< PHINode *, VPLiveOut * > & | getLiveOuts () const |
VPValue * | getSCEVExpansion (const SCEV *S) const |
void | addSCEVExpansion (const SCEV *S, VPValue *V) |
VPBasicBlock * | getPreheader () |
const VPBasicBlock * | getPreheader () const |
Static Public Member Functions | |
static VPlanPtr | createInitialVPlan (const SCEV *TripCount, ScalarEvolution &PSE) |
Create initial VPlan skeleton, having an "entry" VPBasicBlock (wrapping original scalar pre-header) which contains SCEV expansions that need to happen before the CFG is modified; a VPBasicBlock for the vector pre-header, followed by a region for the vector loop, followed by the middle VPBasicBlock. | |
Friends | |
class | VPlanPrinter |
class | VPSlotTracker |
VPlan models a candidate for vectorization, encoding various decisions take to produce efficient output IR, including which branches, basic-blocks and output IR instructions to generate, and their cost.
VPlan holds a Hierarchical-CFG of VPBasicBlocks and VPRegionBlocks rooted at an Entry VPBasicBlock.
|
inline |
|
inline |
Construct a VPlan with original preheader Preheader
and Entry
to the plan.
At the moment, Preheader
and Entry
need to be disconnected, as the bypass blocks between them are not yet modeled in VPlan.
Definition at line 2553 of file VPlan.h.
References assert(), llvm::VPBlockBase::getNumPredecessors(), llvm::VPBlockBase::getNumSuccessors(), and llvm::VPBlockBase::setPlan().
VPlan::~VPlan | ( | ) |
Definition at line 690 of file VPlan.cpp.
References llvm::Block, llvm::VPBlockBase::deleteCFG(), and llvm::vp_depth_first_shallow().
Definition at line 2713 of file VPlan.h.
References assert(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::contains().
Referenced by llvm::vputils::getOrCreateVPValueForSCEVExpr().
|
inline |
Definition at line 2602 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::insert().
Definition at line 2627 of file VPlan.h.
References assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), and llvm::VPValue::isLiveIn().
Referenced by getVPValueOrAddLiveIn().
|
static |
Create initial VPlan skeleton, having an "entry" VPBasicBlock (wrapping original scalar pre-header) which contains SCEV expansions that need to happen before the CFG is modified; a VPBasicBlock for the vector pre-header, followed by a region for the vector loop, followed by the middle VPBasicBlock.
Definition at line 711 of file VPlan.cpp.
References llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::VPBlockUtils::insertBlockAfter(), and llvm::VPTransformState::Plan.
|
inline |
LLVM_DUMP_METHOD void VPlan::dump | ( | ) | const |
Dump the plan to stderr (for debugging).
Definition at line 920 of file VPlan.cpp.
References llvm::dbgs(), and llvm::print().
void VPlan::execute | ( | VPTransformState * | State | ) |
Generate the IR code for this VPlan.
Generate the code inside the preheader and body of the vectorized loop.
Assumes a single pre-header basic-block was created for this. Introduce additional basic-blocks as needed, and fill them all.
Definition at line 766 of file VPlan.cpp.
References llvm::DominatorTreeBase< NodeT, IsPostDom >::addNewBlock(), llvm::Block, llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::VPTransformState::DT, llvm::EnableVPlanNativePath, llvm::VPTransformState::CFGState::ExitBB, GEP, llvm::VPTransformState::get(), llvm::VPBlockBase::getExitingBasicBlock(), llvm::ilist_node_with_parent< NodeTy, ParentTy, Options >::getPrevNode(), llvm::BasicBlock::getSingleSuccessor(), llvm::BasicBlock::getTerminator(), llvm::Instruction::moveBefore(), llvm::VPTransformState::CFGState::PrevBB, llvm::VPTransformState::CFGState::PrevVPBB, llvm::IRBuilderBase::SetInsertPoint(), llvm::VPTransformState::UF, llvm::VPTransformState::VF, llvm::vp_depth_first_shallow(), llvm::VPTransformState::CFGState::VPBB2IRBB, and llvm::VPTransformState::VPValue2Value.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
inline |
Returns the canonical induction recipe of the vector loop.
Definition at line 2689 of file VPlan.h.
References llvm::VPBasicBlock::begin(), llvm::VPBasicBlock::empty(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::VPBlockBase::getSingleSuccessor(), and getVectorLoopRegion().
Referenced by llvm::VPlanTransforms::addActiveLaneMask(), addVPLaneMaskPhiAndUpdateExitBranch(), llvm::VPRecipeBuilder::createHeaderMask(), createScalarIVSteps(), llvm::VPWidenPointerInductionRecipe::execute(), and llvm::VPlanTransforms::optimizeForVFAndUF().
|
inline |
Definition at line 2579 of file VPlan.h.
Referenced by addReplicateRegions(), llvm::VPlanPrinter::dump(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::DomTreeNodeTraits< VPBlockBase >::getEntryNode(), getVectorLoopRegion(), mergeReplicateRegionsIntoSuccessors(), llvm::VPBlockBase::setPlan(), simplifyRecipes(), sinkScalarOperands(), and llvm::VPlanVerifier::verifyPlanIsValid().
|
inline |
Definition at line 2705 of file VPlan.h.
Referenced by llvm::InnerLoopVectorizer::fixVectorizedLoop(), and llvm::VPlanVerifier::verifyPlanIsValid().
std::string VPlan::getName | ( | ) | const |
Return a string with the name of the plan and the applicable VFs and UFs.
Definition at line 890 of file VPlan.cpp.
References llvm::drop_begin(), Name, llvm::VPTransformState::UF, and llvm::VPTransformState::VF.
Referenced by llvm::VPlanPrinter::dump().
|
inline |
The backedge taken count of the original loop.
Definition at line 2589 of file VPlan.h.
Referenced by llvm::VPlanTransforms::addActiveLaneMask(), and llvm::VPRecipeBuilder::createHeaderMask().
|
inline |
Definition at line 2719 of file VPlan.h.
Referenced by llvm::VPlanPrinter::dump(), llvm::LoopVectorizationPlanner::executePlan(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::LoopVectorizePass::processLoop(), and llvm::VPBlockBase::setPlan().
|
inline |
Definition at line 2709 of file VPlan.h.
References llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::lookup().
Referenced by llvm::InnerLoopVectorizer::fixupIVUsers(), and llvm::vputils::getOrCreateVPValueForSCEVExpr().
|
inline |
The trip count of the original loop.
Definition at line 2583 of file VPlan.h.
References assert().
Referenced by llvm::VPlanTransforms::addActiveLaneMask(), addVPLaneMaskPhiAndUpdateExitBranch(), and llvm::LoopVectorizationPlanner::executePlan().
|
inline |
Returns the VPRegionBlock of the vector loop.
Definition at line 2681 of file VPlan.h.
References getEntry().
Referenced by addCanonicalIVRecipes(), addVPLaneMaskPhiAndUpdateExitBranch(), llvm::VPlanTransforms::adjustFixedOrderRecurrences(), llvm::VPlanHCFGBuilder::buildHierarchicalCFG(), llvm::VPlanTransforms::clearReductionWrapFlags(), llvm::VPRecipeBuilder::createHeaderMask(), createScalarIVSteps(), llvm::LoopVectorizationPlanner::executePlan(), llvm::InnerLoopVectorizer::fixCrossIterationPHIs(), llvm::VPLiveOut::fixPhi(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), llvm::VPTransformState::get(), getCanonicalIV(), llvm::VPlanTransforms::optimizeForVFAndUF(), llvm::LoopVectorizePass::processLoop(), llvm::VPlanVerifier::verifyPlanIsValid(), and llvm::VPInterleavedAccessInfo::VPInterleavedAccessInfo().
|
inline |
Definition at line 2684 of file VPlan.h.
References getEntry().
|
inline |
The vector trip count.
Definition at line 2596 of file VPlan.h.
Referenced by addCanonicalIVRecipes().
Returns the VPValue for V
.
OverrideAllowed
can be used to disable /// checking whether it is safe to query VPValues using IR Values.
Definition at line 2637 of file VPlan.h.
References assert(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count().
Referenced by getVPValueOrAddLiveIn(), and llvm::InnerLoopVectorizer::truncateToMinimalBitwidths().
Gets the VPValue for V
or adds a new live-in (if none exists yet) for V
.
Definition at line 2648 of file VPlan.h.
References addVPValue(), assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::count(), getVPValue(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().
Referenced by addCanonicalIVRecipes(), addUsersInExitBlock(), llvm::VPRecipeBuilder::createEdgeMask(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), mapToVPValues(), llvm::VPlanTransforms::optimizeForVFAndUF(), and llvm::LoopVectorizePass::processLoop().
|
inline |
Definition at line 2612 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::size().
Referenced by sinkScalarOperands().
Definition at line 2614 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::contains(), and llvm::SetVector< T, Vector, Set, N >::empty().
Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::VPlanTransforms::optimizeForVFAndUF(), and setUF().
|
inline |
Definition at line 2610 of file VPlan.h.
References llvm::SetVector< T, Vector, Set, N >::count().
Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::VPlanTransforms::optimizeForVFAndUF(), and setVF().
|
inline |
Returns a range mapping the values the range Operands
to their corresponding VPValues.
Definition at line 2673 of file VPlan.h.
References getVPValueOrAddLiveIn(), llvm::map_range(), and Operands.
Referenced by llvm::VPRecipeBuilder::handleReplication().
void VPlan::prepareToExecute | ( | Value * | TripCount, |
Value * | VectorTripCount, | ||
Value * | CanonicalIVStartValue, | ||
VPTransformState & | State | ||
) |
Prepare the plan for execution, setting up the required live-in values.
Definition at line 725 of file VPlan.cpp.
References llvm::all_of(), assert(), llvm::VPTransformState::Builder, llvm::VPTransformState::CFG, llvm::IRBuilderBase::CreateSub(), llvm::IRBuilderBase::CreateVectorSplat(), llvm::ConstantInt::get(), llvm::BasicBlock::getTerminator(), llvm::Value::getType(), llvm::ElementCount::isScalar(), IV, llvm::VPTransformState::CFGState::PrevBB, llvm::VPTransformState::set(), llvm::VPTransformState::UF, and llvm::VPTransformState::VF.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
LLVM_DUMP_METHOD void VPlan::print | ( | raw_ostream & | O | ) | const |
Print this VPlan to O
.
Definition at line 847 of file VPlan.cpp.
References llvm::Block, getName(), and llvm::vp_depth_first_shallow().
Referenced by llvm::operator<<().
LLVM_DUMP_METHOD void VPlan::printDOT | ( | raw_ostream & | O | ) | const |
|
inline |
Definition at line 2700 of file VPlan.h.
References llvm::MapVector< KeyT, ValueT, MapType, VectorType >::erase().
Referenced by llvm::InnerLoopVectorizer::fixFixedOrderRecurrence(), llvm::InnerLoopVectorizer::fixReduction(), and llvm::InnerLoopVectorizer::fixupIVUsers().
Definition at line 2625 of file VPlan.h.
References Name, and llvm::Twine::str().
Referenced by llvm::VPlanHCFGBuilder::buildHierarchicalCFG().
|
inline |
Definition at line 2616 of file VPlan.h.
References assert(), llvm::SetVector< T, Vector, Set, N >::clear(), hasUF(), and llvm::SetVector< T, Vector, Set, N >::insert().
Referenced by llvm::VPlanTransforms::optimizeForVFAndUF().
|
inline |
Definition at line 2604 of file VPlan.h.
References assert(), llvm::SetVector< T, Vector, Set, N >::clear(), hasVF(), and llvm::SetVector< T, Vector, Set, N >::insert().
Referenced by llvm::VPlanTransforms::optimizeForVFAndUF().
|
friend |
|
friend |