llvm::LoopVectorizationPlanner Class Reference

Planner drives the vectorization process after having passed Legality checks. More...

#include "Transforms/Vectorize/LoopVectorizationPlanner.h"

Public Types
enum class	EpilogueVectorizationKind { None , MainLoop , Epilogue }
	Generate the IR code for the vectorized loop captured in VPlan BestPlan according to the best selected VF and UF. More...

Public Member Functions
	LoopVectorizationPlanner (Loop *L, LoopInfo *LI, DominatorTree *DT, const TargetLibraryInfo *TLI, const TargetTransformInfo &TTI, LoopVectorizationLegality *Legal, LoopVectorizationCostModel &CM, VFSelectionContext &Config, InterleavedAccessInfo &IAI, PredicatedScalarEvolution &PSE, const LoopVectorizeHints &Hints, OptimizationRemarkEmitter *ORE)
void	plan (ElementCount UserVF, unsigned UserIC)
	Build VPlans for the specified UserVF and UserIC if they are non-zero or all applicable candidate VFs otherwise.
VPlan &	getPlanFor (ElementCount VF) const
	Return the VPlan for VF.
std::pair< VectorizationFactor, VPlan * >	computeBestVF ()
	Compute and return the most profitable vectorization factor and the corresponding best VPlan.
unsigned	selectInterleaveCount (VPlan &Plan, ElementCount VF, InstructionCost LoopCost)
DenseMap< const SCEV *, Value * >	executePlan (ElementCount VF, unsigned UF, VPlan &BestPlan, InnerLoopVectorizer &LB, DominatorTree *DT, EpilogueVectorizationKind EpilogueVecKind=EpilogueVectorizationKind::None)
void	printPlans (raw_ostream &O)
bool	hasPlanWithVF (ElementCount VF) const
	Look through the existing plans and return true if we have one with vectorization factor VF.
std::unique_ptr< VPlan >	selectBestEpiloguePlan (VPlan &MainPlan, ElementCount MainLoopVF, unsigned IC)
void	emitInvalidCostRemarks (OptimizationRemarkEmitter *ORE)
	Emit remarks for recipes with invalid costs in the available VPlans.
void	addMinimumIterationCheck (VPlan &Plan, ElementCount VF, unsigned UF, ElementCount MinProfitableTripCount) const
	Create a check to Plan to see if the vector loop should be executed based on its trip count.
void	attachRuntimeChecks (VPlan &Plan, GeneratedRTChecks &RTChecks, bool HasBranchWeights) const
	Attach the runtime checks of RTChecks to Plan.
void	updateLoopMetadataAndProfileInfo (Loop *VectorLoop, VPBasicBlock *HeaderVPBB, const VPlan &Plan, bool VectorizingEpilogue, MDNode *OrigLoopID, std::optional< unsigned > OrigAverageTripCount, unsigned OrigLoopInvocationWeight, unsigned EstimatedVFxUF, bool DisableRuntimeUnroll)
	Update loop metadata and profile info for both the scalar remainder loop and VectorLoop, if it exists.

Static Public Member Functions
static bool	getDecisionAndClampRange (const std::function< bool(ElementCount)> &Predicate, VFRange &Range)
	Test a Predicate on a Range of VF's.

Detailed Description

Planner drives the vectorization process after having passed Legality checks.

Definition at line 724 of file LoopVectorizationPlanner.h.

Member Enumeration Documentation

EpilogueVectorizationKind

enum class llvm::LoopVectorizationPlanner::EpilogueVectorizationKind

strong

Generate the IR code for the vectorized loop captured in VPlan BestPlan according to the best selected VF and UF.

TODO: EpilogueVecKind should be removed once the re-use issue has been fixed.

Returns a mapping of SCEVs to their expanded IR values. Note that this is a temporary workaround needed due to the current epilogue handling.

Enumerator
None	Not part of epilogue vectorization.
MainLoop	Vectorizing the main loop of epilogue vectorization.
Epilogue	Vectorizing the epilogue loop.

Definition at line 822 of file LoopVectorizationPlanner.h.

Constructor & Destructor Documentation

LoopVectorizationPlanner()

llvm::LoopVectorizationPlanner::LoopVectorizationPlanner ( Loop * L, LoopInfo * LI, DominatorTree * DT, const TargetLibraryInfo * TLI, const TargetTransformInfo & TTI, LoopVectorizationLegality * Legal, LoopVectorizationCostModel & CM, VFSelectionContext & Config, InterleavedAccessInfo & IAI, PredicatedScalarEvolution & PSE, const LoopVectorizeHints & Hints, OptimizationRemarkEmitter * ORE )

inline

Definition at line 783 of file LoopVectorizationPlanner.h.

Member Function Documentation

addMinimumIterationCheck()

void LoopVectorizationPlanner::addMinimumIterationCheck	(	VPlan &	Plan,
		ElementCount	VF,
		unsigned	UF,
		ElementCount	MinProfitableTripCount ) const

Create a check to Plan to see if the vector loop should be executed based on its trip count.

Definition at line 7319 of file LoopVectorize.cpp.

References llvm::VPlanTransforms::addMinimumIterationCheck(), llvm::hasBranchWeightMD(), llvm::ElementCount::isVector(), MinItersBypassWeights, and RUN_VPLAN_PASS.

Referenced by llvm::LoopVectorizePass::processLoop().

attachRuntimeChecks()

void LoopVectorizationPlanner::attachRuntimeChecks	(	VPlan &	Plan,
		GeneratedRTChecks &	RTChecks,
		bool	HasBranchWeights ) const

Attach the runtime checks of RTChecks to Plan.

Definition at line 7282 of file LoopVectorize.cpp.

References assert(), llvm::VPlanTransforms::attachCheckBlock(), DEBUG_TYPE, llvm::EnableVPlanNativePath, llvm::LoopVectorizeHints::FK_Enabled, llvm::VPlan::isOuterLoop(), and RUN_VPLAN_PASS.

Referenced by llvm::LoopVectorizePass::processLoop().

computeBestVF()

std::pair< VectorizationFactor, VPlan * > LoopVectorizationPlanner::computeBestVF

(

)

Compute and return the most profitable vectorization factor and the corresponding best VPlan.

Also collect all profitable VFs in ProfitableVFs.

Definition at line 5979 of file LoopVectorize.cpp.

References llvm::any_of(), assert(), llvm::calculateRegisterUsageForPlan(), ConsiderRegPressure, llvm::VectorizationFactor::Cost, llvm::dbgs(), llvm::VectorizationFactor::Disabled(), EpilogueVectorizationForceVF, llvm::LoopVectorizeHints::FK_Enabled, llvm::ElementCount::getFixed(), llvm::InstructionCost::getMax(), llvm::VPlan::getSingleVF(), hasPlanWithVF(), hasReplicatorRegion(), llvm::VPlan::hasScalarVFOnly(), llvm::VPlan::hasVF(), I, llvm::VPlan::isOuterLoop(), llvm::ElementCount::isScalar(), LLVM_DEBUG, P, llvm::VectorizationFactor::ScalarCost, llvm::ArrayRef< T >::size(), llvm::TargetTransformInfo::TCK_CodeSize, llvm::TargetTransformInfo::TCK_Latency, llvm::TargetTransformInfo::TCK_RecipThroughput, llvm::TargetTransformInfo::TCK_SizeAndLatency, VectorizationFactor, llvm::VectorizationFactor::Width, and willGenerateVectors().

Referenced by llvm::LoopVectorizePass::processLoop().

emitInvalidCostRemarks()

void LoopVectorizationPlanner::emitInvalidCostRemarks

(

OptimizationRemarkEmitter *

ORE

)

Emit remarks for recipes with invalid costs in the available VPlans.

Definition at line 3175 of file LoopVectorize.cpp.

References A(), llvm::ArrayRef(), assert(), B(), llvm::VPBlockUtils::blocksOnly(), llvm::TypeSwitch< T, ResultT >::Case(), llvm::cast(), llvm::dyn_cast(), llvm::SmallVectorImpl< T >::emplace_back(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::Value::getName(), llvm::RecurrenceDescriptor::getOpcode(), llvm::Instruction::getOpcodeName(), I, Int, llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownLT(), llvm::reportVectorizationInfo(), llvm::sort(), llvm::CallingConv::Tail, llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::try_emplace(), and llvm::vp_depth_first_deep().

Referenced by llvm::LoopVectorizePass::processLoop().

executePlan()

DenseMap< const SCEV *, Value * > LoopVectorizationPlanner::executePlan	(	ElementCount	VF,
		unsigned	UF,
		VPlan &	BestPlan,
		InnerLoopVectorizer &	LB,
		DominatorTree *	DT,
		EpilogueVectorizationKind	EpilogueVecKind = EpilogueVectorizationKind::None )

Definition at line 6090 of file LoopVectorize.cpp.

References llvm::InnerLoopVectorizer::AC, llvm::VPlanTransforms::addBranchWeightToMiddleTerminator(), assert(), llvm::InnerLoopVectorizer::Builder, llvm::cast(), llvm::VPlanTransforms::convertEVLExitCond(), llvm::VPlanTransforms::convertToConcreteRecipes(), llvm::VPlanTransforms::convertToVariableLengthStep(), llvm::InnerLoopVectorizer::createVectorizedLoopSkeleton(), llvm::VPlanTransforms::cse(), DEBUG_TYPE, llvm::VPlanTransforms::dissolveLoopRegions(), Epilogue, estimateElementCount(), llvm::VPlan::execute(), llvm::VPlanTransforms::expandBranchOnTwoConds(), llvm::VPlanTransforms::expandSCEVs(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), llvm::ScalarEvolution::forgetBlockAndLoopDispositions(), llvm::ScalarEvolution::forgetLcssaPhiWithNewPredecessor(), llvm::ScalarEvolution::forgetLoop(), llvm::VPlan::getEntry(), llvm::VPlan::getExitBlocks(), llvm::vputils::getFirstLoopHeader(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::getLoopEstimatedTripCount(), llvm::getMaxVScale(), llvm::VPlan::getScalarPreheader(), llvm::VPBlockBase::getSingleSuccessor(), llvm::VPlan::getVectorPreheader(), llvm::VPlan::getVFxUF(), llvm::hasBranchWeightMD(), llvm::VPlan::hasEarlyExit(), llvm::VPlan::hasUF(), llvm::VPlan::hasVF(), llvm::ElementCount::isScalar(), llvm::ElementCount::isVector(), llvm::VPlanTransforms::materializeBackedgeTakenCount(), llvm::VPlanTransforms::materializeBroadcasts(), llvm::VPlanTransforms::materializeConstantVectorTripCount(), llvm::VPlanTransforms::materializeFactors(), llvm::VPlanTransforms::materializePacksAndUnpacks(), llvm::VPlanTransforms::materializeVectorTripCount(), None, llvm::VPlanTransforms::optimizeForVFAndUF(), llvm::InnerLoopVectorizer::printDebugTracesAtEnd(), llvm::InnerLoopVectorizer::printDebugTracesAtStart(), llvm::VPlanTransforms::removeBranchOnConst(), llvm::VPlanTransforms::removeDeadRecipes(), replaceVPBBWithIRVPBB(), llvm::VPlanTransforms::replaceWideCanonicalIVWithWideIV(), llvm::VPlanTransforms::replicateByVF(), llvm::InnerLoopVectorizer::RTChecks, RUN_VPLAN_PASS, llvm::VPlanTransforms::simplifyKnownEVL(), llvm::VPlanTransforms::simplifyRecipes(), llvm::VPlanTransforms::unrollByUF(), updateLoopMetadataAndProfileInfo(), and llvm::verifyVPlanIsValid().

Referenced by llvm::LoopVectorizePass::processLoop().

getDecisionAndClampRange()

bool LoopVectorizationPlanner::getDecisionAndClampRange ( const std::function< bool(ElementCount)> & Predicate, VFRange & Range )

static

Test a Predicate on a Range of VF's.

Return the value of applying Predicate on Range.Start, possibly decreasing Range.End such that the returned value holds for the entire Range.

Definition at line 1667 of file VPlan.cpp.

References assert(), and Range.

Referenced by llvm::VPlanTransforms::adjustFirstOrderRecurrenceMiddleUsers(), llvm::VPlanTransforms::createPartialReductions(), llvm::VPRecipeBuilder::handleReplication(), llvm::VPlanTransforms::makeMemOpWideningDecisions(), llvm::VPlanTransforms::makeScalarizationDecisions(), llvm::VPRecipeBuilder::tryToCreateWidenNonPhiRecipe(), tryToMatchAndCreateExtendedReduction(), tryToMatchAndCreateMulAccumulateReduction(), and llvm::VPRecipeBuilder::tryToWidenMemory().

getPlanFor()

VPlan & LoopVectorizationPlanner::getPlanFor

(

ElementCount

VF

)

const

Return the VPlan for VF.

At the moment, there is always a single VPlan for each VF.

Definition at line 1681 of file VPlan.cpp.

References assert(), llvm::count_if(), and llvm_unreachable.

Referenced by selectBestEpiloguePlan().

hasPlanWithVF()

bool llvm::LoopVectorizationPlanner::hasPlanWithVF ( ElementCount VF ) const

inline

Look through the existing plans and return true if we have one with vectorization factor VF.

Definition at line 839 of file LoopVectorizationPlanner.h.

References llvm::any_of().

Referenced by computeBestVF(), llvm::LoopVectorizePass::processLoop(), and selectBestEpiloguePlan().

plan()

void LoopVectorizationPlanner::plan	(	ElementCount	UserVF,
		unsigned	UserIC )

Build VPlans for the specified UserVF and UserIC if they are non-zero or all applicable candidate VFs otherwise.

If vectorization and interleaving should be avoided up-front, no plans are generated.

Definition at line 5673 of file LoopVectorize.cpp.

References assert(), llvm::dbgs(), EpilogueVectorizationForceVF, llvm::FixedScalableVFPair::FixedVF, llvm::ElementCount::getFixed(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::ElementCount::getScalable(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownLE(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownLT(), llvm::isPowerOf2_32(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::ElementCount::isVector(), LLVM_DEBUG, printPlans(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::reportVectorizationInfo(), llvm::FixedScalableVFPair::ScalableVF, and useMaskedInterleavedAccesses().

Referenced by llvm::LoopVectorizePass::processLoop().

printPlans()

void LoopVectorizationPlanner::printPlans

(

raw_ostream &

O

)

Definition at line 1838 of file VPlan.cpp.

References llvm::VPlan::print(), llvm::VPlan::printDOT(), and PrintVPlansInDotFormat.

Referenced by plan().

selectBestEpiloguePlan()

std::unique_ptr< VPlan > LoopVectorizationPlanner::selectBestEpiloguePlan	(	VPlan &	MainPlan,
		ElementCount	MainLoopVF,
		unsigned	IC )

Returns

A VPlan for the most profitable epilogue vectorization, with its VF narrowed to the chosen factor. The returned plan is a duplicate. Returns nullptr if epilogue vectorization is not supported or not profitable for the loop.

Definition at line 3481 of file LoopVectorize.cpp.

References assert(), llvm::dbgs(), llvm::VectorizationFactor::Disabled(), llvm::VPlan::duplicate(), EnableEpilogueVectorization, EpilogueVectorizationForceVF, estimateElementCount(), llvm::ElementCount::get(), llvm::ScalarEvolution::getConstant(), llvm::ScalarEvolution::getElementCount(), llvm::VPBlockBase::getExitingBasicBlock(), llvm::ElementCount::getFixed(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getFixedValue(), llvm::ScalarEvolution::getMulExpr(), getPlanFor(), llvm::vputils::getSCEVExprForVPValue(), llvm::VPlan::getTripCount(), llvm::VPlan::getUF(), llvm::ScalarEvolution::getUnsignedRangeMax(), llvm::ScalarEvolution::getURemExpr(), llvm::VPlan::getVectorLoopRegion(), llvm::APInt::getZExtValue(), hasPlanWithVF(), llvm::CmpInst::ICMP_UGT, llvm::CmpInst::ICMP_ULT, llvm::isa(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isFixed(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownGE(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownGT(), llvm::ScalarEvolution::isKnownPredicate(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::SCEV::isZero(), LLVM_DEBUG, llvm::PatternMatch::m_Add(), llvm::SCEVPatternMatch::m_SCEV(), llvm::SCEVPatternMatch::m_scev_c_Mul(), llvm::SCEVPatternMatch::m_SCEVVScale(), llvm::PatternMatch::m_Specific(), and llvm::PatternMatch::match().

Referenced by llvm::LoopVectorizePass::processLoop().

selectInterleaveCount()

unsigned LoopVectorizationPlanner::selectInterleaveCount	(	VPlan &	Plan,
		ElementCount	VF,
		InstructionCost	LoopCost )

Returns

The desired interleave count. If interleave count has been specified by metadata it will be returned. Otherwise, the interleave count is computed and returned. VF and LoopCost are the selected vectorization factor and the cost of the selected VF.

Definition at line 3658 of file LoopVectorize.cpp.

References llvm::any_of(), assert(), llvm::bit_floor(), llvm::VPBlockUtils::blocksOnly(), llvm::calculateRegisterUsageForPlan(), llvm::dbgs(), llvm::dyn_cast(), EnableIndVarRegisterHeur, EnableLoadStoreRuntimeInterleave, estimateElementCount(), F, ForceTargetMaxScalarInterleaveFactor, ForceTargetMaxVectorInterleaveFactor, ForceTargetNumScalarRegs, ForceTargetNumVectorRegs, llvm::VPRegionBlock::getEntry(), llvm::VPBlockBase::getEntryBasicBlock(), getSmallBestKnownTC(), getSmallConstantTripCount(), llvm::InstructionCost::getValue(), llvm::VPlan::getVectorLoopRegion(), llvm::VPlan::hasEarlyExit(), hasFindLastReductionPhi(), llvm::isa(), llvm::IsaPred, llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isNonZero(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::InstructionCost::isValid(), llvm::ElementCount::isVector(), LLVM_DEBUG, MaxNestedScalarReductionIC, llvm::VPBasicBlock::phis(), SmallLoopCost, and llvm::vp_depth_first_deep().

Referenced by llvm::LoopVectorizePass::processLoop().

updateLoopMetadataAndProfileInfo()

void LoopVectorizationPlanner::updateLoopMetadataAndProfileInfo	(	Loop *	VectorLoop,
		VPBasicBlock *	HeaderVPBB,
		const VPlan &	Plan,
		bool	VectorizingEpilogue,
		MDNode *	OrigLoopID,
		std::optional< unsigned >	OrigAverageTripCount,
		unsigned	OrigLoopInvocationWeight,
		unsigned	EstimatedVFxUF,
		bool	DisableRuntimeUnroll )

Update loop metadata and profile info for both the scalar remainder loop and VectorLoop, if it exists.

Keeps all loop hints from the original loop on the vector loop and replaces vectorizer-specific metadata. The loop ID of the original loop OrigLoopID must be passed, together with the average trip count and invocation weight of the original loop (OrigAverageTripCount and OrigLoopInvocationWeight respectively). They cannot be retrieved after the plan has been executed, as the original loop may have been removed.

Definition at line 1732 of file VPlan.cpp.

References llvm::Loop::addIntLoopAttribute(), addRuntimeUnrollDisableMetaData(), llvm::Function::getEntryCount(), llvm::LoopBase< BlockT, LoopT >::getLoopPreheader(), llvm::BasicBlock::getParent(), llvm::VPlan::getScalarPreheader(), LLVMLoopVectorizeFollowupAll, LLVMLoopVectorizeFollowupEpilogue, LLVMLoopVectorizeFollowupVectorized, llvm::makeFollowupLoopID(), llvm::ProfcheckDisableMetadataFixes, llvm::setLoopEstimatedTripCount(), llvm::Loop::setLoopID(), and llvm::TargetTransformInfo::UnrollingPreferences::UnrollVectorizedLoop.

Referenced by executePlan().

---

The documentation for this class was generated from the following files:

• lib/Transforms/Vectorize/LoopVectorizationPlanner.h
• lib/Transforms/Vectorize/LoopVectorize.cpp
• lib/Transforms/Vectorize/VPlan.cpp