Holds state needed to make cost decisions before computing costs per-VF, including the maximum VFs. More...

#include "Transforms/Vectorize/LoopVectorizationPlanner.h"

Public Member Functions
	VFSelectionContext (const TargetTransformInfo &TTI, const LoopVectorizationLegality Legal, const Loop TheLoop, const Function &F, PredicatedScalarEvolution &PSE, DemandedBits DB, OptimizationRemarkEmitter ORE, const LoopVectorizeHints *Hints, bool OptForSize)
std::optional< unsigned >	getVScaleForTuning () const
bool	shouldConsiderRegPressureForVF (ElementCount VF) const
bool	supportsScalableVectors () const
void	collectElementTypesForWidening (const SmallPtrSetImpl< const Value * > *ValuesToIgnore=nullptr)
	Collect element types in the loop that need widening.
std::pair< unsigned, unsigned >	getSmallestAndWidestTypes () const
FixedScalableVFPair	computeFeasibleMaxVF (unsigned MaxTripCount, ElementCount UserVF, unsigned UserIC, bool FoldTailByMasking, bool RequiresScalarEpilogue)
std::optional< unsigned >	getMaxSafeElements () const
	Return maximum safe number of elements to be processed per vector iteration, which do not prevent store-load forwarding and are safe with regard to the memory dependencies.
bool	useOrderedReductions (const RecurrenceDescriptor &RdxDesc) const
	Returns true if we should use strict in-order reductions for the given RdxDesc.
bool	isLegalMaskedLoadOrStore (Instruction *I, ElementCount VF) const
	Returns true if the target machine supports masked loads or stores for `I's` data type and alignment.
bool	isLegalGatherOrScatter (Value *V, ElementCount VF) const
	Returns true if the target machine can represent `V` as a masked gather or scatter operation.
void	collectInLoopReductions ()
	Split reductions into those that happen in the loop, and those that happen outside.
bool	isInLoopReduction (PHINode *Phi) const
	Returns true if the Phi is part of an inloop reduction.
const SmallPtrSetImpl< PHINode * > &	getInLoopReductions () const
	Returns the set of in-loop reduction PHIs.
Instruction *	getInLoopReductionImmediateChain (Instruction *I) const
	Returns the immediate chain operand of in-loop reduction operation `I`, or nullptr if `I` is not an in-loop reduction operation.
bool	runtimeChecksRequired ()
	Check whether vectorization would require runtime checks.
FixedScalableVFPair	computeVPlanOuterloopVF (ElementCount UserVF)
	Returns a scalable VF to use for outer-loop vectorization if the target supports it and a fixed VF otherwise.
void	computeMinimalBitwidths ()
	Compute smallest bitwidth each instruction can be represented with.
const MapVector< Instruction *, uint64_t > &	getMinimalBitwidths () const

Public Attributes
const TTI::TargetCostKind	CostKind
	The kind of cost that we are calculating.
const bool	OptForSize
	Whether this loop should be optimized for size based on function attribute or profile information.

Detailed Description

Holds state needed to make cost decisions before computing costs per-VF, including the maximum VFs.

Definition at line 530 of file LoopVectorizationPlanner.h.

Constructor & Destructor Documentation

◆ VFSelectionContext()

llvm::VFSelectionContext::VFSelectionContext	(	const TargetTransformInfo &	TTI,
		const LoopVectorizationLegality *	Legal,
		const Loop *	TheLoop,
		const Function &	F,
		PredicatedScalarEvolution &	PSE,
		DemandedBits *	DB,
		OptimizationRemarkEmitter *	ORE,
		const LoopVectorizeHints *	Hints,
		bool	OptForSize )

inline

Definition at line 616 of file LoopVectorizationPlanner.h.

References CostKind, and OptForSize.

Member Function Documentation

◆ collectElementTypesForWidening()

void VFSelectionContext::collectElementTypesForWidening ( const SmallPtrSetImpl< const Value * > * ValuesToIgnore = nullptr )

Collect element types in the loop that need widening.

Definition at line 462 of file LoopVectorizationPlanner.cpp.

References assert(), llvm::SmallPtrSetImpl< PtrType >::contains(), llvm::dyn_cast(), llvm::RecurrenceDescriptor::getRecurrenceKind(), llvm::RecurrenceDescriptor::getRecurrenceType(), I, llvm::isa(), llvm::PreferInLoopReductions, T, and useOrderedReductions().

◆ collectInLoopReductions()

void VFSelectionContext::collectInLoopReductions ( )

Split reductions into those that happen in the loop, and those that happen outside.

In-loop reductions are collected into InLoopReductions. InLoopReductionImmediateChains is filled with each in-loop reduction operation and its immediate chain operand for use during cost modelling.

Definition at line 568 of file LoopVectorizationPlanner.cpp.

References llvm::dbgs(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::RecurrenceDescriptor::getRecurrenceKind(), llvm::RecurrenceDescriptor::getRecurrenceType(), llvm::RecurrenceDescriptor::getReductionOpChain(), llvm::RecurrenceDescriptor::hasUsesOutsideReductionChain(), I, llvm::RecurrenceDescriptor::isAnyOfRecurrenceKind(), llvm::RecurrenceDescriptor::isFindIVRecurrenceKind(), llvm::RecurrenceDescriptor::isFindLastRecurrenceKind(), LLVM_DEBUG, llvm::PreferInLoopReductions, and useOrderedReductions().

◆ computeFeasibleMaxVF()

FixedScalableVFPair VFSelectionContext::computeFeasibleMaxVF	(	unsigned	MaxTripCount,
		ElementCount	UserVF,
		unsigned	UserIC,
		bool	FoldTailByMasking,
		bool	RequiresScalarEpilogue )

Returns: An upper bound for the vectorization factors for both fixed and scalable vectorization, where the minimum-known number of elements is a power-of-2 larger than zero. If scalable vectorization is disabled or unsupported, then the scalable part will be equal to ElementCount::getScalable(0). Also sets MaxSafeElements.

Definition at line 320 of file LoopVectorizationPlanner.cpp.

References assert(), llvm::bit_floor(), llvm::dbgs(), DEBUG_TYPE, llvm::ElementCount::getFixed(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::ElementCount::getScalable(), getSmallestAndWidestTypes(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownGT(), llvm::details::FixedOrScalableQuantity< ElementCount, unsigned >::isKnownLE(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), LLVM_DEBUG, and supportsScalableVectors().

◆ computeMinimalBitwidths()

void VFSelectionContext::computeMinimalBitwidths ( )

Compute smallest bitwidth each instruction can be represented with.

The vector equivalents of these instructions should be truncated to this type.

Definition at line 564 of file LoopVectorizationPlanner.cpp.

References llvm::computeMinimumValueSizes().

◆ computeVPlanOuterloopVF()

FixedScalableVFPair VFSelectionContext::computeVPlanOuterloopVF ( ElementCount UserVF )

Returns a scalable VF to use for outer-loop vectorization if the target supports it and a fixed VF otherwise.

Definition at line 626 of file LoopVectorizationPlanner.cpp.

References _, assert(), llvm::dbgs(), llvm::ElementCount::get(), llvm::ElementCount::getFixed(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::FixedScalableVFPair::getNone(), getSmallestAndWidestTypes(), llvm::isPowerOf2_32(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isZero(), LLVM_DEBUG, N, RegSize, llvm::reportVectorizationFailure(), llvm::TargetTransformInfo::RGK_FixedWidthVector, llvm::TargetTransformInfo::RGK_ScalableVector, supportsScalableVectors(), and VPlanBuildOuterloopStressTest.

◆ getInLoopReductionImmediateChain()

Instruction * llvm::VFSelectionContext::getInLoopReductionImmediateChain ( Instruction * I ) const

inline

Returns the immediate chain operand of in-loop reduction operation I, or nullptr if I is not an in-loop reduction operation.

Definition at line 699 of file LoopVectorizationPlanner.h.

References I.

◆ getInLoopReductions()

const SmallPtrSetImpl< PHINode * > & llvm::VFSelectionContext::getInLoopReductions ( ) const

inline

Returns the set of in-loop reduction PHIs.

Definition at line 693 of file LoopVectorizationPlanner.h.

◆ getMaxSafeElements()

std::optional< unsigned > llvm::VFSelectionContext::getMaxSafeElements ( ) const

inline

Return maximum safe number of elements to be processed per vector iteration, which do not prevent store-load forwarding and are safe with regard to the memory dependencies.

Required for EVL-based VPlans to correctly calculate AVL (application vector length) as min(remaining AVL, MaxSafeElements). Set by computeFeasibleMaxVF. TODO: need to consider adjusting cost model to use this value as a vectorization factor for EVL-based vectorization.

Definition at line 664 of file LoopVectorizationPlanner.h.

◆ getMinimalBitwidths()

const MapVector< Instruction *, uint64_t > & llvm::VFSelectionContext::getMinimalBitwidths ( ) const

inline

Returns: The smallest bitwidth each instruction can be represented with.

Definition at line 717 of file LoopVectorizationPlanner.h.

◆ getSmallestAndWidestTypes()

std::pair< unsigned, unsigned > VFSelectionContext::getSmallestAndWidestTypes ( ) const

Returns: The size (in bits) of the smallest and widest types in the code that need to be vectorized. We ignore values that remain scalar such as 64 bit loop indices.

Definition at line 433 of file LoopVectorizationPlanner.cpp.

References _, DL, and T.

Referenced by computeFeasibleMaxVF(), and computeVPlanOuterloopVF().