slpvectorizer::BoUpSLP Class Reference

Bottom Up SLP Vectorizer. More...

Classes
struct	EdgeInfo
	This structure holds any data we need about the edges being traversed during buildTreeRec(). More...
class	LookAheadHeuristics
	A helper class used for scoring candidates for two consecutive lanes. More...
class	ShuffleCostEstimator
	Merges shuffle masks and emits final shuffle instruction, if required. More...
class	ShuffleInstructionBuilder
	Merges shuffle masks and emits final shuffle instruction, if required. More...
struct	StridedPtrInfo
	If we decide to generate strided load / store, this struct contains all the necessary info. More...
class	VLOperands
	A helper data structure to hold the operands of a vector of instructions. More...

Public Types
enum class	LoadsState {   Gather , Vectorize , ScatterVectorize , StridedVectorize ,   CompressVectorize , BlendedLoadVectorize }
	Tracks the state we can represent the loads in the given sequence. More...
using	ValueList = SmallVector<Value *, 8>
using	InstrList = SmallVector<Instruction *, 16>
using	ValueSet = SmallPtrSet<Value *, 16>
using	StoreList = SmallVector<StoreInst *, 8>
using	ExtraValueToDebugLocsMap = SmallDenseSet<Value *, 4>
using	OrdersType = SmallVector<unsigned, 4>

Public Member Functions
	BoUpSLP (const VectorizeOptions &Opts, Function *Func, ScalarEvolution *Se, TargetTransformInfo *Tti, TargetLibraryInfo *TLi, AAResults *Aa, LoopInfo *Li, DominatorTree *Dt, AssumptionCache *AC, DemandedBits *DB, const DataLayout *DL, OptimizationRemarkEmitter *ORE)
Value *	vectorizeTree ()
	Vectorize the tree that starts with the elements in VL.
Value *	vectorizeTree (const ExtraValueToDebugLocsMap &ExternallyUsedValues, Instruction *ReductionRoot=nullptr, ArrayRef< ReductionVectorPart > VectorValuesAndScales={})
	Vectorize the tree but with the list of externally used values ExternallyUsedValues.
InstructionCost	getSpillCost ()
TargetTransformInfo::TargetCostKind	getCostKind () const
InstructionCost	calculateTreeCostAndTrimNonProfitable (ArrayRef< Value * > VectorizedVals={}, Instruction *RdxRoot=nullptr)
	Calculates the cost of the subtrees, trims non-profitable ones and returns final cost.
void	detectBooleanizedNodes ()
	Finds i1 and/or nodes whose operand nodes are booleanized wide leaves (one-use truncs of wide values or zero-tests of values from [0, 1]) and registers them in MinBWs to be emitted in the wide leaf type.
InstructionCost	getTreeCost (InstructionCost TreeCost, ArrayRef< Value * > VectorizedVals={}, InstructionCost ReductionCost=TTI::TCC_Free, Instruction *RdxRoot=nullptr)
void	buildTree (ArrayRef< Value * > Roots, const SmallDenseSet< Value * > &UserIgnoreLst)
	Construct a vectorizable tree that starts at Roots, ignoring users for the purpose of scheduling and extraction in the UserIgnoreLst.
void	buildTree (ArrayRef< Value * > Roots)
	Construct a vectorizable tree that starts at Roots.
void	setNarrowedChainInsts (ArrayRef< Instruction * > Insts)
	Sets the narrowed reduction chain instructions, dropped together with the reduction.
bool	hasRuntimeCheckableBlockers () const
	Returns true if the last buildTree() observed a may-alias memory dependency between two distinct, range-checkable base objects, i.e.
void	setHasRuntimeCheckableBlockers (bool V)
	Records whether a may-alias dependency between distinct, range-checkable base objects has been observed, so the caller can decide to retry with runtime alias checks enabled.
bool	hasNonCheckableMemBlocker () const
	Returns true if the last buildTree() kept a may-alias memory dependency that is not runtime-checkable (call or a non-simple mem access).
void	setHasNonCheckableMemBlocker (bool V)
	Records that a non-runtime-checkable may-alias dependency was kept.
bool	isTryingRuntimeAliasChecks () const
	Returns true if the current vectorization attempt may drop runtime-checkable may-alias dependencies and guard the region with runtime alias checks.
void	setTryRuntimeAliasChecks (bool V)
	Enables or disables dropping runtime-checkable may-alias dependencies in favor of runtime alias checks for the current vectorization attempt.
SaveAndRestore< bool >	splitStoredValues ()
	Splits the stored values of the root stores into halves, vectorized as separate subtrees, while the returned guard is alive.
bool	canSplitStoredValues (ArrayRef< Value * > Stores) const
	Checks if the graph is built for the Stores and their stored values, looking through the vectorized cast node, are vectorized as the alternate node with the different opcodes in the halves.
void	resetRuntimeAliasCheckState ()
	Resets the runtime alias check data.
void	captureRuntimeCheckBodySnapshot ()
	Snapshots RTChecks.BB's body (non-PHI, non-alloca, non-terminator) into RTOrigBodyOrder in program order, for the scalar fallback.
bool	canVersionBlockForRuntimeChecks (BasicBlock *BB) const
	Returns true if BB satisfies the block-level preconditions for runtime alias check versioning (straight-line, duplicable, not a scalar fallback, function not optimized for size).
bool	canVersionForRuntimeChecks ()
	Returns true if the runtime alias checks can be safely emitted to guard the vectorized region.
bool	isScalarFallbackBlock (BasicBlock *BB) const
	Returns true if BB is a scalar fallback block created by runtime alias check versioning.
bool	runtimeChecksFailedForBlock (BasicBlock *BB) const
	Returns true if an optimistic runtime-checks versioning attempt already failed for BB, so further retries in the same block can be skipped.
void	markRuntimeChecksFailedForBlock (BasicBlock *BB)
	Records that an optimistic runtime-checks versioning attempt failed for BB.
InstructionCost	getRuntimeChecksCost () const
	Returns the modeled cost of the runtime alias checks collected during the last (optimistic) buildTree().
bool	hasRuntimeAliasChecks () const
	Returns true if the last (optimistic) buildTree() collected any runtime alias checks that must guard the vectorized region.
bool	isCFGChanged () const
	Returns true if vectorization changed the CFG (i.e.
TreeEntry &	getRootNode ()
const TreeEntry &	getRootNode () const
ArrayRef< Value * >	getRootNodeScalars () const
	Returns the scalars of the root node.
unsigned	findRootLaneForValue (Value *V) const
	Returns the lane the given value is vectorized to in the root node.
std::optional< std::pair< Type *, bool > >	getRootNodeTypeWithNoCast () const
	Returns the type/is-signed info for the root node in the graph without casting.
bool	isSignedMinBitwidthRootNode () const
	Checks if the root graph node can be emitted with narrower bitwidth at codegen and returns it signedness, if so.
FixedVectorType *	getReductionType () const
	Returns reduction type after minbitdth analysis.
bool	isReducedBitcastRoot () const
	Returns true if the tree results in one of the reduced bitcasts variants.
bool	isReducedCmpBitcastRoot () const
	Returns true if the tree results in the reduced cmp bitcast root.
bool	isReductionTree () const
	Returns true if the tree is a reduction tree.
void	buildExternalUses (const ExtraValueToDebugLocsMap &ExternallyUsedValues={})
	Builds external uses of the vectorized scalars, i.e.
void	transformNodes ()
	Transforms graph nodes to target specific representations, if profitable.
void	deleteTree ()
	Clear the internal data structures that are created by 'buildTree'.
unsigned	getTreeSize () const
unsigned	getTreeSizeExcludingGathers () const
	Returns the size of the current tree, stopping at gathers For instance, even though we continue adding tree entries after a split node, this function stops counting at the split node since it is a gather.
bool	hasNonConstantGathers () const
	Returns true if the tree gathers any scalars other than constants.
unsigned	getCanonicalGraphSize () const
	Returns the base graph size, before any transformations.
unsigned	getNumSplatSubtreeEntries () const
	Number of tree entries that form the splat gather subtrees.
void	optimizeGatherSequence ()
	Perform LICM and CSE on the newly generated gather sequences.
std::optional< OrdersType >	findReusedOrderedScalars (const TreeEntry &TE, bool TopToBottom, bool IgnoreReorder)
	Checks if the specified gather tree entry TE can be represented as a shuffled vector entry + (possibly) permutation with other gathers.
std::optional< OrdersType >	findPartiallyOrderedLoads (const TreeEntry &TE)
	Sort loads into increasing pointers offsets to allow greater clustering.
std::optional< OrdersType >	getReorderingData (const TreeEntry &TE, bool TopToBottom, bool IgnoreReorder)
	Gets reordering data for the given tree entry.
bool	isProfitableToReorder () const
	Checks if it is profitable to reorder the current tree.
void	reorderTopToBottom ()
	Reorders the current graph to the most profitable order starting from the root node to the leaf nodes.
void	reorderBottomToTop (bool IgnoreReorder=false)
	Reorders the current graph to the most profitable order starting from leaves to the root.
void	markCopyableDepsForRecalc (TreeEntry &TE)
	Marks the schedule data of the copyable-modeled operands of TE for dependency recalculation at the next bundle scheduling.
unsigned	getVectorElementSize (Value *V)
void	computeMinimumValueSizes ()
	Compute the minimum type sizes required to represent the entries in a vectorizable tree.
unsigned	getMaxVecRegSize () const
int	getCostThreshold () const
unsigned	getMinVecRegSize () const
unsigned	getNumberOfParts (Type *VecTy, Type *ScalarTy, unsigned Limit=std::numeric_limits< unsigned >::max()) const
unsigned	getRegUsageForType (Type *VecTy, Type *ScalarTy, unsigned Limit=std::numeric_limits< unsigned >::max()) const
unsigned	getMinVF (unsigned Sz) const
unsigned	getMaximumVF (unsigned ElemWidth, unsigned Opcode) const
unsigned	canMapToVector (Type *T) const
	Check if homogeneous aggregate is isomorphic to some VectorType.
bool	canVectorStoreInsertValue (const TreeEntry *E) const
FixedVectorType *	getInsertBuildVectorSrcTy (const TreeEntry *E) const
bool	isTreeTinyAndNotFullyVectorizable (bool ForReduction=false) const
bool	isTreeNotExtendable () const
	Checks if the graph and all its subgraphs cannot be better vectorized.
bool	isStridedLoad (ArrayRef< Value * > PointerOps, Type *ScalarTy, Align Alignment, const int64_t Diff, const size_t Sz) const
	Checks if strided loads can be generated out of VL loads with pointers PointerOps:
bool	isWidenedStridedLoad (const TreeEntry *E) const
	Returns true for widened strided loads, where each lane of the strided load covers a group of consecutive scalars.
bool	analyzeConstantStrideCandidate (const ArrayRef< Value * > PointerOps, Type *ElemTy, Align Alignment, const SmallVectorImpl< unsigned > &SortedIndices, const int64_t Diff, Value *Ptr0, StridedPtrInfo &SPtrInfo) const
	Return true if an array of scalar loads can be replaced with a strided load (with constant stride).
bool	analyzeRtStrideCandidate (ArrayRef< Value * > PointerOps, Type *ScalarTy, Align CommonAlignment, SmallVectorImpl< unsigned > &SortedIndices, StridedPtrInfo &SPtrInfo, bool IsLoad) const
	Return true if an array of scalar loads can be replaced with a strided load (with run-time stride).
LoadsState	canVectorizeLoads (ArrayRef< Value * > VL, const Value *VL0, SmallVectorImpl< unsigned > &Order, SmallVectorImpl< Value * > &PointerOps, StridedPtrInfo &SPtrInfo, unsigned *BestVF=nullptr, bool TryRecursiveCheck=true) const
	Checks if the given array of loads can be represented as a vectorized, scatter or just simple gather.
bool	hasSameNode (const InstructionsState &S, ArrayRef< Value * > VL) const
	Checks whether some existing tree entry has scalars equal to VL.
template<typename T>
void	registerNonVectorizableLoads (ArrayRef< T * > VL)
	Registers non-vectorizable sequence of loads.
template<typename T>
bool	areKnownNonVectorizableLoads (ArrayRef< T * > VL) const
	Checks if the given loads sequence is known as not vectorizable.
OptimizationRemarkEmitter *	getORE ()
std::pair< std::optional< int >, int >	findBestRootPair (ArrayRef< std::pair< Value *, Value * > > Candidates, int Limit=LookAheadHeuristics::ScoreFail) const
	Evaluate each pair in Candidates and return index into Candidates for a pair which have highest score deemed to have best chance to form root of profitable tree to vectorize.
bool	isDeleted (Instruction *I) const
	Checks if the instruction is marked for deletion.
std::optional< int64_t >	getPointersDiff (Type *ElemTyA, Value *PtrA, Type *ElemTyB, Value *PtrB, const DataLayout &DL, ScalarEvolution &SE, bool StrictCheck=false, bool CheckType=true) const
	Cached llvm::getPointersDiff(), see SLPAnalysisCache.
SLPAnalysisCache &	getAnalysisCache () const
	Returns the cached analyses results, shared by all the trees built for the function.
bool	isEphemeralValue (const Value *V) const
	Checks if the value is used only by the assume-like intrinsics.
void	eraseInstruction (Instruction *I)
	Removes an instruction from its block and eventually deletes it.
template<typename T>
void	removeInstructionsAndOperands (ArrayRef< T * > DeadVals, ArrayRef< ReductionVectorPart > VectorValuesAndScales)
	Remove instructions from the parent function and clear the operands of DeadVals instructions, marking for deletion trivially dead operands.
bool	isAnalyzedReductionRoot (Instruction *I) const
	Checks if the instruction was already analyzed for being possible reduction root.
void	analyzedReductionRoot (Instruction *I)
	Register given instruction as already analyzed for being possible reduction root.
bool	areAnalyzedReductionVals (ArrayRef< Value * > VL) const
	Checks if the provided list of reduced values was checked already for vectorization.
void	analyzedReductionVals (ArrayRef< Value * > VL)
	Adds the list of reduced values to list of already checked values for the vectorization.
bool	isAnalyzedScalar (const Value *V) const
	Checks if the value was already a part of the analyzed vector node.
bool	isAnalyzedBundle (ArrayRef< Value * > VL) const
	Checks if the given bundle was already rejected as non-vectorizable.
void	analyzedBundle (ArrayRef< Value * > VL)
	Registers the bundle as rejected for the vectorization.
void	clearReductionData ()
	Clear the list of the analyzed reduction root instructions.
bool	isAnyGathered (const SmallDenseSet< Value * > &Vals) const
	Checks if the given value is gathered in one of the nodes.
bool	isGathered (const Value *V) const
	Checks if the given value is gathered in one of the nodes.
bool	isNotScheduled (const Value *V) const
	Checks if the specified value was not schedule.
bool	isReassocScalarVectorized (const Value *V) const
	Check if V is a peeled reassociated scalar still owned by a live (non-deleted, non-gathered) tree entry.
bool	isVectorized (const Value *V) const
	Check if the value is vectorized in the tree.
bool	hasResolvedUser (Instruction *I) const
	Returns true if the role of I is already decided by its user: a deleted user was folded into some other vector by an earlier attempt.
bool	canBuildSplitNode (ArrayRef< Value * > VL, const InstructionsState &LocalState, SmallVectorImpl< Value * > &Op1, SmallVectorImpl< Value * > &Op2, OrdersType &ReorderIndices) const
	Checks if it is legal and profitable to build SplitVectorize node for the given VL.
TargetTransformInfo::CastContextHint	getCastContextHint (const TreeEntry &TE) const
	~BoUpSLP ()
template<typename ResTy, typename BVTy>
std::optional< ResTy >	processExtractedFieldsGather (BVTy &ShuffleBuilder, const TreeEntry *E, Type *ScalarTy, ArrayRef< Value * > VL, ArrayRef< std::pair< const TreeEntry *, unsigned > > SubVectors)
template<typename BVTy, typename ResTy, typename... Args>
ResTy	processBuildVector (const TreeEntry *E, Type *ScalarTy, Args &...Params)

Static Public Member Functions
static bool	isIdentityOrder (ArrayRef< unsigned > Order)
	Does this non-empty order represent an identity order?

Public Attributes
const VectorizeOptions &	Opts

Friends
struct	DenseMapInfo< EdgeInfo >
struct	GraphTraits< BoUpSLP * >
struct	DOTGraphTraits< BoUpSLP * >
class	::HorizontalReduction
raw_ostream &	operator<< (raw_ostream &OS, const BoUpSLP::ScheduleEntity &SE)
raw_ostream &	operator<< (raw_ostream &OS, const BoUpSLP::ScheduleData &SD)
raw_ostream &	operator<< (raw_ostream &OS, const BoUpSLP::ScheduleBundle &Bundle)
raw_ostream &	operator<< (raw_ostream &OS, const BoUpSLP::ScheduleCopyableData &SD)

Detailed Description

Bottom Up SLP Vectorizer.

Definition at line 179 of file SLPVectorizer.cpp.

Member Typedef Documentation

ExtraValueToDebugLocsMap

using slpvectorizer::BoUpSLP::ExtraValueToDebugLocsMap = SmallDenseSet<Value *, 4>

Definition at line 215 of file SLPVectorizer.cpp.

InstrList

using slpvectorizer::BoUpSLP::InstrList = SmallVector<Instruction *, 16>

Definition at line 212 of file SLPVectorizer.cpp.

OrdersType

using slpvectorizer::BoUpSLP::OrdersType = SmallVector<unsigned, 4>

Definition at line 216 of file SLPVectorizer.cpp.

StoreList

using slpvectorizer::BoUpSLP::StoreList = SmallVector<StoreInst *, 8>

Definition at line 214 of file SLPVectorizer.cpp.

ValueList

using slpvectorizer::BoUpSLP::ValueList = SmallVector<Value *, 8>

Definition at line 211 of file SLPVectorizer.cpp.

ValueSet

using slpvectorizer::BoUpSLP::ValueSet = SmallPtrSet<Value *, 16>

Definition at line 213 of file SLPVectorizer.cpp.

Member Enumeration Documentation

LoadsState

enum class slpvectorizer::BoUpSLP::LoadsState

strong

Tracks the state we can represent the loads in the given sequence.

Enumerator
Gather
Vectorize
ScatterVectorize
StridedVectorize
CompressVectorize
BlendedLoadVectorize

Definition at line 202 of file SLPVectorizer.cpp.

Constructor & Destructor Documentation

BoUpSLP()

slpvectorizer::BoUpSLP::BoUpSLP ( const VectorizeOptions & Opts, Function * Func, ScalarEvolution * Se, TargetTransformInfo * Tti, TargetLibraryInfo * TLi, AAResults * Aa, LoopInfo * Li, DominatorTree * Dt, AssumptionCache * AC, DemandedBits * DB, const DataLayout * DL, OptimizationRemarkEmitter * ORE )

inline

Definition at line 220 of file SLPVectorizer.cpp.

References llvm::CodeMetrics::collectEphemeralValues(), llvm::slpvectorizer::getSLPCostKind(), Opts, and llvm::TargetTransformInfo::RGK_FixedWidthVector.

Referenced by ::HorizontalReduction, slpvectorizer::BoUpSLP::LookAheadHeuristics::LookAheadHeuristics(), slpvectorizer::BoUpSLP::ShuffleCostEstimator::ShuffleCostEstimator(), slpvectorizer::BoUpSLP::ShuffleInstructionBuilder::ShuffleInstructionBuilder(), slpvectorizer::BoUpSLP::VLOperands::VLOperands(), and slpvectorizer::BoUpSLP::VLOperands::VLOperands().

~BoUpSLP()

BoUpSLP::~BoUpSLP

(

)

Definition at line 5579 of file SLPVectorizer.cpp.

References assert(), llvm::dbgs(), llvm::SmallVectorImpl< T >::emplace_back(), I, llvm::isa(), llvm::make_isa_range(), llvm::RecursivelyDeleteTriviallyDeadInstructions(), llvm::verifyFunction(), and llvm::wouldInstructionBeTriviallyDead().

Member Function Documentation

analyzeConstantStrideCandidate()

bool BoUpSLP::analyzeConstantStrideCandidate	(	const ArrayRef< Value * >	PointerOps,
		Type *	ElemTy,
		Align	Alignment,
		const SmallVectorImpl< unsigned > &	SortedIndices,
		const int64_t	Diff,
		Value *	Ptr0,
		StridedPtrInfo &	SPtrInfo ) const

Return true if an array of scalar loads can be replaced with a strided load (with constant stride).

It is possible that the load gets "widened". Suppose that originally each load loads k bytes and PointerOps can be arranged as follows (s is constant): b + 0 * s + 0 b + 0 * s + 1 b + 0 * s + 2 ... b + 0 * s + (w - 1)

b + 1 * s + 0 b + 1 * s + 1 b + 1 * s + 2 ... b + 1 * s + (w - 1) ...

b + (n - 1) * s + 0 b + (n - 1) * s + 1 b + (n - 1) * s + 2 ... b + (n - 1) * s + (w - 1)

In this case we will generate a strided load of type <n x (k * w)>.

Parameters

PointerOps	list of pointer arguments of loads.
ElemTy	original scalar type of loads.
Alignment	alignment of the first load.
SortedIndices	is the order of PointerOps as returned by sortPtrAccesses
Diff	Pointer difference between the lowest and the highes pointer in PointerOps as returned by getPointersDiff.
Ptr0	first pointer in PointersOps.
PtrN	last pointer in PointersOps.
SPtrInfo	If the function return true, it also sets all the fields of SPtrInfo necessary to generate the strided load later.

Definition at line 5869 of file SLPVectorizer.cpp.

References assert(), llvm::cast(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::find_if(), llvm::Type::getIntNTy(), llvm::getPointersDiff(), llvm::ConstantInt::getSigned(), llvm::Value::getType(), llvm::slpvectorizer::getWidenedType(), I, isStridedLoad(), llvm::Type::isVectorTy(), llvm::Offset, llvm::seq(), llvm::ArrayRef< T >::size(), slpvectorizer::BoUpSLP::StridedPtrInfo::StrideVal, and slpvectorizer::BoUpSLP::StridedPtrInfo::Ty.

Referenced by canVectorizeLoads().

analyzedBundle()

void slpvectorizer::BoUpSLP::analyzedBundle ( ArrayRef< Value * > VL )

inline

Registers the bundle as rejected for the vectorization.

Definition at line 2253 of file SLPVectorizer.cpp.

References llvm::hash_value().

analyzedReductionRoot()

void slpvectorizer::BoUpSLP::analyzedReductionRoot ( Instruction * I )

inline

Register given instruction as already analyzed for being possible reduction root.

Definition at line 2231 of file SLPVectorizer.cpp.

References I.

analyzedReductionVals()

void slpvectorizer::BoUpSLP::analyzedReductionVals ( ArrayRef< Value * > VL )

inline

Adds the list of reduced values to list of already checked values for the vectorization.

Definition at line 2241 of file SLPVectorizer.cpp.

References llvm::hash_value().

analyzeRtStrideCandidate()

bool BoUpSLP::analyzeRtStrideCandidate	(	ArrayRef< Value * >	PointerOps,
		Type *	ScalarTy,
		Align	CommonAlignment,
		SmallVectorImpl< unsigned > &	SortedIndices,
		StridedPtrInfo &	SPtrInfo,
		bool	IsLoad ) const

Return true if an array of scalar loads can be replaced with a strided load (with run-time stride).

Parameters

PointerOps	list of pointer arguments of loads.
ScalarTy	type of loads.
CommonAlignment	common alignement of loads as computed by computeCommonAlignment<LoadInst>.
SortedIndicies	is a list of indicies computed by this function such that the sequence PointerOps[SortedIndices[0]], / PointerOps[SortedIndicies[1]], ..., PointerOps[SortedIndices[n]] is ordered by the coefficient of the stride. For example, if PointerOps is base + stride, base, base + 2 * stride the SortedIndices will be [1, 0, 2]. We follow the convention that if SortedIndices has to be 0, 1, 2, 3, ... we return empty vector for SortedIndicies.
SPtrInfo	If the function return true, it also sets all the fields of SPtrInfo necessary to generate the strided load later.
IsLoad	Is this a strided load (true) or strided store (false)

Definition at line 5969 of file SLPVectorizer.cpp.

References llvm::Add, llvm::SmallVectorTemplateCommon< T, typename >::begin(), llvm::slpvectorizer::calculateRtStride(), llvm::SmallVectorImpl< T >::clear(), llvm::dyn_cast(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::end(), llvm::enumerate(), llvm::Type::getIntNTy(), llvm::slpvectorizer::getNumElements(), llvm::slpvectorizer::getWidenedType(), I, llvm::detail::DenseSetImpl< ValueT, MapTy >::insert(), llvm::Type::isVectorTy(), llvm::none_of(), llvm::Offset, Opts, llvm::SmallVectorImpl< T >::resize(), llvm::seq(), llvm::ArrayRef< T >::size(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::size(), llvm::detail::DenseSetImpl< ValueT, MapTy >::size(), llvm::SmallVectorTemplateCommon< T, typename >::size(), llvm::sort(), slpvectorizer::BoUpSLP::StridedPtrInfo::StrideSCEV, and slpvectorizer::BoUpSLP::StridedPtrInfo::Ty.

Referenced by canVectorizeLoads().

areAnalyzedReductionVals()

bool slpvectorizer::BoUpSLP::areAnalyzedReductionVals ( ArrayRef< Value * > VL ) const

inline

Checks if the provided list of reduced values was checked already for vectorization.

Definition at line 2236 of file SLPVectorizer.cpp.

References llvm::hash_value().

areKnownNonVectorizableLoads()

template<typename T>

bool slpvectorizer::BoUpSLP::areKnownNonVectorizableLoads ( ArrayRef< T * > VL ) const

inline

Checks if the given loads sequence is known as not vectorizable.

Definition at line 880 of file SLPVectorizer.cpp.

References llvm::hash_value().

Referenced by canVectorizeLoads().

buildExternalUses()

void BoUpSLP::buildExternalUses

(

const ExtraValueToDebugLocsMap &

ExternallyUsedValues = {}

)

Builds external uses of the vectorized scalars, i.e.

the list of vectorized scalars to be extracted, their lanes and their scalar users. ExternallyUsedValues contains additional list of external uses to handle vectorization of reductions.

Definition at line 8030 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::any_of(), assert(), llvm::cast(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::clear(), llvm::dbgs(), llvm::slpvectorizer::doesInTreeUserNeedToExtract(), llvm::dyn_cast(), llvm::SmallVectorImpl< T >::emplace_back(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::detail::DenseSetImpl< ValueT, MapTy >::end(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::find(), llvm::detail::DenseSetImpl< ValueT, MapTy >::find(), llvm::isa(), isDeleted(), isGathered(), isReassocScalarVectorized(), LLVM_DEBUG, llvm::make_isa_range(), llvm::none_of(), llvm::SmallVectorImpl< T >::pop_back_val(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::SCEVExprContains(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::try_emplace(), llvm::Value::users(), and llvm::slpvectorizer::UsesLimit.

buildTree() [1/2]

void BoUpSLP::buildTree

(

ArrayRef< Value * >

Roots

)

Construct a vectorizable tree that starts at Roots.

Definition at line 8380 of file SLPVectorizer.cpp.

References llvm::slpvectorizer::allSameType(), assert(), and deleteTree().

buildTree() [2/2]

void BoUpSLP::buildTree	(	ArrayRef< Value * >	Roots,
		const SmallDenseSet< Value * > &	UserIgnoreLst )

Construct a vectorizable tree that starts at Roots, ignoring users for the purpose of scheduling and extraction in the UserIgnoreLst.

Definition at line 8367 of file SLPVectorizer.cpp.

References llvm::slpvectorizer::allSameType(), assert(), and deleteTree().

calculateTreeCostAndTrimNonProfitable()

InstructionCost BoUpSLP::calculateTreeCostAndTrimNonProfitable	(	ArrayRef< Value * >	VectorizedVals = {},
		Instruction *	RdxRoot = nullptr )

Calculates the cost of the subtrees, trims non-profitable ones and returns final cost.

Definition at line 19523 of file SLPVectorizer.cpp.

References llvm::any_of(), llvm::dbgs(), llvm::dyn_cast(), llvm::InstructionCost::getInvalid(), getRootNode(), I, llvm::isa(), llvm::isa_and_present(), LLVM_DEBUG, llvm::make_filter_range(), and llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::try_emplace().

canBuildSplitNode()

bool BoUpSLP::canBuildSplitNode	(	ArrayRef< Value * >	VL,
		const InstructionsState &	LocalState,
		SmallVectorImpl< Value * > &	Op1,
		SmallVectorImpl< Value * > &	Op2,
		OrdersType &	ReorderIndices ) const

Checks if it is legal and profitable to build SplitVectorize node for the given VL.

Parameters

Op1	first homogeneous scalars.
Op2	second homogeneous scalars.
ReorderIndices	indices to reorder the scalars.

Returns

true if the node was successfully built.

Definition at line 10351 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::SmallVectorImpl< T >::assign(), llvm::cast(), llvm::SmallVectorImpl< T >::clear(), llvm::dbgs(), llvm::dyn_cast(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::enumerate(), llvm::from_range, llvm::ArrayRef< T >::front(), llvm::SmallVectorTemplateCommon< T, typename >::front(), llvm::slpvectorizer::getAltInstrMask(), llvm::slpvectorizer::InstructionsState::getAltOp(), llvm::slpvectorizer::InstructionsState::getAltOpcode(), llvm::slpvectorizer::InstructionsState::getMainOp(), llvm::slpvectorizer::InstructionsState::getOpcode(), llvm::TargetTransformInfo::getOperandInfo(), getRootNode(), llvm::slpvectorizer::getShuffleCost(), llvm::slpvectorizer::getValueType(), llvm::slpvectorizer::getWidenedType(), llvm::slpvectorizer::hasFullVectorsOrPowerOf2(), I, llvm::slpvectorizer::inversePermutation(), llvm::isa(), llvm::slpvectorizer::InstructionsState::isAddSubLikeOp(), llvm::slpvectorizer::isAlternateInstruction(), llvm::Instruction::isBinaryOp(), llvm::slpvectorizer::InstructionsState::isBitwiseLogicOp(), llvm::Instruction::isCast(), llvm::slpvectorizer::InstructionsState::isCmpOp(), isIdentityOrder(), isMainInstruction(), llvm::slpvectorizer::InstructionsState::isMulDivLikeOp(), llvm::slpvectorizer::InstructionsState::isShiftOp(), llvm::Instruction::isUnaryOp(), LLVM_DEBUG, llvm::make_isa_range(), Opts, llvm::PoisonMaskElem, llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::seq(), llvm::SmallBitVector::set(), llvm::ArrayRef< T >::size(), llvm::SmallPtrSetImplBase::size(), llvm::SmallVectorTemplateCommon< T, typename >::size(), llvm::TargetTransformInfo::SK_InsertSubvector, llvm::TargetTransformInfo::SK_PermuteTwoSrc, SmallNodeSize, llvm::SmallBitVector::test(), and llvm::Values.

canMapToVector()

unsigned BoUpSLP::canMapToVector

(

Type *

T

)

const

Check if homogeneous aggregate is isomorphic to some VectorType.

Accepts homogeneous multidimensional aggregate of scalars/vectors like {[4 x i16], [4 x i16]}, { <2 x float>, <2 x float> }, {{{i16, i16}, {i16, i16}}, {{i16, i16}, {i16, i16}}} and so on.

Returns

number of elements in vector if isomorphism exists, 0 otherwise.

Definition at line 13208 of file SLPVectorizer.cpp.

References llvm::cast(), llvm::dyn_cast(), llvm::slpvectorizer::getWidenedType(), llvm::isa(), llvm::Type::isEmptyTy(), llvm::slpvectorizer::isValidElementType(), N, and Opts.

canSplitStoredValues()

bool slpvectorizer::BoUpSLP::canSplitStoredValues ( ArrayRef< Value * > Stores ) const

inline

Checks if the graph is built for the Stores and their stored values, looking through the vectorized cast node, are vectorized as the alternate node with the different opcodes in the halves.

Definition at line 334 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::ArrayRef< T >::back(), llvm::cast(), llvm::ArrayRef< T >::drop_front(), llvm::dyn_cast(), llvm::ArrayRef< T >::front(), getRootNode(), getRootNodeScalars(), I, llvm::isGather(), llvm::ArrayRef< T >::size(), and llvm::ArrayRef< T >::take_front().

canVectorizeLoads()

BoUpSLP::LoadsState BoUpSLP::canVectorizeLoads	(	ArrayRef< Value * >	VL,
		const Value *	VL0,
		SmallVectorImpl< unsigned > &	Order,
		SmallVectorImpl< Value * > &	PointerOps,
		StridedPtrInfo &	SPtrInfo,
		unsigned *	BestVF = nullptr,
		bool	TryRecursiveCheck = true ) const

Checks if the given array of loads can be represented as a vectorized, scatter or just simple gather.

Parameters

VL	list of loads.
VL0	main load value.
Order	returned order of load instructions.
PointerOps	returned list of pointer operands.
BestVF	return best vector factor, if recursive check found better vectorization sequences rather than masked gather.
TryRecursiveCheck	used to check if long masked gather can be represented as a serie of loads/insert subvector, if profitable.

Definition at line 6196 of file SLPVectorizer.cpp.

References llvm::accumulate(), llvm::all_of(), analyzeConstantStrideCandidate(), analyzeRtStrideCandidate(), llvm::any_of(), areKnownNonVectorizableLoads(), llvm::slpvectorizer::arePointersCompatible(), llvm::ArrayRef(), assert(), llvm::SmallVectorTemplateCommon< T, typename >::back(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), BlendedLoadVectorize, C(), canVectorizeLoads(), llvm::slpvectorizer::canWidenLoadsOverUndefLanes(), llvm::cast(), llvm::SmallVectorImpl< T >::clear(), llvm::APInt::clearAllBits(), CompressVectorize, llvm::slpvectorizer::computeBlendedLoadBaseAlignment(), llvm::slpvectorizer::computeCommonAlignment< LoadInst >(), llvm::count_if(), llvm::slpvectorizer::doesNotNeedToBeScheduled(), llvm::dyn_cast(), llvm::SmallVectorImpl< T >::emplace_back(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::End, llvm::SmallVectorTemplateCommon< T, typename >::front(), Gather, GEP, llvm::APInt::getAllOnes(), llvm::slpvectorizer::getBlendedLoadCost(), getCostThreshold(), llvm::slpvectorizer::getFloorFullVectorNumberOfElements(), llvm::slpvectorizer::getGEPCosts(), getMinVF(), llvm::APInt::getOneBitSet(), llvm::TargetTransformInfo::getOperandInfo(), getParent(), llvm::getPointerAddressSpace(), llvm::getPointerOperand(), llvm::getPointersDiff(), llvm::slpvectorizer::getScalarizationOverhead(), llvm::slpvectorizer::getShuffleCost(), llvm::Value::getType(), llvm::getUnderlyingObject(), llvm::slpvectorizer::getWidenedType(), InstructionCost, llvm::isa(), llvm::APInt::isAllOnes(), llvm::IsaPred, llvm::slpvectorizer::isCopyableGEPAddressVector(), llvm::slpvectorizer::isMaskedLoadCompress(), llvm::slpvectorizer::isReverseOrder(), llvm::slpvectorizer::isSelectedBaseLoad(), llvm::APInt::isZero(), llvm_unreachable, Opts, P, llvm::SmallVectorImpl< T >::resize(), ScatterVectorize, llvm::seq(), llvm::APInt::setAllBits(), llvm::APInt::setBits(), llvm::ArrayRef< T >::size(), llvm::SmallVectorTemplateCommon< T, typename >::size(), llvm::TargetTransformInfo::SK_Broadcast, llvm::TargetTransformInfo::SK_InsertSubvector, llvm::TargetTransformInfo::SK_PermuteSingleSrc, llvm::ArrayRef< T >::slice(), llvm::sortPtrAccesses(), StridedVectorize, and Vectorize.

Referenced by canVectorizeLoads(), and getReorderingData().

canVectorStoreInsertValue()

bool BoUpSLP::canVectorStoreInsertValue

(

const TreeEntry *

E

)

const

Returns

true if the vectorized insertvalue result can be stored directly as a vector, i.e. every insertvalue with an external user is consumed by a single store only.

Definition at line 13241 of file SLPVectorizer.cpp.

References llvm::dyn_cast(), llvm::is_contained(), isDeleted(), and IV.

canVersionBlockForRuntimeChecks()

bool BoUpSLP::canVersionBlockForRuntimeChecks

(

BasicBlock *

BB

)

const

Returns true if BB satisfies the block-level preconditions for runtime alias check versioning (straight-line, duplicable, not a scalar fallback, function not optimized for size).

These checks do not depend on the collected checks, so they can gate the (expensive) optimistic retry before any tree is rebuilt.

Definition at line 25978 of file SLPVectorizer.cpp.

References llvm::any_of(), assert(), blockBodyHasVectorInstructions(), llvm::dyn_cast(), llvm::BasicBlock::end(), llvm::BasicBlock::getFirstNonPHIOrAlloca(), llvm::BasicBlock::getTerminatingMustTailCall(), I, llvm::IsaPred, llvm::make_range(), and Opts.

Referenced by canVersionForRuntimeChecks().

canVersionForRuntimeChecks()

bool BoUpSLP::canVersionForRuntimeChecks

(

)

Returns true if the runtime alias checks can be safely emitted to guard the vectorized region.

Definition at line 26014 of file SLPVectorizer.cpp.

References llvm::any_of(), llvm::sampleprof::Base, canVersionBlockForRuntimeChecks(), llvm::SmallPtrSetImplBase::clear(), llvm::SmallPtrSetImpl< PtrType >::contains(), llvm::dyn_cast(), llvm::getBooleanLoopAttribute(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getFixedValue(), llvm::getLoadStorePointerOperand(), llvm::getLoadStoreType(), getRuntimeChecksCost(), llvm::Value::getType(), llvm::getUnderlyingObject(), I, llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::isa(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::InstructionCost::isValid(), isVectorized(), llvm::make_filter_range(), llvm::make_isa_range(), MaxMemDepDistance, Off, Opts, P, llvm::MapVector< KeyT, ValueT, MapType, VectorType, N >::size(), llvm::SmallPtrSetImplBase::size(), and llvm::MapVector< KeyT, ValueT, MapType, VectorType, N >::try_emplace().

captureRuntimeCheckBodySnapshot()

void BoUpSLP::captureRuntimeCheckBodySnapshot

(

)

Snapshots RTChecks.BB's body (non-PHI, non-alloca, non-terminator) into RTOrigBodyOrder in program order, for the scalar fallback.

Definition at line 25937 of file SLPVectorizer.cpp.

References I, and llvm::isa().

clearReductionData()

void slpvectorizer::BoUpSLP::clearReductionData ( )

inline

Clear the list of the analyzed reduction root instructions.

Definition at line 2257 of file SLPVectorizer.cpp.

computeMinimumValueSizes()

void BoUpSLP::computeMinimumValueSizes

(

)

Compute the minimum type sizes required to represent the entries in a vectorizable tree.

Definition at line 29306 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::any_of(), assert(), llvm::bit_ceil(), llvm::SmallVectorImpl< T >::clear(), llvm::ComputeNumSignBits(), llvm::detail::DenseSetImpl< ValueT, MapTy >::contains(), detectBooleanizedNodes(), llvm::dyn_cast(), llvm::ArrayRef< T >::front(), llvm::IntegerType::get(), getNumberOfParts(), getRootNode(), llvm::Type::getScalarType(), getTreeSizeExcludingGathers(), llvm::Value::getType(), llvm::slpvectorizer::getWidenedType(), I, llvm::isa(), llvm::isGather(), llvm::make_filter_range(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().

deleteTree()

void slpvectorizer::BoUpSLP::deleteTree ( )

inline

Clear the internal data structures that are created by 'buildTree'.

Definition at line 505 of file SLPVectorizer.cpp.

Referenced by buildTree(), and buildTree().

detectBooleanizedNodes()

void BoUpSLP::detectBooleanizedNodes

(

)

Finds i1 and/or nodes whose operand nodes are booleanized wide leaves (one-use truncs of wide values or zero-tests of values from [0, 1]) and registers them in MinBWs to be emitted in the wide leaf type.

Definition at line 19395 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::any_of(), llvm::computeKnownBits(), llvm::IntegerType::get(), llvm::Type::getIntegerBitWidth(), I, llvm::Instruction::isBinaryOp(), llvm::isGuaranteedNotToBePoison(), llvm::Type::isIntegerTy(), llvm::reverse(), and llvm::seq().

Referenced by computeMinimumValueSizes().

eraseInstruction()

void slpvectorizer::BoUpSLP::eraseInstruction ( Instruction * I )

inline

Removes an instruction from its block and eventually deletes it.

It's like Instruction::eraseFromParent() except that the actual deletion is delayed until BoUpSLP is destructed.

Definition at line 2132 of file SLPVectorizer.cpp.

References I.

Referenced by optimizeGatherSequence(), removeInstructionsAndOperands(), and vectorizeTree().

findBestRootPair()

std::pair< std::optional< int >, int > slpvectorizer::BoUpSLP::findBestRootPair ( ArrayRef< std::pair< Value *, Value * > > Candidates, int Limit = LookAheadHeuristics::ScoreFail ) const

inline

Evaluate each pair in Candidates and return index into Candidates for a pair which have highest score deemed to have best chance to form root of profitable tree to vectorize.

Return std::nullopt if no candidate scored above the LookAheadHeuristics::ScoreFail.

Parameters

Limit

Lower limit of the cost, considered to be good enough score.

Definition at line 2091 of file SLPVectorizer.cpp.

References slpvectorizer::BoUpSLP::LookAheadHeuristics::getScoreAtLevelRec(), I, Opts, slpvectorizer::BoUpSLP::LookAheadHeuristics::ScoreFail, and llvm::seq().

Referenced by transformNodes().

findPartiallyOrderedLoads()

std::optional< BoUpSLP::OrdersType > BoUpSLP::findPartiallyOrderedLoads

(

const TreeEntry &

TE

)

Sort loads into increasing pointers offsets to allow greater clustering.

Definition at line 6571 of file SLPVectorizer.cpp.

References assert(), llvm::slpvectorizer::clusterSortPtrAccesses(), llvm::dyn_cast(), Opts, llvm::SmallVectorTemplateBase< T, bool >::push_back(), and llvm::SmallVectorImpl< T >::reserve().

Referenced by getReorderingData().

findReusedOrderedScalars()

std::optional< BoUpSLP::OrdersType > BoUpSLP::findReusedOrderedScalars	(	const TreeEntry &	TE,
		bool	TopToBottom,
		bool	IgnoreReorder )

Checks if the specified gather tree entry TE can be represented as a shuffled vector entry + (possibly) permutation with other gathers.

It implements the checks only for possibly ordered scalars (Loads, ExtractElement, ExtractValue), which can be part of the graph.

Parameters

TopToBottom	If true, used for the whole tree rotation, false - for sub-tree rotations.
IgnoreReorder	true, if the order of the root node might be ignored.

Definition at line 5616 of file SLPVectorizer.cpp.

References llvm::SmallBitVector::all(), llvm::all_of(), llvm::SmallBitVector::any(), llvm::any_of(), assert(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), llvm::SmallVectorImpl< T >::clear(), llvm::count(), llvm::dyn_cast(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::end(), llvm::enumerate(), llvm::fill(), llvm::find(), llvm::SmallVectorTemplateCommon< T, typename >::front(), getNumberOfParts(), llvm::slpvectorizer::getNumElems(), llvm::slpvectorizer::getPartNumElems(), llvm::slpvectorizer::getWidenedType(), I, llvm::isa(), llvm::slpvectorizer::isConstant(), llvm::slpvectorizer::isValidElementType(), llvm::not_equal_to(), Opts, P, llvm::PoisonMaskElem, llvm::seq(), llvm::SmallBitVector::set(), llvm::SmallVectorTemplateCommon< T, typename >::size(), llvm::TargetTransformInfo::SK_PermuteSingleSrc, and llvm::SmallBitVector::test().

Referenced by getReorderingData().

findRootLaneForValue()

unsigned slpvectorizer::BoUpSLP::findRootLaneForValue ( Value * V ) const

inline

Returns the lane the given value is vectorized to in the root node.

Definition at line 429 of file SLPVectorizer.cpp.

References getRootNode().

getAnalysisCache()

SLPAnalysisCache & slpvectorizer::BoUpSLP::getAnalysisCache ( ) const

inline

Returns the cached analyses results, shared by all the trees built for the function.

Definition at line 2124 of file SLPVectorizer.cpp.

getCanonicalGraphSize()

unsigned slpvectorizer::BoUpSLP::getCanonicalGraphSize ( ) const

inline

Returns the base graph size, before any transformations.

Definition at line 589 of file SLPVectorizer.cpp.

Referenced by isTreeNotExtendable().

getCastContextHint()

TTI::CastContextHint BoUpSLP::getCastContextHint

(

const TreeEntry &

TE

)

const

Returns

Cast context for the given graph node.

Definition at line 16350 of file SLPVectorizer.cpp.

References llvm::ArrayRef< T >::empty(), llvm::ArrayRef< T >::front(), llvm::TargetTransformInfo::GatherScatter, getCastContextHint(), I, llvm::slpvectorizer::inversePermutation(), llvm::isa(), llvm::ShuffleVectorInst::isReverseMask(), llvm::make_isa_range(), llvm::TargetTransformInfo::Masked, llvm::slpvectorizer::matchGatheredExtractedFields(), llvm::TargetTransformInfo::None, llvm::none_of(), llvm::TargetTransformInfo::Normal, and llvm::TargetTransformInfo::Reversed.

Referenced by getCastContextHint(), and getTreeCost().

getCostKind()

TargetTransformInfo::TargetCostKind slpvectorizer::BoUpSLP::getCostKind ( ) const

inline

Definition at line 259 of file SLPVectorizer.cpp.

Referenced by slpvectorizer::BoUpSLP::ShuffleCostEstimator::ShuffleCostEstimator().

getCostThreshold()

int slpvectorizer::BoUpSLP::getCostThreshold ( ) const

inline

Returns

the -slp-threshold value, 0 if unset.

Definition at line 679 of file SLPVectorizer.cpp.

References Opts.

Referenced by canVectorizeLoads(), getTreeCost(), and isTreeTinyAndNotFullyVectorizable().

getInsertBuildVectorSrcTy()

FixedVectorType * BoUpSLP::getInsertBuildVectorSrcTy

(

const TreeEntry *

E

)

const

Returns

the source vector type for an InsertElement/InsertValue buildvector node E: the inserted vector type for insertelement, or a vector of the inserted scalar type wide enough to cover the highest inserted index for insertvalue.

Definition at line 13273 of file SLPVectorizer.cpp.

References assert(), llvm::cast(), llvm::Value::getType(), llvm::slpvectorizer::getValueType(), llvm::slpvectorizer::getWidenedType(), I, llvm::make_isa_range(), and Opts.

getMaximumVF()

unsigned slpvectorizer::BoUpSLP::getMaximumVF ( unsigned ElemWidth, unsigned Opcode ) const

inline

Definition at line 719 of file SLPVectorizer.cpp.

References Opts.

getMaxVecRegSize()

unsigned slpvectorizer::BoUpSLP::getMaxVecRegSize ( ) const

inline

Definition at line 674 of file SLPVectorizer.cpp.

getMinVecRegSize()

unsigned slpvectorizer::BoUpSLP::getMinVecRegSize ( ) const

inline

Definition at line 682 of file SLPVectorizer.cpp.

Referenced by getMinVF().

getMinVF()

unsigned slpvectorizer::BoUpSLP::getMinVF ( unsigned Sz ) const

inline

Definition at line 715 of file SLPVectorizer.cpp.

References getMinVecRegSize().

Referenced by canVectorizeLoads().

getNumberOfParts()

unsigned slpvectorizer::BoUpSLP::getNumberOfParts ( Type * VecTy, Type * ScalarTy, unsigned Limit = std::numeric_limits<unsigned>::max() ) const

inline

Returns

the number of parts, the type VecTy is split at the codegen phase. The type legalization queries are repeated for the very same types during the analysis, so the results are cached for the function.

Definition at line 689 of file SLPVectorizer.cpp.

References llvm::slpvectorizer::getNumberOfPartsOrRegs(), and Opts.

Referenced by computeMinimumValueSizes(), findReusedOrderedScalars(), getReorderingData(), optimizeGatherSequence(), and processBuildVector().

getNumSplatSubtreeEntries()

unsigned slpvectorizer::BoUpSLP::getNumSplatSubtreeEntries ( ) const

inline

Number of tree entries that form the splat gather subtrees.

Definition at line 592 of file SLPVectorizer.cpp.

getORE()

OptimizationRemarkEmitter * slpvectorizer::BoUpSLP::getORE ( )

inline

Definition at line 884 of file SLPVectorizer.cpp.

getPointersDiff()

std::optional< int64_t > slpvectorizer::BoUpSLP::getPointersDiff ( Type * ElemTyA, Value * PtrA, Type * ElemTyB, Value * PtrB, const DataLayout & DL, ScalarEvolution & SE, bool StrictCheck = false, bool CheckType = true ) const

inline

Cached llvm::getPointersDiff(), see SLPAnalysisCache.

Definition at line 2115 of file SLPVectorizer.cpp.

getReductionType()

FixedVectorType * slpvectorizer::BoUpSLP::getReductionType ( ) const

inline

Returns reduction type after minbitdth analysis.

Definition at line 459 of file SLPVectorizer.cpp.

References llvm::cast(), llvm::IntegerType::get(), getContext(), getRootNode(), getRootNodeScalars(), getType(), and llvm::slpvectorizer::getWidenedType().

getRegUsageForType()

unsigned slpvectorizer::BoUpSLP::getRegUsageForType ( Type * VecTy, Type * ScalarTy, unsigned Limit = std::numeric_limits<unsigned>::max() ) const

inline

Returns

the number of parts, the type VecTy is split at the codegen phase. The type legalization queries are repeated for the very same types during the analysis, so the results are cached for the function.

Definition at line 703 of file SLPVectorizer.cpp.

References llvm::slpvectorizer::getNumberOfPartsOrRegs(), and Opts.

getReorderingData()

std::optional< BoUpSLP::OrdersType > BoUpSLP::getReorderingData	(	const TreeEntry &	TE,
		bool	TopToBottom,
		bool	IgnoreReorder )

Gets reordering data for the given tree entry.

If the entry is vectorized

• just return ReorderIndices, otherwise check if the scalars can be reordered and return the most optimal order.

  Returns

  std::nullopt if ordering is not important, empty order, if identity order is important, or the actual order.

  Parameters

  TopToBottom	If true, include the order of vectorized stores and insertelement nodes, otherwise skip them.
  IgnoreReorder	true, if the root node order can be ignored.

Definition at line 6647 of file SLPVectorizer.cpp.

References llvm::slpvectorizer::addMask(), llvm::all_of(), llvm::slpvectorizer::allConstant(), llvm::slpvectorizer::allSameType(), llvm::any_of(), llvm::ArrayRef(), assert(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), BlendedLoadVectorize, canVectorizeLoads(), llvm::cast(), llvm::Instruction::comesBefore(), CompressVectorize, llvm::count_if(), llvm::Data, llvm::divideCeil(), llvm::dyn_cast(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::end(), llvm::enumerate(), llvm::find(), llvm::find_if_not(), findPartiallyOrderedLoads(), findReusedOrderedScalars(), llvm::slpvectorizer::fixupOrderingIndices(), llvm::SmallVectorTemplateCommon< T, typename >::front(), llvm::PoisonValue::get(), llvm::slpvectorizer::getElementIndex(), llvm::slpvectorizer::getExtractIndex(), getNumberOfParts(), llvm::Value::getNumUses(), getParent(), llvm::ilist_detail::node_parent_access< NodeTy, ParentTy >::getParent(), llvm::slpvectorizer::getShuffleCost(), llvm::slpvectorizer::getValueType(), llvm::slpvectorizer::getWidenedType(), I, II, llvm::slpvectorizer::inversePermutation(), llvm::isa(), llvm::slpvectorizer::isAlternateInstruction(), llvm::IsaPred, llvm::Instruction::isBinaryOp(), isConstant(), isIdentityOrder(), llvm::ShuffleVectorInst::isOneUseSingleSourceMask(), llvm::slpvectorizer::isSplat(), llvm::make_filter_range(), llvm::make_isa_range(), Opts, P, llvm::PoisonMaskElem, llvm::slpvectorizer::reorderOrder(), llvm::seq(), llvm::SmallBitVector::set(), llvm::SmallVectorTemplateCommon< T, typename >::size(), llvm::TargetTransformInfo::SK_PermuteSingleSrc, llvm::stable_sort(), StridedVectorize, llvm::SmallBitVector::test(), llvm::transform(), llvm::Value::use_empty(), llvm::Value::user_begin(), llvm::V1, Vectorize, and llvm::zip().

Referenced by reorderBottomToTop(), and reorderTopToBottom().

getRootNode() [1/2]

TreeEntry & slpvectorizer::BoUpSLP::getRootNode ( )

inline

Definition at line 415 of file SLPVectorizer.cpp.

References assert().

Referenced by calculateTreeCostAndTrimNonProfitable(), canBuildSplitNode(), canSplitStoredValues(), computeMinimumValueSizes(), findRootLaneForValue(), getReductionType(), getRootNodeScalars(), getRootNodeTypeWithNoCast(), getSpillCost(), getTreeCost(), isProfitableToReorder(), isReducedBitcastRoot(), isReducedCmpBitcastRoot(), isSignedMinBitwidthRootNode(), isTreeNotExtendable(), isTreeTinyAndNotFullyVectorizable(), reorderBottomToTop(), reorderTopToBottom(), transformNodes(), and vectorizeTree().

getRootNode() [2/2]

const TreeEntry & slpvectorizer::BoUpSLP::getRootNode ( ) const

inline

Definition at line 420 of file SLPVectorizer.cpp.

References assert().

getRootNodeScalars()

ArrayRef< Value * > slpvectorizer::BoUpSLP::getRootNodeScalars ( ) const

inline

Returns the scalars of the root node.

Definition at line 426 of file SLPVectorizer.cpp.

References getRootNode().

Referenced by canSplitStoredValues(), getReductionType(), getTreeCost(), and isProfitableToReorder().

getRootNodeTypeWithNoCast()

std::optional< std::pair< Type *, bool > > slpvectorizer::BoUpSLP::getRootNodeTypeWithNoCast ( ) const

inline

Returns the type/is-signed info for the root node in the graph without casting.

Definition at line 435 of file SLPVectorizer.cpp.

References llvm::cast(), llvm::SmallVectorTemplateCommon< T, typename >::front(), llvm::IntegerType::get(), llvm::Value::getContext(), getRootNode(), llvm::Value::getType(), and llvm::Type::isIntegerTy().

getRuntimeChecksCost()

InstructionCost BoUpSLP::getRuntimeChecksCost

(

)

const

Returns the modeled cost of the runtime alias checks collected during the last (optimistic) buildTree().

Definition at line 26171 of file SLPVectorizer.cpp.

References llvm::Type::getInt1Ty(), and llvm::CmpInst::ICMP_ULT.

Referenced by canVersionForRuntimeChecks().

getSpillCost()

InstructionCost BoUpSLP::getSpillCost

(

)

Returns

the cost incurred by unwanted spills and fills, caused by holding live values over call sites.

Definition at line 18783 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::slpvectorizer::allConstant(), llvm::SmallVectorImpl< T >::append(), assert(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::at(), BlockSize, llvm::cast(), Cleanup, llvm::Instruction::comesBefore(), llvm::SmallPtrSetImpl< PtrType >::contains(), llvm::dyn_cast(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::find(), llvm::find_if_not(), llvm::First, Gather, llvm::IntegerType::get(), llvm::IntrinsicCostAttributes::getArgTypes(), llvm::Type::getContext(), llvm::BasicBlock::getFirstNonPHIOrAlloca(), llvm::BasicBlock::getParent(), llvm::ilist_detail::node_parent_access< NodeTy, ParentTy >::getParent(), getRootNode(), llvm::BasicBlock::getTerminator(), llvm::slpvectorizer::getWidenedType(), I, II, llvm::InnerAnalysisManagerProxy< AnalysisManagerT, IRUnitT, ExtraArgTs... >::Key, llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::isa(), llvm::IsaPred, isVectorized(), llvm::Type::isVectorTy(), llvm::Last, llvm::make_filter_range(), llvm::make_isa_range(), Operands, Opts, llvm::SmallVectorImpl< T >::pop_back_val(), llvm::pred_begin(), llvm::pred_end(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::BasicBlock::size(), and llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::try_emplace().

Referenced by getTreeCost().

getTreeCost()

InstructionCost BoUpSLP::getTreeCost	(	InstructionCost	TreeCost,
		ArrayRef< Value * >	VectorizedVals = {},
		InstructionCost	ReductionCost = TTI::TCC_Free,
		Instruction *	RdxRoot = nullptr )

Returns

the vectorization cost of the subtree that starts at VL. A negative number means that this is profitable.

Definition at line 20475 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::any_of(), areTwoInsertFromSameBuildVector(), assert(), llvm::SmallVectorImpl< T >::assign(), llvm::ArrayRef< T >::back(), llvm::sampleprof::Base, llvm::SmallVectorTemplateCommon< T, typename >::begin(), C(), llvm::cast(), llvm::cast_if_present(), llvm::detail::DenseSetImpl< ValueT, MapTy >::contains(), llvm::count_if(), llvm::Data, llvm::dbgs(), llvm::dump(), llvm::dyn_cast(), llvm::dyn_cast_if_present(), llvm::SmallVectorImpl< T >::emplace_back(), llvm::ArrayRef< T >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::ArrayRef< T >::end(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::SmallVectorTemplateCommon< T, typename >::end(), llvm::enumerate(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::find(), llvm::find_if(), llvm::find_if_not(), llvm::slpvectorizer::findInnermostNonInvariantLoop(), llvm::ArrayRef< T >::front(), llvm::SmallVectorTemplateCommon< T, typename >::front(), llvm::IntegerType::get(), getCastContextHint(), llvm::Type::getContext(), getCostThreshold(), getElementIndex(), llvm::slpvectorizer::getExtractWithExtendCost(), llvm::ExtractValueInst::getIndexedType(), llvm::InstructionCost::getInvalid(), llvm::User::getOperand(), getParent(), llvm::BasicBlock::getParent(), llvm::ilist_detail::node_parent_access< NodeTy, ParentTy >::getParent(), getRootNode(), getRootNodeScalars(), llvm::Type::getScalarType(), llvm::slpvectorizer::getShuffleCost(), getSpillCost(), llvm::BasicBlock::getTerminator(), getType(), llvm::InsertElementInst::getType(), llvm::getUnderlyingObject(), llvm::slpvectorizer::getValueType(), llvm::slpvectorizer::getVectorInstrCost(), llvm::slpvectorizer::getWidenedType(), llvm::APInt::getZero(), llvm::has_single_bit(), hasHighRegisterPressure(), llvm::Value::hasOneUse(), I, II, llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::insert(), llvm::detail::DenseSetImpl< ValueT, MapTy >::insert(), llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::is_contained(), llvm::isa(), llvm::isa_and_nonnull(), llvm::BasicBlock::isEHPad(), llvm::slpvectorizer::isFirstInsertElement(), llvm::ShuffleVectorInst::isIdentityMask(), llvm::isKnownNonNegative(), llvm::InstructionCost::isValid(), isVectorized(), llvm::isVectorizedTy(), LLVM_DEBUG, llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::lookup(), llvm::MIPatternMatch::m_OneUse(), llvm::PatternMatch::m_Value(), llvm::PatternMatch::m_ZExtOrSExt(), llvm::make_filter_range(), llvm::PatternMatch::match(), llvm::MutableArrayRef(), llvm::TargetTransformInfo::None, llvm::none_of(), Opts, P, performExtractsShuffleAction(), PHI, llvm::PoisonMaskElem, llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::seq(), llvm::ArrayRef< T >::size(), llvm::SmallVectorTemplateCommon< T, typename >::size(), llvm::TargetTransformInfo::SK_PermuteSingleSrc, llvm::TargetTransformInfo::SK_PermuteTwoSrc, llvm::TargetTransformInfo::TCC_Basic, llvm::TargetTransformInfo::TCC_Free, llvm::TargetTransformInfo::TCK_CodeSize, llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::try_emplace(), llvm::Value::user_back(), llvm::slpvectorizer::UsesLimit, llvm::Vector, and llvm::ViewGraph().

getTreeSize()

unsigned slpvectorizer::BoUpSLP::getTreeSize ( ) const

inline

Definition at line 551 of file SLPVectorizer.cpp.

Referenced by isTreeNotExtendable().

getTreeSizeExcludingGathers()

unsigned slpvectorizer::BoUpSLP::getTreeSizeExcludingGathers ( ) const

inline

Returns the size of the current tree, stopping at gathers For instance, even though we continue adding tree entries after a split node, this function stops counting at the split node since it is a gather.

This matches the behavior of MinBWs computation.

Definition at line 558 of file SLPVectorizer.cpp.

References llvm::seq(), llvm::SmallBitVector::set(), and llvm::SmallBitVector::test().

Referenced by computeMinimumValueSizes().

getVectorElementSize()

unsigned BoUpSLP::getVectorElementSize

(

Value *

V

)

Returns

The vector element size in bits to use when vectorizing the expression tree ending at V. If V is a store, the size is the width of the stored value. Otherwise, the size is the width of the largest loaded value reaching V. This method is used by the vectorizer to calculate vectorization factors.

Definition at line 28756 of file SLPVectorizer.cpp.

References llvm::dyn_cast(), llvm::SmallVectorImpl< T >::emplace_back(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::Type::getInt1Ty(), llvm::CallBase::getIntrinsicID(), llvm::Value::getType(), getVectorElementSize(), I, llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::isa(), Opts, llvm::SmallVectorImpl< T >::pop_back_val(), and llvm::Store.

Referenced by getVectorElementSize().

hasNonCheckableMemBlocker()

bool slpvectorizer::BoUpSLP::hasNonCheckableMemBlocker ( ) const

inline

Returns true if the last buildTree() kept a may-alias memory dependency that is not runtime-checkable (call or a non-simple mem access).

Such a dependency cannot be dropped, so a runtime-checks retry cannot unblock the region and would be pure overhead.

Definition at line 311 of file SLPVectorizer.cpp.

hasNonConstantGathers()

bool slpvectorizer::BoUpSLP::hasNonConstantGathers ( ) const

inline

Returns true if the tree gathers any scalars other than constants.

Definition at line 580 of file SLPVectorizer.cpp.

References llvm::slpvectorizer::allConstant(), and llvm::any_of().

hasResolvedUser()

bool slpvectorizer::BoUpSLP::hasResolvedUser ( Instruction * I ) const

inline

Returns true if the role of I is already decided by its user: a deleted user was folded into some other vector by an earlier attempt.

Definition at line 2299 of file SLPVectorizer.cpp.

References llvm::any_of(), and I.

hasRuntimeAliasChecks()

bool slpvectorizer::BoUpSLP::hasRuntimeAliasChecks ( ) const

inline

Returns true if the last (optimistic) buildTree() collected any runtime alias checks that must guard the vectorized region.

Definition at line 409 of file SLPVectorizer.cpp.

hasRuntimeCheckableBlockers()

bool slpvectorizer::BoUpSLP::hasRuntimeCheckableBlockers ( ) const

inline

Returns true if the last buildTree() observed a may-alias memory dependency between two distinct, range-checkable base objects, i.e.

a dependency that could be turned into a runtime alias check.

Definition at line 296 of file SLPVectorizer.cpp.

hasSameNode()

bool slpvectorizer::BoUpSLP::hasSameNode ( const InstructionsState & S, ArrayRef< Value * > VL ) const

inline

Checks whether some existing tree entry has scalars equal to VL.

S is the common opcode of VL when one exists; an empty S means the values have no common opcode (mixed buildvector/gather candidates).

Definition at line 844 of file SLPVectorizer.cpp.

References llvm::any_of(), llvm::slpvectorizer::InstructionsState::getMainOp(), llvm::SmallPtrSetImpl< PtrType >::insert(), and isConstant().

isAnalyzedBundle()

bool slpvectorizer::BoUpSLP::isAnalyzedBundle ( ArrayRef< Value * > VL ) const

inline

Checks if the given bundle was already rejected as non-vectorizable.

Definition at line 2249 of file SLPVectorizer.cpp.

References llvm::hash_value().

isAnalyzedReductionRoot()

bool slpvectorizer::BoUpSLP::isAnalyzedReductionRoot ( Instruction * I ) const

inline

Checks if the instruction was already analyzed for being possible reduction root.

Definition at line 2226 of file SLPVectorizer.cpp.

References I.

isAnalyzedScalar()

bool slpvectorizer::BoUpSLP::isAnalyzedScalar ( const Value * V ) const

inline

Checks if the value was already a part of the analyzed vector node.

Definition at line 2245 of file SLPVectorizer.cpp.

isAnyGathered()

bool slpvectorizer::BoUpSLP::isAnyGathered ( const SmallDenseSet< Value * > & Vals ) const

inline

Checks if the given value is gathered in one of the nodes.

Definition at line 2264 of file SLPVectorizer.cpp.

References llvm::any_of(), and llvm::detail::DenseSetImpl< ValueT, MapTy >::contains().

isCFGChanged()

bool slpvectorizer::BoUpSLP::isCFGChanged ( ) const

inline

Returns true if vectorization changed the CFG (i.e.

a block was versioned with runtime alias checks). When true, CFG analyses must not be preserved.

Definition at line 413 of file SLPVectorizer.cpp.

isDeleted()

bool slpvectorizer::BoUpSLP::isDeleted ( Instruction * I ) const

inline

Checks if the instruction is marked for deletion.

Definition at line 2111 of file SLPVectorizer.cpp.

References I.

Referenced by buildExternalUses(), canVectorStoreInsertValue(), and optimizeGatherSequence().

isEphemeralValue()

bool slpvectorizer::BoUpSLP::isEphemeralValue ( const Value * V ) const

inline

Checks if the value is used only by the assume-like intrinsics.

Definition at line 2127 of file SLPVectorizer.cpp.

isGathered()

bool slpvectorizer::BoUpSLP::isGathered ( const Value * V ) const

inline

Checks if the given value is gathered in one of the nodes.

Definition at line 2268 of file SLPVectorizer.cpp.

Referenced by buildExternalUses().

isIdentityOrder()

bool slpvectorizer::BoUpSLP::isIdentityOrder ( ArrayRef< unsigned > Order )

inlinestatic

Does this non-empty order represent an identity order?

Identity should be represented as an empty order, so this is used to decide if we can canonicalize a computed order. Undef elements (represented as size) are ignored.

Definition at line 603 of file SLPVectorizer.cpp.

References llvm::all_of(), assert(), llvm::ArrayRef< T >::empty(), llvm::enumerate(), P, and llvm::ArrayRef< T >::size().

Referenced by canBuildSplitNode(), getReorderingData(), reorderBottomToTop(), and reorderTopToBottom().

isNotScheduled()

bool slpvectorizer::BoUpSLP::isNotScheduled ( const Value * V ) const

inline

Checks if the specified value was not schedule.

Definition at line 2272 of file SLPVectorizer.cpp.

isProfitableToReorder()

bool BoUpSLP::isProfitableToReorder

(

)

const

Checks if it is profitable to reorder the current tree.

If the tree does not contain many profitable reordable nodes, better to skip it to save compile time.

Definition at line 7036 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::any_of(), llvm::ArrayRef(), llvm::count_if(), llvm::ArrayRef< T >::empty(), getRootNode(), getRootNodeScalars(), llvm::IsaPred, llvm::Instruction::isBinaryOp(), llvm::slpvectorizer::isCommutative(), llvm::isGather(), llvm::none_of(), Opts, and llvm::size().

isReassocScalarVectorized()

bool slpvectorizer::BoUpSLP::isReassocScalarVectorized ( const Value * V ) const

inline

Check if V is a peeled reassociated scalar still owned by a live (non-deleted, non-gathered) tree entry.

Definition at line 2278 of file SLPVectorizer.cpp.

References llvm::any_of().

Referenced by buildExternalUses(), and isVectorized().

isReducedBitcastRoot()

bool slpvectorizer::BoUpSLP::isReducedBitcastRoot ( ) const

inline

Returns true if the tree results in one of the reduced bitcasts variants.

Definition at line 474 of file SLPVectorizer.cpp.

References getRootNode().

isReducedCmpBitcastRoot()

bool slpvectorizer::BoUpSLP::isReducedCmpBitcastRoot ( ) const

inline

Returns true if the tree results in the reduced cmp bitcast root.

Definition at line 485 of file SLPVectorizer.cpp.

References getRootNode().

isReductionTree()

bool slpvectorizer::BoUpSLP::isReductionTree ( ) const

inline

Returns true if the tree is a reduction tree.

Definition at line 492 of file SLPVectorizer.cpp.

isScalarFallbackBlock()

bool slpvectorizer::BoUpSLP::isScalarFallbackBlock ( BasicBlock * BB ) const

inline

Returns true if BB is a scalar fallback block created by runtime alias check versioning.

Definition at line 387 of file SLPVectorizer.cpp.

isSignedMinBitwidthRootNode()

bool slpvectorizer::BoUpSLP::isSignedMinBitwidthRootNode ( ) const

inline

Checks if the root graph node can be emitted with narrower bitwidth at codegen and returns it signedness, if so.

Definition at line 454 of file SLPVectorizer.cpp.

References getRootNode().

isStridedLoad()

bool BoUpSLP::isStridedLoad	(	ArrayRef< Value * >	PointerOps,
		Type *	ScalarTy,
		Align	Alignment,
		const int64_t	Diff,
		const size_t	Sz ) const

Checks if strided loads can be generated out of VL loads with pointers PointerOps:

1. Target with strided load support is detected.
2. The number of loads is greater than MinProfitableStridedLoads, or the potential stride <= MaxProfitableStride and the potential stride is power-of-2 (to avoid perf regressions for the very small number of loads) and max distance > number of loads, or potential stride is -1.
3. The loads are ordered, or number of unordered loads <= MaxProfitableUnorderedLoads, or loads are in reversed order. (this check is to avoid extra costs for very expensive shuffles).
4. Any pointer operand is an instruction with the users outside of the current graph (for masked gathers extra extractelement instructions might be required).

Definition at line 5829 of file SLPVectorizer.cpp.

References llvm::any_of(), llvm::slpvectorizer::getWidenedType(), llvm::has_single_bit(), I, llvm::make_isa_range(), and Opts.

Referenced by analyzeConstantStrideCandidate().

isTreeNotExtendable()

bool BoUpSLP::isTreeNotExtendable

(

)

const

Checks if the graph and all its subgraphs cannot be better vectorized.

It may happen, if all gather nodes are loads and they cannot be "clusterized". In this case even subgraphs cannot be vectorized more effectively than the base graph.

Definition at line 18749 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::slpvectorizer::allConstant(), llvm::slpvectorizer::allSameBlock(), llvm::ArrayRef(), llvm::count_if(), getCanonicalGraphSize(), getRootNode(), llvm::slpvectorizer::getSameOpcode(), getTreeSize(), llvm::isa(), llvm::IsaPred, isSplat(), and llvm::seq().

isTreeTinyAndNotFullyVectorizable()

bool BoUpSLP::isTreeTinyAndNotFullyVectorizable

(

bool

ForReduction = false

)

const

Returns

True if the VectorizableTree is both tiny and not fully vectorizable. We do not vectorize such trees.

Definition at line 18372 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::slpvectorizer::allConstant(), llvm::slpvectorizer::allSameBlock(), llvm::any_of(), llvm::ArrayRef(), assert(), llvm::cast(), llvm::count_if(), llvm::dbgs(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::APInt::getAllOnes(), getCostThreshold(), getRootNode(), llvm::slpvectorizer::getWidenedType(), I, llvm::isa(), llvm::IsaPred, llvm::isGather(), isSplat(), LLVM_DEBUG, llvm::make_isa_range(), llvm::none_of(), Opts, llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::DebugCounter::shouldExecute(), and llvm::SmallVectorTemplateCommon< T, typename >::size().

isTryingRuntimeAliasChecks()

bool slpvectorizer::BoUpSLP::isTryingRuntimeAliasChecks ( ) const

inline

Returns true if the current vectorization attempt may drop runtime-checkable may-alias dependencies and guard the region with runtime alias checks.

Definition at line 319 of file SLPVectorizer.cpp.

isVectorized()

bool slpvectorizer::BoUpSLP::isVectorized ( const Value * V ) const

inline

Check if the value is vectorized in the tree.

Definition at line 2288 of file SLPVectorizer.cpp.

References llvm::any_of(), assert(), and isReassocScalarVectorized().

Referenced by canVersionForRuntimeChecks(), getSpillCost(), getTreeCost(), processBuildVector(), and vectorizeTree().

isWidenedStridedLoad()

bool BoUpSLP::isWidenedStridedLoad

(

const TreeEntry *

E

)

const

Returns true for widened strided loads, where each lane of the strided load covers a group of consecutive scalars.

Reversed widened strided loads must be emitted with a positive stride and a reorder shuffle.

Definition at line 5861 of file SLPVectorizer.cpp.

References llvm::FixedVectorType::getNumElements(), and slpvectorizer::BoUpSLP::StridedPtrInfo::Ty.

markCopyableDepsForRecalc()

void BoUpSLP::markCopyableDepsForRecalc

(

TreeEntry &

TE

)

Marks the schedule data of the copyable-modeled operands of TE for dependency recalculation at the next bundle scheduling.

Definition at line 7164 of file SLPVectorizer.cpp.

Referenced by ::HorizontalReduction, reorderBottomToTop(), and reorderTopToBottom().

markRuntimeChecksFailedForBlock()

void slpvectorizer::BoUpSLP::markRuntimeChecksFailedForBlock ( BasicBlock * BB )

inline

Records that an optimistic runtime-checks versioning attempt failed for BB.

Definition at line 399 of file SLPVectorizer.cpp.

optimizeGatherSequence()

void BoUpSLP::optimizeGatherSequence

(

)

Perform LICM and CSE on the newly generated gather sequences.

Definition at line 27296 of file SLPVectorizer.cpp.

References A(), llvm::any_of(), assert(), llvm::SmallVectorImpl< T >::assign(), B(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), llvm::SmallPtrSetImplBase::clear(), llvm::dbgs(), llvm::dyn_cast(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::end(), eraseInstruction(), llvm::ilist_node_impl< OptionsT >::getIterator(), getNumberOfParts(), llvm::BasicBlock::getTerminator(), llvm::slpvectorizer::getWidenedType(), I, llvm::is_contained(), llvm::isa(), isDeleted(), LLVM_DEBUG, llvm::make_early_inc_range(), llvm::make_filter_range(), llvm::make_isa_range(), N, llvm::PoisonMaskElem, llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::SmallVectorImpl< T >::reserve(), llvm::ArrayRef< T >::size(), llvm::SmallVectorTemplateCommon< T, typename >::size(), and llvm::sort().

processBuildVector()

template<typename BVTy, typename ResTy, typename... Args>

ResTy slpvectorizer::BoUpSLP::processBuildVector	(	const TreeEntry *	E,
		Type *	ScalarTy,
		Args &...	Params )

Definition at line 23351 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::any_of(), llvm::SmallVectorImpl< T >::append(), llvm::ArrayRef(), assert(), llvm::SmallVectorImpl< T >::assign(), llvm::ArrayRef< T >::back(), llvm::SmallVectorTemplateCommon< T, typename >::back(), llvm::ArrayRef< T >::begin(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), llvm::cast(), llvm::SmallVectorImpl< T >::clear(), llvm::copy(), llvm::count_if(), llvm::dbgs(), llvm::dyn_cast(), llvm::ArrayRef< T >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::ArrayRef< T >::end(), llvm::SmallVectorTemplateCommon< T, typename >::end(), llvm::enumerate(), llvm::equal(), llvm::find_if(), llvm::find_if_not(), llvm::for_each(), llvm::ArrayRef< T >::front(), llvm::SmallVectorTemplateCommon< T, typename >::front(), llvm::PoisonValue::get(), llvm::UndefValue::get(), getNumberOfParts(), llvm::slpvectorizer::getNumElems(), llvm::slpvectorizer::getPartNumElems(), llvm::Value::getType(), llvm::slpvectorizer::getWidenedType(), I, if(), llvm::slpvectorizer::inversePermutation(), llvm::is_contained(), llvm::isa(), llvm::IsaPred, isConstant(), llvm::ShuffleVectorInst::isExtractSubvectorMask(), llvm::isGuaranteedNotToBePoison(), llvm::ShuffleVectorInst::isIdentityMask(), isSplat(), isVectorized(), LLVM_DEBUG, llvm::make_filter_range(), llvm::make_isa_range(), llvm::MutableArrayRef(), llvm::none_of(), P, llvm::PoisonMaskElem, llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::slpvectorizer::reorderScalars(), llvm::seq(), llvm::slpvectorizer::shortBundleName(), llvm::ArrayRef< T >::size(), llvm::SmallVectorTemplateCommon< T, typename >::size(), llvm::TargetTransformInfo::SK_PermuteSingleSrc, llvm::TargetTransformInfo::SK_PermuteTwoSrc, llvm::ArrayRef< T >::slice(), std::swap(), llvm::transform(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::try_emplace(), llvm::V1, llvm::Values, and llvm::zip().

processExtractedFieldsGather()

template<typename ResTy, typename BVTy>

std::optional< ResTy > slpvectorizer::BoUpSLP::processExtractedFieldsGather	(	BVTy &	ShuffleBuilder,
		const TreeEntry *	E,
		Type *	ScalarTy,
		ArrayRef< Value * >	VL,
		ArrayRef< std::pair< const TreeEntry *, unsigned > >	SubVectors )

Definition at line 23332 of file SLPVectorizer.cpp.

References llvm::Type::getIntegerBitWidth(), llvm::Type::isIntegerTy(), and llvm::slpvectorizer::matchGatheredExtractedFields().

registerNonVectorizableLoads()

template<typename T>

void slpvectorizer::BoUpSLP::registerNonVectorizableLoads ( ArrayRef< T * > VL )

inline

Registers non-vectorizable sequence of loads.

Definition at line 874 of file SLPVectorizer.cpp.

References llvm::hash_value().

removeInstructionsAndOperands()

template<typename T>

void slpvectorizer::BoUpSLP::removeInstructionsAndOperands ( ArrayRef< T * > DeadVals, ArrayRef< ReductionVectorPart > VectorValuesAndScales )

inline

Remove instructions from the parent function and clear the operands of DeadVals instructions, marking for deletion trivially dead operands.

Definition at line 2139 of file SLPVectorizer.cpp.

References llvm::all_of(), assert(), llvm::cast(), llvm::dyn_cast(), llvm::ArrayRef< T >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), eraseInstruction(), I, llvm::detail::DenseSetImpl< ValueT, MapTy >::insert(), llvm::isInstructionTriviallyDead(), llvm::none_of(), llvm::SmallVectorImpl< T >::pop_back_val(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::salvageDebugInfo(), llvm::Value::use_empty(), and llvm::wouldInstructionBeTriviallyDead().

reorderBottomToTop()

void BoUpSLP::reorderBottomToTop

(

bool

IgnoreReorder = false

)

Reorders the current graph to the most profitable order starting from leaves to the root.

It allows to rotate small subgraphs and reduce the number of reshuffles if the leaf nodes use the same order. In this case we can merge the orders and just shuffle user node instead of shuffling its operands. Plus, even the leaf nodes have different orders, it allows to sink reordering in the graph closer to the root node and merge it later during analysis.

Definition at line 7564 of file SLPVectorizer.cpp.

References AbstractManglingParser< Derived, Alloc >::NumOps, AbstractManglingParser< Derived, Alloc >::Ops, llvm::all_of(), llvm::any_of(), assert(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), llvm::SmallPtrSetImplBase::clear(), llvm::SmallVectorImpl< T >::clear(), llvm::slpvectorizer::combineOrders(), llvm::detail::DenseSetImpl< ValueT, MapTy >::contains(), llvm::detail::DenseSetImpl< ValueT, MapTy >::count(), llvm::count_if(), llvm::Data, llvm::dyn_cast(), llvm::ArrayRef< T >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::SmallPtrSetImpl< PtrType >::erase(), llvm::find_if_not(), llvm::slpvectorizer::fixupOrderingIndices(), Gather, getReorderingData(), getRootNode(), llvm::slpvectorizer::getValueType(), llvm::getVectorIntrinsicIDForCall(), I, llvm::detail::DenseSetImpl< ValueT, MapTy >::insert(), llvm::SetVector< T, Vector, Set, N >::insert(), llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::SmallPtrSetImpl< PtrType >::insert_range(), llvm::slpvectorizer::inversePermutation(), llvm::isa(), isConstant(), isIdentityOrder(), llvm::isVectorIntrinsicWithScalarOpAtArg(), llvm::make_filter_range(), llvm::make_second_range(), markCopyableDepsForRecalc(), llvm::Intrinsic::not_intrinsic, Opts, P, llvm::PoisonMaskElem, llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::slpvectorizer::reorderOrder(), llvm::slpvectorizer::reorderReuses(), llvm::slpvectorizer::reorderScalars(), llvm::seq(), llvm::ArrayRef< T >::size(), llvm::SmallVectorTemplateCommon< T, typename >::size(), llvm::SetVector< T, Vector, Set, N >::takeVector(), llvm::transform(), and Users.

reorderTopToBottom()

void BoUpSLP::reorderTopToBottom

(

)

Reorders the current graph to the most profitable order starting from the root node to the leaf nodes.

The best order is chosen only from the nodes of the same size (vectorization factor). Smaller nodes are considered parts of subgraph with smaller VF and they are reordered independently. We can make it because we still need to extend smaller nodes to the wider VF and we can merge reordering shuffles with the widening shuffles.

Definition at line 7172 of file SLPVectorizer.cpp.

References llvm::slpvectorizer::addMask(), llvm::any_of(), assert(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), llvm::cast(), Cleanup, llvm::slpvectorizer::combineOrders(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::count(), llvm::ArrayRef< T >::empty(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::erase(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::find(), llvm::slpvectorizer::fixupOrderingIndices(), llvm::for_each(), llvm::slpvectorizer::getAltInstrMask(), getReorderingData(), getRootNode(), llvm::slpvectorizer::getValueType(), llvm::slpvectorizer::getWidenedType(), I, llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::insert(), llvm::slpvectorizer::inversePermutation(), llvm::is_contained(), llvm::isa(), isIdentityOrder(), llvm::make_filter_range(), markCopyableDepsForRecalc(), Opts, llvm::PoisonMaskElem, llvm::slpvectorizer::reorderOrder(), llvm::slpvectorizer::reorderScalars(), llvm::ArrayRef< T >::size(), llvm::transform(), and llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::try_emplace().

resetRuntimeAliasCheckState()

void slpvectorizer::BoUpSLP::resetRuntimeAliasCheckState ( )

inline

Resets the runtime alias check data.

Definition at line 362 of file SLPVectorizer.cpp.

runtimeChecksFailedForBlock()

bool slpvectorizer::BoUpSLP::runtimeChecksFailedForBlock ( BasicBlock * BB ) const

inline

Returns true if an optimistic runtime-checks versioning attempt already failed for BB, so further retries in the same block can be skipped.

Definition at line 393 of file SLPVectorizer.cpp.

setHasNonCheckableMemBlocker()

void slpvectorizer::BoUpSLP::setHasNonCheckableMemBlocker ( bool V )

inline

Records that a non-runtime-checkable may-alias dependency was kept.

Definition at line 314 of file SLPVectorizer.cpp.

setHasRuntimeCheckableBlockers()

void slpvectorizer::BoUpSLP::setHasRuntimeCheckableBlockers ( bool V )

inline

Records whether a may-alias dependency between distinct, range-checkable base objects has been observed, so the caller can decide to retry with runtime alias checks enabled.

Definition at line 303 of file SLPVectorizer.cpp.

setNarrowedChainInsts()

void slpvectorizer::BoUpSLP::setNarrowedChainInsts ( ArrayRef< Instruction * > Insts )

inline

Sets the narrowed reduction chain instructions, dropped together with the reduction.

Definition at line 289 of file SLPVectorizer.cpp.

References llvm::ArrayRef< T >::begin(), and llvm::ArrayRef< T >::end().

setTryRuntimeAliasChecks()

void slpvectorizer::BoUpSLP::setTryRuntimeAliasChecks ( bool V )

inline

Enables or disables dropping runtime-checkable may-alias dependencies in favor of runtime alias checks for the current vectorization attempt.

Definition at line 323 of file SLPVectorizer.cpp.

splitStoredValues()

SaveAndRestore< bool > slpvectorizer::BoUpSLP::splitStoredValues ( )

inlinenodiscard

Splits the stored values of the root stores into halves, vectorized as separate subtrees, while the returned guard is alive.

Definition at line 327 of file SLPVectorizer.cpp.

References llvm::SaveAndRestore().

transformNodes()

void BoUpSLP::transformNodes

(

)

Transforms graph nodes to target specific representations, if profitable.

Definition at line 14591 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::any_of(), llvm::count_if(), llvm::dyn_cast(), llvm::SmallVectorImpl< T >::emplace_back(), llvm::ArrayRef< T >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::equal(), findBestRootPair(), llvm::slpvectorizer::InstructionsState::getMainOp(), llvm::User::getNumOperands(), getRootNode(), I, llvm::is_contained(), llvm::isa(), llvm::isGather(), llvm::slpvectorizer::matchGatheredExtractedFields(), P, llvm::slpvectorizer::reorderScalars(), slpvectorizer::BoUpSLP::LookAheadHeuristics::ScoreSplatLoads, llvm::seq(), and llvm::sort().

vectorizeTree() [1/2]

Value * BoUpSLP::vectorizeTree

(

)

Vectorize the tree that starts with the elements in VL.

Returns the vectorized root.

Definition at line 26365 of file SLPVectorizer.cpp.

References vectorizeTree().

Referenced by vectorizeTree(), and vectorizeTree().

vectorizeTree() [2/2]

Value * BoUpSLP::vectorizeTree	(	const ExtraValueToDebugLocsMap &	ExternallyUsedValues,
		Instruction *	ReductionRoot = nullptr,
		ArrayRef< ReductionVectorPart >	VectorValuesAndScales = {} )

Vectorize the tree but with the list of externally used values ExternallyUsedValues.

Values in this MapVector can be replaced but the generated extractvalue instructions.

Definition at line 26371 of file SLPVectorizer.cpp.

References llvm::all_of(), llvm::any_of(), areTwoInsertFromSameBuildVector(), assert(), llvm::SmallVectorImpl< T >::assign(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), llvm::cast(), llvm::detail::DenseSetImpl< ValueT, MapTy >::contains(), llvm::detail::DenseSetImpl< ValueT, MapTy >::count(), llvm::slpvectorizer::createExtractVector(), llvm::Data, llvm::dbgs(), llvm::dyn_cast(), llvm::SmallVectorImpl< T >::emplace_back(), llvm::ArrayRef< T >::empty(), llvm::BasicBlock::end(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::SmallVectorTemplateCommon< T, typename >::end(), eraseInstruction(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::find(), llvm::find_if(), getElementIndex(), llvm::ilist_node_impl< OptionsT >::getIterator(), llvm::slpvectorizer::getNumElements(), llvm::User::getOperand(), llvm::ilist_detail::node_parent_access< NodeTy, ParentTy >::getParent(), getRootNode(), llvm::InsertElementInst::getType(), llvm::Value::getType(), llvm::slpvectorizer::getWidenedType(), I, II, llvm::InnerAnalysisManagerProxy< AnalysisManagerT, IRUnitT, ExtraArgTs... >::Key, llvm::detail::DenseSetImpl< ValueT, MapTy >::insert(), llvm::is_contained(), llvm::isa(), llvm::isa_and_nonnull(), llvm::Type::isIntOrIntVectorTy(), llvm::isKnownNonNegative(), isVectorized(), llvm::isVectorizedTy(), IV, LLVM_DEBUG, llvm::make_filter_range(), llvm::mayHaveNonDefUseDependency(), llvm::Instruction::moveAfter(), llvm::none_of(), Opts, PHI, llvm::PoisonMaskElem, llvm::slpvectorizer::redirectDbgValues(), llvm::Value::replaceAllUsesWith(), llvm::User::replaceUsesOfWith(), llvm::seq(), llvm::SmallVectorTemplateCommon< T, typename >::size(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::try_emplace(), llvm::User::User(), llvm::Value::users(), and vectorizeTree().

Friends And Related Symbol Documentation

::HorizontalReduction

friend class ::HorizontalReduction

friend

Definition at line 4219 of file SLPVectorizer.cpp.

References AbstractManglingParser< Derived, Alloc >::NumOps, llvm::all_of(), llvm::any_of(), assert(), llvm::ArrayRef< T >::begin(), llvm::SmallVectorTemplateCommon< T, typename >::begin(), BoUpSLP(), llvm::cast(), llvm::SetVector< T, Vector, Set, N >::clear(), llvm::SmallVectorImpl< T >::clear(), llvm::Instruction::comesBefore(), llvm::count(), llvm::Data, llvm::dbgs(), llvm::dyn_cast(), slpvectorizer::BoUpSLP::EdgeInfo::EdgeIdx, llvm::SmallVectorImpl< T >::emplace_back(), llvm::ArrayRef< T >::empty(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::empty(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::ArrayRef< T >::end(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::erase(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::find(), llvm::find(), llvm::for_each(), llvm::slpvectorizer::getLocation(), llvm::slpvectorizer::getNumberOfPotentiallyCommutativeOps(), llvm::User::getOperand(), llvm::ilist_detail::node_parent_access< NodeTy, ParentTy >::getParent(), llvm::getUnderlyingObject(), llvm::getVectorIntrinsicIDForCall(), I, llvm::SetVector< T, Vector, Set, N >::insert(), llvm::SmallPtrSetImpl< PtrType >::insert(), llvm::is_contained(), llvm::isa(), llvm::slpvectorizer::isCommutableOperand(), llvm::slpvectorizer::isCommutative(), llvm::slpvectorizer::InstructionsState::isExpandedBinOp(), llvm::isNotVisibleOnUnwind(), llvm::isVectorIntrinsicWithScalarOpAtArg(), LLVM_DEBUG, llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::lookup(), llvm::make_filter_range(), llvm::make_isa_range(), llvm::make_range(), markCopyableDepsForRecalc(), MinScheduleRegionSize, llvm::none_of(), llvm::User::operands(), P, PHI, llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::seq(), llvm::BasicBlock::size(), llvm::SmallVectorTemplateCommon< T, typename >::size(), llvm::DenseMapBase< StorageT, KeyT, ValueT, KeyInfoT, BucketT >::try_emplace(), slpvectorizer::BoUpSLP::EdgeInfo::UserTE, and verify.

DenseMapInfo< EdgeInfo >

friend struct DenseMapInfo< EdgeInfo >

friend

Definition at line 884 of file SLPVectorizer.cpp.

DOTGraphTraits< BoUpSLP * >

friend struct DOTGraphTraits< BoUpSLP * >

friend

Definition at line 4210 of file SLPVectorizer.cpp.

GraphTraits< BoUpSLP * >

friend struct GraphTraits< BoUpSLP * >

friend

Definition at line 4210 of file SLPVectorizer.cpp.

operator<< [1/4]

raw_ostream & operator<< ( raw_ostream & OS, const BoUpSLP::ScheduleBundle & Bundle )

friend

Definition at line 4101 of file SLPVectorizer.cpp.

operator<< [2/4]

raw_ostream & operator<< ( raw_ostream & OS, const BoUpSLP::ScheduleCopyableData & SD )

friend

Definition at line 4210 of file SLPVectorizer.cpp.

operator<< [3/4]

raw_ostream & operator<< ( raw_ostream & OS, const BoUpSLP::ScheduleData & SD )

friend

Definition at line 3994 of file SLPVectorizer.cpp.

operator<< [4/4]

raw_ostream & operator<< ( raw_ostream & OS, const BoUpSLP::ScheduleEntity & SE )

friend

Definition at line 3831 of file SLPVectorizer.cpp.

Member Data Documentation

Opts

const VectorizeOptions& slpvectorizer::BoUpSLP::Opts

Definition at line 218 of file SLPVectorizer.cpp.

Referenced by analyzeRtStrideCandidate(), blockBodyHasVectorInstructions(), BoUpSLP(), canBuildSplitNode(), canMapToVector(), canVectorizeLoads(), canVersionBlockForRuntimeChecks(), canVersionForRuntimeChecks(), compareCmp(), findBestRootPair(), findPartiallyOrderedLoads(), findReusedOrderedScalars(), forEachOperandChainCandidate(), getCostThreshold(), getInsertBuildVectorSrcTy(), getMaximumVF(), getNumberOfParts(), getRegUsageForType(), getReorderingData(), getSpillCost(), getTreeCost(), getVectorElementSize(), hasHighRegisterPressure(), isNonVectorizableInst(), isPoorThroughputOp(), isProfitableToReorder(), isStridedLoad(), isTreeTinyAndNotFullyVectorizable(), reorderBottomToTop(), reorderTopToBottom(), tryToFindDuplicates(), tryToVectorizeSequence(), and vectorizeTree().

---

The documentation for this class was generated from the following file:

• lib/Transforms/Vectorize/SLPVectorizer.cpp