LLVM 20.0.0git
|
VPTransformState holds information passed down when "executing" a VPlan, needed for generating the output IR. More...
#include "Transforms/Vectorize/VPlan.h"
Classes | |
struct | CFGState |
Hold state information used when constructing the CFG of the output IR, traversing the VPBasicBlocks and generating corresponding IR BasicBlocks. More... | |
struct | DataState |
Public Member Functions | |
VPTransformState (ElementCount VF, unsigned UF, LoopInfo *LI, DominatorTree *DT, IRBuilderBase &Builder, InnerLoopVectorizer *ILV, VPlan *Plan, LLVMContext &Ctx) | |
Value * | get (VPValue *Def, unsigned Part, bool IsScalar=false) |
Get the generated vector Value for a given VPValue Def and a given Part if IsScalar is false, otherwise return the generated scalar for Part . | |
Value * | get (VPValue *Def, const VPIteration &Instance) |
Get the generated Value for a given VPValue and given Part and Lane. | |
bool | hasVectorValue (VPValue *Def, unsigned Part) |
bool | hasScalarValue (VPValue *Def, VPIteration Instance) |
void | set (VPValue *Def, Value *V, unsigned Part, bool IsScalar=false) |
Set the generated vector Value for a given VPValue and a given Part, if IsScalar is false. | |
void | reset (VPValue *Def, Value *V, unsigned Part) |
Reset an existing vector value for Def and a given Part . | |
void | set (VPValue *Def, Value *V, const VPIteration &Instance) |
Set the generated scalar V for Def and the given Instance . | |
void | reset (VPValue *Def, Value *V, const VPIteration &Instance) |
Reset an existing scalar value for Def and a given Instance . | |
void | addNewMetadata (Instruction *To, const Instruction *Orig) |
Add additional metadata to To that was not present on Orig . | |
void | addMetadata (Value *To, Instruction *From) |
Add metadata from one instruction to another. | |
void | setDebugLocFrom (DebugLoc DL) |
Set the debug location in the builder using the debug location DL . | |
void | packScalarIntoVectorValue (VPValue *Def, const VPIteration &Instance) |
Construct the vector value of a scalarized value V one lane at a time. | |
Public Attributes | |
ElementCount | VF |
The chosen Vectorization and Unroll Factors of the loop being vectorized. | |
unsigned | UF |
std::optional< VPIteration > | Instance |
Hold the indices to generate specific scalar instructions. | |
struct llvm::VPTransformState::DataState | Data |
struct llvm::VPTransformState::CFGState | CFG |
LoopInfo * | LI |
Hold a pointer to LoopInfo to register new basic blocks in the loop. | |
IRBuilderBase & | Builder |
Hold a reference to the IRBuilder used to generate output IR code. | |
InnerLoopVectorizer * | ILV |
Hold a pointer to InnerLoopVectorizer to reuse its IR generation methods. | |
VPlan * | Plan |
Pointer to the VPlan code is generated for. | |
Loop * | CurrentVectorLoop = nullptr |
The loop object for the current parent region, or nullptr. | |
LoopVersioning * | LVer = nullptr |
LoopVersioning. | |
DenseMap< const SCEV *, Value * > | ExpandedSCEVs |
Map SCEVs to their expanded values. | |
VPTypeAnalysis | TypeAnalysis |
VPlan-based type analysis. | |
VPTransformState holds information passed down when "executing" a VPlan, needed for generating the output IR.
VPTransformState::VPTransformState | ( | ElementCount | VF, |
unsigned | UF, | ||
LoopInfo * | LI, | ||
DominatorTree * | DT, | ||
IRBuilderBase & | Builder, | ||
InnerLoopVectorizer * | ILV, | ||
VPlan * | Plan, | ||
LLVMContext & | Ctx | ||
) |
void VPTransformState::addMetadata | ( | Value * | To, |
Instruction * | From | ||
) |
Add metadata from one instruction to another.
This includes both the original MDs from From
and additional ones (
Definition at line 374 of file VPlan.cpp.
References addNewMetadata(), From, and llvm::propagateMetadata().
Referenced by llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenLoadEVLRecipe::execute(), llvm::VPWidenStoreRecipe::execute(), and llvm::VPWidenStoreEVLRecipe::execute().
void VPTransformState::addNewMetadata | ( | Instruction * | To, |
const Instruction * | Orig | ||
) |
Add additional metadata to To
that was not present on Orig
.
Currently this is used to add the noalias annotations based on the inserted memchecks. Use this for instructions that are cloned into the vector loop.
Definition at line 366 of file VPlan.cpp.
References llvm::LoopVersioning::annotateInstWithNoAlias(), and LVer.
Referenced by addMetadata(), and llvm::InnerLoopVectorizer::scalarizeInstruction().
Value * VPTransformState::get | ( | VPValue * | Def, |
const VPIteration & | Instance | ||
) |
Get the generated Value for a given VPValue and given Part and Lane.
Definition at line 232 of file VPlan.cpp.
References assert(), Builder, llvm::IRBuilderBase::CreateExtractElement(), Data, llvm::VPLane::getFirstLane(), hasScalarValue(), hasVectorValue(), Instance, llvm::vputils::isUniformAfterVectorization(), llvm::VPTransformState::DataState::PerPartOutput, llvm::VPTransformState::DataState::PerPartScalars, and VF.
Get the generated vector Value for a given VPValue Def
and a given Part
if IsScalar
is false, otherwise return the generated scalar for Part
.
\See set.
Definition at line 259 of file VPlan.cpp.
References assert(), B, Builder, CFG, llvm::IRBuilderBase::CreateVectorSplat(), Data, llvm::VectorType::get(), llvm::PoisonValue::get(), get(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::VPBlockBase::getSinglePredecessor(), llvm::BasicBlock::getTerminator(), llvm::VPlan::getVectorLoopRegion(), hasScalarValue(), hasVectorValue(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::isScalable(), llvm::ElementCount::isScalar(), llvm::vputils::isUniformAfterVectorization(), llvm::vputils::onlyFirstLaneUsed(), packScalarIntoVectorValue(), llvm::VPTransformState::DataState::PerPartOutput, llvm::VPTransformState::DataState::PerPartScalars, Plan, llvm::IRBuilderBase::restoreIP(), llvm::IRBuilderBase::saveIP(), set(), llvm::IRBuilderBase::SetInsertPoint(), VF, and llvm::VPTransformState::CFGState::VPBB2IRBB.
Referenced by createAndCollectMergePhiForReduction(), llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPScalarCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPWidenPHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReductionEVLRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenLoadEVLRecipe::execute(), llvm::VPWidenStoreRecipe::execute(), llvm::VPWidenStoreEVLRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPEVLBasedIVPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPlan::execute(), llvm::LoopVectorizationPlanner::executePlan(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::VPLiveOut::fixPhi(), llvm::InnerLoopVectorizer::fixupIVUsers(), get(), packScalarIntoVectorValue(), and llvm::InnerLoopVectorizer::scalarizeInstruction().
|
inline |
Definition at line 295 of file VPlan.h.
References Data, I, Instance, llvm::VPTransformState::DataState::PerPartScalars, and VF.
Referenced by llvm::VPPredInstPHIRecipe::execute(), and get().
Definition at line 289 of file VPlan.h.
References Data, I, and llvm::VPTransformState::DataState::PerPartOutput.
Referenced by llvm::VPPredInstPHIRecipe::execute(), and get().
void VPTransformState::packScalarIntoVectorValue | ( | VPValue * | Def, |
const VPIteration & | Instance | ||
) |
Construct the vector value of a scalarized value V
one lane at a time.
Definition at line 406 of file VPlan.cpp.
References Builder, llvm::IRBuilderBase::CreateInsertElement(), get(), Instance, set(), and VF.
Referenced by llvm::VPReplicateRecipe::execute(), and get().
|
inline |
Reset an existing vector value for Def
and a given Part
.
Definition at line 322 of file VPlan.h.
References assert(), Data, and llvm::VPTransformState::DataState::PerPartOutput.
Referenced by llvm::VPPredInstPHIRecipe::execute().
|
inline |
|
inline |
Set the generated vector Value for a given VPValue and a given Part, if IsScalar
is false.
If IsScalar
is true, set the scalar in (Part, 0).
Definition at line 307 of file VPlan.h.
References assert(), Data, llvm::ElementCount::isScalar(), llvm::VPTransformState::DataState::PerPartOutput, set(), UF, and VF.
Referenced by llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPScalarCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPWidenPHIRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReductionEVLRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenLoadEVLRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPCanonicalIVPHIRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPEVLBasedIVPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), get(), packScalarIntoVectorValue(), llvm::InnerLoopVectorizer::scalarizeInstruction(), and set().
void VPTransformState::setDebugLocFrom | ( | DebugLoc | DL | ) |
Set the debug location in the builder using the debug location DL
.
Definition at line 385 of file VPlan.cpp.
References Builder, llvm::DILocation::cloneByMultiplyingDuplicationFactor(), llvm::dbgs(), DL, llvm::EnableFSDiscriminator, llvm::IRBuilderBase::GetInsertBlock(), llvm::details::FixedOrScalableQuantity< LeafTy, ValueTy >::getKnownMinValue(), llvm::BasicBlock::getParent(), LLVM_DEBUG, llvm::IRBuilderBase::SetCurrentDebugLocation(), llvm::Function::shouldEmitDebugInfoForProfiling(), UF, and VF.
Referenced by llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenLoadEVLRecipe::execute(), llvm::VPWidenStoreRecipe::execute(), llvm::VPWidenStoreEVLRecipe::execute(), and llvm::InnerLoopVectorizer::scalarizeInstruction().
IRBuilderBase& llvm::VPTransformState::Builder |
Hold a reference to the IRBuilder used to generate output IR code.
Definition at line 409 of file VPlan.h.
Referenced by llvm::VPlan::createInitialVPlan(), llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPWidenPHIRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReductionEVLRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenLoadEVLRecipe::execute(), llvm::VPWidenStoreRecipe::execute(), llvm::VPWidenStoreEVLRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPEVLBasedIVPHIRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPlan::execute(), llvm::VPBasicBlock::execute(), llvm::VPIRBasicBlock::execute(), llvm::LoopVectorizationPlanner::executePlan(), get(), packScalarIntoVectorValue(), llvm::VPlan::prepareToExecute(), llvm::InnerLoopVectorizer::scalarizeInstruction(), and setDebugLocFrom().
struct llvm::VPTransformState::CFGState llvm::VPTransformState::CFG |
Referenced by llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPCanonicalIVPHIRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPEVLBasedIVPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPlan::execute(), llvm::VPBasicBlock::execute(), llvm::VPIRBasicBlock::execute(), llvm::VPRegionBlock::execute(), llvm::LoopVectorizationPlanner::executePlan(), llvm::VPBasicBlock::executeRecipes(), llvm::InnerLoopVectorizer::fixNonInductionPHIs(), llvm::VPLiveOut::fixPhi(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), get(), and llvm::VPlan::prepareToExecute().
Loop* llvm::VPTransformState::CurrentVectorLoop = nullptr |
The loop object for the current parent region, or nullptr.
Definition at line 418 of file VPlan.h.
Referenced by llvm::VPReductionPHIRecipe::execute(), llvm::VPBasicBlock::execute(), and llvm::VPRegionBlock::execute().
struct llvm::VPTransformState::DataState llvm::VPTransformState::Data |
Referenced by get(), hasScalarValue(), hasVectorValue(), reset(), and set().
Map SCEVs to their expanded values.
Populated when executing VPExpandSCEVRecipes.
Definition at line 429 of file VPlan.h.
Referenced by llvm::VPExpandSCEVRecipe::execute(), and llvm::LoopVectorizationPlanner::executePlan().
InnerLoopVectorizer* llvm::VPTransformState::ILV |
Hold a pointer to InnerLoopVectorizer to reuse its IR generation methods.
Definition at line 412 of file VPlan.h.
Referenced by llvm::VPReplicateRecipe::execute().
std::optional<VPIteration> llvm::VPTransformState::Instance |
Hold the indices to generate specific scalar instructions.
Null indicates that all instances are to be generated, using either scalar or vector instructions.
Definition at line 267 of file VPlan.h.
Referenced by llvm::VPInstruction::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReductionEVLRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPBranchOnMaskRecipe::execute(), llvm::VPPredInstPHIRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPDerivedIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPBasicBlock::execute(), llvm::VPRegionBlock::execute(), get(), hasScalarValue(), packScalarIntoVectorValue(), reset(), and set().
LoopInfo* llvm::VPTransformState::LI |
Hold a pointer to LoopInfo to register new basic blocks in the loop.
Definition at line 406 of file VPlan.h.
Referenced by llvm::VPBasicBlock::execute(), and llvm::VPRegionBlock::execute().
LoopVersioning* llvm::VPTransformState::LVer = nullptr |
It's only set up (non-null) if memchecks were used.
This is currently only used to add no-alias metadata based on the memchecks. The actually versioning is performed manually.
Definition at line 425 of file VPlan.h.
Referenced by addNewMetadata(), and llvm::LoopVectorizationPlanner::executePlan().
VPlan* llvm::VPTransformState::Plan |
Pointer to the VPlan code is generated for.
Definition at line 415 of file VPlan.h.
Referenced by llvm::LoopVectorizationPlanner::buildVPlans(), llvm::VPlan::createInitialVPlan(), llvm::VPlanPrinter::dump(), llvm::InnerLoopVectorizer::fixVectorizedLoop(), get(), llvm::vputils::getOrCreateVPValueForSCEVExpr(), llvm::LoopVectorizationPlanner::getPlanFor(), llvm::vputils::isHeaderMask(), llvm::LoopVectorizationPlanner::printPlans(), and llvm::VPInterleavedAccessInfo::VPInterleavedAccessInfo().
VPTypeAnalysis llvm::VPTransformState::TypeAnalysis |
VPlan-based type analysis.
Definition at line 432 of file VPlan.h.
Referenced by llvm::VPWidenRecipe::execute(), and llvm::InnerLoopVectorizer::scalarizeInstruction().
unsigned llvm::VPTransformState::UF |
Definition at line 262 of file VPlan.h.
Referenced by createAndCollectMergePhiForReduction(), llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPScalarCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenSelectRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPBlendRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReductionEVLRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenLoadEVLRecipe::execute(), llvm::VPWidenStoreRecipe::execute(), llvm::VPWidenStoreEVLRecipe::execute(), llvm::VPExpandSCEVRecipe::execute(), llvm::VPCanonicalIVPHIRecipe::execute(), llvm::VPActiveLaneMaskPHIRecipe::execute(), llvm::VPEVLBasedIVPHIRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPlan::execute(), llvm::VPRegionBlock::execute(), llvm::LoopVectorizationPlanner::executePlan(), llvm::VPlan::getName(), llvm::VPlan::prepareToExecute(), set(), and setDebugLocFrom().
ElementCount llvm::VPTransformState::VF |
The chosen Vectorization and Unroll Factors of the loop being vectorized.
Definition at line 261 of file VPlan.h.
Referenced by llvm::LoopVectorizationPlanner::buildVPlans(), llvm::VPlan::cost(), llvm::VPBasicBlock::cost(), llvm::VPRegionBlock::cost(), llvm::VPInstruction::execute(), llvm::VPWidenRecipe::execute(), llvm::VPWidenCastRecipe::execute(), llvm::VPWidenCallRecipe::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPVectorPointerRecipe::execute(), llvm::VPWidenIntOrFpInductionRecipe::execute(), llvm::VPWidenPointerInductionRecipe::execute(), llvm::VPFirstOrderRecurrencePHIRecipe::execute(), llvm::VPReductionPHIRecipe::execute(), llvm::VPInterleaveRecipe::execute(), llvm::VPReductionRecipe::execute(), llvm::VPReductionEVLRecipe::execute(), llvm::VPReplicateRecipe::execute(), llvm::VPWidenLoadRecipe::execute(), llvm::VPWidenLoadEVLRecipe::execute(), llvm::VPWidenStoreEVLRecipe::execute(), llvm::VPWidenCanonicalIVRecipe::execute(), llvm::VPScalarIVStepsRecipe::execute(), llvm::VPlan::execute(), llvm::VPRegionBlock::execute(), llvm::LoopVectorizationPlanner::executePlan(), get(), llvm::VPlan::getName(), llvm::LoopVectorizationPlanner::getPlanFor(), hasScalarValue(), packScalarIntoVectorValue(), llvm::VPlan::prepareToExecute(), reset(), set(), and setDebugLocFrom().