|
LLVM 23.0.0git
|
InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF). More...
Public Member Functions | |
| InnerLoopVectorizer (Loop *OrigLoop, PredicatedScalarEvolution &PSE, LoopInfo *LI, DominatorTree *DT, const TargetTransformInfo *TTI, AssumptionCache *AC, ElementCount VecWidth, unsigned UnrollFactor, LoopVectorizationCostModel *CM, GeneratedRTChecks &RTChecks, VPlan &Plan) | |
| virtual | ~InnerLoopVectorizer ()=default |
| virtual BasicBlock * | createVectorizedLoopSkeleton () |
| Creates a basic block for the scalar preheader. | |
| void | fixVectorizedLoop (VPTransformState &State) |
| Fix the vectorized code, taking care of header phi's, and more. | |
| void | fixNonInductionPHIs (VPTransformState &State) |
Fix the non-induction PHIs in Plan. | |
Protected Member Functions | |
| BasicBlock * | createScalarPreheader (StringRef Prefix) |
Create and return a new IR basic block for the scalar preheader whose name is prefixed with Prefix. | |
| virtual void | printDebugTracesAtStart () |
| Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested. | |
| virtual void | printDebugTracesAtEnd () |
Protected Attributes | |
| Loop * | OrigLoop |
| The original loop. | |
| PredicatedScalarEvolution & | PSE |
| A wrapper around ScalarEvolution used to add runtime SCEV checks. | |
| LoopInfo * | LI |
| Loop Info. | |
| DominatorTree * | DT |
| Dominator Tree. | |
| const TargetTransformInfo * | TTI |
| Target Transform Info. | |
| AssumptionCache * | AC |
| Assumption Cache. | |
| ElementCount | VF |
| The vectorization SIMD factor to use. | |
| unsigned | UF |
| The vectorization unroll factor to use. | |
| IRBuilder | Builder |
| The builder that we use. | |
| LoopVectorizationCostModel * | Cost |
| The profitablity analysis. | |
| GeneratedRTChecks & | RTChecks |
| Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable. | |
| VPlan & | Plan |
| VPBasicBlock * | VectorPHVPBB |
The vector preheader block of Plan, used as target for check blocks introduced during skeleton creation. | |
Friends | |
| class | LoopVectorizationPlanner |
InnerLoopVectorizer vectorizes loops which contain only one basic block to a specified vectorization factor (VF).
This class performs the widening of scalars into vectors, or multiple scalars. This class also implements the following features:
Definition at line 546 of file LoopVectorize.cpp.
|
inline |
Definition at line 548 of file LoopVectorize.cpp.
References AC, Builder, llvm::cast(), Cost, DT, LI, OrigLoop, Plan, PSE, RTChecks, TTI, UF, VectorPHVPBB, and VF.
Referenced by llvm::InnerLoopAndEpilogueVectorizer::InnerLoopAndEpilogueVectorizer().
|
virtualdefault |
|
protected |
Create and return a new IR basic block for the scalar preheader whose name is prefixed with Prefix.
Definition at line 2331 of file LoopVectorize.cpp.
References assert(), Cost, DT, llvm::BasicBlock::getTerminator(), LI, OrigLoop, llvm::SplitBlock(), and VF.
Referenced by llvm::EpilogueVectorizerEpilogueLoop::createVectorizedLoopSkeleton(), and createVectorizedLoopSkeleton().
|
virtual |
Creates a basic block for the scalar preheader.
Both EpilogueVectorizerMainLoop and EpilogueVectorizerEpilogueLoop overwrite the method to create additional blocks and checks needed for epilogue vectorization.
Reimplemented in llvm::EpilogueVectorizerEpilogueLoop.
Definition at line 2369 of file LoopVectorize.cpp.
References createScalarPreheader(), and llvm::BasicBlock::getSinglePredecessor().
Referenced by llvm::LoopVectorizationPlanner::executePlan().
| void InnerLoopVectorizer::fixNonInductionPHIs | ( | VPTransformState & | State | ) |
Fix the non-induction PHIs in Plan.
Definition at line 2516 of file LoopVectorize.cpp.
References llvm::PHINode::addIncoming(), llvm::VPBlockUtils::blocksOnly(), Builder, llvm::cast(), llvm::dyn_cast(), llvm::VPPhiAccessors::incoming_values_and_blocks(), P, Plan, and llvm::vp_depth_first_shallow().
Referenced by fixVectorizedLoop().
| void InnerLoopVectorizer::fixVectorizedLoop | ( | VPTransformState & | State | ) |
Fix the vectorized code, taking care of header phi's, and more.
Definition at line 2499 of file LoopVectorize.cpp.
References fixNonInductionPHIs(), llvm::vputils::getFirstLoopHeader(), and legacyCSE().
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
inlineprotectedvirtual |
Reimplemented in llvm::EpilogueVectorizerEpilogueLoop, and llvm::EpilogueVectorizerMainLoop.
Definition at line 584 of file LoopVectorize.cpp.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
inlineprotectedvirtual |
Allow subclasses to override and print debug traces before/after vplan execution, when trace information is requested.
Reimplemented in llvm::EpilogueVectorizerEpilogueLoop, and llvm::EpilogueVectorizerMainLoop.
Definition at line 583 of file LoopVectorize.cpp.
Referenced by llvm::LoopVectorizationPlanner::executePlan().
|
friend |
Definition at line 575 of file LoopVectorize.cpp.
References LoopVectorizationPlanner.
Referenced by LoopVectorizationPlanner.
|
protected |
Assumption Cache.
Definition at line 604 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop(), llvm::EpilogueVectorizerMainLoop::EpilogueVectorizerMainLoop(), llvm::LoopVectorizationPlanner::executePlan(), llvm::InnerLoopAndEpilogueVectorizer::InnerLoopAndEpilogueVectorizer(), and InnerLoopVectorizer().
|
protected |
The builder that we use.
Definition at line 615 of file LoopVectorize.cpp.
Referenced by llvm::LoopVectorizationPlanner::executePlan(), fixNonInductionPHIs(), and InnerLoopVectorizer().
|
protected |
The profitablity analysis.
Definition at line 620 of file LoopVectorize.cpp.
Referenced by createScalarPreheader(), and InnerLoopVectorizer().
|
protected |
Dominator Tree.
Definition at line 598 of file LoopVectorize.cpp.
Referenced by createScalarPreheader(), llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop(), llvm::EpilogueVectorizerMainLoop::EpilogueVectorizerMainLoop(), llvm::InnerLoopAndEpilogueVectorizer::InnerLoopAndEpilogueVectorizer(), and InnerLoopVectorizer().
|
protected |
Loop Info.
Definition at line 595 of file LoopVectorize.cpp.
Referenced by createScalarPreheader(), llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop(), llvm::EpilogueVectorizerMainLoop::EpilogueVectorizerMainLoop(), llvm::InnerLoopAndEpilogueVectorizer::InnerLoopAndEpilogueVectorizer(), and InnerLoopVectorizer().
|
protected |
The original loop.
Definition at line 587 of file LoopVectorize.cpp.
Referenced by createScalarPreheader(), llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop(), llvm::EpilogueVectorizerMainLoop::EpilogueVectorizerMainLoop(), llvm::InnerLoopAndEpilogueVectorizer::InnerLoopAndEpilogueVectorizer(), InnerLoopVectorizer(), llvm::EpilogueVectorizerEpilogueLoop::printDebugTracesAtEnd(), and llvm::EpilogueVectorizerMainLoop::printDebugTracesAtEnd().
|
protected |
Definition at line 626 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerEpilogueLoop::createVectorizedLoopSkeleton(), llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop(), llvm::EpilogueVectorizerMainLoop::EpilogueVectorizerMainLoop(), fixNonInductionPHIs(), llvm::InnerLoopAndEpilogueVectorizer::InnerLoopAndEpilogueVectorizer(), and InnerLoopVectorizer().
|
protected |
A wrapper around ScalarEvolution used to add runtime SCEV checks.
Applies dynamic knowledge to simplify SCEV expressions and converts them to a more usable form.
Definition at line 592 of file LoopVectorize.cpp.
Referenced by llvm::EpilogueVectorizerEpilogueLoop::EpilogueVectorizerEpilogueLoop(), llvm::EpilogueVectorizerMainLoop::EpilogueVectorizerMainLoop(), llvm::InnerLoopAndEpilogueVectorizer::InnerLoopAndEpilogueVectorizer(), and InnerLoopVectorizer().
|
protected |
Structure to hold information about generated runtime checks, responsible for cleaning the checks, if vectorization turns out unprofitable.
Definition at line 624 of file LoopVectorize.cpp.
Referenced by llvm::LoopVectorizationPlanner::executePlan(), and InnerLoopVectorizer().
|
protected |
Target Transform Info.
Definition at line 601 of file LoopVectorize.cpp.
Referenced by llvm::InnerLoopAndEpilogueVectorizer::InnerLoopAndEpilogueVectorizer(), and InnerLoopVectorizer().
|
protected |
The vectorization unroll factor to use.
Each scalar is vectorized to this many different vector instructions.
Definition at line 612 of file LoopVectorize.cpp.
Referenced by InnerLoopVectorizer().
|
protected |
The vector preheader block of Plan, used as target for check blocks introduced during skeleton creation.
Definition at line 630 of file LoopVectorize.cpp.
Referenced by InnerLoopVectorizer().
|
protected |
The vectorization SIMD factor to use.
Each vector will have this many vector elements.
Definition at line 608 of file LoopVectorize.cpp.
Referenced by createScalarPreheader(), and InnerLoopVectorizer().