clang  3.9.0
Public Member Functions | Public Attributes | List of all members
clang::format::FormatToken Struct Reference

A wrapper around a Token storing information about the whitespace characters preceding it. More...

#include <FormatToken.h>

Collaboration diagram for clang::format::FormatToken:
[legend]

Public Member Functions

 FormatToken ()
 
bool is (tok::TokenKind Kind) const
 
bool is (TokenType TT) const
 
bool is (const IdentifierInfo *II) const
 
bool is (tok::PPKeywordKind Kind) const
 
template<typename A , typename B >
bool isOneOf (A K1, B K2) const
 
template<typename A , typename B , typename... Ts>
bool isOneOf (A K1, B K2, Ts...Ks) const
 
template<typename T >
bool isNot (T Kind) const
 
template<typename A , typename... Ts>
bool startsSequence (A K1, Ts...Tokens) const
 true if this token starts a sequence with the given tokens in order, following the Next pointers, ignoring comments. More...
 
template<typename A , typename... Ts>
bool endsSequence (A K1, Ts...Tokens) const
 true if this token ends a sequence with the given tokens in order, following the Previous pointers, ignoring comments. More...
 
bool isStringLiteral () const
 
bool isObjCAtKeyword (tok::ObjCKeywordKind Kind) const
 
bool isAccessSpecifier (bool ColonRequired=true) const
 
bool isSimpleTypeSpecifier () const
 Determine whether the token is a simple-type-specifier. More...
 
bool isObjCAccessSpecifier () const
 
bool opensScope () const
 Returns whether Tok is ([{ or a template opening <. More...
 
bool closesScope () const
 Returns whether Tok is )]} or a template closing >. More...
 
bool isMemberAccess () const
 Returns true if this is a "." or "->" accessing a member. More...
 
bool isUnaryOperator () const
 
bool isBinaryOperator () const
 
bool isTrailingComment () const
 
bool isFunctionLikeKeyword () const
 Returns true if this is a keyword that can be used like a function call (e.g. More...
 
SourceLocation getStartOfNonWhitespace () const
 Returns actual token start location without leading escaped newlines and whitespace. More...
 
prec::Level getPrecedence () const
 
FormatTokengetPreviousNonComment () const
 Returns the previous token ignoring comments. More...
 
const FormatTokengetNextNonComment () const
 Returns the next token ignoring comments. More...
 
bool opensBlockOrBlockTypeList (const FormatStyle &Style) const
 Returns true if this tokens starts a block-type list, i.e. More...
 
bool closesBlockOrBlockTypeList (const FormatStyle &Style) const
 Same as opensBlockOrBlockTypeList, but for the closing token. More...
 

Public Attributes

Token Tok
 The Token. More...
 
unsigned NewlinesBefore = 0
 The number of newlines immediately before the Token. More...
 
bool HasUnescapedNewline = false
 Whether there is at least one unescaped newline before the Token. More...
 
SourceRange WhitespaceRange
 The range of the whitespace immediately preceding the Token. More...
 
unsigned LastNewlineOffset = 0
 The offset just past the last '
' in this token's leading whitespace (relative to WhiteSpaceStart). More...
 
unsigned ColumnWidth = 0
 The width of the non-whitespace parts of the token (or its first line for multi-line tokens) in columns. More...
 
unsigned LastLineColumnWidth = 0
 Contains the width in columns of the last line of a multi-line token. More...
 
bool IsMultiline = false
 Whether the token text contains newlines (escaped or not). More...
 
bool IsFirst = false
 Indicates that this is the first token of the file. More...
 
bool MustBreakBefore = false
 Whether there must be a line break before this token. More...
 
StringRef TokenText
 The raw text of the token. More...
 
bool IsUnterminatedLiteral = 0
 Set to true if this token is an unterminated literal. More...
 
BraceBlockKind BlockKind = BK_Unknown
 Contains the kind of block if this token is a brace. More...
 
TokenType Type = TT_Unknown
 
unsigned SpacesRequiredBefore = 0
 The number of spaces that should be inserted before this token. More...
 
bool CanBreakBefore = false
 true if it is allowed to break before this token. More...
 
bool ClosesTemplateDeclaration = false
 true if this is the ">" of "template<..>". More...
 
unsigned ParameterCount = 0
 Number of parameters, if this is "(", "[" or "<". More...
 
unsigned BlockParameterCount = 0
 Number of parameters that are nested blocks, if this is "(", "[" or "<". More...
 
tok::TokenKind ParentBracket = tok::unknown
 If this is a bracket ("<", "(", "[" or "{"), contains the kind of the surrounding bracket. More...
 
std::unique_ptr< TokenRoleRole
 A token can have a special role that can carry extra information about the token's formatting. More...
 
ParameterPackingKind PackingKind = PPK_Inconclusive
 If this is an opening parenthesis, how are the parameters packed? More...
 
unsigned TotalLength = 0
 The total length of the unwrapped line up to and including this token. More...
 
unsigned OriginalColumn = 0
 The original 0-based column of this token, including expanded tabs. More...
 
unsigned UnbreakableTailLength = 0
 The length of following tokens until the next natural split point, or the next token that can be broken. More...
 
unsigned BindingStrength = 0
 The binding strength of a token. More...
 
unsigned NestingLevel = 0
 The nesting level of this token, i.e. More...
 
unsigned SplitPenalty = 0
 Penalty for inserting a line break before this token. More...
 
unsigned LongestObjCSelectorName = 0
 If this is the first ObjC selector name in an ObjC method definition or call, this contains the length of the longest name. More...
 
SmallVector< prec::Level, 4 > FakeLParens
 Stores the number of required fake parentheses and the corresponding operator precedence. More...
 
unsigned FakeRParens = 0
 Insert this many fake ) after this token for correct indentation. More...
 
bool StartsBinaryExpression = false
 true if this token starts a binary expression, i.e. More...
 
bool EndsBinaryExpression = false
 true if this token ends a binary expression. More...
 
unsigned OperatorIndex = 0
 Is this is an operator (or "."/"->") in a sequence of operators with the same precedence, contains the 0-based operator index. More...
 
FormatTokenNextOperator = nullptr
 If this is an operator (or "."/"->") in a sequence of operators with the same precedence, points to the next operator. More...
 
bool PartOfMultiVariableDeclStmt = false
 Is this token part of a DeclStmt defining multiple variables? More...
 
FormatTokenMatchingParen = nullptr
 If this is a bracket, this points to the matching one. More...
 
FormatTokenPrevious = nullptr
 The previous token in the unwrapped line. More...
 
FormatTokenNext = nullptr
 The next token in the unwrapped line. More...
 
SmallVector< AnnotatedLine *, 1 > Children
 If this token starts a block, this contains all the unwrapped lines in it. More...
 
FormatDecision Decision = FD_Unformatted
 Stores the formatting decision for the token once it was made. More...
 
bool Finalized = false
 If true, this token has been fully formatted (indented and potentially re-formatted inside), and we do not allow further formatting changes. More...
 

Detailed Description

A wrapper around a Token storing information about the whitespace characters preceding it.

Definition at line 113 of file FormatToken.h.

Constructor & Destructor Documentation

clang::format::FormatToken::FormatToken ( )
inline

Definition at line 114 of file FormatToken.h.

Member Function Documentation

bool clang::format::FormatToken::closesBlockOrBlockTypeList ( const FormatStyle Style) const
inline

Same as opensBlockOrBlockTypeList, but for the closing token.

Definition at line 438 of file FormatToken.h.

References MatchingParen, and opensBlockOrBlockTypeList().

Referenced by clang::format::ContinuationIndenter::canBreak(), and clang::format::ContinuationIndenter::mustBreak().

bool clang::format::FormatToken::closesScope ( ) const
inline

Returns whether Tok is )]} or a template closing >.

Definition at line 341 of file FormatToken.h.

References isOneOf().

Referenced by clang::format::startsSegmentOfBuilderTypeCall().

template<typename A , typename... Ts>
bool clang::format::FormatToken::endsSequence ( K1,
Ts...  Tokens 
) const
inline

true if this token ends a sequence with the given tokens in order, following the Previous pointers, ignoring comments.

Definition at line 310 of file FormatToken.h.

References Tokens.

Referenced by clang::format::AnnotatedLine::endsWith().

const FormatToken* clang::format::FormatToken::getNextNonComment ( ) const
inline

Returns the next token ignoring comments.

Definition at line 421 of file FormatToken.h.

References is(), Next, and Tok.

Referenced by clang::format::CommaSeparatedList::precomputeFormattingInfos().

prec::Level clang::format::FormatToken::getPrecedence ( ) const
inline
FormatToken* clang::format::FormatToken::getPreviousNonComment ( ) const
inline

Returns the previous token ignoring comments.

Definition at line 413 of file FormatToken.h.

References is(), Previous, and Tok.

Referenced by clang::format::CommaSeparatedList::formatAfterToken(), and clang::format::CommaSeparatedList::precomputeFormattingInfos().

SourceLocation clang::format::FormatToken::getStartOfNonWhitespace ( ) const
inline

Returns actual token start location without leading escaped newlines and whitespace.

This can be different to Tok.getLocation(), which includes leading escaped newlines.

Definition at line 404 of file FormatToken.h.

References clang::SourceRange::getEnd(), and WhitespaceRange.

Referenced by clang::format::WhitespaceManager::replaceWhitespaceInToken().

bool clang::format::FormatToken::is ( tok::TokenKind  Kind) const
inline
bool clang::format::FormatToken::is ( TokenType  TT) const
inline

Definition at line 283 of file FormatToken.h.

bool clang::format::FormatToken::is ( const IdentifierInfo II) const
inline

Definition at line 284 of file FormatToken.h.

References clang::Token::getIdentifierInfo(), and Tok.

bool clang::format::FormatToken::is ( tok::PPKeywordKind  Kind) const
inline
bool clang::format::FormatToken::isAccessSpecifier ( bool  ColonRequired = true) const
inline

Definition at line 320 of file FormatToken.h.

References is(), isOneOf(), and Next.

bool clang::format::FormatToken::isBinaryOperator ( ) const
inline

Definition at line 369 of file FormatToken.h.

References clang::prec::Comma, and getPrecedence().

bool clang::format::FormatToken::isFunctionLikeKeyword ( ) const
inline

Returns true if this is a keyword that can be used like a function call (e.g.

sizeof, typeid, ...).

Definition at line 381 of file FormatToken.h.

References clang::Token::getKind(), and Tok.

bool clang::format::FormatToken::isMemberAccess ( ) const
inline

Returns true if this is a "." or "->" accessing a member.

Definition at line 347 of file FormatToken.h.

References isOneOf().

Referenced by clang::format::ContinuationIndenter::canBreak(), clang::format::ContinuationIndenter::mustBreak(), and clang::format::startsSegmentOfBuilderTypeCall().

template<typename T >
bool clang::format::FormatToken::isNot ( Kind) const
inline
bool clang::format::FormatToken::isObjCAccessSpecifier ( ) const
inline

Definition at line 328 of file FormatToken.h.

References is(), isObjCAtKeyword(), and Next.

bool clang::format::FormatToken::isObjCAtKeyword ( tok::ObjCKeywordKind  Kind) const
inline

Definition at line 316 of file FormatToken.h.

References clang::Token::isObjCAtKeyword(), and Tok.

Referenced by isObjCAccessSpecifier().

template<typename A , typename B >
bool clang::format::FormatToken::isOneOf ( K1,
K2 
) const
inline
template<typename A , typename B , typename... Ts>
bool clang::format::FormatToken::isOneOf ( K1,
K2,
Ts...  Ks 
) const
inline

Definition at line 295 of file FormatToken.h.

References is(), and isOneOf().

bool clang::format::FormatToken::isSimpleTypeSpecifier ( ) const

Determine whether the token is a simple-type-specifier.

Definition at line 42 of file FormatToken.cpp.

References clang::Token::getKind(), and Tok.

bool clang::format::FormatToken::isStringLiteral ( ) const
inline

Definition at line 314 of file FormatToken.h.

References clang::Token::getKind(), clang::tok::isStringLiteral(), and Tok.

bool clang::format::FormatToken::isTrailingComment ( ) const
inline
bool clang::format::FormatToken::isUnaryOperator ( ) const
inline

Definition at line 353 of file FormatToken.h.

References clang::Token::getKind(), and Tok.

bool clang::format::FormatToken::opensBlockOrBlockTypeList ( const FormatStyle Style) const
inline

Returns true if this tokens starts a block-type list, i.e.

a list that should be indented with a block indent.

Definition at line 430 of file FormatToken.h.

References clang::format::BK_Block, BlockKind, clang::format::FormatStyle::Cpp11BracedListStyle, is(), and NestingLevel.

Referenced by closesBlockOrBlockTypeList().

bool clang::format::FormatToken::opensScope ( ) const
inline

Returns whether Tok is ([{ or a template opening <.

Definition at line 336 of file FormatToken.h.

References isOneOf().

Referenced by clang::format::TokenAnnotator::calculateFormattingInformation(), and clang::format::ContinuationIndenter::canBreak().

template<typename A , typename... Ts>
bool clang::format::FormatToken::startsSequence ( K1,
Ts...  Tokens 
) const
inline

true if this token starts a sequence with the given tokens in order, following the Next pointers, ignoring comments.

Definition at line 303 of file FormatToken.h.

References Tokens.

Referenced by clang::format::AnnotatedLine::startsWith().

Member Data Documentation

unsigned clang::format::FormatToken::BindingStrength = 0

The binding strength of a token.

This is a combined value of operator precedence, parenthesis nesting, etc.

Definition at line 217 of file FormatToken.h.

Referenced by clang::format::TokenAnnotator::calculateFormattingInformation().

BraceBlockKind clang::format::FormatToken::BlockKind = BK_Unknown
unsigned clang::format::FormatToken::BlockParameterCount = 0

Number of parameters that are nested blocks, if this is "(", "[" or "<".

Definition at line 189 of file FormatToken.h.

bool clang::format::FormatToken::CanBreakBefore = false
SmallVector<AnnotatedLine *, 1> clang::format::FormatToken::Children

If this token starts a block, this contains all the unwrapped lines in it.

Definition at line 272 of file FormatToken.h.

Referenced by clang::format::AnnotatedLine::AnnotatedLine(), clang::format::TokenAnnotator::calculateFormattingInformation(), and clang::format::AnnotatedLine::~AnnotatedLine().

bool clang::format::FormatToken::ClosesTemplateDeclaration = false

true if this is the ">" of "template<..>".

Definition at line 178 of file FormatToken.h.

Referenced by clang::format::ContinuationIndenter::mustBreak().

unsigned clang::format::FormatToken::ColumnWidth = 0

The width of the non-whitespace parts of the token (or its first line for multi-line tokens) in columns.

We need this to correctly measure number of columns a token spans.

Definition at line 139 of file FormatToken.h.

Referenced by clang::format::TokenAnnotator::calculateFormattingInformation(), clang::format::CodePointsBetween(), and clang::format::CommaSeparatedList::formatAfterToken().

FormatDecision clang::format::FormatToken::Decision = FD_Unformatted

Stores the formatting decision for the token once it was made.

Definition at line 275 of file FormatToken.h.

Referenced by clang::format::WhitespaceManager::replaceWhitespace().

bool clang::format::FormatToken::EndsBinaryExpression = false

true if this token ends a binary expression.

Definition at line 246 of file FormatToken.h.

Referenced by clang::format::ContinuationIndenter::mustBreak().

SmallVector<prec::Level, 4> clang::format::FormatToken::FakeLParens

Stores the number of required fake parentheses and the corresponding operator precedence.

If multiple fake parentheses start at a token, this vector stores them in reverse order, i.e. inner fake parenthesis first.

Definition at line 238 of file FormatToken.h.

unsigned clang::format::FormatToken::FakeRParens = 0

Insert this many fake ) after this token for correct indentation.

Definition at line 240 of file FormatToken.h.

bool clang::format::FormatToken::Finalized = false

If true, this token has been fully formatted (indented and potentially re-formatted inside), and we do not allow further formatting changes.

Definition at line 280 of file FormatToken.h.

Referenced by clang::format::WhitespaceManager::addUntouchableToken(), clang::format::WhitespaceManager::replaceWhitespace(), and clang::format::WhitespaceManager::replaceWhitespaceInToken().

bool clang::format::FormatToken::HasUnescapedNewline = false

Whether there is at least one unescaped newline before the Token.

Definition at line 127 of file FormatToken.h.

Referenced by clang::format::TokenAnnotator::calculateFormattingInformation(), and clang::format::CommaSeparatedList::precomputeFormattingInfos().

bool clang::format::FormatToken::IsFirst = false
bool clang::format::FormatToken::IsMultiline = false

Whether the token text contains newlines (escaped or not).

Definition at line 146 of file FormatToken.h.

Referenced by clang::format::TokenAnnotator::calculateFormattingInformation().

bool clang::format::FormatToken::IsUnterminatedLiteral = 0

Set to true if this token is an unterminated literal.

Definition at line 164 of file FormatToken.h.

unsigned clang::format::FormatToken::LastLineColumnWidth = 0

Contains the width in columns of the last line of a multi-line token.

Definition at line 143 of file FormatToken.h.

unsigned clang::format::FormatToken::LastNewlineOffset = 0

The offset just past the last '
' in this token's leading whitespace (relative to WhiteSpaceStart).

0 if there is no '
'.

Definition at line 134 of file FormatToken.h.

Referenced by clang::format::ContinuationIndenter::addTokenToState().

unsigned clang::format::FormatToken::LongestObjCSelectorName = 0

If this is the first ObjC selector name in an ObjC method definition or call, this contains the length of the longest name.

This being set to 0 means that the selectors should not be colon-aligned, e.g. because several of them are block-type.

Definition at line 231 of file FormatToken.h.

FormatToken* clang::format::FormatToken::MatchingParen = nullptr
bool clang::format::FormatToken::MustBreakBefore = false
unsigned clang::format::FormatToken::NestingLevel = 0
unsigned clang::format::FormatToken::NewlinesBefore = 0

The number of newlines immediately before the Token.

This can be used to determine what the user wrote in the original code and thereby e.g. leave an empty line between two function definitions.

Definition at line 123 of file FormatToken.h.

Referenced by clang::format::WhitespaceManager::addUntouchableToken(), clang::format::JavaScriptImportSorter::analyze(), clang::format::UnwrappedLineFormatter::format(), and isTrailingComment().

FormatToken* clang::format::FormatToken::Next = nullptr
FormatToken* clang::format::FormatToken::NextOperator = nullptr

If this is an operator (or "."/"->") in a sequence of operators with the same precedence, points to the next operator.

Definition at line 254 of file FormatToken.h.

Referenced by clang::format::getLengthToNextOperator().

unsigned clang::format::FormatToken::OperatorIndex = 0

Is this is an operator (or "."/"->") in a sequence of operators with the same precedence, contains the 0-based operator index.

Definition at line 250 of file FormatToken.h.

unsigned clang::format::FormatToken::OriginalColumn = 0

The original 0-based column of this token, including expanded tabs.

The configured TabWidth is used as tab width.

Definition at line 208 of file FormatToken.h.

Referenced by clang::format::WhitespaceManager::addUntouchableToken(), and clang::format::UnwrappedLineFormatter::format().

ParameterPackingKind clang::format::FormatToken::PackingKind = PPK_Inconclusive

If this is an opening parenthesis, how are the parameters packed?

Definition at line 200 of file FormatToken.h.

unsigned clang::format::FormatToken::ParameterCount = 0

Number of parameters, if this is "(", "[" or "<".

This is initialized to 1 as we don't need to distinguish functions with 0 parameters from functions with 1 parameter. Thus, we can simply count the number of commas.

Definition at line 185 of file FormatToken.h.

Referenced by clang::format::ContinuationIndenter::mustBreak().

tok::TokenKind clang::format::FormatToken::ParentBracket = tok::unknown

If this is a bracket ("<", "(", "[" or "{"), contains the kind of the surrounding bracket.

Definition at line 193 of file FormatToken.h.

bool clang::format::FormatToken::PartOfMultiVariableDeclStmt = false

Is this token part of a DeclStmt defining multiple variables?

Only set if Type == TT_StartOfName.

Definition at line 259 of file FormatToken.h.

FormatToken* clang::format::FormatToken::Previous = nullptr
std::unique_ptr<TokenRole> clang::format::FormatToken::Role

A token can have a special role that can carry extra information about the token's formatting.

Definition at line 197 of file FormatToken.h.

Referenced by clang::format::TokenAnnotator::calculateFormattingInformation(), and clang::format::AnnotatedLine::~AnnotatedLine().

unsigned clang::format::FormatToken::SpacesRequiredBefore = 0

The number of spaces that should be inserted before this token.

Definition at line 172 of file FormatToken.h.

Referenced by clang::format::TokenAnnotator::annotate(), and clang::format::TokenAnnotator::calculateFormattingInformation().

unsigned clang::format::FormatToken::SplitPenalty = 0

Penalty for inserting a line break before this token.

Definition at line 224 of file FormatToken.h.

Referenced by clang::format::TokenAnnotator::calculateFormattingInformation().

bool clang::format::FormatToken::StartsBinaryExpression = false

true if this token starts a binary expression, i.e.

has at least one fake l_paren with a precedence greater than prec::Unknown.

Definition at line 244 of file FormatToken.h.

Token clang::format::FormatToken::Tok
StringRef clang::format::FormatToken::TokenText
unsigned clang::format::FormatToken::TotalLength = 0
TokenType clang::format::FormatToken::Type = TT_Unknown
unsigned clang::format::FormatToken::UnbreakableTailLength = 0

The length of following tokens until the next natural split point, or the next token that can be broken.

Definition at line 212 of file FormatToken.h.

SourceRange clang::format::FormatToken::WhitespaceRange

The documentation for this struct was generated from the following files: