Read tokens in infinite loop

Question

I have a code that reads tokens a text. I’m using Visual Studio 2013 in an extensibility project.

When reading the text through scanner and collect the tokens, the program enters an infinite loop:

public class TestScanner : IScanner
{
    private TestSource m_source;
    private int m_offset;

    public TestScanner(LanguageService service, IVsTextLines buffer)
    {
        this.m_source = new TestSource(service, buffer, new TestColorizer(service, buffer, this));
    }

    public bool ScanTokenAndProvideInfoAboutIt(TokenInfo tokenInfo, ref int state)
    {
        bool bFound = false;
        if (tokenInfo != null)
        {
            bFound = this.m_source.GetNextToken(m_offset, tokenInfo, ref state);
            if (bFound)
            {
                m_offset = tokenInfo.EndIndex + 1;
            }
        }
        return bFound;
    }

    public void SetSource(string source, int offset)
    {
        m_source.SetText(source);
        m_offset = offset;
    }
}

I used base this link to check the usefulness of the interface:

Iscanner.Scantokenandprovideinfoaboutit Method

Returns true if a token was Parsed from the Current line and information returned; otherwise, Returns false indicating no more tokens on the Current line.

I am not using the class directly in the code, I implement the interface through the class TestScanner and VS uses it to read the code (although I have the Colorizer, but without any override).

What I hope will happen:

• SetSource send me the lines of code of the text, I recognize the tokens and, when there is no more, jump to next line, as it says in the documentation, just return false in ScanTokenAndProvideInfoAboutIt.

What happens

• In fact, ScanTokenAndProvideInfoAboutIt returns true the first time, and the value of m_offset is modified, when passing the second time in the method, m_offset has the value of the end of the line (13 specifically, using the string "using System;")
• Like m_offset is equal to or greater than the string this.m_source.Getnexttoken returns false, sequentially in ScanTokenAndProvideInfoAboutIt, that is, I state that I have completed the collection and recognition of tokens.
• However SetSource sends me the same string that I did the last collection of tokens, thus entering an infinite loop.

---

I’d like to know why SetSource repeats the call with the same argument (same line), even if I returned false. It would be a bug of Visual Studio?

What logic do I need to use to say that I just recognized the Line Tokens?

Note: If you need more code just comment!

Answer 1

I ended up finding the source of the infinite loop.

m_source.SetText(source);

This generates a future call to the parser himself TestScanner, was not a directly recursive call, which made localization even more difficult.