tags and huge external lib

[jfouche]

Hi Eran

I wanted to test boost.spirit... and as I defined boost in my include file for tags, it begans to parse it.
It's awfully long (I started to retag workspace 2 hours ago ).
I tried to remove boost from the includes path, but the current parsing continue.
The problem is that I can't kill codelite_indexer.exe in the Windows task manager. I'll have to wait...
I don't know if there is a solution to fix the 'to big lib' problem (tell CodeLite to stop retag if it takes to much time).
Just to let you know about this...

[eranif]

boost is not the problem. The problem is 2 files ... in boost: vector250 and vector100 each generates tens of thousands entries...
(both files are auto-generated)

On my machine, I have boost in the path, and it parses it un-noticable
But I am using the more standard headers: boost/shared_ptr.hpp and the like.

Can you give me an example of include files to add to my project so it will start parse like crazy?

EDIT: I tried to use the example code from wikipedia to boost.spirit and codelite parsed the files in < second
this is the code I was using:

Code: Select all

#include <boost/spirit.hpp>
#include <boost/spirit/actor.hpp>
#include <string>
#include <iostream>
 
using namespace std;
using namespace boost::spirit;
 
int main()
{
    string input;
 
    cout << "Input a line.\n";
    getline(cin, input);
 
    cout << "Got '" << input << "'.\n";
 
    unsigned count = 0;
 
 /*  
    Next line parses the input (input.c_str()),
        using a parser constructed with the following semantics
        (identation matches source for clarity):
 
     Zero or more occurrences of (
          literal string "cat" ( when matched, increment the counter "count" )
      or  any character (to move on finding the next occurrence of "cat")
     )
 */
     parse(input.c_str(),
        *(  str_p("cat") [ increment_a(count) ]
          | anychar_p
         ));
 /*
     The parser is constructed by the compiler using operator
     overloading and template matching, so the actual work is
     done within spirit::parse(), and the expression starting
     with * only initializes the rule object that the parse
     function uses.
  */
 
    // last, show results.
    cout << "The input had " << count
              << " occurrences of 'cat'\n";
    return 0;
}

Eran

[jfouche]

Hi Eran

Parsing is finished now, and the workspace database is 115 Mo (310 files and 76000 tags, which doesn't seems so huge).
Here is the file :

Code: Select all

#include <boost/config/warning_disable.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>

#include <iostream>
#include <string>
#include <complex>

///////////////////////////////////////////////////////////////////////////////
//  Our complex number parser/compiler
///////////////////////////////////////////////////////////////////////////////
//[tutorial_complex_number
namespace client
{
    template <typename Iterator>
    bool parse_complex(Iterator first, Iterator last, std::complex<double>& c)
    {
        using boost::spirit::qi::double_;
        using boost::spirit::qi::_1;
        using boost::spirit::qi::phrase_parse;
        using boost::spirit::ascii::space;
        using boost::phoenix::ref;

        double rN = 0.0;
        double iN = 0.0;
        bool r = phrase_parse(first, last,

            //  Begin grammar
            (
                    '(' >> double_[ref(rN) = _1]
                        >> -(',' >> double_[ref(iN) = _1]) >> ')'
                |   double_[ref(rN) = _1]
            ),
            //  End grammar

            space);

        if (!r || first != last) // fail if we did not get a full match
            return false;
        c = std::complex<double>(rN, iN);
        return r;
    }
}
//]

////////////////////////////////////////////////////////////////////////////
//  Main program
////////////////////////////////////////////////////////////////////////////
int
main()
{
    std::cout << "/////////////////////////////////////////////////////////\n\n";
    std::cout << "\t\tA complex number micro parser for Spirit...\n\n";
    std::cout << "/////////////////////////////////////////////////////////\n\n";

    std::cout << "Give me a complex number of the form r or (r) or (r,i) \n";
    std::cout << "Type [q or Q] to quit\n\n";

    std::string str;
    while (getline(std::cin, str))
    {
        if (str.empty() || str[0] == 'q' || str[0] == 'Q')
            break;

        std::complex<double> c;
        if (client::parse_complex(str.begin(), str.end(), c))
        {
            std::cout << "-------------------------\n";
            std::cout << "Parsing succeeded\n";
            std::cout << "got: " << c << std::endl;
            std::cout << "\n-------------------------\n";
        }
        else
        {
            std::cout << "-------------------------\n";
            std::cout << "Parsing failed\n";
            std::cout << "-------------------------\n";
        }
    }

    std::cout << "Bye... :-) \n\n";
    return 0;
}

[eranif]

Thanks for this file, like I guessed: its those vectorXXX.hpp files which takes too long to parse.

Other than those files, the parsing is the same speed as other files (very reasonable time)

I can think of several fixes here:
- If codelite_indexer does not respond in XXX seconds - kill it (which will cause the current file to be skipped - small price to pay)
- put in the code a quick and ugly hack to avoid parsing these files
- Maybe placing a memory limit (dont allow more than XXX tags from a given file)

What do you think?

Eran

[jfouche]

- If codelite_indexer does not respond in XXX seconds - kill it (which will cause the current file to be skipped - small price to pay)

Why not. I'm asking myself what could happen at each workspace retag (or at least when saving the workspace file which include the evil .h file) ? Does the following include files will be parse also, to allow a good code completion anyway ? Are you sure you can kill the process ? From the task manager, I didn't succed !

- put in the code a quick and ugly hack to avoid parsing these files

Arghhhh

- Maybe placing a memory limit (dont allow more than XXX tags from a given file)

That will probably decrease parsing performances, because you have to ask for the number of tags in the database during parsing. But that sounds pretty good. We can suppose that this kind of file is not important for code completion, but for internal use of the lib. Moreover, it has the advantage to not trying to retag the evil file each time (unlike the 1st solution).

[eranif]

Are you sure you can kill the process ? From the task manager, I didn't succed !

ofc I can. I am doing it already (whenver you are closing the tags settings dialog the indexer process is restarted)
You may think that the process is not killed, is simply being restarted again by codelite (notice that it has a different PID)

That will probably decrease parsing performances, because you have to ask for the number of tags in the database during parsing

I can get the number of tags from the indexer: the number of tags equals the number of lines returned

Eran

[jfouche]

Hi Eran

I think the 3rd solution is the better one, mostly because it will not try to parse the file each time. I just would like to be sure the parsing of the evil file can stop even if the file is not fully parsed (does lex allow it) ?

BTW : I thought CL developpment was near to be finished... But the TODO list continue. Do you want me to add a feature request on sourceforge ? It would great to have a roadmap based on the sourceforge bugs / requests (like TRAC). The one on the CodeLite web site is out f date now (2.0 is released). I don't think sourceforge provide something like this. Never mind...

[eranif]

I think that SF provides TRAC, I need to enable it. If I will enable it, will you be able to maintain it?

Eran

[jfouche]

If I will enable it, will you be able to maintain it?

Yes, I can do this. I just have to beware of your commits (with CommitMonitor, it's easy), and you just have to let me know the feature you want to implement for a given version (I'll probably ask you from time to times). Let's have a try...

[eranif]

Hi,

CodeLite Trac can be found here:

https://sourceforge.net/apps/trac/codelite/

I gave you permissions on codelite project (I am not sure they sufficient though) - let me know if you can update the Trac pages

Eran