Using the parser in other open source projects

[sreinst1]

Doesn't the update of the outline view depend on the process of Ctags and the database update? They must be completed before you can update the outline...

It does.

This is what the worker thread does:
- Invoke ctags for the modified file
- Reads ctags output (ctags is running as daemon process, --filter=yes option of ctags)
- Performs a 'diff' operation between the tags that exists in the database for this file, and the new ones received from ctags.
- update the database with the new tags
- Send 3 events to the outline view:
- SymbolDelete - the event also contains list of tags that should be removed from the outline view
- SymbolUpdate - the event also contains list of entries that their display/icon should be modified
- SymbolNew - new items that should be added to the tree

My OS: WinXP Professional
CPU: 3.0 Ghz, single core

Probably this is why you see this delay. I will look into the code to see where it can be optimized..

Eran

Thanks. You have really done a great job in CodeLite, both in the ideas and the implementation. Thanks! You don't have to make further changes for me, as I'm probably going to stick with jEdit, for various reasons. But I learned (and still do) a lot from CodeLite, and I hope I will be able to contribute some features back to CodeLite.

[sreinst1]

1. Is it okay for me to take the parser (and lexer) of CodeLite, and port them to ANTLR (a parser generator in Java)?

As long as you comply to the GPL license.

What does it mean to comply to the GPL license? Does it just mean that the grammar file I create for ANTLR will also have a GPL license header and associated file attached? If I decide to do this, it will be in order to use it in a jEdit plugin, which is itself open source. But ANTLR itself has BSD license.

[eranif]

It means that if you decide to use my grammar file, and you link against it or any of its generated code - you should provide the source code for your application as well

Eran

[sreinst1]

It means that if you decide to use my grammar file, and you link against it or any of its generated code - you should provide the source code for your application as well

Eran

Ok, thanks. I will not link against it, I will simply use it to create an ANTLR grammar file, probably with Java actions.

[eranif]

I am not sure if you are heading the ctags way, but due to your report about latency when saving file on your machine, I implemented something that I planned while ago:

I create a library out of ctags, so instead of invoking it as daemon process ( the current way that codelite uses it), I simply make a simple API call to DLL / Shared Object. This improved performance by factor ~3.5

For example:
Creating external database out of wx library, now takes about 12.1 secs instead of 45 seconds with the previous implementation

Eran

[sreinst1]

I am not sure if you are heading the ctags way, but due to your report about latency when saving file on your machine, I implemented something that I planned while ago:

I create a library out of ctags, so instead of invoking it as daemon process ( the current way that codelite uses it), I simply make a simple API call to DLL / Shared Object. This improved performance by factor ~3.5

For example:
Creating external database out of wx library, now takes about 12.1 secs instead of 45 seconds with the previous implementation

Eran

Amazing! Very nice work. So I guess you replace the tag file creation with a direct pluggable tag collector that you implemented in CodeLite? This way you can avoiding both the overhead of launching a separate process and the overhead of creating a file on disk. I cannot do the same in jEdit - it's a java app.

[sreinst1]

I am not sure if you are heading the ctags way, but due to your report about latency when saving file on your machine, I implemented something that I planned while ago:

I create a library out of ctags, so instead of invoking it as daemon process ( the current way that codelite uses it), I simply make a simple API call to DLL / Shared Object. This improved performance by factor ~3.5

For example:
Creating external database out of wx library, now takes about 12.1 secs instead of 45 seconds with the previous implementation

Eran

Amazing! Very nice work. So I guess you replace the tag file creation with a direct pluggable tag collector that you implemented in CodeLite? This way you can avoiding both the overhead of launching a separate process and the overhead of creating a file on disk. I cannot do the same in jEdit - it's a java app.

Oops! I missed some very important feedback for you on this. While using Ctags as a library can greatly improve performance, note that it can also effect the stability of CodeLite. During my short experience with CodeLite, there were many invocations of the Ctags executable (the modified version used by CodeLite) that actually crashed (with an Access Violation or something). Somehow CodeLite handles this pretty well and the crash popups that I get disappear within less than a second, and CodeLite keeps calling Ctags for other files. But, if you use Ctags as a library, this will crash CodeLite instead of just the Ctags process.

[eranif]

Oops! I missed some very important feedback for you on this. While using Ctags as a library can greatly improve performance, note that it can also effect the stability of CodeLite. During my short experience with CodeLite, there were many invocations of the Ctags executable (the modified version used by CodeLite) that actually crashed (with an Access Violation or something). Somehow CodeLite handles this pretty well and the crash popups that I get disappear within less than a second, and CodeLite keeps calling Ctags for other files. But, if you use Ctags as a library, this will crash CodeLite instead of just the Ctags process.

Indeed, you are correct. This a risk which I am now testing to see if it worth embedding into the next release. Currently, CodeLite detects crashes and restarts ctags. I have not encounter any crashes yet of ctags on Windows/Linux but now during my testings on Mac, I am facing some crashes all inside ctags.so.

So I may revert my code back to use the executable.

Eran

[sreinst1]

Oops! I missed some very important feedback for you on this. While using Ctags as a library can greatly improve performance, note that it can also effect the stability of CodeLite. During my short experience with CodeLite, there were many invocations of the Ctags executable (the modified version used by CodeLite) that actually crashed (with an Access Violation or something). Somehow CodeLite handles this pretty well and the crash popups that I get disappear within less than a second, and CodeLite keeps calling Ctags for other files. But, if you use Ctags as a library, this will crash CodeLite instead of just the Ctags process.

Indeed, you are correct. This a risk which I am now testing to see if it worth embedding into the next release. Currently, CodeLite detects crashes and restarts ctags. I have not encounter any crashes yet of ctags on Windows/Linux but now during my testings on Mac, I am facing some crashes all inside ctags.so.

So I may revert my code back to use the executable.

Eran

Note two things:
1. On my WinXP system, CodeLite's invocations of Ctags crashed quite a lot. Our system contains some IDL (Interface Description Language) files, maybe Ctags crashed on them, I don't know because the popup disappears about half a second after it appears.
2. As far as I know (and this is quite new to me), you can even overcome such crashes of Ctags as a library using the SEH (Structured Exception Handling) on Windows, and possibly using similar elements in other operating systems. On Windows, you can even catch access violations as exceptions and handle them. So it is possible that you can overcome such problems and continue to use Ctags as a library.