Update workspace tags

[jfouche]

Hi

Do you think that this feature could be interresting ?
Instead of completly retag a workspace, it could be smarter (take less time on big workspace) to just update tags with the changed files. This is usefull when a file is modified out of codelite (by svn update for example).
I think this feature need to modify the tag format to add a DateTime information.
As I think it is to complicated for me, it is just a question (for the moment...).

[eranif]

Do you think that this feature could be interresting ?

The tagging system is now under a compelete re-design:

The main change:
I managed to built ctags as shared object/dll, the idea is to remove the non efficient IPC (inter process communication) between codelite and ctags process in the following way:
- New binary will be created 'codelite_indexer' - which uses ctags as library (i.e. it links against ctags) and accepts from codelite only file name. Once a file name is received, it will be retagged and the output is sent back to codelite over NamedPipe/Unix domain socket.

This way the performance of the tagging is improved significantly, since the current implementation, "talks" to ctags over stdin/out using polling with sleep in the middle.

The next phase, is to let the indexer to save results into the database directly thus eliminating the IPC to minimum.

You might ask your self: "why cant we link against libctags? why do we need another executable?" - the answer is simple: it appears as ctags is not stable as you might think, I managed to get him crash while parsing some files, linking against it will lead to codelite instability. using the indexer, codelite will perform another task: It will be the indexer watchdog (as it currently does for ctags exe)

nstead of completly retag a workspace,

CodeLite offers 3 levels of retagging:
- Workspace - from the menu: Workspace -> retag workspace or from the workspace context menu
- Project - project context menu
- Single file (right click on the file context menu and select retag file OR Ctrl-Shift-G)

Eran

[sdolim]

Eran,

What do you think of this:

Add a new table to the db with columns (path, date). Each row will store the full path of a tagged file and the date/time it was last tagged. Each time a file is tagged, update (or insert) its corresponding row in this table with the current date.

Now when retagging the workspace, it would be possible to check each file's last modification time (from stat) against the time it was last tagged (from the table). So you can skip files that haven't changed since they were tagged. Retagging the workspace should go much faster this way. There can be an option (checkbox maybe) to force retagging of all files regardless of these times.

Scott

[eranif]

the full path is already kept in db just adding last modified

Eran

[eranif]

Some progress report:
The SVN version already uses codelite_indexer (ctags-le removed from SVN) which shows great performance improvements:
re-tagging codelite workspace on my laptop takes 58 seconds with codelite_indexer it now takes ~20 secdonds

The next step is to move the database writing to the indexer and thus eliminating the IPC for the tags from the indexer and codelite.

The SVN version is stable and works on all 3 major platforms (Mac/Win/Linux)

Eran

[eranif]

Another update:

After I implemented the inserting into the database in the indexer, I did not notice any performance improvements, so I decided to rollback this "improvement" (only the part that the indexer writes directly into the database).

However, I added another table to the database which holds the last_retagged timestamp, so now re-tagging is performed in a smart way: only files that were modified since the last time they were re-tagged are now being retagged. this indeeded reduces the overhead of re-tagging workspace to the bare minimum.

Ofc, this functionality can be turned off and on from settings -> tags settings -> advance -> use full re-tagging

Next: Identifies points in the code which re-tagging make sense (like after svn-update, svn-apply patch, opening workspace etc.)

Eran

[sdolim]

This feature is working very nicely for me.

My suggestion on when to auto-retag the workspace is in the app-activated event handler (the same place that call Mainbook::ReloadExternallyModified). That way you don't have to worry about it in all the other places (like after svn-update, svn-apply patch, opening workspace etc.).

Scott

[eranif]

My suggestion on when to auto-retag the workspace is in the app-activated event handler (the same place that call Mainbook::ReloadExternallyModified). That way you don't have to worry about it in all the other places (like after svn-update, svn-apply patch, opening workspace etc.).

I am not sure on this one since on Windows it does not trigger until you switch to another application and then go back to codelite.

For example:
When you do update from svn for a file which is currently being opened on Linux you get the prompt immediately however on windows, you wont get the reload until you switch back and forth applications OR you open a any dialog and dismiss it

I also prefer to keep it selective and ofc make it configurable for people who prefer not to have it enabled.

Eran

[sdolim]

OK. But it will still catch all cases where files are modified by a tool outside of codelite. E.g. I often do "svn update" in my terminal -- when I switch back to CL, the app activation handler would make sure my tags db is updated.

[eranif]

E.g. I often do "svn update" in my terminal

Well, start doing it from within codelite

Anyways, for now, codelite automatically retags files after svn-update & svn-apply patch.

This option is configurable from the 'plugins -> subversion -> options -> general -> keep workspace tags up-to-date...'
I will probably add this option for 'Open workspace' as well.

Eran