Preprocessing and scintilla_22

[jfouche]

I tried last scite editor, to test the new preprocessor handling in scintilla. That's great.
Here is what I understood :
The preprocessor definitions must be entered as keyword 5 for c++, and the lexer.cpp.track.preprocessor, and lexer.cpp.update.preprocessor should be set to 1 as describe here.
I gave a look to the lexcpp file, but it's not so easy to understand. But It gaves me some ideas to improve the preprocessor handling in codelite :

1 - The first step I see is to allow the user to add preprocessor definitions and CodeLite will add it to the keyword5.
2 - Much more sophisticated : The CodeLite preprocessor may use the lexcpp code to improve the preprocessing in order to find itself all other preprocessing definitions in the headers found by include_finder.
3 - ... and it shall improve the way how CodeLite register the good definition of token (if there are multiple definitions according to preprocessor definitions) :

Code: Select all

#define FOO
...

#ifdef FOO
class Foo {
...
};
#else
class Foo {
...
};
#endif

as currently, CodeLite takes the first one (or the last one, I don't know).

I you have some idea how to implement this, I would be happy to help you on this subject as it will greatly improve this Editor. Let me know

[eranif]

Actually I already put some thoughts on scintilla 2.2 here are the directions that I had in mind:

- Scintilla already implements all the logic of the pre-processing calculations - so codelite only needs to feed it with the available macros (and their values)
- Currently codelite has an infrastructure to parse macro ( to see it in action: Settings | Tags Settings | Advanced and click on the 'Parse!' button)

Suggested flow:

- File is opened
- A request is sent to the parser-thread to parse all macros included from this file
- Once the parse is done, an event is sent back to the editor (the requested file name is sent back with teh event to make sure that the calling editor is still opened)
- If the requesting editor is "still alive" the macros are then "feed" into scintilla

Of course we need to move codelite to scintilla 2.2

The main work was already done. I just need to merge it into trunk
see here:

http://codelite.svn.sourceforge.net/vie ... NTILLA_22/

Eran

[jfouche]

Hi Eran

The main problem I see with what you suggest is that the point #3 will not work (goto definition may send you to a class definition grayed, due to he fact it's not in the good preprocessor block). But that seems to be a good first step.

I tried the Settings | Tags Settings | Advanced and click on the 'Parse!' button : It seems that the first definition is the only one taken in account (I suppose that the DB definition do not allow you to define 2 times the same token, so the 2nd one is not inserted in DB). To go further, I created the following file :
jeremie.h

Code: Select all

#ifndef FOO_H_INCLUDED
#define FOO_H_INCLUDED

#define BAR

#define TEST 1

#ifndef BAR
	#define FOO_1
	struct Foo {
	   int i;
	};
#else
	#define FOO_2
	struct Foo {
	   int i;
	   int j;
	};
#endif

#define Y 89

#endif // FOO_H_INCLUDED

I put this file in c:\MinGW\include. I checked that C:\MinGW\include is in the list of the include directories (Tags options / Include file). I added jeremie.h in the Tags options / Advanced / headers list, and pressed the Parse button. The file created did not contain my own preprocessor definitions.

[eranif]

(goto definition may send you to a class definition grayed, due to he fact it's not in the good preprocessor block). But that seems to be a good first step.

this is because there are actually two preprocessors: ctags and codelite
codelite's one is more accurate, while ctags will always takes the 'else' branch (unless #if 0 is found)

I put this file in c:\MinGW\include. I checked that C:\MinGW\include is in the list of the include directories (Tags options / Include file). I added jeremie.h in the Tags options / Advanced / headers list, and pressed the Parse button. The file created did not contain my own preprocessor definitions.

This is because for that purpose (for the 'replacements') there is no added values for macros with constant values (like all the macros you defined)
so codelite filtered them out (there were parsed, they were simply been removed before being pasted into the output file)

If you will change any macro to a function like macro, you will see it:
For example:

Code: Select all

#define  M_FOO(x) x

The API call PPScan(const wxFileName &file, bool foCC) is called with forCC set to true.
To retrieve all macros pass this argument as 'false'

Eran

[jfouche]

I tried SCINTILLA_22 branch, but all preprocessor blocks are grayed, probably due to the fact that no preprocessor definition are define as keyword5. I went to Settings -> Syntax highlight and fonts -> C++ -> keyword4 (which seems to be pp def, as saw in LexCPP line 349), to try to enter my own preprocessor definition, without success.
How can I enable it, if you ever succeed ?

[jfouche]

I made it work, added the foolowing code in context_cpp.cpp :

*** C:/_Perso/projets/codelite_SCINTILLA_22/LiteEditor/context_cpp.cpp Fri Oct 29 10:14:14 2010
--- C:/_Perso/projets/codelite/LiteEditor/context_cpp.cpp Sun Oct 24 13:56:05 2010
***************
*** 1840,1856 ****
doxyKeyWords.Replace(wxT("\r"), wxT(" "));
rCtrl.SetKeyWords(2, doxyKeyWords);

- //{ JFO
- // preprocessor keywords
- wxString ppKeyWords = lexPtr->GetKeyWords(4);
- ppKeyWords.Replace(wxT("\n"), wxT(" "));
- ppKeyWords.Replace(wxT("\r"), wxT(" "));
- rCtrl.SetKeyWords(4, ppKeyWords);
-
- rCtrl.SetProperty(wxT("lexer.cpp.track.preprocessor"), wxT("1"));
- rCtrl.SetProperty(wxT("lexer.cpp.update.preprocessor"), wxT("1"));
- //} JFO
-
DoApplySettings( lexPtr );

//create all images used by the cpp context
--- 1851,1856 ----

sorry about the patch format, which is not easy to apply, but I do not have SVN currenty, so I made it with WinMerge.

[jfouche]

Hello Eran

What is the command line you use to transform your lex (.l) & yacc (.y) files to cpp file ? Do you use flex.exe (the one you provide with CodeLite/MinGW) ?

I tried :

flex -B -f -L -opp.cpp -Cem -8 pp.l pp.y

But the result is not the same at all

Edit : I found :

yacc -l -t -p pp_ pp.y

But I realized that the pp.l and pp.y from the CodeLite directory are not the good one. The reference is in PreProcessor/filecrowler. So I have to copy the y.tab.c to CodeLite\pp.cpp, insn't it ?
I still have problems with flex : I have some diffs with the folowing command line :

flex -B -L -Cem -opp_lexer.jfo.cpp pp.l

--- C:/_Perso/projets/codelite/SCINTILLA_22/PreProcessor/fileCrawler/pp_lexer.cpp Sat Oct 30 09:49:59 2010
+++ C:/_Perso/projets/codelite/SCINTILLA_22/PreProcessor/fileCrawler/pp_lexer.jfo.cpp Sat Oct 30 14:38:51 2010
@@ -1,22 +1,3 @@
-#define yy_create_buffer pp__create_buffer
-#define yy_delete_buffer pp__delete_buffer
-#define yy_scan_buffer pp__scan_buffer
-#define yy_scan_string pp__scan_string
-#define yy_scan_bytes pp__scan_bytes
-#define yy_flex_debug pp__flex_debug
-#define yy_init_buffer pp__init_buffer
-#define yy_flush_buffer pp__flush_buffer
-#define yy_load_buffer_state pp__load_buffer_state
-#define yy_switch_to_buffer pp__switch_to_buffer
-#define yyin pp_in
-#define yyleng pp_leng
-#define yylex pp_lex
-#define yyout pp_out
-#define yyrestart pp_restart
-#define yytext pp_text
-#define yylineno pp_lineno
-#define yywrap pp_wrap
-
/* A lexical scanner generated by flex */

/* Scanner skeleton version:
@@ -829,7 +810,7 @@
*yy_state_ptr++ = yy_current_state;
++yy_cp;
}
- while ( yy_base[yy_current_state] != 413 );
+ while ( yy_current_state != 165 );

yy_find_action:
yy_current_state = *--yy_state_ptr;

[jfouche]

Well, I finaly succeeded
But I had to change a little bit the PP workspace. I added the following line as pre build step :

yacc -l -t -p pp_ pp.y
cmd /C "move /Y y.tab.c pp.cpp"
flex -B -L -Cem -opp_lexer.content.cpp pp.l
cmd /C "copy /Y /b pp_lexer.hdr.h+pp_lexer.content.cpp pp_lexer.cpp"

where pp_lexer.hdr.h is

Code: Select all

#define yy_create_buffer pp__create_buffer
#define yy_delete_buffer pp__delete_buffer
#define yy_scan_buffer pp__scan_buffer
#define yy_scan_string pp__scan_string
#define yy_scan_bytes pp__scan_bytes
#define yy_flex_debug pp__flex_debug
#define yy_init_buffer pp__init_buffer
#define yy_flush_buffer pp__flush_buffer
#define yy_load_buffer_state pp__load_buffer_state
#define yy_switch_to_buffer pp__switch_to_buffer
#define yyin pp_in
#define yyleng pp_leng
#define yylex pp_lex
#define yyout pp_out
#define yyrestart pp_restart
#define yytext pp_text
#define yylineno pp_lineno
#define yywrap pp_wrap

I made some tests, and give you a feedback in next post (probably this evening)...

[eranif]

You should not need to change anything, it should all be set already.

Open the workspace under PreProcessor/

select the PP.workspace open it hit F7 and its done (including cpoying all relevant files)

The yacc commands are already there (they are under 'custom makefile steps') + the post commands
Eran

[jfouche]

Me again

The work to do is RefactoringEngine::GetInterestingMacros(text), which is empty right now.
So, if I understand, we have to :

- A request is sent to the parser-thread to parse all macros included from this file
- Once the parse is done, an event is sent back to the editor (the requested file name is sent back with teh event to make sure that the calling editor is still opened)
- If the requesting editor is "still alive" the macros are then "feed" into scintilla

from there.