make tcc reentrant

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
43 messages Options
123
Reply | Threaded
Open this post in threaded view
|

make tcc reentrant

Ulrich Schmidt
Hi all.

i try to write a lua binding for tcc. To work out propperly, the tcc lib
needs to be reentrant.

I took a look into the sources and found some comments (XXX:...) and
started with removing

the static var tcc_state. As a result allmost all lib functions needs a
1st parameter of

type TCCState*. I did this in my own local branch and tcc is still
running :).

But this is a really HUGE change. in addition most of the local vars in
tccpp, tccgen, ... needs

to be moved to TCCState. I can do that but at some points i will have
some questions and i

can only test on windows and probably on linux.

My 1st question is: Are you interested in these changes or should i do
this locally?

I would like to this together with you.


Greetings.

Ulrich.


_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Christian Jullien-3
Hi,

Even if I don't currently need it, with multi-core, multi-thread modern CPU, it is always a good principle to make reentrant any software components.
I personally vote for it but, as it is a major change, especially regarding the API, I prefer if it starts after 0.9.27. BTW, 0.9.27 is about 2 years old. It's probably time to freeze it and start 0.9.28 with possibly reentrant feature as main goal.

Maintainers, wdyt?

C.

-----Original Message-----
From: Tinycc-devel [mailto:tinycc-devel-bounces+eligis=[hidden email]] On Behalf Of Ulrich Schmidt
Sent: Tuesday, December 03, 2019 09:05
To: TCC mailing list
Subject: [Tinycc-devel] make tcc reentrant

Hi all.

i try to write a lua binding for tcc. To work out propperly, the tcc lib
needs to be reentrant.

I took a look into the sources and found some comments (XXX:...) and
started with removing

the static var tcc_state. As a result allmost all lib functions needs a
1st parameter of

type TCCState*. I did this in my own local branch and tcc is still
running :).

But this is a really HUGE change. in addition most of the local vars in
tccpp, tccgen, ... needs

to be moved to TCCState. I can do that but at some points i will have
some questions and i

can only test on windows and probably on linux.

My 1st question is: Are you interested in these changes or should i do
this locally?

I would like to this together with you.


Greetings.

Ulrich.


_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel


_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Ulrich Schmidt
Hi.

I dod not change the api. Only the internal storage of internal static
vars need to be moved to the compilation state struct.


Ulrich.


Am 03.12.2019 um 09:30 schrieb Christian Jullien:

> Hi,
>
> Even if I don't currently need it, with multi-core, multi-thread modern CPU, it is always a good principle to make reentrant any software components.
> I personally vote for it but, as it is a major change, especially regarding the API, I prefer if it starts after 0.9.27. BTW, 0.9.27 is about 2 years old. It's probably time to freeze it and start 0.9.28 with possibly reentrant feature as main goal.
>
> Maintainers, wdyt?
>
> C.
>
> -----Original Message-----
> From: Tinycc-devel [mailto:tinycc-devel-bounces+eligis=[hidden email]] On Behalf Of Ulrich Schmidt
> Sent: Tuesday, December 03, 2019 09:05
> To: TCC mailing list
> Subject: [Tinycc-devel] make tcc reentrant
>
> Hi all.
>
> i try to write a lua binding for tcc. To work out propperly, the tcc lib
> needs to be reentrant.
>
> I took a look into the sources and found some comments (XXX:...) and
> started with removing
>
> the static var tcc_state. As a result allmost all lib functions needs a
> 1st parameter of
>
> type TCCState*. I did this in my own local branch and tcc is still
> running :).
>
> But this is a really HUGE change. in addition most of the local vars in
> tccpp, tccgen, ... needs
>
> to be moved to TCCState. I can do that but at some points i will have
> some questions and i
>
> can only test on windows and probably on linux.
>
> My 1st question is: Are you interested in these changes or should i do
> this locally?
>
> I would like to this together with you.
>
>
> Greetings.
>
> Ulrich.
>
>
> _______________________________________________
> Tinycc-devel mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/tinycc-devel
>
>
> _______________________________________________
> Tinycc-devel mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Sean Conner
In reply to this post by Ulrich Schmidt
It was thus said that the Great Ulrich Schmidt once stated:
> Hi all.
>
> i try to write a lua binding for tcc. To work out propperly, the tcc lib
> needs to be reentrant.

  I never had a problem with TCC not being reentrant when I made my Lua
bindings for it (way back in 2012 [1]).  But then again, I never really used
this module for production work.  If you are intersted:

        The C code wrapper:

                https://github.com/spc476/lua-conmanorg/blob/master/src/tcc.c

        The Lua module which uses the above module to load C-based Lua
modules via source code:

        https://github.com/spc476/lua-conmanorg/blob/master/lua/cc.lua

  -spc (Hope it helps some ... )

[1] Here's the thread where I first bring it up:

        https://lists.nongnu.org/archive/html/tinycc-devel/2012-08/msg00001.html


_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Ulrich Schmidt
Thank you, Sean. I'l take a look.

Your Idea of making tcc_relocate_ex() global is probably worth thinking
about.

I wrote my own binding and ran into trouble like you did and i
understood not enouth about tcc internals. (TCC doc is not the best
source of information.) Hence i looked into the sources and figured out:

If more than one app uses libtcc, the compilation processes interfere
with each other causing compilation faults. If you use libtcc only once
at a time, all is fine.

 From my point of view a lua module needs to usable multiple times parallel.

Ulrich.

Am 03.12.2019 um 10:09 schrieb Sean Conner:

> It was thus said that the Great Ulrich Schmidt once stated:
>> Hi all.
>>
>> i try to write a lua binding for tcc. To work out propperly, the tcc lib
>> needs to be reentrant.
>    I never had a problem with TCC not being reentrant when I made my Lua
> bindings for it (way back in 2012 [1]).  But then again, I never really used
> this module for production work.  If you are intersted:
>
> The C code wrapper:
>
> https://github.com/spc476/lua-conmanorg/blob/master/src/tcc.c
>
> The Lua module which uses the above module to load C-based Lua
> modules via source code:
>
> https://github.com/spc476/lua-conmanorg/blob/master/lua/cc.lua
>
>    -spc (Hope it helps some ... )
>
> [1] Here's the thread where I first bring it up:
>
> https://lists.nongnu.org/archive/html/tinycc-devel/2012-08/msg00001.html
>
>
> _______________________________________________
> Tinycc-devel mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

uso ewin
On Tue, Dec 3, 2019 at 11:02 AM Ulrich Schmidt <[hidden email]> wrote:

>
> Thank you, Sean. I'l take a look.
>
> Your Idea of making tcc_relocate_ex() global is probably worth thinking
> about.
>
> I wrote my own binding and ran into trouble like you did and i
> understood not enouth about tcc internals. (TCC doc is not the best
> source of information.) Hence i looked into the sources and figured out:
>
> If more than one app uses libtcc, the compilation processes interfere
> with each other causing compilation faults. If you use libtcc only once
> at a time, all is fine.
>
>  From my point of view a lua module needs to usable multiple times parallel.
>
> Ulrich.
>

For parallel it could be possible to optionally change global as
_Thread_local variable,
this will make tcc unable to compile itself if the option is enable,
as it didn't support _Thread_local,
but could be easier than making tcc reentrant, I don't know which
approach is better
but maybe it's worth thinking about this option too.

about reentrantcy, I'm not a maintainer either, I'm not against this
change, as it's a big change
maybe it's possible to make this in an incremental way by first making
small patchs that will
move the global to TCCState.

Also I like the idea of a freeze in order to release a 0.28  soon

Matthias,

> Am 03.12.2019 um 10:09 schrieb Sean Conner:
> > It was thus said that the Great Ulrich Schmidt once stated:
> >> Hi all.
> >>
> >> i try to write a lua binding for tcc. To work out propperly, the tcc lib
> >> needs to be reentrant.
> >    I never had a problem with TCC not being reentrant when I made my Lua
> > bindings for it (way back in 2012 [1]).  But then again, I never really used
> > this module for production work.  If you are intersted:
> >
> >       The C code wrapper:
> >
> >               https://github.com/spc476/lua-conmanorg/blob/master/src/tcc.c
> >
> >       The Lua module which uses the above module to load C-based Lua
> > modules via source code:
> >
> >       https://github.com/spc476/lua-conmanorg/blob/master/lua/cc.lua
> >
> >    -spc (Hope it helps some ... )
> >
> > [1]   Here's the thread where I first bring it up:
> >
> >       https://lists.nongnu.org/archive/html/tinycc-devel/2012-08/msg00001.html
> >
> >
> > _______________________________________________
> > Tinycc-devel mailing list
> > [hidden email]
> > https://lists.nongnu.org/mailman/listinfo/tinycc-devel
>
> _______________________________________________
> Tinycc-devel mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Ulrich Schmidt

Am 03.12.2019 um 11:49 schrieb uso ewin:

> On Tue, Dec 3, 2019 at 11:02 AM Ulrich Schmidt <[hidden email]> wrote:
>> Thank you, Sean. I'l take a look.
>>
>> Your Idea of making tcc_relocate_ex() global is probably worth thinking
>> about.
>>
>> I wrote my own binding and ran into trouble like you did and i
>> understood not enouth about tcc internals. (TCC doc is not the best
>> source of information.) Hence i looked into the sources and figured out:
>>
>> If more than one app uses libtcc, the compilation processes interfere
>> with each other causing compilation faults. If you use libtcc only once
>> at a time, all is fine.
>>
>>   From my point of view a lua module needs to usable multiple times parallel.
>>
>> Ulrich.
>>
> For parallel it could be possible to optionally change global as
> _Thread_local variable,
> this will make tcc unable to compile itself if the option is enable,
> as it didn't support _Thread_local,
> but could be easier than making tcc reentrant, I don't know which
> approach is better
> but maybe it's worth thinking about this option too.
>
> about reentrantcy, I'm not a maintainer either, I'm not against this
> change, as it's a big change
> maybe it's possible to make this in an incremental way by first making
> small patchs that will
> move the global to TCCState.
>
> Also I like the idea of a freeze in order to release a 0.28  soon
>
> Matthias,

I agree with you. I changed all the function parameters in the i386 and
X64 related sources. I could push my branch so anyone can see how it's
look like. In 2 source files i already placed comments which static vars
need to be moved into TCCState.

Ulrich.


_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Karl Yerkes
i have a project that needs this and i've been studying the source of tcc to figure out how to do it. i would like to see how you plan to do this.

i would test on macOS. 

-- karl

On Tue, Dec 3, 2019 at 2:59 AM Ulrich Schmidt <[hidden email]> wrote:

Am 03.12.2019 um 11:49 schrieb uso ewin:
> On Tue, Dec 3, 2019 at 11:02 AM Ulrich Schmidt <[hidden email]> wrote:
>> Thank you, Sean. I'l take a look.
>>
>> Your Idea of making tcc_relocate_ex() global is probably worth thinking
>> about.
>>
>> I wrote my own binding and ran into trouble like you did and i
>> understood not enouth about tcc internals. (TCC doc is not the best
>> source of information.) Hence i looked into the sources and figured out:
>>
>> If more than one app uses libtcc, the compilation processes interfere
>> with each other causing compilation faults. If you use libtcc only once
>> at a time, all is fine.
>>
>>   From my point of view a lua module needs to usable multiple times parallel.
>>
>> Ulrich.
>>
> For parallel it could be possible to optionally change global as
> _Thread_local variable,
> this will make tcc unable to compile itself if the option is enable,
> as it didn't support _Thread_local,
> but could be easier than making tcc reentrant, I don't know which
> approach is better
> but maybe it's worth thinking about this option too.
>
> about reentrantcy, I'm not a maintainer either, I'm not against this
> change, as it's a big change
> maybe it's possible to make this in an incremental way by first making
> small patchs that will
> move the global to TCCState.
>
> Also I like the idea of a freeze in order to release a 0.28  soon
>
> Matthias,

I agree with you. I changed all the function parameters in the i386 and
X64 related sources. I could push my branch so anyone can see how it's
look like. In 2 source files i already placed comments which static vars
need to be moved into TCCState.

Ulrich.


_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

uso ewin
In reply to this post by Ulrich Schmidt
On Tue, Dec 3, 2019 at 11:55 AM Ulrich Schmidt <[hidden email]> wrote:

>
>
> Am 03.12.2019 um 11:49 schrieb uso ewin:
> > On Tue, Dec 3, 2019 at 11:02 AM Ulrich Schmidt <[hidden email]> wrote:
> >> Thank you, Sean. I'l take a look.
> >>
> >> Your Idea of making tcc_relocate_ex() global is probably worth thinking
> >> about.
> >>
> >> I wrote my own binding and ran into trouble like you did and i
> >> understood not enouth about tcc internals. (TCC doc is not the best
> >> source of information.) Hence i looked into the sources and figured out:
> >>
> >> If more than one app uses libtcc, the compilation processes interfere
> >> with each other causing compilation faults. If you use libtcc only once
> >> at a time, all is fine.
> >>
> >>   From my point of view a lua module needs to usable multiple times parallel.
> >>
> >> Ulrich.
> >>
> > For parallel it could be possible to optionally change global as
> > _Thread_local variable,
> > this will make tcc unable to compile itself if the option is enable,
> > as it didn't support _Thread_local,
> > but could be easier than making tcc reentrant, I don't know which
> > approach is better
> > but maybe it's worth thinking about this option too.
> >
> > about reentrantcy, I'm not a maintainer either, I'm not against this
> > change, as it's a big change
> > maybe it's possible to make this in an incremental way by first making
> > small patchs that will
> > move the global to TCCState.
> >
> > Also I like the idea of a freeze in order to release a 0.28  soon
> >
> > Matthias,
>
> I agree with you. I changed all the function parameters in the i386 and
> X64 related sources. I could push my branch so anyone can see how it's
> look like. In 2 source files i already placed comments which static vars
> need to be moved into TCCState.
>
> Ulrich.
>
That would be nice, If you have your code on a public repository,
you can just send a link on the ML and peoples might review it
>
> _______________________________________________
> Tinycc-devel mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Ulrich Schmidt
In reply to this post by Karl Yerkes
Am 03.12.2019 um 17:16 schrieb Karl Yerkes:
 > i have a project that needs this and i've been studying the source of
tcc to figure out how to do it. i would like to see how you plan to do this.
 >
 > i would test on macOS.
 >
While doing the changes we need a freeze (other changes should be delayed).
The 1st step i already started:
- removing the static tcc_state, hence all functions calling the error
function
     needs to be able to know about the TCCState. All affected functions
need a
     TCCState* as a first parameter. This first parameter allways is
named "s1".
     I started this change for all X86 related files.
- Mark all static vars that needs to be moved to TCCState in all source
files
     with a comment. This could be done as a teamwork.
- The move process should be done in 2 steps. First move local static
vars into
     a new struct. Static vars referenced by multiple source files will
be moved
     directly into TCCState.
     Step2: Move all local struct fields to TCCState.

All this should be done in tiny steps while some people are testing on
different
platforms.

Looks like a lot of work. :)

Ulrich.


_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Ulrich Schmidt
In reply to this post by uso ewin
Am 03.12.2019 um 18:08 schrieb uso ewin:
> That would be nice, If you have your code on a public repository,
> you can just send a link on the ML and peoples might review it

Here are my changes (WIP): https://github.com/Fuzzlix/tinycc
I see my repo as a temporary one. The work should probably done on the
main repo.

Ulrich.


_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Michael Matz-4
In reply to this post by Ulrich Schmidt
Hello,

On Tue, 3 Dec 2019, Ulrich Schmidt wrote:

> i try to write a lua binding for tcc. To work out propperly, the tcc lib
> needs to be reentrant.

As demonstrated down-thread, that isn't correct.  It doesn't _need_ to be,
it would be an feature.  As usual with features it needs to be measured
against the downsides.  The downsides for your proposed changes are the
following at least:
1) more complicated/boiler-platy source code of TCC (a TCCState
   argument almost everywhere)
2) slower code: most of the time the indirection through a pointer
   variable (the state) in comparison to a direct access to a static
   variable doesn't matter.  But it does matter for the symbol/token
   table (and potentially for the register/evaluation stack).  I have
   measured this years ago for the token table, so this might or might not
   still be the case.

So, while I can see the wish for this feature, I don't necessarily see
that tcc should be changed to accomodate.

If anything I would expect a _complete_ transition to exist, in order to
measure the impact.  The worst thing that could happen is if someone added
TCCState arguments everywhere, moved some static variables to that state,
and then leaves: none of the features of this whole excercise would be
had, but all the downsides would be there.

And yes, this is a big project.  I really think it would be better
if you simply write a wrapper for libtcc that ensures single-threadedness
and that regards TCCState as a singleton.  I think such thing would be
well-suited in the TCC sources itself.

(In a way it seems prudent for a tiny C compiler to only be usable as a
singleton)


Ciao,
Michael.

>
> I took a look into the sources and found some comments (XXX:...) and
> started with removing
>
> the static var tcc_state. As a result allmost all lib functions needs a
> 1st parameter of
>
> type TCCState*. I did this in my own local branch and tcc is still
> running :).
>
> But this is a really HUGE change. in addition most of the local vars in
> tccpp, tccgen, ... needs
>
> to be moved to TCCState. I can do that but at some points i will have
> some questions and i
>
> can only test on windows and probably on linux.
>
> My 1st question is: Are you interested in these changes or should i do
> this locally?
>
> I would like to this together with you.
>
>
> Greetings.
>
> Ulrich.
>
>
> _______________________________________________
> Tinycc-devel mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/tinycc-devel
>

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Karl Yerkes
In reply to this post by Ulrich Schmidt
that repo disappeared. is this topic dead?

i would settle for a something less than fully reentrant. for instance, if i could just link to two essentially identical, but separate copies of libtcc, then i could be happy. to do this, i need to be able to rename all the functions and globals. i'm writing a program that will find and rename all symbols, so i hope to be able to link to two separate copies of libtcc that don't share symbols. 


On Tue, Dec 3, 2019 at 11:44 AM Ulrich Schmidt <[hidden email]> wrote:
Am 03.12.2019 um 18:08 schrieb uso ewin:
> That would be nice, If you have your code on a public repository,
> you can just send a link on the ML and peoples might review it

Here are my changes (WIP): https://github.com/Fuzzlix/tinycc
I see my repo as a temporary one. The work should probably done on the
main repo.

Ulrich.


_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Agathoklis D.E Chatzimanikas
In reply to this post by Michael Matz-4
On Fri, Dec 06, at 03:42 Michael Matz wrote:

> Hello,
>
> On Tue, 3 Dec 2019, Ulrich Schmidt wrote:
>
> > i try to write a lua binding for tcc. To work out propperly, the tcc lib
> > needs to be reentrant.
>
> As demonstrated down-thread, that isn't correct.  It doesn't _need_ to be,
> it would be an feature.  As usual with features it needs to be measured
> against the downsides.  The downsides for your proposed changes are the
> following at least:
> 1) more complicated/boiler-platy source code of TCC (a TCCState
>    argument almost everywhere)
> 2) slower code: most of the time the indirection through a pointer
>    variable (the state) in comparison to a direct access to a static
>    variable doesn't matter.  But it does matter for the symbol/token
>    table (and potentially for the register/evaluation stack).  I have
>    measured this years ago for the token table, so this might or might not
>    still be the case.
>
> So, while I can see the wish for this feature, I don't necessarily see
> that tcc should be changed to accomodate.
>
> If anything I would expect a _complete_ transition to exist, in order to
> measure the impact.  The worst thing that could happen is if someone added
> TCCState arguments everywhere, moved some static variables to that state,
> and then leaves: none of the features of this whole excercise would be
> had, but all the downsides would be there.
>
> And yes, this is a big project.  I really think it would be better
> if you simply write a wrapper for libtcc that ensures single-threadedness
> and that regards TCCState as a singleton.  I think such thing would be
> well-suited in the TCC sources itself.
>
> (In a way it seems prudent for a tiny C compiler to only be usable as a
> singleton)

I maybe understand, that a C compiler would be best as a singleton, as a
state can influence unexpectedly the other states (unless (perhaps like in
this case as it would be under Lua control) there is a mechanism to handle
gracefully any errors)), and i can trust you about the "direct access to static
variables", but how about an (probably predefined (even with a compile option)
__array__ of states (under a single-thread), and expose it generiously and with
a pleasure to the user responsibility, without tcc guilties (if any)?
Perhaps can even implement this mechanism (to have a control of the environment)
by it's own.

Anyway C is unsafe by default (if it is that worry).

> Ciao,
> Michael.

Best,
  Αγαθοκλής

> >
> > I took a look into the sources and found some comments (XXX:...) and
> > started with removing
> >
> > the static var tcc_state. As a result allmost all lib functions needs a
> > 1st parameter of
> >
> > type TCCState*. I did this in my own local branch and tcc is still
> > running :).
> >
> > But this is a really HUGE change. in addition most of the local vars in
> > tccpp, tccgen, ... needs
> >
> > to be moved to TCCState. I can do that but at some points i will have
> > some questions and i
> >
> > can only test on windows and probably on linux.
> >
> > My 1st question is: Are you interested in these changes or should i do
> > this locally?
> >
> > I would like to this together with you.
> >
> >
> > Greetings.
> >
> > Ulrich.

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Charles Lohr
Is there a reason you don't just compile tcc in tcc to make the tcc instance that is basically then reentrant?  I've used this trick a while on things other than tcc, turning all global or static variables in any C program into an object that can be created or deleted or duplicated in a process space.

Charles

On Fri, Dec 6, 2019 at 11:46 AM ag <[hidden email]> wrote:
On Fri, Dec 06, at 03:42 Michael Matz wrote:
> Hello,
>
> On Tue, 3 Dec 2019, Ulrich Schmidt wrote:
>
> > i try to write a lua binding for tcc. To work out propperly, the tcc lib
> > needs to be reentrant.
>
> As demonstrated down-thread, that isn't correct.  It doesn't _need_ to be,
> it would be an feature.  As usual with features it needs to be measured
> against the downsides.  The downsides for your proposed changes are the
> following at least:
> 1) more complicated/boiler-platy source code of TCC (a TCCState
>    argument almost everywhere)
> 2) slower code: most of the time the indirection through a pointer
>    variable (the state) in comparison to a direct access to a static
>    variable doesn't matter.  But it does matter for the symbol/token
>    table (and potentially for the register/evaluation stack).  I have
>    measured this years ago for the token table, so this might or might not
>    still be the case.
>
> So, while I can see the wish for this feature, I don't necessarily see
> that tcc should be changed to accomodate.
>
> If anything I would expect a _complete_ transition to exist, in order to
> measure the impact.  The worst thing that could happen is if someone added
> TCCState arguments everywhere, moved some static variables to that state,
> and then leaves: none of the features of this whole excercise would be
> had, but all the downsides would be there.
>
> And yes, this is a big project.  I really think it would be better
> if you simply write a wrapper for libtcc that ensures single-threadedness
> and that regards TCCState as a singleton.  I think such thing would be
> well-suited in the TCC sources itself.
>
> (In a way it seems prudent for a tiny C compiler to only be usable as a
> singleton)

I maybe understand, that a C compiler would be best as a singleton, as a
state can influence unexpectedly the other states (unless (perhaps like in
this case as it would be under Lua control) there is a mechanism to handle
gracefully any errors)), and i can trust you about the "direct access to static
variables", but how about an (probably predefined (even with a compile option)
__array__ of states (under a single-thread), and expose it generiously and with
a pleasure to the user responsibility, without tcc guilties (if any)?
Perhaps can even implement this mechanism (to have a control of the environment)
by it's own.

Anyway C is unsafe by default (if it is that worry).

> Ciao,
> Michael.

Best,
  Αγαθοκλής

> >
> > I took a look into the sources and found some comments (XXX:...) and
> > started with removing
> >
> > the static var tcc_state. As a result allmost all lib functions needs a
> > 1st parameter of
> >
> > type TCCState*. I did this in my own local branch and tcc is still
> > running :).
> >
> > But this is a really HUGE change. in addition most of the local vars in
> > tccpp, tccgen, ... needs
> >
> > to be moved to TCCState. I can do that but at some points i will have
> > some questions and i
> >
> > can only test on windows and probably on linux.
> >
> > My 1st question is: Are you interested in these changes or should i do
> > this locally?
> >
> > I would like to this together with you.
> >
> >
> > Greetings.
> >
> > Ulrich.

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

grischka
In reply to this post by Michael Matz-4
Michael Matz wrote:

> Hello,
>
> On Tue, 3 Dec 2019, Ulrich Schmidt wrote:
>
>> i try to write a lua binding for tcc. To work out propperly, the tcc lib
>> needs to be reentrant.
>
> As demonstrated down-thread, that isn't correct.  It doesn't _need_ to be,
> it would be an feature.  As usual with features it needs to be measured
> against the downsides.  The downsides for your proposed changes are the
> following at least:
> 1) more complicated/boiler-platy source code of TCC (a TCCState
>    argument almost everywhere)
> 2) slower code: most of the time the indirection through a pointer
>    variable (the state) in comparison to a direct access to a static
>    variable doesn't matter.  But it does matter for the symbol/token
>    table (and potentially for the register/evaluation stack).  I have
>    measured this years ago for the token table, so this might or might not
>    still be the case.

I would like to add

   3) would get in the way with "git blame" / "show origin of line" almost always.

   4) would break probably every patch that people might have but didn't yet commit.

On the other hand I could imagine a "tinier" solution where everything
is made reentrant, except compilation itself.  Such that in order to be
thread-safe one would need to put a semaphore just around the "tcc_compile()"
function.

-- grischka

>
> So, while I can see the wish for this feature, I don't necessarily see
> that tcc should be changed to accomodate.
>
> If anything I would expect a _complete_ transition to exist, in order to
> measure the impact.  The worst thing that could happen is if someone added
> TCCState arguments everywhere, moved some static variables to that state,
> and then leaves: none of the features of this whole excercise would be
> had, but all the downsides would be there.
>
> And yes, this is a big project.  I really think it would be better
> if you simply write a wrapper for libtcc that ensures single-threadedness
> and that regards TCCState as a singleton.  I think such thing would be
> well-suited in the TCC sources itself.
>
> (In a way it seems prudent for a tiny C compiler to only be usable as a
> singleton)
>
>
> Ciao,
> Michael.
>
>> I took a look into the sources and found some comments (XXX:...) and
>> started with removing
>>
>> the static var tcc_state. As a result allmost all lib functions needs a
>> 1st parameter of
>>
>> type TCCState*. I did this in my own local branch and tcc is still
>> running :).
>>
>> But this is a really HUGE change. in addition most of the local vars in
>> tccpp, tccgen, ... needs
>>
>> to be moved to TCCState. I can do that but at some points i will have
>> some questions and i
>>
>> can only test on windows and probably on linux.
>>
>> My 1st question is: Are you interested in these changes or should i do
>> this locally?
>>
>> I would like to this together with you.
>>
>>
>> Greetings.
>>
>> Ulrich.

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Christian Jullien-3
In reply to this post by Michael Matz-4
> 2) slower code: most of the time the indirection through a pointer
   variable (the state) in comparison to a direct access to a static
   variable doesn't matter.

In fact, I experimented the opposite. When moving all global variables to a struct, my Lisp was around 1% faster because globals are now close together and more often accessible from L1 cache.
It has of course no effect when global is a pointer which introduces the same indirection.
It is true for aggressive optimizers which are likely to put struct pointer to a register.
So it may be faster for tcc compiled by gcc, clang or vc++ but slower when tcc is compiled by tcc.

C.

-----Original Message-----
From: Tinycc-devel [mailto:tinycc-devel-bounces+eligis=[hidden email]] On Behalf Of Michael Matz
Sent: Friday, December 06, 2019 16:42
To: TCC mailing list
Subject: Re: [Tinycc-devel] make tcc reentrant

Hello,

On Tue, 3 Dec 2019, Ulrich Schmidt wrote:

> i try to write a lua binding for tcc. To work out propperly, the tcc lib
> needs to be reentrant.

As demonstrated down-thread, that isn't correct.  It doesn't _need_ to be,
it would be an feature.  As usual with features it needs to be measured
against the downsides.  The downsides for your proposed changes are the
following at least:
1) more complicated/boiler-platy source code of TCC (a TCCState
   argument almost everywhere)
2) slower code: most of the time the indirection through a pointer
   variable (the state) in comparison to a direct access to a static
   variable doesn't matter.  But it does matter for the symbol/token
   table (and potentially for the register/evaluation stack).  I have
   measured this years ago for the token table, so this might or might not
   still be the case.

So, while I can see the wish for this feature, I don't necessarily see
that tcc should be changed to accomodate.

If anything I would expect a _complete_ transition to exist, in order to
measure the impact.  The worst thing that could happen is if someone added
TCCState arguments everywhere, moved some static variables to that state,
and then leaves: none of the features of this whole excercise would be
had, but all the downsides would be there.

And yes, this is a big project.  I really think it would be better
if you simply write a wrapper for libtcc that ensures single-threadedness
and that regards TCCState as a singleton.  I think such thing would be
well-suited in the TCC sources itself.

(In a way it seems prudent for a tiny C compiler to only be usable as a
singleton)


Ciao,
Michael.

>
> I took a look into the sources and found some comments (XXX:...) and
> started with removing
>
> the static var tcc_state. As a result allmost all lib functions needs a
> 1st parameter of
>
> type TCCState*. I did this in my own local branch and tcc is still
> running :).
>
> But this is a really HUGE change. in addition most of the local vars in
> tccpp, tccgen, ... needs
>
> to be moved to TCCState. I can do that but at some points i will have
> some questions and i
>
> can only test on windows and probably on linux.
>
> My 1st question is: Are you interested in these changes or should i do
> this locally?
>
> I would like to this together with you.
>
>
> Greetings.
>
> Ulrich.
>
>
> _______________________________________________
> Tinycc-devel mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/tinycc-devel
>

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel


_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Domingo Alvarez Duarte-2
In reply to this post by Charles Lohr

Hello !

I went to the trouble of making tcc fully reentrant but it was not accepted https://github.com/mingodad/tinycc which is sad.

Cheers !

On 6/12/19 22:16, Charles Lohr wrote:
Is there a reason you don't just compile tcc in tcc to make the tcc instance that is basically then reentrant?  I've used this trick a while on things other than tcc, turning all global or static variables in any C program into an object that can be created or deleted or duplicated in a process space.

Charles

On Fri, Dec 6, 2019 at 11:46 AM ag <[hidden email]> wrote:
On Fri, Dec 06, at 03:42 Michael Matz wrote:
> Hello,
>
> On Tue, 3 Dec 2019, Ulrich Schmidt wrote:
>
> > i try to write a lua binding for tcc. To work out propperly, the tcc lib
> > needs to be reentrant.
>
> As demonstrated down-thread, that isn't correct.  It doesn't _need_ to be,
> it would be an feature.  As usual with features it needs to be measured
> against the downsides.  The downsides for your proposed changes are the
> following at least:
> 1) more complicated/boiler-platy source code of TCC (a TCCState
>    argument almost everywhere)
> 2) slower code: most of the time the indirection through a pointer
>    variable (the state) in comparison to a direct access to a static
>    variable doesn't matter.  But it does matter for the symbol/token
>    table (and potentially for the register/evaluation stack).  I have
>    measured this years ago for the token table, so this might or might not
>    still be the case.
>
> So, while I can see the wish for this feature, I don't necessarily see
> that tcc should be changed to accomodate.
>
> If anything I would expect a _complete_ transition to exist, in order to
> measure the impact.  The worst thing that could happen is if someone added
> TCCState arguments everywhere, moved some static variables to that state,
> and then leaves: none of the features of this whole excercise would be
> had, but all the downsides would be there.
>
> And yes, this is a big project.  I really think it would be better
> if you simply write a wrapper for libtcc that ensures single-threadedness
> and that regards TCCState as a singleton.  I think such thing would be
> well-suited in the TCC sources itself.
>
> (In a way it seems prudent for a tiny C compiler to only be usable as a
> singleton)

I maybe understand, that a C compiler would be best as a singleton, as a
state can influence unexpectedly the other states (unless (perhaps like in
this case as it would be under Lua control) there is a mechanism to handle
gracefully any errors)), and i can trust you about the "direct access to static
variables", but how about an (probably predefined (even with a compile option)
__array__ of states (under a single-thread), and expose it generiously and with
a pleasure to the user responsibility, without tcc guilties (if any)?
Perhaps can even implement this mechanism (to have a control of the environment)
by it's own.

Anyway C is unsafe by default (if it is that worry).

> Ciao,
> Michael.

Best,
  Αγαθοκλής

> >
> > I took a look into the sources and found some comments (XXX:...) and
> > started with removing
> >
> > the static var tcc_state. As a result allmost all lib functions needs a
> > 1st parameter of
> >
> > type TCCState*. I did this in my own local branch and tcc is still
> > running :).
> >
> > But this is a really HUGE change. in addition most of the local vars in
> > tccpp, tccgen, ... needs
> >
> > to be moved to TCCState. I can do that but at some points i will have
> > some questions and i
> >
> > can only test on windows and probably on linux.
> >
> > My 1st question is: Are you interested in these changes or should i do
> > this locally?
> >
> > I would like to this together with you.
> >
> >
> > Greetings.
> >
> > Ulrich.

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Ulrich Schmidt
Hello.

After reading all your posts. I see there are 2 scenarios/goals in the
comunity.

scenario 1:
I want a tiny and fast c compiler (tcc/tcc.exe).
In this case the TCCSTate struct is not needed at all and dereferencing
TCCState fields slows down execution. Making TCCStruct fields local
would speed things up. libtcc.c needs to become compiled statically into
tcc.exe. You can execute tcc many times parallel without trouble and it
is fast as it can be.

scenario 2:
I want to use libtcc in my own application(libtcc.so/libtcc.so).
This library HAS to be reentrant and we need the TCCStruct and all
compiling state related values needs to be stored inside this struct.

These 2 scenarios result in different decisions about TCCState. Many
years ago (more than 6) someone decited to make tcc functionality
useable from other apps, put all relevant conde into libbtcc.c, defined
a api,created this TCCStruct and started the transition process. (you
can still find the comments //XXX: in the sources) this process never
finished.

The question is, how we deal with the situation. Right now i would not
provide a separate libtcc until the transition process came to a end.
Right now libtcc needs to be compiled statically into tcc.

So how can we comply with scenario 1 AND 2 at the same time? (just a
quick thought)
- make the state values local or struct fields by using defines and
macros...

- The in scenario1 unloved 1 function parameter can be ignored and the
values stored locally:
   #define LIBPARAM1
   #define VALUE(v) v
   ..

   somereturn somefunc(LIBPARAM1 ...) {
     VALUE(count) = 0;
     ...

- in scenario 2 we define the 2 macros different:
   #define LIBPARAM1 TCCState *s1,
   #define VALUE(v) s1->v

Right now tcc does not fit scenario 1 nor 2.

Ulrich.


_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: make tcc reentrant

Daniel Glöckner-2
Hi,

On Sat, Dec 07, 2019 at 11:34:11AM +0100, Ulrich Schmidt wrote:

> - The in scenario1 unloved 1 function parameter can be ignored and the
> values stored locally:
>   #define LIBPARAM1
>   #define VALUE(v) v
>   ..
>
>   somereturn somefunc(LIBPARAM1 ...) {
>     VALUE(count) = 0;
>     ...
>
> - in scenario 2 we define the 2 macros different:
>   #define LIBPARAM1 TCCState *s1,
>   #define VALUE(v) s1->v

We need a macro

#if scenario == 1
#define PASSPARAM1
#else
#define PASSPARAM1 s1,
#endif

Grouping all globals in the TCCState structure might be beneficial
even if we don't switch to those macros, because of the cache locality
Christian wrote about. The VALUE macro in scenario 1 would then be
#define VALUE(v) global_state.v

Btw, has anyone thought about handling TCCState like the context is
handled in the OpenGL API? It has one function to select the context
to use for following operations and the operations themselves don't
have a parameter for the context. If the pointer to the TCCState is
stored in a normal or in a TLS variable is up to us and might even
be a compile time option (that is set to "normal" for compilation
with tcc as long as tcc doesn't know how to generate TLS variables).

Best regards,

  Daniel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
123