Almost added a feature, but I broke things

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Almost added a feature, but I broke things

Joshua Scholar
If anyone who knows this project better than me can help it would be wonderful.

I decided to add a feature I figured a lot of people would like, that libtcc can hold a virtual read-only file system, so that you don't actually need to have an include, lib and libtcc directory for a project to use libtcc.

Warning, I'm a windows user, so I'm probably doing this on a different system than most of you.

Now that I think about it, my choice to do it by embedding a zip file and linking miniunzip and libz in was probably a poor one for efficiency.  Those parts are working, so I'll leave it alone until I get the bugs I caused out, unzipping, buffering the unzipped data every time, allocating and deallocating the memory for those buffers and putting a global lock on the miniunzip calls waste time, and if people want to use libtcc as something like a jit, they probably would prefer the speed to the memory. 

Also, and even more convincing, embedding libz and the latest version of minizip (which has forked off from libz) into the project and making it work on every platform would be a nightmare.

But never mind that for now, my current state is:
tcc works for making .o files and .a files and the results are byte equivalent the previous version - and it doesn't need the directories to exist to do it.

But libtcc_test silently fails, and I can't get this version of tcc to make exe files and -run doesn't work.

But since I didn't make a visual studio solution file, I'm just working off the command line, I haven't been able to use a debugger and I'm lost for why everything compiles without complaint and runs without complaint, but not everything works.

To get help from the compiler I replaced the int type for posix handles with a struct type for a wrapped handle that can read from this virtual file system.  Since the types can't be substituted without error, I should have caught all the spots where I need to change things.

Joshua Scholar

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: Almost added a feature, but I broke things

Kyryl Melekhin
>I figured a lot of people would like, that >libtcc can hold a virtual read-only file >system, so that you don't actually need to >have an include, lib and libtcc directory for >a project to use libtcc.

Is that so? I can kind of see where you are coming from, but this is not the right way to solve it. It creates unnecessary dependency bloat, and even a lot more headache for anybody willing to take time and read though that. If you want I can send you my custom amalgamated version of TCC which is literally 1 c file, does not depend on tcclib, because it has it in it's source already, all the assembly routines for stuff like alloca, are written in extended asm syntax and they work. There is no need to run make to compile, just gcc tinycc.c and some platform flags is enough. You also don't need to worry about header files, because you can always make your own, just write extern printf(...) bla bla and it will work. That's basically how I use tcc, it's so much easier to work with and maintain especially now that i have wrote a special version of tcc which can correctly amalgamate any C source code just like compiler sees it, no assumptions because tcc implements complete preprocessor spec. 

On Thu, Dec 24, 2020, 06:29 Joshua Scholar <[hidden email]> wrote:
If anyone who knows this project better than me can help it would be wonderful.

I decided to add a feature I figured a lot of people would like, that libtcc can hold a virtual read-only file system, so that you don't actually need to have an include, lib and libtcc directory for a project to use libtcc.

Warning, I'm a windows user, so I'm probably doing this on a different system than most of you.

Now that I think about it, my choice to do it by embedding a zip file and linking miniunzip and libz in was probably a poor one for efficiency.  Those parts are working, so I'll leave it alone until I get the bugs I caused out, unzipping, buffering the unzipped data every time, allocating and deallocating the memory for those buffers and putting a global lock on the miniunzip calls waste time, and if people want to use libtcc as something like a jit, they probably would prefer the speed to the memory. 

Also, and even more convincing, embedding libz and the latest version of minizip (which has forked off from libz) into the project and making it work on every platform would be a nightmare.

But never mind that for now, my current state is:
tcc works for making .o files and .a files and the results are byte equivalent the previous version - and it doesn't need the directories to exist to do it.

But libtcc_test silently fails, and I can't get this version of tcc to make exe files and -run doesn't work.

But since I didn't make a visual studio solution file, I'm just working off the command line, I haven't been able to use a debugger and I'm lost for why everything compiles without complaint and runs without complaint, but not everything works.

To get help from the compiler I replaced the int type for posix handles with a struct type for a wrapped handle that can read from this virtual file system.  Since the types can't be substituted without error, I should have caught all the spots where I need to change things.

Joshua Scholar
_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: Almost added a feature, but I broke things

tinycc-devel mailing list

>Is that so? I can kind of see where you are coming from, but this is not the right way to solve it. It creates unnecessary dependency bloat, and even a lot >more headache for anybody willing to take time and read though that. If you want I can send you my custom amalgamated version of TCC which is >literally 1 c file, does not depend on tcclib, because it has it in it's source already, all the assembly routines for stuff like alloca, are written in extended >asm syntax and they work. There is no need to run make to compile, just gcc tinycc.c and some platform flags is enough. You also don't need to worry >about header files, because you can always make your own, just write extern printf(...) bla bla and it will work. That's basically how I use tcc, it's so much >easier to work with and maintain especially now that i have wrote a special version of tcc which can correctly amalgamate any C source code just like >compiler sees it, no assumptions because tcc implements complete preprocessor spec.

That sounds really cool Kyryl. I'd be interested to see it. Can you please share your special version of tcc that's able to amalgamate any C source code?

On 24/12/2020 15:38, Kyryl Melekhin wrote:
>I figured a lot of people would like, that >libtcc can hold a virtual read-only file >system, so that you don't actually need to >have an include, lib and libtcc directory for >a project to use libtcc.

Is that so? I can kind of see where you are coming from, but this is not the right way to solve it. It creates unnecessary dependency bloat, and even a lot more headache for anybody willing to take time and read though that. If you want I can send you my custom amalgamated version of TCC which is literally 1 c file, does not depend on tcclib, because it has it in it's source already, all the assembly routines for stuff like alloca, are written in extended asm syntax and they work. There is no need to run make to compile, just gcc tinycc.c and some platform flags is enough. You also don't need to worry about header files, because you can always make your own, just write extern printf(...) bla bla and it will work. That's basically how I use tcc, it's so much easier to work with and maintain especially now that i have wrote a special version of tcc which can correctly amalgamate any C source code just like compiler sees it, no assumptions because tcc implements complete preprocessor spec. 

On Thu, Dec 24, 2020, 06:29 Joshua Scholar <[hidden email]> wrote:
If anyone who knows this project better than me can help it would be wonderful.

I decided to add a feature I figured a lot of people would like, that libtcc can hold a virtual read-only file system, so that you don't actually need to have an include, lib and libtcc directory for a project to use libtcc.

Warning, I'm a windows user, so I'm probably doing this on a different system than most of you.

Now that I think about it, my choice to do it by embedding a zip file and linking miniunzip and libz in was probably a poor one for efficiency.  Those parts are working, so I'll leave it alone until I get the bugs I caused out, unzipping, buffering the unzipped data every time, allocating and deallocating the memory for those buffers and putting a global lock on the miniunzip calls waste time, and if people want to use libtcc as something like a jit, they probably would prefer the speed to the memory. 

Also, and even more convincing, embedding libz and the latest version of minizip (which has forked off from libz) into the project and making it work on every platform would be a nightmare.

But never mind that for now, my current state is:
tcc works for making .o files and .a files and the results are byte equivalent the previous version - and it doesn't need the directories to exist to do it.

But libtcc_test silently fails, and I can't get this version of tcc to make exe files and -run doesn't work.

But since I didn't make a visual studio solution file, I'm just working off the command line, I haven't been able to use a debugger and I'm lost for why everything compiles without complaint and runs without complaint, but not everything works.

To get help from the compiler I replaced the int type for posix handles with a struct type for a wrapped handle that can read from this virtual file system.  Since the types can't be substituted without error, I should have caught all the spots where I need to change things.

Joshua Scholar
_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: Almost added a feature, but I broke things

tinycc-devel mailing list
In reply to this post by Kyryl Melekhin

>Is that so? I can kind of see where you are coming from, but this is not the right way to solve it. It creates unnecessary dependency bloat, and even a lot >more headache for anybody willing to take time and read though that. If you want I can send you my custom amalgamated version of TCC which is >literally 1 c file, does not depend on tcclib, because it has it in it's source already, all the assembly routines for stuff like alloca, are written in extended >asm syntax and they work. There is no need to run make to compile, just gcc tinycc.c and some platform flags is enough. You also don't need to worry >about header files, because you can always make your own, just write extern printf(...) bla bla and it will work. That's basically how I use tcc, it's so much >easier to work with and maintain especially now that i have wrote a special version of tcc which can correctly amalgamate any C source code just like >compiler sees it, no assumptions because tcc implements complete preprocessor spec.

That sounds really cool Kyryl. I'd be interested to see your special version of TinyC that's able to amalgamate any C source code. Is it the same amalgamated tinycc.c or a different version?



On 24/12/2020 15:38, Kyryl Melekhin wrote:
>I figured a lot of people would like, that >libtcc can hold a virtual read-only file >system, so that you don't actually need to >have an include, lib and libtcc directory for >a project to use libtcc.

Is that so? I can kind of see where you are coming from, but this is not the right way to solve it. It creates unnecessary dependency bloat, and even a lot more headache for anybody willing to take time and read though that. If you want I can send you my custom amalgamated version of TCC which is literally 1 c file, does not depend on tcclib, because it has it in it's source already, all the assembly routines for stuff like alloca, are written in extended asm syntax and they work. There is no need to run make to compile, just gcc tinycc.c and some platform flags is enough. You also don't need to worry about header files, because you can always make your own, just write extern printf(...) bla bla and it will work. That's basically how I use tcc, it's so much easier to work with and maintain especially now that i have wrote a special version of tcc which can correctly amalgamate any C source code just like compiler sees it, no assumptions because tcc implements complete preprocessor spec. 

On Thu, Dec 24, 2020, 06:29 Joshua Scholar <[hidden email]> wrote:
If anyone who knows this project better than me can help it would be wonderful.

I decided to add a feature I figured a lot of people would like, that libtcc can hold a virtual read-only file system, so that you don't actually need to have an include, lib and libtcc directory for a project to use libtcc.

Warning, I'm a windows user, so I'm probably doing this on a different system than most of you.

Now that I think about it, my choice to do it by embedding a zip file and linking miniunzip and libz in was probably a poor one for efficiency.  Those parts are working, so I'll leave it alone until I get the bugs I caused out, unzipping, buffering the unzipped data every time, allocating and deallocating the memory for those buffers and putting a global lock on the miniunzip calls waste time, and if people want to use libtcc as something like a jit, they probably would prefer the speed to the memory. 

Also, and even more convincing, embedding libz and the latest version of minizip (which has forked off from libz) into the project and making it work on every platform would be a nightmare.

But never mind that for now, my current state is:
tcc works for making .o files and .a files and the results are byte equivalent the previous version - and it doesn't need the directories to exist to do it.

But libtcc_test silently fails, and I can't get this version of tcc to make exe files and -run doesn't work.

But since I didn't make a visual studio solution file, I'm just working off the command line, I haven't been able to use a debugger and I'm lost for why everything compiles without complaint and runs without complaint, but not everything works.

To get help from the compiler I replaced the int type for posix handles with a struct type for a wrapped handle that can read from this virtual file system.  Since the types can't be substituted without error, I should have caught all the spots where I need to change things.

Joshua Scholar
_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: Almost added a feature, but I broke things

Joshua Scholar
In reply to this post by Kyryl Melekhin
It took me a while to understand your post, I've been up all night debugging.  And I've isolated my bug... 

Anyway, I see what you did, and it's cool, but it doesn't match my use-case which is for built in compiler, they call them "jits" these days, although what I want isn't technically a jit.
I mean more of a system where you can keep adding code even as the old code is still running, the sort of thing that used to be more popular with languages like Lisp, or Smalltalk or Forth.  That's not called a jits generally let programs be interpreted for a while and then compile code while it's running.  These old languages compiled code as soon as it was entered, it's just that you could add code without deleting the data that was already there, and in the case of the more interactive systems, you could even have some of the code already running while you're adding to it.  And there wasn't an inherent distinction between the development environment or even the windowing system and the user's code.  I once saw someone change how selected text is rendered in a Smalltalk system by changing the editor's code from within the editor.

That's the sort of system where you don't want all your source code concatenated and compiled at once like you're doing, you want to compile a bunch of tiny routines and have them access each other.

Actually I can see there's a problem with the run time system in that case, because you want routines to share run time code that's already sitting in memory, not link to a new run time system each time.

C isn't designed for that.

I asked about that before and didn't get an answer, probably because what I was imagining is so foreign to how C works.

C isn't designed to link to a run time system once, and then slowly add more routines to a live system, reusing the symbol table for the existing run time and the already compiled routines.  

I guess I'm going to have to replace the runtime code with stubs that share.

But I can see a use for some of what you're doing, concatenating header files and doing any preprocessing on them only once.  A kind of not very sophisticated precompiled header.

Josh Scholar

On Thu, Dec 24, 2020 at 7:39 AM Kyryl Melekhin <[hidden email]> wrote:
>I figured a lot of people would like, that >libtcc can hold a virtual read-only file >system, so that you don't actually need to >have an include, lib and libtcc directory for >a project to use libtcc.

Is that so? I can kind of see where you are coming from, but this is not the right way to solve it. It creates unnecessary dependency bloat, and even a lot more headache for anybody willing to take time and read though that. If you want I can send you my custom amalgamated version of TCC which is literally 1 c file, does not depend on tcclib, because it has it in it's source already, all the assembly routines for stuff like alloca, are written in extended asm syntax and they work. There is no need to run make to compile, just gcc tinycc.c and some platform flags is enough. You also don't need to worry about header files, because you can always make your own, just write extern printf(...) bla bla and it will work. That's basically how I use tcc, it's so much easier to work with and maintain especially now that i have wrote a special version of tcc which can correctly amalgamate any C source code just like compiler sees it, no assumptions because tcc implements complete preprocessor spec. 

On Thu, Dec 24, 2020, 06:29 Joshua Scholar <[hidden email]> wrote:
If anyone who knows this project better than me can help it would be wonderful.

I decided to add a feature I figured a lot of people would like, that libtcc can hold a virtual read-only file system, so that you don't actually need to have an include, lib and libtcc directory for a project to use libtcc.

Warning, I'm a windows user, so I'm probably doing this on a different system than most of you.

Now that I think about it, my choice to do it by embedding a zip file and linking miniunzip and libz in was probably a poor one for efficiency.  Those parts are working, so I'll leave it alone until I get the bugs I caused out, unzipping, buffering the unzipped data every time, allocating and deallocating the memory for those buffers and putting a global lock on the miniunzip calls waste time, and if people want to use libtcc as something like a jit, they probably would prefer the speed to the memory. 

Also, and even more convincing, embedding libz and the latest version of minizip (which has forked off from libz) into the project and making it work on every platform would be a nightmare.

But never mind that for now, my current state is:
tcc works for making .o files and .a files and the results are byte equivalent the previous version - and it doesn't need the directories to exist to do it.

But libtcc_test silently fails, and I can't get this version of tcc to make exe files and -run doesn't work.

But since I didn't make a visual studio solution file, I'm just working off the command line, I haven't been able to use a debugger and I'm lost for why everything compiles without complaint and runs without complaint, but not everything works.

To get help from the compiler I replaced the int type for posix handles with a struct type for a wrapped handle that can read from this virtual file system.  Since the types can't be substituted without error, I should have caught all the spots where I need to change things.

Joshua Scholar
_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: Almost added a feature, but I broke things

Kyryl Melekhin
In reply to this post by tinycc-devel mailing list
https://raw.githubusercontent.com/kyx0r/klec/master/utils/tinycc.c

on x86_64 linux compile like so gcc tinycc.c -ldl -lpthread
then you can use it like so:

if the project already has something resembling a unity build:
a.out -E file1.c > file2.c

if the project does incremental build:
use cat on every C file in correct order ignore the headers.
cat *.c > file1.c
a.out -E file1.c > file2.c


To better understand how it works I recommend to try it on tcc
code base first. Tcc code actually is layed out in an easy format
despite make forcing the incremental build.
a.out -E tcc.c > tinycc.c
This current version the result should amount to approx 87K loc

Some comments:
1. If the headers in C source are not guarded properly
it will not work, but so will your project not compile also. So
this does not happen. Header exclusion works the same way a compiler
would exclude it.
2. Problem of amalgamation is not as trivial as you think it might be,
because of the nature of how C works your header might be guarded but
you can also have code outside of that guard, in any case my program will
exclude the code properly. Especially that #endif is used to terminate any kind
of preprocessing expression.
3. By default headers with <> (system headers) are not amalgamated.
You can enable that, just read the source code in tcc_preprocess.
This program is powered by strategically placing printfs inside tcc
and some compiler logic changes of the default option -E, so don't try
to use this tcc for anything except what is meant to be.
4. While in most cases the resulting file will compile, some projects might be
weird and still require some manual tweaks and edits. Also you might need
to spend some time to clean the code from extra newlines that might get
created.
5. You can also configure it in source code to instead of processing everything
actually do the preprocess and exclude all the unnecessary junk, so for example
with that tcc source for x64 linux will be about ~35K instead of 87K.

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: Almost added a feature, but I broke things

Joshua Scholar
I guess I misunderstood, I thought that your file was all of the header files and the runtime placed into one source file so that you could have tcc or libtcc be part of a project and make the product/or tcc  be a single executable that doesn't need directories.
ie. without needing link files for the run time and without needing a directory tree of include files, (or the same for a program using libtcc internally).

But this is just another way of BUILDING tcc. 

I wanted a project that USES libtcc without needing a directory of standard includes and without needing lib files - ie a single file product.  I wanted to USE libtcc, I didn't want a project that that BUILDS tcc.  

And since your file starts with a number of standard includes, your file doesn't even stop people from needing all of directories of includes, it just saves them from needing make? Or this is so that people can use the equivalent of libtcc without knowing how to link a library into their own program?

So this isn't for making products that use lib tcc, it's for developers who hate make and for some strange reason don't want to link?

I don't get it, what sort of meta-product needs to compile tcc over and over?

Is this just to save time on running make on your personal project that changes tcc?  And the cost of that is that tcc is one huge file?  That still doesn't tell me why you smashed tcc's headers into it since you need to include tcc compatible versions of the header files anyway.  You don't want people to need make, but they'll still need gnu C for the headers?



Anyway I GOT MY ADDON WORKING, there were just a bunch of typos.

But I'm gonna rewrite it.  Instead of making the embedded directory a zip file and adding a bunch of libraries to handle zip files, I'm gonna make it a tar file (uncompressed),  write a program that converts a tar file into a .c file (so my program won't need an external tar file) and just write c code that can make an index into a tar file in memory.

And I'll keep my change to libtcc so that it searches my virtual file first.

That will mean that the feature needs minimal changes to the project and it will make a compiler that's much faster than a version that reads from a compressed archive.

Just add a flag to the make (I have it as -s on the windows bat file make) and it will build versions of tcc and libtcc that don't need directories of include and libraries.
There are some things that won't work.  You can't load a dynamic link library out of an archive.

But my idea of using libtcc as a jit needs work on exporting the run time, I think.  Though I noticed libtcc reading a file called "msvcrt.def", so maybe it's doing something clever and reusing the runtime library of embedded program? 

On Thu, Dec 24, 2020 at 3:22 PM Kyryl Melekhin <[hidden email]> wrote:
https://raw.githubusercontent.com/kyx0r/klec/master/utils/tinycc.c

on x86_64 linux compile like so gcc tinycc.c -ldl -lpthread
then you can use it like so:

if the project already has something resembling a unity build:
a.out -E file1.c > file2.c

if the project does incremental build:
use cat on every C file in correct order ignore the headers.
cat *.c > file1.c
a.out -E file1.c > file2.c


To better understand how it works I recommend to try it on tcc
code base first. Tcc code actually is layed out in an easy format
despite make forcing the incremental build.
a.out -E tcc.c > tinycc.c
This current version the result should amount to approx 87K loc

Some comments:
1. If the headers in C source are not guarded properly
it will not work, but so will your project not compile also. So
this does not happen. Header exclusion works the same way a compiler
would exclude it.
2. Problem of amalgamation is not as trivial as you think it might be,
because of the nature of how C works your header might be guarded but
you can also have code outside of that guard, in any case my program will
exclude the code properly. Especially that #endif is used to terminate any kind
of preprocessing expression.
3. By default headers with <> (system headers) are not amalgamated.
You can enable that, just read the source code in tcc_preprocess.
This program is powered by strategically placing printfs inside tcc
and some compiler logic changes of the default option -E, so don't try
to use this tcc for anything except what is meant to be.
4. While in most cases the resulting file will compile, some projects might be
weird and still require some manual tweaks and edits. Also you might need
to spend some time to clean the code from extra newlines that might get
created.
5. You can also configure it in source code to instead of processing everything
actually do the preprocess and exclude all the unnecessary junk, so for example
with that tcc source for x64 linux will be about ~35K instead of 87K.

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: Almost added a feature, but I broke things

Kyryl Melekhin
This isn't a build system you are misunderstanding a lot of things, it's a special tool that can save you a lot of time if you ever need to correctly combine C source or libraries into one file. The best build system is having no build system at all. Also files are annoying to a lot of people and hurt your understanding of the code as a whole. Thats why single header libraries are so popular. Sometimes the best way to read the source code of some C project is to combine everything into one file and explore the code in its entirety without wasting your time memorizing where every function is located, what file and subfolders. Sometimes to gain complete understanding of complex system it's better to strip it down to the "suckless" level. Also if your editor can't handle large files, it's better to reconsider your tools or write your own that suck less. There is always some kind of missing data and indirection between the original software authors and you, what made sense to them when building a project will not be something you will be able to understand on your own. Most of the time developers split code into files for no reason whatsoever and cryptic names dont help. I don't recall ever having filename help me in understanding the code of example. It's just for esthetic organizational purposes only, but they are useless when it comes to being productive. Also dont even tell me about multicore builds, those are sucky too, most of the time you will be hit by O(n) slow linker, which makes whole thing even slower in some cases. That's why I use Tcc, compiler should be fast enough to handle massive codebases without a chug. If tcc compiles 200K loc of code in less than a second, I don't think any reasonable user land applications should have any justifications for using build systems because such a compiler exists.

Anyway, I was actually replying to the person who asked me about the amalgamation, but I figured might as well tell you bit about my philosophy that I've come to form over the years. 

On Thu, Dec 24, 2020, 19:19 Joshua Scholar <[hidden email]> wrote:
I guess I misunderstood, I thought that your file was all of the header files and the runtime placed into one source file so that you could have tcc or libtcc be part of a project and make the product/or tcc  be a single executable that doesn't need directories.
ie. without needing link files for the run time and without needing a directory tree of include files, (or the same for a program using libtcc internally).

But this is just another way of BUILDING tcc. 

I wanted a project that USES libtcc without needing a directory of standard includes and without needing lib files - ie a single file product.  I wanted to USE libtcc, I didn't want a project that that BUILDS tcc.  

And since your file starts with a number of standard includes, your file doesn't even stop people from needing all of directories of includes, it just saves them from needing make? Or this is so that people can use the equivalent of libtcc without knowing how to link a library into their own program?

So this isn't for making products that use lib tcc, it's for developers who hate make and for some strange reason don't want to link?

I don't get it, what sort of meta-product needs to compile tcc over and over?

Is this just to save time on running make on your personal project that changes tcc?  And the cost of that is that tcc is one huge file?  That still doesn't tell me why you smashed tcc's headers into it since you need to include tcc compatible versions of the header files anyway.  You don't want people to need make, but they'll still need gnu C for the headers?



Anyway I GOT MY ADDON WORKING, there were just a bunch of typos.

But I'm gonna rewrite it.  Instead of making the embedded directory a zip file and adding a bunch of libraries to handle zip files, I'm gonna make it a tar file (uncompressed),  write a program that converts a tar file into a .c file (so my program won't need an external tar file) and just write c code that can make an index into a tar file in memory.

And I'll keep my change to libtcc so that it searches my virtual file first.

That will mean that the feature needs minimal changes to the project and it will make a compiler that's much faster than a version that reads from a compressed archive.

Just add a flag to the make (I have it as -s on the windows bat file make) and it will build versions of tcc and libtcc that don't need directories of include and libraries.
There are some things that won't work.  You can't load a dynamic link library out of an archive.

But my idea of using libtcc as a jit needs work on exporting the run time, I think.  Though I noticed libtcc reading a file called "msvcrt.def", so maybe it's doing something clever and reusing the runtime library of embedded program? 

On Thu, Dec 24, 2020 at 3:22 PM Kyryl Melekhin <[hidden email]> wrote:
https://raw.githubusercontent.com/kyx0r/klec/master/utils/tinycc.c

on x86_64 linux compile like so gcc tinycc.c -ldl -lpthread
then you can use it like so:

if the project already has something resembling a unity build:
a.out -E file1.c > file2.c

if the project does incremental build:
use cat on every C file in correct order ignore the headers.
cat *.c > file1.c
a.out -E file1.c > file2.c


To better understand how it works I recommend to try it on tcc
code base first. Tcc code actually is layed out in an easy format
despite make forcing the incremental build.
a.out -E tcc.c > tinycc.c
This current version the result should amount to approx 87K loc

Some comments:
1. If the headers in C source are not guarded properly
it will not work, but so will your project not compile also. So
this does not happen. Header exclusion works the same way a compiler
would exclude it.
2. Problem of amalgamation is not as trivial as you think it might be,
because of the nature of how C works your header might be guarded but
you can also have code outside of that guard, in any case my program will
exclude the code properly. Especially that #endif is used to terminate any kind
of preprocessing expression.
3. By default headers with <> (system headers) are not amalgamated.
You can enable that, just read the source code in tcc_preprocess.
This program is powered by strategically placing printfs inside tcc
and some compiler logic changes of the default option -E, so don't try
to use this tcc for anything except what is meant to be.
4. While in most cases the resulting file will compile, some projects might be
weird and still require some manual tweaks and edits. Also you might need
to spend some time to clean the code from extra newlines that might get
created.
5. You can also configure it in source code to instead of processing everything
actually do the preprocess and exclude all the unnecessary junk, so for example
with that tcc source for x64 linux will be about ~35K instead of 87K.

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: Almost added a feature, but I broke things

Joshua Scholar
Heh, people told me I was crazy to make my chess engine a single file.

On Thu, Dec 24, 2020 at 5:00 PM Kyryl Melekhin <[hidden email]> wrote:
This isn't a build system you are misunderstanding a lot of things, it's a special tool that can save you a lot of time if you ever need to correctly combine C source or libraries into one file. The best build system is having no build system at all. Also files are annoying to a lot of people and hurt your understanding of the code as a whole. Thats why single header libraries are so popular. Sometimes the best way to read the source code of some C project is to combine everything into one file and explore the code in its entirety without wasting your time memorizing where every function is located, what file and subfolders. Sometimes to gain complete understanding of complex system it's better to strip it down to the "suckless" level. Also if your editor can't handle large files, it's better to reconsider your tools or write your own that suck less. There is always some kind of missing data and indirection between the original software authors and you, what made sense to them when building a project will not be something you will be able to understand on your own. Most of the time developers split code into files for no reason whatsoever and cryptic names dont help. I don't recall ever having filename help me in understanding the code of example. It's just for esthetic organizational purposes only, but they are useless when it comes to being productive. Also dont even tell me about multicore builds, those are sucky too, most of the time you will be hit by O(n) slow linker, which makes whole thing even slower in some cases. That's why I use Tcc, compiler should be fast enough to handle massive codebases without a chug. If tcc compiles 200K loc of code in less than a second, I don't think any reasonable user land applications should have any justifications for using build systems because such a compiler exists.

Anyway, I was actually replying to the person who asked me about the amalgamation, but I figured might as well tell you bit about my philosophy that I've come to form over the years. 

On Thu, Dec 24, 2020, 19:19 Joshua Scholar <[hidden email]> wrote:
I guess I misunderstood, I thought that your file was all of the header files and the runtime placed into one source file so that you could have tcc or libtcc be part of a project and make the product/or tcc  be a single executable that doesn't need directories.
ie. without needing link files for the run time and without needing a directory tree of include files, (or the same for a program using libtcc internally).

But this is just another way of BUILDING tcc. 

I wanted a project that USES libtcc without needing a directory of standard includes and without needing lib files - ie a single file product.  I wanted to USE libtcc, I didn't want a project that that BUILDS tcc.  

And since your file starts with a number of standard includes, your file doesn't even stop people from needing all of directories of includes, it just saves them from needing make? Or this is so that people can use the equivalent of libtcc without knowing how to link a library into their own program?

So this isn't for making products that use lib tcc, it's for developers who hate make and for some strange reason don't want to link?

I don't get it, what sort of meta-product needs to compile tcc over and over?

Is this just to save time on running make on your personal project that changes tcc?  And the cost of that is that tcc is one huge file?  That still doesn't tell me why you smashed tcc's headers into it since you need to include tcc compatible versions of the header files anyway.  You don't want people to need make, but they'll still need gnu C for the headers?



Anyway I GOT MY ADDON WORKING, there were just a bunch of typos.

But I'm gonna rewrite it.  Instead of making the embedded directory a zip file and adding a bunch of libraries to handle zip files, I'm gonna make it a tar file (uncompressed),  write a program that converts a tar file into a .c file (so my program won't need an external tar file) and just write c code that can make an index into a tar file in memory.

And I'll keep my change to libtcc so that it searches my virtual file first.

That will mean that the feature needs minimal changes to the project and it will make a compiler that's much faster than a version that reads from a compressed archive.

Just add a flag to the make (I have it as -s on the windows bat file make) and it will build versions of tcc and libtcc that don't need directories of include and libraries.
There are some things that won't work.  You can't load a dynamic link library out of an archive.

But my idea of using libtcc as a jit needs work on exporting the run time, I think.  Though I noticed libtcc reading a file called "msvcrt.def", so maybe it's doing something clever and reusing the runtime library of embedded program? 

On Thu, Dec 24, 2020 at 3:22 PM Kyryl Melekhin <[hidden email]> wrote:
https://raw.githubusercontent.com/kyx0r/klec/master/utils/tinycc.c

on x86_64 linux compile like so gcc tinycc.c -ldl -lpthread
then you can use it like so:

if the project already has something resembling a unity build:
a.out -E file1.c > file2.c

if the project does incremental build:
use cat on every C file in correct order ignore the headers.
cat *.c > file1.c
a.out -E file1.c > file2.c


To better understand how it works I recommend to try it on tcc
code base first. Tcc code actually is layed out in an easy format
despite make forcing the incremental build.
a.out -E tcc.c > tinycc.c
This current version the result should amount to approx 87K loc

Some comments:
1. If the headers in C source are not guarded properly
it will not work, but so will your project not compile also. So
this does not happen. Header exclusion works the same way a compiler
would exclude it.
2. Problem of amalgamation is not as trivial as you think it might be,
because of the nature of how C works your header might be guarded but
you can also have code outside of that guard, in any case my program will
exclude the code properly. Especially that #endif is used to terminate any kind
of preprocessing expression.
3. By default headers with <> (system headers) are not amalgamated.
You can enable that, just read the source code in tcc_preprocess.
This program is powered by strategically placing printfs inside tcc
and some compiler logic changes of the default option -E, so don't try
to use this tcc for anything except what is meant to be.
4. While in most cases the resulting file will compile, some projects might be
weird and still require some manual tweaks and edits. Also you might need
to spend some time to clean the code from extra newlines that might get
created.
5. You can also configure it in source code to instead of processing everything
actually do the preprocess and exclude all the unnecessary junk, so for example
with that tcc source for x64 linux will be about ~35K instead of 87K.

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
Reply | Threaded
Open this post in threaded view
|

Re: Almost added a feature, but I broke things

tinycc-devel mailing list
In reply to this post by Kyryl Melekhin
Thank you Kyryl for the amalgama code and the notes, really good stuff!
I'll have fun playing with this over the holidays

On 24/12/2020 18:21, Kyryl Melekhin wrote:

> https://raw.githubusercontent.com/kyx0r/klec/master/utils/tinycc.c
>
> on x86_64 linux compile like so gcc tinycc.c -ldl -lpthread
> then you can use it like so:
>
> if the project already has something resembling a unity build:
> a.out -E file1.c > file2.c
>
> if the project does incremental build:
> use cat on every C file in correct order ignore the headers.
> cat *.c > file1.c
> a.out -E file1.c > file2.c
>
>
> To better understand how it works I recommend to try it on tcc
> code base first. Tcc code actually is layed out in an easy format
> despite make forcing the incremental build.
> a.out -E tcc.c > tinycc.c
> This current version the result should amount to approx 87K loc
>
> Some comments:
> 1. If the headers in C source are not guarded properly
> it will not work, but so will your project not compile also. So
> this does not happen. Header exclusion works the same way a compiler
> would exclude it.
> 2. Problem of amalgamation is not as trivial as you think it might be,
> because of the nature of how C works your header might be guarded but
> you can also have code outside of that guard, in any case my program will
> exclude the code properly. Especially that #endif is used to terminate any kind
> of preprocessing expression.
> 3. By default headers with <> (system headers) are not amalgamated.
> You can enable that, just read the source code in tcc_preprocess.
> This program is powered by strategically placing printfs inside tcc
> and some compiler logic changes of the default option -E, so don't try
> to use this tcc for anything except what is meant to be.
> 4. While in most cases the resulting file will compile, some projects might be
> weird and still require some manual tweaks and edits. Also you might need
> to spend some time to clean the code from extra newlines that might get
> created.
> 5. You can also configure it in source code to instead of processing everything
> actually do the preprocess and exclude all the unnecessary junk, so for example
> with that tcc source for x64 linux will be about ~35K instead of 87K.
>
> _______________________________________________
> Tinycc-devel mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/tinycc-devel

_______________________________________________
Tinycc-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel