memory monitoring and leak debugging? (should the advice be in a web page?)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

memory monitoring and leak debugging? (should the advice be in a web page?)

Daniel Ortmann
Hello all,
I am new to Chicken Scheme and experimenting with binding scheme to a C
scanner built with Flex.  The results are fast but I feel the need to
monitor memory use and watch for leaks.

The only relevant thing I find on call-cc.org is this url:
http://wiki.call-cc.org/chicken-for-emacs-lisp-programmers#tooling

What are your experiences, tools, and practices with debugging mixed
Scheme + C code?

Thank you!


_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users
Reply | Threaded
Open this post in threaded view
|

Re: memory monitoring and leak debugging? (should the advice be in a web page?)

Jörg F. Wittenberger
Hello Daniel,

welcome here.

Since CHICKEN compiles into C, all the tools you are used with C to use
are still there.

Personally I'm not a fan of fancy debuggers, since most of the things I
write tend to depend on external (network) events.  I'd welcome tips
how to automate those jobs using better tools than printing log
messages.

Memory use in code mixing C and CHICKEN Scheme can be hairy.  I tend to
recommend to abstain from calling back from C into Scheme until you
know what you are doing.
http://wiki.call-cc.org/man/5/C%20interface#notes

Otherwise I used to run my code under valgrind, which helped me a lot
to catch some errors.

Best Regards

/Jörg

Am Tue, 6 Aug 2019 10:37:06 -0500
schrieb Daniel Ortmann <[hidden email]>:

> Hello all,
> I am new to Chicken Scheme and experimenting with binding scheme to a
> C scanner built with Flex.  The results are fast but I feel the need
> to monitor memory use and watch for leaks.
>
> The only relevant thing I find on call-cc.org is this url:
> http://wiki.call-cc.org/chicken-for-emacs-lisp-programmers#tooling
>
> What are your experiences, tools, and practices with debugging mixed
> Scheme + C code?



_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users
Reply | Threaded
Open this post in threaded view
|

Re: memory monitoring and leak debugging? (should the advice be in a web page?)

Kristian Lein-Mathisen

Hi Daniel and welcome to the Chicken mailing list.

Another thing to keep in mind is that the Chicken garbage-collector will move objects around during gc. That can cause a lot of hard-to-find bugs, and is probably one of the reasons for Joerg advice on not calling back into Scheme from your foreign-lambdas.

Another cause of problems with the relocations is that you cannot reference Chicken objects in C long-term, because the gc won't be able to update those pointers.

Having said that, the ffi in Chicken is really nice to use, so much that I often use it to explore new C api's.

And also, I sometimes use valgrind to check for memory leaks.

Best of luck!
K.

On Tue, Aug 6, 2019, 22:18 Jörg F. Wittenberger <[hidden email]> wrote:
Hello Daniel,

welcome here.

Since CHICKEN compiles into C, all the tools you are used with C to use
are still there.

Personally I'm not a fan of fancy debuggers, since most of the things I
write tend to depend on external (network) events.  I'd welcome tips
how to automate those jobs using better tools than printing log
messages.

Memory use in code mixing C and CHICKEN Scheme can be hairy.  I tend to
recommend to abstain from calling back from C into Scheme until you
know what you are doing.
http://wiki.call-cc.org/man/5/C%20interface#notes

Otherwise I used to run my code under valgrind, which helped me a lot
to catch some errors.

Best Regards

/Jörg

Am Tue, 6 Aug 2019 10:37:06 -0500
schrieb Daniel Ortmann <[hidden email]>:

> Hello all,
> I am new to Chicken Scheme and experimenting with binding scheme to a
> C scanner built with Flex.  The results are fast but I feel the need
> to monitor memory use and watch for leaks.
>
> The only relevant thing I find on call-cc.org is this url:
> http://wiki.call-cc.org/chicken-for-emacs-lisp-programmers#tooling
>
> What are your experiences, tools, and practices with debugging mixed
> Scheme + C code?



_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users

_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users
Reply | Threaded
Open this post in threaded view
|

Re: memory monitoring and leak debugging? (should the advice be in a web page?)

Dan Leslie
Isn't it possible to pin items, and avoid these relocation and garbage collection issues, with object-evict?


-Dan


Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, August 7, 2019 1:00 PM, Kristian Lein-Mathisen <[hidden email]> wrote:


Hi Daniel and welcome to the Chicken mailing list.

Another thing to keep in mind is that the Chicken garbage-collector will move objects around during gc. That can cause a lot of hard-to-find bugs, and is probably one of the reasons for Joerg advice on not calling back into Scheme from your foreign-lambdas.

Another cause of problems with the relocations is that you cannot reference Chicken objects in C long-term, because the gc won't be able to update those pointers.

Having said that, the ffi in Chicken is really nice to use, so much that I often use it to explore new C api's.

And also, I sometimes use valgrind to check for memory leaks.

Best of luck!
K.

On Tue, Aug 6, 2019, 22:18 Jörg F. Wittenberger <[hidden email]> wrote:
Hello Daniel,

welcome here.

Since CHICKEN compiles into C, all the tools you are used with C to use
are still there.

Personally I'm not a fan of fancy debuggers, since most of the things I
write tend to depend on external (network) events.  I'd welcome tips
how to automate those jobs using better tools than printing log
messages.

Memory use in code mixing C and CHICKEN Scheme can be hairy.  I tend to
recommend to abstain from calling back from C into Scheme until you
know what you are doing.

Otherwise I used to run my code under valgrind, which helped me a lot
to catch some errors.

Best Regards

/Jörg

Am Tue, 6 Aug 2019 10:37:06 -0500
schrieb Daniel Ortmann <[hidden email]>:

> Hello all,
> I am new to Chicken Scheme and experimenting with binding scheme to a
> C scanner built with Flex.  The results are fast but I feel the need
> to monitor memory use and watch for leaks.
>
> The only relevant thing I find on call-cc.org is this url:
>
> What are your experiences, tools, and practices with debugging mixed
> Scheme + C code?



_______________________________________________
Chicken-users mailing list


_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users
Reply | Threaded
Open this post in threaded view
|

Re: memory monitoring and leak debugging? (should the advice be in a web page?)

Daniel Ortmann
In reply to this post by Kristian Lein-Mathisen
I have Scheme calling the C yylex() and can easily get all of the token ints.

I am trying to return '(tok,lineno) '(tok,lineno,intval) or '(tok,lineno,strval)
    C_word *ptr, *sptr;
    C_word lst;
...
        case TOK_G_R_SECT:

            printf("gettoken:(TOK_G_R_SECT,line,text) = (%2d,%3d,'%s')\n", tok, lineno, yylval.sptr);
            ptr = C_alloc(C_SIZEOF_LIST(3));
            lst = C_list(&ptr, 3, C_fix(tok), C_fix(lineno), C_fix(0)); /*C_string2(&sptr, yylval.sptr));*/
            return callin(lst);

Question 1:
Does this stumbling attempt fall under that warning about not calling from C to Scheme?  Or is that warning for something else?

Question 2:
Do I need a bit better magic words to get that 'lst =' line working?  Or am I missing something more major?

Thank you!


On 8/7/19 3:00 PM, Kristian Lein-Mathisen wrote:

Hi Daniel and welcome to the Chicken mailing list.

Another thing to keep in mind is that the Chicken garbage-collector will move objects around during gc. That can cause a lot of hard-to-find bugs, and is probably one of the reasons for Joerg advice on not calling back into Scheme from your foreign-lambdas.

Another cause of problems with the relocations is that you cannot reference Chicken objects in C long-term, because the gc won't be able to update those pointers.

Having said that, the ffi in Chicken is really nice to use, so much that I often use it to explore new C api's.

And also, I sometimes use valgrind to check for memory leaks.

Best of luck!
K.

On Tue, Aug 6, 2019, 22:18 Jörg F. Wittenberger <[hidden email]> wrote:
Hello Daniel,

welcome here.

Since CHICKEN compiles into C, all the tools you are used with C to use
are still there.

Personally I'm not a fan of fancy debuggers, since most of the things I
write tend to depend on external (network) events.  I'd welcome tips
how to automate those jobs using better tools than printing log
messages.

Memory use in code mixing C and CHICKEN Scheme can be hairy.  I tend to
recommend to abstain from calling back from C into Scheme until you
know what you are doing.
http://wiki.call-cc.org/man/5/C%20interface#notes

Otherwise I used to run my code under valgrind, which helped me a lot
to catch some errors.

Best Regards

/Jörg

Am Tue, 6 Aug 2019 10:37:06 -0500
schrieb Daniel Ortmann <[hidden email]>:

> Hello all,
> I am new to Chicken Scheme and experimenting with binding scheme to a
> C scanner built with Flex.  The results are fast but I feel the need
> to monitor memory use and watch for leaks.
>
> The only relevant thing I find on call-cc.org is this url:
> http://wiki.call-cc.org/chicken-for-emacs-lisp-programmers#tooling
>
> What are your experiences, tools, and practices with debugging mixed
> Scheme + C code?



_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users

_______________________________________________
Chicken-users mailing list
[hidden email]
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.nongnu.org_mailman_listinfo_chicken-2Dusers&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Dkq-B4wf3U4UeyggH34kSw8HWPy7CqI3u3RwPXz2f5o&m=uGR1RxlutvwNyGUJ8DkkNMO13p4pIcq2C8qZP_SlF2Q&s=zCUp8Z1mO63DzrUtaQUUKKKpB3z_x70a3OgFBS5Unn4&e= 


_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users
Reply | Threaded
Open this post in threaded view
|

Re: memory monitoring and leak debugging? (should the advice be in a web page?)

Kristian Lein-Mathisen
In reply to this post by Dan Leslie

That is possible, but it is quite hard to use correctly in my opinion. It's also not in core anymore. If I understand things correctly, there is a performance penalty in every new gc root. Maybe some of the core team members can comment. To me, object-evict has been a last-resort (and as such, I've never used it) but I don't know how others might feel about it.

K.


On Wed, Aug 7, 2019, 22:52 Dan Leslie <[hidden email]> wrote:
Isn't it possible to pin items, and avoid these relocation and garbage collection issues, with object-evict?


-Dan


Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, August 7, 2019 1:00 PM, Kristian Lein-Mathisen <[hidden email]> wrote:


Hi Daniel and welcome to the Chicken mailing list.

Another thing to keep in mind is that the Chicken garbage-collector will move objects around during gc. That can cause a lot of hard-to-find bugs, and is probably one of the reasons for Joerg advice on not calling back into Scheme from your foreign-lambdas.

Another cause of problems with the relocations is that you cannot reference Chicken objects in C long-term, because the gc won't be able to update those pointers.

Having said that, the ffi in Chicken is really nice to use, so much that I often use it to explore new C api's.

And also, I sometimes use valgrind to check for memory leaks.

Best of luck!
K.

On Tue, Aug 6, 2019, 22:18 Jörg F. Wittenberger <[hidden email]> wrote:
Hello Daniel,

welcome here.

Since CHICKEN compiles into C, all the tools you are used with C to use
are still there.

Personally I'm not a fan of fancy debuggers, since most of the things I
write tend to depend on external (network) events.  I'd welcome tips
how to automate those jobs using better tools than printing log
messages.

Memory use in code mixing C and CHICKEN Scheme can be hairy.  I tend to
recommend to abstain from calling back from C into Scheme until you
know what you are doing.

Otherwise I used to run my code under valgrind, which helped me a lot
to catch some errors.

Best Regards

/Jörg

Am Tue, 6 Aug 2019 10:37:06 -0500
schrieb Daniel Ortmann <[hidden email]>:

> Hello all,
> I am new to Chicken Scheme and experimenting with binding scheme to a
> C scanner built with Flex.  The results are fast but I feel the need
> to monitor memory use and watch for leaks.
>
> The only relevant thing I find on call-cc.org is this url:
>
> What are your experiences, tools, and practices with debugging mixed
> Scheme + C code?



_______________________________________________
Chicken-users mailing list


_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users
Reply | Threaded
Open this post in threaded view
|

Re: memory monitoring and leak debugging? (should the advice be in a web page?)

felix.winkelmann
> That is possible, but it is quite hard to use correctly in my opinion. It's
> also not in core anymore. If I understand things correctly, there is a
> performance penalty in every new gc root. Maybe some of the core team
> members can comment. To me, object-evict has been a last-resort (and as
> such, I've never used it) but I don't know how others might feel about it.
>

"object-evict" moves an object into statically allocated memory, it does
not create a GC root. There are issues with object-identity for symbols
and also the eviction recurses into all data contained in the object. It
is most useful for number arrays and data that contains only numeric
or simple values.

But Kristian is quite right - it's a low-level tool for very specific
situations. Using "foreign-lambda*" with callbacks into Scheme
is a cleaner way of handling allocation inside foreign code.


felix


_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users
Reply | Threaded
Open this post in threaded view
|

Re: memory monitoring and leak debugging? (should the advice be in a web page?)

Kristian Lein-Mathisen
In reply to this post by Daniel Ortmann

Hi Daniel,

Sorry for the late reply, I'm on vacation.



Q1: no, you are not calling back into Scheme (you'd need define-external or something).
Q2: What it seems you are doing there is construct a compound object using the Chicken C-api. I try to avoid doing that.

Constructing Scheme objects in C is easy to get wrong. Everything is allocated on the stack so I think you have to use foreign-primitive to be safe, not foreign-lambda (I think). This may be an easier way:

You can define-external a simple scheme procedure that takes the three ints and returns list of them:

(define-external
  (list_int3 (int a) (int b) (int c))
    scheme-object
   (list a b c))
;;C_word  list_int3(int t0,int t1,int t2)

And use this to produce your compound object. However, keep in mind that the GC will move your objects around: Don't store lst anywhere in C between Scheme callbacks because it will be invalidated after gc. Note that this does violates the "don't use callbacks" advice!

Another trick which is useful to be aware of is using let-location. You can allocate  objects in Scheme and pass them as pointers to your C procedures, which in turn can mutate them and we can multiple return values from Scheme side. I'm doing that here, for example:


My advice for beginners would be:

## don't allocate compound scheme objects in C because it is hard
but allocating "C objects" is usually not that hard though. if a library allocates a struct for you, go for it. and you can even use set-finalizer! on it.

## avoid Scheme callbacks if you can because it makes things hard

## whenever you can, allocate in Scheme and mutate in C
because it is generally safe and fast. I'm doing that here, for example: https://github.com/kristianlm/chicken-minissh/blob/8d5cccb9a561704aacf1464a8e3d0315499ec2c6/chacha20.scm#L10

## box C structs in define-record's even if you feel it is slow
so you don't start mixing up types

Safety is always harder and more important that it seems. I've had so many cases where I have considered my bindings complete only to find I get the occational segfault - typically caused by the GC moving object around. But reproducing GC-related bugs can be hard.

Hope this helps a bit!
K.

On Wed, Aug 7, 2019, 23:10 Daniel Ortmann <[hidden email]> wrote:
I have Scheme calling the C yylex() and can easily get all of the token ints.

I am trying to return '(tok,lineno) '(tok,lineno,intval) or '(tok,lineno,strval)
    C_word *ptr, *sptr;
    C_word lst;
...
        case TOK_G_R_SECT:

            printf("gettoken:(TOK_G_R_SECT,line,text) = (%2d,%3d,'%s')\n", tok, lineno, yylval.sptr);
            ptr = C_alloc(C_SIZEOF_LIST(3));
            lst = C_list(&ptr, 3, C_fix(tok), C_fix(lineno), C_fix(0)); /*C_string2(&sptr, yylval.sptr));*/
            return callin(lst);

Question 1:
Does this stumbling attempt fall under that warning about not calling from C to Scheme?  Or is that warning for something else?

Question 2:
Do I need a bit better magic words to get that 'lst =' line working?  Or am I missing something more major?

Thank you!


On 8/7/19 3:00 PM, Kristian Lein-Mathisen wrote:

Hi Daniel and welcome to the Chicken mailing list.

Another thing to keep in mind is that the Chicken garbage-collector will move objects around during gc. That can cause a lot of hard-to-find bugs, and is probably one of the reasons for Joerg advice on not calling back into Scheme from your foreign-lambdas.

Another cause of problems with the relocations is that you cannot reference Chicken objects in C long-term, because the gc won't be able to update those pointers.

Having said that, the ffi in Chicken is really nice to use, so much that I often use it to explore new C api's.

And also, I sometimes use valgrind to check for memory leaks.

Best of luck!
K.

On Tue, Aug 6, 2019, 22:18 Jörg F. Wittenberger <[hidden email]> wrote:
Hello Daniel,

welcome here.

Since CHICKEN compiles into C, all the tools you are used with C to use
are still there.

Personally I'm not a fan of fancy debuggers, since most of the things I
write tend to depend on external (network) events.  I'd welcome tips
how to automate those jobs using better tools than printing log
messages.

Memory use in code mixing C and CHICKEN Scheme can be hairy.  I tend to
recommend to abstain from calling back from C into Scheme until you
know what you are doing.
http://wiki.call-cc.org/man/5/C%20interface#notes

Otherwise I used to run my code under valgrind, which helped me a lot
to catch some errors.

Best Regards

/Jörg

Am Tue, 6 Aug 2019 10:37:06 -0500
schrieb Daniel Ortmann <[hidden email]>:

> Hello all,
> I am new to Chicken Scheme and experimenting with binding scheme to a
> C scanner built with Flex.  The results are fast but I feel the need
> to monitor memory use and watch for leaks.
>
> The only relevant thing I find on call-cc.org is this url:
> http://wiki.call-cc.org/chicken-for-emacs-lisp-programmers#tooling
>
> What are your experiences, tools, and practices with debugging mixed
> Scheme + C code?



_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users

_______________________________________________
Chicken-users mailing list
[hidden email]
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.nongnu.org_mailman_listinfo_chicken-2Dusers&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Dkq-B4wf3U4UeyggH34kSw8HWPy7CqI3u3RwPXz2f5o&m=uGR1RxlutvwNyGUJ8DkkNMO13p4pIcq2C8qZP_SlF2Q&s=zCUp8Z1mO63DzrUtaQUUKKKpB3z_x70a3OgFBS5Unn4&e= 

_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users

_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users