Possible approach to preloading samples / presets

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Possible approach to preloading samples / presets

Marcus Weseloh
Hi all,

When we implemented the dynamic sample loading, we also discussed if
it would be good to provide a way to preload samples. Back then it was
mainly meant as a way to preload all presets from a MIDI file, so that
loading the samples dynamically wouldn't stall the playback. We
decided that it wasn't worth the effort, as nobody had expressed an
interest or seen a real need for it.

Well, I do have a real need for preloading presets (or rather samples)
now. My use-case is the following: I'm building musical instruments
that make use of FluidSynth for sound generation and those instruments
are used in a real-time performance context. Players have the ability
to switch between different instrument setups at the touch of a button
and want to make use of this feature even in the middle of a
performance. Those instrument setups also change the channel presets
in FluidSynth, and as the SoundFonts reside on an SD-Card, switching
between setups sometimes takes a few seconds. That is way too long for
my users, who would like to switch setups between beats, in the
fraction of a second.

You could say that I shouldn't use dynamic sample loading in that
case, but memory on the instrument is limited and I would rather not
load all samples from a huge soundfont just because the player uses a
single preset from that font. It also makes boot-up of the instrument
*much* faster, so I really want a solution that works with dynamic
sample loading.

So currently I'm using a hackish way to implement preloading of sample
data. I have increased the number of MIDI channels in the synth to 64
and use the otherwise unused channels 16-63 as "preload slots". When a
user triggers the "preload" function, I go through all presets in all
setups and assign each unique preset to the next available "preload
slot channel", thereby loading the samples into the sample cache. That
method works, but it does feel like a hack. It also puts an arbitrary
limit on the number of presets I can preload, regardless of their
actual size in memory.

So I would really like to extend FluidSynth to allow for preloading of
presets / samples via a public API. But before I make a suggestion in
that direction, I wanted to ask your opinion on a a couple of
questions:

1. Would such a preload feature even be interesting to other users? Or
is the use-case so unique to my application that it wouldn't make
sense to add it to FluidSynth?

2. If you think it could be added, is that because you would have a
use-case for it as well? And if so, is it similar to mine (preloading
an externally determined list of presets) or do you need it for
something else (like preloading all presets from a MIDI file, for
example)?

The answers to those two questions would greatly influence the
approach for this feature. Obviously the answer to 1. would be most
important whether to approach it at all. But the answer to 2. might
influence how to go about it. It could be implemented as a preload
(load before use) or as "pinning" (don't unload after use). It might
also affect where to extend FS: a public API for the sample cache,
extending the preset selection functions, extending the soundfont
loader interface, ...

It would be great to hear your opinion about this!

All the best
Marcus

_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: Possible approach to preloading samples / presets

fluid-dev mailing list
Understood. Yes, it would be nice to hear at least one other person who
has a similar use case.

I also have a preference of implementing this. But before, allow me to
ask the following questions:

> You could say that I shouldn't use dynamic sample loading in that
case

Why do you *need* to use (or support) "huge soundfonts" on a device that
apparently has memory constraints? Wouldn't a smaller SF that entirely
fits into memory do as well?

And if you really need that "huge soundfonts", wouldn't it be possible
to truncate the SF by deleting samples that you'll never need?

Tom

_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: Possible approach to preloading samples / presets

Ceresa Jean-Jacques
In reply to this post by Marcus Weseloh

Hi,

 

>Well, I do have a real need for preloading presets (or rather samples) now.

 

I understand your need and I'am not surprised you need this now because you are using a memory constrained system.

Having a way to preload sample is also useful when preparing a set of presets that doesnt' belong to the same soundfont file.

Of course we could use the cut/copy/paste functions of any soundfont editor to do the job, but using the capabilities of the synth's soundfont loader could help to achieve

the result much faster because of the feedback given by the synth on which those preset must be played. In summary, I found useful any information exposed to the application

by the target soundfont loader which could help to determine an externally list of preset scattered on multiple soundfont files.

This preset list could be used directly for live performance (Marcus's use case) and also by a soundfont editor to automatize the building of a custom SoundFont file that gathers all those presets.

It will be really nice to hear opinions of organ musicians or live performance musicians.

jjc.

 


_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: Possible approach to preloading samples / presets

Marcus Weseloh
In reply to this post by fluid-dev mailing list
Am Fr., 23. Okt. 2020 um 21:17 Uhr schrieb Tom M. via fluid-dev
<[hidden email]>:
> Why do you *need* to use (or support) "huge soundfonts" on a device that
> apparently has memory constraints? Wouldn't a smaller SF that entirely
> fits into memory do as well?

First let me clarify: I'm not hugely memory constrained. The device
has 1G of RAM, of which around 700 - 800 MB are theoretically usable
for Soundfonts.

The reason I want to use dynamic sample loading is because I'm not in
control of the Soundfonts on the device. I ship with a few
preconfigured soundfonts and instrument setups, but the user has the
ability to upload new soundfonts to the device and configure new
setups. And as the device uses 9 synth channels and the user can
freely assign any preset from any soundfonts to these channels, an
instrument setup could load nine different soundfonts for a single
setup.

> And if you really need that "huge soundfonts", wouldn't it be possible
> to truncate the SF by deleting samples that you'll never need?

Yes, that would be possible of course. But it would require my users
to do that themselves. Or I would have to restrict the Soundfonts
which can be used with the instrument, which I really don't want to
do.

Does that answer your question?

Cheers
Marcus

_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: Possible approach to preloading samples / presets

fluid-dev mailing list
Ok, I see.

So, my preference of implementing this is to use the pinning approach.

I'm thinking of a function like

fluid_synth_pin_preset(synth, fluid_preset_t*)

The function will attempt to pin all samples of the given preset and
load them into memory, if they are currently unloaded. "to pin" in this
context means preventing them from being unloaded by an upcoming channel
prog change. If successful, FLUID_OK is returned.

If the preset doesn't support pinning (because it's a DLS preset, which
doesn't use the sample cache currently), FLUID_FAILED is returned.

If the preset supports pinning, but dynamic-sample-loading is disabled,
FLUID_OK is still returned. Not an error, it is already loaded and will
never be unloaded.

No exposing of the internal sample cache is needed.

Ofc, we'll need a way to unpin presets. I'd vote for a separate "unpin"
function. This function should unset the is_pinned flag. Again, if the
preset doesn't support pinning, FLUID_FAILED is returned. Ideally, the
function also unloads the samples if the given preset is no longer
referenced by any channel. In this case, care must be taken to avoid a
race condition with the rendering thread: If you unload the samples from
memory (because the ref count was zero) and just right afterwards, the
rendering thread selects the sample again, then just make sure that the
sample is first being unloaded cleanly and then loaded back cleanly.
Can't remember if you were already using some kind of synchronization
like that, didn't had a look.

Thoughts?


Tom

_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: Possible approach to preloading samples / presets

Marcus Weseloh
Hi,

Am So., 25. Okt. 2020 um 11:34 Uhr schrieb Tom M. via fluid-dev
<[hidden email]>:
> So, my preference of implementing this is to use the pinning approach.
> I'm thinking of a function like
> fluid_synth_pin_preset(synth, fluid_preset_t*)

Yes, sounds good. And quite similar to the approach I used in the
earlier approach to preload all samples from a MIDI file (which we
didn't take, because nobody needed it).
I will prepare an RFC pull request for further discussion.

Cheers
Marcus

_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: Possible approach to preloading samples / presets

Marcus Weseloh
In reply to this post by fluid-dev mailing list
Hm, looking at the code and thinking about this some more...

You suggested something like
  fluid_synth_pin_preset(synth, fluid_preset_t*)

The problem here is the fluid_preset_t pointer. Exposing a public API
that consumes fluid_preset_t pointers will introduce some problems, as
there is no thread-safe way of retrieving and using those pointers for
external applications (outside of the synthesis thread) and without
implementing a separate Soundfont loader.

What do you think about:
fluid_synth_pin_preset(synth, sfont_id, bank, program)
fluid_synth_unpin_preset(synth, sfont_id, bank, program)

Cheers
Marcus

Am So., 25. Okt. 2020 um 11:34 Uhr schrieb Tom M. via fluid-dev
<[hidden email]>:

>
> Ok, I see.
>
> So, my preference of implementing this is to use the pinning approach.
>
> I'm thinking of a function like
>
> fluid_synth_pin_preset(synth, fluid_preset_t*)
>
> The function will attempt to pin all samples of the given preset and
> load them into memory, if they are currently unloaded. "to pin" in this
> context means preventing them from being unloaded by an upcoming channel
> prog change. If successful, FLUID_OK is returned.
>
> If the preset doesn't support pinning (because it's a DLS preset, which
> doesn't use the sample cache currently), FLUID_FAILED is returned.
>
> If the preset supports pinning, but dynamic-sample-loading is disabled,
> FLUID_OK is still returned. Not an error, it is already loaded and will
> never be unloaded.
>
> No exposing of the internal sample cache is needed.
>
> Ofc, we'll need a way to unpin presets. I'd vote for a separate "unpin"
> function. This function should unset the is_pinned flag. Again, if the
> preset doesn't support pinning, FLUID_FAILED is returned. Ideally, the
> function also unloads the samples if the given preset is no longer
> referenced by any channel. In this case, care must be taken to avoid a
> race condition with the rendering thread: If you unload the samples from
> memory (because the ref count was zero) and just right afterwards, the
> rendering thread selects the sample again, then just make sure that the
> sample is first being unloaded cleanly and then loaded back cleanly.
> Can't remember if you were already using some kind of synchronization
> like that, didn't had a look.
>
> Thoughts?
>
>
> Tom
>
> _______________________________________________
> fluid-dev mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/fluid-dev

_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: Possible approach to preloading samples / presets

fluid-dev mailing list
> there is no thread-safe way of retrieving and using those pointers for
external applications (outside of the synthesis thread)

Valid point, indeed.

> What do you think about:
> fluid_synth_pin_preset(synth, sfont_id, bank, program)

Ok for me. Looking forward to it :)


Tom

_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: Possible approach to preloading samples / presets

Marcus Weseloh
Hi,

there is a first RFC pull-request here:
https://github.com/FluidSynth/fluidsynth/pull/698

It implements the fluid_synth_pin_preset and fluid_synt_unpin_presets
by hooking into the preset notify callbacks that are already used by
the dynamic-sample-loading to load and unload the preset. Advantage is
that the changes can be mostly kept to the default loader.
Disadvantage is that the public API can not give any indication if
pinning or unpinning has actually succeeded.

Looking forward to your comments!

BTW: I haven't tested that code yet, only made sure it actually
compiles. If you think this approach has merit, I will develop it
further and also add tests.

Cheers
Marcus

_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev