About Supporting 2GiB+ Soundfonts on Windows

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

About Supporting 2GiB+ Soundfonts on Windows

fluid-dev mailing list
We have a bug report on GitHub [1]: A user reports that loading Soundfonts
>2GiB fails on Win10 64bit. There are two factors making up the root cause:

1. The ANSI C file API, namely ftell() and fseek(), use "long" as data type
for specifying offsets in the file.
2. Even on 64bit Windows, "long" is only a 32bit integer.

As a consequence, files >2GiB cannot be accessed through the standard C
library on Windows.

Of course, there are non-standard extensions: POSIX has ftello64() and
fseeko64(), Windows has _ftelli64() and _fseeki64().

BUT: we cannot simply switch to use those extension functions, because the
file operations are exposed via the our public API [2].

That is, switching to the 64bit extension functions would require to adopt our
API to use a 64bit integer, rather than "long". And THIS IS THE PROBLEM: There
is not signed 64bit integer in C89!

To fix this issue correctly, I currently see no other possibility than moving
fluidsynth to C99 and use "long long" in our public headers. Downsides:
* Moving to C99 would require VS2015.
* And people, who are consuming our header in a C++ project were required to
use C++11 in order to support the type "long long".

We cannot simply use size_t, because fseek requires a signed integer in order
to seek backwards in the file.

The current workaround is to build fluidsynth with CygWin, which has "long"
set to 64bit.

Any ideas how to solve this? Opinions on C99?

[1] https://github.com/FluidSynth/fluidsynth/issues/628
[2] http://www.fluidsynth.org/api/
sfont_8h.html#ac5cf57119fee0cde7422c86beafe0917

Tom




_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: About Supporting 2GiB+ Soundfonts on Windows

Dan Eble
On Mar 16, 2020, at 09:57, Tom M. via fluid-dev <[hidden email]> wrote:
> The current workaround is to build fluidsynth with CygWin, which has "long"
> set to 64bit.
>
> Any ideas how to solve this? Opinions on C99?

If you decided to require C99, I would think you would want to use int64_t and its relatives for portability.

Dan


_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: About Supporting 2GiB+ Soundfonts on Windows

Reinhold Hoffmann
We (still) use Visual Studio 2010 and use __int3264 which redirects to
__int64 for 64 bit.
 
===================================================
code snipset from BaseTsd.h which is part of VS2010
===================================================

//
// The INT_PTR is guaranteed to be the same size as a pointer.  Its
// size with change with pointer size (32/64).  It should be used
// anywhere that a pointer is cast to an integer type. UINT_PTR is
// the unsigned variation.
//
// __int3264 is intrinsic to 64b MIDL but not to old MIDL or to C compiler.
//

#if defined(_WIN64)
    typedef __int64 INT_PTR, *PINT_PTR;
    typedef unsigned __int64 UINT_PTR, *PUINT_PTR;

    typedef __int64 LONG_PTR, *PLONG_PTR;
    typedef unsigned __int64 ULONG_PTR, *PULONG_PTR;

    #define __int3264   __int64

#else
    typedef _W64 int INT_PTR, *PINT_PTR;
    typedef _W64 unsigned int UINT_PTR, *PUINT_PTR;

    typedef _W64 long LONG_PTR, *PLONG_PTR;
    typedef _W64 unsigned long ULONG_PTR, *PULONG_PTR;

    #define __int3264   __int32

#endif
===================================================

May be this can be used to be more downward compatible to older VS than
VS2015.

Reinhold

-----Ursprüngliche Nachricht-----
Von: fluid-dev [mailto:fluid-dev-bounces+reinhold=[hidden email]]
Im Auftrag von Dan Eble
Gesendet: Montag, 16. März 2020 23:24
An: FluidSynth mailing list
Cc: Tom M.
Betreff: Re: [fluid-dev] About Supporting 2GiB+ Soundfonts on Windows

On Mar 16, 2020, at 09:57, Tom M. via fluid-dev <[hidden email]>
wrote:
> The current workaround is to build fluidsynth with CygWin, which has
"long"
> set to 64bit.
>
> Any ideas how to solve this? Opinions on C99?

If you decided to require C99, I would think you would want to use int64_t
and its relatives for portability.
—
Dan


_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev


_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: About Supporting 2GiB+ Soundfonts on Windows

fluid-dev mailing list
> If you decided to require C99, I would think you would want to use int64_t and its relatives for portability.

I actually had "long long" in mind, because int64_t would require
fluidsynth's header to include stdint.h, and whenever possible I would
like to avoid polluting our header with any C system header (for the
sake of portability to C++). Also note that int64_t is an optional
type and may only be provided if the implementation supports it. (The
latter doesn't really make sense in practice, because long long is an
intrinsic type guaranteed to be at least 64 bits wide. And because
long long must be available, int64_t will also be available.)

Based on Reinhold's thought, the idea would be to use something like

#if defined(_MSVC_VER < VisualStudio2015)
typedef __int64 fluid_long_long_t; // even on 32bit windows
#else
typedef long long fluid_long_long_t;
#endif

in our public header. Which basically means: Fluidsynth's source code
stays C89. The public header will be compatible with C99 (or C++11)
UNLESS it's being compiled with an old VS in which case we provide a
fallback. This should provide enough portability while still retaining
ABI compatibility between and MSVC and MinGW compiled fluidsynth.

Should it really be that simple or am I missing smth.?

Tom

_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: About Supporting 2GiB+ Soundfonts on Windows

fluid-dev mailing list
Hello,

although it is true that you cannot handle files larger than 2GB with current code, there is one thing that's strange in that bug report.
If you really have not large file support, the first function to fail should be fseek() and not ftell(), so the first message you should read in the console should be "Seek to end of file failed" and not "Get end of file position failed". And this is what happens on Windows with oldest common runtime, try to believe. And when this happens, this means that MSVCRT does not also provide functions like _fseeki64/_ftelli64, this is unavoidable. But if the bug report says that the function that fails is ftell(), probably it means that the program is using a CRT with large file support. In this case, you can bypass the limitation of ftell() in this way:

fsize = fcbs->ftell(sf->sffd);
if (fsize == FLUID_FAILED)
{
    FLUID_LOG(FLUID_WARN, "Get end of file position failed");
    /* Try alternative way for large file support */
    fsize = get_large_file_size(sf->sffd);
    if (fsize == FLUID_FAILED)
    {
        FLUID_LOG(FLUID_ERR, "Get large file size failed");
        goto error_exit;
    }
}

and then you can add a function like this one (I used traditional fseek/ftell instead of _fluid_file_callbacks_t for simplicity):

int64_t get_large_file_size(FILE *fp)
{
    int     res = fseek(fp, 0L, SEEK_END);
    long    offset = 0x40000000L; /* 1GiB */
    long    rel;
    int64_t size = 0;

    if (res < 0)
        return res;

    for (;;)
    {
        res = fseek(fp, -offset, SEEK_CUR);

        if (res < 0)
            return res;

        size += offset;

        rel = ftell(fp);
        if (rel < 0)
            continue;

        return size + rel;
    }
    return size;
}

and you won't need to change the ABI of the library, at least for now. Technically, I don't think that it would be a big problem: "filesize" is compared with "chunk.size", which is 32bit so, if I understood correctly, this limits the size of a single SF2 file to 4GiB by design and this also means that there will be a maximum of 3 calls to fseek/ftell before computing the size of the large file. I have not tested on Unix, but if LIBC provides large file support, I expect that it will also work on it.
What do you think?

Sincerely.

Carlo Bramini.

> Il 17 marzo 2020 alle 9.44 "Tom M. via fluid-dev" <[hidden email]> ha scritto:
>
>
> > If you decided to require C99, I would think you would want to use int64_t and its relatives for portability.
>
> I actually had "long long" in mind, because int64_t would require
> fluidsynth's header to include stdint.h, and whenever possible I would
> like to avoid polluting our header with any C system header (for the
> sake of portability to C++). Also note that int64_t is an optional
> type and may only be provided if the implementation supports it. (The
> latter doesn't really make sense in practice, because long long is an
> intrinsic type guaranteed to be at least 64 bits wide. And because
> long long must be available, int64_t will also be available.)
>
> Based on Reinhold's thought, the idea would be to use something like
>
> #if defined(_MSVC_VER < VisualStudio2015)
> typedef __int64 fluid_long_long_t; // even on 32bit windows
> #else
> typedef long long fluid_long_long_t;
> #endif
>
> in our public header. Which basically means: Fluidsynth's source code
> stays C89. The public header will be compatible with C99 (or C++11)
> UNLESS it's being compiled with an old VS in which case we provide a
> fallback. This should provide enough portability while still retaining
> ABI compatibility between and MSVC and MinGW compiled fluidsynth.
>
> Should it really be that simple or am I missing smth.?
>
> Tom
>
> _______________________________________________
> fluid-dev mailing list
> [hidden email]
> https://lists.nongnu.org/mailman/listinfo/fluid-dev

_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: About Supporting 2GiB+ Soundfonts on Windows

fluid-dev mailing list
There is no guarantee that sf->sffd is a FILE* . Again, this is exposed via
our API. sf->sffd is a void*, which could be some user defined handle, which,
when passing to fseek(), would result in an access violation and in a
horrible, possibly-hard-to-reproduce crash of the entire application.

I do not quite understand why you think that fseek() should fail. It only
commands to seek to the end of the file. How this is accomplished is up to the
CRT. Even if _fseeki64/_ftelli64 are not available (as it seems to be the case
on Windows XP), the implemenation could just use _lseeki64/_ltelli64 under the
hood [1].

And yes, due to the RIFF chunks SF2 is limited to 4GiB in total.

[1] https://de.osdn.net/projects/mingw/lists/archive/users/2018-April/
000035.html


Tom




_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: About Supporting 2GiB+ Soundfonts on Windows

Ceresa Jean-Jacques
In reply to this post by fluid-dev mailing list

>Based on Reinhold's thought, the idea would be to use something like

#if defined(_MSVC_VER < VisualStudio2015)
typedef __int64 fluid_long_long_t; // even on 32bit windows
#else
typedef long long fluid_long_long_t;
#endif

>Should it really be that simple or am I missing smth.?

 

You are not missing something, and this should be really that simple.

Verified using VS2010 on Windows XP,  __int64  is really an integer value on 64 bit, (even on 32bits CPU).

__int64   v2_pos_max = 0x7FFFFFFFFFFFFFFF;  // +-9223372036854775807
__int64   v2_neg_min = 0x8000000000000000;     //  -9223372036854775808

jjc

 

 

 

 

> Message du 17/03/20 09:45
> De : "Tom M. via fluid-dev" <[hidden email]>
> A : "FluidSynth mailing list" <[hidden email]>
> Copie à : "Tom M." <[hidden email]>
> Objet : Re: [fluid-dev] About Supporting 2GiB+ Soundfonts on Windows
>
> > If you decided to require C99, I would think you would want to use int64_t and its relatives for portability. I actually had "long long" in mind, because int64_t would require fluidsynth's header to include stdint.h, and whenever possible I would like to avoid polluting our header with any C system header (for the sake of portability to C++). Also note that int64_t is an optional type and may only be provided if the implementation supports it. (The latter doesn't really make sense in practice, because long long is an intrinsic type guaranteed to be at least 64 bits wide. And because long long must be available, int64_t will also be available.) Based on Reinhold's thought, the idea would be to use something like #if defined(_MSVC_VER < VisualStudio2015) typedef __int64 fluid_long_long_t; // even on 32bit windows #else typedef long long fluid_long_long_t; #endif in our public header. Which basically means: Fluidsynth's source code stays C89. The public header will be compatible with C99 (or C++11) UNLESS it's being compiled with an old VS in which case we provide a fallback. This should provide enough portability while still retaining ABI compatibility between and MSVC and MinGW compiled fluidsynth. Should it really be that simple or am I missing smth.? Tom _______________________________________________ fluid-dev mailing list [hidden email] https://lists.nongnu.org/mailman/listinfo/fluid-dev

_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: About Supporting 2GiB+ Soundfonts on Windows

Dan Eble
In reply to this post by fluid-dev mailing list
On Mar 17, 2020, at 04:44, Tom M. via fluid-dev <[hidden email]> wrote:

> latter doesn't really make sense in practice, because long long is an
> intrinsic type guaranteed to be at least 64 bits wide. And because
> long long must be available, int64_t will also be available.)

That describes int_least64_t.  In an implementation where long long is wider than 64 bits, int64_t can not be a long long.

> typedef long long fluid_long_long_t;
...
> Should it really be that simple or am I missing smth.?

What you might be missing is that the width of fluid_long_long_t could then differ between implementations.  I can't say whether that would cause problems for you, but I have seen other projects where the size of int on different platforms was a key factor in many compiler warnings, and naturally, the warnings differed by platform too.  One developer might fix a warning for himself and end up creating one for another developer.

Regards,

Dan


_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: About Supporting 2GiB+ Soundfonts on Windows

fluid-dev mailing list
In reply to this post by fluid-dev mailing list
Hello!

> There is no guarantee that sf->sffd is a FILE* . Again, this is exposed via
> our API. sf->sffd is a void*, which could be some user defined handle, which,
> when passing to fseek(), would result in an access violation and in a
> horrible, possibly-hard-to-reproduce crash of the entire application.

Yes, I know ;)
As I have written, the function was a test that I made with traditional fseek/ftell, because I did an experiment for detecting the size of a file larger than 2 GiB, by using only C89/C90 functions. That example function surely needs to be improved: since you confirmed that the limit of the size is 4GiB, it is required to count the number of hops and if they are more than 3, it should return an error, but actually it depends if it is a good idea or not.

Sincerely.

Carlo Bramini.

_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev
Reply | Threaded
Open this post in threaded view
|

Re: About Supporting 2GiB+ Soundfonts on Windows

fluid-dev mailing list
Thanks to all of you for your input. FYI, a PR for this issue is now available:

https://github.com/FluidSynth/fluidsynth/pull/629


Tom



_______________________________________________
fluid-dev mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/fluid-dev