Building the openssl egg on MacOS

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Building the openssl egg on MacOS

Lassi Kortela
On MacOS Mojave, "chicken-install openssl" fails because the OS doesn't
ship any pkg-config definition file for its version of the openssl
library. The pkg-config definition is supposed to be in a file called
'openssl.pc' but 'sudo find / -name openssl.pc 2>/dev/null' turns up no
such file for the OpenSSL that comes with the OS.

(In this version of MacOS the openssl library is actually the OpenSSL
compatibility wrapper of the LibreSSL library: "/usr/bin/openssl
version" says "LibreSSL 2.6.5". Even though the library is LibreSSL,
it's still designed to ship with a file named 'openssl.pc' for
compatibility with OpenSSL. But MacOS doesn't have that file.)

The error chicken-install gives is:

     OpenSSL >= 1.0.2 seems to be unavailable

Using the helpful verbose mode of chicken-install I figured out that the
source of the error is the pkg-config invocation in the shell script
<~/.cache/chicken-install/openssl/build-openssl>.

The easiest workaround is to install a copy of OpenSSL or LibreSSL from
the popular Homebrew package manager and build the egg using that copy:

     brew install openssl
     export PKG_CONFIG_PATH="$(brew --prefix openssl)/lib/pkgconfig"
     chicken-install openssl

Or:

     brew install libressl
     export PKG_CONFIG_PATH="$(brew --prefix libressl)/lib/pkgconfig"
     chicken-install openssl

In principle one could use the openssl library that ships with MacOS to
build the openssl egg. However, on this OS version I can't find the
<openssl/ssl.h> C header file anywhere in the file system, even though
the library itself is installed as </usr/lib/libssl.dylib>. I installed
Apple's command-line developer tools using "sudo xcode-select
--install". As far as I can tell, I don't have the full GUI version of
XCode anymore with this OS upgrade. I think the GUI version is still
available free of charge but it may now require a Mac App Store login to
install. From Chicken's point of view, we unfortunately can't assume
that people who use Chicken have the full version of XCode.

Many/most MacOS users of intarweb might stumble onto this problem now
that HTTPS websites are everywhere, so would it make sense to add
MacOS-specific checks to the build-openssl script? Since it seems tricky
to reliably find the system OpenSSL header files, maybe it should
suggest that people use Homebrew as the easiest alternative. I can write
and test a patch for the 'build-openssl' shell script if it helps (well,
I already wrote most of it :-)

_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users
Reply | Threaded
Open this post in threaded view
|

Re: Building the openssl egg on MacOS

Vasilij Schneidermann-2
Hello Lassi,

I maintain the openssl egg these days.

> On MacOS Mojave, "chicken-install openssl" fails because the OS doesn't ship
> any pkg-config definition file for its version of the openssl library. The
> pkg-config definition is supposed to be in a file called 'openssl.pc' but
> 'sudo find / -name openssl.pc 2>/dev/null' turns up no such file for the
> OpenSSL that comes with the OS.
>
> (In this version of MacOS the openssl library is actually the OpenSSL
> compatibility wrapper of the LibreSSL library: "/usr/bin/openssl version"
> says "LibreSSL 2.6.5". Even though the library is LibreSSL, it's still
> designed to ship with a file named 'openssl.pc' for compatibility with
> OpenSSL. But MacOS doesn't have that file.)
Ugh.  I've always had the impression macOS gets worse with each release,
but this is ridiculous, almost as if they expect everyone to use XCode
for development...

> The easiest workaround is to install a copy of OpenSSL or LibreSSL from the
> popular Homebrew package manager and build the egg using that copy:
>
>     brew install openssl
>     export PKG_CONFIG_PATH="$(brew --prefix openssl)/lib/pkgconfig"
>     chicken-install openssl
>
> Or:
>
>     brew install libressl
>     export PKG_CONFIG_PATH="$(brew --prefix libressl)/lib/pkgconfig"
>     chicken-install openssl
This is what I recommend to everyone who has to work on that kind of
system.  It's sad, but the least painful way of getting work done.

> In principle one could use the openssl library that ships with MacOS to
> build the openssl egg. However, on this OS version I can't find the
> <openssl/ssl.h> C header file anywhere in the file system, even though the
> library itself is installed as </usr/lib/libssl.dylib>.

The reason for this is because on macOS you're supposed to use
"frameworks" instead which contain all that information, much like an
.app contains all the files associated with a program.  There's even a
`-framework` option for `csc` which might just make this work... I can't
test this though because I've abandoned that OS many years ago.

> I installed Apple's command-line developer tools using "sudo
> xcode-select --install". As far as I can tell, I don't have the full
> GUI version of XCode anymore with this OS upgrade. I think the GUI
> version is still available free of charge but it may now require a Mac
> App Store login to install. From Chicken's point of view, we
> unfortunately can't assume that people who use Chicken have the full
> version of XCode.

I hereby reiterate my point that doing development on macOS involves
much sadness, such as creating a developer account to do development.
I'm afraid there isn't much else you can do, unless you somehow get gcc
and the rest of the toolchain working without that.

> Many/most MacOS users of intarweb might stumble onto this problem now that
> HTTPS websites are everywhere, so would it make sense to add MacOS-specific
> checks to the build-openssl script? Since it seems tricky to reliably find
> the system OpenSSL header files, maybe it should suggest that people use
> Homebrew as the easiest alternative. I can write and test a patch for the
> 'build-openssl' shell script if it helps (well, I already wrote most of it
> :-)

Before you do that, there is some work I've done on a few more eggs I
maintain, I got fed up with writing user-unfriendly shell scripts that I
rewrote the non-Windows version to use a Scheme program instead doing
basic version detection, falling back to environment variables and
finally bailing out with an error.  You can find the latest version of
it at the breadline repository [1].  Please let me know if that fulfills
your wishes and if not, whether it can be made to do so.  If yes, then
I'd be willing to migrate the openssl egg towards such a script as well.
The reason I haven't done so is because unlike the other eggs I maintain
it's something I'd rather not touch unnecessarily, breakages to it will
be far more annoying to handle than anything else.  And honestly
speaking, OpenSSL isn't nice to deal with either :>

Vasilij

[1] https://raw.githubusercontent.com/wasamasa/breadline/master/build-breadline.scm

_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Building the openssl egg on MacOS

Norman Gray-2

Greetings.

On 15 Jul 2019, at 19:53, Vasilij Schneidermann wrote:

>  And honestly
> speaking, OpenSSL isn't nice to deal with either

Indeed!

I spent a _little_ while digging into the details of this a few months
ago, and my understanding is this:

   * Apple were suffering version-skew hell because (reportedly) the
OpenSSL folk kept changing the ABI for the library.
   * So they lost patience and deprecated OpenSSL on macOS
   * ...to the extent that there are now compiler-visible deprecation
markers in the relevant still-visible header files.
   * The doctrine is that one should use Apple's own encryption
frameworks (as you noted, Vasilij)
   * (which work fine, in the pretty basic uses I've made of them, but
they don't pretend to be OpenSSL)
   * ...and act as if there were no OpenSSL library at all on macOS.
   * As Lassi noted, there is still a LibreSSL library on the system in
fact, but I believe it is intended to be strictly for legacy use.

So if you have a tool which depends on Open/LibreSSL, then you need to
get it on your machine either from source, or using the package manager
of your choice, and not even try using the system one.

If I've misunderstood or misrepresented one of the steps here, I'm sure
someone will cheerfully correct me.

Best wishes,

Norman


--
Norman Gray  :  https://nxg.me.uk
SUPA School of Physics and Astronomy, University of Glasgow, UK

_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users
Reply | Threaded
Open this post in threaded view
|

Re: Building the openssl egg on MacOS

Lassi Kortela
> I spent a _little_ while digging into the details of this a few months
> ago, and my understanding is this:
>
>     * Apple were suffering version-skew hell because (reportedly) the
> OpenSSL folk kept changing the ABI for the library.
>     * So they lost patience and deprecated OpenSSL on macOS

Sounds quite plausible.

>     * ...to the extent that there are now compiler-visible deprecation
> markers in the relevant still-visible header files.
>     * The doctrine is that one should use Apple's own encryption
> frameworks (as you noted, Vasilij)
>     * (which work fine, in the pretty basic uses I've made of them, but
> they don't pretend to be OpenSSL)
>     * ...and act as if there were no OpenSSL library at all on macOS.
>     * As Lassi noted, there is still a LibreSSL library on the system in
> fact, but I believe it is intended to be strictly for legacy use.

This is the impression I got as well.

The thing is, not only are the Apple frameworks non-portable to any
other OS but Apple keeps changing, deprecating and merging frameworks
from release to release. They just deprecated OpenGL (and didn't adopt
Vulkan) in favor of some brand new framework that doesn't work on any
non-Apple OS! And Swift may hold the world record for the number of
deprecation warnings in any tool. So Apple's own programming interfaces
are some of the least stable ever :)

> So if you have a tool which depends on Open/LibreSSL, then you need to
> get it on your machine either from source, or using the package manager
> of your choice, and not even try using the system one.

Based on the non-portability and difficulty of using Apple's frameworks,
and their history of changing them endlessly, I suggest adopting a
simple policy to direct all Mac users to Homebrew :)

Of SSL libraries, LibreSSL sounds like it will be the most stable and
secure choice. The OpenSSL compatibility makes it extra nice.

_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users
Reply | Threaded
Open this post in threaded view
|

Re: Building the openssl egg on MacOS

Lassi Kortela
In reply to this post by Vasilij Schneidermann-2
> I maintain the openssl egg these days.

Very nice, thanks for your efforts!

> Ugh.  I've always had the impression macOS gets worse with each release,
> but this is ridiculous, almost as if they expect everyone to use XCode
> for development...

I find that the usability gets better with each release (still dreading
the inevitable downfall) but from a programmer's point of view it does
indeed get worse. Mojave locked down some files so even with sudo you
can't touch them, deprecated other stuf, and retroactively made it so
you need the App Store login to install the full XCode.

>> The easiest workaround is to install a copy of OpenSSL or LibreSSL from the
>> popular Homebrew package manager and build the egg using that copy:
>
> This is what I recommend to everyone who has to work on that kind of
> system.  It's sad, but the least painful way of getting work done.

Cool, I didn't know that.

Homebrew is actually really nice to use, but your point stands about the
state of MacOS itself.

> I hereby reiterate my point that doing development on macOS involves
> much sadness, such as creating a developer account to do development.
> I'm afraid there isn't much else you can do, unless you somehow get gcc
> and the rest of the toolchain working without that.

The command-line compilers (clang, swiftc) are available without an App
Store login as before ("sudo xcode-select --install"). But the XCode IDE
is apparently not. As for the OpenSSL headers, I don't know whether they
are intentionally gone for good or only available from the App Store
(and can't tell which is worse :)

>> would it make sense to add MacOS-specific checks to the build-openssl script?
>
> Before you do that, there is some work I've done on a few more eggs I
> maintain, I got fed up with writing user-unfriendly shell scripts that I
> rewrote the non-Windows version to use a Scheme program instead

That sounds like a wise move. And for us who maintain Scheme-related
infrastructure, it's nice to get the chance to actually write some
Scheme code every now and then :) It so often happens that the C/shell
parts take the most time.

> basic version detection, falling back to environment variables and
> finally bailing out with an error.  You can find the latest version of
> it at the breadline repository [1].  Please let me know if that fulfills
> your wishes and if not, whether it can be made to do so.  If yes, then
> I'd be willing to migrate the openssl egg towards such a script as well.
> The reason I haven't done so is because unlike the other eggs I maintain
> it's something I'd rather not touch unnecessarily, breakages to it will
> be far more annoying to handle than anything else.  And honestly
> speaking, OpenSSL isn't nice to deal with either :>

Thank you for inviting feedback! I'll definitely help work things out.

I was thinking, even without considering OpenSSL at all, dealing with
pkg-config by itself is complicated enough that it might be worth baking
special pkg-config support into chicken-install (or whatever the right
internal module is that chicken-install uses).

Some of the BSD's re-implemented the original glib-based pkg-config as
"pkgconf". Luckily, pkgconf installs a pkg-config command as a
compatibility wrapper and it seems quite feature-complete (in the same
way that libressl is with openssl, or clang is with gcc). I haven't had
any problems with pkgconf.

But several OSes don't ship with pkg-config by default, so users have to
install it. In order to be user-friendly, it'd be nice if Chicken
advised them on how to do it. (If Chicken has all the pkg-config
business in one place, instead of separately in each egg, then it's a
very modest amount of code to provide a user-friendly experience.)

I would tentatively recommend supporting pkg-config as the only way to
specify compiler and linker flags for C libraries. I've always thought
that users should have the option to give flags manually instead of
relying on pkg-config. But today I realized that pkg-config .pc files
are basically simple .ini files that contain little more than those same
flags. So if users want manual overrides, we could just ask them to
write their own .pc file and set PKG_CONFIG_PATH to find it.

Also .pc files specify the library version. E.g. the openssl egg
requires openssl 1.0.2 or later, which it can check from the .pc file.
(It's not reliable to check using the "openssl" shell command, since
that may be a different version.) With user-supplied CFLAGS and LDFLAGS
we can't easily make this version check. The .pc file is unlikely to
have the wrong version number unless the user deliberately messes it up
or makes up an arbitrary version number.

But is stuff detailed enough that it should be discussed on the
"hackers" mailing list?

_______________________________________________
Chicken-users mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/chicken-users