Patch to compile against Botan 2.x

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Patch to compile against Botan 2.x

Jack Lloyd-2

I've attached a patch for building Monotone against Botan 2.x

This isn't quite complete for 2.2, because a lot of Monotone assumes
Botan::SecureVector is a type, and that botan.h includes more than it
does in 2.0-2.2. I made a change upstream
https://github.com/randombit/botan/commit/3f7dba2c4455bf53dae89d088bd56cdf9b2c94fe
to make some changes to botan.h to make this patch simpler, and in the
thought it will likely ease transition for other projects. This will
be included in 2.3 which is coming out next week.

Two other relevant pieces of information:

- All support for Botan 1.10 ends at the end of this year

- Botan now uses semantic versioning, so all Botan 2.x releases should
  be forward compatible. It is anticipated 2.x will be supported until
  at least 2021.

With this patch Monotone builds for me on Arch Linux x86-64. I ran the
tests, some passed and some failed, I did not investigate this further.

Hope this helps,
  Jack Lloyd


_______________________________________________
Monotone-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/monotone-devel

monotone_for_botan2.patch (22K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Patch to compile against Botan 2.x

Markus Wanner-2
On 09/26/2017 09:47 PM, Jack Lloyd wrote:
> I've attached a patch for building Monotone against Botan 2.x

wow, that's great! Thank you, Jack.

I'll have a look this weekend and will try to commit this.

Kind Regards

Markus Wanner

_______________________________________________
Monotone-devel mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/monotone-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patch to compile against Botan 2.x

Markus Wanner-2
In reply to this post by Jack Lloyd-2
Hello Jack,

On 09/26/2017 09:47 PM, Jack Lloyd wrote:
> I've attached a patch for building Monotone against Botan 2.x

thanks again and sorry for not getting back to you earlier, as promised.

I committed your patch to net.venge.monotone.botan and continued from
there on. Unfortunately, it didn't quite compile against any newer Botan
version, so I continued to work on it.

Plus, I too started some Botan 1.11 compatibility efforts back some
time.. so now I'm facing two heads on that branch and I'm still in the
process of merging those.

> This isn't quite complete for 2.2, because a lot of Monotone assumes
> Botan::SecureVector is a type, and that botan.h includes more than it
> does in 2.0-2.2.

I added a new header file (src/botan.hh) to deduplicate some of the
conditional imports.

> I made a change upstream
> https://github.com/randombit/botan/commit/3f7dba2c4455bf53dae89d088bd56cdf9b2c94fe
> to make some changes to botan.h to make this patch simpler, and in the
> thought it will likely ease transition for other projects. This will
> be included in 2.3 which is coming out next week.

Cool, thanks. Given this won't appear before 2.3, I think monotone still
needs dedicated includes for e.g. botan/filters.h to support 2.0 - 2.2,
right?

Another unrelated question: You changed a couple of (not necessarily
secure) byte vectors to DataSource_Memory. Whereas I figured I might
simply use a vector<Botan::bytes>. What's the difference?

> Two other relevant pieces of information:
>
> - All support for Botan 1.10 ends at the end of this year

I don't known about other distros, but it's what Debian stable currently
ships. And given it's just been released, I fear that statement will
hold true for another roughly 2 years.

I think it would make sense to require at least 1.10 from now on and
drop everything older than that. I'm hesitant dropping 1.10 just yet.
What do others think about dropping support for older Botan versions?

> - Botan now uses semantic versioning, so all Botan 2.x releases should
>   be forward compatible. It is anticipated 2.x will be supported until
>   at least 2021.

+1, I welcome that simplification.

Kind Regards

Markus Wanner

_______________________________________________
Monotone-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/monotone-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patch to compile against Botan 2.x

Jack Lloyd-2
On Sun, Oct 15, 2017 at 07:36:23PM +0200, Markus Wanner wrote:

> I committed your patch to net.venge.monotone.botan and continued from
> there on. Unfortunately, it didn't quite compile against any newer Botan
> version, so I continued to work on it.

Ah sorry about that, I will take another look at the patch and see
what happened.

> I added a new header file (src/botan.hh) to deduplicate some of the
> conditional imports.

That's a good idea.

> Cool, thanks. Given this won't appear before 2.3, I think monotone still
> needs dedicated includes for e.g. botan/filters.h to support 2.0 - 2.2,
> right?

Yes. It would probably be better to just include filters.h where
needed, instead of assuming botan.h pulls it in.

> Another unrelated question: You changed a couple of (not necessarily
> secure) byte vectors to DataSource_Memory. Whereas I figured I might
> simply use a vector<Botan::bytes>. What's the difference?

Hmm I'll have to look at the patch and get back to you on that, I
remember there was a good reason for that change but do not recall
what now.

> I don't known about other distros, but it's what Debian stable currently
> ships. And given it's just been released, I fear that statement will
> hold true for another roughly 2 years.

I know :( This is also true for EPEL7, Ubuntu 16.04, etc. Botan 2.x
package for Fedora/RHEL is created and past review, but now stuck in
some kind of limbo with RedHat legal review (they are paranoid about
patents). I contacted the current Debian maintainer for Botan 1.10
about submitting a new package for 2.x but have not heard back.

I think what's going to happen in practice is that after end of this
year we'll still continue to fix truly critical bugs in 1.10 (ie
remote code exec) if they arise, but stop backporting fixes for side
channels.

> I think it would make sense to require at least 1.10 from now on and
> drop everything older than that. I'm hesitant dropping 1.10 just yet.
> What do others think about dropping support for older Botan versions?

The only supported Linux/BSD distro I know of that is still shipping
1.8 is RHEL6/EPEL6. Everyone else is either 2.x or 1.10. 1.8 is good
and dead by now (there was a final patch roundup release in 2016 but
besides that the last release was in 2012).

Jack

_______________________________________________
Monotone-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/monotone-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patch to compile against Botan 2.x

Markus Wanner-2
Jack,

On 15.10.2017 20:49, Jack Lloyd wrote:
> Ah sorry about that, I will take another look at the patch and see
> what happened.

it might have been caused by my changes, as I've applied your patch onto
some different base revision, IIRC.

Anyways, I've now merged our work in net.venge.monotone.botan and at
least mtn now compiles (for me, too) against Botan 2.0. Not the test
suite, though.

And with simply invoking `mtn status`, I already get some CRC-32 error
(presumably the gzip stuff, which I'd love to get rid of...).

2751100d is the newest revision. I fear I'll have to defer further work
on that branch to next week.

Kind Regards

Markus Wanner

_______________________________________________
Monotone-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/monotone-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patch to compile against Botan 2.x

grarpamp
In reply to this post by Jack Lloyd-2
Here is what FreeBSD packages ship...

"origin":"devel/monotone","version":"1.1_8" has maintainer listed
"origin":"devel/monotone-viz","version":"1.0.2_7" needs maintainer
"security/botan110","version":"1.10.13_6" same maintainer as monotone
 used by: monone, monotone-viz, bundy, softhsm, encryptpad

_______________________________________________
Monotone-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/monotone-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patch to compile against Botan 2.x

Markus Wanner-2
In reply to this post by Markus Wanner-2
Hi,

I finally made some progress on the nvm.botan branch. I figured two
important things:

First, the CRC32 calculation was broken due to comparing it with the
entire 8 bytes of the footer (which couldn't ever match). That was easy
to fix.

Second, our custom gzip filter sets the timestamp in the gzip header to
all all zeroes. More importantly, the parser *requires* it to be all
zeroes. This does not work with more standard gzip behaviour (like the
one implemented with Botan's (De)Compression_Filter that's available
since 2.0 and newer).

I already made the parser more tolerant (in nvm.botan). However, we
cannot simply start writing non-zero timestamps without breaking
backwards-compatibility. Therefore, I left the custom gzip code in place
and in use, even when compiled against newer Botan versions.

It's also worth mentioning that I've already dropped support for Botan
1.6.x and 1.7.x. However, the 1.8.x still works, so I kept it for now.
I'm tempted to drop it as soon as it poses problems, though. (I haven't
even tested Botan 1.6, so the decision is pretty arbitrary.)

All of the unit tests now pass with the following Botan versions I'm
currently testing against: 1.8.15, 1.10.17, 2.0.1, 2.1.0, 2.2.0, and
2.3.0. However, I'm still facing Botan-version dependent failures on
various functional tests. I'll investigate on those, next.

Kind Regards

Markus Wanner

_______________________________________________
Monotone-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/monotone-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patch to compile against Botan 2.x

Markus Wanner-2
On 11/08/2017 10:39 PM, Markus Wanner wrote:
> All of the unit tests now pass with the following Botan versions I'm
> currently testing against: 1.8.15, 1.10.17, 2.0.1, 2.1.0, 2.2.0, and
> 2.3.0. However, I'm still facing Botan-version dependent failures on
> various functional tests. I'll investigate on those, next.

I corrected the PKCS #5 key writing by specifying "SHA-160" instead of
"SHA-1" for Botan versions 2.0 and newer.

With that last fix all functional tests now pass and nvm.botan is ready
to be merged on mainline. Anybody up for a quick review?

Kind Regards

Markus Wanner

_______________________________________________
Monotone-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/monotone-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patch to compile against Botan 2.x

Markus Wanner-2
On 11/11/2017 11:24 AM, Markus Wanner wrote:
> I corrected the PKCS #5 key writing by specifying "SHA-160" instead of> "SHA-1" for Botan versions 2.0 and newer.

I figured I forgot to actually commit this (had the commit log prepared
and everything, but...). Done now: 6e44856c.

Kind Regards

Markus

_______________________________________________
Monotone-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/monotone-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patch to compile against Botan 2.x

Markus Wanner-2
In reply to this post by Markus Wanner-2
On 11/11/2017 11:24 AM, Markus Wanner wrote:
> With that last fix all functional tests now pass and nvm.botan is ready
> to be merged on mainline. Anybody up for a quick review?

I count the lack of objections as a frantic "Hell, yes, go merge!" and
did so. The main branch should now compile against all of the Botan 2.x
series.

Kind Regards

Markus Wanner

_______________________________________________
Monotone-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/monotone-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patch to compile against Botan 2.x

Stephen Leake-3
Markus Wanner <[hidden email]> writes:

> On 11/11/2017 11:24 AM, Markus Wanner wrote:
>> With that last fix all functional tests now pass and nvm.botan is ready
>> to be merged on mainline. Anybody up for a quick review?
>
> I count the lack of objections as a frantic "Hell, yes, go merge!" and
> did so. The main branch should now compile against all of the Botan 2.x
> series.

Thanks for your work on keeping mtn up to date.

Sadly, the public server I've been using (ada-france) seems to be going
away, and I'll be forced to switch to git for Gnu Savannah (for my Emacs
projects).

But I plan to continue using mtn for local work, as long as it keeps
working.

--
-- Stephe

_______________________________________________
Monotone-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/monotone-devel
Reply | Threaded
Open this post in threaded view
|

Re: Patch to compile against Botan 2.x

Jack Lloyd-2
In reply to this post by Markus Wanner-2
Hi Markus,

Sorry for the long delayed reply, this ended up going into a mail folder I don't
usually look in.

On Wed, Nov 08, 2017 at 10:39:55PM +0100, Markus Wanner wrote:

> Second, our custom gzip filter sets the timestamp in the gzip header to
> all all zeroes. More importantly, the parser *requires* it to be all
> zeroes. This does not work with more standard gzip behaviour (like the
> one implemented with Botan's (De)Compression_Filter that's available
> since 2.0 and newer).
>
> I already made the parser more tolerant (in nvm.botan). However, we
> cannot simply start writing non-zero timestamps without breaking
> backwards-compatibility. Therefore, I left the custom gzip code in place
> and in use, even when compiled against newer Botan versions.

Thanks for pointing this out. For 2.4 I'm going to change the code so that by
default the time field is zeros, and then the user can optionally supply a time
if they want it set in the header. (Ironically I just recently opened a bug in
Python because the tarfile module has the same behavior of always using the
system clock in the gzip header with no way to override it, oops.)

One (albeit hackish) fix would be to just zero out the timestamp field in the
header before writing. (Oh but this might break the CRC? Not sure.) However in
practice you're probably better of just continuing to use your custom filter, as
not all distributions that package Botan include the compression code.

> All of the unit tests now pass with the following Botan versions I'm
> currently testing against: 1.8.15, 1.10.17, 2.0.1, 2.1.0, 2.2.0, and
> 2.3.0. However, I'm still facing Botan-version dependent failures on
> various functional tests. I'll investigate on those, next.

From the NEWS file it looks like this has landed on nvm now, thank you!

The good news is, Botan 2.x should remain API stable and is supported through at
least 2021. And even in future major releases, API changes will be limited to
features that are actively marked as deprecated. So if you do notice any problem
caused by using a future version of Botan it's almost certainly a bug in Botan,
so just mail me or open a Github issue and we'll take care of it.

Jack

_______________________________________________
Monotone-devel mailing list
[hidden email]
https://lists.nongnu.org/mailman/listinfo/monotone-devel