sks build times

[Andy Ruddock-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

I'm currently rebuilding the database for keyserver.rainydayz.org.
I'm using the normalbuild rather than the fastbuild.
I'm presuming that the process displays the time taken after it's
processed each file, which means that the rebuild is going to take about
three weeks.
As it's a slow machine I reckoned the extra time spent building the
database would be compensated for it running faster when it was finished.
How much faster is the fastbuild process and how much slower does the
server run when it's finished?
Would it not be possible to get a file dump of the database and start
with that?

- --
Andy Ruddock
- ------------
[hidden email] (GPG Key ID 0xEEC3AFB3)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCgAGBQJL5o6+AAoJEAC80KTuw6+zl4YP/jBKTjD2TkGrdnDC7FpDmWGd
1biWFQ4OA3G71BG8YwQ1MybC5N4vXGjDtpDdIbQpMdK143RBL/+XVbpr74xY8xG+
Oha6IRclDb0lJXa0qz4JbC02QHZ9wTsDLj5RkbCbNdEQxm4Rd/H6VVS+vLHOpVMj
1GoSaUui1318DKxW0EXfBzflNorEMr4SgnKATIHp7N23hCbjGQ25HUb9cFBYcc9d
sjJiGx65RYpIdzM1fOfGj4AOIAZkdF74WHbHR2vdo+BHokdJLtA4jkCbxxZHPdGN
8aLjsIWAtAs6ZfUW3qckoNmHc/hXZUcD6BxOButJVY0w/7jviXAPwH9F0x1sEvXK
1DRAbRtX8m0i8ovn/lUkab+MpSKi5+xc2B2gUbdOrIcEZllmtvYDmSFmz8RoJC63
U093mq1SDSnictypvbJkJGoxukac8usHdozyoCwets1XJG9/Pv65dibm+RAEA7nE
abrwehO51cvlrwChMaJ8ls+X8f35S+87oHI7NSz77kfRIT3Gj/WRckCvOEl85uQi
CEJpC0bBtJrh+/1tki9F3XUFP4/E9/+BEzEmkWNgURCa4hqbX4z8Q1vnij2mbY2e
Lyea7kPS12o+hPGX7FjOKbu3wxdGmPd5zv0neG8WHmzOL6qLP0bNSvu0afmc60Ho
6yATm3B1V3TqlDtK0iUu
=/TK2
-----END PGP SIGNATURE-----


_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

[John Clizbe-3]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Andy Ruddock wrote:
> Hi,
>
> I'm currently rebuilding the database for keyserver.rainydayz.org.
> I'm using the normalbuild rather than the fastbuild.
> I'm presuming that the process displays the time taken after it's
> processed each file, which means that the rebuild is going to take about
> three weeks.

It's the time after processing each set of files, i.e. the value you have -n set
to on the sks build command line. On a 1.5G Hz Sempron box with 1GB RAM, I can
load just under 200,000 keys per pass without starting to swap. So, if I'm using
a dump from pramberger.at where each file is 25k keys, I use -n 7. From
linux.prato.it, each dump is 15k keys, and then I use -n 13.

ls -ltu on your dump directory will tell you the last access time for each file.
Add -r and you'll easily get the last file read in.

build takes about 6 hours on this box and pbuild approx 2.5 hrs.

> As it's a slow machine I reckoned the extra time spent building the
> database would be compensated for it running faster when it was finished.
> How much faster is the fastbuild process and how much slower does the
> server run when it's finished?

I don't know that this has ever been measured. I've never built with fastbuild

> Would it not be possible to get a file dump of the database and start
> with that?

I guess that could be done, but it would be significantly larger than the
existing key dumps.

- --
John P. Clizbe                      Inet:John (a) gingerbear.net
You can't spell fiasco without SCO. hkp://keyserver.gingerbear.net  or
     mailto:[hidden email]?subject=HELP

Q:"Just how do the residents of Haiku, Hawai'i hold conversations?"
A:"An odd melody / island voices on the winds / surplus of vowels"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11-svn5331-2010-05-07 (Windows XP)
Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl!
Comment: Be part of the £€37 ECHELON -- Use Strong Encryption.
Comment: It's YOUR right - for the time being.
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iF4EAREIAAYFAkvnGp8ACgkQ614Z89ZWmCV4dgD/W4wRWGaxV6CeXWXklYEcEMiq
d7WdVfnmO7TqJwd6/9QA/2a9//vjnn5J0Dbw2izWGV9djWTdRDSuQe2c1XIVRVI4
=VgPN
-----END PGP SIGNATURE-----


_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

[Kim Minh Kaplan]

Andy Ruddock writes:

> Hi,
>
> I'm currently rebuilding the database for keyserver.rainydayz.org.
> I'm using the normalbuild rather than the fastbuild.
> I'm presuming that the process displays the time taken after it's
> processed each file, which means that the rebuild is going to take about
> three weeks.
> As it's a slow machine I reckoned the extra time spent building the
> database would be compensated for it running faster when it was finished.
> How much faster is the fastbuild process and how much slower does the
> server run when it's finished?

When I had a quick glance at this[1], the default settings for Berkeley
DB's page size seamed way too low; you could try increasing it.  But
once the key database is fully populated it occupies more room than one
built with fastbuild.  I suspect that performances will not change much
in the day to day operation of your SKS server, but please tell us what
your experience is.

> Would it not be possible to get a file dump of the database and start
> with that?

It should be but I am not sure what kind of gain you'd expect.

[1] http://lists.gnu.org/archive/html/sks-devel/2009-08/msg00058.html
--
Kim Minh

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel

[Jeff Johnson-12]

On May 12, 2010, at 3:14 PM, Kim Minh Kaplan wrote:
>
> It should be but I am not sure what kind of gain you'd expect.
>

The gain is not in performance, but rather "referential integrity"
and a simpler easier to maintain data store w/o a key dump.

The KDB store is tied to one specific key dump.

(aside)
There's another conscious design choice related to "referential intergrity"
splitting the PTRee from the KDB store. Both stores must be maintained
in parallel (or the PTree store must be rebuilt when/as needed).

But -fastbuild (what I use) performs quite well, and the PTree store can
be rebuilt if/when needed, and a new keydump can be loaded whenever, gud enuf.

73 de Jeff

_______________________________________________
Sks-devel mailing list
[hidden email]
http://lists.nongnu.org/mailman/listinfo/sks-devel