Is it safe to lower SHARD_MIN_BLOCK_SIZE?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Is it safe to lower SHARD_MIN_BLOCK_SIZE?

Dmitry Antipov
I'm trying to investigate a few issues found on disperse volumes with sharding
enabled (https://github.com/gluster/glusterfs/issues/1148 for example). Since
testing on huge files may be slow and consume a lot of system resources, I'm
trying to scale SHARD_MIN_BLOCK_SIZE down to very small values, say 4K, and
use 1-4M files for testing. Since this tweak doesn't break replica 3 volumes,
I expect that it should be valid for disperse volumes as well. Is that so?

Dmitry
_______________________________________________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968




Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel

Reply | Threaded
Open this post in threaded view
|

Re: Is it safe to lower SHARD_MIN_BLOCK_SIZE?

Krutika Dhananjay
So SHARD_MIN_BLOCK_SIZE is currently hardcoded to 4MB.
If you want to reduce that value further (purely for the sake of testing, not recommended otherwise), you will need to change its value in the source code here - https://github.com/gluster/glusterfs/blob/43ed6fa34b259b568ae1442b65a5004a894a2654/xlators/features/shard/src/shard.h#L21
and recompile the code. And then set features.shard-block-size to whatever value you intend to use.
I'd also suggest disabling performance.quick-read before you run your tests.

-Krutika

On Mon, Apr 13, 2020 at 7:44 PM Dmitry Antipov <[hidden email]> wrote:
I'm trying to investigate a few issues found on disperse volumes with sharding
enabled (https://github.com/gluster/glusterfs/issues/1148 for example). Since
testing on huge files may be slow and consume a lot of system resources, I'm
trying to scale SHARD_MIN_BLOCK_SIZE down to very small values, say 4K, and
use 1-4M files for testing. Since this tweak doesn't break replica 3 volumes,
I expect that it should be valid for disperse volumes as well. Is that so?

Dmitry
_______________________________________________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968




Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel


_______________________________________________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968




Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel

Reply | Threaded
Open this post in threaded view
|

Re: Is it safe to lower SHARD_MIN_BLOCK_SIZE?

Dmitry Antipov
On 4/14/20 12:16 PM, Krutika Dhananjay wrote:

> So SHARD_MIN_BLOCK_SIZE is currently hardcoded to 4MB.
> If you want to reduce that value further (purely for the sake of testing, not recommended otherwise), you will need to change its value in the source code here -
> https://github.com/gluster/glusterfs/blob/43ed6fa34b259b568ae1442b65a5004a894a2654/xlators/features/shard/src/shard.h#L21
> and recompile the code. And then set features.shard-block-size to whatever value you intend to use.
> I'd also suggest disabling performance.quick-read before you run your tests.

Thanks.

https://github.com/gluster/glusterfs/issues/1148 updated, looks like a heap corruption.

Dmitry
_______________________________________________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968




Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel