Re: RDMA in GlusterFs

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: RDMA in GlusterFs

RAFI KC
Hi Liu,

RDMA support in GlusterFS has been dropped from the latest release and removed from the codebase. You can find more details here in the mail sent as the proposal to drop the feature[1].

<snippet from proposal mail>
Gluster
started supporting RDMA while ib-verbs was still new, and very high-end
infra around that time were using Infiniband. Engineers did work with
Mellanox, and got the technology into GlusterFS for better data migration,
data copy. While current day kernels support very good speed with IPoIB
module itself, and there are no more bandwidth for experts in these area to
maintain the feature, we recommend migrating over to TCP (IP based) network
for your volume.If you are successfully using RDMA transport, do get in
touch with us to prioritize the migration plan for your volume. Plan is to
work on this after the release, so by version 6.0, we will have a cleaner
transport code, which just needs to support one type.
</end od snippet>

That said, Officially GlusterFs doesn't support RDMA on master branch.

[1] : https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html


Regards
Rafi KC

>
>
> Does anyone use RDMA in Glusterfs before? What’s RDMA usability in
> Glusterfs now?
>
>
>
> *From:* Liu, Changcheng
> *Sent:* Tuesday, October 1, 2019 11:24 PM
> *To:* 'Yaniv Kaul' <[hidden email]>; 'Amar Tumballi' <[hidden email]>
> *Cc:* '[hidden email]' <[hidden email]>
> *Subject:* RE: RDMA in GlusterFs
>
>
>
> Does anyone know whether Glusterfs support RDMA or not in master branch
> now?
>
>
>
> *From:* Liu, Changcheng
> *Sent:* Tuesday, October 1, 2019 12:16 AM
> *To:* Yaniv Kaul <[hidden email]>; Amar Tumballi <[hidden email]>
> *Cc:* [hidden email]
> *Subject:* RDMA in GlusterFs
>
>
>
> Hi Yaniv Kaul & Amar Tumbali
>
> Upstream question:
>
>        Does Glusterfs support RDMA after below PR is merged?
>
>              ibverbs/rdma: remove from build:
> https://github.com/gluster/glusterfs/commit/8f37561d665b
>
>
>
> B.R.
>
> Changcheng
>
_______________________________________________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel

Reply | Threaded
Open this post in threaded view
|

Re: RDMA in GlusterFs

Amar Tumballi-2
If it is for academic purposes, I suggest using glusterfs-3.12.x or glusterfs-5.x series with RDMA... That would give an idea.

-Amar

On Tue, Oct 8, 2019 at 7:44 PM Liu, Changcheng <[hidden email]> wrote:
Thanks Rafi-KC & Tumbali.
I'm working on developing RDMA in Ceph and doing some investigating about RDMA usage in distributed storage system.
It's unlucky that GlusterFs won't support RDMA now.

For IPoIB, I'll try to evaluate its performance versus RDMA(RoCEv2 & iWARP).

-----Original Message-----
From: Rafi Kavungal Chundattu Parambil [mailto:[hidden email]]
Sent: Monday, October 7, 2019 5:07 PM
To: Liu, Changcheng <[hidden email]>
Cc: [hidden email]; [hidden email]
Subject: Re: RDMA in GlusterFs

Hi Liu,

RDMA support in GlusterFS has been dropped from the latest release and removed from the codebase. You can find more details here in the mail sent as the proposal to drop the feature[1].

<snippet from proposal mail>
Gluster
started supporting RDMA while ib-verbs was still new, and very high-end infra around that time were using Infiniband. Engineers did work with Mellanox, and got the technology into GlusterFS for better data migration, data copy. While current day kernels support very good speed with IPoIB module itself, and there are no more bandwidth for experts in these area to maintain the feature, we recommend migrating over to TCP (IP based) network for your volume.If you are successfully using RDMA transport, do get in touch with us to prioritize the migration plan for your volume. Plan is to work on this after the release, so by version 6.0, we will have a cleaner transport code, which just needs to support one type.
</end od snippet>

That said, Officially GlusterFs doesn't support RDMA on master branch.

[1] : https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html


Regards
Rafi KC
>
>
> Does anyone use RDMA in Glusterfs before? What’s RDMA usability in
> Glusterfs now?
>
>
>
> *From:* Liu, Changcheng
> *Sent:* Tuesday, October 1, 2019 11:24 PM
> *To:* 'Yaniv Kaul' <[hidden email]>; 'Amar Tumballi'
> <[hidden email]>
> *Cc:* '[hidden email]' <[hidden email]>
> *Subject:* RE: RDMA in GlusterFs
>
>
>
> Does anyone know whether Glusterfs support RDMA or not in master
> branch now?
>
>
>
> *From:* Liu, Changcheng
> *Sent:* Tuesday, October 1, 2019 12:16 AM
> *To:* Yaniv Kaul <[hidden email]>; Amar Tumballi <[hidden email]>
> *Cc:* [hidden email]
> *Subject:* RDMA in GlusterFs
>
>
>
> Hi Yaniv Kaul & Amar Tumbali
>
> Upstream question:
>
>        Does Glusterfs support RDMA after below PR is merged?
>
>              ibverbs/rdma: remove from build:
> https://github.com/gluster/glusterfs/commit/8f37561d665b
>
>
>
> B.R.
>
> Changcheng
>

_______________________________________________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel