【replace-brick failed but make there’re two same client-id of the gluster cluster, which lead can’t mount the gluster anymore】

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

【replace-brick failed but make there’re two same client-id of the gluster cluster, which lead can’t mount the gluster anymore】

Frank Yu

Hi guys, 

I met a terrible situations need all your helps. I have a production cluster running well at first. the version of gluster is 3.12.15-1.el7.x86_64, the cluster has 12 nodes, 12 brick(disk) per nodes, there is one distributed-replicate volume, with 144 bricks(48 * 3). 

then there is a node crushed(the node named nodeA), and all it’s disk can’t be used anymore, but since the os of nodes run on kvm machine, so it came back with 12 new disks. 

I try to replace the first brick of nodeA with cmd ‘gluster volume replace-brick VOLUMENAME nodeA:/mnt/data-1/data  nodeA:/mnt/data-1/data01 commit force’, after some times, it failed with error “Error : Request timed out”.

here came the problem, both ‘nodeA:/mnt/data-1/data’ and ‘nodeA:/mnt/data-1/data01’ show in the output of cmd ‘gluster volume info’

When I try to mount gluster to client with fuse, it report error like below: 

[2019-08-12 12:27:42.395440] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.15 (args: /usr/sbin/glusterfs --volfile-server=xxxxx --volfile-id=/training-data-ali /mnt/glusterfs)

[2019-08-12 12:27:42.400015] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction

[2019-08-12 12:27:42.404994] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1

[2019-08-12 12:27:42.415971] E [MSGID: 101179] [graph.y:153:new_volume] 0-parser: Line 1381: volume ‘VOLUME-NAME-client-74' defined again

[2019-08-12 12:27:42.416124] E [MSGID: 100026] [glusterfsd.c:2358:glusterfs_process_volfp] 0-: failed to construct the graph

[2019-08-12 12:27:42.416376] E [graph.c:1102:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x532) [0x55898e35e092] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x150) [0x55898e357da0] -->/lib64/libglusterfs.so.0(glusterfs_graph_destroy+0x84) [0x7f95f7318754] ) 0-graph: invalid argument: graph [Invalid argument]

[2019-08-12 12:27:42.416425] W [glusterfsd.c:1375:cleanup_and_exit] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x532) [0x55898e35e092] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x163) [0x55898e357db3] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55898e35732b] ) 0-: received signum (-1), shutting down

[2019-08-12 12:27:42.416455] I [fuse-bridge.c:5852:fini] 0-fuse: Unmounting '/mnt/glusterfs'.

[2019-08-12 12:27:42.429655] I [fuse-bridge.c:5857:fini] 0-fuse: Closing fuse connection to '/mnt/glusterfs-aliyun'.

[2019-08-12 12:27:42.429759] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f95f6140e25] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55898e3574b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55898e35732b] ) 0-: received signum (15), shutting down

So, how can I solve error “Line 1381: volume ‘VOLUME-NAME-client-74' defined again” 

Frank Yu


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/836554017

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/486278655

Gluster-devel mailing list
[hidden email]