Toggle storage.linux-aio and volume restart

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Toggle storage.linux-aio and volume restart

Dmitry Antipov
Is it expected that toggling storage.linux-aio should take an effect immediately without
doing the full volume restart (with 'gluster volume stop' and 'gluster volume start')?

It seems that if the volume was started with AIO enabled, it's possible to disable it
with 'gluster volume set XXX storage.linux-aio off', but reverting the latter back to
'on' makes no effect until volume restart.

Dmitry
-------

Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk

Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel

Reply | Threaded
Open this post in threaded view
|

Re: Toggle storage.linux-aio and volume restart

Amar Tumballi-3
Depends on if posix_reconfigure() handles this option or not. If it handles it, then without volume restart, we can use it.

-Amar

On Tue, Dec 8, 2020 at 6:59 PM Dmitry Antipov <[hidden email]> wrote:
Is it expected that toggling storage.linux-aio should take an effect immediately without
doing the full volume restart (with 'gluster volume stop' and 'gluster volume start')?

It seems that if the volume was started with AIO enabled, it's possible to disable it
with 'gluster volume set XXX storage.linux-aio off', but reverting the latter back to
'on' makes no effect until volume restart.

Dmitry
-------

Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk

Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel



--
--
Container Storage made easy!


-------

Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk

Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel

Reply | Threaded
Open this post in threaded view
|

Re: Toggle storage.linux-aio and volume restart

Dmitry Antipov
On 12/8/20 4:47 PM, Amar Tumballi wrote:

> Depends on if posix_reconfigure() handles this option or not. If it handles it, then without volume restart, we can use it.

Sure it handles. But I do not understand how the following becomes possible:

Thread 20 "glfs_iotwr003" hit Breakpoint 1, posix_writev (frame=0x7fb068003bf8, this=0x7fb09c008b10, fd=0x7fb0781ee528, vector=0x7fb090293750, count=1,
     offset=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at posix-inode-fd-ops.c:1929
1929    int32_t op_errno = 0;
(gdb) bt
#0  posix_writev (frame=0x7fb068003bf8, this=0x7fb09c008b10, fd=0x7fb0781ee528, vector=0x7fb090293750, count=1, offset=653123584, flags=0, iobref=0x7fb0a4088120,
     xdata=0x7fb090394ac8) at posix-inode-fd-ops.c:1929
#1  0x00007fb0bbd9b422 in default_writev (frame=frame@entry=0x7fb068003bf8, this=<optimized out>, fd=fd@entry=0x7fb0781ee528, vector=vector@entry=0x7fb090293750,
     count=count@entry=1, off=off@entry=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at defaults.c:2550
#2  0x00007fb0a90405fb in changelog_writev (frame=frame@entry=0x7fb068003ae8, this=0x7fb09c00ec80, fd=fd@entry=0x7fb0781ee528, vector=vector@entry=0x7fb090293750,
     count=count@entry=1, offset=offset@entry=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at changelog.c:1763
#3  0x00007fb0a90c9eaf in br_stub_writev (frame=0x7fb068019968, this=0x7fb09c0110b0, fd=0x7fb0781ee528, vector=0x7fb090293750, count=1, offset=653123584, flags=0,
     iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at bit-rot-stub.c:2112
#4  0x00007fb0bbd9b422 in default_writev (frame=frame@entry=0x7fb068019968, this=<optimized out>, fd=fd@entry=0x7fb0781ee528, vector=vector@entry=0x7fb090293750,
     count=count@entry=1, off=off@entry=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at defaults.c:2550
#5  0x00007fb0a8ffd4d4 in pl_writev (frame=0x7fb0680039d8, this=0x7fb09c014d70, fd=0x7fb0781ee528, vector=0x7fb090293750, count=1, offset=653123584, flags=0,
     iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at posix.c:2377
#6  0x00007fb0a8fcd99b in worm_writev (frame=0x7fb0680039d8, this=<optimized out>, fd=0x7fb0781ee528, vector=0x7fb090293750, count=1, offset=653123584, flags=0,
     iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at worm.c:429
#7  0x00007fb0a8fbed0f in ro_writev (frame=frame@entry=0x7fb0680039d8, this=<optimized out>, fd=fd@entry=0x7fb0781ee528, vector=vector@entry=0x7fb090293750,
     count=count@entry=1, off=off@entry=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at read-only-common.c:374
#8  0x00007fb0a8fa60a6 in leases_writev (frame=frame@entry=0x7fb0680038c8, this=0x7fb09c01ab10, fd=fd@entry=0x7fb0781ee528, vector=vector@entry=0x7fb090293750,
     count=count@entry=1, off=off@entry=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at leases.c:132
#9  0x00007fb0a8f90d4a in up_writev (frame=frame@entry=0x7fb068003338, this=0x7fb09c01c8d0, fd=fd@entry=0x7fb0781ee528, vector=vector@entry=0x7fb090293750,
     count=count@entry=1, off=off@entry=653123584, flags=0, iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at upcall.c:124
#10 0x00007fb0bbdb08c9 in default_writev_resume (frame=0x7fb09022fbc8, this=0x7fb09c01e640, fd=0x7fb0781ee528, vector=0x7fb090293750, count=1, off=653123584, flags=0,
     iobref=0x7fb0a4088120, xdata=0x7fb090394ac8) at defaults.c:1831
#11 0x00007fb0bbd38ba8 in call_resume_wind (stub=<optimized out>) at call-stub.c:1922
#12 0x00007fb0bbd38d95 in call_resume (stub=0x7fb0900b7a08) at call-stub.c:2392
#13 0x00007fb0a8f72740 in iot_worker (data=0x7fb09c05b490) at io-threads.c:232
#14 0x00007fb0bbab63f9 in start_thread (arg=0x7fb0a8200640) at pthread_create.c:463
#15 0x00007fb0bb6f5903 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) p this
$1 = (xlator_t *) 0x7fb09c008b10                                                           ;; xlator in subject
(gdb) p this->fops->readv
$2 = (fop_readv_t) 0x7fb0a90ff530 <posix_readv>
(gdb) p this->fops->writev
$3 = (fop_writev_t) 0x7fb0a91002e0 <posix_writev>
(gdb) p this->private
$4 = (void *) 0x7fb09c07ce40
(gdb) p ((struct posix_private *)(this->private))->aio_configured                          ;; has AIO configured
$5 = true
(gdb) p ((struct posix_private *)(this->private))->aio_init_done                           ;; and initialized (i.e. AIO processing thread is running)
$6 = true
(gdb) info threads
   Id   Target Id                                            Frame
   ...
   12   Thread 0x7fb0a8bf5640 (LWP 447802) "glfs_posixaio"   syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
   ...
(gdb) thr 12                                                                               ;; this is the one
[Switching to thread 12 (Thread 0x7fb0a8bf5640 (LWP 447802))]
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38 cmpq $-4095, %rax /* Check %rax for error.  */
(gdb) bt
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007fb0a90b530b in __io_getevents_0_4 (ctx=0x7fb0a8bf6000, min_nr=1, nr=16, events=0x7fb0a8bf3d60, timeout=0x0) at io_getevents.c:25
#2  0x00007fb0a90b5361 in io_getevents_0_4 (ctx=<optimized out>, min_nr=min_nr@entry=1, nr=nr@entry=16, events=events@entry=0x7fb0a8bf3d60, timeout=timeout@entry=0x0)
     at io_getevents.c:54
#3  0x00007fb0a90eab62 in posix_aio_thread (data=0x7fb09c008b10) at posix-aio.c:422        ;; xlator passed to AIO thread
#4  0x00007fb0bbab63f9 in start_thread (arg=0x7fb0a8bf5640) at pthread_create.c:463
#5  0x00007fb0bb6f5903 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) p (xlator_t *)0x7fb09c008b10                                                         ;; xlator in subject
$7 = (xlator_t *) 0x7fb09c008b10
(gdb) p ((xlator_t *)0x7fb09c008b10)->fops->readv
$8 = (fop_readv_t) 0x7fb0a90ff530 <posix_readv>
(gdb) p ((xlator_t *)0x7fb09c008b10)->fops->writev
$9 = (fop_writev_t) 0x7fb0a91002e0 <posix_writev>

IOW if aio_configured is true, fops->readv and fops->writev should be set to posix_aio_readv()
and posix_aio_writev(), respectively. But the whole picture looks like something in xlator
graph silently reverts fops->readv and fops->writev back to posix_xxxv() defaults.

Dmitry
-------

Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk

Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel

Reply | Threaded
Open this post in threaded view
|

Re: Toggle storage.linux-aio and volume restart

Ravishankar N

On 08/12/20 9:15 pm, Dmitry Antipov wrote:
> IOW if aio_configured is true, fops->readv and fops->writev should be
> set to posix_aio_readv()
> and posix_aio_writev(), respectively. But the whole picture looks like
> something in xlator
> graph silently reverts fops->readv and fops->writev back to
> posix_xxxv() defaults.
Looks like the (priv->io_uring_configured) check (I added this) which
comes after the (priv->aio_configure check) in posix_reconfigure() is
overwriting this.

-Ravi

-------

Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk

Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel

Reply | Threaded
Open this post in threaded view
|

Re: Toggle storage.linux-aio and volume restart

Ravishankar N

On 09/12/20 10:39 am, Ravishankar N wrote:

>
> On 08/12/20 9:15 pm, Dmitry Antipov wrote:
>> IOW if aio_configured is true, fops->readv and fops->writev should be
>> set to posix_aio_readv()
>> and posix_aio_writev(), respectively. But the whole picture looks
>> like something in xlator
>> graph silently reverts fops->readv and fops->writev back to
>> posix_xxxv() defaults.
> Looks like the (priv->io_uring_configured) check (I added this) which
> comes after the (priv->aio_configure check) in posix_reconfigure() is
> overwriting this.

And considering we do not have graph switch implemented on the server
side, it is undesirable to toggle the fop dispatch table when there
could be in-fight fops. Perhaps we should disallow changing linux-aio on
the fly, like how it is done for linux-io_uring (see the check added in
glusterd_op_stage_set_volume() for io_uring).

-Ravi

>
> -Ravi
>
> -------
>
> Community Meeting Calendar:
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
>
> Gluster-devel mailing list
> [hidden email]
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>

-------

Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk

Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel

Reply | Threaded
Open this post in threaded view
|

Repository access and review policy

Dmitry Antipov
In reply to this post by Dmitry Antipov
Is it possible to request reviews for pull requests from the persons without write access?
Considering myself as an example, this may be https://github.com/gluster/glusterfs/pull/1953.

Dmitry
-------

Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk

Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel

Reply | Threaded
Open this post in threaded view
|

Re: Repository access and review policy

Amar Tumballi-3
Add the Reviewers in comment for now!

(@amarts is me on github). Lets see how better we can handle this.

-Amar

On Tue, Jan 12, 2021 at 10:46 PM Dmitry Antipov <[hidden email]> wrote:
Is it possible to request reviews for pull requests from the persons without write access?
Considering myself as an example, this may be https://github.com/gluster/glusterfs/pull/1953.

Dmitry
-------

Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk

Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel



--
--
Container Storage made easy!


-------

Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk

Gluster-devel mailing list
[hidden email]
https://lists.gluster.org/mailman/listinfo/gluster-devel