RetroSearch Browse

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/vmware/docker-volume-vsphere/issues/1675 below:

Volume stays attached if the ESX of the vm goes down and vsphere HA moves vm to different ESX · Issue #1675 · vmware-archive/vsphere-storage-for-docker · GitHub

Test Steps:

1. Created a volume.
2. Ran a container with volume mounted and wrote data to it.
3. Turned on vsphere HA.
4. While container is running, powered-off the esx on which vm resides. 
5. vSphere HA power-off's the vm and restarts it on a different ESX.
6. After vm successfully comes back up, volume shows up as 'attached' even though there are no containers running.
Also volume removal fails with error: Volume is in use.

Logs have been attached at the end.

Steps and their output are as follows:

Created a volume.

root@sc-rdops-vm02-dhcp-52-237:~# docker volume create -d vsphere:latest --name TestVol
TestVol
root@sc-rdops-vm02-dhcp-52-237:~#
root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
DRIVER              VOLUME NAME
vsphere:latest      TestVol@sharedVmfs-0
root@sc-rdops-vm02-dhcp-52-237:~# docker volume inspect TestVol@sharedVmfs-0
[
    {
        "Driver": "vsphere:latest",
        "Labels": null,
        "Mountpoint": "/mnt/vmdk/TestVol@sharedVmfs-0",
        "Name": "TestVol@sharedVmfs-0",
        "Options": {},
        "Scope": "global",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "13MB",
                "size": "100MB"
            },
            "clone-from": "None",
            "created": "Tue Aug  1 19:10:53 2017",
            "created by VM": "ubuntu-VM0.1",
            "datastore": "sharedVmfs-0",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "detached"
        }
    }
]
root@sc-rdops-vm02-dhcp-52-237:~#

[root@sc-rdops-vm01-dhcp-0-74:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls
Volume   Datastore     VMGroup   Capacity  Used  Filesystem  Policy  Disk Format  Attached-to  Access      Attach-as               Created By    Created Date
-------  ------------  --------  --------  ----  ----------  ------  -----------  -----------  ----------  ----------------------  ------------  ------------------------
TestVol  sharedVmfs-0  _DEFAULT  100MB     13MB  ext4        N/A     thin         detached     read-write  independent_persistent  ubuntu-VM0.1  Tue Aug  1 19:10:53 2017

[root@sc-rdops-vm01-dhcp-0-74:~]

Turned on vSphere HA.
Ran the container with volume mounted and then wrote the data to it.

root@sc-rdops-vm02-dhcp-52-237:~#  docker run -it --volume-driver=vsphere:latest -v TestVol@sharedVmfs-0:/vol1 --name ubHA ubuntu
root@c4dacf722bf7:/#
root@c4dacf722bf7:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vol1
root@c4dacf722bf7:/#
root@c4dacf722bf7:/# echo "Hello" > vol1/hello
root@c4dacf722bf7:/# cat vol1/hello
Hello
root@c4dacf722bf7:/#
root@c4dacf722bf7:/# echo "World" >> vol1/hello
root@c4dacf722bf7:/# cat vol1/hello
Hello
World
root@c4dacf722bf7:/#

Admin cli volume ls also shows volume as attached.

[root@sc-rdops-vm01-dhcp-0-74:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls
Volume   Datastore     VMGroup   Capacity  Used  Filesystem  Policy  Disk Format  Attached-to   Access      Attach-as               Created By    Created Date
-------  ------------  --------  --------  ----  ----------  ------  -----------  ------------  ----------  ----------------------  ------------  ------------------------
TestVol  sharedVmfs-0  _DEFAULT  100MB     15MB  ext4        N/A     thin         ubuntu-VM0.1  read-write  independent_persistent  ubuntu-VM0.1  Tue Aug  1 19:10:53 2017

[root@sc-rdops-vm01-dhcp-0-74:~]

Now while the container is running, shut-down the esx.
vSphere HA powers-off and restarts the docker host on a different ESX .
After vm has successfully restarted on different ESX, I do see the original volume - TestVol@sharedVmfs-0

root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
DRIVER              VOLUME NAME
vsphere:latest      TestVol@sharedVmfs-0
root@sc-rdops-vm02-dhcp-52-237:~#
root@sc-rdops-vm02-dhcp-52-237:~# docker volume inspect TestVol@sharedVmfs-0
[
    {
        "Driver": "vsphere:latest",
        "Labels": null,
        "Mountpoint": "/mnt/vmdk/TestVol@sharedVmfs-0",
        "Name": "TestVol@sharedVmfs-0",
        "Options": {},
        "Scope": "global",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "attached to VM": "ubuntu-VM0.1",
            "attachedVMDevice": {
                "ControllerPciSlotNumber": "160",
                "Unit": "0"
            },
            "capacity": {
                "allocated": "15MB",
                "size": "100MB"
            },
            "clone-from": "None",
            "created": "Tue Aug  1 19:10:53 2017",
            "created by VM": "ubuntu-VM0.1",
            "datastore": "sharedVmfs-0",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "attached"
        }
    }
]
root@sc-rdops-vm02-dhcp-52-237:~#
root@sc-rdops-vm02-dhcp-52-237:~# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
root@sc-rdops-vm02-dhcp-52-237:~#
root@sc-rdops-vm02-dhcp-52-237:~# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                            PORTS               NA                                                             MES
c4dacf722bf7        ubuntu              "/bin/bash"         11 minutes ago      Exited (255) About a minute ago                       ub                                                             HA
root@sc-rdops-vm02-dhcp-52-237:~#

Admin cli volume ls

[root@sc-rdops-vm01-dhcp-18-68:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls
Volume   Datastore     VMGroup   Capacity  Used  Filesystem  Policy  Disk Format  Attached-to   Access      Attach-as               Created By    Created Date
-------  ------------  --------  --------  ----  ----------  ------  -----------  ------------  ----------  ----------------------  ------------  ------------------------
TestVol  sharedVmfs-0  _DEFAULT  100MB     15MB  ext4        N/A     thin         ubuntu-VM0.1  read-write  independent_persistent  ubuntu-VM0.1  Tue Aug  1 19:10:53 2017

[root@sc-rdops-vm01-dhcp-18-68:~]

Removed the container but volume still shows-up as attached. Hence not able to remove volume and we have a left-over volume.

root@sc-rdops-vm02-dhcp-52-237:~# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
root@sc-rdops-vm02-dhcp-52-237:~#
root@sc-rdops-vm02-dhcp-52-237:~# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                       PORTS               NAMES
c4dacf722bf7        ubuntu              "/bin/bash"         16 minutes ago      Exited (255) 6 minutes ago                       ubHA
root@sc-rdops-vm02-dhcp-52-237:~#
root@sc-rdops-vm02-dhcp-52-237:~# docker rm c4dacf722bf7
c4dacf722bf7
root@sc-rdops-vm02-dhcp-52-237:~#
root@sc-rdops-vm02-dhcp-52-237:~# docker volume inspect TestVol@sharedVmfs-0
[
    {
        "Driver": "vsphere:latest",
        "Labels": null,
        "Mountpoint": "/mnt/vmdk/TestVol@sharedVmfs-0",
        "Name": "TestVol@sharedVmfs-0",
        "Options": {},
        "Scope": "global",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "attached to VM": "ubuntu-VM0.1",
            "attachedVMDevice": {
                "ControllerPciSlotNumber": "160",
                "Unit": "0"
            },
            "capacity": {
                "allocated": "15MB",
                "size": "100MB"
            },
            "clone-from": "None",
            "created": "Tue Aug  1 19:10:53 2017",
            "created by VM": "ubuntu-VM0.1",
            "datastore": "sharedVmfs-0",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "attached"     <<<<<<<<<<<<<<<<<<<<<<<<<
        }
    }
]
root@sc-rdops-vm02-dhcp-52-237:~#
root@sc-rdops-vm02-dhcp-52-237:~# docker volume rm TestVol@sharedVmfs-0
Error response from daemon: unable to remove volume: remove TestVol@sharedVmfs-0: VolumeDriver.Remove: Failed to remove volume TestVol, in use by VM = ubuntu-VM0.1.
root@sc-rdops-vm02-dhcp-52-237:~#

Logs:
docker.txt
docker-volume-vsphere.txt
vmdk_ops.txt

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4