Hi,
We've encountered issue twice in 2 weeks with one of our hosted server.
"Bootbank cannot be found at path '/bootbank'" was displayed and every VM were grayed out.
Once it happens, nothing more is logged.
The only solution is to restart the ESX.
Below, few log lines before the issue happens:
vmkernel.log
2018-04-10T23:04:49.964Z cpu9:36122)WARNING: CBT: 2080: Unsupported ioctl 61
2018-04-10T23:04:49.964Z cpu9:36122)WARNING: CBT: 2080: Unsupported ioctl 60
2018-04-10T23:04:49.964Z cpu9:36122)WARNING: CBT: 2080: Unsupported ioctl 44
2018-04-10T23:04:49.964Z cpu9:36122)VSCSI: 4047: handle 8823(vscsi0:1):Creating Virtual Device for world 36114 (FSS handle 26283523) numBlocks=346030080 (bs=512)
2018-04-10T23:04:49.964Z cpu9:36122)VSCSI: 273: handle 8823(vscsi0:1):Input values: res=0 limit=-2 bw=-1 Shares=1000
2018-04-10T23:04:49.964Z cpu9:36122)Vmxnet3: 15414: Disable Rx queuing; queue size 256 is larger than Vmxnet3RxQueueLimit limit of 128.
2018-04-10T23:04:49.964Z cpu9:36122)Vmxnet3: 15664: Using default queue delivery for vmxnet3 for port 0x200000b
2018-04-10T23:04:49.964Z cpu9:36122)NetPort: 1575: enabled port 0x200000b with mac 00:50:56:0a:c6:77
2018-04-10T23:04:50.044Z cpu0:34630)FSS: 6264: Failed to open file 'stark.vmdk'; Requested flags 0x40001, world: 34630 [hostd-worker], (Existing flags 0x8, world: 36123 [vmx-vcpu-1:stark]): Busy
2018-04-10T23:30:01.300Z cpu5:36114)ScsiDeviceIO: 2636: Cmd(0x43b5cdf81140) 0x1a, CmdSN 0x15b95 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-10T23:45:08.131Z cpu8:35869)WARNING: ScsiDeviceIO: 1243: Device naa.600605b00a093f001d12d2537f796f0f performance has deteriorated. I/O latency increased from average value of 3532 microseconds to 106428 microseconds.
2018-04-10T23:45:08.322Z cpu8:35869)WARNING: ScsiDeviceIO: 1243: Device naa.600605b00a093f001d12d2537f796f0f performance has deteriorated. I/O latency increased from average value of 3532 microseconds to 215223 microseconds.
2018-04-10T23:45:22.669Z cpu11:35870)ScsiDeviceIO: 1217: Device naa.600605b00a093f001d12d2537f796f0f performance has improved. I/O latency reduced from 215223 microseconds to 41912 microseconds.
2018-04-10T23:46:40.296Z cpu3:36071)ScsiDeviceIO: 1217: Device naa.600605b00a093f001d12d2537f796f0f performance has improved. I/O latency reduced from 41912 microseconds to 8113 microseconds.
2018-04-10T23:47:18.503Z cpu7:35361)ScsiDeviceIO: 1217: Device naa.600605b00a093f001d12d2537f796f0f performance has improved. I/O latency reduced from 8113 microseconds to 6887 microseconds.
2018-04-11T00:09:37.572Z cpu11:34569)ScsiDeviceIO: 2636: Cmd(0x43b5c0c53fc0) 0x1a, CmdSN 0x15bde from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T01:01:01.590Z cpu10:36073)ScsiDeviceIO: 2636: Cmd(0x43b5cf322d40) 0x1a, CmdSN 0x15c33 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T01:05:56.656Z cpu4:35939)VSCSI: 2590: handle 8236(vscsi0:0):Reset request on FSS handle 2624663 (0 outstanding commands) from (vmm0:tosca_20151005)
2018-04-11T01:05:56.656Z cpu9:32945)VSCSI: 2868: handle 8236(vscsi0:0):Reset [Retries: 0/0] from (vmm0:tosca_20151005)
2018-04-11T01:05:56.656Z cpu2:32945)VSCSI: 2661: handle 8236(vscsi0:0):Completing reset (0 outstanding commands)
2018-04-11T01:06:07.488Z cpu8:35937)VSCSI: 2590: handle 8236(vscsi0:0):Reset request on FSS handle 2624663 (0 outstanding commands) from (vmm0:tosca_20151005)
2018-04-11T01:06:07.488Z cpu2:32945)VSCSI: 2868: handle 8236(vscsi0:0):Reset [Retries: 0/0] from (vmm0:tosca_20151005)
2018-04-11T01:06:07.488Z cpu2:32945)VSCSI: 2661: handle 8236(vscsi0:0):Completing reset (0 outstanding commands)
2018-04-11T01:20:36.446Z cpu3:35871)VSCSI: 2590: handle 8224(vscsi0:0):Reset request on FSS handle 3148725 (0 outstanding commands) from (vmm0:CPV4_20151005)
2018-04-11T01:20:36.447Z cpu2:32945)VSCSI: 2868: handle 8224(vscsi0:0):Reset [Retries: 0/0] from (vmm0:CPV4_20151005)
2018-04-11T01:20:36.447Z cpu2:32945)VSCSI: 2661: handle 8224(vscsi0:0):Completing reset (0 outstanding commands)
2018-04-11T01:20:47.572Z cpu6:35867)VSCSI: 2590: handle 8224(vscsi0:0):Reset request on FSS handle 3148725 (0 outstanding commands) from (vmm0:CPV4_20151005)
2018-04-11T01:20:47.572Z cpu2:32945)VSCSI: 2868: handle 8224(vscsi0:0):Reset [Retries: 0/0] from (vmm0:CPV4_20151005)
2018-04-11T01:20:47.572Z cpu2:32945)VSCSI: 2661: handle 8224(vscsi0:0):Completing reset (0 outstanding commands)
2018-04-11T01:39:37.596Z cpu6:34569)ScsiDeviceIO: 2636: Cmd(0x43b5c0ba0800) 0x1a, CmdSN 0x15c7c from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T01:40:07.862Z cpu5:35867)WARNING: ScsiDeviceIO: 1243: Device naa.600605b00a093f001d12d2537f796f0f performance has deteriorated. I/O latency increased from average value of 3528 microseconds to 107444 microseconds.
2018-04-11T01:40:14.208Z cpu11:35716)ScsiDeviceIO: 1217: Device naa.600605b00a093f001d12d2537f796f0f performance has improved. I/O latency reduced from 107444 microseconds to 21401 microseconds.
2018-04-11T01:40:53.656Z cpu3:35554)ScsiDeviceIO: 1217: Device naa.600605b00a093f001d12d2537f796f0f performance has improved. I/O latency reduced from 21401 microseconds to 6909 microseconds.
2018-04-11T02:30:21.006Z cpu5:35735)ScsiDeviceIO: 2636: Cmd(0x43b5ce082c00) 0x1a, CmdSN 0x15cd1 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T03:09:37.604Z cpu11:35468)ScsiDeviceIO: 2636: Cmd(0x43b5c0bd3940) 0x1a, CmdSN 0x15d1a from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T04:01:01.557Z cpu8:35941)ScsiDeviceIO: 2636: Cmd(0x43b5c0c01480) 0x1a, CmdSN 0x15d6f from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T04:40:26.413Z cpu7:32789)ScsiDeviceIO: 2636: Cmd(0x43b5cdfa66c0) 0x1a, CmdSN 0x15db3 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T05:39:37.637Z cpu10:35512)ScsiDeviceIO: 2636: Cmd(0x43b5c0b35c40) 0x1a, CmdSN 0x15e0f from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T06:10:39.541Z cpu6:35973)ScsiDeviceIO: 2636: Cmd(0x43b5c0be3200) 0x1a, CmdSN 0x15e59 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T07:09:37.679Z cpu8:36072)ScsiDeviceIO: 2636: Cmd(0x43b5c0ba1b80) 0x1a, CmdSN 0x15eb5 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T07:50:42.305Z cpu4:34636)ScsiDeviceIO: 2636: Cmd(0x43b5c0b8d000) 0x1a, CmdSN 0x15ef9 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T08:39:37.695Z cpu8:34569)ScsiDeviceIO: 2636: Cmd(0x43b5c0c6dd80) 0x4d, CmdSN 0xe41 from world 34569 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2018-04-11T09:20:44.247Z cpu8:35870)ScsiDeviceIO: 2636: Cmd(0x43b5c1bb58c0) 0x1a, CmdSN 0x15f97 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T10:09:37.721Z cpu7:35869)ScsiDeviceIO: 2636: Cmd(0x43b5c0c1af00) 0x4d, CmdSN 0xe4d from world 34569 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2018-04-11T11:00:46.202Z cpu5:35869)ScsiDeviceIO: 2636: Cmd(0x43b5cf32b800) 0x1a, CmdSN 0x16037 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T11:39:37.745Z cpu11:36073)ScsiDeviceIO: 2636: Cmd(0x43b5cdf82ac0) 0x1a, CmdSN 0x16080 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T12:30:48.551Z cpu9:34217)ScsiDeviceIO: 2636: Cmd(0x43b5cdfa3240) 0x1a, CmdSN 0x160d5 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T13:09:37.769Z cpu6:32790)ScsiDeviceIO: 2636: Cmd(0x43b5c1bb34c0) 0x1a, CmdSN 0x1611e from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T14:01:01.554Z cpu8:35890)ScsiDeviceIO: 2636: Cmd(0x43b5cdc2aec0) 0x1a, CmdSN 0x16173 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T14:40:51.762Z cpu7:35890)ScsiDeviceIO: 2636: Cmd(0x43b5ceb34780) 0x1a, CmdSN 0x161b7 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T15:39:37.833Z cpu9:35416)ScsiDeviceIO: 2636: Cmd(0x43b5ceb32080) 0x1a, CmdSN 0x16213 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2018-04-11T16:10:53.686Z cpu5:35581)ScsiDeviceIO: 2636: Cmd(0x43b5cdc2ad40) 0x1a, CmdSN 0x16255 from world 0 to dev "naa.600605b00a093f001d12d2537f796f0f" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
We've made hardware test (Hard drive especially) but everything seems to be OK.
Thanks.