XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!

CitrixNewbJD

[root@xen2 log]# tail -n 100 xensource.log
Dec 27 14:31:16 xen2 xapi: [debug|xen2|1450823 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:82887bba0b96|api_effect] PBD.remove_from_other_config
Dec 27 14:31:16 xen2 xapi: [debug|xen2|1450825 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:09e2aa2687dd|api_effect] PBD.remove_from_other_config
Dec 27 14:31:16 xen2 xapi: [debug|xen2|1450827 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:13a7f9b859e0|api_effect] PBD.add_to_other_config
Dec 27 14:31:16 xen2 xapi: [debug|xen2|1450829 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:3270fba588b8|api_effect] PBD.add_to_other_config
Dec 27 14:31:16 xen2 xapi: [ info|xen2|1450838 UNIX /var/xapi/xapi|session.logout D:2ad4d14e2a85|xapi] Session.destroy trackid=402d2a79fc43297a8dc25f7fa36844de
Dec 27 14:31:16 xen2 xapi: [ info|xen2|1450839 UNIX /var/xapi/xapi|session.login_with_password D:82f44b61ad08|xapi] Session.create trackid=9b5f5e4e28b67bce328bb7a29f8c6e61 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:16 xen2 xapi: [debug|xen2|1450839 UNIX /var/xapi/xapi|session.login_with_password D:82f44b61ad08|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:16 xen2 xapi: [debug|xen2|1450840 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:df5edc881477 created by task D:82f44b61ad08
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450858 INET 0.0.0.0:80|dispatch:PBD.remove_from_other_config D:9d9efd457fb7|api_effect] PBD.remove_from_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450860 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:04c9188b6703|api_effect] PBD.remove_from_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450861 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:49cf176e5e46|api_effect] PBD.remove_from_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450863 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:aeb76c4c5540|api_effect] PBD.add_to_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450865 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:400507c7a850|api_effect] PBD.add_to_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450862 INET 0.0.0.0:80|dispatch:PBD.remove_from_other_config D:7f07845eed3f|api_effect] PBD.remove_from_other_config
Dec 27 14:31:17 xen2 xapi: [ info|xen2|1450875 UNIX /var/xapi/xapi|session.logout D:32207e9ca5af|xapi] Session.destroy trackid=9b5f5e4e28b67bce328bb7a29f8c6e61
Dec 27 14:31:17 xen2 xapi: [ info|xen2|1450876 UNIX /var/xapi/xapi|session.login_with_password D:266336f0c5ca|xapi] Session.create trackid=cbe2ea2161a008b31c123bba3d3088f6 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450876 UNIX /var/xapi/xapi|session.login_with_password D:266336f0c5ca|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450877 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:a754d9534632 created by task D:266336f0c5ca
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450885 INET 0.0.0.0:80|dispatch:PBD.add_to_other_config D:5165cfdc6eb0|api_effect] PBD.add_to_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450897 INET 0.0.0.0:80|dispatch:PBD.add_to_other_config D:f540965ad467|api_effect] PBD.add_to_other_config
Dec 27 14:31:17 xen2 xapi: [ info|xen2|1450910 INET 0.0.0.0:80|session.logout D:41f9af099127|xapi] Session.destroy trackid=fe675f6a258246bd1b69c3f7a5c3e99d
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450911 INET 0.0.0.0:80||dummytaskhelper] task dispatch:session.get_uuid D:68bea1da5826 created by task D:e5fb02a3454d
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] Marking SR for PBD.plug (task=OpaqueRef:1a2d6e65-ffde-4e66-1904-557a00063c92)
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|backtrace] Raised at xapi_sr_operations.ml:137.33-72 -> message_forwarding.ml:2950.5-67 -> message_forwarding.ml:283.5-8
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] Caught exception while OTHER_OPERATION_IN_PROGRESS: [ SR; OpaqueRef:37ddc678-2fc1-67d3-a5c8-c53499df2864; plug ] in message forwarder: marking SR for PBD.plug
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|backtrace] Raised at message_forwarding.ml:286.8-9 -> threadext.ml:20.20-24 -> threadext.ml:20.62-65 -> message_forwarding.ml:149.19-25
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] PBD.plug locking failed: caught transient failure OTHER_OPERATION_IN_PROGRESS: [ SR; OpaqueRef:37ddc678-2fc1-67d3-a5c8-c53499df2864; plug ]
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] Waiting for up to 1.695234 seconds before retrying...
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450924 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:d3520cfe0fe4|api_effect] PBD.remove_from_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450926 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:45e6f1af395a|api_effect] PBD.remove_from_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450928 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:1657b7fd0b3b|api_effect] PBD.add_to_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450930 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:7a61e7bc2274|api_effect] PBD.add_to_other_config
Dec 27 14:31:17 xen2 xapi: [ info|xen2|1450939 UNIX /var/xapi/xapi|session.logout D:f26e50a00a23|xapi] Session.destroy trackid=cbe2ea2161a008b31c123bba3d3088f6
Dec 27 14:31:17 xen2 xapi: [ info|xen2|1450940 UNIX /var/xapi/xapi|session.login_with_password D:b7f1535237f8|xapi] Session.create trackid=90b7aa6ac3991339a1bd4f0642b76680 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450940 UNIX /var/xapi/xapi|session.login_with_password D:b7f1535237f8|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450941 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:17f47ccae7b3 created by task D:b7f1535237f8
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450959 INET 0.0.0.0:80|dispatch:PBD.remove_from_other_config D:e7ca4a8dec5b|api_effect] PBD.remove_from_other_config
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450961 INET 0.0.0.0:80|dispatch:PBD.remove_from_other_config D:4a97c8a92518|api_effect] PBD.remove_from_other_config
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450963 INET 0.0.0.0:80|dispatch:PBD.add_to_other_config D:5f0b0e1f91ad|api_effect] PBD.add_to_other_config
Dec 27 14:31:18 xen2 xapi: [ info|xen2|1450970 UNIX /var/xapi/xapi|session.logout D:33f81c0e1f75|xapi] Session.destroy trackid=90b7aa6ac3991339a1bd4f0642b76680
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450971 INET 0.0.0.0:80|dispatch:PBD.add_to_other_config D:3dc55dedcbbb|api_effect] PBD.add_to_other_config
Dec 27 14:31:18 xen2 xapi: [ info|xen2|1450984 INET 0.0.0.0:80|session.logout D:3f6e5af1e625|xapi] Session.destroy trackid=db210ab1f145ee422323350b21bd74dd
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450985 INET 0.0.0.0:80||dummytaskhelper] task dispatch:session.get_uuid D:e79b78e7a6a5 created by task D:e91fc341b52a
Dec 27 14:31:18 xen2 xapi: [ info|xen2|1450998 UNIX /var/xapi/xapi|session.login_with_password D:35cee6258606|xapi] Session.create trackid=d10e206dd14c6d49f4fe098a4f0766fb pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450998 UNIX /var/xapi/xapi|session.login_with_password D:35cee6258606|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450999 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:6ba2db342589 created by task D:35cee6258606
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1451017 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:34d31751b3ca|api_effect] PBD.remove_from_other_config
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1451019 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:02dd5072cc69|api_effect] PBD.remove_from_other_config
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1451021 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:a8db6c31efc3|api_effect] PBD.add_to_other_config
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1451023 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:def7f44f171f|api_effect] PBD.add_to_other_config
Dec 27 14:31:19 xen2 xapi: [ info|xen2|1451032 UNIX /var/xapi/xapi|session.logout D:142a6672fcff|xapi] Session.destroy trackid=d10e206dd14c6d49f4fe098a4f0766fb
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] Marking SR for PBD.plug (task=OpaqueRef:1a2d6e65-ffde-4e66-1904-557a00063c92)
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|backtrace] Raised at xapi_sr_operations.ml:137.33-72 -> message_forwarding.ml:2950.5-67 -> message_forwarding.ml:283.5-8
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] Caught exception while OTHER_OPERATION_IN_PROGRESS: [ SR; OpaqueRef:37ddc678-2fc1-67d3-a5c8-c53499df2864; plug ] in message forwarder: marking SR for PBD.plug
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|backtrace] Raised at message_forwarding.ml:286.8-9 -> threadext.ml:20.20-24 -> threadext.ml:20.62-65 -> message_forwarding.ml:149.19-25
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] PBD.plug locking failed: caught transient failure OTHER_OPERATION_IN_PROGRESS: [ SR; OpaqueRef:37ddc678-2fc1-67d3-a5c8-c53499df2864; plug ]
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] Waiting for up to 12.666722 seconds before retrying...
Dec 27 14:31:19 xen2 xcp-rrdd: [debug|xen2|0 monitor|main|rrdd_stats] system stats: MemTotal: 762248 KiB; MemFree: 142620 KiB; Buffered: 30580 KiB; Cached: 223716 KiB; SwapTotal: 524280 KiB; SwapFree: 524280 KiB
Dec 27 14:31:19 xen2 xcp-rrdd: [debug|xen2|0 monitor|main|rrdd_stats] Clock drift: -0
Dec 27 14:31:19 xen2 xcp-rrdd: [debug|xen2|0 monitor|main|rrdd_stats] xcp-rrdd stats (n = 1): size: 68164 KiB; rss: 5388 KiB; data: 60788 KiB; stack: 88 KiB
Dec 27 14:31:19 xen2 xcp-rrdd: [debug|xen2|0 monitor|main|rrdd_stats] xapi stats (n = 2): size: 361788 KiB; rss: 73772 KiB; data: 330832 KiB; stack: 220 KiB
Dec 27 14:31:19 xen2 xapi: [ info|xen2|1451033 UNIX /var/xapi/xapi|session.login_with_password D:4619a8559d1e|xapi] Session.create trackid=f662a35431bf0ddd237a3447e7ef4f77 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1451033 UNIX /var/xapi/xapi|session.login_with_password D:4619a8559d1e|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1451034 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:c9f8df7140fc created by task D:4619a8559d1e
Dec 27 14:31:19 xen2 xcp-rrdd: [debug|xen2|0 monitor|main|rrdd_stats] xenopsd stats (n = 1): size: 114976 KiB; rss: 4348 KiB; data: 109736 KiB; stack: 88 KiB
Dec 27 14:31:19 xen2 xapi: [debug|xen2|6 dbflush [/var/xapi/state.db]||sql] XML backend [/var/xapi/state.db] -- Write buffer flushed. Time: 0.022087
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1289694 UNIX /var/xapi/xapi|PBD.plug R:48b3b6331cca|xapi] Marking SR for PBD.plug (task=OpaqueRef:48b3b633-1cca-1112-4280-62e0c95bd0e0)
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1289694 UNIX /var/xapi/xapi|PBD.plug R:48b3b6331cca|backtrace] Raised at xapi_sr_operations.ml:137.33-72 -> message_forwarding.ml:2950.5-67 -> message_forwarding.ml:283.5-8
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1289694 UNIX /var/xapi/xapi|PBD.plug R:48b3b6331cca|xapi] Caught exception while OTHER_OPERATION_IN_PROGRESS: [ SR; OpaqueRef:37ddc678-2fc1-67d3-a5c8-c53499df2864; plug ] in message forwarder: marking SR for PBD.plug
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1289694 UNIX /var/xapi/xapi|PBD.plug R:48b3b6331cca|backtrace] Raised at message_forwarding.ml:286.8-9 -> threadext.ml:20.20-24 -> threadext.ml:20.62-65 -> message_forwarding.ml:149.19-25
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1289694 UNIX /var/xapi/xapi|PBD.plug R:48b3b6331cca|xapi] PBD.plug locking failed: caught transient failure OTHER_OPERATION_IN_PROGRESS: [ SR; OpaqueRef:37ddc678-2fc1-67d3-a5c8-c53499df2864; plug ]
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1289694 UNIX /var/xapi/xapi|PBD.plug R:48b3b6331cca|xapi] Waiting for up to 14.806154 seconds before retrying...
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451058 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:12c10588e48e|api_effect] PBD.remove_from_other_config
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451059 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:f7ca1fefa964|api_effect] PBD.remove_from_other_config
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451060 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:c60e0a7885a6|api_effect] PBD.add_to_other_config
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451061 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:7d41478633a2|api_effect] PBD.add_to_other_config
Dec 27 14:31:20 xen2 xapi: [ info|xen2|1451071 UNIX /var/xapi/xapi|session.logout D:d99c9d2cbad6|xapi] Session.destroy trackid=f662a35431bf0ddd237a3447e7ef4f77
Dec 27 14:31:20 xen2 xapi: [ info|xen2|1451072 UNIX /var/xapi/xapi|session.login_with_password D:99d40118134c|xapi] Session.create trackid=41be80a325ec2708158b61cd0f319b9d pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451072 UNIX /var/xapi/xapi|session.login_with_password D:99d40118134c|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451073 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:09c20f39baca created by task D:99d40118134c
Dec 27 14:31:20 xen2 xapi: [ info|xen2|1451093 INET 0.0.0.0:80|session.logout D:cc28177ddc28|xapi] Session.destroy trackid=b468f6dc63292261522b234fe5ee5ad2
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451094 INET 0.0.0.0:80||dummytaskhelper] task dispatch:session.get_uuid D:2d0986801026 created by task D:0761c6bbf357
Dec 27 14:31:21 xen2 xapi: [ info|xen2|1451112 UNIX /var/xapi/xapi|session.logout D:fcd319dfb72c|xapi] Session.destroy trackid=41be80a325ec2708158b61cd0f319b9d
Dec 27 14:31:21 xen2 xapi: [ info|xen2|1451113 UNIX /var/xapi/xapi|session.login_with_password D:218ca8a8a66a|xapi] Session.create trackid=061c385d53b9ad60f9b61a224d40b830 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451113 UNIX /var/xapi/xapi|session.login_with_password D:218ca8a8a66a|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451114 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:ea5152d18048 created by task D:218ca8a8a66a
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451132 INET 0.0.0.0:80|dispatch:PBD.remove_from_other_config D:e9897d53a20e|api_effect] PBD.remove_from_other_config
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451134 INET 0.0.0.0:80|dispatch:PBD.remove_from_other_config D:7f7b526dec2e|api_effect] PBD.remove_from_other_config
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451136 INET 0.0.0.0:80|dispatch:PBD.add_to_other_config D:1bfda339ff57|api_effect] PBD.add_to_other_config
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451138 INET 0.0.0.0:80|dispatch:PBD.add_to_other_config D:b5333aacf0f0|api_effect] PBD.add_to_other_config
Dec 27 14:31:21 xen2 xapi: [ info|xen2|1451151 INET 0.0.0.0:80|session.logout D:9e312895529a|xapi] Session.destroy trackid=5c0c47b10d423763811399642ffdfafc
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451152 INET 0.0.0.0:80||dummytaskhelper] task dispatch:session.get_uuid D:d5ccd057b9d5 created by task D:7abf77ad1741
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451157 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:5bd0ee2d790b|api_effect] PBD.remove_from_other_config
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451158 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:6d3b73bc7395|api_effect] PBD.remove_from_other_config
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451161 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:eb4ac5803e5a|api_effect] PBD.add_to_other_config
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451162 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:a07c440a7f09|api_effect] PBD.add_to_other_config
Dec 27 14:31:21 xen2 xapi: [ info|xen2|1451169 UNIX /var/xapi/xapi|session.logout D:0c2a055e01cf|xapi] Session.destroy trackid=061c385d53b9ad60f9b61a224d40b830
Dec 27 14:31:21 xen2 xapi: [ info|xen2|1451173 UNIX /var/xapi/xapi|session.login_with_password D:e4b0fd1684b5|xapi] Session.create trackid=9b647868c555905ed290099de411b7f4 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451173 UNIX /var/xapi/xapi|session.login_with_password D:e4b0fd1684b5|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451175 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:7e21e3a33f60 created by task D:e4b0fd1684b5

CitrixNewbJD

@scottalanmiller

[root@xen2 log]# tail -n 100 xen
tail: error reading `xen': Is a directory

scottalanmiller

@CitrixNewbJD hardware seems to be fine.

scottalanmiller

How about this command:

xe pbd-list

CitrixNewbJD

@scottalanmiller

[root@xen2 log]# xe pbd-list
uuid ( RO)                  : 360fee57-907f-2c6a-e15d-166412f7bd7e
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): e58b6f8a-225d-2a83-642d-0610812efbea
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; SCSIid: 360024e800070ed06000004314a967309; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : c4ee5bb1-f824-52c4-dc09-c2c50034801e
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): e8c1da6c-9016-fa1d-b357-e1bc070b4d2b
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; SCSIid: 360024e800070ed06000007f54dba7bfb; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : 5dedd15d-45c6-c38d-7ed1-0eb8bd7a4620
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): d457d592-56c2-73e0-379a-26a55be27538
         device-config (MRO): iso_path: /XenISO; location: //192.168.1.230/Zipfiles; type: cifs; username: spindleinc\mgould; cifspassword_secret: c74bebaa-bbb0-e2d8-9a19-e11433375cd4
    currently-attached ( RO): false


uuid ( RO)                  : 5bd81e5b-895e-d385-bd28-da4a4b2603f8
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): 169601bb-0d24-006a-a02f-e6ccb65a560f
         device-config (MRO): device: /dev/sdb4
    currently-attached ( RO): false


uuid ( RO)                  : ebf92343-b5b1-d530-ebd8-235365c3d961
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): ff6303ba-0852-6f9c-b59f-ddb2cbc4d07b
         device-config (MRO): multihomelist: 192.168.130.108:3260,192.168.130.105:3260,192.168.130.107:3260,192.168.130.106:3260; SCSIid: 360024e800054baef0000ddd550c85082; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800054baef0000000050c83cdb; port: 3260; target: 192.168.130.108
    currently-attached ( RO): false


uuid ( RO)                  : 0f241fac-bf50-5141-69aa-7c31ac0fb406
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): ec1432db-9f89-2a04-987e-263f0d30c66f
         device-config (MRO): iso_path: /Software/Microsoft/Windows/Server/2012; location: //192.168.1.100/files; type: cifs; username: spindleinc\mgould; cifspassword_secret: 77701771-8d4e-3de7-a562-90542842828a
    currently-attached ( RO): false


uuid ( RO)                  : 6fd14ddd-9595-88fe-592f-1d289d06ebbc
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): 7e001a4b-4f2d-15fe-4e7a-653062b4cdcd
         device-config (MRO): location: /dev/xapi/cd
    currently-attached ( RO): false


uuid ( RO)                  : e97a8dbc-eec8-dcde-63ef-fe0f350ed73d
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): 4e97fda1-40cb-8a4a-baba-184a9e81fee5
         device-config (MRO): location: /opt/xensource/packages/iso; legacy_mode: true
    currently-attached ( RO): false


uuid ( RO)                  : 7ab79e27-cbe1-04b0-4c14-f8e89c200411
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): 4e97fda1-40cb-8a4a-baba-184a9e81fee5
         device-config (MRO): location: /opt/xensource/packages/iso; legacy_mode: true
    currently-attached ( RO): false


uuid ( RO)                  : cacb5cd4-3894-8fbc-e3ce-6b342c11df95
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): e8c1da6c-9016-fa1d-b357-e1bc070b4d2b
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; SCSIid: 360024e800070ed06000007f54dba7bfb; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : 09b657e0-974e-7a9e-99f8-425e8da32c7b
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): dbdb8c1c-30b6-5e24-991d-7f42c452fd07
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; SCSIid: 360024e80007b786a000004134a9670f3; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : 38ac4cf3-15b4-b319-2fc7-44eff01b93f6
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): 49447156-88bc-e5cf-7578-c0a5775d6dc7
         device-config (MRO): device: /dev/disk/by-id/scsi-360024e8072e05b0011fe4fc707bd94eb-part4
    currently-attached ( RO): false


uuid ( RO)                  : 8ffccb24-5c1e-27a9-879d-01e89b419fca
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): 85d27d5d-19cd-df91-62e2-f64cfb7153cf
         device-config (MRO): location: /dev/xapi/block
    currently-attached ( RO): false


uuid ( RO)                  : 6d49b0e1-5d6a-9779-1f23-5c4414121518
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): 5cdb03b4-746d-4ee1-8a45-3b3a33895e69
         device-config (MRO): type: cifs; location: //192.168.1.230/zipfiles; cifspassword_secret: 25876439-7bf0-75be-f977-17ce22ec3536; username: jason; iso_path: /microsoft
    currently-attached ( RO): false


uuid ( RO)                  : 24695844-f828-d80e-a002-08107734c527
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): e58b6f8a-225d-2a83-642d-0610812efbea
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; SCSIid: 360024e800070ed06000004314a967309; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : 80f89268-eb01-5460-5ed9-e65b8fe62875
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): ff6303ba-0852-6f9c-b59f-ddb2cbc4d07b
         device-config (MRO): multihomelist: 192.168.130.108:3260,192.168.130.107:3260,192.168.130.105:3260,192.168.130.106:3260; SCSIid: 360024e800054baef0000ddd550c85082; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800054baef0000000050c83cdb; port: 3260; target: 192.168.130.108
    currently-attached ( RO): false


uuid ( RO)                  : 965ead53-f00f-158a-db6d-84865dcdb38d
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): d457d592-56c2-73e0-379a-26a55be27538
         device-config (MRO): iso_path: /XenISO; location: //192.168.1.230/Zipfiles; type: cifs; username: spindleinc\mgould; cifspassword_secret: a44f6bfd-dd64-768d-56e6-cb1c002c5d8c
    currently-attached ( RO): false


uuid ( RO)                  : dbef4921-c3f1-2cb8-6548-d462745bb1ea
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): 5cdb03b4-746d-4ee1-8a45-3b3a33895e69
         device-config (MRO): type: cifs; location: //192.168.1.230/zipfiles; cifspassword_secret: d7287cdb-b3a8-8583-aba4-a2c24d0394d5; username: jason; iso_path: /microsoft
    currently-attached ( RO): false


uuid ( RO)                  : 9b968975-b85e-5369-86f3-d924c9d10871
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): dbdb8c1c-30b6-5e24-991d-7f42c452fd07
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; SCSIid: 360024e80007b786a000004134a9670f3; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : c15826f2-0f0b-6fcf-87ab-36b90754368f
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): ba8349a9-87d5-19b0-36b6-c12f4e5b40a9
         device-config (MRO): location: /dev/xapi/cd
    currently-attached ( RO): false


uuid ( RO)                  : 65de2fdc-49dd-c0a4-b397-c44e42e6b998
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): b03b52b1-e403-1649-9595-8f7c1b442896
         device-config (MRO): location: /dev/xapi/block
    currently-attached ( RO): false


uuid ( RO)                  : b602d816-38fc-d5f8-6880-943a3ba824d3
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): ec1432db-9f89-2a04-987e-263f0d30c66f
         device-config (MRO): iso_path: /Software/Microsoft/Windows/Server/2012; location: //192.168.1.100/files; type: cifs; username: spindleinc\mgould; cifspassword_secret: 8ca82529-cbac-4818-db7e-ccdcbb85c193
    currently-attached ( RO): false

momurda

Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?

CitrixNewbJD

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?

mgould is still with us. I don't know if it was set up with AD integration, but I would assume so based on this. It's not a disabled account, but we can't get to AD.

I have no idea why we have 5 ISO libraries.

momurda

Oh, you cant authenticate to ad but all your SR need it to attach
That is the problem right there i think.

scottalanmiller

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?

mgould is still with us. I don't know if it was set up with AD integration, but I would assume so based on this. It's not a disabled account, but we can't get to AD.

That's not good. AD was built on top of a system that was merged with all shared dependencies... so AD needs Xen which needs the PowerVault, which needs Xen which needs AD. So the circular issue potentially? There is no AD to fail over to if anything goes wrong with the Xen cluster has no redundancy. No matter how many copies run on XenServer here, there is only one disk array holding it all.

scottalanmiller

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Oh, you cant authenticate to ad but all your SR need it to attach
That is the problem right there i think.

That would do it. Excellent catch.

CitrixNewbJD

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?

mgould is still with us. I don't know if it was set up with AD integration, but I would assume so based on this. It's not a disabled account, but we can't get to AD.

That's not good. AD was built on top of a system that was merged with all shared dependencies... so AD needs Xen which needs the PowerVault, which needs Xen which needs AD. So the circular issue potentially? There is no AD to fail over to if anything goes wrong with the Xen cluster has no redundancy. No matter how many copies run on XenServer here, there is only one disk array holding it all.

Does this mean it's irreparable?

scottalanmiller

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?

mgould is still with us. I don't know if it was set up with AD integration, but I would assume so based on this. It's not a disabled account, but we can't get to AD.

That's not good. AD was built on top of a system that was merged with all shared dependencies... so AD needs Xen which needs the PowerVault, which needs Xen which needs AD. So the circular issue potentially? There is no AD to fail over to if anything goes wrong with the Xen cluster has no redundancy. No matter how many copies run on XenServer here, there is only one disk array holding it all.

Does this mean it's irreparable?

No, just really REALLY silly.

scottalanmiller

@momurda from a quick glance those appear to be all CIFS ISO repos that are AD tied, so we should be able to just remove those to fix that, right? I don't see anything else.

momurda

@CitrixNewbJD
No i dont think so.
You got some work to do though
Probably need to do:
log into the san and change the creds for iscsi, disable ad authentication in xc, and then probably forget/reattach the sr

NerdyDad

I always try to make sure that I have an administrative local user to every system, just in case I cannot get to AD.

scottalanmiller

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD
No i dont think so.
You got some work to do though
Probably need to do:
log into the san and change the creds for iscsi, disable ad authentication in xc, and then probably forget/reattach the sr

I didn't see it on the iSCSI, darn it. Still fixable, but more work.

scottalanmiller

@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

I always try to make sure that I have an administrative local user to every system, just in case I cannot get to AD.

That, and never have AD on top of something that depends on it. That's like locking your keys in your car.

NerdyDad

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

I always try to make sure that I have an administrative local user to every system, just in case I cannot get to AD.

That, and never have AD on top of something that depends on it. That's like locking your keys in your car.

Yup. Our SAN is isolated from the rest of the house network physically. All authentication between the SAN and the host should never even touch AD just in case of things like this.

Sometimes playing what-if is a good exercise to keep bad things from happening as far as security, reliability, and recovery.

momurda

@scottalanmiller
Yes the cifs repos can be forgotten for now. I doubt there is anything important there in the iso repo.
@CitrixNewbJD
Before you do anything else, make sure you have the root pw for your xs servers as disabling AD integration will force you to use them.

momurda

Unless you have a DC that you can get working, ie one not attached to your shared storage. Perhaps you have a physical dc somewhere? Unlikely i know.