System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7
-
Here is the memory (sar -r) output from the same time period as before:
# sar -r Linux 3.10.0-327.10.1.el7.x86_64 (dny-lnx-sc) 12/15/2016 _x86_64_ (1 CPU) 12:00:02 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 12:10:02 AM 71740 945180 92.95 5740 204672 776528 37.60 312564 561148 1504 12:20:02 AM 144500 872420 85.79 1376 127600 781836 37.85 316908 485200 4568 12:30:03 AM 89184 927736 91.23 4636 201208 775784 37.56 350084 518004 152 12:40:02 AM 82012 934908 91.94 8584 203812 777824 37.66 350192 524276 44 12:50:02 AM 86608 930312 91.48 9340 203928 772360 37.39 345288 524636 80 01:00:02 AM 70352 946568 93.08 11212 191552 797024 38.59 368356 517972 56 01:10:02 AM 208864 808056 79.46 2956 75616 783968 37.96 287288 460324 96 01:20:01 AM 207828 809092 79.56 3572 75928 783984 37.96 287652 460900 40 01:30:02 AM 208676 808244 79.48 4192 76176 784652 37.99 285684 461516 112 01:40:02 AM 197316 819604 80.60 5076 77908 783236 37.92 296752 463596 40 01:50:01 AM 202932 813988 80.04 5728 84604 770016 37.28 287944 467048 68 02:00:02 AM 202192 814728 80.12 6364 84848 769776 37.27 287960 467692 16 02:10:02 AM 218764 798156 78.49 1164 74132 769776 37.27 287572 450392 84 02:20:02 AM 215884 801036 78.77 1824 79160 766748 37.12 286404 454144 16 02:30:01 AM 228212 788708 77.56 2472 79480 755612 36.58 273168 454780 44 02:40:01 AM 232876 784044 77.10 3092 79732 747760 36.20 268544 455388 40 02:50:02 AM 224900 792020 77.88 4360 85724 750860 36.35 269420 461892 96 03:00:02 AM 211420 805500 79.21 5036 86016 763656 36.97 281940 462584 76 03:10:02 AM 190960 825960 81.22 5860 92388 776328 37.59 301156 463384 92 03:20:02 AM 225756 791164 77.80 3812 92836 745776 36.11 294012 436016 72 03:30:01 AM 219584 797336 78.41 4868 93232 755232 36.56 298700 436308 100 03:40:01 AM 192624 824296 81.06 6268 95096 774804 37.51 327724 434328 40 03:50:02 AM 191772 825148 81.14 6840 95276 774980 37.52 328004 434916 40 04:00:02 AM 203592 813328 79.98 7284 95456 762264 36.90 315888 435500 12 04:10:02 AM 173216 843704 82.97 8452 95888 792852 38.39 344348 436560 72 04:20:03 AM 204336 812584 79.91 1648 75312 792840 38.39 299548 449096 28 04:30:02 AM 197472 819448 80.58 2692 83148 788304 38.17 299716 456252 52 04:40:01 AM 206972 809948 79.65 3284 83400 779868 37.76 289200 456852 0 04:50:02 AM 208324 808596 79.51 3944 83764 775296 37.54 287860 457508 100 05:00:01 AM 194676 822244 80.86 4540 84056 788288 38.16 300584 458096 120 05:10:02 AM 172312 844608 83.06 6256 88568 806008 39.02 319976 460064 16 05:20:02 AM 227148 789772 77.66 3488 73876 770952 37.33 283756 441564 104 05:30:01 AM 228532 788388 77.53 4816 74400 765496 37.06 281904 442748 80 05:40:02 AM 205388 811532 79.80 5332 83248 778088 37.67 294216 451912 112 05:50:02 AM 191296 825624 81.19 5848 90192 783284 37.92 303504 455272 92 06:00:02 AM 198684 818236 80.46 6316 91972 770468 37.30 291712 457276 52 06:10:02 AM 226980 789940 77.68 616 71632 775328 37.54 278996 437272 16 06:20:02 AM 214080 802840 78.95 1072 71800 784244 37.97 291384 436748 92 06:30:01 AM 212752 804168 79.08 1560 71856 784636 37.99 291920 437136 92 06:40:02 AM 224592 792328 77.91 2068 71980 771836 37.37 279792 437652 76 06:50:02 AM 221240 795680 78.24 2548 72152 776904 37.61 282056 438200 108 07:00:04 AM 74956 941964 92.63 460 68796 955132 46.24 434580 433956 12 07:10:02 AM 206272 810648 79.72 6340 90724 770120 37.29 286604 450160 36 07:20:02 AM 217708 799212 78.59 6852 90852 757704 36.68 274888 450644 16 07:20:02 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 07:30:01 AM 216592 800328 78.70 7316 90956 758196 36.71 275624 451036 24 07:40:02 AM 215924 800996 78.77 7768 91056 758196 36.71 275760 451492 20 07:50:02 AM 196432 820488 80.68 8268 97304 771348 37.34 288812 457800 84 08:00:02 AM 206632 810288 79.68 8712 98588 759180 36.76 277156 459476 28 08:10:02 AM 138312 878608 86.40 9844 142392 789244 38.21 334080 468152 88 Average: 189987 826933 81.32 4933 97843 777645 37.65 301575 459691 184 08:11:58 AM LINUX RESTART 08:20:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 08:30:01 AM 72464 944456 92.87 3996 58628 437852 21.20 445604 442244 128 08:40:02 AM 70796 946124 93.04 3804 60536 859512 41.61 433572 461436 124 08:50:01 AM 74840 942080 92.64 3224 57956 893008 43.23 438624 451276 128 09:00:01 AM 83976 932944 91.74 6396 60736 873680 42.30 429480 451792 88 09:10:01 AM 185324 831596 81.78 2624 64992 880036 42.61 283204 496412 536 09:20:01 AM 182560 834360 82.05 6248 74140 867036 41.98 280912 501500 84 09:30:02 AM 180472 836448 82.25 9100 75668 863968 41.83 278748 505580 68 09:40:01 AM 173340 843580 82.95 12404 76340 867016 41.98 283360 508012 92 09:50:01 AM 160064 856856 84.26 15596 77248 874812 42.35 292548 511856 144 10:00:01 AM 94168 922752 90.74 20012 128740 905900 43.86 303000 565368 136 10:10:02 AM 216724 800196 78.69 1608 61384 880864 42.65 272124 473656 44 10:20:01 AM 164216 852704 83.85 6268 90640 897888 43.47 313292 484224 120 10:30:01 AM 152996 863924 84.95 9504 92672 905148 43.82 319888 488796 88 Average: 139380 877540 86.29 7753 75360 846671 40.99 336489 487858 137You can see that memory was never exhausted, even during the 100% CPU load time.
-
Here is the paging activity. Yes the page file is being used, but not heavily. There was no excessing paging during the times of performance issues:
# sar -B Linux 3.10.0-327.10.1.el7.x86_64 (dny-lnx-sc) 12/15/2016 _x86_64_ (1 CPU) 12:00:02 AM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff 12:10:02 AM 1070.88 475.19 313.52 18.61 303.18 229.64 2.30 227.53 98.10 12:20:02 AM 370.40 229.51 206.97 0.52 283.77 128.90 0.00 117.97 91.52 12:30:03 AM 739.50 639.48 115.19 0.78 350.09 271.70 0.00 259.53 95.52 12:40:02 AM 3.36 47.00 94.66 0.06 54.44 0.00 0.00 0.00 0.00 12:50:02 AM 0.00 9.23 93.99 0.00 54.01 0.00 0.00 0.00 0.00 01:00:02 AM 0.04 23.73 130.96 0.00 75.68 6.19 0.00 6.16 99.50 01:10:02 AM 138.80 24.04 202.75 0.38 240.10 106.09 0.00 93.01 87.67 01:20:01 AM 0.35 8.15 96.68 0.00 52.07 0.00 0.00 0.00 0.00 01:30:02 AM 0.27 7.73 111.38 0.00 58.43 0.00 0.00 0.00 0.00 01:40:02 AM 8.53 11.16 289.74 0.20 126.59 0.00 0.00 0.00 0.00 01:50:01 AM 0.40 8.22 116.28 0.00 72.54 0.00 0.00 0.00 0.00 02:00:02 AM 0.26 7.31 90.02 0.00 49.75 0.00 0.00 0.00 0.00 02:10:02 AM 63.54 9.81 188.55 0.26 156.20 32.83 0.00 24.31 74.04 02:20:02 AM 9.54 8.48 115.26 0.07 67.49 0.00 0.00 0.00 0.00 02:30:01 AM 0.36 8.91 116.85 0.00 72.74 0.00 0.00 0.00 0.00 02:40:01 AM 0.27 7.98 94.15 0.00 53.94 0.00 0.00 0.00 0.00 02:50:02 AM 11.25 11.48 169.47 0.01 81.96 0.00 0.00 0.00 0.00 03:00:02 AM 0.26 9.15 149.14 0.00 73.15 0.00 0.00 0.00 0.00 03:10:02 AM 11.69 8.54 136.34 0.10 70.33 0.00 0.00 0.00 0.00 03:20:02 AM 52.71 21.43 164.83 0.21 152.25 25.04 0.00 17.44 69.64 03:30:01 AM 0.20 14.92 170.39 0.00 81.22 0.00 0.00 0.00 0.00 03:40:01 AM 4.14 11.31 128.74 0.01 63.29 0.00 0.00 0.00 0.00 03:50:02 AM 0.15 7.17 85.33 0.00 53.95 0.00 0.00 0.00 0.00 04:00:02 AM 0.37 4.52 43.55 0.00 37.19 0.00 0.00 0.00 0.00 04:10:02 AM 0.92 10.29 72.12 0.00 35.06 0.00 0.00 0.00 0.00 04:20:03 AM 62.63 11.58 149.51 0.27 148.09 40.53 0.00 30.23 74.57 04:30:02 AM 15.06 8.68 123.79 0.03 73.41 0.00 0.00 0.00 0.00 04:40:01 AM 0.36 7.96 92.66 0.00 55.38 0.00 0.00 0.00 0.00 04:50:02 AM 0.44 8.41 90.09 0.00 50.81 0.00 0.00 0.00 0.00 05:00:01 AM 0.31 7.99 99.26 0.00 50.50 0.00 0.00 0.00 0.00 05:10:02 AM 9.71 10.33 141.26 0.10 74.02 0.00 0.00 0.00 0.00 05:20:02 AM 48.96 14.03 164.96 0.08 159.73 30.39 0.00 22.40 73.70 05:30:01 AM 2.17 7.82 90.30 0.00 51.34 0.00 0.00 0.00 0.00 05:40:02 AM 15.94 6.61 101.93 0.01 51.49 0.00 0.00 0.00 0.00 05:50:02 AM 0.88 7.12 127.64 0.01 72.83 0.00 0.00 0.00 0.00 06:00:02 AM 3.55 6.36 90.42 0.05 56.44 0.00 0.00 0.00 0.00 06:10:02 AM 95.21 8.28 203.59 0.34 185.54 47.92 0.00 39.88 83.23 06:20:02 AM 0.03 5.98 102.36 0.00 61.96 0.00 0.00 0.00 0.00 06:30:01 AM 0.00 6.31 109.01 0.00 72.76 0.00 0.00 0.00 0.00 06:40:02 AM 0.00 5.99 99.20 0.00 71.66 0.00 0.00 0.00 0.00 06:50:02 AM 0.13 6.12 93.71 0.00 59.18 0.00 0.00 0.00 0.00 07:00:04 AM 469.46 186.17 211.96 0.67 187.31 141.09 0.00 121.64 86.21 07:10:02 AM 189.35 14.65 238.19 0.67 278.61 43.89 0.00 34.50 78.60 07:20:02 AM 0.01 6.17 95.28 0.00 69.84 0.00 0.00 0.00 0.00 07:20:02 AM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff 07:30:01 AM 0.00 5.77 89.42 0.00 58.34 0.00 0.00 0.00 0.00 07:40:02 AM 0.00 5.86 89.96 0.00 58.88 0.00 0.00 0.00 0.00 07:50:02 AM 11.22 6.36 116.90 0.01 72.79 0.00 0.00 0.00 0.00 08:00:02 AM 2.25 5.50 88.66 0.03 67.45 0.00 0.00 0.00 0.00 08:10:02 AM 83.13 6.81 166.69 0.25 78.10 0.00 0.00 0.00 0.00 Average: 71.30 40.58 132.09 0.48 99.66 22.51 0.05 20.27 89.88 08:11:58 AM LINUX RESTART 08:20:01 AM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff 08:30:01 AM 6460.45 1452.55 860.54 10.95 2203.45 2695.77 1.84 1968.47 72.97 08:40:02 AM 3843.51 992.95 1509.57 23.78 2091.99 1544.96 5.08 1063.32 68.60 08:50:01 AM 178.13 106.49 398.48 0.75 419.55 94.83 0.00 63.27 66.72 09:00:01 AM 18.67 52.13 185.53 0.17 230.14 9.80 0.00 5.19 52.98 09:10:01 AM 244.59 246.77 245.09 0.73 323.11 201.56 0.13 109.10 54.09 09:20:01 AM 15.81 44.15 113.35 0.10 87.69 0.00 0.00 0.00 0.00 09:30:02 AM 0.68 37.97 94.69 0.02 77.13 0.00 0.00 0.00 0.00 09:40:01 AM 0.62 43.77 104.97 0.01 79.69 0.00 0.00 0.00 0.00 09:50:01 AM 4.18 40.15 135.47 0.10 101.44 0.00 0.00 0.00 0.00 10:00:01 AM 81.33 44.61 172.65 0.16 200.52 0.00 0.00 0.00 0.00 10:10:02 AM 208.11 56.65 211.51 0.62 314.71 112.92 0.35 101.51 89.62 10:20:01 AM 52.47 44.80 433.22 0.19 345.24 0.00 0.00 0.00 0.00 10:30:01 AM 0.95 44.14 186.85 0.00 262.01 0.00 0.00 0.00 0.00 Average: 823.62 239.63 350.69 2.78 505.34 345.80 0.55 245.66 70.93 -
Here is the disk IOPS during the period:
# sar -b Linux 3.10.0-327.10.1.el7.x86_64 (dny-lnx-sc) 12/15/2016 _x86_64_ (1 CPU) 12:00:02 AM tps rtps wtps bread/s bwrtn/s 12:10:02 AM 37.78 35.51 2.26 2141.75 950.39 12:20:02 AM 60.72 59.03 1.69 740.81 459.02 12:30:03 AM 49.76 44.11 5.65 1478.99 1278.96 12:40:02 AM 4.00 0.10 3.91 6.73 94.01 12:50:02 AM 1.01 0.00 1.01 0.00 18.47 01:00:02 AM 2.50 0.01 2.49 0.07 47.46 01:10:02 AM 6.86 4.90 1.96 277.61 48.09 01:20:01 AM 1.09 0.08 1.01 0.70 16.29 01:30:02 AM 1.00 0.07 0.94 0.54 15.46 01:40:02 AM 1.48 0.34 1.14 17.05 22.31 01:50:01 AM 1.06 0.08 0.98 0.80 16.44 02:00:02 AM 0.95 0.07 0.88 0.52 14.62 02:10:02 AM 2.72 1.71 1.01 127.08 19.63 02:20:02 AM 1.43 0.40 1.03 19.09 16.96 02:30:01 AM 1.13 0.09 1.04 0.71 17.83 02:40:01 AM 1.02 0.07 0.95 0.54 15.96 02:50:02 AM 1.60 0.45 1.15 22.49 22.96 03:00:02 AM 1.14 0.07 1.07 0.52 18.29 03:10:02 AM 1.87 0.87 0.99 23.37 17.08 03:20:02 AM 3.42 1.64 1.78 105.42 42.86 03:30:01 AM 1.48 0.05 1.43 0.41 29.83 03:40:01 AM 1.67 0.44 1.23 8.28 22.61 03:50:02 AM 0.87 0.04 0.83 0.29 14.34 04:00:02 AM 0.59 0.04 0.55 0.74 9.05 04:10:02 AM 1.26 0.23 1.03 1.85 20.58 04:20:03 AM 2.61 1.76 0.84 125.26 23.16 04:30:02 AM 1.56 0.51 1.06 30.12 17.36 04:40:01 AM 1.03 0.09 0.94 0.73 15.92 04:50:02 AM 1.10 0.11 0.99 0.89 16.81 05:00:01 AM 1.03 0.08 0.95 0.63 15.98 05:10:02 AM 2.18 1.03 1.16 19.42 20.66 05:20:02 AM 4.03 2.23 1.80 97.92 28.06 05:30:01 AM 1.33 0.39 0.95 4.33 15.65 05:40:02 AM 1.11 0.29 0.82 31.88 13.22 05:50:02 AM 0.94 0.04 0.90 1.75 14.25 06:00:02 AM 0.92 0.11 0.81 7.09 12.71 06:10:02 AM 3.43 2.45 0.98 190.42 16.55 06:20:02 AM 0.76 0.01 0.75 0.06 11.95 06:30:01 AM 0.78 0.00 0.78 0.00 12.62 06:40:02 AM 0.78 0.00 0.78 0.00 11.97 06:50:02 AM 0.83 0.01 0.82 0.26 12.23 07:00:04 AM 18.36 11.59 6.77 938.93 372.35 07:10:02 AM 8.82 7.30 1.52 378.69 29.30 07:20:02 AM 0.81 0.00 0.81 0.01 12.34 07:20:02 AM tps rtps wtps bread/s bwrtn/s 07:30:01 AM 0.78 0.00 0.78 0.00 11.55 07:40:02 AM 0.78 0.00 0.78 0.00 11.72 07:50:02 AM 0.93 0.12 0.81 22.45 12.71 08:00:02 AM 0.81 0.06 0.75 4.49 11.00 08:10:02 AM 5.67 4.84 0.83 166.27 13.62 Average: 5.09 3.73 1.35 142.61 81.15 08:11:58 AM LINUX RESTART 08:20:01 AM tps rtps wtps bread/s bwrtn/s 08:30:01 AM 162.30 151.14 11.15 12920.88 2905.10 08:40:02 AM 129.66 120.51 9.15 7687.04 1985.90 08:50:01 AM 10.27 6.22 4.05 356.26 212.99 09:00:01 AM 5.04 1.12 3.92 37.33 104.27 09:10:01 AM 9.50 6.04 3.46 489.18 493.55 09:20:01 AM 5.15 1.93 3.22 31.62 88.30 09:30:02 AM 2.77 0.03 2.74 1.36 75.94 09:40:01 AM 3.22 0.06 3.16 1.24 87.54 09:50:01 AM 3.14 0.28 2.87 8.37 80.30 10:00:01 AM 7.41 4.23 3.18 162.67 89.22 10:10:02 AM 7.35 4.22 3.13 416.22 113.30 10:20:01 AM 5.79 2.59 3.20 104.94 89.60 10:30:01 AM 3.10 0.02 3.08 1.90 88.28 10:40:01 AM 15.26 12.06 3.20 637.15 92.53 Average: 25.61 21.40 4.21 1574.71 451.49 -
HTOP when things are healthy:
-
TOP when healthy:
-
GLANCES when healthy:
-
We know that the process that takes the system resources is mono which is screenconnects parent container.
-
Looks like i'm running the same version:-
On CentOS7 ( I think it's all up to date with patches etc)Not seeing this problem.
Running on XenServer 7 - 2vCPU 2GB RAM 40GB HD Space -
@hobbit666 take a look at your SAR output, any chance that you have one threading engine hitting the wall and one that is not?
-
We are resizing the instance up to 2 vCPU and 2GB RAM right now to see what happens with more resources available at ScreenConnect support's request. But it is not using the resources available to it now, we are afraid of masking the issue and getting the system limping along rather than fixing it.
-
@DustinB3403 said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
Any chance you can clone this VM, and downgrade the SC version to the previous version. And monitor both, putting them side by side?
No, can't downgrade that easily as the Client version would be different and not likely to re-sync
-
Resized and the issue remains. Look at that CPU load for an idle box!!
-
Sorry @scottalanmiller i'm a Linux newb
How do I run the SAR stuff?
-
@hobbit666 said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
Sorry @scottalanmiller i'm a Linux newb
How do I run the SAR stuff?
If it is installed, form CLI just run
sar -
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
@hobbit666 said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
Sorry @scottalanmiller i'm a Linux newb
How do I run the SAR stuff?
If it is installed, form CLI just run
sarIf it is not installed
$ yum install sysstat -
Example of
$ sar$ sar Linux 3.10.0-327.10.1.el7.x86_64 (dny-lnx-sc) 12/15/2016 _x86_64_ (2 CPU) 10:56:35 AM LINUX RESTART 11:00:02 AM CPU %user %nice %system %iowait %steal %idle 11:10:01 AM all 22.30 0.00 0.85 0.16 3.05 73.64 Average: all 22.30 0.00 0.85 0.16 3.05 73.64Or as above from @scottalanmiller with
$ sar -b -
This is all i'm getting from SAR
[root@screenconnect6 ~]# sar
Linux 3.10.0-327.28.3.el7.x86_64 (screenconnect6) 15/12/16 x86_64 (2 CPU)16:20:43 LINUX RESTART
-
@scottalanmiller said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
Resized and the issue remains. Look at that CPU load for an idle box!!
How do you get that pretty picture??
Funny I was about to start a thread about tuning Linux systems for resources so I can free some up for more VM's
-
@hobbit666 said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
This is all i'm getting from SAR
[root@screenconnect6 ~]# sar
Linux 3.10.0-327.28.3.el7.x86_64 (screenconnect6) 15/12/16 x86_64 (2 CPU)16:20:43 LINUX RESTART
Takes ten minutes to accumulate the first line of data.
-
@hobbit666 said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
@scottalanmiller said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
Resized and the issue remains. Look at that CPU load for an idle box!!
How do you get that pretty picture??
Command is htop, it's in the EPEL. Then I just screenshot it. It's the default terminal on Ubuntu 16.10.
