System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7

gjacobse

After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.

Dashrender

@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.

what data will you lose?

gjacobse

@Dashrender said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.

what data will you lose?

We should not lose any data. Just some down time while the process is completeed.

gjacobse

Process done:

sudo service screenconnect stop

mv /opt/screenconnect/Session.db /opt/screenconnect/Session.db.bak

sudo service screenconnect start

gjacobse

]$ ls -l
total 29724
-rw-r--r--  1 root root      454 Aug 25 19:15 ExtensionConfiguration.xml
drwxr-xr-x  2 root root     4096 Nov  8 09:44 Helper
-rw-r--r--  1 root root      654 Aug 25 19:15 License.xml
-rw-r--r--  1 root root    23699 Dec 16 11:26 Role.xml
-rw-r--r--  1 root root  1232896 Dec 19 20:14 Session.db
-rw-r--r--  1 root root 24956928 Dec 19 20:11 Session.db.bak
-rw-r--r--  1 root root    32768 Dec 19 20:18 Session.db-shm
-rw-r--r--  1 root root  4128272 Dec 19 20:18 Session.db-wal
-rw-r--r--  1 root root     4030 Oct 25 08:55 SessionEventTrigger.xml
-rw-r--r--  1 root root     5178 Dec  2 08:13 SessionGroup.xml
drwxr-xr-x 14 root root     4096 Dec 12 12:55 Toolbox
-rw-r--r--  1 root root        0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e6cea952.xml
-rw-r--r--  1 root root    21908 Dec 19 20:13 User.xml

gjacobse

@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
-rw-r--r-- 1 root root 244,855,752 Dec 14 16:25 Session.db-wal
-rw-r--r--  1 root root 78,403,632 Dec 16 08:33 Session.db-wal
rw-r--r--  1 root root  4,128,272 Dec 17 02:17 Session.db-wal
Going back and doing some comparisons and noticed this -

(commas added for 'dramatic effect'
-rw-r--r--  1 root root 25,082,592 Dec 18 10:39 Session.db-wal

Session.db-wal is not the right file to be looking at... Should be looking at session.db

gjacobse

@Dashrender said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.

what data will you lose?

The only thing we seem to have lost is Support Sessions, as they are temporary anyway.

scottalanmiller

@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

@Dashrender said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.

what data will you lose?

The only thing we seem to have lost is Support Sessions, as they are temporary anyway.

Taht would be expected.

scottalanmiller

So how are we looking after a few hours?

gjacobse

@scottalanmiller said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

So how are we looking after a few hours?

We are at 320 of the 540 devices previously logg d.

I expect this as some devices are off,...

scottalanmiller

Any CPU spikes?

gjacobse

DB modification was done at 8:00pm(ish) last night, SAR going back to Midnight

]$ sar
Linux 3.10.0-327.10.1.el7.x86_64 (server)   12/20/2016      _x86_64_        (1 CPU)

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM     all      0.42      0.00      0.07      0.02      0.02     99.48
12:20:01 AM     all      1.46      0.00      0.16      0.07      0.02     98.29
12:30:01 AM     all      0.34      0.00      0.05      0.01      0.01     99.59
12:40:01 AM     all      1.02      0.00      0.09      0.08      0.03     98.78
12:50:01 AM     all      0.34      0.00      0.05      0.02      0.01     99.58
01:00:01 AM     all      1.01      0.00      0.10      0.07      0.01     98.81
01:10:01 AM     all      0.64      0.00      0.11      0.02      0.04     99.19
01:20:01 AM     all      1.12      0.00      0.09      1.03      0.06     97.71
01:30:01 AM     all      0.39      0.00      0.05      0.02      0.03     99.51
01:40:01 AM     all      1.06      0.00      0.09      0.07      0.02     98.75
01:50:01 AM     all      0.35      0.00      0.05      0.01      0.01     99.57
02:00:01 AM     all      0.96      0.00      0.09      0.08      0.02     98.85
02:10:01 AM     all      0.57      0.00      0.11      0.02      0.02     99.29
02:20:01 AM     all      1.08      0.00      0.10      0.18      0.01     98.62
02:30:01 AM     all      0.34      0.00      0.05      0.02      0.02     99.58
02:40:01 AM     all      1.02      0.00      0.09      0.07      0.02     98.80
02:50:01 AM     all      0.32      0.00      0.05      0.11      0.01     99.51
03:00:01 AM     all      1.00      0.00      0.10      0.06      0.02     98.82
03:10:01 AM     all      0.39      0.00      0.06      0.02      0.02     99.52
03:20:01 AM     all      1.28      0.02      0.15      0.14      0.03     98.38
03:30:01 AM     all      0.34      0.00      0.05      0.02      0.02     99.56
03:40:01 AM     all      1.05      0.00      0.10      0.08      0.03     98.75
03:50:01 AM     all      0.33      0.00      0.06      0.02      0.02     99.58
04:00:01 AM     all      1.04      0.00      0.10      0.07      0.03     98.76
04:10:01 AM     all      1.07      0.00      0.14      0.04      0.07     98.67
04:20:02 AM     all      1.17      0.00      0.13      0.08      0.06     98.57
04:30:01 AM     all      0.44      0.31      0.12      0.04      0.04     99.06
04:40:01 AM     all      1.07      0.00      0.11      0.08      0.04     98.70
04:50:01 AM     all      0.47      0.00      0.07      0.02      0.02     99.42
05:00:01 AM     all      1.04      0.00      0.11      0.17      0.05     98.64
05:10:01 AM     all      0.76      0.00      0.12      0.04      0.05     99.02
05:20:01 AM     all      1.02      0.00      0.09      0.07      0.06     98.76
05:30:01 AM     all      0.55      0.00      0.07      0.03      0.04     99.31

05:30:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
05:40:01 AM     all      1.10      0.00      0.12      0.07      0.03     98.68
05:50:01 AM     all      0.55      0.00      0.07      0.03      0.04     99.31
06:00:01 AM     all      1.02      0.00      0.09      0.07      0.04     98.78
06:10:01 AM     all      1.16      0.00      0.18      0.05      0.07     98.55
06:20:01 AM     all      1.43      0.00      0.16      0.09      0.11     98.22
06:30:01 AM     all      0.48      0.00      0.06      0.02      0.05     99.39
06:40:01 AM     all      1.06      0.00      0.09      0.28      0.07     98.51
06:50:01 AM     all      1.13      0.00      0.09      0.03      0.08     98.67
07:00:01 AM     all      8.95      0.00      0.17      0.34      0.17     90.37
07:10:01 AM     all      3.81      0.00      0.15      0.04      0.18     95.82
07:20:01 AM     all      9.25      0.00      0.15      0.17      0.22     90.21
07:30:01 AM     all      3.97      0.00      0.10      0.04      0.16     95.72
07:40:01 AM     all     10.44      0.00      0.17      0.10      0.29     89.00
07:50:01 AM     all      4.62      0.00      0.10      0.05      0.18     95.05
08:00:01 AM     all     10.71      0.00      0.15      0.09      0.14     88.91
08:10:01 AM     all      5.20      0.00      0.12      0.07      0.22     94.39
Average:        all      1.84      0.01      0.10      0.09      0.06     97.90

]$ ls -l
total 30856
-rw-r--r--  1 root root      454 Aug 25 19:15 ExtensionConfiguration.xml
drwxr-xr-x  2 root root     4096 Nov  8 09:44 Helper
-rw-r--r--  1 root root      654 Aug 25 19:15 License.xml
-rw-r--r--  1 root root    23699 Dec 16 11:26 Role.xml
-rw-r--r--  1 root root  2383872 Dec 20 08:09 Session.db
-rw-r--r--  1 root root 24956928 Dec 19 20:11 Session.db.bak
-rw-r--r--  1 root root    32768 Dec 20 08:16 Session.db-shm
-rw-r--r--  1 root root  4136512 Dec 20 08:16 Session.db-wal
-rw-r--r--  1 root root     4030 Oct 25 08:55 SessionEventTrigger.xml
-rw-r--r--  1 root root     5178 Dec  2 08:13 SessionGroup.xml
drwxr-xr-x 14 root root     4096 Dec 12 12:55 Toolbox
-rw-r--r--  1 root root        0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e6cea952.xml
-rw-r--r--  1 root root    21908 Dec 19 23:07 User.xml

Currently 373 devices of the orginial 540

gjacobse

@gjacobse

I expect that there are so 'few' devices is that they get powered off at the end of the day, and have not been powered on as of yet to call home.

gjacobse

Now showing 405 devices connected.

scottalanmiller

@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

Now showing 405 devices connected.

How about the CPU?

gjacobse

@scottalanmiller said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

Now showing 405 devices connected.

How about the CPU?

0_1482249010171_2016-12-20 10_49_41-gene@dny-lnx-sc_~.png

gjacobse

09:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
09:10:01 AM     all      8.39      0.00      0.28      0.07      0.79     90.48
09:20:01 AM     all     14.24      0.00      0.24      0.14      0.79     84.58
09:30:01 AM     all      7.18      0.00      0.14      0.08      0.44     92.15
09:40:01 AM     all     12.75      0.00      0.23      0.13      0.53     86.35
09:50:01 AM     all     15.59      0.00      0.24      0.10      0.36     83.71
10:00:01 AM     all     27.39      0.00      0.28      0.10      0.54     71.69
10:10:01 AM     all     18.61      0.00      0.24      0.21      0.61     80.33
10:20:01 AM     all     24.03      0.00      0.30      0.15      0.71     74.80
10:30:01 AM     all      8.70      0.00      0.14      0.09      0.47     90.60
10:40:01 AM     all     20.68      0.00      0.27      0.15      0.95     77.95
10:50:01 AM     all     10.40      0.00      0.27      0.08      0.80     88.45
Average:        all      4.82      0.01      0.13      0.09      0.17     94.78

scottalanmiller

That's higher than I would have expected, but SAR is what we need, not htop.

scottalanmiller

There we go, and still too high.

gjacobse

@scottalanmiller said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:

There we go, and still too high.

In some regard yes I would agree - However as part of the cycling, I have updated about 15 clients that were on the previous build to the latest. So that could be part of the increased CPU as it was pushing out the update.