WebUI always crashing and slow

What is your memory usage if you look at statistics?

Can you see the memory usage climb over time? Do you use tmsh as default shell for users? That could be a real memory hog.

/Patrik

Hello Patrik,

I compared the usage of this unit with other boxes and I see that it does have unusually higher usage. System Memory Other is at 50%.

Other is usage of processes other than the TMM. Can you paste the output of

top

o

Q

Here's what I got:

top - 15:42:06 up 251 days, 15:38, 4 users, load average: 1.41, 1.28, 1.25 Tasks: 581 total, 5 running, 576 sleeping, 0 stopped, 0 zombie Cpu(s): 8.0%us, 11.5%sy, 0.0%ni, 80.4%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 49555080k total, 49191028k used, 364052k free, 610976k buffers Swap: 1048568k total, 20280k used, 1028288k free, 1724880k cached

PID USER PR NI RES VIRT SHR S %CPU %MEM TIME+ COMMAND 18062 root 1 -19 133m 42.9g 112m S 107.3 0.3 86700:10 tmm 22998 root RT 0 0 0 0 S 80.1 0.0 41847:24 enforcer/13 23015 root RT 0 0 0 0 R 80.1 0.0 41843:06 enforcer/17 23044 root RT 0 0 0 0 R 80.1 0.0 41843:41 enforcer/21 18061 root 1 -19 135m 42.9g 114m S 21.5 0.3 51534:42 tmm 9083 root 20 0 45m 56624 19m S 8.3 0.1 28890:21 bcm56xxd 7942 root 20 0 18m 26876 10m S 5.6 0.0 17383:59 csyncd 9082 root 20 0 12m 170m 6716 S 5.3 0.0 11422:53 chmand 9311 root 20 0 109m 172m 19m S 3.0 0.2 10720:16 mcpd 7365 root 20 0 1280 2844 832 S 2.6 0.0 0:00.08 rsync 12259 root 20 0 880 1952 696 S 2.3 0.0 5579:06 LCDd 6673 root 20 0 1436 2708 856 R 0.7 0.0 0:00.07 top 7795 root 20 0 47m 65644 3316 S 0.7 0.1 8101:25 statsd 8249 root 20 0 38m 40452 30m S 0.7 0.1 2489:19 sod 17362 tomcat 20 0 232m 357m 5388 S 0.7 0.5 1531:28 java 53 root 20 0 0 0 0 S 0.3 0.0 806:33.25 ksoftirqd/12 106 root 20 0 0 0 0 S 0.3 0.0 26:46.29 events/7 3507 apache 20 0 18m 69452 3404 S 0.3 0.0 0:00.81 httpd 9080 root 20 0 8424 28040 3856 S 0.3 0.0 335:41.76 alertd 9544 root 25 5 28m 92852 20m S 0.3 0.1 3406:22 merged 1 root 20 0 584 2180 528 S 0.0 0.0 9:46.92 init 2 root 20 0 0 0 0 S 0.0 0.0 0:03.45 kthreadd 3 root RT 0 0 0 0 S 0.0 0.0 0:00.04 migration/0 4 root 20 0 0 0 0 S 0.0 0.0 0:02.60 ksoftirqd/0 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 6 root RT 0 0 0 0 S 0.0 0.0 0:14.00 watchdog/0 7 root RT 0 0 0 0 S 0.0 0.0 0:31.18 migration/1 8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1 9 root 20 0 0 0 0 S 0.0 0.0 6:41.09 ksoftirqd/1 10 root RT 0 0 0 0 S 0.0 0.0 0:10.88 watchdog/1 11 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/2 12 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/2 13 root 20 0 0 0 0 S 0.0 0.0 3:58.88 ksoftirqd/2 14 root RT 0 0 0 0 S 0.0 0.0 0:08.64 watchdog/2 15 root RT 0 0 0 0 S 0.0 0.0 7:51.92 migration/3 16 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/3 17 root 20 0 0 0 0 S 0.0 0.0 4:25.52 ksoftirqd/3

Send the output of this command too?

ps aux | awk '{if($6 > 10000){$6=$6/1000 " MB";print $6 " " $11}}'

Please use the pre-formatted option in the forum (mark the text and hit CTRL + k). It's easier to read.

/Patrik

This is what I got.

0 MB COMMAND
20.292 MB /usr/sbin/httpd
20.14 MB /usr/sbin/httpd
52.8 MB -tmsh
11.148 MB /usr/sbin/restorecond
44.812 MB /usr/bin/overdog
19.128 MB /usr/bin/promptstatusd
49.272 MB /usr/bin/statsd
20.572 MB /usr/bin/lacpd
19.732 MB /usr/sbin/logstatd
28.988 MB /usr/bin/merged
39.556 MB /usr/bin/sod
12.32 MB /usr/bin/chmand
111.692 MB /usr/bin/mcpd
47.204 MB /usr/bin/icrd
33.176 MB /usr/bin/fpdd
17.072 MB /usr/bin/lind
20.496 MB /usr/bin/devmgmtd
18.296 MB /usr/bin/lldpd
27.74 MB /usr/bin/clusterd
24.36 MB /usr/share/cbr/bin/cbrd
19.64 MB /usr/bin/istatsd
20.456 MB /usr/sbin/httpd
52.164 MB /usr/bin/scriptd
19.032 MB /usr/bin/eventd
19.084 MB /usr/bin/csyncd
19.548 MB /usr/bin/evrouted
221.028 MB /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.19.x86_64/jre/bin/java
22.024 MB /usr/sbin/sflow_agent
18.204 MB /usr/share/iprepd/bin/iprepd
17.64 MB /usr/bin/syscalld
20.312 MB /usr/sbin/httpd
20.08 MB /usr/sbin/httpd
20.184 MB /usr/sbin/httpd
138.784 MB tmm.0
136.648 MB tmm.6
20.392 MB /usr/sbin/httpd
20.312 MB /usr/sbin/httpd
20.096 MB /usr/bin/tmipsecd
19.6 MB /usr/bin/vxland
20.332 MB /usr/sbin/httpd
25.18 MB splunkd
21.16 MB /usr/sbin/httpd
17.332 MB /usr/sbin/fcgi-
18.96 MB /usr/local/www/mcpq/mcpq
69.7 MB /usr/local/www/iControl/iControlPortal.cgi
203.504 MB /usr/lib/jvm/jre/bin/java
46.744 MB /usr/bin/bcm56xxd
20.944 MB /usr/bin/bigd
20.312 MB /usr/sbin/httpd
18.304 MB /usr/bin/audit_forwarder
27.784 MB /usr/sbin/snmpd

What is splunkd doing there? :)

/Patrik

Please return the output of

ls -al /opt

which splunkd

find / -name *splunk*

Never seen splunk installed on an F5 before. Maybe someone else has?

/Patrik

I have no idea to be honest, but other units have it too but not experiencing the same problem.

Also, for fun, can you run

pfile 

This would show which files it has open.

/Patrik