CirrusHi SanYang ,
try to run this Command :
tmsh show sys tmm-trafficTo see how is the traffic distributed accross TMMs instances
This Article will help you : https://my.f5.com/manage/s/article/K36483224
- if you found uneven balancing throught tTMM instances , maybe you will need something Like ( F5 DAG ) to adjust this traffic ditribution , For DAG please have a look in this : https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-service-provider-generic-message-administration-13-0-0/5.html
you can configure it on vlans ( GUI or CLI ).
- If you define the correct DAG , you shouldn't see that one of TMM instances CPU is 100 % , Correct DAG value protects your from Context-swiching ( which means one of your TMM is 100% so it couldn't handle the data traffic for clients so this leads to redirect connrctions to another TMM or drop it , also if TMM is fully utilized it may request more resources to handle this high pressure on it whereas the other TMMs not handle this quantity of data traffic )
This is from my perspective , but I want to know the updates till reach the optimal value of TMM CPU maybe I am not fully correct, so I will keep the attention for this thread here 🙂
CirrusHi Mohamed_Ahmed_Kansoh ,
It seems to be evenly distributed.
# tmsh show sys tmm-traffic
-------------------------------------------------
Sys::TMM: 0.0
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 49.6T 38.5T
Bits Out 17.9T 17.9T
Packets In 86.2G 68.8G
Packets Out 2.0G 1.7G
Current Connections 311 279
Maximum Connections 20.2K 20.2K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 66.9G 66.9G
Total Requests 58.6M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 490.0K
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 537
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.2
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 50.4T 41.0T
Bits Out 21.4T 21.5T
Packets In 96.6G 78.1G
Packets Out 2.4G 2.0G
Current Connections 1.9K 1.7K
Maximum Connections 63.8K 63.3K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 76.5G 76.5G
Total Requests 70.4M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 34
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 504
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.4
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 53.8T 44.0T
Bits Out 20.5T 20.6T
Packets In 100.6G 81.7G
Packets Out 2.3G 1.9G
Current Connections 58.7K 58.2K
Maximum Connections 62.8K 62.1K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 80.5G 80.5G
Total Requests 67.3M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 33
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 585
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.6
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 80.9T 46.5T
Bits Out 20.1T 20.2T
Packets In 149.2G 86.6G
Packets Out 2.2G 1.9G
Current Connections 1.9K 1.7K
Maximum Connections 20.4K 20.3K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 85.5G 85.5G
Total Requests 65.7M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 4
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 519
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.8
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 46.4T 37.1T
Bits Out 20.1T 20.2T
Packets In 89.3G 72.0G
Packets Out 2.2G 1.9G
Current Connections 1.8K 1.5K
Maximum Connections 20.5K 20.4K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 70.6G 70.6G
Total Requests 65.8M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 10
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 547
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.10
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 271.8T 257.5T
Bits Out 20.0T 20.1T
Packets In 132.0G 115.3G
Packets Out 2.2G 1.9G
Current Connections 1.9K 1.6K
Maximum Connections 20.4K 20.4K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 70.4G 70.4G
Total Requests 65.5M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 19
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 582
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.12
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 55.1T 42.5T
Bits Out 20.1T 20.3T
Packets In 98.7G 76.9G
Packets Out 2.2G 1.9G
Current Connections 1.7K 1.4K
Maximum Connections 20.4K 20.4K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 75.8G 75.8G
Total Requests 65.7M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 42
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 538
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.14
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 52.9T 40.6T
Bits Out 20.0T 20.1T
Packets In 92.3G 74.9G
Packets Out 2.2G 1.9G
Current Connections 1.7K 1.5K
Maximum Connections 20.3K 20.3K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 73.1G 73.1G
Total Requests 65.5M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 22
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 557
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.16
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 64.9T 47.9T
Bits Out 20.0T 20.1T
Packets In 110.0G 83.4G
Packets Out 2.2G 1.9G
Current Connections 1.8K 1.5K
Maximum Connections 20.3K 20.3K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 81.8G 81.8G
Total Requests 65.6M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 19
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 532
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.18
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 47.1T 37.7T
Bits Out 20.0T 20.1T
Packets In 81.1G 68.3G
Packets Out 2.2G 1.9G
Current Connections 1.8K 1.6K
Maximum Connections 20.3K 20.3K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 66.9G 66.9G
Total Requests 65.5M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 27
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 536
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.20
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 50.0T 38.0T
Bits Out 19.8T 19.9T
Packets In 89.6G 70.3G
Packets Out 2.2G 1.8G
Current Connections 1.7K 1.5K
Maximum Connections 20.0K 19.9K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 68.8G 68.8G
Total Requests 64.7M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 10
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 490
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.22
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 47.1T 37.5T
Bits Out 19.8T 19.9T
Packets In 89.2G 71.1G
Packets Out 2.2G 1.8G
Current Connections 1.7K 1.5K
Maximum Connections 20.3K 20.2K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 70.0G 70.0G
Total Requests 64.9M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 30
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 527
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.24
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 49.8T 39.3T
Bits Out 19.9T 20.0T
Packets In 90.7G 73.4G
Packets Out 2.2G 1.9G
Current Connections 1.8K 1.6K
Maximum Connections 20.0K 20.0K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 72.3G 72.3G
Total Requests 64.9M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 39
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 546
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0Hi SanYang,
As I don't see errors in passing traffic and somehow balanced traffic for all TMM instances ( maybe only TMM0.10 handles high connections so it's CPU is +60% ) but over all TMMs CPU is within 20%.
So this is an indication of that TMM0 represents the Total CPU & memory usage in your blade and because you have till 24 tmm instances it gave you 100 % as this is a high number of TMMs.
This is Clarified in this Article : https://my.f5.com/manage/s/article/K02895424
Now , you should keen on in the overall CPU & memory usage as average between all TMMs cores.
If you go to ( Statistics >>> Performance reports >>> Chech the CPU & memory graphs )
Also adjust time range ( Last 3 hours then Last week , then last month )
I'm sure you will find the overall Cpu is consistent following the same trend for all other graphs , I mean you will find the overall/average CPU for the ( last month/week/day/hours ) not high or reached to 100%
Do u know if you found this overall CPU reached to 100 % your box will failover immediately.
So I need you to check that performance report and let me know if my analysis is correct or not.
Thanks
CirrusHi Mohamed_Ahmed_Kansoh ,
Thanks for your reply.
I checked the charts.
CPU0 has been running at 100% for a while now...
Hello SanYang,
yes this does make sense with the TMM0 CPU value in CLI.
But as I see in the historical graphs last monday and tuesday it was normal.
I see now you need to open a case with F5 support , to validate the health of TMM0.
maybe this is the reason : https://my.f5.com/manage/s/article/K02895424
but you need to verify this , because if one of your TMMs crashes ( data plane ones I mean ) this will effect badly in your overall performance.
So I recommend opening a case.
then when you get the RCA for this high utilization you can share it with us.