Forum Discussion

David_M's avatar
David_M
Icon for Cirrostratus rankCirrostratus
Sep 04, 2019

Investigating a bigip reboot from LTM logs

Hello,

So I see the boot marker, but nothing indicates what lead to the reboot.

The boot_market indicates the reboot, right? 

2019-08-26T03:50:03+04:00 ASMHOST notice boot_marker : ---===[ HD1.3 - BIG-IP 13.1.0.5 Build 0.47.5 ]===---
Aug 26 03:50:12 ASMHOST info mprov:2210:: Invoked as: /usr/bin/mprov.pl (pid=2210) --logicaldisk --boot --quiet
Aug 26 03:50:12 ASMHOST info mprov:2210:: 'Checking for and completing any logical disk transactions:'
Aug 26 03:50:13 ASMHOST info mprov:2244:: Invoked as: /usr/bin/mprov.pl (pid=2244) --diskmgmt --boot --quiet
Aug 26 03:50:13 ASMHOST info mprov:2244:: 'Checking for and completing any disk management transactions:'
Aug 26 03:50:14 ASMHOST info mprov:2277:: Invoked as: /usr/bin/mprov.pl (pid=2277) --boot --quiet
Aug 26 03:50:14 ASMHOST info mprov:2277:: '/bin/mkdir -p /dev/mprov/tam'
Aug 26 03:50:14 ASMHOST info mprov:2277:: '/bin/mkdir -p /dev/mprov/tmm'
Aug 26 03:50:14 ASMHOST info mprov:2277:: '/bin/mkdir -p /dev/mprov/afm'
Aug 26 03:50:14 ASMHOST info mprov:2277:: '/bin/mkdir -p /dev/mprov/vcmp'
Aug 26 03:50:14 ASMHOST info mprov:2277:: '/bin/mkdir -p /dev/mprov/apm'
Aug 26 03:50:14 ASMHOST info mprov:2277:: '/bin/mkdir -p /dev/mprov/avr'
Aug 26 03:50:14 ASMHOST info mprov:2277:: '/bin/mkdir -p /dev/mprov/am'
Aug 26 03:50:14 ASMHOST info mprov:2277:: '/bin/mkdir -p /dev/mprov/asm'
Aug 26 03:50:14 ASMHOST info mprov:2277:: '/bin/mkdir -p /dev/mprov/ltm'
Aug 26 03:50:14 ASMHOST info mprov:2277:: '/bin/mkdir -p /dev/mprov/dos'
Aug 26 03:50:14 ASMHOST info mprov:2277:: '/bin/mkdir -p /dev/mprov/pem'
Aug 26 03:50:14 ASMHOST info mprov:2277:: '/bin/mkdir -p /dev/mprov/fps'
Aug 26 03:50:14 ASMHOST warning mprov:2277:: 'Requested memory type hugetlbfs-1G is unsupported'
Aug 26 03:50:16 ASMHOST info mprov:2277:: '/usr/bin/numactl --interleave=all /bin/echo 4140 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages'
Aug 26 03:50:16 ASMHOST info mprov:2277:: '/bin/mount -t hugetlbfs -o pagesize=2M none /dev/mprov/apm 2>&1 1>/dev/null'
Aug 26 03:50:16 ASMHOST info mprov:2277:: '/bin/mount -t hugetlbfs -o pagesize=2M none /dev/mprov/asm 2>&1 1>/dev/null'
Aug 26 03:50:16 ASMHOST info mprov:2277:: '/bin/mount -t hugetlbfs -o pagesize=2M none /dev/mprov/avr 2>&1 1>/dev/null'
Aug 26 03:50:16 ASMHOST info mprov:2277:: '/bin/mount -t hugetlbfs -o pagesize=2M none /dev/mprov/tmm 2>&1 1>/dev/null'
Aug 26 03:50:16 ASMHOST info mprov:2277:: '/usr/bin/setdb Provision.Action none -skipsyntax'
Aug 26 03:50:16 ASMHOST info mprov:2277:: 'Provisioning successful.'
Aug 26 03:50:17 ASMHOST info mprov:2418:: Invoked as: /usr/bin/mprov.pl (pid=2418) --legacy --quiet
Aug 26 03:50:17 ASMHOST info mprov:2418:: 'Provisioning (legacy update) successful.'

then at 3:51 I get this

Aug 26 03:51:49 ASMHOST warning sod[6198]: 01140029:4: HA proc_running tmm fails action is go offline and down links.
Aug 26 03:51:49 ASMHOST warning sod[6198]: 01140029:4: HA proc_running bd fails action is go offline and down links.
Aug 26 03:51:49 ASMHOST warning sod[6198]: 01140029:4: HA proc_running datasyncd fails action is go offline and down links.
Aug 26 03:51:49 ASMHOST notice sod[6198]: 010c007f:5: Receiving status updates from peer device ASM01.fqdn (172.16.9.81) (Online).
Aug 26 03:51:49 ASMHOST info devmgmtd[6194]: 015a0000:6: updateCaBundle complete
Aug 26 03:51:49 ASMHOST info devmgmtd[6194]: 015a0000:6: updateCaBundle complete
Aug 26 03:51:49 ASMHOST info mprov:9163:: 'Provisioning (legacy update) successful.'
Aug 26 03:51:49 ASMHOST notice chmand[7233]: 012a0005:5: Platform marketing name: BIG-IP 4200
Aug 26 03:51:49 ASMHOST info scriptd[6603]: 0114002b:6: HA daemon_heartbeat scriptd enabled.
Aug 26 03:51:49 ASMHOST notice mcpd[8067]: 01070404:5: Add a new Publication for publisherID scriptd-publisher and filterType (nil)
Aug 26 03:51:49 ASMHOST warning chmand[7233]: 012a0004:4: mgmt interface enable/disable not available for this platform
Aug 26 03:51:49 ASMHOST notice chmand[7233]: 012a0005:5: mgmtIpDel: deleting IP (172.16.9.82/24), interface (mgmt)
Aug 26 03:51:49 ASMHOST notice chmand[7233]: 012a0005:5: mgmtIpAdd: adding IP (172.16.9.82/24), interface (mgmt)
Aug 26 03:51:49 ASMHOST notice chmand[7233]: 012a0005:5: FPGA type requested = 0
Aug 26 03:51:49 ASMHOST notice chmand[7233]: 012a0005:5: FPGA vers requested = Latest
Aug 26 03:51:50 ASMHOST notice iprepd[6526]: 015c0009:5: IP Reputation has no license currently
Aug 26 03:51:50 ASMHOST notice errdefsd[7498]: 0194001d:5: Errdefsd is starting

Before 3:50 all I see is the normal irule logging with new sessions coming in etc.

application delivery
BIG-IP
LTM

1 Reply

Sort By
  • JG's avatar
    JG
    Icon for Cumulonimbus rankCumulonimbus
    Sep 05, 2019

    There should be more info about the reboot in the logs. The reboot conditions, on v11.6.4, are as follows:

    # tmsh show /sys ha-status all-properties 
-------------------------------------------------------------------------------------------------------------------------------------------------------
Sys::HA Status
Slot  Feature               Key                                        Action                        Fail  Feature  Take  Client  Proc          Timeout
                                                                                                           Enabled  Act   Data                  (sec)
-------------------------------------------------------------------------------------------------------------------------------------------------------
...
1     nic-failsafe          tmm                                        reboot                        no    yes      no    0       tmm           0
1     nic-failsafe          tmm1                                       reboot                        no    yes      no    0       tmm1          0
1     nic-failsafe          tmm2                                       reboot                        no    yes      no    0       tmm2          0
1     nic-failsafe          tmm3                                       reboot                        no    yes      no    0       tmm3          0
1     nic-failsafe          tmm4                                       reboot                        no    yes      no    0       tmm4          0
1     nic-failsafe          tmm5                                       reboot                        no    yes      no    0       tmm5          0
1     nic-failsafe          tmm6                                       reboot                        no    yes      no    0       tmm6          0
1     nic-failsafe          tmm7                                       reboot                        no    yes      no    0       tmm7          0
1     nic-failsafe          tmm8                                       reboot                        no    yes      no    0       tmm8          0
1     nic-failsafe          tmm9                                       reboot                        no    yes      no    0       tmm9          0
1     reboot-request        sod                                        reboot                        no    yes      no    0       sod           0
1     software-update       lind                                       reboot                        no    yes      no    0       lind          0

    . Not sure about your situation.