Forum Discussion

Vishal_96707's avatar
Vishal_96707
Icon for Nimbostratus rankNimbostratus
Feb 27, 2009

Urgent Help - Sharepoint search crawl failing

We have sharepoint 2003 and sharepoint 2007 environments setup on F5. We are migrating from sharepoint 2003 to SP 2007. We had configured iRule which was capturing "404" and "File Not Foud" msgs from 2003 environments and redirecting to 2007 environments. We are not having issues with the sharepoint search crawl. The sharepoint search crawl fails with the iRules however everything works fine without iRules. Can someone help me? We are in the process of getting new environment in production soon.

The below logic is combined in one iRule. 

when HTTP_REQUEST { 
 set MYHOST [HTTP::host] 
 set MYURL [HTTP::uri] 
 } 
 when HTTP_RESPONSE { 
 if { [HTTP::status] ends_with "404" } 
 {HTTP::redirect "https://intranet.bmc.com/_layouts/redirects/Default.aspx?URL=$MYHOST$MYURL"}}

when HTTP_REQUEST { 
    set MYHOST [HTTP::host] 
    set MYURL [HTTP::uri] 
 } 
  
 when HTTP_RESPONSE { 
 if { [HTTP::payload] contains "File Not Found" } 
  {HTTP::redirect "https://intranet.bmc.com/_layouts/redirects/Default.aspx?URL=$MYHOST$MYURL"}}

Thanks in advance,
application delivery
dev
devops
iRules

6 Replies

Sort By
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Feb 27, 2009
    What are the symptoms of the issue? Do you think the problem is just with the URIs that are being redirected? If so, do you know whether the spider will follow 302 redirects? Maybe the spider only follows 301 redirects. If that's the case, you could change HTTP::redirect... to something like:

     

     

    HTTP::respond 301 Location "https://intranet.bmc.com/_layouts/redirects/Default.aspx?URL=$MYHOST$MYURL"

     

     

    If the problem isn't with just the File Not Found redirects, can you provide more details?

     

     

    Aaron
  • Vishal_96707's avatar
    Vishal_96707
    Icon for Nimbostratus rankNimbostratus
    Feb 27, 2009

    http://sptest.bmc.com/SMTIndia

     

    Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled. (The item was deleted because it was either not found or the crawler was denied access to it.)

     

     

     

    Here is what we are getting on the index server. If you remove the iRule everything woks fine. This is the only data that i have. As per the sharepoint team the account used for crawling has got Admin rights. If you hit the same url manually and use same credentials it works fine. Let me know if you need more info.
  • Vishal_96707's avatar
    Vishal_96707
    Icon for Nimbostratus rankNimbostratus
    Feb 27, 2009
    Symptoms,

     

    - SP 2007 index server is crawling SP2003 Portal server

     

    - I have checked using tcpdump and there was only one place where the redirect was done by F5 using iRule. All the 401 Unauthorized messages are returned by IIS server. All these messages start only when you apply the iRule

     

    - If you remove the iRules and crawl it works fine.

     

    - As per the app team, they start seeing access denied messages frequently and the crawl fails
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Feb 27, 2009
    Does the issue occur on every request or just the ones that trigger the iRule redirect? Are you redirecting the spider to a site which it doesn't have access to?

     

     

    Aaron
  • Vishal_96707's avatar
    Vishal_96707
    Icon for Nimbostratus rankNimbostratus
    Feb 27, 2009
    Some more tests..

     

     

    I tried to log iRules however it is not matching with any of the index server error messages.

     

     

    Index Server

     

    http://sptest.bmc.com/CCB/Lists/Objectives

     

    Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled. (The item was deleted because it was either not found or the crawler was denied access to it.)

     

     

    LTM

     

    : Found HTTP Status 404 and the uri is sptest.bmc.com /default_files/filelist.xml Redirecting

     

    Feb 27 06:28:21 tmm tmm[1084]: Rule File_Not_Found-404-Redirect : Found HTTP Status 404 and the uri is sptest.bmc.com /Project%20Status%20for%20Sharepoint%20new_files/filelist.xml Redirecting

     

    Feb 27 06:28:21 tmm tmm[1084]: Rule File_Not_Found-404-Redirect : Found HTTP Status 404 and the uri is sptest.bmc.com /default_files/filelist.xml Redirecting

     

    Feb 27 06:28:23 tmm tmm[1084]: Rule File_Not_Found-404-Redirect : Found HTTP Status 404 and the uri is sptest.bmc.com /default_files/filelist.xml Redirecting

     

    Feb 27 06:28:26 tmm tmm[1084]: Rule File_Not_Found-404-Redirect : Found HTTP Status 404 and the uri is sptest.bmc.com /default_files/filelist.xml Redirecting

     

    Feb 27 06:28:27 tmm tmm[1084]: Rule File_Not_Found-404-Redirect : Found HTTP Status 404 and the uri is sptest.bmc.com /ESM/_vti_bin/sitedata.asmx Redirecting

     

     

    There is something with the iRules since once we remove the iRule it starts working. Even dont get 401 Unauthorized messages
  • Vishal_96707's avatar
    Vishal_96707
    Icon for Nimbostratus rankNimbostratus
    Mar 09, 2009
    The issue has been reolved by bypassing index server hitting redirection logic in iRule.