I agree with jquadri about the overall solution: if you can use Least Connections or Dynamic Ratio (maybe combined with persistence) to load balance new vs. existing users to the least loaded server, that'll be faster than inspecting requests.
About your question:
If your app sets a cookie or URI query parameter for users that are logged in that would be better than reading POSTs. Or if you're don't mind making the differentiation more general, and just checking to see if they're already on the site (not neccessarily "logged in") you could enable cookie persistence and check for the BIG-IP cookie to determine if they're new or not.
There are some iRules on CodeShare that might give you some ideas on how to handle your situation. They don't have the exact same use-case, but some ideas might be reusable:
http://devcentral.f5.com/wiki/default.aspx/iRules/HTTP_delay_and_validate_clients_using_javascript__cookies_when_CPU_is_overloaded.html
Also check out this iRule:
http://devcentral.f5.com/Default.aspx?tabid=109
Regardless of how you determine users are new vs. existing, I suggest only doing that check once. Even if you read POST data, after you've determined they're already logged in I suggest setting a cookie so you don't have to inspect every POST from the same user.
To answer your question specifically, there are different tiers of 'cost' for inspecting more data. For example, inspecting 5 bytes is probably the same as inspecting 10, but inspecting 100 is probably more than 5, and 200 may be the same as 100, but 500 may be more than 200. I'm not sure of the exact cost difference, because usually other factors dominate -- you can determine the speed of your particular iRule using one of the links below -- but I do know that in some cases there's no additional cost for inspecting a few more bytes, though in general it does cost more to inspect more data.
Once you have the data, how you do the inspecting is going to be an important factor in performance. If you can check stuff in the header (i.e. URI, cookies) that's faster than inspecting data. If you have to inspect data, whether you use starts_with vs. contains vs. scan will matter. If you're using starts_with, there's probably no reason to check 100 bytes -- just check enough bytes to be unique.
Another idea would be to use caching to offload the server:
http://devcentral.f5.com/Default.aspx?tabid=63&articleType=ArticleView&articleId=283
Maybe even use an iRule to force caching of more content that wouldn't normally be cachable (or maybe all content except POSTs) during the overload case, so the users at least get a response, even if it's a cached response.
You might have an iRule that checks to see if the client is sending an if-modified-since request, and immediately respond with a 304 "not modified" when the server is overloaded so requested for freshness are handled by BIG-IP (of course, you should only do this if the content really doesn't change). So offloading the server when it's overloaded might be a good idea, too.
You can find recommendations on how to write fast iRules here:
http://devcentral.f5.com/wiki/default.aspx/iRules/HowToWriteFastRules.html
And you can see how to check the performance of your iRule here:
http://devcentral.f5.com/Default.aspx?tabid=63&articleType=ArticleView&articleId=123
Other ideas that might be helping (persist users based on payload content, detect web page load time to make automatically engage the special mode you're takling about, etc):
http://devcentral.f5.com/wiki/default.aspx/iRules/LTMMaintenanceWindow.html
http://devcentral.f5.com/wiki/default.aspx/iRules/Persist_client_on_response_content_with_stream.html
Check page load time:
http://devcentral.f5.com/Default.aspx?tabid=53&view=topic&postid=24456
Good luck!