I don't think it is possible to distinguish, why would you need to do that? I believe a subtable with all keys removed simply ceases to exist...
if {[table keys -subtable sometable -count] ==0} {
subtable is empty or does not exist
}
if you really need to do this then you can place a dummy key in your subtable and ccheck for it using "-mustexist" option.
Extract from Wiki: https://devcentral.f5.com/wiki/iRules.table.ashx?lc=1
"
If -mustexist is specified, and the key does not already exist, no action will be taken, and an empty string will be returned. Cannot be specified with -excl.If -excl is specified, and the key already exists, the key will not be inserted, and the existing value will be returned. Cannot be specified with -mustexist.
"
Hope this helps,
Sam