I think the poster might be looking to validate "files" uploaded via HTTP using the POST method. If that's the case, the options are somewhat limited. The filename is passed as one parameter and the "file" content is passed in a second parameter as encoded binary data. There isn't anything stopping a malicious user from naming a file with a .txt extension but then uploading any arbitrary content. And there isn't much of a way to validate what the encoded binary data contains.
Here is some detail from RFC1867:
http://www.ietf.org/rfc/rfc1867.txt
6. Examples
Suppose the server supplies the following HTML:
> ACTION="http://server.dom/cgi/handle"
ENCTYPE="multipart/form-data"
METHOD=POST>
What is your name?
What files are you sending?
and the user types "Joe Blow" in the name field, and selects a text
file "file1.txt" for the answer to 'What files are you sending?'
The client might send back the following data:
Content-type: multipart/form-data, boundary=AaB03x
--AaB03x
content-disposition: form-data; name="field1"
Joe Blow
--AaB03x
content-disposition: form-data; name="pics"; filename="file1.txt"
Content-Type: text/plain
... contents of file1.txt ...
--AaB03x--
Content-Type: text/plain
... contents of file1.txt ...
--BbC04y
Content-disposition: attachment; filename="file2.gif"
Content-type: image/gif
Content-Transfer-Encoding: binary
...contents of file2.gif...
--BbC04y--
--AaB03x--
ASM provides some functionality along these lines. You can easily validate the size of the file data parameter and the value of the filename parameter.
If you're still interested in trying this in an iRule, can you provide more detail on exactly what you're trying to validate so we can provide some suggestions?
Aaron