Search
Joe Pruitt - A Software Architect's take on Network Security
You are here: DevCentral > Weblogs

posted on Wednesday, May 06, 2009 9:48 AM

PowerShell_unix PowerShell is definitely gaining momentum in the windows scripting world but I still hear folks wanting to rely on Unix based tools to get their job done.  In this series of posts I’m going to look at converting some of the more popular Unix based tools to PowerShell.

cut

The Unix “cut” command is used to extract sections from each link of input.  Extraction of line segments can be done by bytes, characters, or fields separated by a delimiter.  A range must be provided in each case which consists of one of N, N-M, N- (N to the end of the line), or –M (beginning of the line to M), where N and M are counted from 1 (there is no zeroth value).

For PowerShell, I’ve omitted support for bytes but the rest of the features is included.  The Parse-Range function is used to parse the above range specification.  It takes as input a range specifier and returns an array of indices that the range contains.  Then, the In-Range function is used to determine if a given index is included in the parsed range. 

The real work is done in the Do-Cut function.  In there, input error conditions are checked.  Then for each file supplied, lines are extracted and processed with the given input specifiers.  For character ranges, each character is processed and if it’s index in the line is in the given range, it is appended to the output line.  For field ranges, the line is split into tokens using the delimiter specifier (default is a TAB).  Each field is processed and if it’s index is in the included range, the field is appended to the output with the given output_delimiter specifier (which defaults to the input delimiter).

The options to the Unix cut command are implemented with the following PowerShell arguments:

Unix PowerShell Description
FILE -filespec The files to process.
-c -characters Output only this range of characters.
-f -fields Output only these fields specified by given range.
-d -delimiter Use DELIM instead of TAB for input field delimiter.
-s -only_delimited Do not print lines not containing delimiters.
--output-delimiter -output_delimiter Use STRING as the output deflimiter.

 

   1: #----------------------------------------------------------------
   2: # Cut.ps1
   3: #----------------------------------------------------------------
   4: param
   5: (
   6:   [string]$filespec = $null,
   7:   [string]$characters = $null,      # Output only this range of characters
   8:   [string]$fields = $null,          # Output Only this field range (ie 1-2; 3,4)
   9:   [string]$delimiter = $null,        # Use DELIM instead of TAB for field delimiter
  10:   [bool]$only_delimited = $false,   # Do not print lines not containing delimiter
  11:   [string]$output_delimiter = $null # Use STRING as output delimiter 
  12: );
  13:  
  14: $script:IN_DELIMITER = "`t";
  15: if ( $delimiter ) { $script:IN_DELIMITER = $delimiter; }
  16: $script:OUT_DELIMITER = $delimiter;
  17: if ( $output_delimiter ) { $script:OUT_DELIMITER = $output_delimiter; }
  18:  
  19: $script:MAX_COLUMNS = 1000;
  20:  
  21: #----------------------------------------------------------------
  22: # function Show-Error
  23: #----------------------------------------------------------------
  24: function Show-Error()
  25: {
  26:   param([string]$msg = $null);
  27:   if ( $msg )
  28:   {
  29:     Write-Host $msg;
  30:     exit;
  31:   }
  32: }
  33:  
  34: #----------------------------------------------------------------
  35: # function Parse-Range
  36: #----------------------------------------------------------------
  37: function Parse-Range()
  38: {
  39:   param([string]$range);
  40:   
  41:   [int[]]$indices = @();
  42:   if ( $range )
  43:   {
  44:     $tokens = $range.Split(',');
  45:     foreach ($token in $tokens )
  46:     {
  47:       if ( $token.Contains("-") )
  48:       {
  49:         $subtokens = $token.Split('-');
  50:         if ( $subtokens.Length -ne 2 )
  51:         {
  52:           Show-Error "Cut: Invalid character or field list."
  53:           exit;
  54:         }
  55:         else
  56:         {
  57:           if ( ($subtokens[0].Length -eq 0) -and ($subtokens[1].Length -gt 0) )
  58:           {
  59:             # -N
  60:             $indices += @(1 .. $subtokens[1]);
  61:           }
  62:           elseif ( ($subtokens[1].Length -eq 0) -and ($subtokens[0].Length -gt 0) )
  63:           {
  64:             # N-
  65:             $indices += @($subtokens[0] .. $script:MAX_COLUMNS);
  66:           }
  67:           else
  68:           {
  69:             # N-N
  70:             if ( $subtokens[1] -lt $subtokens[0] )
  71:             {
  72:               Show-Error "Cut: Invalid character or field list";
  73:             }
  74:             else
  75:             {
  76:               $indices += @($subtokens[0] .. $subtokens[1]);
  77:             }
  78:           }
  79:         }
  80:       }
  81:       else
  82:       {
  83:        $indices += @($token);
  84:       }
  85:     }
  86:   }
  87:   $indices;
  88: }
  89:  
  90: #----------------------------------------------------------------
  91: # function In-Range
  92: #----------------------------------------------------------------
  93: function In-Range()
  94: {
  95:   param
  96:   (
  97:     [int]$index = 0,
  98:     [int[]]$range = $null
  99:   );
 100:   $inrange = $true;
 101:   
 102:   if ( $range )
 103:   {
 104:     $inrange = $false;
 105:     foreach ($i in $range )
 106:     {
 107:       if ( $i -eq $index )
 108:       {
 109:         $inrange = $true;
 110:       }
 111:     }
 112:   }
 113:   $inrange;
 114: }
 115:  
 116: #----------------------------------------------------------------
 117: # function Do-Cut
 118: #----------------------------------------------------------------
 119: function Do-Cut()
 120: {
 121:   param
 122:   (
 123:     [string]$filespec = $null,
 124:     [string]$characters = $null,
 125:     [string]$fields = $null,
 126:     [string]$delimiter = $null,
 127:     [bool]$only_delimited = $false,
 128:     [string]$output_delimiter = $null
 129:   );
 130:   
 131:   if ( $filespec )
 132:   {
 133:     # Check parameters
 134:     if ( $characters -and $fields )
 135:     {
 136:       Show-Error "Cut: only one type of list may be specified.";
 137:     }
 138:     elseif ( !$characters -and !$fields )
 139:     {
 140:       Show-Error "Cut: You must specify a list of characters or fields.";
 141:     }
 142:     else
 143:     {
 144:       if ( $characters )
 145:       {
 146:         $range = Parse-Range -range $characters;
 147:       }
 148:       elseif ( $fields )
 149:       {
 150:         $range = Parse-Range -range $fields;
 151:       }
 152:       
 153:       $files = @(Get-ChildItem -Path $filespec);
 154:       foreach ($file in $files)
 155:       {
 156:         $lines = Get-Content -Path $file;
 157:         foreach ($line in $lines)
 158:         {
 159:           $line_out = $line;
 160:           if ( $characters )
 161:           {
 162:             $line_out = "";
 163:             for($i = 1; $i -le $line.Length; $i++)
 164:             {
 165:               if ( In-Range $i $range )
 166:               {
 167:                 $line_out += $line[$i-1];
 168:               }
 169:             }
 170:           }
 171:           elseif ( $fields )
 172:           {
 173:             $line_out = "";
 174:             $line_fields = $line.Split($script:IN_DELIMITER);
 175:             if ( $only_delimited -and ($line_fields.Length -eq 1) )
 176:             {
 177:               # Skip this line since it didn't contain the delimiter
 178:               continue;
 179:             }
 180:             else
 181:             {
 182:               for($i=1; $i -le $line_fields.Length; $i++)
 183:               {
 184:                 if ( In-Range $i $range )
 185:                 {
 186:                   if ( $line_out.Length ) { $line_out += $script:OUT_DELIMITER; }
 187:                   $line_out += $line_fields[$i-1];
 188:                 }
 189:               }
 190:             }
 191:           }
 192:           $line_out;
 193:         }
 194:       }
 195:     }
 196:   }
 197: }
 198:  
 199: Do-Cut -filespec $filespec -characters $characters -fields $fields `
 200:   -delimiter $delimiter -only_delimited $only_delimited `
 201:   -output_delimiter $output_delimiter;

You can download the script here: Cut.ps1



Feedback

5/6/2009 11:49 AM
Gravatar Or you could just stick with unix and avoid having to write your own version of basic tools that have existed for 20 years. :) Microsoft SUA, Win32 GNU tools, Cygwin, VMware.

I won't claim unix is without faults, and in the realm of scripting it's easy to show that PowerShell is far more consistent than unix shell scripting. This consistency sometimes comes at the cost being pedantic, but even so, I can't say this makes PowerShell a worse solution for all users.

However, I can say without a doubt that Microsoft's vast technical gap created by a long history of providing a 2nd-class CLI wasn't completely closed by releasing PowerShell. As a heavy user of resource kit tools, sysinternals, 3rd party tools, and a light user of PowerShell, I still find myself routinely using unix CLI tools because they're faster and more feature-rich. Look at bash tab-completion alone and you'll see how far Microsoft has to go. Not to mention grep, wget, etc.

PowerShell has some nice qualities for sure, but to be honest it's not "there" yet for hard-core CLI users -- still too few built-in tools and too much awkward syntax. Let's all hope Microsoft continues their CLI investment so we don't have to cobble together our own versions of cut 5 years from now. :)
UnixUser
5/6/2009 2:17 PM
Gravatar @UnixUser - Great point. PowerShell has been a big jump from Command.exe and the rich integration with the new server platforms will make it more prevalent in the future. I too rely on my set of unix tools (grep, curl, wget, etc) and that is what this series of posts was geared to address.

Again, thanks for the great feedback and if you do take a look at some of these scripts I've written, I'd love to hear feedback on their implementation/usage.

Cheers!

-Joe
Joe Pruitt
6/18/2009 10:52 AM
Gravatar Hello,

I am very new to using Powershell (we've used unix in the past) and this is exactly what I need.

Would you by any chance have an example of how an entry should look? I can't seem to get it working, after trying many different ways.

Currently I have

Do-Cut -filespec ./test.csv -fields "1,10,15" -delimiter ","

Sorry this is probably extremely simple, but I'm stuck. Thanks.

Tony
6/18/2009 12:25 PM
Gravatar That should work, are you seeing that it isn't? If you could post a few lines of your test.csv file, I'll test it out.

-Joe
Joe Pruitt
6/29/2009 1:26 PM
Gravatar Do-Cut -filespec ./test.csv -fields "1,10,15" -delimiter ","

try this, i dont know but maybe you can differantiate

=cut-filespec ./test.csv -fields "1,10,15" -delimiter ","
Teknoloji
7/15/2009 2:49 PM
Gravatar PowerShell has some nice qualities for sure, but to be honest it's not "there" yet for hard-core CLI users -- still too few built-in tools and too much awkward syntax
tooth
7/15/2009 2:51 PM
Gravatar Can you provide examples of the most common delimiters? I am always confused at how to represent newline characters in windows.
Techachino
7/15/2009 3:32 PM
Gravatar Wow. Great post and nice script. Keep up the great work. :)
Novo PC
7/15/2009 4:35 PM
Gravatar Great stuff mate. Please some examples of the most common delimiters. Here also are a newbe in this. Thanks.
Diseño web Ciudad Real
7/15/2009 6:06 PM
Gravatar Thank you for the download. I just within the last couple of months started diving into Unix a bit more and have made some good headway. Its a bit unforgiving at first, but once you begin to grasp the concept of certain actions a bit, it gets easier. :)

Anyways, I will give this a shot and see what I can come up with.

Kyle
comsoft
7/15/2009 6:28 PM
Gravatar Great...
i like this script...
hope many people love it too..
:)

Adi
7/15/2009 6:39 PM
Gravatar Grt. work. Pls. keep posting such informative articles.
Your Finance Guide
7/29/2009 8:17 PM
Gravatar Very nice blog.please update more
vbulletin services
9/8/2009 12:27 AM
Gravatar Hi,
When i try to pass the follwoing command to a psc1 file I get an error due to | delimiter

please help me on this issue.

.bat file content:

powershell.exe -PSConsoleFile C:\idm_log\one.psc1 -nologo -command "get-wmiobject -class

MSFT_SIPESUserSetting | where-object { $_.PrimaryURI -eq 'sip:Navin@OCSR2.COM' } | {

$_.Enabled = $True; $_.put() | out-null }"

Error Msg
Expressions are only permitted as the first element of a pipeline.
At line:1 char:147
+ get-wmiobject -class MSFT_SIPESUserSetting | where-object { $_.PrimaryURI -eq
'sip:Navin@OCSR2.COM' } | { $_.Enabled = $True; $_.put() | out-null } <<<<


what shoule be the content of the bat file or how shoule be the command passed to the psc1 file.
nvn
9/8/2009 11:51 AM
Gravatar I think your issue isn't with the passing in of the command but rather with how your command is formulated. When I tried to enter just the "get-wmiobject ..." directly into powershell I got the same error. The issue is with the code after the second pipe. I'd try something like this:

"Get-WMIObject -class MSFT_SPIESUserSetting | where-object { $_.PrimaryURI -eq 'sip:Navin@OCSR2.COM' } | foreach-object -process { $_.Enabled = $True; $_.put() | out-null; }"

I've put a foreach-object before your last expression block and it seems to work for me.

Let me know how it goes...

-Joe
Joe
11/6/2009 9:27 AM
Gravatar Thank you, Nice Script

www.scripterworld.com/.../...command-examples.html
Bhagya
11/6/2009 10:16 AM
Gravatar Thank you,Nice Script

Unix cut command with examples
Bhagya
6/26/2010 5:23 AM
Gravatar Hi nice post, i have come across your site once before when searching for something so i was just wondering something. I love your theme, would it happen to be a free one i can download, or is it a custom one you had made? In a few weeks i will be launching my own site, i’m not great with designs but i really like the style of your site so it would be cool if i could find (or pay for) something with a similar look...
Kasino Bonus Guide
7/27/2010 7:44 AM
Gravatar Hello,

I wanted to try your command to do the following unix command:
cut -d: -f1 /etc/passwd

I copied /etc/passwd file in passwd.txt and was running the following command, but I am not getting the desired output

Do-Cut -filespec .\passwd.txt -fields "1" -delimiter ":" ;

Am I missing something here?

Thanks.
shivashis
7/17/2010 2:15 PM
Gravatar Thank you.

Btw
[code]70: if ( $subtokens[1] -lt $subtokens[0] )[/code]
should be
[code]70: if ( [int] $subtokens[1] -lt [int] $subtokens[0] )[/code]
otherwise ranges like 4-10 or 40-5 are wrong.
sevenforce
7/17/2010 2:22 PM
Gravatar Thank you.

Btw
70: if ( $subtokens[1] -lt $subtokens[0] )
should be
70: if ( [int] $subtokens[1] -lt [int] $subtokens[0] )
otherwise ranges like 4-10 or 40-5 are wrong.
sevenforce
7/19/2010 12:10 PM
Gravatar @sevenforce, thanks for the fix, I'll test it out and get the post updated.

-Joe
Joe

Let Me Know What You Think


Please use the form below if you have any comments, questions, or suggestions.

Title:
 
Name:
 
Email: (so we can show your gravatar)
Website:
Comment: Allowed tags: blockquote, a, strong, em, p, u, strike, super, sub, code
 
Please add 4 and 2 and type the answer here:

Blog Stats

Posts:379
Comments:1067
Stories:1
Trackbacks:301
  

Article Categories

  iRules
  

Image Galleries

  

Joe's bookshelf: read

The Lost Gate
4 of 5 stars
This one started slow but I got really got into it about 1/3 of the way through. If you are an Ender's Game fan, you'll probably like this one as well.

goodreads.com


82,243 Members in 102 Countries and Growing!

Join DevCentral Today!

About DevCentral

DevCentral has been a successful, thriving community for many years. We have always strived to bring you the best technical documentation, discussion forums, blogs, media and much more that we can.

So dive in, get familiar with DevCentral. We hope you like it, we hope it makes your job easier, and lets you get that much more power out of the community. To learn more, make sure to check out the Getting Started section. And if you have any problems, or think something could be easier to use, drop us a line to let us know.

Got It !

We've received your comment and transmitted it directly to DevCentral HQ.

Thanks for taking time to let us know what's on your mind. At DevCentral | Community Matters!

Get In Touch With Us

Have questions, suggestions or just want to get something off your chest?

Use our handy form below to Direct Connect with DevCentral Mission Control.

Send Us Feedback       or