Using sed Capture Groups (Linux/Mac)

This will be a short one and belongs to the TIL bin: until 2 days ago I did not even know about capture groups and how to use it. So, I knew how to replace a matching string/pattern in sed

bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v/swallow_speed/' 
As of today swallow_speed=23kph at STD
bash-3.2$ 

And how to replace from a given pattern all the way to the end of the line. Or starting from the beginning of the line to said pattern:

bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*/swallow_speed=42/' 
As of today swallow_speed=42
bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/^.*swallow_v=/Can you believe that swallow_speed=/' 
Can you believe that swallow_speed=23kph at STD
bash-3.2$ 

If you are curious, the .* in the search pattern means "any character or list of characters here, be it zero or a lot of characters". The dot (.) does the any part and the asterisk (*) the how many. I think this is from regex, but don't quote me on that. Fine, what if I want to replace everything between two patterns, but leaving the second pattern alone? Tricky. You see, replacing everything between the two patterns, inclusive is not that hard

raub@desktop:~$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*k/swallow_v=25/'
As of today swallow_v=25ph at STD
raub@desktop:~$ 

But to preserve the second pattern we need to use the Capture Groups mentioned in the title of this article. And that makes sense because if it is on the title I better use it. So, we are supposed to surround the capture group pattern with parenthesis and then we can refer to them. If it does not make sense, I too was confused, so let's keep on using our test string:

raub@desktop:~$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*(k)/swallow_v=25$1/'
As of today swallow_v=23kph at STD
raub@desktop:~$ 

Er, it does not seem to have worked according to the plan. In fact, it was supposed to at least grab swallow_v=23k but as you can see it did not find the pattern. Is $1 the proper way to output the captured string? Going nowhere slowly.

After much soul searching, I found the -e requires the parenthesis to be escaped. And, the capture group pattern is output using \1 instead of $1.So we try again:

raub@desktop:~$ echo "As of today swallow_v=23kph at STD" | sed -e "s/swallow_v=.*\(k\)/swallow_v=25\1/"
As of today swallow_v=25kph at STD
raub@desktop:~$ 

much better!

What about the Mac? Same thing (the bash-3.2$ you have seen all day is it; the raub@desktop:~$ is the Linux box):

bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*\(k\)/swallow_v=25\1/' 
As of today swallow_v=25kph at STD
bash-3.2$ 

I am not going to say it is perfect though:

bash-3.2$ echo "As of today swallow_v=23kph at STD" | sed -e 's/swallow_v=.*\([[:blank:]]\)/swallow_v=25\1/' 
As of today swallow_v=25 STD
bash-3.2$ echo "As of today swallow_v=23kph at    STD" | sed -e 's/swallow_v=.*\([[:blank:]]\)/swallow_v=25\1/' 
As of today swallow_v=25 STD
bash-3.2$ 

References

• A site I think has lots of interesting sed examples.

Network packet capturing in Windows without extra programs

One of the things that separate the Linux from Windows is that

When you want to take a look at what is happening on the network, you want to listen to the wire. In Linux you can run tcpdump, wireshark (GUI) and tshark (console), or even wiping up a script in python or bash. So, it can be done with something that comes with the OS by default (most of distros come with python and bash and a lot also have tcpdump) or can be easily added (wireshark).

Then we have Windows... it better be since the title of this article hints that it might be involved. Common sense and standard practices dictate that if you want to do packet capture in that OS, you should buy or download a program/app such as (surprise!) wireshark or something that was created specifically for Windows. Which is fine... unless you are running in a server. Ask yourself: why should we install and run wireshark in a Web Server? And probably leave it there in case we might need it again, so someone can have it ready to go after breaking into the system (this is related to my pet peeve about developing or at least leaving development software on production servers in general and web servers specifically). Or worse: search the web and download a suspicious packet capture app because it had "EZ" on its name and a cute turtle as its logo? That smells like a security risk besides adding weight to your server; ideally you should only have the packages and programs you need.

That looks like a bit of a drag. It would be really nice if we could do packet capturing in Windows without needing to install yet another program. Perhaps even using what is built-in the host.

One can dream...

Starting the capture

So, let's see how to do it then. The command we want is netsh trace, which will need to be run with escalated privileges because it is accessing the network port. Here is how we would capture everything and save it to the file pickles.etl:

netsh trace start persistent=yes capture=yes tracefile=pickles.etl

There are a few useful options we might want to know:

• maxsize : max size of log file before it gets overwritten
  
  • maxsize=250 MB is the default
    
  • maxsize = 0 unlimited
    
  NOTE: if you use this option you also need to add the option filemode or it will not work.

  filemode={circular|append|single}

  Ex:

  netsh trace start persistent=yes capture=yes tracefile=stuff maxsize=0  filemode=append

• persistent : Keep on logging after a reboot
  persistent = no (default)

Stopping capturing

netsh trace stop
Correlating traces ... done
Merging traces ... done
Generating data collection ...

Don't look at me like that; you guessed what the stop command was while I was typing this. Anyway, we end up with two documents:

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        4/24/2017   3:59 PM         650033 pickles.cab
-a----        4/24/2017   3:58 PM         524288 pickles.etl

But for now we only care about the .etl one.

Converting file to something less proprietary

Note:I deleted my original capture after I posted this and forgot to put screen captures. So I had to add the images later.

Unfortunately, this time we will need to download something called Microsoft Message Analyzer. On my defense, it is (currently) a free Microsoft product. You will need to install it as admin because otherwise it will not allow everyone on the machine to run it as the error message states:

Thing is, I would rather have only me able to run it but I am not given an option as implied in the above image. But I digress.

So do install it and then run it. It will take some time to load everything up and be ready for business.

The way MessageAnayzer shows packets is different than Wireshark, which does not mean it is bad. But all I want is to convert it, so we open the file pickles.etl.

As I said, it does look different than wireshark. But I know wireshark better so let's do some exporting: Hit File->Save As and you then will be able to export it:

Save it as a .pcap file and then wireshark will be happy. Yes I know it required to install an extra program in the end but this can be done in our desktop, not on the machine we did the packet acquisition. Would you agree we have a working solution that met our requirements?