Category Archives: Uncategorized

Using ‘jq’ to query JSON

jq

jq is great for CLI manipulation of JSON data!  Handy little bit of knowledge here, stashing so i can remember.

You’re probably familiar with the fact that you can use the command line tool jq to do pretty printing of JSON on the command line, like this:

$ cat file.json  | jq "." | head -n 30
[
  {
    "collection": "state_of_oklahoma",
    "ip_address": "156.110.191.1",
    "port": 3389,
    "protocol": "tcp",
    "service": "RDP",
    "first_seen": null,
    "last_seen": null,
    "timestamp": "2019-05-12T02:02:29+00:00"
  },
  {
    "collection": "state_of_oklahoma",
    "ip_address": "156.110.192.1",
    "port": 3389,
    "protocol": "tcp",
    "service": "RDP",
    "first_seen": null,
    "last_seen": null,
    "timestamp": "2019-05-12T02:02:41+00:00"
  },

But did you know you can use it to query attributes? For instance, if you want to get the ip_address of all items running RDP, you can query like such:

$ cat file.json | jq -c '.[]| select(.service=="RDP") | .ip_address'

This essentially tells jq to iterate through the outer array, selecting any hash with the “service” field set to “RDP, and then print the ip_address field. And the output?

$ cat file.json | jq -c '.[]| select(.service=="RDP") | .ip_address'
"156.110.191.1"
"156.110.192.1"
...

Also, if you want those quotes removed, add a ‘-r’ switch:

$ cat file.json | jq -r -c '.[]| select(.service=="RDP") | .ip_address'

Hope this is helpful, check out the jq tutorial for more info !

You and Your Research

I was recently pointed to a great speech entitled “You and Your Research” given by Richard Hamming of Bell Labs (and Hamming codes!) fame. It’s essentially Hamming giving his insight on how to do great work as a scientist. I think it’s relevant for anyone doing infosec research today.

Here are my key takeaways:

  • Work on important problems.
  • Luck favors the prepared mind.
  • Courage is a characteristic of the successful.
  • Plant acorns to grow oak trees.
  • Follow the greats in your field.
  • Every defect can be looked at as an asset.
  • Knowledge and productivity are like compound interest.
  • Keep track of flaws in your theories.
  • Work on problems you’re committed to.
  • Get emotionally involved, otherwise your subconscious goofs off.
  • Reach out to people outside of your field.
  • Pursue opportunity when its presented.
  • Find and know the important problems in your field.
  • Practice makes perfect.
  • Schedule some dedicated time to make “great thoughts time”
  • Open doors -> more input -> finding the right problems.
  • Zoom out to see the larger problem.
  • You want others to stand on the shoulders of your work.
  • It’s not sufficient to do a job, you have to sell it.
  • Write clearly and well so that people will read it.
  • Learn to give formal talks.
  • Learn to give informal talks.
  • When giving a talk, start slowly and paint a general picture of why its imporant, and give a sketch of what was done.
  • Educate your boss, get other people to ask for what you need.
  • Take advantage of the systems around you to scale yourself.
  • Know thyself & watch thy ego.
  • The appearance of conforming gets you a long way.
  • Don’t spend effort needlessly fighting the system and don’t fool yourself by creating alibis for disappointment.
  • A little extra effort goes a long way with people.

Powerful words to guide your career.

password hangover

Just saw the Hangover 2. – funny (and true) bit on passwords…

as an international drug dealer tranfers money between accounts:

“your password is bologna1?”

“it used to be bologna, but they make you include a stupid number now”

*facepalm*

sadly (… or happily, depending on your perspective  :] ) , weak passwords are still common…. metasploit has some awesome modules to test passwords:

jcran@disko:~/framework/modules$ find . |grep _login | grep -v svn

  • ./auxiliary/scanner/snmp/snmp_login.rb
  • ./auxiliary/scanner/mssql/mssql_login.rb
  • ./auxiliary/scanner/postgres/postgres_login.rb
  • ./auxiliary/scanner/http/wordpress_login_enum.rb
  • ./auxiliary/scanner/http/axis_login.rb
  • ./auxiliary/scanner/http/tomcat_mgr_login.rb
  • ./auxiliary/scanner/http/http_login.rb
  • ./auxiliary/scanner/http/frontpage_login.rb
  • ./auxiliary/scanner/ftp/ftp_login.rb
  • ./auxiliary/scanner/vnc/vnc_login.rb
  • ./auxiliary/scanner/ssh/ssh_login_pubkey.rb
  • ./auxiliary/scanner/ssh/ssh_login.rb
  • ./auxiliary/scanner/telnet/telnet_login.rb
  • ./auxiliary/scanner/sap/sap_mgmt_con_brute_login.rb
  • ./auxiliary/scanner/lotus/lotus_domino_login.rb
  • ./auxiliary/scanner/mysql/mysql_login.rb
  • ./auxiliary/scanner/rservices/rsh_login.rb
  • ./auxiliary/scanner/rservices/rlogin_login.rb
  • ./auxiliary/scanner/rservices/rexec_login.rb
  • ./auxiliary/scanner/smb/smb_login.rb
  • ./auxiliary/scanner/oracle/isqlplus_login.rb
  • ./auxiliary/scanner/oracle/oracle_login.rb
  • ./auxiliary/fuzzers/tds/tds_login_username.rb
  • ./auxiliary/fuzzers/tds/tds_login_corrupt.rb
  • ./auxiliary/fuzzers/smb/smb_ntlm1_login_corrupt.rb
  • ./auxiliary/admin/oracle/oracle_login.rb
  • ./exploits/windows/imap/mailenable_login.rb
  • ./exploits/windows/imap/mercury_login.rb
  • ./exploits/windows/http/hp_power_manager_login.rb

pianobar is epic!

@mrbarrett just pointed me at pianobar, a command-line client for pandora. It gets rid of the need for flash player, and allows you to bypass some of the limitations of the web client (you can skip more than 5 songs!).

To install (on Ubuntu):


sudo apt-get install git-core libao-dev libfaad-dev libmad0-dev
git clone git://github.com/PromyLOPh/pianobar.git
cd pianobar
make
sudo make install

enjoy!

Loading only the Metasploit modules you use

The framework is taking quite a while to load on my machine these days, so i decided to stop loading all modules by default, and load only those modules i need. Here’s the process:

Simply comment the module-loading lines in the framework file lib/msf/base/simple/framework.rb, so :


if (Msf::Config.module_directory)

framework.modules.add_module_path(Msf::Config.module_directory)

end

becomes


#if (Msf::Config.module_directory)

#framework.modules.add_module_path(Msf::Config.module_directory)

#end

Or you can apply this patch: http://www.0x0e.org/x/framework-no-default-mods.patch

Alos make sure to remove the ~/.msf3/modcache directory.

Then, mirroring the framework modules directory structure, copy the modules you’d like to load into your .msf3/modules directory. For example, if you wanted to load only the psexec module and the reverse_tcp payload, copy

  • modules/exploits/windows/smb/psexec.rb into ~/.msf3/modules/exploits/windows/smb/psexec.rb
  • modules/payloads/stagers/windows/reverse_tcp.rb into ~/.msf3/modules/payloads/stagers/windows/reverse_tcp.rb

You should now see a load-time speed improvement on the order of:

Before:

<pre>jcran@disko:~/framework$ time ./msfconsole -r exit.rc

                |                    |      _) |
 __ `__ \   _ \ __|  _` |  __| __ \  |  _ \  | __|
 |   |   |  __/ |   (   |\__ \ |   | | (   | | |
_|  _|  _|\___|\__|\__,_|____/ .__/ _|\___/ _|\__|
                              _|


       =[ metasploit v3.5.1-dev [core:3.5 api:1.0]
+ -- --=[ 630 exploits - 310 auxiliary
+ -- --=[ 215 payloads - 27 encoders - 8 nops
       =[ svn r10985 updated today (2010.11.11)

resource (exit.rc)> exit
resource (exit.rc)> exit

real	0m42.750s
user	0m40.710s
sys	0m0.820s</pre>

After:

<pre>jcran@disko:~/framework$ time ./msfconsole -r exit.rc

                 o                       8         o   o
                 8                       8             8
ooYoYo. .oPYo.  o8P .oPYo. .oPYo. .oPYo. 8 .oPYo. o8  o8P
8' 8  8 8oooo8   8  .oooo8 Yb..   8    8 8 8    8  8   8
8  8  8 8.       8  8    8   'Yb. 8    8 8 8    8  8   8
8  8  8 `Yooo'   8  `YooP8 `YooP' 8YooP' 8 `YooP'  8   8
..:..:..:.....:::..::.....::.....:8.....:..:.....::..::..:
::::::::::::::::::::::::::::::::::8:::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


       =[ metasploit v3.5.1-dev [core:3.5 api:1.0]
+ -- --=[ 1 exploits - 0 auxiliary
+ -- --=[ 1 payloads - 0 encoders - 0 nops
       =[ svn r10985 updated today (2010.11.11)

resource (exit.rc)> exit
resource (exit.rc)> exit

real	0m12.232s
user	0m11.340s
sys	0m0.510s</pre>


Not huge, but definitely an improvement.

IP List to KML generator (Create a google map from a list of IPs)

Pretty simple, it takes a file with a list of ips, one/line and generates a kml file. Very handy if you’re working on a large pentest and want to track down (and visualize) where a particular host is located. It uses the Yahoo GeoIP API to grab location data.

#!/usr/bin/ruby

require 'net/http'
require 'rexml/document'

include REXML

def getAddress(ip)
#takes an ip and returns an xml blob with city/state
# example: http://ipinfodb.com/ip_query.php?ip=65.23.23.33
 url = "http://ipinfodb.com/ip_query.php?ip=" + ip
 #puts "DEBUG: URL: #{url.to_s}"
 resp = Net::HTTP.get(URI.parse(url))
 #print "DEBUG: got " + resp
return resp
end

def getCoordinates(address)
#takes a hash with city, state address and returns a hash w/ coords
url = "http://local.yahooapis.com/MapsService/V1/geocode"
 params = {
 "appid" => "GwLDY.bV34HH7gkBDs97p_5U5P_tBfXBnfDyYFwpTRLwZDEvgj8BOQqws1JOCFPyhTQR",
 "street" => "",
 "city" => address["city"],
 "state" => address["state"]
 }
 #puts "DEBUG: URL: #{url.to_s}"
 resp = Net::HTTP.post_form(URI.parse(url), params)
 resp_text = resp.body
 #print "DEBUG: got " + resp_text
return resp_text
end

def parseAddress(xml)
#takes an xml blob with city / state & returns a hash with address,city,state
 doc = Document.new xml
 root = doc.root

 city = root.elements["City"].get_text.to_s
 state = root.elements["RegionName"].get_text.to_s
 country = root.elements["CountryCode"].get_text.to_s

 #puts "DEBUG: city: " + city
 #puts "DEBUG: state: " + state
 #puts "DEBUG: country: " + country

 toReturn = Hash["city" => city, "state" => state, "country" => country]
 return toReturn
end

def parseCoordinates(xml)
#takes an xml blob with coordinates & returns a hash with long/lat
 doc = REXML::Document.new xml
 root = doc.root

 long = REXML::XPath.first( doc, "//Longitude" ).get_text.to_s
 lat = REXML::XPath.first( doc, "//Latitude" ).get_text.to_s

 toReturn = Hash["long" => long, "lat" => lat]

 #puts "DEBUG: long: " + long
 #puts "DEBUG: lat: " + lat

 return toReturn
end

def genKML(ips)
 kml = ""
 kml = kml + "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
 kml = kml + "<kml xmlns=\"http://www.opengis.net/kml/2.2\">\n"
 kml = kml + "<Document>\n"
 ips.each do |ip|
 ip = ip.to_s.chomp
 kmlplacemark = mip(ip,"error.log")

 if kmlplacemark.to_s != "" then
 #        puts "DEBUG: adding non-blank placemark" + kmlplacemark
 kml = kml + kmlplacemark
 else
 #        puts "DEBUG: unable to map ip: " + ip + "\n"
 end
 end

 kml = kml + "</Document>\n"
 kml = kml + "</kml>\n"
end

def genPlacemark(ip,address,coordinates)
 xml = ""
 xml = xml + "    <Placemark>\n"
 xml = xml + "        <name>" + ip + "</name>\n"
 xml = xml + "        <description>"
 xml = xml + address["city"] + ", "
 xml = xml + address["state"] + ", "
 xml = xml + address["country"]
 xml = xml + "</description>\n"
 xml = xml + "        <Point>\n"
 xml = xml + "            <coordinates>" +
 coordinates["long"]  + "," +
 coordinates["lat"] + ",0</coordinates>\n"
 xml = xml + "        </Point>\n"
 xml = xml + "    </Placemark>\n"
end


def mip(ip,errorfile)
 begin
 if (ip != "") then
 xmlAddress = getAddress(ip)
 objAddress = parseAddress(xmlAddress)

 if (objAddress["state"] != "") then
 xmlCoordinates = getCoordinates(objAddress)
 objCoordinates = parseCoordinates(xmlCoordinates)

 kmlplacemark = genPlacemark(ip,objAddress,objCoordinates)
 else
 File.open(errorfile, 'w') {|f| f.write(ip) }

 end
 end
 rescue
 kmlplacemark = ""
 end

return kmlplacemark
end

def mips(file)
 counter = 0
 ips = Array.new

 File.open(file, "r") do |infile|

 while (line = infile.gets)
 #puts "mapping #{counter}: #{line}"

 ips[counter] = line

 counter = counter + 1
 end
 end
 kml = genKML(ips)
 return kml
end


kml = mips(ARGV[0])
out = File.new(ARGV[0]+".kml", "w")
out.puts kml