Aug 25, 2018 • python,tools,intel,cli

One of the tools I use daily is Machinae. Machinae is a python command line tool that allows quick intel gathering on ips, domains, hashes and more. Developed in Python3 by HurricaneLabs Machinae makes quick work of searching and correlating intel sources.

Once installed using Machinae is as simple as typing its name followed by the object you want to search for. Machinae automatically determines the object type whether it be ip, domain, or hash. It then hits the sources for that specific object type. Below is an example of me running Machiane against this site.

$ machinae
* Information for
* Observable type: fqdn (Auto-detected: True)

[+] Results
    [-] IP:
    [-] Country: Netherlands
    [-] Country Code: NL
    [-] Latitude: 52.3824
    [-] Longitude: 4.8995
[-] No URLVoid Results
[-] No W3bin Results
[+] McAfee Threat Results
    [-] Web Risk: Unverified
    [-] Last Seen: 2018-08-25
[+] Fortinet Category Results
    [-] URL Category: Newly Observed Domain
[+] URL Unshorten Results
    [-] Unshortened URL:
[+] Redirect Check Results
    [-] Status Code: 301
    [-] Status Code: 200
[-] No Malc0de Results
[+] VirusTotal Passive DNS Results
    [-] Data: ('2018-08-25', '')
    [-] Data: ('2016-07-12', '')
[-] No Domain Lookup Results
[+] PassiveTotal Passive DNS Results
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Data:
    [-] Query Value:
[+] PassiveTotal Whois Results
    [-] Updated: 2017-09-11T21:49:44.000-0700
    [-] Domain:
    [-] Billing: {}
    [-] Zone: {}
    [-] Name Servers: ['', '', '', '']
    [-] Registered: 2013-10-11T11:15:39.000-0700
    [-] Last Loaded At: 2017-12-08T18:49:18.214-0800
    [-] Whois Server:
    [-] Contact Email:
    [-] Admin:
    [-] Expires: 2018-10-11T11:15:39.000-0700
    [-] Registrar: Google Inc.
    [-] Tech:
    [-] Registrant:
[-] No PassiveTotal Components Results
[-] No PassiveTotal Trackers Results
[-] No Ransomware Tracker Results
[-] No VxStream Sandbox Domain Search Results
[-] No VxStream Sandbox URL Search Results
[-] No Malware Domain List Results
[+] Results
    [-] report:

As you can see this gives us just about everything there is to know about

The main reason I enjoy Machinae is that it is super easy to extend the sources. Machinae uses a yaml configuration file to store the site data in. The default configuration contains plenty of good sources however I have probably tripled the number of sources in my own. If there is a site you use for intel gathering and it either has an API or can be scraped using regex then it most likely can be added to Machinae.

Here is an example of a site from my config file for a Json based API site.

  default: True
    - fqdn
    - url
      url: ''
      method: post
        API-Key: APIKEY_HERE
        Accept: application/json
        url: '{target}'
      - key: result
          key: message
          value: "Submission Successful"
        pretty_name: report

As you can see when defining a site its as easy as naming the site, choosing what objects the site works on, and then defining the request and how to extract the results. You can also turn sites off with the default key, setting this to false would ensure it is not searched unless specified by name when running Machinae (machinae -s SITENAME OBJECT).

Here is an example of a site from my config file for a Webscraper based site.

  name: Fortinet Category
  default: True
    - fqdn
    - url
      url: '{target}'
      method: get
    - regex: 'Category:\s(.+)</h4>'
        - fortinet_category
      pretty_name: URL Category

The layout is the exact same as the first example, however instead of “json” we are using “webscraper” and the results extraction is a bit different. In the results section here, rather than supplying the keys from the json object we want returned, we are giving a regex string that is used to extract the data we want from the page and setting it to a value.

As you can see after adding a lot of sources the output may get a bit long winded. I suggest going through your sources and determining the best ones for you by default and set the others to default: False. Also I commited a change to Machinae to add a --list-sites flag so you can display a table of all of your sites, the objects they run on, and if they are on by default. Here is what that looks like.

$ machinae --list-sites
SITE                                    NAME                                    OTYPES                                  DEFAULT
ipstack                                                    ipv4, fqdn                              True
ipinfoio                                                     ipv4, ipv6                              True
urlvoid                                 URLVoid                                 fqdn                                    True
w3bin                                   W3bin                                   fqdn                                    True
mcafee_threat_domain                    McAfee Threat                           fqdn                                    True
mcafee_threat_ip                        McAfee Threat                           ipv4                                    True
fortinet_classify                       Fortinet Category                       fqdn, url                               True
fraudguard                              FraudGuard                              ipv4                                    True
ipwhois                                 IP Whois                                ipv4                                    True
spamhaus_ip                             Spamhaus Zen BL                         ipv4                                    False
spamhaus_domain                         Spamhaus Domain BL                      fqdn                                    False
ipvoid                                  IPVoid                                  ipv4                                    False
toolsvoid_unshorten                     URL Unshorten                           fqdn, url                               False                            URL Unshorten                           fqdn, url                               True
redirect_check                          Redirect Check                          fqdn, url                               True
malc0de                                 Malc0de                                 ipv4, fqdn, hash                        True
AbuseIPDB                               AbuseIPDB                               ipv4                                    True
sans                                    SANS                                    ipv4                                    True
telize                                  Telize GeoIP                            ipv4                                    False
maxmind                                 MaxMind GeoIP2 Precision                ipv4                                    False
vt_ip                                   VirusTotal Passive DNS                  ipv4                                    True
vt_domain                               VirusTotal Passive DNS                  fqdn                                    True
vt_url                                  VirusTotal URL Report                   url                                     True
vt_hash                                 VirusTotal File Report                  hash, hash.sha1, hash.sha256            True
reputation_authority                    Reputation Authority                    ipv4                                    False
threatexpert                            ThreatExpert                            hash                                    True
vxvault                                 VxVault                                 hash                                    True
projecthoneypot                         ProjectHoneypot                         ipv4                                    False
stopforumspam                           StopForumSpam                           email                                   False
cymru_mhr                               Cymru MHR                               hash, hash.sha1, hash.sha256            True
icsi_notary                             ICSI Certificate Notary                 sslfp                                   False
totalhash_ip                            TotalHash                               ipv4                                    False
domaintools_parsed_whois                DomainTools Whois                       fqdn                                    False
domaintools_reverse_whois               DomainTools Reverse Whois               email                                   False
domaintools_reputation                  DomainTools Reputation                  fqdn                                    False
dnsdb_ip                                Farsight DNSDB                          ipv4, ipv6                              False
dnsdb_fqdn                              Farsight DNSDB                          fqdn                                    False
cif                                     Collective Intelligence Framework       ipv4, fqdn, email, hash                 False
cymon_ip_events                IP Events                      ipv4                                    True
cymon_ip_domains               IP Domains                     ipv4                                    True
cymon_ip_urls                  IP URLs                        ipv4                                    True
cymon_domain                   Domain Lookup                  fqdn                                    True
cymon_url                      URL Lookup                     url                                     False
threatcrowd_ip_report                   ThreatCrowd IP Report                   ipv4                                    True
passivetotal_pdns                       PassiveTotal Passive DNS                fqdn, ipv4                              True
passivetotal_whois                      PassiveTotal Whois                      fqdn                                    True
passivetotal_sslcert                    PassiveTotal SSL Certificate History    ipv4                                    True
passivetotal_components                 PassiveTotal Components                 fqdn                                    True
passivetotal_trackers                   PassiveTotal Trackers                   fqdn                                    True
shodan                                  Shodan                                  ipv4                                    True
alienvaultotx_ipv4_geo                  Alien Vault OTX Geolocation             ipv4                                    False
alienvaultotx_ipv4_malware              Alien Vault OTX Malware Callback        ipv4                                    False
alienvaultotx_ipv4_url                  Alien Vault OTX URL                     ipv4                                    False
alienvaultotx_ipv4_passive_dns          Alien Vault OTX Passive DNS             ipv4                                    False
alienvaultotx_ipv4_http_meta            Alien Vault OTX HTTP Meta Data          ipv4                                    False
alienvaultotx_ipv6_geo                  Alien Vault OTX Geolocation             ipv6                                    False
alienvaultotx_ipv6_malware              Alien Vault OTX Malware Callback        ipv6                                    False
alienvaultotx_ipv6_url                  Alien Vault OTX URL                     ipv6                                    False
alienvaultotx_ipv6_passive_dns          Alien Vault OTX Passive DNS             ipv6                                    False
alienvaultotx_domain_geo                Alien Vault OTX Geolocation             fqdn                                    False
alienvaultotx_domain_malware            Alien Vault OTX Malware Callback        fqdn                                    False
alienvaultotx_domain_url                Alien Vault OTX URL                     fqdn                                    False
alienvaultotx_domain_passive_dns        Alien Vault OTX Passive DNS             fqdn                                    False
alienvaultotx_domain_http_meta          Alien Vault OTX HTTP Meta Data          fqdn                                    False
alienvaultotx_domain_whois              Alien Vault OTX Whois                   fqdn                                    False
alienvaultotx_url_url                   Alien Vault OTX URL                     url                                     False
alienvaultotx_file_analysis             Alien Vault OTX File Analysis           hash, hash.sha256, hash.sha1            False
ransomwaretracker                       Ransomware Tracker                      fqdn                                    True
torexitnodes                            TOR Exit Nodes                          ipv4                                    False
vxstream_hash                           VxStream Sandbox Hash Lookup            hash, hash.sha1, hash.sha256            True
vxstream_ipv4_search                    VxStream Sandbox Host Search            ipv4                                    False
vxstream_domain_search                  VxStream Sandbox Domain Search          fqdn                                    True
vxstream_url_search                     VxStream Sandbox URL Search             fqdn                                    True
malware_domain_list                     Malware Domain List                     fqdn, ipv4                              True
hackertarget_mtr                        HackerTarget MTR                        fqdn, ipv4                              False
hackertarget_nping                      HackerTarget Nping                      fqdn, ipv4                              False
hackertarget_dig                        HackerTarget Dig                        fqdn                                    False
hackertarget_rdns                       HackerTarget RDns                       fqdn, ipv4                              False
hackertarget_hostsearch                 HackerTarget Host Search                fqdn                                    False
hackertarget_shareddns                  HackerTarget Shared NS                  fqdn                                    False
hackertarget_zonetransfer               HackerTarget Zone Transfer              fqdn                                    False
hackertarget_whois                      HackerTarget Whois                      fqdn, ipv4                              False
hackertarget_geoip                      HackerTarget GeoIP                      fqdn, ipv4                              False
hackertarget_rip                        HackerTarget Reverse IP                 fqdn, ipv4                              False
hackertarget_nmap                       HackerTarget Nmap                       fqdn, ipv4                              False
hackertarget_http_headers               HackerTarget Http Headers               fqdn, url, ipv4                         False
hackertarget_page_links                 HackerTarget Page Links                 fqdn, url, ipv4                         False
urlscan                                                     fqdn, url                               True
greynoise                               Grey Noise                              ipv4                                    True

Machinae is a great tool that saves a lot of time when gathering intel. If you find yourself needing to quickly lookup intel regularly Machinae can likely help with that.

To learn more about Machinae check it out over at HurricaneLabs Github page: