Machinae
One of the tools I use daily is Machinae. Machinae is a python command line tool that allows quick intel gathering on ips, domains, hashes and more. Developed in Python3 by HurricaneLabs Machinae makes quick work of searching and correlating intel sources.
Once installed using Machinae is as simple as typing its name followed by the object you want to search for. Machinae automatically determines the object type whether it be ip, domain, or hash. It then hits the sources for that specific object type. Below is an example of me running Machiane against this site.
$ machinae bizarrechaos.com
********************************************************************************
* Information for bizarrechaos.com
* Observable type: fqdn (Auto-detected: True)
********************************************************************************
[+] ipstack.com Results
[-] IP: 185.199.108.153
[-] Country: Netherlands
[-] Country Code: NL
[-] Latitude: 52.3824
[-] Longitude: 4.8995
[-] No URLVoid Results
[-] No W3bin Results
[+] McAfee Threat Results
[-] Web Risk: Unverified
[-] Last Seen: 2018-08-25
[+] Fortinet Category Results
[-] URL Category: Newly Observed Domain
[+] URL Unshorten Results
[-] Unshortened URL: https://bizarrechaos.com/
[+] Redirect Check Results
[-] Status Code: 301
[-] Status Code: 200
[-] No Malc0de Results
[+] VirusTotal Passive DNS Results
[-] Data: ('2018-08-25', '185.199.108.153')
[-] Data: ('2016-07-12', '192.30.252.153')
[-] No Cymon.io Domain Lookup Results
[+] PassiveTotal Passive DNS Results
[-] Data: vern.ns.cloudflare.com
[-] Data: alt1.gmr-smtp-in.l.google.com
[-] Data: dns@cloudflare.com
[-] Data: sysadmin@hambiscu.it
[-] Data: 104.28.31.93
[-] Data: 185.199.108.153
[-] Data: alt4.gmr-smtp-in.l.google.com
[-] Data: dns2.stabletransit.com
[-] Data: reza.ns.cloudflare.com
[-] Data: dns1.stabletransit.com
[-] Data: 166.78.174.253
[-] Data: alt2.gmr-smtp-in.l.google.com
[-] Data: 162.242.253.165
[-] Data: ns-cloud-b2.googledomains.com
[-] Data: gmr-smtp-in.l.google.com
[-] Data: alt3.gmr-smtp-in.l.google.com
[-] Data: ns-cloud-b1.googledomains.com
[-] Data: dns-admin@google.com
[-] Data: 192.30.252.154
[-] Data: ns-cloud-b3.googledomains.com
[-] Data: 104.28.30.93
[-] Data: ns-cloud-b1.googledomains.com
[-] Data: ns-cloud-b4.googledomains.com
[-] Query Value: bizarrechaos.com
[+] PassiveTotal Whois Results
[-] Updated: 2017-09-11T21:49:44.000-0700
[-] Domain: bizarrechaos.com
[-] Billing: {}
[-] Zone: {}
[-] Name Servers: ['ns-cloud-b1.googledomains.com', 'ns-cloud-b2.googledomains.com', 'ns-cloud-b3.googledomains.com', 'ns-cloud-b4.googledomains.com']
[-] Registered: 2013-10-11T11:15:39.000-0700
[-] Last Loaded At: 2017-12-08T18:49:18.214-0800
[-] Whois Server: whois.google.com
[-] Contact Email: ks2nmosgdrsh@contactprivacy.email
[-] Admin: ks2nmosgdrsh@contactprivacy.email
[-] Expires: 2018-10-11T11:15:39.000-0700
[-] Registrar: Google Inc.
[-] Tech: ks2nmosgdrsh@contactprivacy.email
[-] Registrant: ks2nmosgdrsh@contactprivacy.email
[-] No PassiveTotal Components Results
[-] No PassiveTotal Trackers Results
[-] No Ransomware Tracker Results
[-] No VxStream Sandbox Domain Search Results
[-] No VxStream Sandbox URL Search Results
[-] No Malware Domain List Results
[+] Urlscan.io Results
[-] report: https://urlscan.io/result/ec12653c-6c84-4489-9889-58a592d65ea7/
As you can see this gives us just about everything there is to know about bizarrechaos.com.
The main reason I enjoy Machinae is that it is super easy to extend the sources. Machinae uses a yaml configuration file to store the site data in. The default configuration contains plenty of good sources however I have probably tripled the number of sources in my own. If there is a site you use for intel gathering and it either has an API or can be scraped using regex then it most likely can be added to Machinae.
Here is an example of a site from my config file for a Json based API site.
urlscan:
name: Urlscan.io
default: True
otypes:
- fqdn
- url
json:
request:
url: 'https://urlscan.io/api/v1/scan/'
method: post
headers:
API-Key: APIKEY_HERE
Accept: application/json
data:
url: '{target}'
results:
- key: result
onlyif:
key: message
value: "Submission Successful"
pretty_name: report
As you can see when defining a site its as easy as naming the site, choosing what objects the site works on, and then defining the request and how to extract the results. You can also turn sites off with the default key, setting this to false would ensure it is not searched unless specified by name when running Machinae (machinae -s SITENAME OBJECT).
Here is an example of a site from my config file for a Webscraper based site.
fortinet_classify:
name: Fortinet Category
default: True
otypes:
- fqdn
- url
webscraper:
request:
url: 'http://www.fortiguard.com/webfilter?q={target}'
method: get
results:
- regex: 'Category:\s(.+)</h4>'
values:
- fortinet_category
pretty_name: URL Category
The layout is the exact same as the first example, however instead of “json” we are using “webscraper” and the results extraction is a bit different. In the results section here, rather than supplying the keys from the json object we want returned, we are giving a regex string that is used to extract the data we want from the page and setting it to a value.
As you can see after adding a lot of sources the output may get a bit long winded.
I suggest going through your sources and determining the best ones for you by default and set the others to default: False.
Also I commited a change to Machinae to add a --list-sites flag so you can display a table of all of your sites, the objects they run on, and if they are on by default. Here is what that looks like.
$ machinae --list-sites
SITE NAME OTYPES DEFAULT
ipstack ipstack.com ipv4, fqdn True
ipinfoio ipinfo.io ipv4, ipv6 True
urlvoid URLVoid fqdn True
w3bin W3bin fqdn True
mcafee_threat_domain McAfee Threat fqdn True
mcafee_threat_ip McAfee Threat ipv4 True
fortinet_classify Fortinet Category fqdn, url True
fraudguard FraudGuard ipv4 True
ipwhois IP Whois ipv4 True
spamhaus_ip Spamhaus Zen BL ipv4 False
spamhaus_domain Spamhaus Domain BL fqdn False
ipvoid IPVoid ipv4 False
toolsvoid_unshorten URL Unshorten fqdn, url False
unshorten.me URL Unshorten fqdn, url True
redirect_check Redirect Check fqdn, url True
malc0de Malc0de ipv4, fqdn, hash True
AbuseIPDB AbuseIPDB ipv4 True
sans SANS ipv4 True
telize Telize GeoIP ipv4 False
maxmind MaxMind GeoIP2 Precision ipv4 False
vt_ip VirusTotal Passive DNS ipv4 True
vt_domain VirusTotal Passive DNS fqdn True
vt_url VirusTotal URL Report url True
vt_hash VirusTotal File Report hash, hash.sha1, hash.sha256 True
reputation_authority Reputation Authority ipv4 False
threatexpert ThreatExpert hash True
vxvault VxVault hash True
projecthoneypot ProjectHoneypot ipv4 False
stopforumspam StopForumSpam email False
cymru_mhr Cymru MHR hash, hash.sha1, hash.sha256 True
icsi_notary ICSI Certificate Notary sslfp False
totalhash_ip TotalHash ipv4 False
domaintools_parsed_whois DomainTools Whois fqdn False
domaintools_reverse_whois DomainTools Reverse Whois email False
domaintools_reputation DomainTools Reputation fqdn False
dnsdb_ip Farsight DNSDB ipv4, ipv6 False
dnsdb_fqdn Farsight DNSDB fqdn False
cif Collective Intelligence Framework ipv4, fqdn, email, hash False
cymon_ip_events Cymon.io IP Events ipv4 True
cymon_ip_domains Cymon.io IP Domains ipv4 True
cymon_ip_urls Cymon.io IP URLs ipv4 True
cymon_domain Cymon.io Domain Lookup fqdn True
cymon_url Cymon.io URL Lookup url False
threatcrowd_ip_report ThreatCrowd IP Report ipv4 True
passivetotal_pdns PassiveTotal Passive DNS fqdn, ipv4 True
passivetotal_whois PassiveTotal Whois fqdn True
passivetotal_sslcert PassiveTotal SSL Certificate History ipv4 True
passivetotal_components PassiveTotal Components fqdn True
passivetotal_trackers PassiveTotal Trackers fqdn True
shodan Shodan ipv4 True
alienvaultotx_ipv4_geo Alien Vault OTX Geolocation ipv4 False
alienvaultotx_ipv4_malware Alien Vault OTX Malware Callback ipv4 False
alienvaultotx_ipv4_url Alien Vault OTX URL ipv4 False
alienvaultotx_ipv4_passive_dns Alien Vault OTX Passive DNS ipv4 False
alienvaultotx_ipv4_http_meta Alien Vault OTX HTTP Meta Data ipv4 False
alienvaultotx_ipv6_geo Alien Vault OTX Geolocation ipv6 False
alienvaultotx_ipv6_malware Alien Vault OTX Malware Callback ipv6 False
alienvaultotx_ipv6_url Alien Vault OTX URL ipv6 False
alienvaultotx_ipv6_passive_dns Alien Vault OTX Passive DNS ipv6 False
alienvaultotx_domain_geo Alien Vault OTX Geolocation fqdn False
alienvaultotx_domain_malware Alien Vault OTX Malware Callback fqdn False
alienvaultotx_domain_url Alien Vault OTX URL fqdn False
alienvaultotx_domain_passive_dns Alien Vault OTX Passive DNS fqdn False
alienvaultotx_domain_http_meta Alien Vault OTX HTTP Meta Data fqdn False
alienvaultotx_domain_whois Alien Vault OTX Whois fqdn False
alienvaultotx_url_url Alien Vault OTX URL url False
alienvaultotx_file_analysis Alien Vault OTX File Analysis hash, hash.sha256, hash.sha1 False
ransomwaretracker Ransomware Tracker fqdn True
torexitnodes TOR Exit Nodes ipv4 False
vxstream_hash VxStream Sandbox Hash Lookup hash, hash.sha1, hash.sha256 True
vxstream_ipv4_search VxStream Sandbox Host Search ipv4 False
vxstream_domain_search VxStream Sandbox Domain Search fqdn True
vxstream_url_search VxStream Sandbox URL Search fqdn True
malware_domain_list Malware Domain List fqdn, ipv4 True
hackertarget_mtr HackerTarget MTR fqdn, ipv4 False
hackertarget_nping HackerTarget Nping fqdn, ipv4 False
hackertarget_dig HackerTarget Dig fqdn False
hackertarget_rdns HackerTarget RDns fqdn, ipv4 False
hackertarget_hostsearch HackerTarget Host Search fqdn False
hackertarget_shareddns HackerTarget Shared NS fqdn False
hackertarget_zonetransfer HackerTarget Zone Transfer fqdn False
hackertarget_whois HackerTarget Whois fqdn, ipv4 False
hackertarget_geoip HackerTarget GeoIP fqdn, ipv4 False
hackertarget_rip HackerTarget Reverse IP fqdn, ipv4 False
hackertarget_nmap HackerTarget Nmap fqdn, ipv4 False
hackertarget_http_headers HackerTarget Http Headers fqdn, url, ipv4 False
hackertarget_page_links HackerTarget Page Links fqdn, url, ipv4 False
urlscan Urlscan.io fqdn, url True
greynoise Grey Noise ipv4 True
Machinae is a great tool that saves a lot of time when gathering intel. If you find yourself needing to quickly lookup intel regularly Machinae can likely help with that.
To learn more about Machinae check it out over at HurricaneLabs Github page: https://github.com/HurricaneLabs/machinae