Search Tools API
The Netlas Search Tools API allows you to perform search queries across any Netlas data collection from third-party applications and scripts.
Before you start building your integration, you need to decide which data you want to retrieve. It's important to distinguish between retrieving data for individual hosts and data for search queries.
- Host API: This specific endpoint in the Netlas API is designed to retrieve detailed information about a single IP address or domain. It is ideal for targeted queries where aggregated data about a particular host is required.
- Search Query API: This set of endpoints within the Netlas API enables complex and diverse queries across different parameters and data indices. These endpoints are suited for broader searches, utilizing Search Query Language.
| Host API | Search Query API |
|---|---|
| Search by IP or domain only | Complex queries with various search arguments |
| Returns specific data for an IP or domain | Returns multiple results based on the query |
| Aggregated data from all data collections | Specific data from chosen data collections or indices |
| Corresponds to IP/Domain Info tool | Corresponds to Responses, DNS, WHOIS, SSL certs search |
Use Cases for Host API
- SIEM System Plugin: Automatically enrich security logs in any SIEM system with detailed IP/domain data from Netlas, providing contextual insights for security events.
- Firewall Management Plugin: Develop a plugin for firewall solutions that utilizes Netlas data to automate security rule creation based on IP reputation, known threats, and geolocation.
- CMS Plugin: Identify a site visitor’s company by IP address for lead generation, analytics, or content adaptation.
- Remote Employee Compliance Checks: Verify the IP address information of remote employees to ensure they connect from secure locations.
- Browser Plugin: Access security-related information about web resources and their hosting servers while browsing.
Use Cases for Search Query API
- Reconnaissance Script: Automate the collection and analysis of external attack surface data to understand the target environment.
- Non-intrusive Assessment: Gather information about a system's security posture without direct interaction, avoiding detection.
- OSINT Profiling: Collect publicly available information to create detailed profiles of organizations or systems.
- Malware Resources Search: Identify and track potential threats by searching for online resources that host malware.
- Phishing Resources Search: Identify and analyze websites used in phishing campaigns to help protect against fraudulent activities.
Host API
You can fetch data for any valid IP address or domain name. This API endpoint aggregates data from all available data collections related to the queried IP or domain.
Query using Python SDK
import netlas
import json
# Define the API key and initialize the Netlas client
netlas_connection = netlas.Netlas(api_key="YOUR_API_KEY")
# IP query example
ip_info = netlas_connection.host(host="135.125.237.168")
print(json.dumps(ip_info))
# Domain query example
domain_info = netlas_connection.host(host="app.netlas.io")
print(json.dumps(domain_info))
A JSON document with the following data will be returned:
| IP query | Domain query |
|---|---|
| PTR, Subnet and Autonomous system data | DNS records |
| Geolocation | Related domains |
| Privacy flags (VPN/Tor/Proxy) | Domain WHOIS data |
| Organization | Threat intelligence data |
| IP WHOIS data | Exposed WEB ports & software, including detected CVE |
| Related domains | |
| Threat intelligence data | |
| Exposed ports & software, including detected CVE |
JSON documents with IP/domain information
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 | |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 | |
Filtering Output
You can limit the amount of output data by using the include option to specify a list of fields to be included in the response, or exclude to filter out certain fields. However, both options cannot be used simultaneously in the same request.
Filtering the output with query options
import netlas
import json
# Define the API key and initialize the Netlas client
netlas_connection = netlas.Netlas(api_key="YOUR_API_KEY")
domain_info = netlas_connection.host(
host="google.com",
fields="whois.registrant.organization"
)
print(json.dumps(domain_info))
Use jq utility to filter CLI tool and curl output.
Both filtering methods can also be used for the Search Query API.
YAML Output
By default, the CLI tool outputs data in YAML format, which is more readable for humans. Additionally, Netlas employs the Pygments library for YAML syntax highlighting. However, using YAML in automation scripts is not recommended for two main reasons:
- Complex Data Parsing: Extracting specific data from YAML is more challenging compared to JSON due to its structured complexity.
- Additional Formatting Handling: ANSI escape codes used for formatting and color in the CLI output must be filtered out, adding an extra step in data processing.
This is why we use the --format json option with the CLI tool in the examples provided. If you still need to utilize YAML output, you can strip ANSI escape characters using the sed utility:
Filtering CLI tool formated YAML output
Explanation:
netlas count "host:135.125.237.168" | sed 's/\x1b\[[0-9;]*m//g'
\x1b: This is the escape character, indicating the start of an ANSI sequence.\[[0-9;]*m: This pattern matches the sequence starting with[, followed by any combination of digits and semicolons, and ending withm, which is typical for color and style escape codes.
Search Query API
Each Netlas Search tool is associated with a set of API endpoints that facilitate different search operations:
/api/[search_tool]/— This endpoint allows you to make search queries, returning up to 20 results per request./api/[search_tool]/download/— Fetches search results in a stream format. You must specify thesizeof the stream as a parameter in the request./api/[search_tool]_count/— Calculates the total number of results for a specific query. If there are fewer than 1000 results, the method returns the exact count. For counts exceeding 1000 results, the number is estimated with a small margin of error not exceeding 3%./api/[search_tool]_facet/— Enables facet searches, which organize search results into groups based on specified criteria.
A complete list of available endpoints is detailed in the API specification.
Regardless of the specific search tool being used, endpoints of the same category exhibit consistent behavior. Both the Netlas SDK and the CLI Tool provide corresponding methods and commands to interact with these endpoints effectively.
Responses available for the query, Search command
#!/bin/bash
# Define the API key
API_KEY="YOUR_API_KEY"
# Define the query
QUERY="host:135.125.237.168"
# Get the count of responses for the given query
COUNT=$(netlas count "$QUERY" --apikey "$API_KEY" --format json \
| jq ".count")
# Check if there are any responses and search for them if there are
if [ "$COUNT" -gt 0 ]; then
echo "Responses for $QUERY"
# Calculate the number of pages needed based on 20 results per page
NUM_PAGES=$((($COUNT + 19) / 20))
# Loop through each page and fetch the results
for ((PAGE=0; PAGE<NUM_PAGES; PAGE++)); do
RESULTS=$(netlas search "$QUERY" --datatype "response" \
--page "$PAGE" --include "uri" \
--apikey "$API_KEY" --format json)
# Check if items exist in the results and print URIs
if [[ $RESULTS == *"items"* ]]; then
echo "$RESULTS" | jq -r '.items[].data.uri'
fi
# Wait for a second to avoid throttling
sleep 1
done
else
echo "No responses found."
fi
Responses available for the query, Search endpoint
#!/bin/bash
# Define the API key and the base URL for the Netlas API
API_KEY="YOUR_API_KEY"
BASE_URL="https://app.netlas.io/api"
# Define the query
QUERY="host:135.125.237.168"
# Get the count of responses for the given query
COUNT=$(curl -s -H "X-API-Key: $API_KEY" \
"$BASE_URL/responses_count/?q=$QUERY" \
| jq ".count")
# Check if there are any responses and search for them if there are
if [ "$COUNT" -gt 0 ]; then
echo "Responses for $QUERY"
# Calculate the number of pages needed based on 20 results per page
NUM_PAGES=$((($COUNT + 19) / 20))
# Loop through each page and fetch the results
for ((PAGE=0; PAGE<NUM_PAGES; PAGE++)); do
OFFSET=$(($PAGE*20))
RESULTS=$(curl -s -H "X-API-Key: $API_KEY" \
"$BASE_URL/responses/?q=$QUERY&start=$OFFSET&fields=uri" \
| jq -r ".items[].data.uri")
# Check if items exist in the results and print URIs
if [ -n "$RESULTS" ]; then
echo "$RESULTS"
fi
# Wait for a second to avoid throttling
sleep 1
done
else
echo "No responses found."
fi
Responses available for the query, Search method
import netlas
import time
# Define the API key and initialize the Netlas client
netlas_connection = netlas.Netlas(api_key='YOUR_API_KEY')
# Define the query
query = "host:135.125.237.168"
# Get the count of responses for the given query
cnt_of_res = netlas_connection.count(query=query, datatype='response')
# Check if there are any responses and search for them if there are
if cnt_of_res['count'] > 0:
print("Responses for " + query)
# Calculate the number of pages needed based on 20 results per page
num_pages = (cnt_of_res['count'] + 19) // 20 # rounding up
# Loop through each page and fetch the results
for page in range(num_pages):
search_results = netlas_connection.search(
query=query, datatype='response', page=page, fields='uri'
)
if 'items' in search_results:
# iterate over data and print: IP address, port, path and protocol
for response in search_results['items']:
print(response['data']['uri'])
# wait for a second to avoid throtling
time.sleep(1)
else:
print("No responses found.")
Search vs. Download
The Search and Download methods in the Netlas API cater to different needs, primarily based on the volume of data required:
-
Search Method: Depending on your pricing plan, you can retrieve between 200 and 4000 search results. This method requires making a separate request for every 20 results, which is ideal for situations where pagination is needed or when the expected number of results is relatively low. This approach is used in our web application to support user interfaces that include pagination.
-
Download Method: Unlike the Search method, the Download method is not technically limited to a specific number of results per request. Instead, it is governed only by your pricing plan, allowing for the retrieval of large data sets in a single request. This method is particularly useful for handling extensive search queries that yield a large number of results, eliminating the need for repeated requests with incremental offsets. The Netlas SDK and CLI tool additionally include a
download_all()method and an--allkey that allow you to query all available results.
Here's an example illustrating the simplicity and conciseness of using the Download method:
Responses available for the query, download endpoint
#!/bin/bash
# Define the API key and the base URL for the Netlas API
API_KEY="YOUR_API_KEY"
BASE_URL="https://app.netlas.io/api"
# Define the query
QUERY="host:*.netlas.io"
# Get the count of responses for the given query
COUNT=$(curl -s -H "X-API-Key: $API_KEY" \
"$BASE_URL/responses_count/?q=$QUERY" \
| jq ".count")
# Check if there are any responses and download them if there are
if [ "$COUNT" -gt 0 ]; then
curl -s -X POST "$BASE_URL/responses/download/" \
-H "Content-Type: application/json" \
-H "X-API-Key: $API_KEY" \
-d '{
"q": "'"${QUERY}"'",
"fields": ["uri"],
"source_type": "include",
"size": '"${COUNT}"'
}' | jq -r ".[].data.uri"
fi
Responses available for the query, download_all() method
import netlas
import json
# Define the API key and initialize the Netlas client
netlas_connection = netlas.Netlas(api_key='YOUR_API_KEY')
# Define the query
query = "host:*.netlas.io"
# Download all available responses
for resp in netlas_connection.download_all(query):
# decode from binary stream
response = json.loads(resp.decode('utf-8'))
print(response['data']['uri'])
The code utilizing the Download method is simpler and more concise because it eliminates the need for looped requests with offsets; all results are obtained in a single request.
How to download large sets of search results that exceed download limits? Read the FAQ →
Indices
By default, queries target the most up-to-date index. However, other indices may also be available to you. Utilizing indices through the API is analogous to using indices in the web application.
Available responses indices
#!/bin/bash
# Define the API key and the base URL for the Netlas API
API_KEY="YOUR_API_KEY"
BASE_URL="https://app.netlas.io/api"
# Fetch the indices and filter the results
curl -s -H "X-API-Key: $API_KEY" "$BASE_URL/indices/" \
| jq -r '.[] | select(.type | contains("responses")) | "\(.id): \(.name)"' \
| sort -g
Available responses indices
import netlas
# Define the API key and initialize the Netlas client
api_key = 'YOUR_API_KEY'
netlas_connection = netlas.Netlas(api_key=api_key)
# Fetch the indices
indices = netlas_connection.indices()
# Filter indices where the type contains 'responses' and sort by id
filtered_indices = [index for index in indices if 'responses' in index['type']]
sorted_indices = sorted(filtered_indices, key=lambda x: x['id'])
# Print the results
for index in sorted_indices:
print(f"{index['id']}: {index['name']}")
One of the examples provided below demonstrates how to perform a historical search and compare results from different indices.
Examples
The examples below assume that the API key has been saved by the user, as described in the SDK installation instructions.
IP to Company
In the example below, the Host API is used to query for the organization name and country associated with IP addresses.
ip2company-cli.sh
#!/bin/bash
# Check if an IP address is provided
if [ "$#" -ne 1 ]; then
echo "Usage: $0 <IP_ADDRESS>"
exit 1
fi
# Make the API call using CLI tool, assuming that API key is saved
response=$(netlas host "$1" --format json)
# Extract organization and country using jq
organization=$(echo $response | jq -r '.organization // "n/a"')
country=$(echo $response | jq -r '.geo.country // "n/a"')
# Conditional output based on data availability
if [ "$organization" = "n/a" ] && [ "$country" = "n/a" ]; then
echo "No organization information found for the IP: $1"
else
echo "$organization, $country"
fi
ip2company-curl.sh
#!/bin/bash
# Check if an IP address is provided
if [ "$#" -ne 1 ]; then
echo "Usage: $0 <IP_ADDRESS>"
exit 1
fi
# Define the API key and the URL for query
API_KEY="YOUR_API_KEY"
API_URL="https://app.netlas.io/api/host/$1/"
# Make the API call using curl with the API key in the header
response=$(curl -s -H "X-API-Key: $API_KEY" "$API_URL")
# Extract organization and country using jq
organization=$(echo $response | jq -r '.organization // "n/a"')
country=$(echo $response | jq -r '.geo.country // "n/a"')
# Conditional output based on data availability
if [ "$organization" = "n/a" ] && [ "$country" = "n/a" ]; then
echo "No organization information found for the IP: $1"
else
echo "$organization, $country"
fi
ip2company.py
import sys
import netlas
# Check if an IP address is provided
if len(sys.argv) < 2:
print("Usage: python ip2company.py <IP_ADDRESS>")
sys.exit(1)
# Initialize the Netlas API assuming that API key is saved
api_key = netlas.helpers.get_api_key()
netlas_api = netlas.Netlas(api_key=api_key)
# Make the API call to get host information
try:
response = netlas_api.host(host=sys.argv[1])
# Check if the response contains the necessary data
if response and 'organization' in response:
organization_name = response.get('organization', 'n/a')
country = response.get('geo', {}).get('country', 'n/a')
# Default to 'n/a' if not available
print(f"{organization_name}, {country}")
else:
print("No organization information found for the IP:", sys.argv[1])
except Exception as e:
print("An error occurred:", str(e))
Abuse Contact
This script retrieves the abuse contact for a domain or IP address using the Host API. For domains, it uses the registrar's email, and for IP addresses, it retrieves the abuse email from the IP WHOIS data.
Please note, data availability depends on your pricing plan
If your pricing plan does not include access to contact information such as phone numbers and email addresses, this example will not be applicable.
abuse_contact-cli.sh
#!/bin/bash
# Check if a domain or IP address is provided
if [ "$#" -ne 1 ]; then
echo "Usage: $0 <DOMAIN_OR_IP>"
exit 1
fi
# Make the API call using CLI tool, assuming that API key is saved
response=$(netlas host $1 --format json)
# Determine the type and extract the relevant abuse contact
if echo "$response" | jq -e '.ip' > /dev/null; then
abuse_contact=$(echo "$response" | jq -r '.whois.abuse // "n/a"')
elif echo "$response" | jq -e '.domain' > /dev/null; then
abuse_contact=$(echo "$response" | jq -r '.whois.registrar.email // "n/a"')
else
abuse_contact="n/a"
fi
echo "$abuse_contact"
~ # bash abuse_contact-cli.sh dns.google
[email protected]
~ # bash abuse_contact-cli.sh 8.8.8.8
[email protected]
abuse_contact-curl.sh
#!/bin/bash
# Check if a domain or IP address is provided
if [ "$#" -ne 1 ]; then
echo "Usage: $0 <DOMAIN_OR_IP>"
exit 1
fi
# Define the API key and the URL for query
API_KEY="YOUR_API_KEY"
API_URL="https://app.netlas.io/api/host/$1/"
# Make the API call using curl with the API key in the header
response=$(curl -s -H "X-API-Key: $API_KEY" "$API_URL")
# Determine the type and extract the relevant abuse contact
if echo "$response" | jq -e '.ip' > /dev/null; then
abuse_contact=$(echo "$response" | jq -r '.whois.abuse // "n/a"')
elif echo "$response" | jq -e '.domain' > /dev/null; then
abuse_contact=$(echo "$response" | jq -r '.whois.registrar.email // "n/a"')
else
abuse_contact="n/a"
fi
echo "$abuse_contact"
~ # bash abuse_contact-curl.sh dns.google
[email protected]
~ # bash abuse_contact-curl.sh 8.8.8.8
[email protected]
abuse_contact.py
import sys
import netlas
# Check if a domain or IP address is provided
if len(sys.argv) < 2:
print("Usage: python abuse_contact.py <DOMAIN_OR_IP>")
sys.exit(1)
# Initialize the Netlas API, assuming that API key is saved
api_key = netlas.helpers.get_api_key()
netlas_api = netlas.Netlas(api_key=api_key)
# Make the API call to get host information
try:
response = netlas_api.host(host=sys.argv[1])
# Determine if the identifier is a domain or IP and extract the relevant abuse contact
if 'domain' in response and 'whois' in response:
if 'registrar' in response['whois'] and 'email' in response['whois']['registrar']:
abuse_contact = response['whois']['registrar']['email']
else:
abuse_contact = 'n/a'
elif 'ip' in response and 'whois' in response:
abuse_contact = response['whois']['abuse'] if 'abuse' in response['whois'] else 'n/a'
else:
abuse_contact = 'n/a'
print(abuse_contact)
except Exception as e:
print("An error occurred:", str(e))
~ # python abuse_contact.py dns.google
[email protected]
~ # python abuse_contact.py 8.8.8.8
[email protected]
Subdomain Enumeration
This example leverages the Search Query API to access the DNS Registry data collection. It performs a search based on user input and downloads all available results.
subdomains-cli.sh
#!/bin/bash
# Check if a domain seqrch query is provided
if [ "$#" -ne 1 ]; then
printf "Usage:\n\t$0 \'<*.domain.com>\'\n\t$0 \'<domain.*>\'\n"
exit 1
fi
# Download all available results
netlas download "domain:$1" --all --datatype domain --include "domain" \
| jq -r ".data.domain"
subdomains-curl.sh
#!/bin/bash
# Check if a domain seqrch query is provided
if [ "$#" -ne 1 ]; then
printf "Usage:\n\t$0 \'<*.domain.com>\'\n\t$0 \'<domain.*>\'\n"
exit 1
fi
# Define the API key and the base URL for the Netlas API
API_KEY="YOUR_API_KEY"
BASE_URL="https://app.netlas.io/api"
# Get the count of domains for the given query
COUNT=$(curl -s -H "X-API-Key: $API_KEY" \
"${BASE_URL}/domains_count/?q=domain:$1" | jq ".count")
# Check if there are any results and download them if there are
if [ "$COUNT" -gt 0 ]; then
curl -s -X POST "$BASE_URL/domains/download/" \
-H "Content-Type: application/json" \
-H "X-API-Key: $API_KEY" \
-d '{
"q": "domain:'"${1}"'",
"fields": ["domain"],
"source_type": "include",
"size": '"${COUNT}"'
}' | jq -r ".[].data.domain"
fi
subdomains.py
import sys
import netlas
import json
# Check if a domain seqrch query is provided
if len(sys.argv) != 2:
print("Usage:\n" +
"\tpython subdomains.py '<*.domain.com>'\n" +
"\tpython subdomains.py '<domain.*>'"
)
sys.exit(1)
# Initialize the Netlas API assuming that API key is saved
api_key = netlas.helpers.get_api_key()
netlas_api = netlas.Netlas(api_key=api_key)
# Define the query
query = "domain:" + sys.argv[1]
try:
# Download all available results
for resp in netlas_api.download_all(
query=query,
datatype='domain',
fields='domain'
):
# decode from binary stream
response = json.loads(resp.decode('utf-8'))
print(response['data']['domain'])
except Exception as e:
print("An error occurred:", str(e))
Always choose the download method to fetch a large amount of results.
This example uses the Download method to accommodate potentially large datasets that could exceed the 4000 result limit imposed by the Search method.
Related Domains
This script takes a domain as input, queries Domain WHOIS data for the organization name of a registrant, and searches for domains with the same organization name. If the number of search results exceeds 1,000, user confirmation is required to proceed with the download. Use -f or --force to bypass the confirmation dialogue.
related_domains-cli.sh
#!/bin/bash
# Check if a domain is provided
if [ "$#" -lt 1 ]; then
echo "Usage: $0 <DOMAIN> [-f|--force]"
echo " <DOMAIN> - the domain to query"
echo " -f, --force - force downloading all results without prompt, even if results are more than 1000"
exit 1
fi
force_download=false
if [ "$#" -eq 2 ] && ([[ "$2" == "-f" ]] || [[ "$2" == "--force" ]]); then
force_download=true
fi
# Fetch organization name from WHOIS data
organization=$(netlas search "domain:$1" \
--format json --datatype whois-domain \
| jq -r '.items[0].data.registrant.organization')
if [ "$organization" == "null" ] || [ -z "$organization" ]; then
echo "No organization found for the domain."
exit 1
fi
# Create query
domains_query="registrant.organization:\"$organization\""
#Count domains with the same organization name
count=$(netlas count "$domains_query" \
--format json --datatype whois-domain | jq ".count")
# Check if there are any results and optionally ask for confirmation
if [ "$count" -eq 0 ]; then
echo "No related domains found."
exit 1
elif [ "$count" -gt 1000 ] && ! $force_download; then
echo "Found $count results for organization: $organization."
read -p "Do you really want to download all results? (Y/n): " user_input
if [[ "$user_input" != "Y" && "$user_input" != "y" ]]; then
echo "Download canceled."
exit 0
fi
fi
# Fetching domains registered to the same organization
netlas download "$domains_query" \
--datatype whois-domain --include "domain" \
--count "$count" \
| jq -r ".data.domain"
related_domains-curl.sh
#!/bin/bash
# Check if a domain is provided
if [ "$#" -lt 1 ]; then
echo "Usage: $0 <DOMAIN> [-f|--force]"
echo " <DOMAIN> - the domain to query"
echo " -f, --force - force downloading all results without prompt, even if results are more than 1000"
exit 1
fi
force_download=false
if [ "$#" -eq 2 ] && ([[ "$2" == "-f" ]] || [[ "$2" == "--force" ]]); then
force_download=true
fi
# Define the API key and the base URL for the Netlas API
API_KEY="YOUR_API_KEY"
BASE_URL="https://app.netlas.io/api"
# Fetch organization name from WHOIS data
org_response=$(curl -s -H "X-API-Key: $API_KEY" \
"$BASE_URL/whois_domains/?q=domain:$1")
organization=$(echo "$org_response" \
| jq -r '.items[0].data.registrant.organization')
if [ "$organization" == "null" ] || [ -z "$organization" ]; then
echo "No organization found for the domain."
exit 1
fi
# Create query with escaped quotes for download endpoint and URL-encoded query
domains_query="registrant.organization:\\\""$organization"\\\""
domains_query_encoded=$(echo -n "registrant.organization:\""$organization"\"" \
| jq -Rr @uri)
#Count domains with the same organization name
count_response=$(curl -s -H "X-API-Key: $API_KEY" \
"$BASE_URL/whois_domains_count/?q=$domains_query_encoded")
count=$(echo "$count_response" | jq ".count")
# Check if there are any results and optionally ask for confirmation
if [ "$count" -eq 0 ]; then
echo "No related domains found."
exit 1
elif [ "$count" -gt 1000 ] && ! $force_download; then
echo "Found $count results for organization: $organization."
read -p "Do you really want to download all results? (Y/n): " user_input
if [[ "$user_input" != "Y" && "$user_input" != "y" ]]; then
echo "Download canceled."
exit 0
fi
fi
# Fetching domains registered to the same organization
curl -s -X POST "$BASE_URL/whois_domains/download/" \
-H "Content-Type: application/json" \
-H "X-API-Key: $API_KEY" \
-d '{
"q": "'"${domains_query}"'",
"fields": ["domain"],
"source_type": "include",
"size": '"${count}"'
}' | jq -r ".[].data.domain"
related_domains.py
import sys
import netlas
import json
# Check if a domain and optionally a force flag are provided
if len(sys.argv) < 2 or (len(sys.argv) == 3 and sys.argv[2] not in ['-f', '--force']):
print("Usage: python related_domains.py <DOMAIN> [-f|--force]")
print("\t<DOMAIN> - the domain to query")
print("\t-f, --force - force downloading all results without prompt, even if results are more than 1000")
sys.exit(1)
force_download = '-f' in sys.argv or '--force' in sys.argv
# Initialize the Netlas API assuming that API key is saved
api_key = netlas.helpers.get_api_key()
netlas_api = netlas.Netlas(api_key=api_key)
# Get registrant.organization by domain name
try:
org_query = "domain:" + sys.argv[1]
response = netlas_api.search(query=org_query, datatype='whois-domain')
organization = response['items'][0]['data']['registrant']['organization']
except Exception as e:
print("An error occurred:", str(e))
sys.exit(2)
# Fetching domains registered to the same organization
try:
# Get the count of domains
domains_query = 'registrant.organization:"' + organization + '"'
cnt_of_res = netlas_api.count(query=domains_query, datatype='whois-domain')
# Check if there are any results and optionally ask for confirmation
if cnt_of_res['count'] > 0:
if cnt_of_res['count'] > 1000 and not force_download:
print(f"Found {cnt_of_res['count']} results for organization: {organization}.")
user_input = input("Do you really want to download all results? (Y/n): ")
if user_input != 'Y':
print("Download canceled.")
sys.exit(0)
# Downloading domains
for resp in netlas_api.download(
query=domains_query,
size=cnt_of_res['count'],
datatype='whois-domain',
fields='domain'
):
# decode from binary stream
response = json.loads(resp.decode('utf-8'))
print(response['data']['domain'])
else:
print("No results for the given domain.")
except Exception as e:
print("An error occurred:", str(e))
Please note, data availability depends on your pricing plan
The Domain WHOIS Search tool is only available on paid pricing plans, and exceeding the allowed number of results may result in an error.
Exposed Ports History
This script uses the Netlas Search API to retrieve and display open ports for a specified IP address across indices available to the user, sorted in historical order.
ip_history-cli.sh
#!/bin/bash
# Check if an IP address is provided
if [ "$#" -lt 1 ]; then
echo "Usage: bash ip_history.sh <IP_ADDRESS>"
exit 1
fi
# Define the query
query="host:$1"
# Fetch the indices and sort by ID
indices=$(netlas indices --format json \
| jq '.[] | select(.type | contains("responses")) | {id, name}' \
| jq -s 'sort_by(.id)')
# Iterate through indices to fetch exposed ports data
echo "$indices" | jq -c '.[]' | while read -r idx; do
index_id=$(echo "$idx" | jq -r '.id')
index_name=$(echo "$idx" | jq -r '.name')
# Get count of results
cnt_of_res=$(netlas count "$query" --indices "$index_id" --format json \
| jq '.count')
# Check if there are any results and download them if there are
if [ "$cnt_of_res" -gt 0 ]; then
printf '%s\t' "Index #$index_id - $index_name"
responses=$(netlas download "$query" \
--include "port,protocol" --count "$cnt_of_res" \
--indices "$index_id" | jq -c '.[]')
exposed_ports=()
while read -r resp; do
port=$(echo "$resp" | jq -r '.port')
protocol=$(echo "$resp" | jq -r '.protocol')
# Check for uniqueness and add to array if not already present
if [[ ! " ${exposed_ports[@]} " =~ " "${port}/${protocol}" " ]]; then
exposed_ports+=("${port}/${protocol}")
fi
done <<< "$responses" # Pass responses as input to the while loop
# Sort and output data
sorted_ports=$(printf '%s\n' "${exposed_ports[@]}" | sort -t '/' -k1,1n -k2,2)
echo $(tr '\n' ' ' <<< "$sorted_ports")
fi
# Wait a second to avoid throttling
sleep 1
done
~ # bash ip_history-cli.sh 1.1.1.1
Index #83 - responses-2022-09-21: 53/dns 80/http 443/https
Index #84 - responses-2023-01-03: 53/dns 80/http 443/https 2095/http 8080/http
Index #86 - responses_*_2023-05-11: 53/dns 53/dns_tcp 80/http 443/https 2095/http 8080/http
Index #89 - responses_*_2023-09-22: 53/dns 53/dns_tcp 80/http 443/https 2095/http 8080/http
Index #103 - responses_*_2024-01-22: 53/dns 53/dns_tcp 80/http 443/https 2095/http 8080/http
Index #110 - responses_*_2024-04-25: 53/dns 53/dns_tcp 80/http 443/https 2095/http 2096/https 8080/http 8443/http 8443/https
ip_history-curl.sh
#!/bin/bash
# Check if an IP address is provided
if [ "$#" -lt 1 ]; then
echo "Usage: bash ip_history.sh <IP_ADDRESS>"
exit 1
fi
# Define the API key and the base URL for the Netlas API
API_KEY="YOUR_API_KEY"
BASE_URL="https://app.netlas.io/api"
# Define the query
query="host:$1"
# Fetch the indices and sort by ID
indices=$(curl -s "$BASE_URL/indices/" \
| jq '.[] | select(.type | contains("responses")) | {id, name}' \
| jq -s 'sort_by(.id)')
# Iterate through indices to fetch exposed ports data
echo "$indices" | jq -c '.[]' | while read -r idx; do
index_id=$(echo "$idx" | jq -r '.id')
index_name=$(echo "$idx" | jq -r '.name')
# Get count of results
cnt_of_res=$(curl -s -H "X-API-Key: $API_KEY" \
"$BASE_URL/responses_count/?q=$query&indices=$index_id" | jq '.count')
# Check if there are any results and download them if there are
if [ "$cnt_of_res" -gt 0 ]; then
printf '%s\t' "Index #$index_id - $index_name"
responses=$(curl -s -X POST "$BASE_URL/responses/download/" \
-H "X-API-Key: $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"q": "'"$query"'",
"fields": ["port","protocol"],
"source_type": "include",
"size": '"${cnt_of_res}"',
"indices": "'"$index_id"'"
}')
exposed_ports=()
responses=$(echo "$responses" | jq -c '.[]')
while read -r resp; do
port=$(echo "$resp" | jq -r '.data.port')
protocol=$(echo "$resp" | jq -r '.data.protocol')
# Check for uniqueness and add to array if not already present
if [[ ! " ${exposed_ports[@]} " =~ " "${port}/${protocol}" " ]]; then
exposed_ports+=("${port}/${protocol}")
fi
done <<< "$responses" # Pass responses as input to the while loop
# Sort and output data
sorted_ports=$(printf '%s\n' "${exposed_ports[@]}" \
| sort -t '/' -k1,1n -k2,2)
echo $(tr '\n' ' ' <<< "$sorted_ports")
fi
# Wait a second to avoid throttling
sleep 1
done
~ # bash ip_history-curl.sh 1.1.1.1
Index #83 - responses-2022-09-21: 53/dns 80/http 443/https
Index #84 - responses-2023-01-03: 53/dns 80/http 443/https 2095/http 8080/http
Index #86 - responses_*_2023-05-11: 53/dns 53/dns_tcp 80/http 443/https 2095/http 8080/http
Index #89 - responses_*_2023-09-22: 53/dns 53/dns_tcp 80/http 443/https 2095/http 8080/http
Index #103 - responses_*_2024-01-22: 53/dns 53/dns_tcp 80/http 443/https 2095/http 8080/http
Index #110 - responses_*_2024-04-25: 53/dns 53/dns_tcp 80/http 443/https 2095/http 2096/https 8080/http 8443/http 8443/https
ip_history.py
import sys
import time
import netlas
import json
# Check if an IP address is provided
if len(sys.argv) < 2:
print("Usage: python ip_history.py <IP_ADDRESS>")
sys.exit(1)
# Initialize the Netlas API assuming that API key is saved
api_key = netlas.helpers.get_api_key()
netlas_api = netlas.Netlas(api_key=api_key)
# Define the query
query = "host:" + sys.argv[1]
try:
# Fetch the indices
indices = netlas_api.indices()
# Filter indices where the type contains 'responses' and sort by id
filtered_indices = [index for index in indices if 'responses' in index['type']]
sorted_indices = sorted(filtered_indices, key=lambda x: x['id'])
# Iterating though indices to fetch exposed ports data
for index in sorted_indices:
cnt_of_res = netlas_api.count(query=query, indices=str(index['id']))
# Check if there are any results and download them if there are
if cnt_of_res['count'] > 0:
exposed_ports = []
print(f"Index #{index['id']} - {index['name']}:", end="\t")
for resp in netlas_api.download(
query=query, fields='port,protocol',
size=cnt_of_res['count'], indices=str(index['id'])
):
response = json.loads(resp.decode('utf-8'))
# Fill a list with unique ports
port = f"{response['data']['port']}/{response['data']['protocol']}"
if port not in exposed_ports:
exposed_ports.append(port)
# Sort and output data
sorted_ports = sorted(
exposed_ports,
key=lambda x: (int(x.split('/')[0]), x.split('/')[1])
)
print(' '.join(sorted_ports))
# Wait a second to avoid throttling
time.sleep(1)
except Exception as e:
print("An error occurred:", str(e))
~ # python ip_history.py 1.1.1.1
Index #83 - responses-2022-09-21: 53/dns 80/http 443/https
Index #84 - responses-2023-01-03: 53/dns 80/http 443/https 2095/http 8080/http
Index #86 - responses_*_2023-05-11: 53/dns 53/dns_tcp 80/http 443/https 2095/http 8080/http
Index #89 - responses_*_2023-09-22: 53/dns 53/dns_tcp 80/http 443/https 2095/http 8080/http
Index #103 - responses_*_2024-01-22: 53/dns 53/dns_tcp 80/http 443/https 2095/http 8080/http
Index #110 - responses_*_2024-04-25: 53/dns 53/dns_tcp 80/http 443/https 2095/http 2096/https 8080/http 8443/http 8443/https
See Also
Explore more examples of using the Netlas API in these resources:
- Netlas Cookbook contains a lot of automation examples, both using the command line and the Python SDK.
- Netlas Scripts repo contains automations, which are discussed in detail in our blog on Medium.