When you're facing some suspicious files while performing forensic investigations or analyzing malware components, it's always interesting to know these files are legit or malicious/modified. One of the key sources to verify hashes is provided by NIST and is called the NSLR project ("National Software Reference Library")[1]. They build "Reference Data Set" (RDS) of information that can be queried to verify a file hash. These RDS are available to download[2] but, as you may expect, there are huge (they are provided as ISO files between 500MB to 4GB!)

CIRCL[3], the Luxembourg CERT, has a good reputation to offer/participate in services like MISP, a passive DNS service, etc. They are now offering an API to query the NIST RDS via HTTP or DNS requests!

How does it work? You can use the FQDN hashlookup.circl.lu like this:

PS C:\Users\xavie> curl.exe -X GET https://hashlookup.circl.lu/lookup/md5/8ED4B4ED952526D89899E723F3488DE4 -H 'Accept: application/json'
{
  "CRC32": "7A5407CA", 
  "FileName": "wow64_microsoft-windows-i..timezones.resources_31bf3856ad364e35_10.0.16299.579_de-de_f24979c73226184d.manifest", 
  "FileSize": "2520", 
  "MD5": "8ED4B4ED952526D89899E723F3488DE4", 
  "OpSystemCode": {"MfgCode": "1006", "OpSystemCode": "362", "OpSystemName": "TBD", "OpSystemVersion": "none"},
  "ProductCode": {
    "ApplicationType": "Security", 
    "Language": "Multilanguage", 
    "MfgCode": "608", 
    "OpSystemCode": "868", 
    "ProductCode": "190742", 
    "ProductName": "Cumulative Update for Windows Server 2016 for x64 (KB4338817)", 
    "ProductVersion": "1709"
  },
  "SHA-1": "00000079FD7AAC9B2F9C988C50750E1F50B27EB5", "SpecialCode": ""
}

You can also query the database via a DNS request (my preferred method!):

root@remnux:/# dig +short -t TXT 8ED4B4ED952526D89899E723F3488DE4.dns.hashlookup.circl.lu | jq -r . | jq .
{
  "CRC32": "7A5407CA",
  "FileName": "wow64_microsoft-windows-i..timezones.resources_31bf3856ad364e35_10.0.16299.579_de-de_f24979c73226184d.manifest",
  "FileSize": "2520",
"MD5": "8ED4B4ED952526D89899E723F3488DE4",
"OpSystemCode": {"MfgCode": "1006", "OpSystemCode": "362", "OpSystemName": "TBD", "OpSystemVersion": "none"},
"ProductCode": {
  "ApplicationType": "Security",
  "Language": "Multilanguage",
  "MfgCode": "608",
  "OpSystemCode": "868",
  "ProductCode": "190742",
  "ProductName": "Cumulative Update for Windows Server 2016 for x64 (KB4338817)",
  "ProductVersion": "1709"
  },
  "SHA-1": "00000079FD7AAC9B2F9C988C50750E1F50B27EB5", "SpecialCode": ""
}

SHA1 and MD5 hashes are supported and you can also submit bulk requests. More documentation is available here[4].

If you need to scan a complete filesystem, my recommendation is of course to use your own local copy of the NIST databases! Many forensic tools allow you to compare hashes against a local database. But when you quickly need to check a single file or a restricted set of files, this service is perfect! Thank you to CIRCL for providing this to the community! The website says also that more databases will be added in the future!

[1] https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl
[2] https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl/nsrl-download/current-rds
[3] https://circl.lu
[4] https://gist.github.com/adulau/4191d44e30fc01df38f1d5fe605fa920#file-hashlookup-circl-lu-md

Xavier Mertens (@xme)
Senior ISC Handler - Freelance Cyber Security Consultant
PGP Key