GOSINT - open source solution for managing indicators of compromise (IoC)
One of the hottest topics in cyber security recently has been the Security Operations Center (SOC), which do not translate very well into Russian as "security monitoring centers", thereby detracting from one of the important functions of the SOC related to responding to information security incidents. But today I would not like to go into terminological disputes, but briefly tell you about one of the projects that was implemented in our internal SOC - GOSINT, a system of indicators of compromise compromise (IoC). In fact, Cisco IS does not use the term “Security Operations Center,” replacing it with a more civic Cisco Security Incident Response Team (CSIRT). So GOSINT is one of our many open source projects , which is designed to collect and unify structured and unstructured information about threats (threat intelligence). I have already talked about how we monitor the security of our internal infrastructure, now it’s time to talk about what helps us do it most effectively.
The illustration above shows some milestones of our IS monitoring and response service, which was created as a separate unit about two decades ago. During this time we have gained tremendous experience in the field of not only creating but also operating SOCs, which we share both in the form of such notes, articles and presentations, and as a separate service, within which we help our customers either build or conduct an audit of an existing IS monitoring center (contact, if that). If you look closely at the illustration, then the last milestone on it is marked by our activity in putting our open source solutions into open access, which we developed as part of the activities of the information security service. One of these projects was the GOSINT system, to which we assign a very important and very difficult task - to collect from separate sources indicators of compromise, which then enrich security events collected from our perimeter, infrastructure, mobile devices and clouds. And what's the problem, you ask? You can create feeds with indicators even in Cisco Firepower with the Threat Intelligence Director installed.
But with such feeds, there are a couple of nuances that emerge in the process of practical work with them. First, in order to quickly obtain data on the latest threats, there must be a lot of sources. How much do you think the Cisco Information Security Service monitors for Threat Intelligence feed sources? I will not give an exact figure, but there are several thousand of them. If all of them are started up on Cisco Firepower with Threat Intelligence Director, then the device will “sway” in performance, trying to process tens and hundreds of thousands of indicators coming in from all sources per day. The second problem related to the indicators of compromise is their poor quality. According to our estimates, up to 80% of all indicators are either outdated information, or false positives, or just ordinary garbage, which, however, requires appropriate processing.
To build a Threat Intelligence process in your organization, you can use both commercial and free, freely available Internet sources of threat information that will be downloaded to your TI platform or directly to SIEM or log management systems. GOSINT is a tool that collects indicators of compromise from various sources, checks them, scavenges them, and then feeds Threat Intelligence management solutions you use. It can be both commercial solutions and freely distributed. For example, Cisco uses MITRE and MISP CRITs . As the Threat Intelligence segment evolves and new data formats emerge and interchange indicators are exchanged, the GOSINT platform is also expanding to support them. Today, the sources we support include OpenPhish , AlienVault OTX , Abuse.ch , Emerging Threats , malc0de , SANS ICS , Cisco Talos , pastebin, and many others.
GOSINT plays a crucial role in the Threat Intelligence process — it automates the process of checking and passing on indicators of compromise to production. Even having received them from domestic feeds / newsletters suppliers, for example, from GosSOPKI or FinCERT, from Kaspersky Lab or BI.ZONE, from Perspective Monitoring or Group-IB, you have to decide whether to give these indicators the green light or reject them from further processing. GOSINT just automates this task, either automatically cutting off garbage, or helping the analysts of your SOC or the response service make the right decision. This is done by cross-checking the indicator through various sources of Threat Intelligence.
The number of sources of indicators of compromise in GOSINT is not limited (several thousand, like in Cisco, this solution handles without problems) and they can connect in different ways. As a rule, the corresponding APIs are used. For example, Cisco Umbrella , VirusTotal, or Twitter API . By receiving information on each indicator, GOSINT can label each of them appropriately so that SOC analysts or response services can use this information in the future when analyzing incidents.
GOSINT includes a so-called recipe manager, which makes it easy to create handlers for the resulting indicators. For example, you get a hash of a file that someone recognized as malicious. To reduce the number of false positives and reduce the load on your IB systems, using the Recipe Manager, you can run a hash check on various sources of Threat Intelligence and, if three of them (and the trigger threshold, and the number of sources to check are configured), gave a positive verdict ( a match is found), then this hash is recognized as relevant and is put into operation - transmitted to MISP, CRITs, Cisco Threat Intelligence Director, Arcsight or QRadar.
GOSINT has another great Ad Hoc Input feature - loading all indicators of compromise from a specific URL. For example, you need to load IoC on a specific hacker campaign or on a new malicious code in order to check their presence in your infrastructure using the tool used for Threat Hunting. If you use the Cisco Threat Response, which I have already mentioned , you copy all the indicators from the threat description and paste them into the Cisco Threat Response start window, which then starts looking for traces in your infrastructure.
If you do not have Cisco Threat Response and you use other Threat Hunting solutions, then GOSINT can be integrated with them just due to the Ad Hoc Input function, which automates the collection of indicators for a specific malicious code or hacker company from the specified URL.
It doesn't make much sense to talk about GOSINT - it's not a Stealthwatch with a bunch of features that help detect threats or anomalies on the internal network or in the cloud infrastructure. We developed GOSINT to automate only one function, which was highly demanded by the information security service of Cisco. We successfully solved this task and were able to embed GOSINT into our Threat Intelligence process. If working with indicators of compromise is included in your daily work, if you do not have the finances to buy expensive and, unfortunately, not very effective Threat Intelligence platforms (we tried many solutions, but eventually settled on our own development), then try GOSINT. You can download it from our GitHub security monitoring and incident response service page - https://github.com/ciscocsirt/GOSINT . Given that we are talking about open source, you can independently finish this solution to your needs. GOSINT is written in Go with the control console in JavaScript (but keep in mind that we regularly update GOSINT and your changes may be lost during the update process described in the documentation).
If you need to conduct an audit of your SOC (and we have a wealth of experience in such work around the world) or build it from scratch (including within the framework of the “own-outsourcing SOC” hybrid scheme), then we can also help you to do it. We do not sell any SIEM and therefore are vendor-independent on which SIEM solutions you use or plan to use in your monitoring center. As part of the services offered, we can recommend an effective combination of various technological solutions used in SOC, commercial or open source, including GOSINT. For details on these types of services, you can contact either your Cisco manager or write to security-request at cisco dot com
The illustration above shows some milestones of our IS monitoring and response service, which was created as a separate unit about two decades ago. During this time we have gained tremendous experience in the field of not only creating but also operating SOCs, which we share both in the form of such notes, articles and presentations, and as a separate service, within which we help our customers either build or conduct an audit of an existing IS monitoring center (contact, if that). If you look closely at the illustration, then the last milestone on it is marked by our activity in putting our open source solutions into open access, which we developed as part of the activities of the information security service. One of these projects was the GOSINT system, to which we assign a very important and very difficult task - to collect from separate sources indicators of compromise, which then enrich security events collected from our perimeter, infrastructure, mobile devices and clouds. And what's the problem, you ask? You can create feeds with indicators even in Cisco Firepower with the Threat Intelligence Director installed.
But with such feeds, there are a couple of nuances that emerge in the process of practical work with them. First, in order to quickly obtain data on the latest threats, there must be a lot of sources. How much do you think the Cisco Information Security Service monitors for Threat Intelligence feed sources? I will not give an exact figure, but there are several thousand of them. If all of them are started up on Cisco Firepower with Threat Intelligence Director, then the device will “sway” in performance, trying to process tens and hundreds of thousands of indicators coming in from all sources per day. The second problem related to the indicators of compromise is their poor quality. According to our estimates, up to 80% of all indicators are either outdated information, or false positives, or just ordinary garbage, which, however, requires appropriate processing.
To build a Threat Intelligence process in your organization, you can use both commercial and free, freely available Internet sources of threat information that will be downloaded to your TI platform or directly to SIEM or log management systems. GOSINT is a tool that collects indicators of compromise from various sources, checks them, scavenges them, and then feeds Threat Intelligence management solutions you use. It can be both commercial solutions and freely distributed. For example, Cisco uses MITRE and MISP CRITs . As the Threat Intelligence segment evolves and new data formats emerge and interchange indicators are exchanged, the GOSINT platform is also expanding to support them. Today, the sources we support include OpenPhish , AlienVault OTX , Abuse.ch , Emerging Threats , malc0de , SANS ICS , Cisco Talos , pastebin, and many others.
GOSINT plays a crucial role in the Threat Intelligence process — it automates the process of checking and passing on indicators of compromise to production. Even having received them from domestic feeds / newsletters suppliers, for example, from GosSOPKI or FinCERT, from Kaspersky Lab or BI.ZONE, from Perspective Monitoring or Group-IB, you have to decide whether to give these indicators the green light or reject them from further processing. GOSINT just automates this task, either automatically cutting off garbage, or helping the analysts of your SOC or the response service make the right decision. This is done by cross-checking the indicator through various sources of Threat Intelligence.
The number of sources of indicators of compromise in GOSINT is not limited (several thousand, like in Cisco, this solution handles without problems) and they can connect in different ways. As a rule, the corresponding APIs are used. For example, Cisco Umbrella , VirusTotal, or Twitter API . By receiving information on each indicator, GOSINT can label each of them appropriately so that SOC analysts or response services can use this information in the future when analyzing incidents.
GOSINT includes a so-called recipe manager, which makes it easy to create handlers for the resulting indicators. For example, you get a hash of a file that someone recognized as malicious. To reduce the number of false positives and reduce the load on your IB systems, using the Recipe Manager, you can run a hash check on various sources of Threat Intelligence and, if three of them (and the trigger threshold, and the number of sources to check are configured), gave a positive verdict ( a match is found), then this hash is recognized as relevant and is put into operation - transmitted to MISP, CRITs, Cisco Threat Intelligence Director, Arcsight or QRadar.
GOSINT has another great Ad Hoc Input feature - loading all indicators of compromise from a specific URL. For example, you need to load IoC on a specific hacker campaign or on a new malicious code in order to check their presence in your infrastructure using the tool used for Threat Hunting. If you use the Cisco Threat Response, which I have already mentioned , you copy all the indicators from the threat description and paste them into the Cisco Threat Response start window, which then starts looking for traces in your infrastructure.
If you do not have Cisco Threat Response and you use other Threat Hunting solutions, then GOSINT can be integrated with them just due to the Ad Hoc Input function, which automates the collection of indicators for a specific malicious code or hacker company from the specified URL.
It doesn't make much sense to talk about GOSINT - it's not a Stealthwatch with a bunch of features that help detect threats or anomalies on the internal network or in the cloud infrastructure. We developed GOSINT to automate only one function, which was highly demanded by the information security service of Cisco. We successfully solved this task and were able to embed GOSINT into our Threat Intelligence process. If working with indicators of compromise is included in your daily work, if you do not have the finances to buy expensive and, unfortunately, not very effective Threat Intelligence platforms (we tried many solutions, but eventually settled on our own development), then try GOSINT. You can download it from our GitHub security monitoring and incident response service page - https://github.com/ciscocsirt/GOSINT . Given that we are talking about open source, you can independently finish this solution to your needs. GOSINT is written in Go with the control console in JavaScript (but keep in mind that we regularly update GOSINT and your changes may be lost during the update process described in the documentation).
If you need to conduct an audit of your SOC (and we have a wealth of experience in such work around the world) or build it from scratch (including within the framework of the “own-outsourcing SOC” hybrid scheme), then we can also help you to do it. We do not sell any SIEM and therefore are vendor-independent on which SIEM solutions you use or plan to use in your monitoring center. As part of the services offered, we can recommend an effective combination of various technological solutions used in SOC, commercial or open source, including GOSINT. For details on these types of services, you can contact either your Cisco manager or write to security-request at cisco dot com
Source: https://habr.com/ru/post/438132/