Policy sources¶
Policy sources represent collections of local and remote pre-formatted data. This includes Suricata rulesets, Intelligence Threat feeds, and other relevant data that can be use with the Input framework.
Corelight-update collects data from these sources, along with data from third-party integrations, to be processed according to the respective management settings. For more details, see:
A policy data source differs from third-party integration, as policy data sources must be pre-formatted content you can download using an unauthenticated, basic-authenticated, or token-authenticated URL.
The URL for a remote policy source must be accessible via HTTPS or HTTP. No other protocols are supported.
See Using a proxy with Corelight-update for details about using a proxy to download remote sources.
Locally managed sources¶
In addition to downloading content from external sources for your sensors, Corelight-update will also accept locally-sourced content and configurations that can be applied at a Global-level, or at a Policy-level.
Corelight-update provides folders for input, threat intel, and suricata data at the Global-level and Policy-level, where you can place pre-formatted content to be processed. The following is a list of folder locations files can be placed for automatic processing.
/etc/corelight-update/global/global-input /etc/corelight-update/global/global-intel /etc/corelight-update/global/global-suricata /etc/corelight-update/configs/<policy_name>/local-input /etc/corelight-update/configs/<policy_name>/local-intel /etc/corelight-update/configs/<policy_name>/local-suricata
For example, if an intel file is placed in the global-intel
folder, the contents are added to the published intel file for all policies. If an intel file is placed in a policy local-intel
folder, the contents are automatically added to the published intel file only for that policy.
The following functions do not require any additional configuration:
Local Intel folders¶
All Zeek compatible formatted files in the
global-intel
folder are added to all policies as an intel file.Any Zeek compatible formatted files placed in a
local-intel
folder is added to that policy as an intel file.Any intel files in the
global-intel
,local-intel
, or generated by an enabled integration are automatically merged into a single intel.dat file.
Local Suricata folders¶
Any Suricata formatted “.rules” or “.rules.tar.gz” ruleset files placed in the
global-suricata
folder are available to all policies.Any Suricata formatted “.rules” or “.rules.tar.gz” ruleset placed in a
local-suricata
folder are available to that policy.Any ruleset file in the
global-suricata
,local-suricata
, or generated by an enabled integration are automatically processed and merged into a single suricata.rules file.
Local Input folders¶
Any Zeek compatible formatted files placed in the
global-input
folder are available to all policies.Any Zeek compatible formatted files placed in a
local-input
folder are available to that policy.Any input files in the
global-input
,local-input
, or generated by an enabled integration, (with the same name) will automatically get merged into a single input file with that name.
To review the order that the configurations are processed in, see Order of operations.
Remote source settings¶
The following fields are available for configuring a remote policy source:
sources: - name: url: source_type: global_cache: auth_type: auth_token: auth_token_header: username: encrypted_pass: ignore_tls:
The policy source
source_type
field can be set to eithersuricata
,intel
, orinput
. When using theintel
orinput
source type, the URL must provide the data in a Zeek compatible format. Forsuricata
, the URL must provide the data in the Suricata rule format.The
global_cache
is disabled (false) by default for all sources. Ifglobal_cache
is disabled, that source will be download once for each policy that uses it.The
auth_type
field can be set tobasic
,token
, or left empty forno auth
.
Overview of adding policy sources¶
Determine the access url and authentication required for the policy data source.
For basic-authenticated sources, use the CLI command
corelight-update encrypt <password>
to generate an encrypted password to store in the policy configuration.Configure the policy data source settings under the
sources:
section of the Corelight-updatedb-config
file.
Processing a policy source¶
When Corelight-update processes a policy source, it:
Checks the global cache for the target filename.
If the file is present, use the file to process the source.
If the file is not present in the global cache:
Check for a policy level cache of the file and generate an
If-Modified-Since
HTTP header.Attempt to download the file using the
If-Modified-Since
HTTP header.If a new file is downloaded, create or update the policy-level cache.
Use the policy-level cache to process the source.
Caution
Matches are made in the global cache using only the filename, not the full URL.
Default policy sources¶
The default policy provided for Corelight-update includes the pre-configured Suricata rulesets:
Corelight Labs Suricata Rules
ET/Open ruleset:
You will find the following pre-configured policy source settings in the db-config
example file:
sources: - name: "Corelight" url: "https://feed.corelight.com/corelight.rules" source_type: "suricata" global_cache: true - name: "ET/Open" url: "https://rules.emergingthreats.net/open/suricata-6.0/emerging.rules.tar.gz" source_type: "suricata" global_cache: false
For more source ideas, see Commonly used Suricata rulesets
Threat intelligence source example¶
The threat intel sources managed with Corelight-update must provide their data in a Zeek compatible format.
The following example includes settings for the ThreatQ and MISP threat intel sources:
sources: - name: ThreatQ url: https://string.experience.threatq.online/api/export/c8299290f2d4319923e2eb/?token=aasTjqMXwJ4u source_type: intel global_cache: false - name: MISP url: https://misp/attributes/bro/download/all source_type: intel global_cache: false auth_type: token auth_token: BVkgNaFh27IGelkIuEAiPBBlDsOp9cjd auth_token_header: Authorization ignore_tls: true
Note
For more details on these examples, see Third-party configuration guides.
Third-party integrations¶
A third-party integration is a data source that might require custom formatting or parsing of the data for use with a sensor, or has unique authentication requirements.
For more details, and a complete list of the current integrations and their respective configurations, see Third-party integrations settings