Policy sources

Policy sources represent collections of local and remote pre-formatted data. This includes Suricata rulesets, Intelligence Threat feeds, and other relevant data that can be use with the Input framework.

Corelight-update collects data from these sources, along with data from third-party integrations, to be processed according to the respective management settings. For more details, see:

A policy data source differs from third-party integration, as policy data sources must be pre-formatted content you can download using an unauthenticated, basic-authenticated, or token-authenticated URL.

The URL for a remote policy source must be accessible via HTTPS or HTTP. No other protocols are supported.

See Using a proxy with Corelight-update for details about using a proxy to download remote sources.

Locally managed sources

In addition to downloading content from external sources for your sensors, Corelight-update will also accept locally-sourced content and configurations that can be applied at a Global-level, or at a Policy-level.

Corelight-update provides folders for input, threat intel, and suricata data at the Global-level and Policy-level, where you can place pre-formatted content to be processed. The following is a list of folder locations files can be placed for automatic processing.

/etc/corelight-update/global/global-input
/etc/corelight-update/global/global-intel
/etc/corelight-update/global/global-suricata
/etc/corelight-update/configs/<policy_name>/local-input
/etc/corelight-update/configs/<policy_name>/local-intel
/etc/corelight-update/configs/<policy_name>/local-suricata

For example, if an intel file is placed in the global-intel folder, the contents are added to the published intel file for all policies. If an intel file is placed in a policy local-intel folder, the contents are automatically added to the published intel file only for that policy.

The following functions do not require any additional configuration:

Local Intel folders

  • All Zeek compatible formatted files in the global-intel folder are added to all policies as an intel file.

  • Any Zeek compatible formatted files placed in a local-intel folder is added to that policy as an intel file.

  • Any intel files in the global-intel, local-intel, or generated by an enabled integration are automatically merged into a single intel.dat file.

Local Suricata folders

  • Any Suricata formatted “.rules” or “.rules.tar.gz” ruleset files placed in the global-suricata folder are available to all policies.

  • Any Suricata formatted “.rules” or “.rules.tar.gz” ruleset placed in a local-suricata folder are available to that policy.

  • Any ruleset file in the global-suricata, local-suricata, or generated by an enabled integration are automatically processed and merged into a single suricata.rules file.

Local Input folders

  • Any Zeek compatible formatted files placed in the global-input folder are available to all policies.

  • Any Zeek compatible formatted files placed in a local-input folder are available to that policy.

  • Any input files in the global-input, local-input, or generated by an enabled integration, (with the same name) will automatically get merged into a single input file with that name.

To review the order that the configurations are processed in, see Order of operations.

Remote source settings

The following fields are available for configuring a remote policy source:

sources:
  - name:
    url:
    source_type:
    global_cache:
    auth_type:
    auth_token:
    auth_token_header:
    username:
    encrypted_pass:
    ignore_tls:
  • The policy source source_type field can be set to either suricata, intel, or input. When using the intel or input source type, the URL must provide the data in a Zeek compatible format. For suricata, the URL must provide the data in the Suricata rule format.

  • The global_cache is disabled (false) by default for all sources. If global_cache is disabled, that source will be download once for each policy that uses it.

  • The auth_type field can be set to basic, token, or left empty for no auth.

Overview of adding policy sources

  1. Determine the access url and authentication required for the policy data source.

  2. For basic-authenticated sources, use the CLI command corelight-update encrypt <password> to generate an encrypted password to store in the policy configuration.

  3. Configure the policy data source settings under the sources: section of the Corelight-update db-config file.

Processing a policy source

When Corelight-update processes a policy source, it:

  1. Checks the global cache for the target filename.

    1. If the file is present, use the file to process the source.

    2. If the file is not present in the global cache:

      1. Check for a policy level cache of the file and generate an If-Modified-Since HTTP header.

      2. Attempt to download the file using the If-Modified-Since HTTP header.

        • If a new file is downloaded, create or update the policy-level cache.

      3. Use the policy-level cache to process the source.

Caution

Matches are made in the global cache using only the filename, not the full URL.

Default policy sources

The default policy provided for Corelight-update includes the pre-configured Suricata rulesets:

  • Corelight Labs Suricata Rules

  • ET/Open ruleset:

You will find the following pre-configured policy source settings in the db-config example file:

sources:
  - name: "Corelight"
    url: "https://feed.corelight.com/corelight.rules"
    source_type: "suricata"
    global_cache: true
  - name: "ET/Open"
    url: "https://rules.emergingthreats.net/open/suricata-6.0/emerging.rules.tar.gz"
    source_type: "suricata"
    global_cache: false

For more source ideas, see Commonly used Suricata rulesets

Threat intelligence source example

The threat intel sources managed with Corelight-update must provide their data in a Zeek compatible format.

The following example includes settings for the ThreatQ and MISP threat intel sources:

sources:
  - name: ThreatQ
    url: https://string.experience.threatq.online/api/export/c8299290f2d4319923e2eb/?token=aasTjqMXwJ4u
    source_type: intel
    global_cache: false
  - name: MISP
    url: https://misp/attributes/bro/download/all
    source_type: intel
    global_cache: false
    auth_type: token
    auth_token: BVkgNaFh27IGelkIuEAiPBBlDsOp9cjd
    auth_token_header: Authorization
    ignore_tls: true

Note

For more details on these examples, see Third-party configuration guides.

Third-party integrations

A third-party integration is a data source that might require custom formatting or parsing of the data for use with a sensor, or has unique authentication requirements.

For more details, and a complete list of the current integrations and their respective configurations, see Third-party integrations settings