Splunk¶

Splunk collects machine-generated data from various sources across an organization, and indexing it in a searchable repository. This allows users to search, analyze, and visualize this data in real-time using a web-based interface.

Data is categorized as hot, warm, cold, or frozen:

Hot data is usually located on SSD.
Warm and cold data can be stored on a remote storage like S3 compatible object storage.
Frozen data is historic, and not indexed. Frozen data often resides on tape for ultra low cost. However, value is lost because it is not searchable, and difficult to retrieve.

Prerequisites¶

Install Splunk Enterprise with SmartStore. Scality has validated version 9.3.1 and higher.

Best Practices¶

Do not modify the configuration files in the default Splunk folders. Modifications should be done in local folders.

Architecture Support¶

ARTESCA deployed on a single site.
Enabling S3 Replication is not supported.
Enabling versioning on buckets is not supported.
Enabling Lifecycle Expiration on buckets is not supported.

Splunk Enterprise Integration¶

Create an Account for the Splunk application.
Create a Bucket to be used with Splunk application.
Create IAM Users for the Splunk application.

Create an IAM policy to grant the Splunk application access to S3 repository, and manage data from an object repository.

Note

Replace bucketname with the name of the bucket that will be used with the Splunk application.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "SplunkPolicy",
           "Effect": "Allow",
           "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetBucketAcl",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload"
           ],
           "Resource": [
               "arn:aws:s3:::bucketname/*",
               "arn:aws:s3:::bucketname"
           ]
       },
       {
           "Sid": "Splunk",
           "Effect": "Allow",
           "Action": [
               "s3:ListAllMyBuckets",
               "s3:ListBucket"
           ],
           "Resource": "*"
       }
    ]
}

Create Access Key and Secret Key for the IAM user.

Configure SmartStore¶

Set the configuration for smartstore in the splunk server located in /opt/splunk/etc/master-apps/_cluster/local/indexes.conf.

[default]
remotePath = volume:remote_store/$_index_name

repFactor = auto

# Configure the remote volume
[volume:remote_store]
storageType = remote.
remote.s3.access_key = <ARTESCA IAM User Access Key>
remote.s3.secret_key = <ARTESCA IAM User Secret Key>
remote.s3.endpoint = https://s3.artesca-test.com
remote.s3.supports_versioning = false
remote.s3.list_objects_version = v2
remote.s3.signature_version = v2
remote.s3.auth_region =

Performance¶

A standard configuration with a 3-node ARTESCA using SSDs can achieve a performance of 3GB/s.

Storage Profile¶

Average object size varies by the size of indexes. Splunk recommends using the following formula to determine the size of S3-compatible object stores:

Remote Object Store sizing = Daily Ingest Rate * Compression Ratio * Retention period performance