Apache NiFi

Apache NiFi is an open-source data integration platform that automates data flows between systems through a web-based drag-and-drop interface. When you integrate Apache NiFi with Scality ARTESCA, NiFi processors can read from and write to S3-compatible object storage, enabling automated data pipelines for ingestion, transformation, and distribution on on-premises infrastructure.

Prerequisites

• Apache NiFi installed and running. Refer to the Apache NiFi getting started documentation.

• A Linux server with network access to the ARTESCA S3 endpoint.

Set Up S3 Resources for Apache NiFi

These steps prepare the ARTESCA account, bucket, Identity and Access Management (IAM) identity, and access policy that Apache NiFi authenticates against.

1. Create an Account for Apache NiFi.

2. Create a bucket to store flow data.

3. Create IAM users for Apache NiFi.

4. Create an IAM policy to grant Apache NiFi access to the S3 bucket.

   The following policy grants Apache NiFi least-privilege access. s3:ListAllMyBuckets requires Resource: "*" and is split into a separate statement from the bucket-scoped object and metadata actions.

   Note

   Replace each bucket_name placeholder in the policy with the actual name of your bucket.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "s3:ListAllMyBuckets",
           "s3:ListBucket"
         ],
         "Resource": "*"
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:PutObject",
           "s3:GetObject",
           "s3:DeleteObject",
           "s3:GetBucketLocation",
           "s3:GetBucketVersioning",
           "s3:GetBucketObjectLockConfiguration"
         ],
         "Resource": [
           "arn:aws:s3:::bucket_name",
           "arn:aws:s3:::bucket_name/*"
         ]
       }
     ]
   }
   

5. Create IAM User Access Keys.

Install Certificates

Install the ARTESCA Certificate Authority (CA) on the NiFi host so the NiFi S3 processors trust the ARTESCA S3 endpoint.

1. Download the CA Root certificate.

2. Import the Certificate.

3. On Ubuntu, copy the certificate to the system CA trust store and update the trust chain.

   sudo cp /<YOUR_CERTIFICATE_PATH>/artesca-ca.pem /usr/local/share/ca-certificates/artesca-ca.crt
   sudo update-ca-certificates
   

Configure NiFi for ARTESCA Operations

These steps set up the AWS credentials controller service and configure the NiFi flow that writes data to and reads data from ARTESCA.

1. Access the NiFi web UI (for example, https://nifi.local:8443/nifi).

2. Create an AWSCredentialsProviderControllerService.

   1. Right-click the canvas and click Configure.

   2. Select the Controller Services tab.

   3. Click + to add a new service.

   4. Select AWSCredentialsProviderControllerService.

   

3. Configure the service with your ARTESCA credentials.

   Note

   Replace <YOUR_ACCESS_KEY> and <YOUR_SECRET_KEY> with the access key and secret key generated when you created the IAM user access keys.

   • Access Key ID: <YOUR_ACCESS_KEY>

   • Secret Access Key: <YOUR_SECRET_KEY>

   

4. Enable the service by clicking the lightning bolt icon next to it.

   

5. Create a sample comma-separated values (CSV) dataset. Create a file named employees.csv with the following content:

   id,name,email,department,salary,hire_date,status,manager_id,location,address,phone,emergency_contact,skills,projects,notes,employee_id
   1,John Doe,john.doe@company.com,Engineering,75000,2023-01-15,Active,5.0,New York,"123 Main St, New York, NY 10001, Apartment 4B, Near Central Park with excellent public transportation access",555-0101,Jane Doe (spouse) 555-0102,"Java,Python,AWS,Docker,Kubernetes,Microservices,Spring Boot,PostgreSQL,Redis,Elasticsearch","Project Alpha - Cloud Migration involving 50+ microservices, Project Beta - API Development with OAuth2 implementation, Project Gamma - Database optimization reducing query time by 60%",Excellent team player with strong technical skills and 8 years of experience. Led successful migration of legacy monolithic systems to cloud-native microservices architecture. Recognized for innovative solutions and mentoring 5 junior developers. Consistently delivers high-quality code and participates in architectural decisions.,1
   2,Jane Smith,jane.smith@company.com,Marketing,68000,2023-02-20,Active,6.0,Los Angeles,"456 Oak Ave, Los Angeles, CA 90210, Suite 12, Hollywood Hills area with panoramic city views",555-0201,Bob Smith (brother) 555-0202,"SEO,SEM,Analytics,Content Marketing,Social Media Management,Adobe Creative Suite,Google Analytics,HubSpot,Salesforce Marketing Cloud","Campaign Gamma - Brand Awareness reaching 2M+ users, Project Delta - Digital Transformation increasing conversion rates by 45%, Project Epsilon - Influencer Marketing generating $500K revenue",Creative marketer with proven track record in digital campaigns and 6 years of experience. Increased brand engagement by 150% through innovative social media strategies. Expert in data-driven marketing approaches with strong analytical skills. Led cross-functional teams to deliver successful campaigns.,2
   3,Mike Johnson,mike.j@company.com,Sales,72000,2023-03-10,Active,7.0,Chicago,"789 Pine St, Chicago, IL 60601, Floor 15, Downtown location with Lake Michigan views",555-0301,Mary Johnson (mother) 555-0302,"Salesforce,CRM,Negotiation,Lead Generation,Account Management,B2B Sales,Enterprise Solutions,Contract Management","Deal Epsilon - Enterprise Sales worth $2M annually, Project Zeta - Customer Retention improving renewal rates by 35%, Project Eta - Territory Expansion adding 50 new accounts",Top performer in sales division with consistent quota achievement over 120% for 3 consecutive years. Developed new client acquisition strategies that increased revenue by 30%. Strong relationship builder with Fortune 500 enterprise clients. Excellent communication and presentation skills.,3
   4,Sarah Wilson,sarah.w@company.com,HR,65000,2023-04-05,Active,8.0,New York,"321 Elm Dr, New York, NY 10002, Garden apartment with private parking space",555-0401,Tom Wilson (husband) 555-0402,"Recruitment,HRIS,Employee Relations,Training,Performance Management,Compensation Analysis,Diversity & Inclusion,Workday,BambooHR","Project Eta - Talent Acquisition reducing time-to-hire by 40%, Initiative Theta - Employee Engagement improving satisfaction scores by 25%, Project Iota - Diversity Program increasing representation by 30%",Experienced HR professional specializing in talent acquisition and employee development with 7 years of experience. Reduced hiring time by 40% through process optimization and automation. Champion of workplace diversity and inclusion initiatives. Strong interpersonal skills and conflict resolution expertise.,4
   5,David Brown,david.b@company.com,Engineering,78000,2023-05-12,Active,,New York,"654 Maple Rd, New York, NY 10003, Brownstone building with rooftop access",555-0501,Lisa Brown (wife) 555-0502,"React,Node.js,MongoDB,Kubernetes,Docker,GraphQL,TypeScript,AWS,Jenkins,Git","Project Iota - Frontend Redesign improving user experience by 40%, Project Kappa - Microservices Architecture handling 1M+ requests daily, Project Lambda - CI/CD Pipeline reducing deployment time by 70%","Senior engineer with expertise in full-stack development and 9 years of experience. Architected scalable microservices platform serving millions of users daily. Passionate about code quality, best practices, and modern development methodologies. Mentors team members and leads technical discussions.",5
   6,Lisa Davis,lisa.d@company.com,Finance,70000,2023-06-18,Active,,Los Angeles,"987 Birch Ln, Los Angeles, CA 90211, Luxury condo with pool and fitness center",555-0601,Carol Davis (sister) 555-0602,"Financial Analysis,Excel,SAP,Budgeting,Forecasting,Financial Modeling,QuickBooks,Tableau,Power BI,Risk Management","Project Mu - Budget Planning for $50M annual budget, Project Nu - Cost Optimization saving $2M annually, Project Xi - Financial Reporting automation reducing manual work by 60%",Detail-oriented finance professional with 5 years of experience in corporate finance. Expert in financial modeling and analysis with strong Excel and SAP skills. Successfully managed budgets exceeding $50M and identified cost-saving opportunities. Strong analytical mindset and attention to detail.,6
   7,Tom Miller,tom.m@company.com,Operations,63000,2023-07-22,Active,,Chicago,"246 Cedar Ave, Chicago, IL 60602, Historic building with modern amenities",555-0701,Sue Miller (daughter) 555-0702,"Process Improvement,Lean Six Sigma,Project Management,Supply Chain,Logistics,ERP Systems,Quality Assurance,Vendor Management","Project Omicron - Process Optimization reducing operational costs by 20%, Project Pi - Supply Chain Management improving delivery times by 30%, Project Rho - Quality System implementation achieving ISO certification",Operations specialist with 6 years of experience in process improvement and supply chain management. Certified Lean Six Sigma Black Belt with proven track record in operational excellence. Led multiple process optimization projects resulting in significant cost savings and efficiency improvements.,7
   8,Amy Taylor,amy.t@company.com,Marketing,69000,2023-08-30,Active,6.0,Los Angeles,"135 Spruce St, Los Angeles, CA 90212, Modern apartment with smart home features",555-0801,Jim Taylor (father) 555-0802,"Digital Marketing,PPC,Facebook Ads,Google Ads,Email Marketing,Marketing Automation,Mailchimp,Hootsuite,Canva,A/B Testing","Project Sigma - PPC Campaign generating 300% ROI, Project Tau - Email Marketing with 25% open rates, Project Upsilon - Social Media Strategy increasing followers by 200%",Digital marketing specialist with 4 years of experience in performance marketing. Expert in PPC campaigns and social media advertising with strong analytical skills. Consistently achieves high ROI on marketing campaigns and stays current with latest digital marketing trends and tools.,8
   9,Chris Anderson,chris.a@company.com,Sales,71000,2023-09-14,Active,7.0,Chicago,"579 Willow Way, Chicago, IL 60603, Loft apartment in trendy neighborhood",555-0901,Pat Anderson (spouse) 555-0902,"B2B Sales,SaaS Sales,Cold Outreach,LinkedIn Sales Navigator,Pipedrive,Outreach.io,Sales Analytics,Territory Management","Project Phi - Territory Development adding $1.5M in pipeline, Project Chi - Sales Process Optimization improving close rates by 25%, Project Psi - Customer Success program reducing churn by 40%",Results-driven sales professional with 5 years of experience in B2B SaaS sales. Consistently exceeds quota and has strong track record in new business development. Expert in consultative selling and building long-term customer relationships. Strong communication and negotiation skills.,9
   10,Emma White,emma.w@company.com,Engineering,76000,2023-10-25,Active,5.0,New York,"802 Ash Blvd, New York, NY 10004, High-rise apartment with city skyline views",555-1001,Ryan White (brother) 555-1002,"Python,Django,PostgreSQL,Redis,Celery,REST APIs,Machine Learning,Data Science,Pandas,NumPy,Scikit-learn","Project Omega - Data Pipeline processing 10TB daily, Project Alpha-2 - Machine Learning model improving predictions by 35%, Project Beta-2 - API Performance optimization reducing response time by 50%",Data engineer with strong background in machine learning and 6 years of experience. Built scalable data pipelines processing terabytes of data daily. Expert in Python ecosystem and modern data engineering tools. Passionate about leveraging data to drive business decisions and innovation.,10
   11,Robert Garcia,robert.g@company.com,Engineering,74000,2023-11-08,Active,5.0,New York,"159 Hickory Hill, New York, NY 10005, Duplex with private entrance and garden",555-1101,Maria Garcia (wife) 555-1102,"JavaScript,Vue.js,Node.js,Express,MongoDB,Socket.io,Redis,AWS Lambda,Serverless,GraphQL","Project Gamma-2 - Real-time Chat Application serving 100K+ users, Project Delta-2 - Serverless Architecture reducing infrastructure costs by 45%, Project Epsilon-2 - Mobile App Backend supporting iOS and Android",Full-stack developer with 7 years of experience specializing in JavaScript and modern web technologies. Built real-time applications serving hundreds of thousands of users. Expert in serverless architectures and cloud-native development. Strong problem-solving skills and attention to performance optimization.,11
   12,Michelle Lee,michelle.l@company.com,Marketing,67000,2023-12-01,Active,6.0,Los Angeles,"753 Poplar Plaza, Los Angeles, CA 90213, Corner unit with balcony and mountain views",555-1201,Kevin Lee (husband) 555-1202,"Content Marketing,Copywriting,WordPress,SEO,Google Analytics,Social Media,Influencer Marketing,Brand Management,PR,Event Marketing","Project Zeta-2 - Content Strategy increasing organic traffic by 180%, Project Eta-2 - Brand Refresh improving brand recognition by 50%, Project Theta-2 - Event Marketing generating 500+ qualified leads",Content marketing specialist with 5 years of experience in brand management and digital content creation. Expert copywriter with proven track record in increasing organic traffic and brand engagement. Strong creative skills combined with analytical mindset to measure content performance and ROI.,12
   13,Kevin Martinez,kevin.m@company.com,Sales,70000,2024-01-10,Active,7.0,Chicago,"426 Magnolia Manor, Chicago, IL 60604, Townhouse with garage and basement",555-1301,Sofia Martinez (daughter) 555-1302,"Enterprise Sales,Solution Selling,Proposal Writing,Presentation Skills,CRM Management,Sales Forecasting,Channel Partnerships,Trade Shows","Project Iota-2 - Enterprise Account Management worth $3M annually, Project Kappa-2 - Channel Partner Program increasing revenue by 60%, Project Lambda-2 - Sales Training program improving team performance by 30%",Senior sales executive with 8 years of experience in enterprise software sales. Consistently ranks in top 10% of sales organization with strong track record in large deal closure. Expert in solution selling and building strategic partnerships. Excellent presentation and communication skills.,13
   14,Jennifer Taylor,jen.t@company.com,HR,64000,2024-02-14,Active,8.0,New York,"918 Sycamore Square, New York, NY 10006, Studio apartment with modern appliances",555-1401,Mark Taylor (brother) 555-1402,"HR Generalist,Payroll,Benefits Administration,FMLA,Workers Compensation,HR Compliance,Employment Law,Onboarding,Exit Interviews","Project Mu-2 - HRIS Implementation streamlining HR processes by 50%, Project Nu-2 - Benefits Program redesign saving company $200K annually, Project Xi-2 - Compliance Audit achieving 100% compliance rating",HR generalist with 4 years of experience in all aspects of human resources. Expert in HR compliance and employment law with strong attention to detail. Successfully managed payroll for 500+ employees and implemented new HRIS system. Excellent interpersonal skills and employee advocacy.,14
   15,Mark Wilson,mark.w@company.com,Finance,72000,2024-03-20,Active,6.0,Los Angeles,"371 Redwood Ridge, Los Angeles, CA 90214, Two-bedroom condo with pool and spa access",555-1501,Linda Wilson (mother) 555-1502,"Corporate Finance,Investment Analysis,Financial Planning,Excel,VBA,Bloomberg Terminal,Capital Budgeting,M&A Analysis,Valuation Models","Project Omicron-2 - Investment Analysis for $10M acquisition, Project Pi-2 - Financial Model development for new product line, Project Rho-2 - Cost-benefit analysis for major system upgrade",Finance analyst with 6 years of experience in corporate finance and investment analysis. Expert in financial modeling and valuation with strong Excel and Bloomberg skills. Successfully evaluated multiple acquisition opportunities and provided strategic financial guidance to senior management.,15
   16,Anthony Davis,anthony.d@company.com,Engineering,77000,2024-04-15,Active,5.0,New York,"642 Chestnut Circle, New York, NY 10007, Penthouse apartment with terrace",555-1601,Nicole Davis (wife) 555-1602,"C++,Python,Linux,Embedded Systems,IoT,Firmware Development,Real-time Systems,Hardware Integration,Debugging,Performance Optimization","Project Sigma-2 - IoT Platform connecting 50K+ devices, Project Tau-2 - Embedded Software for automotive industry, Project Upsilon-2 - Real-time System optimization improving latency by 80%",Embedded systems engineer with 10 years of experience in firmware development and IoT solutions. Expert in C++ and low-level programming with strong hardware integration skills. Built IoT platforms connecting tens of thousands of devices. Passionate about performance optimization and real-time systems.,16
   17,Rachel Johnson,rachel.j@company.com,Marketing,66000,2024-05-10,Active,6.0,Los Angeles,"185 Walnut Way, Los Angeles, CA 90215, Garden apartment with private patio",555-1701,Steve Johnson (husband) 555-1702,"Brand Management,Market Research,Consumer Insights,Surveys,Focus Groups,Statistical Analysis,SPSS,Tableau,Presentation Skills","Project Phi-2 - Market Research for new product launch, Project Chi-2 - Brand Positioning study improving market share by 15%, Project Psi-2 - Consumer Behavior analysis driving product improvements",Market research specialist with 5 years of experience in consumer insights and brand management. Expert in statistical analysis and survey methodology with strong presentation skills. Led multiple market research projects that directly influenced product development and marketing strategy.,17
   18,Brandon Lee,brandon.l@company.com,Sales,73000,2024-06-05,Active,7.0,Chicago,"729 Dogwood Drive, Chicago, IL 60605, Converted warehouse loft with exposed brick",555-1801,Jessica Lee (girlfriend) 555-1802,"Inside Sales,Lead Qualification,CRM,Sales Development,Cold Calling,Email Outreach,Sales Analytics,Territory Planning","Project Omega-2 - Lead Generation program creating 1000+ qualified leads monthly, Project Alpha-3 - Sales Development process improving conversion rates by 40%, Project Beta-3 - Territory Analysis optimizing coverage and performance",Inside sales specialist with 4 years of experience in lead generation and qualification. Expert in CRM systems and sales development processes with strong phone and email communication skills. Consistently exceeds lead generation targets and maintains high conversion rates.,18
   19,Nicole Brown,nicole.b@company.com,HR,63000,2024-07-20,Active,8.0,New York,"436 Elmwood Estates, New York, NY 10008, Two-story townhouse with private garage",555-1901,Tyler Brown (son) 555-1902,"Training and Development,Learning Management Systems,Instructional Design,e-Learning,Performance Coaching,Talent Development,Succession Planning","Project Gamma-3 - Leadership Development program training 100+ managers, Project Delta-3 - e-Learning Platform implementation increasing training completion by 75%, Project Epsilon-3 - Performance Management system redesign",Training and development specialist with 6 years of experience in corporate learning and talent development. Expert in instructional design and e-learning platforms with strong coaching skills. Successfully designed and implemented leadership development programs that improved employee performance and retention.,19
   20,Daniel Martinez,daniel.m@company.com,Finance,71000,2024-08-12,Active,6.0,Los Angeles,"593 Beechwood Boulevard, Los Angeles, CA 90216, Modern high-rise with concierge service",555-2001,Carmen Martinez (wife) 555-2002,"Financial Reporting,SEC Compliance,GAAP,Financial Statements,Auditing,Internal Controls,Excel,SAP,Hyperion,Financial Analysis","Project Zeta-3 - SEC Reporting automation reducing preparation time by 60%, Project Eta-3 - Internal Controls enhancement achieving SOX compliance, Project Theta-3 - Financial Close process optimization reducing cycle time by 5 days",Senior financial analyst with 7 years of experience in financial reporting and SEC compliance. Expert in GAAP and financial statement preparation with strong attention to detail. Successfully led financial close process improvements and maintained 100% compliance with regulatory requirements.,20
   

6. Create a NiFi flow with the following processors connected in sequence:

   Upload flow: GenerateFlowFile → ReplaceText → PutS3Object → LogMessage

   Download flow: ListS3 → FetchS3Object → LogMessage

7. Configure the GenerateFlowFile processor.

   • Run Schedule: 60 sec

   • File Size: 1 KB

   • Batch Size: 1

8. Configure the ReplaceText processor.

   • Search Value: .*

   • Replacement Strategy: Always Replace

   • Replacement Value: paste the CSV content from the previous step.

9. Configure the PutS3Object processor.

   Note

   Replace <YOUR_ENDPOINT> with the ARTESCA S3 endpoint hostname. Replace all instances of <bucket_name> with the actual bucket name.

   • Bucket: <bucket_name>

   • Object Key: test-data/employees_${now():format('yyyy-MM-dd_HH-mm-ss')}.csv

   • AWS Credentials Provider Service: select the service configured earlier.

   • Region: us-east-1

   • Endpoint Override URL: https://<YOUR_ENDPOINT>

   • Signer Override: AWSS3V4SignerType

   

10. Configure the ListS3 processor.

    • Bucket: <bucket_name>

    • AWS Credentials Provider Service: select the service configured earlier.

    • Region: us-east-1

    • Endpoint Override URL: https://<YOUR_ENDPOINT>

    

11. Configure the FetchS3Object processor.

    • AWS Credentials Provider Service: select the service configured earlier.

    • Region: us-east-1

    • Endpoint Override URL: https://<YOUR_ENDPOINT>

    

12. Configure the LogMessage processor.

    • Log Level: INFO

    • Log Message: Successfully processed S3 object: ${filename}

13. Connect the processors in this order:

    • GenerateFlowFile → ReplaceText

    • ReplaceText → PutS3Object

    • PutS3Object → LogMessage (success relationship)

    • ListS3 → FetchS3Object

    • FetchS3Object → LogMessage

Verify the Apache NiFi and ARTESCA Integration

Confirm that NiFi successfully writes to and reads from the target bucket on ARTESCA.

1. Start the flow by right-clicking each processor and selecting Start.

2. Monitor the processors for successful execution and review NiFi logs for errors.

   

3. Access the ARTESCA management interface and navigate to the NiFi bucket.

   

4. Verify the uploaded CSV files appear in the test-data/ prefix.