Network Firewall¶
ARTESCA comes with built-in settings for network firewall, which should be configured according to your needs and environment.
After installation, ARTESCA implements firewall rules for internal traffic (between containers), which is transferred over the workload plane network.
By default, it also configures restricting rules for host traffic on both control plane and workload plane networks. This document details each of these rules and how they are implemented.
System Constraints¶
ARTESCA implements a firewall through iptables rules, which are managed actively by a component called Calico. To prevent conflicting rules from coexisting and breaking ARTESCA networking, do not attempt to use other iptables-based firewalling technologies, such as firewalld.
Network Definition¶
ARTESCA distinguishes internal traffic into two networks: control plane and workload plane. For implementing its firewall, ARTESCA combines two types of resources:
HostEndpoints, which describe a host interface with a set of IP addresses
GlobalNetworkPolicies, which are rulesets for filtering IP traffic traversing selected HostEndpoints
When ARTESCA enables the host-level firewall, it does so by creating HostEndpoints: one for workload plane, and one for control plane (if it differs, otherwise only the workload endpoint is created). Once a HostEndpoint exists, only explicitly allowed communications can transit through its interface.
Important
Currently, ARTESCA assumes that all data virtual IPs belong on the same HostEndpoint as the workload plane real IP.
Refer to the change data listening IP procedure for hardware or software for more information.
Respectively, the management IP is declared on all control plane HostEndpoint. Consequently, policies defined for each plane are applied to their corresponding virtual IPs.
Refer to the change management IP procedure for hardware or software for more information.
Default Policies¶
ARTESCA defines firewall rules for network traffic transiting through host interfaces. Although 3 node deployments have almost identical nodes (same services running on each one), the rules are defined to apply on node endpoints depending on the node roles (e.g. “master” nodes run the Kubernetes control plane services).
These rules will be detailed below, and can be summed up by the following tables (destination/source can also be filtered, refer to detailed explanations below):
Policy workload-plane
Network |
Role |
Protocol(Port) |
Traffic Type |
Direction(s) |
Source |
Destination |
|---|---|---|---|---|---|---|
workload plane |
(any) |
TCP(:80) |
Ingress HTTP |
Ingress |
(any) |
N/A |
workload plane |
(any) |
TCP(:443) |
Ingress HTTPS |
Ingress |
(any) |
N/A |
workload plane |
(any) |
TCP(:179) |
Calico BGP |
Ingress |
In cluster |
N/A |
workload plane |
(any) |
VRRP() |
keepalived |
Ingress, Egress |
N/A |
N/A |
workload plane |
(any) |
TCP(*) |
All Egress TCP |
Egress |
N/A |
(any) |
workload plane |
(any) |
UDP(*) |
All Egress UDP |
Egress |
N/A |
(any) |
Policy control-plane-bootstrap
Network |
Role |
Protocol(Port) |
Traffic Type |
Direction(s) |
Source |
Destination |
|---|---|---|---|---|---|---|
control plane |
bootstrap |
TCP(:4505) |
Salt publisher |
Ingress |
(any) |
N/A |
control plane |
bootstrap |
TCP(:4506) |
Salt request server |
Ingress |
(any) |
N/A |
control plane |
bootstrap |
TCP(:4507) |
Salt API |
Ingress |
(any) |
N/A |
control plane |
bootstrap |
TCP(:8428) |
Artifacts server |
Ingress |
(any) |
N/A |
control plane |
bootstrap |
TCP(:22) |
SSH |
Egress |
N/A |
In cluster |
Policy control-plane-registry
Network |
Role |
Protocol(Port) |
Traffic Type |
Direction(s) |
Source |
Destination |
|---|---|---|---|---|---|---|
control plane |
registry |
TCP(:8080) |
Registry (containers and OS packages) |
Ingress |
(any) |
N/A |
control plane |
registry |
TCP(:8428) |
Artifacts server |
Egress |
N/A |
Bootstrap node |
Policy control-plane-master
Network |
Role |
Protocol(Port) |
Traffic Type |
Direction(s) |
Source |
Destination |
|---|---|---|---|---|---|---|
control plane |
master |
TCP(:6443) |
Kubernetes API server |
Ingress |
(any) |
N/A |
control plane |
master |
TCP(:8443) |
Control plane Ingress controller |
Ingress |
(any) |
N/A |
control plane |
master |
TCP(:10257) |
Kubernetes controller manager metrics |
Ingress |
Infra nodes |
N/A |
control plane |
master |
TCP(:10259) |
Kubernetes scheduler metrics |
Ingress |
Infra nodes |
N/A |
Policy control-plane-etcd
Network |
Role |
Protocol(Port) |
Traffic Type |
Direction(s) |
Source |
Destination |
|---|---|---|---|---|---|---|
control plane |
etcd |
TCP(:2379) |
etcd client |
Ingress |
Etcd nodes |
N/A |
control plane |
etcd |
TCP(:2380) |
etcd peer |
Ingress, Egress |
Etcd nodes |
Etcd nodes |
control plane |
etcd |
TCP(:2381) |
etcd metrics |
Ingress |
Infra nodes |
N/A |
Policy control-plane-infra
Network |
Role |
Protocol(Port) |
Traffic Type |
Direction(s) |
Source |
Destination |
|---|---|---|---|---|---|---|
control plane |
infra |
TCP(:10257) |
Kubernetes controller manager metrics |
Egress |
N/A |
Master nodes |
control plane |
infra |
TCP(:10259) |
Kubernetes scheduler metrics |
Egress |
N/A |
Master nodes |
control plane |
infra |
TCP(:2381) |
etcd metrics |
Egress |
N/A |
Etcd nodes |
control plane |
infra |
TCP(:9100) |
Node exporter (metrics) |
Egress |
N/A |
In cluster |
control plane |
infra |
TCP(:10249) |
Kubernetes proxy metrics |
Egress |
N/A |
In cluster |
control plane |
infra |
TCP(:10250) |
Kubelet metrics |
Egress |
N/A |
In cluster |
Policy control-plane-general
Network |
Role |
Protocol(Port) |
Traffic Type |
Direction(s) |
Source |
Destination |
|---|---|---|---|---|---|---|
control plane |
(any) |
TCP(:22) |
SSH |
Ingress |
Bootstrap node |
N/A |
control plane |
(any) |
TCP(:9100) |
Node exporter (metrics) |
Ingress |
Infra nodes |
N/A |
control plane |
(any) |
TCP(:10249) |
Kubernetes proxy metrics |
Ingress |
Infra nodes |
N/A |
control plane |
(any) |
TCP(:10250) |
Kubelet API |
Ingress, Egress |
In cluster |
(any) |
control plane |
(any) |
TCP(:10256) |
Kubernetes proxy healthcheck |
Ingress, Egress |
In cluster |
In cluster |
control plane |
(any) |
TCP(:2379) |
etcd client |
Egress |
N/A |
In cluster |
control plane |
(any) |
TCP(:6443) |
Kubernetes API server |
Egress |
N/A |
In cluster |
control plane |
(any) |
TCP(:4505) |
Salt publisher |
Egress |
N/A |
Bootstrap node |
control plane |
(any) |
TCP(:4506) |
Salt request server |
Egress |
N/A |
Bootstrap node |
control plane |
(any) |
TCP(:8080) |
Registry (containers and OS packages) |
Egress |
N/A |
Registry nodes |
control plane |
(any) |
TCP(:8443) |
Control plane Ingress controller |
Egress |
N/A |
Master nodes |
control plane |
(any) |
TCP(*) |
All Egress TCP |
Egress |
N/A |
(any) |
control plane |
(any) |
UDP(*) |
All Egress UDP |
Egress |
N/A |
(any) |
Workload Plane¶
Workload plane traffic can be divided into three categories:
Ingress requests to workload services (S3 API for instance)
Internal communication between pods
Internal communication between hosts (only for VIP management)
Note
Control and Workload planes are fully open for egress traffic, to allow access to external services (e.g. to retrieve Scality artifacts).
Virtual Network for Pods and Services¶
ARTESCA manages a virtual network, where each pod has its own virtual interface, by forwarding traffic through the workload plane network. This allows pods to communicate with each other without them needing to use real IPs and reserve host-level ports.
To ensure this implementation is secure, ARTESCA enforces that only registered pod IPs are allowed to be forwarded, and further source and destination checks are implemented by default for pod-to-pod communication, using NetworkPolicy Kubernetes resources.
Such traffic does not need to be explicitly allowed with policy rules, as demonstrated below for other kinds of traffic.
Host Network¶
At the host level, the workload plane network is used for BGP between Calico nodes (to implement the aforementioned virtual network), VRRP between keepalived instances (for balancing the workload plane VIPs), and ingress traffic (e.g. accessing S3 endpoints).
The workload-plane policy:
applies to workload plane endpoints
has order 200
allows ingress traffic for:
TCP to port(s) 80, 443 (used for workload plane ingress)
112 (used for keepalived VRRP)
TCP to port(s) 179 from workload plane endpoints (used for Calico BGP)
allows egress traffic for:
112 (used for keepalived VRRP)
TCP (used for all TCP egress)
UDP (used for all UDP egress)
This schema depicts ingress traffic received by the Ingress controller (on ports 80 and 443), or rejected when attempting to reach another process on another port:
Control Plane¶
Some control plane services are required to be exposed directly on the host network, since they predate the existence of the virtual network described above.
Important
Only a subset of the services exposed on the host network require explicit rules, because other such services never receive requests from other host-level services.
Bootstrap Services¶
Warning
SSH from bootstrap using default metalk8s key is forbidden.
The first server installed by ARTESCA exposes a few specific services needed for host management (implemented with Saltstack). This first server also requires SSH access to other machines.
The control-plane-bootstrap policy:
applies to control plane endpoints with role(s) bootstrap
has order 100
allows ingress traffic for:
TCP to port(s) 4505, 4506, 4507 from control plane endpoints (used for salt-master)
TCP to port(s) 8428 (used for artifacts-server)
UDP to port(s) 123 (used for NTP for hardware appliances)
allows egress traffic for:
TCP to control plane endpoints with port(s) 22 (used for ssh)
This schema shows all bootstrap-related traffic through control plane:
Another service, which gets replicated to other hosts, is the registry, responsible for serving system packages and container images to the cluster.
The control-plane-registry policy:
applies to control plane endpoints serving the registry
has order 103
allows ingress traffic for:
TCP to port(s) 8080 (used for registry)
In the schema below, you can see how any node can query any a registry instance, and how registry replicas will query the bootstrap node to retrieve their artifacts:
Kubernetes Control Plane¶
ARTESCA is running on Kubernetes, which needs a few services to communicate with each other for high availability.
The control-plane-master policy:
applies to control plane endpoints with role(s) master
has order 101
allows ingress traffic for:
TCP to port(s) 6443 from control plane endpoints (used for kube-apiserver)
TCP to port(s) 8443 (used for control-plane ingress)
TCP to port(s) 10257 from control plane endpoints with role(s) infra (used for kube-controller-manager metrics)
TCP to port(s) 10259 from control plane endpoints with role(s) infra (used for kube-scheduler metrics)
Here’s an overview of the traffic between kube-apiserver and kubelet on various nodes:
Kubernetes requires a distributed persistence layer, implemented with etcd. Its members communicate on the host network, since they must exist before Kubernetes API can, which in turn means the virtual network isn’t available.
The control-plane-etcd policy:
applies to control plane endpoints with role(s) etcd
has order 102
allows ingress traffic for:
TCP to port(s) 2379 from control plane endpoints with role(s) etcd (used for etcd client)
TCP to port(s) 2380 from control plane endpoints (used for etcd peer)
TCP to port(s) 2381 from control plane endpoints with role(s) infra (used for etcd metrics)
allows egress traffic for:
TCP to port(s) 2380 from control plane endpoints with role(s) etcd (used for etcd peer)
Metrics¶
Some components export their metrics on the control plane host address, which are scraped by Prometheus.
The control-plane-infra policy:
applies to control plane endpoints with role(s) infra
has order 104
allows egress traffic for:
TCP to control plane endpoints with role(s) master with port(s) 10257 (used for kube-controller-manager metrics)
TCP to control plane endpoints with role(s) master with port(s) 10259 (used for kube-scheduler metrics)
TCP to control plane endpoints with role(s) etcd with port(s) 2381 (used for etcd metrics)
TCP to port(s) 9100 from control plane endpoints (used for node-exporter)
TCP to control plane endpoints with port(s) 10249 (used for kube-proxy metrics)
TCP to control plane endpoints with port(s) 10250 (used for kubelet metrics)
Common Rules¶
Some services run on each ARTESCA server, such as kubelet, which receive ingress traffic from other servers, or salt-minion, which emit egress traffic to the bootstrap server.
The control-plane-general policy:
applies to control plane endpoints
has order 105
allows ingress traffic for:
TCP to port(s) 10250 from control plane endpoints (used for kubelet)
TCP to port(s) 10256 from control plane endpoints (used for kube-proxy healthcheck)
TCP to port(s) 9100 from control plane endpoints with role(s) infra (used for node-exporter)
TCP to port(s) 10249 from control plane endpoints with role(s) infra (used for kube-proxy metrics)
TCP to port(s) 22 from control plane endpoints with role(s) bootstrap (used for ssh)
allows egress traffic for:
TCP to port(s) 2379 from control plane endpoints (used for etcd client)
TCP to control plane endpoints with role(s) bootstrap with port(s) 4505, 4506 (used for saltstack)
TCP to control plane endpoints with port(s) 6443 (used for kube-apiserver)
TCP to control plane endpoints serving the registry with port(s) 8080 (used for registry)
TCP to control plane endpoints with role(s) bootstrap with port(s) 8428 (used for artifacts-server)
TCP to port(s) 10250 (used for kubelet)
TCP to port(s) 10256 from control plane endpoints (used for kube-proxy healthcheck)
TCP to control plane endpoints with role(s) master with port(s) 8443 (used for control-plane ingress)
UDP to port(s) 123 (used for all NTP egress)
TCP (used for all TCP egress)
UDP (used for all UDP egress)
