eBPF: Cilium on FHIR® - A Star Wars Story

Anakin Skywalker challenged the high ground and has been terribly injured on Mustafar. He is a relatively new employee of the Galactic Empire, covered by a Large Group Planetary Plan and now has an active encounter in progress for emergent medical services in the Grand Medical Facility on Coruscant. The EMR deployed for the Galactic Health System is Powered by InterSystems FHIR Server running on Kubernetes protected by Cilium.

Let’s recreate the technical landscape, to be performed in the style of Isovalent Labs…

Kind Cluster

Let’s fire up a 3-node cluster, and disable the CNI so we can replace it with Cilium.

cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
networking:
  disableDefaultCNI: true
EOF

This will provision the kind cluster, 3 nodes wide with a single control plane.

Cilium

Cilium is an open-source project that provides networking, security, and observability for containerized environments. It uses a Linux kernel technology called eBPF (extended Berkeley Packet Filter) to inject security, networking, and observability logic into the kernel. In other words, it wields the force.

cilium install --version v1.16.0
cilium status --wait

Hubble

Hubble is a “clown suit” for Cilium, providing ridiculous visibility to what powers Cilium are in play in real time.

cilium enable hubble

InterSystems FHIR Workload

InterSystems is the GOAT of interoperability, and transforms Healthcare Data like a protocol Droid.

kubectl apply -f https://raw.githubusercontent.com/sween/basenube/main/scenarios/ciliumfhir/deploy/cilium-fhir-starwars.yaml

The resulting workload has 4 deployments:

GrandMedicalFacility: Integrated Delivery Network based in Coruscant, runs Epic and utilizes InterSystems I4H as a FHIR Server.
MedicalDroid FX-6: Supplied Vader with a blood transfusion and trained in cybernetic procedures.
MedicalDroid DD-13: Three-legged droid designed to install cybernetic implants.
MedicalDroid 2-1B: Specialized in neurosurgery and cybernetic limb replacement.

Sith-uation

Palpatine accompanied the fallen jedi to the facility, and upon arrival helped registration admit him as Darth Vader.

cat > vader.json << 'EOF'
{
  "name": [
    {
      "use": "official",
      "family": "Vader",
      "given": [
        "Darth"
      ]
    }
  ],
  "gender": "male",
  "id": "DarthVader",
  "birthDate": "1977-05-25",
  "resourceType": "Patient"
}
EOF

curl -v -X PUT \
  -H "Content-Type: application/fhir+json" \
  -d @vader.json \
  "http://coruscanthealth:52773/intersystems/fhir/r5/Patient/DarthVader"

Darth Vader is now registered, and can be seen throughout the Health System…

Vader Registered

There is a problem though!

Shortly after registration, a Galactic IT Outage has occurred, making the Identity Provider for the Health System unavailable. The InterSystems FHIR Resource Server is SMART enabled, and since the IDP is down, EMR launches are impossible without the necessary JWT token.

Hubble Map

The care team cannot access the patient record, getting nothing but 401’s and 403’s. Inspecting the Hubble flows with Layer 7 information reveals the sithuation.

L7 Flows

Adding some debug logging to the InterSystems FHIR endpoint confirms it:

zn "USER"
Set ^FSLogChannel("all")=1
zn "%SYS"
Set ^%ISCLOG=5
Set ^%ISCLOG("Category","HSFHIR")=5
Set ^%ISCLOG("Category","HSFHIRServer")=5
Set ^%ISCLOG("Category","OAuth2")=5
Set ^%ISCLOG("Category","OAuth2Server")=5

zw^FSLOG
...
^FSLOG(379555)="DispatchRequest^HS.FHIRServer.Service^944|Msg|Dispatch interaction read for Patient/DarthVader|09/19/2024 10:48:20.833339AM"
^FSLOG(379556)="DispatchRequest^HS.FHIRServer.Service^944|Msg|Request Completed in .000186 secs: Patient/DarthVader|09/19/2024 10:48:20.833450AM"
^FSLOG(379557)="processRequest^HS.FHIRServer.RestHandler^944|Msg|Response Status: 401, Json: Patient|09/19/2024 10:48:20.833454AM"
...

With SMART route enforcement unavailable, we’ll use Cilium to protect the endpoints while Vader gets immediate attention. We’ll go Rogue One and hand off the route protection to Cilium.

First, institute a Deny All policy:

cat <<EOF | kubectl apply -n galactic-fhir -f-
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: "denyall-coruscanthealth"
spec:
  description: "Block all the traffic (except DNS) by default"
  egress:
  - toEndpoints:
    - matchLabels:
        io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: '53'
        protocol: UDP
      rules:
        dns:
        - matchPattern: '*'
  endpointSelector:
    matchExpressions:
    - key: io.kubernetes.pod.namespace
      operator: NotIn
      values:
      - kube-system
EOF

Cilium is now dropping traffic: Cilium Dropping Traffic

Now, let’s open up the FHIR endpoint on the InterSystems pod by disabling the OAuth2 client:

set app = "/intersystems/fhir/r5"
Set strategy = ##class(HS.FHIRServer.API.InteractionsStrategy).GetStrategyForEndpoint(app)
// 7 = Mass Openness
Set configData.DebugMode = 7
Set configData = strategy.GetServiceConfigData()
Do strategy.SaveServiceConfigData(configData)

Lastly, create a CiliumNetworkPolicy to allow pods with the label org: empire access to Darth Vader’s record:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l7-visibility"
spec:
  endpointSelector:
    matchLabels:
      org: empire
  egress:
  - toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": galactic-fhir
    toPorts:
    - ports:
      - port: "52773"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/intersystems/fhir/r5/Patient/DarthVader"
        - method: "HEAD"
          path: "/intersystems/fhir/r5/Patient/DarthVader"

Argo View

Hubble View

Looks like we may be able to get back to iRacing, I think we are good… except…

Yeah, looks like the payer is getting dropped: Policy Dropped

Let’s add another policy, allowing org: payer access to Vader’s route:

cat <<EOF | kubectl apply -n galactic-fhir -f-
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "l7-visibility-payer"
spec:
  endpointSelector:
    matchLabels:
      org: payer
  egress:
  - toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": galactic-fhir
    toPorts:
    - ports:
      - port: "52773"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/intersystems/fhir/r5/Patient/DarthVader"
        - method: "HEAD"
          path: "/intersystems/fhir/r5/Patient/DarthVader"
EOF