vedchitnis

Actual excerpt from our chat which led to this TIL:

If the pod speaks and no service hears it, did it really return a 200 OK?

I was building a feature where one app (let’s call it publisher) needed to broadcast a message to all pods of another app (listener) in a k8s cluster AND listen to what the listener pods have to say!

Seems simple, right? Just use the k8s service DNS and call it a day?

Hah. No.

Yes, I know I could’ve decoupled this system — added a message queue, maybe some external broker, something neat and robust. I even considered explaining it away as needing a fairly synchronous fan-out pattern that gives me back immediate responses from each pod…

But honestly, I just wanted it to be easy and quick :D

default behaviour

When I hit http://listener.default.svc.cluster.local, k8s did what it does best — load balance the request to a single pod behind that service.

So even with n replicas of listener, only one would get the call. That’s not broadcasting. That’s whispering into a crowd and hoping the right person hears you, then possibly needing to repeat this multiple times so that all the people hear you at least once, not the best approach.

goal

Send an API call (like /metrics) to every listener pod, collect their responses, and do something useful with it.
So, broadcasting. But the real kind.

plan

Use the Kubernetes API inside publisher to list all pods with the given labels
Grab each pod’s IP address
Make a direct HTTP call to each pod:
http://<pod-ip>:<port>/metrics
Aggregate responses.
Handle errors gracefully (I need to handle pods that might fail).

here we go

I made the /broadcast API filterable to allow for some extension later on:

By default, it broadcasts to all pods with app=listener
But I can pass custom filters like:
```
/broadcast?label=app=listener&label=zone=us-east
```
and it will only ping pods matching all those labels.

Each listener pod exposes a /metrics endpoint like:

{
  "pod": "listener-x4f8s",
  "timestamp": "2025-04-12T14:23:41Z",
  "memoryMB": 91.2,
  "anyOther_Data_I_Want": {}
}

So when I hit /broadcast, the publisher stitches together a JSON list of all their responses. Something like:

[
  { "pod": "listener-x4f8s", "memoryMB": 91.2 },
  { "pod": "listener-a9v4h", "memoryMB": 93.7 },
  { "pod": "listener-n4g0k", "memoryMB": 90.5 }
]

notes

Kubernetes services aren’t meant for broadcasting — they route traffic to a single pod per request.
To actually broadcast, I had to interact with the Kubernetes API directly and fetch pod IPs.
Feign clients and service discovery weren’t helpful here. I needed plain RestTemplate/WebClient fan-out.
Label-based filtering gave me runtime targeting control over which pods to include.

code

You can find the full source code, deployment YAMLs here.

Here’s a brief walkthrough of how I did this:

1. The /metrics endpoint in listener:

@GetMapping("/metrics")
public Map<String, Object> metrics() {
    Map<String, Object> data = new HashMap<>();
    data.put("pod", hostname);
    data.put("timestamp", Instant.now().toString());
    data.put("memoryMB", getMemoryInMB());
    return data;
}

2. Using the Kubernetes API in publisher to list pods:

String labelSelector = (label != null && !label.isEmpty())
		? String.join(",", label)
		: "app=listener";

V1PodList pods = coreV1Api.listNamespacedPod(
		"default", null, null, null, null,
		labelSelector, null, null, null, null, null);

3. Broadcasting requests to each pod:

List<Map<String, Object>> results = new ArrayList<>();
RestTemplate restTemplate = new RestTemplate();
for (V1Pod pod : pods.getItems()) {
	String podIp = pod.getStatus().getPodIP();
	try {
		Map response = restTemplate.getForObject(
				"http://" + podIp + ":8080/metrics", Map.class);
		results.add(response);
	} catch (Exception e) {
		Map<String, Object> failed = new HashMap<>();
		failed.put("pod", podIp);
		failed.put("error", "unreachable");
		results.add(failed);
	}
}

This gave me a dead-simple way to fan out requests in a cluster with full control over error handling and response shaping.

What’s Next?

Now that the broadcasting works reliably, it’s time to introduce some failure — on purpose.

Again another excerpt from the group chat

Honestly no clue, Some pods will respond.
Some will return errors.
Some will delay just enough or more to cause a timeout.

Coming soon: “TIL: When My Posds Got Moody”

Shouting in a k8s void