HasnainWaris

Continuing my Kubernetes series, this post focuses on scheduling – how Kubernetes decides where your pods actually run – along with monitoring and logging to keep track of what’s happening in your cluster. Understanding these concepts gives you control over pod placement and visibility into your cluster’s health.

Scheduling

Manual Scheduling

By default, Kubernetes uses its scheduler to place pods on nodes. But you can bypass this and manually schedule pods if needed.

Set the node at pod creation time in the spec:

spec:
  nodeName: node01
  containers:
  - image: nginx
    name: nginx

Important note: you can only set this at creation time. An existing pod can only be moved to a different node using a binding resource.

Labels and Selectors

Labels and selectors are how you group and filter resources in Kubernetes. They’re key-value pairs that help organize and identify resources.

Labels are set on resources like this:

metadata:
  labels:
    env: prod
    app: nginx
    tier: frontend

Selectors are used to call or filter based on those labels. You can have multiple labels on a resource and filter by any of them.

Filter pods by label with kubectl get pods --selector env=dev for example, filtering our pods where the env is set to dev. This becomes especially useful when working with services, deployments, and other resources that need to target specific pods.

Taints and Tolerations

Taints and tolerations control the relationship between pods and nodes – restricting or preferring where pods can be placed.

Taints are set on nodes to repel pods. Tolerations are set on pods to allow them onto tainted nodes.

Think of it like a security clearance system. The node has a restriction (taint), and only pods with the right permission (toleration) can be scheduled there.

Add a taint to a node with kubectl taint nodes node1 key1=value1:NoSchedule.

Important distinction: taints don’t guarantee a pod will land on a specific node – they just prevent pods without the right toleration from being scheduled there. The pod might still end up on another untainted node.

Node Selectors: Simple Pod-to-Node Assignment

Node selectors let you restrict pods to run on specific nodes based on labels.

First, label your node with kubectl label nodes node-name labelkey=value.

Then reference it in your pod spec:

spec:
  nodeSelector:
    labelkey: value

The limitation here is that it’s a simple one-to-one relationship. You can’t express complex logic like “run on nodes with label X OR Y” or “avoid nodes with label Z.” That’s where node affinity comes in.

Node Affinity: Advanced Scheduling

Node affinity is the more powerful way to control pod placement. It gives you complex matching rules and different enforcement levels.

Node Affinity Types:

requiredDuringSchedulingIgnoredDuringExecution – Hard requirement. The pod won’t be scheduled if the rules aren’t met. Like saying “MUST run on nodes with this label.”

preferredDuringSchedulingIgnoredDuringExecution – Soft preference. The scheduler tries to match but will still schedule the pod if it can’t. Like saying “I’d PREFER nodes with this label, but it’s okay if not.”

There’s also a planned type (not yet available): requiredDuringSchedulingRequiredDuringExecution – This would evict pods if node labels change and no longer match the requirements.

Key detail: “IgnoredDuringExecution” means if a pod is already running and the node’s labels change, the pod keeps running and won’t be evicted.

Operators Available:

• In – label value must be in the list
• NotIn – label value must NOT be in the list
• Exists – label key must exist (value doesn’t matter)

Example:

spec:
  containers:
  - image: nginx
    name: nginx
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: color
            operator: In
            values:
            - blue

This pod will only be scheduled on nodes labeled with color=blue.

Resource Requests and Limits

The scheduler considers resource availability before placing pods. You can specify how much CPU and memory your pod needs (requests) and the maximum it can use (limits).

Set resource requests in your pod spec:

resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "128Mi"
    cpu: "500m"

The scheduler uses requests to find nodes with enough available resources. Limits prevent pods from consuming more than their allocation.

If you see a pod with status OOMKilled, it means the pod ran out of memory and hit its limit. This is your signal to either optimize the application or increase the memory limit.

DaemonSets

While ReplicaSets help you run multiple instances of a pod across the cluster, DaemonSets ensure at least one replica runs on EACH node.

Common use cases include monitoring agents, log collectors, or networking components that need to run on every node in the cluster.

The YAML structure is similar to a ReplicaSet, and DaemonSets use the default scheduler to place pods.

Static Pods

On a node without a kube-apiserver – just kubelet – you can create static pods by placing YAML files in /etc/kubernetes/manifests. These are managed directly by kubelet, not through the API server.

You can only create pods this way, not other resources like deployments or services.

When you run kubectl get pods, static pods will have the node name appended to their name – that’s how you identify them.

Multiple Schedulers

Kubernetes lets you run multiple scheduler profiles. When creating a pod, you can specify which scheduler to use based on your needs. This is useful if you have custom scheduling requirements or want different scheduling logic for different workloads.

Priority Classes

You can set priority on pods to influence scheduling order. Higher priority pods get scheduled first.

Kubernetes has default priority classes, and you can create custom ones. The default priority value is 0 – higher numbers mean higher priority.

How Scheduling Actually Works

Understanding the scheduling phases helps when troubleshooting:

1. Scheduling Queue – Pod enters the queue, ordered by priority
2. Filtering – Nodes that don’t meet requirements are filtered out
3. Scoring – Remaining nodes are scored based on various factors
4. Binding – The highest-scoring node is selected and the pod is bound to it

Each phase has scheduling plugins handling that section of the process.

Monitoring Your Cluster

Now that we understand how pods get scheduled, let’s talk about keeping an eye on them once they’re running.

Kubernetes doesn’t come with a built-in monitoring solution, but there are solid open-source options like Metrics Server, Prometheus, and the Grafana stack.

Metrics Server is a cluster add-on that collects and aggregates resource usage data – CPU and memory – from all nodes and pods. It’s lightweight and perfect for basic monitoring needs.

Once Metrics Server is installed, you can use kubectl to view resource consumption.

kubectl top nodes shows CPU and memory usage across your nodes.

kubectl top pods shows resource usage for all pods in the current namespace.

This is incredibly useful for monitoring resource consumption, checking node health, and tracking pod performance. You can quickly spot which pods are consuming the most resources or if a node is running hot.

Managing Logs

Logs are critical for troubleshooting and understanding what’s happening inside your pods.

The basic command to view pod logs is kubectl logs pod-name.

If your pod has multiple containers, specify which one with kubectl logs pod-name -c container-name.

Follow logs in real-time with kubectl logs -f pod-name.

View logs from a previous instance of the container (useful if it crashed) with kubectl logs pod-name –previous.

Coming from a sysadmin background where I’m used to tailing /var/log files, kubectl logs takes some getting used to. But it’s actually more convenient – no SSHing into nodes, no hunting through directories. Just point kubectl at the pod and you’ve got your logs.

Troubleshooting Tips

• Use kubectl describe pod pod-name to see scheduling events and why a pod might be pending
• Check kubectl get events to view recent cluster events in a namespace
• Use kubectl logs for application logs and debugging runtime issues
• Combine kubectl top with kubectl describe to correlate resource usage with pod behavior

Wrapping Up

Scheduling in Kubernetes gives you control over where your workloads run – from simple node selectors to complex affinity rules, taints and tolerations, and resource management. Pair that with proper monitoring and logging, and you’ve got visibility into both placement decisions and runtime behavior.

These tools let you optimize for performance, cost, and availability while being able to troubleshoot issues when they inevitably come up.

In the next post, I’ll continue the series covering storage, ConfigMaps, Secrets, and more. See you then!