Mas Cloud Native Blog: July 2020

I got the following error when trying to do my install:

[root@unicamaster Helm-updated]# desc hcl-unica-platform-75d94f5546-grzmw

Name: hcl-unica-platform-75d94f5546-grzmw

Namespace: default

Priority: 0

Node: unicanode1/192.168.1.252

Start Time: Fri, 31 Jul 2020 02:59:19 +0800

Labels: app.kubernetes.io/instance=hcl

app.kubernetes.io/name=unica-platform

pod-template-hash=75d94f5546

Annotations: <none>

Status: Pending

IP: 10.44.0.9

IPs:

IP: 10.44.0.9

Controlled By: ReplicaSet/hcl-unica-platform-75d94f5546

Init Containers:

install:

Container ID:

Image: 192.168.1.251:5000/unica-install:68

Image ID:

Port: <none>

Host Port: <none>

Command:

/bin/sh

-c

Args:

chmod 755 /docker/unica && ./entrypoint.sh

State: Waiting

Reason: ImagePullBackOff

Ready: False

Restart Count: 0

Environment Variables from:

hcl-unica-common-env ConfigMap Optional: false

hcl-unica-platform-env ConfigMap Optional: false

hcl-unica-campaign-env ConfigMap Optional: false

hcl-unica-plan-env ConfigMap Optional: false

hcl-unica-interact-env ConfigMap Optional: false

hcl-unica-offer-env ConfigMap Optional: false

hcl-unica-birt-env ConfigMap Optional: false

Environment: <none>

Mounts:

/docker/unica from volume-mount (rw)

/var/run/secrets/kubernetes.io/serviceaccount from default-token-whztp (ro)

Containers:

unica-platform:

Container ID:

Image: 192.168.1.251:5000/tomcat-unica-platform:12

Image ID:

Port: 9123/TCP

Host Port: 0/TCP

State: Waiting

Reason: PodInitializing

Ready: False

Restart Count: 0

Readiness: exec [/bin/bash -ce ] delay=20s timeout=30s period=40s #success=1 #failure=3

Environment Variables from:

hcl-unica-common-env ConfigMap Optional: false

hcl-unica-platform-env ConfigMap Optional: false

hcl-unica-campaign-env ConfigMap Optional: false

hcl-unica-plan-env ConfigMap Optional: false

hcl-unica-interact-env ConfigMap Optional: false

hcl-unica-offer-env ConfigMap Optional: false

hcl-unica-birt-env ConfigMap Optional: false

Environment:

CONTAINER_NAME: unica-platform

WLP_HTTP_PORT: 9123

WLP_HTTPS_PORT: 9445

Mounts:

/docker/unica from volume-mount (rw)

/var/run/secrets/kubernetes.io/serviceaccount from default-token-whztp (ro)

Conditions:

Type Status

Initialized False

Ready False

ContainersReady False

PodScheduled True

Volumes:

volume-mount:

Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

ClaimName: hcl-unica

ReadOnly: false

default-token-whztp:

Type: Secret (a volume populated by a Secret)

SecretName: default-token-whztp

Optional: false

QoS Class: BestEffort

Node-Selectors: <none>

Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s

node.kubernetes.io/unreachable:NoExecute for 300s

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Scheduled <unknown> default-scheduler Successfully assigned default/hcl-unica-platform-75d94f5546-grzmw to unicanode1

Normal Pulling 27s (x3 over 108s) kubelet, unicanode1 Pulling image "192.168.1.251:5000/unica-install:68"

Warning Failed 27s (x3 over 70s) kubelet, unicanode1 Failed to pull image "192.168.1.251:5000/unica-install:68": rpc error: code = Unknown desc = Error while pulling image: Get http://192.168.1.251:5000/v1/repositories/unica-install/images: dial tcp 192.168.1.251:5000: connect: connection refused

Warning Failed 27s (x3 over 70s) kubelet, unicanode1 Error: ErrImagePull

Normal BackOff 0s (x4 over 69s) kubelet, unicanode1 Back-off pulling image "192.168.1.251:5000/unica-install:68"

Warning Failed 0s (x4 over 69s) kubelet, unicanode1 Error: ImagePullBackOff

It turns out my registry is not running:

[root@unicamaster unica.12.0.0.1]# docker push 192.168.1.251:5000/unica-install:68

The push refers to a repository [192.168.1.251:5000/unica-install]

Put http://192.168.1.251:5000/v1/repositories/unica-install/: dial tcp 192.168.1.251:5000: connect: connection refused

Do a docker ps to confirm:

[root@unicamaster Helm-updated]# docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

561d0bebfaef k8s.gcr.io/coredns@sha256:2c8d61c46f484d881db43b34d13ca47a269336e576c81cf007ca740fa9ec0800 "/coredns -conf /e..." 2 hours ago Up 2 hours k8s_coredns_coredns-66bff467f8-28rmt_kube-system_7f1f52e1-e298-4619-b8ad-e191c0cc8f34_0

d5a99cd720a3 k8s.gcr.io/pause:3.2 "/pause" 2 hours ago Up 2 hours k8s_POD_coredns-66bff467f8-28rmt_kube-system_7f1f52e1-e298-4619-b8ad-e191c0cc8f34_0

1c8df48edf2c docker.io/weaveworks/weave-npc@sha256:0f6166e000faa500ccc0df53caae17edd3110590b7b159007a5ea727cdfb1cef "/usr/bin/launch.sh" 2 hours ago Up 2 hours k8s_weave-npc_weave-net-qz7bf_kube-system_ad764c24-49eb-489e-974b-eb89fb11f1af_0

74b52284da08 docker.io/weaveworks/weave-kube@sha256:703a045a58377cb04bc85d0f5a7c93356d5490282accd7e5b5d7a99fe2ef09e2 "/home/weave/launc..." 2 hours ago Up 2 hours k8s_weave_weave-net-qz7bf_kube-system_ad764c24-49eb-489e-974b-eb89fb11f1af_0

7d9b898e6c3c k8s.gcr.io/pause:3.2 "/pause" 2 hours ago Up 2 hours k8s_POD_weave-net-qz7bf_kube-system_ad764c24-49eb-489e-974b-eb89fb11f1af_0

c4e408af5a01 c3d62d6fe412 "/usr/local/bin/ku..." 2 hours ago Up 2 hours k8s_kube-proxy_kube-proxy-pw8v7_kube-system_7090da86-7f12-4e98-8f37-0c877123ea58_4

36003ec653c5 k8s.gcr.io/pause:3.2 "/pause" 2 hours ago Up 2 hours k8s_POD_kube-proxy-pw8v7_kube-system_7090da86-7f12-4e98-8f37-0c877123ea58_4

cde8fb2ca72f 56acd67ea15a "kube-apiserver --..." 2 hours ago Up 2 hours k8s_kube-apiserver_kube-apiserver-unicamaster_kube-system_45e531058954ac50479dd9379a75b049_3

16b77b15f562 ffce5e64d915 "kube-controller-m..." 2 hours ago Up 2 hours k8s_kube-controller-manager_kube-controller-manager-unicamaster_kube-system_a498455ef3ba2eb7d5fd01ed8271dea6_4

4efbb4b34108 303ce5db0e90 "etcd --advertise-..." 2 hours ago Up 2 hours k8s_etcd_etcd-unicamaster_kube-system_2beb97d61fe2d8b34880654b5e6314ff_4

575e2943ec29 0e0972b2b5d1 "kube-scheduler --..." 2 hours ago Up 2 hours k8s_kube-scheduler_kube-scheduler-unicamaster_kube-system_3dd66788a2c7782d910d05ea37b91678_4

baf40fb36f52 k8s.gcr.io/pause:3.2 "/pause" 2 hours ago Up 2 hours k8s_POD_kube-apiserver-unicamaster_kube-system_45e531058954ac50479dd9379a75b049_4

c0b6e188820b k8s.gcr.io/pause:3.2 "/pause" 2 hours ago Up 2 hours k8s_POD_kube-scheduler-unicamaster_kube-system_3dd66788a2c7782d910d05ea37b91678_4

11fbfd0936a5 k8s.gcr.io/pause:3.2 "/pause" 2 hours ago Up 2 hours k8s_POD_kube-controller-manager-unicamaster_kube-system_a498455ef3ba2eb7d5fd01ed8271dea6_4

48994c1405f4 k8s.gcr.io/pause:3.2 "/pause" 2 hours ago Up 2 hours k8s_POD_etcd-unicamaster_kube-system_2beb97d61fe2d8b34880654b5e6314ff_4

1448a8fa1aae ffce5e64d915 "kube-controller-m..." 47 hours ago Exited (2) 2 hours ago k8s_kube-controller-manager_kube-controller-manager-unicamaster_kube-system_a498455ef3ba2eb7d5fd01ed8271dea6_3

dacda2fa2526 k8s.gcr.io/pause:3.2 "/pause" 47 hours ago Exited (0) 2 hours ago k8s_POD_kube-controller-manager-unicamaster_kube-system_a498455ef3ba2eb7d5fd01ed8271dea6_3

d78ca1419012 0e0972b2b5d1 "kube-scheduler --..." 47 hours ago Exited (2) 2 hours ago k8s_kube-scheduler_kube-scheduler-unicamaster_kube-system_3dd66788a2c7782d910d05ea37b91678_3

b5bd2e6284ba k8s.gcr.io/pause:3.2 "/pause" 47 hours ago Exited (0) 2 hours ago k8s_POD_kube-scheduler-unicamaster_kube-system_3dd66788a2c7782d910d05ea37b91678_3

321a1e5074b0 56acd67ea15a "kube-apiserver --..." 47 hours ago Exited (0) 2 hours ago k8s_kube-apiserver_kube-apiserver-unicamaster_kube-system_45e531058954ac50479dd9379a75b049_2

7e2de05ead5c c3d62d6fe412 "/usr/local/bin/ku..." 47 hours ago Exited (2) 2 hours ago k8s_kube-proxy_kube-proxy-pw8v7_kube-system_7090da86-7f12-4e98-8f37-0c877123ea58_3

1c4220043c1f 303ce5db0e90 "etcd --advertise-..." 47 hours ago Exited (0) 2 hours ago k8s_etcd_etcd-unicamaster_kube-system_2beb97d61fe2d8b34880654b5e6314ff_3

d087f25acde8 k8s.gcr.io/pause:3.2 "/pause" 47 hours ago Exited (0) 2 hours ago k8s_POD_kube-apiserver-unicamaster_kube-system_45e531058954ac50479dd9379a75b049_3

345519cf5a8c k8s.gcr.io/pause:3.2 "/pause" 47 hours ago Exited (0) 2 hours ago k8s_POD_etcd-unicamaster_kube-system_2beb97d61fe2d8b34880654b5e6314ff_3

bad7f6b45de0 k8s.gcr.io/pause:3.2 "/pause" 47 hours ago Exited (0) 2 hours ago k8s_POD_kube-proxy-pw8v7_kube-system_7090da86-7f12-4e98-8f37-0c877123ea58_3

8111ced3b898 registry:2 "/entrypoint.sh /e..." 2 days ago Exited (2) 2 hours ago registry

b6ac25ad59d1 hello-world "/hello" 3 days ago Exited (0) 3 days ago hungry_lumiere

[root@unicamaster Helm-updated]# ^C

I removed registry:

[root@unicamaster Helm-updated]# docker rm registry

registry

But cannot recreate it:

[root@unicamaster Helm-updated]# docker run -d -p 5000:5000 --restart=always --name registry registry:2

b03e7c2b76d9dc3ce2c222e8ccc30d10cc09ac3c31f45105b16424718750fb59

/usr/bin/docker-current: Error response from daemon: driver failed programming external connectivity on endpoint registry (8dcc7fdb228341ae1222c052905cdce61086380e21c3b6289c7f35e148696535): (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 5000 -j DNAT --to-destination 172.17.0.2:5000 ! -i docker0: iptables: No chain/target/match by that name.

(exit status 1)).

firewalld needs to be up

https://success.docker.com/article/iptables-error-when-starting-container-with-docker-run

but i can't start it:

[root@unicamaster Helm-updated]# systemctl status firewalld.service

● firewalld.service - firewalld - dynamic firewall daemon

Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)

Active: inactive (dead)

Docs: man:firewalld(1)

[root@unicamaster Helm-updated]# systemctl start firewalld.service

Job for firewalld.service failed because a timeout was exceeded. See "systemctl status firewalld.service" and "journalctl -xe" for details.

journalctl -xe

Jul 31 05:50:22 unicamaster systemd[1]: Failed to start firewalld - dynamic firewall daemon.

-- Subject: Unit firewalld.service has failed

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit firewalld.service has failed.

-- The result is failed.

Jul 31 05:50:22 unicamaster systemd[1]: Unit firewalld.service entered failed state.

Jul 31 05:50:22 unicamaster systemd[1]: firewalld.service failed.

Jul 31 05:50:22 unicamaster systemd[1]: Reached target Network (Pre).

-- Subject: Unit network-pre.target has finished start-up

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit network-pre.target has finished starting up.

-- The start-up result is done.

Found this solution:

https://www.seimaxim.com/clients/knowledgebase/42/firewalld-service-is-failing-and-unable-to-start-in-RHEL-7.html

Check if any extra unwanted firewalld processes are still running on the system:

# ps -ef|grep firewalld |grep -v grep
root      2643     1 19 12:03 ?        00:00:02 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
root     11429     1  0 Jun13 ?        00:00:04 /usr/bin/python -Es /usr/sbin/firewalld

In the above example, two processes of firewalld are running; one of them is an old process with PID=11429 (running since June 13). Kill all the running processes:
```
# kill -KILL <PID>
```
Check if the iptables service is running:
```
# systemctl status iptables.service
```

If yes, then stop and disable the service:

# systemctl stop iptables.service
# systemctl disable iptables.service

Restart the firewalld service:
```
# systemctl start firewalld.service
```

And now it can start:

[root@unicamaster Helm-updated]# ps -ef|grep firewalld |grep -v grep

root 21659 1 0 03:13 ? 00:00:00 /usr/bin/python -Es /usr/sbin/firewalld

[root@unicamaster Helm-updated]# kill -9 21659

[root@unicamaster Helm-updated]# ps -ef|grep firewalld |grep -v grep

[root@unicamaster Helm-updated]# systemctl status iptables.service

Unit iptables.service could not be found.

[root@unicamaster Helm-updated]# systemctl start firewalld.service

[root@unicamaster Helm-updated]# systemctl status iptables.service

Unit iptables.service could not be found.

[root@unicamaster Helm-updated]# systemctl status firewalld.service

● firewalld.service - firewalld - dynamic firewall daemon

Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)

Active: active (running) since Fri 2020-07-31 06:01:43 +08; 12s ago

Docs: man:firewalld(1)

Main PID: 16350 (firewalld)

Tasks: 2

CGroup: /system.slice/firewalld.service

└─16350 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Jul 31 06:01:43 unicamaster systemd[1]: Starting firewalld - dynamic firewall daemon...

Jul 31 06:01:43 unicamaster systemd[1]: Started firewalld - dynamic firewall daemon.

After that reboot and problem solved

Mas Cloud Native Blog

Thursday, 30 July 2020

Useful commands

Failed to pull image

Tuesday, 28 July 2020

docker push http: server gave HTTP response to HTTPS client

PVC in terminating state

Report Abuse