Wednesday, 10 June 2020

Troubleshooting networking (between nodes)

I think this is one of the hardest (maybe I don't know enough). But here is a good start:

https://gravitational.com/blog/troubleshooting-kubernetes-networking/

Basically, curl is a very important tool since most pods have very limited commands you can run.

And here are some very basic services and how it works within pods:
https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies

For discovering services, check out this DNS docs:
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

Deleting images in docker

docker images

docker rmi -f 1ee7db3527cf
docker images
docker rmi -f 2006138de05d
docker rmi -f 743901726c7f

docker rmi -f 093aaee064e4

Tuesday, 9 June 2020

About /etc/docker/daemon.json and /etc/default/docker

It needs to be on nodes as well.

/etc/docker/daemon.json


{
    "insecure-registries": ["10.0.2.6:5000"]
}

/etc/default/docker
DOCKER_OPTS="--insecure-registry 10.0.2.6:5000"

MountVolume.SetUp failed for volume "default-token-t7d5k" : failed to sync secret cache: timed out waiting for the condition

If you get the error "MountVolume.SetUp failed for volume "default-token-t7d5k" : failed to sync secret cache: timed out waiting for the condition":

[root@unicamaster software]# kubectl describe pod unica-platform-b5688dcb5-4wptl
Name:         unica-platform-b5688dcb5-4wptl
Namespace:    default
Priority:     0
Node:         unicanode1/10.0.2.5
Start Time:   Wed, 10 Jun 2020 07:16:42 +0530
Labels:       app.kubernetes.io/instance=unica
              app.kubernetes.io/name=unica-platform
              pod-template-hash=b5688dcb5
Annotations:  <none>
Status:       Pending
IP:           10.44.0.4
IPs:
  IP:           10.44.0.4
Controlled By:  ReplicaSet/unica-platform-b5688dcb5
Init Containers:
  install:
    Container ID:  docker://aff6ed9523034dfc36232aba8a5e9f8dacbfd1fa6ce4f697a1b23d57dfd31c18
    Image:         10.0.2.6:5000/unica-install:68
    Image ID:      docker-pullable://10.0.2.6:5000/unica-install@sha256:db069e6cbcd3370bbaa0e7bd80a29a83ab82e70b03a2537572ef27a6e4046233
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
    Args:
      chmod 755 /docker/unica && ./entrypoint.sh
    State:          Running
      Started:      Wed, 10 Jun 2020 07:17:50 +0530
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      unica-common-env    ConfigMap  Optional: false
      unica-platform-env  ConfigMap  Optional: false
      unica-campaign-env  ConfigMap  Optional: false
      unica-interact-env  ConfigMap  Optional: false
    Environment:          <none>
    Mounts:
      /docker/unica from volume-mount (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-t7d5k (ro)
Containers:
  unica-platform:
    Container ID:  
    Image:          10.0.2.6:5000/tomcat-unica-platform:12
    Image ID:      
    Port:           9123/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1600Mi
    Requests:
      cpu:      1500m
      memory:   1300Mi
    Readiness:  exec [/bin/bash -ce ] delay=20s timeout=30s period=40s #success=1 #failure=3
    Environment Variables from:
      unica-common-env    ConfigMap  Optional: false
      unica-platform-env  ConfigMap  Optional: false
      unica-campaign-env  ConfigMap  Optional: false
      unica-interact-env  ConfigMap  Optional: false
    Environment:
      CONTAINER_NAME:  unica-platform
      WLP_HTTP_PORT:   9123
      WLP_HTTPS_PORT:  9445
    Mounts:
      /docker/unica from volume-mount (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-t7d5k (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  volume-mount:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  unica
    ReadOnly:   false
  default-token-t7d5k:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-t7d5k
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age    From                 Message
  ----     ------       ----   ----                 -------
  Normal   Scheduled    5m54s  default-scheduler    Successfully assigned default/unica-platform-b5688dcb5-4wptl to unicanode1
  Warning  FailedMount  5m52s  kubelet, unicanode1  MountVolume.SetUp failed for volume "default-token-t7d5k" : failed to sync secret cache: timed out waiting for the condition
  Normal   Pulling      5m51s  kubelet, unicanode1  Pulling image "10.0.2.6:5000/unica-install:68"
  Normal   Pulled       4m47s  kubelet, unicanode1  Successfully pulled image "10.0.2.6:5000/unica-install:68"
  Normal   Created      4m46s  kubelet, unicanode1  Created container install
  Normal   Started      4m46s  kubelet, unicanode1  Started container install
[root@unicamaster software]# kubectl describe pod unica-platform-b5688dcb5-4wptl
Name:         unica-platform-b5688dcb5-4wptl
Namespace:    default
Priority:     0
Node:         unicanode1/10.0.2.5
Start Time:   Wed, 10 Jun 2020 07:16:42 +0530
Labels:       app.kubernetes.io/instance=unica
              app.kubernetes.io/name=unica-platform
              pod-template-hash=b5688dcb5
Annotations:  <none>
Status:       Pending
IP:           10.44.0.4
IPs:
  IP:           10.44.0.4
Controlled By:  ReplicaSet/unica-platform-b5688dcb5
Init Containers:
  install:
    Container ID:  docker://aff6ed9523034dfc36232aba8a5e9f8dacbfd1fa6ce4f697a1b23d57dfd31c18
    Image:         10.0.2.6:5000/unica-install:68
    Image ID:      docker-pullable://10.0.2.6:5000/unica-install@sha256:db069e6cbcd3370bbaa0e7bd80a29a83ab82e70b03a2537572ef27a6e4046233
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
    Args:
      chmod 755 /docker/unica && ./entrypoint.sh
    State:          Running
      Started:      Wed, 10 Jun 2020 07:17:50 +0530
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      unica-common-env    ConfigMap  Optional: false
      unica-platform-env  ConfigMap  Optional: false
      unica-campaign-env  ConfigMap  Optional: false
      unica-interact-env  ConfigMap  Optional: false
    Environment:          <none>
    Mounts:
      /docker/unica from volume-mount (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-t7d5k (ro)
Containers:
  unica-platform:
    Container ID:  
    Image:          10.0.2.6:5000/tomcat-unica-platform:12
    Image ID:      
    Port:           9123/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1600Mi
    Requests:
      cpu:      1500m
      memory:   1300Mi
    Readiness:  exec [/bin/bash -ce ] delay=20s timeout=30s period=40s #success=1 #failure=3
    Environment Variables from:
      unica-common-env    ConfigMap  Optional: false
      unica-platform-env  ConfigMap  Optional: false
      unica-campaign-env  ConfigMap  Optional: false
      unica-interact-env  ConfigMap  Optional: false
    Environment:
      CONTAINER_NAME:  unica-platform
      WLP_HTTP_PORT:   9123
      WLP_HTTPS_PORT:  9445
    Mounts:
      /docker/unica from volume-mount (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-t7d5k (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  volume-mount:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  unica
    ReadOnly:   false
  default-token-t7d5k:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-t7d5k
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age    From                 Message
  ----     ------       ----   ----                 -------
  Normal   Scheduled    7m28s  default-scheduler    Successfully assigned default/unica-platform-b5688dcb5-4wptl to unicanode1
  Warning  FailedMount  7m26s  kubelet, unicanode1  MountVolume.SetUp failed for volume "default-token-t7d5k" : failed to sync secret cache: timed out waiting for the condition
  Normal   Pulling      7m25s  kubelet, unicanode1  Pulling image "10.0.2.6:5000/unica-install:68"
  Normal   Pulled       6m21s  kubelet, unicanode1  Successfully pulled image "10.0.2.6:5000/unica-install:68"
  Normal   Created      6m20s  kubelet, unicanode1  Created container install
  Normal   Started      6m20s  kubelet, unicanode1  Started container install





Check the node:
[root@unicanode1 docker]# kubectl get secrets
error: stat /root/.kube/config: not a directory

Means setup was incomplete, need to do the following:
[root@unicanode1 ~]# mkdir -p $HOME/.kube

On master:
[root@unicamaster .kube]# scp -r config unicanode1:/root/.kube
root@unicanode1's password:
config

On node:
[root@unicanode1 .kube]# ls -la
total 8
drwxr-xr-x  2 root root   20 Jun 10 07:35 .
dr-xr-x---. 6 root root  251 Jun 10 07:33 ..
-rw-------  1 root root 5444 Jun 10 07:35 config
[root@unicanode1 .kube]# vi config
[root@unicanode1 .kube]# kubectl get secrets
NAME                                      TYPE                                  DATA   AGE
chart-unica-tls                           Opaque                                2      18m
default-token-t7d5k                       kubernetes.io/service-account-token   3      2d23h
nginx-nginx-ingress-backend-token-jmnzw   kubernetes.io/service-account-token   3      17h
nginx-nginx-ingress-token-9q5b5           kubernetes.io/service-account-token   3      17h

Should be fine now..


Kubelet http server gave http response to https client (ImagePullBackOff status on a Kubernetes pod)


[root@unicamaster docker]# kubectl get pods
NAME                                                   READY   STATUS                  RESTARTS   AGE
listener-0                                             0/1     Init:0/1                0          46m
nginx-nginx-ingress-controller-7fc9499f7b-96h99        1/1     Running                 0          90m
nginx-nginx-ingress-default-backend-679f548db6-tq2vz   1/1     Running                 0          90m
unica-campaign-79b746c5b-jb8rz                         0/1     Init:0/1                0          46m
unica-interact-fd5d44b49-xbw52                         0/1     Init:0/1                0          46m
unica-platform-b5688dcb5-4bzz4                         0/1     Init:ImagePullBackOff   0          46m
[root@unicamaster docker]# helm ls --all unica
NAME     REVISION    UPDATED                     STATUS      CHART          APP VERSION    NAMESPACE
unica    1           Tue Jun  9 15:05:08 2020    DEPLOYED    unica-0.5.0    1.0            default 




[root@unicamaster software]# kubectl describe pod unica-platform-b5688dcb5-4bzz4
Name:         unica-platform-b5688dcb5-4bzz4
Namespace:    default
Priority:     0
Node:         unicanode1/10.0.2.5
Start Time:   Tue, 09 Jun 2020 15:05:09 +0530
Labels:       app.kubernetes.io/instance=unica
              app.kubernetes.io/name=unica-platform
              pod-template-hash=b5688dcb5
Annotations:  <none>
Status:       Pending
IP:           10.44.0.5
IPs:
  IP:           10.44.0.5
Controlled By:  ReplicaSet/unica-platform-b5688dcb5
Init Containers:
  install:
    Container ID: 
    Image:         10.0.2.6:5000/unica-install:68
    Image ID:     
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
    Args:
      chmod 755 /docker/unica && ./entrypoint.sh
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      unica-common-env    ConfigMap  Optional: false
      unica-platform-env  ConfigMap  Optional: false
      unica-campaign-env  ConfigMap  Optional: false
      unica-interact-env  ConfigMap  Optional: false
    Environment:          <none>
    Mounts:
      /docker/unica from volume-mount (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-t7d5k (ro)
Containers:
  unica-platform:
    Container ID:  
    Image:          10.0.2.6:5000/tomcat-unica-platform:12
    Image ID:      
    Port:           9123/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1600Mi
    Requests:
      cpu:      1500m
      memory:   1300Mi
    Readiness:  exec [/bin/bash -ce ] delay=20s timeout=30s period=40s #success=1 #failure=3
    Environment Variables from:
      unica-common-env    ConfigMap  Optional: false
      unica-platform-env  ConfigMap  Optional: false
      unica-campaign-env  ConfigMap  Optional: false
      unica-interact-env  ConfigMap  Optional: false
    Environment:
      CONTAINER_NAME:  unica-platform
      WLP_HTTP_PORT:   9123
      WLP_HTTPS_PORT:  9445
    Mounts:
      /docker/unica from volume-mount (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-t7d5k (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  volume-mount:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  unica
    ReadOnly:   false
  default-token-t7d5k:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-t7d5k
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                 Message
  ----     ------     ----                   ----                 -------
  Normal   Scheduled  <unknown>              default-scheduler    Successfully assigned default/unica-platform-b5688dcb5-4bzz4 to unicanode1
  Normal   Pulling    38m (x4 over 39m)      kubelet, unicanode1  Pulling image "10.0.2.6:5000/unica-install:68"
  Warning  Failed     38m (x4 over 39m)      kubelet, unicanode1  Failed to pull image "10.0.2.6:5000/unica-install:68": rpc error: code = Unknown desc = Error response from daemon: Get https://10.0.2.6:5000/v2/: http: server gave HTTP response to HTTPS client
  Warning  Failed     38m (x4 over 39m)      kubelet, unicanode1  Error: ErrImagePull
  Normal   BackOff    14m (x107 over 39m)    kubelet, unicanode1  Back-off pulling image "10.0.2.6:5000/unica-install:68"
  Warning  Failed     4m34s (x151 over 39m)  kubelet, unicanode1  Error: ImagePullBackOff

Means you need to set daemon.json:

If you didn't set the daemon.json correctly:
[root@unicamaster docker]# systemctl restart docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
[root@unicamaster docker]# vi daemon.json
[root@unicamaster docker]# systemctl restart docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
[root@unicamaster docker]# systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2020-06-09 15:48:38 IST; 58s ago
     Docs: https://docs.docker.com
  Process: 5452 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE)
 Main PID: 5452 (code=exited, status=1/FAILURE)

Correct syntax:
{
        "insecure-registries" : ["10.0.2.6:5000"]
}

NOTE: Need to restart on node1 as well

Error: could not find tiller - Helm init

My first post on on Cloud Native... this is with Helm.

[root@unicamaster software]#   helm install --name nginx stable/nginx-ingress -f ./omnix-unica/extra-configs/nginx-conf.yaml
Error: could not find tiller
[root@unicamaster software]# cd tmp
[root@unicamaster tmp]# l s-l
bash: l: command not found...
[root@unicamaster tmp]# l s-l
bash: l: command not found...
[root@unicamaster tmp]# ls -l
total 0
[root@unicamaster tmp]# gunzip -c ../helm-v2.15.2-linux-amd64.tar.gz | tar -xvf -
linux-amd64/
linux-amd64/tiller
linux-amd64/helm
linux-amd64/README.md
linux-amd64/LICENSE
[root@unicamaster tmp]# ls -l
total 0
drwxr-xr-x 2 root root 64 Oct 30  2019 linux-amd64
[root@unicamaster tmp]# cd *
[root@unicamaster linux-amd64]# ls -l
total 76720
-rwxr-xr-x 1 root root 38789120 Oct 30  2019 helm
-rw-r--r-- 1 root root    11343 Oct 30  2019 LICENSE
-rw-r--r-- 1 root root     3288 Oct 30  2019 README.md
-rwxr-xr-x 1 root root 39755776 Oct 30  2019 tiller
[root@unicamaster linux-amd64]# tiller
bash: tiller: command not found...
[root@unicamaster linux-amd64]# ./tiller
[main] 2020/06/09 14:12:58 Starting Tiller v2.15.2 (tls=false)
[main] 2020/06/09 14:12:58 GRPC listening on :44134
[main] 2020/06/09 14:12:58 Probes listening on :44135
[main] 2020/06/09 14:12:58 Storage driver is ConfigMap
[main] 2020/06/09 14:12:58 Max history per release is 0
^C
[root@unicamaster linux-amd64]# l s-l
bash: l: command not found...
[root@unicamaster linux-amd64]# cp -p tiller /usr/local/bin
[root@unicamaster linux-amd64]# ls -l /usr/local/bin
total 76704
lrwxrwxrwx 1 root root       30 Apr 27 11:19 db2greg -> /opt/ibm/db2/V11.1/bin/db2greg
lrwxrwxrwx 1 root root       32 Apr 27 11:19 db2ls -> /opt/ibm/db2/V11.1/install/db2ls
-rwxr-xr-x 1 root root 38789120 Oct 30  2019 helm
-rwxr-xr-x 1 root root 39755776 Oct 30  2019 tiller
[root@unicamaster linux-amd64]# pwd
/opt/software/tmp/linux-amd64
[root@unicamaster linux-amd64]# cd ..
[root@unicamaster tmp]# cd ..
[root@unicamaster software]# pwd
/opt/software
[root@unicamaster software]# l s-l
bash: l: command not found...
[root@unicamaster software]# pwd
/opt/software
[root@unicamaster software]# ls -l
total 23952
-rwxr-x--- 1 root root 24525846 Apr 27 11:10 helm-v2.15.2-linux-amd64.tar.gz
drwxr-x--- 6 root root      288 Jun  9 13:56 omnix-unica
drwxr-xr-x 3 root root       25 Jun  9 14:12 tmp
drwxr-xr-x 2 root root      135 Jun  9 12:51 unica.12.0.0.0
drwxr-xr-x 2 root root       69 Jun  9 13:26 unica.12.0.0.1
[root@unicamaster software]#   helm install --name nginx stable/nginx-ingress -f ./omnix-unica/extra-configs/nginx-conf.yaml
Error: could not find tiller
[root@unicamaster software]# help init
bash: help: no help topics match `init'.  Try `help help' or `man -k init' or `info init'.
[root@unicamaster software]# helm init
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
[root@unicamaster software]#   helm install --name nginx stable/nginx-ingress -f ./omnix-unica/extra-configs/nginx-conf.yaml
Error: could not find a ready tiller pod
[root@unicamaster software]#  kubectl create serviceaccount --namespace kube-system tiller
serviceaccount/tiller created
[root@unicamaster software]#   helm install --name nginx stable/nginx-ingress -f ./omnix-unica/extra-configs/nginx-conf.yaml
Error: release nginx failed: namespaces "default" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "default"
[root@unicamaster software]# kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io/tiller-cluster-rule created
[root@unicamaster software]# kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
deployment.apps/tiller-deploy patched
[root@unicamaster software]# helm version
Client: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}
[root@unicamaster software]#   helm install --name nginx stable/nginx-ingress -f ./omnix-unica/extra-configs/nginx-conf.yaml
NAME:   nginx
LAST DEPLOYED: Tue Jun  9 14:20:49 2020
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                 AGE
nginx-nginx-ingress  1s

==> v1/ClusterRoleBinding
NAME                 AGE
nginx-nginx-ingress  1s

==> v1/ConfigMap
NAME                            DATA  AGE
nginx-nginx-ingress-controller  1     1s

==> v1/Deployment
NAME                                 READY  UP-TO-DATE  AVAILABLE  AGE
nginx-nginx-ingress-controller       0/1    1           0          1s
nginx-nginx-ingress-default-backend  0/1    1           0          1s

==> v1/Pod(related)
NAME                                                  READY  STATUS             RESTARTS  AGE
nginx-nginx-ingress-controller-7fc9499f7b-96h99       0/1    ContainerCreating  0         0s
nginx-nginx-ingress-default-backend-679f548db6-tq2vz  0/1    ContainerCreating  0         1s

==> v1/Role
NAME                 AGE
nginx-nginx-ingress  1s

==> v1/RoleBinding
NAME                 AGE
nginx-nginx-ingress  1s

==> v1/Service
NAME                                    TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
nginx-nginx-ingress-controller          LoadBalancer  10.97.49.16    <pending>    80:32500/TCP,443:31138/TCP  1s
nginx-nginx-ingress-controller-metrics  ClusterIP     10.108.50.227  <none>       9913/TCP                    1s
nginx-nginx-ingress-default-backend     ClusterIP     10.97.62.215   <none>       80/TCP                      1s

==> v1/ServiceAccount
NAME                         SECRETS  AGE
nginx-nginx-ingress          1        1s
nginx-nginx-ingress-backend  1        1s


NOTES:
The nginx-ingress controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w nginx-nginx-ingress-controller'

An example Ingress that makes use of the controller:

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

PVC in terminating state

Referencee:https://veducate.co.uk/kubernetes-pvc-terminating/  I still had errors performing deployment, till I released that although I pre...