Доброе утро, виртуалки на vsphere, kubeadm, calico. Часть нод потеряли

интернал/екстереал айпи, не обновляют status.addresses(там только хостнейм) флаги кубелета(с айпишкой), рестарт его, ребут ноды - не помогают, в логах кубелета никаких ошибок связанных с этим делом - нет. Можно починить, если удалить ноду, поправить ямл и применить, но вариант такой себе. Может кто то сталкивался?

P.s. Кластер не разворачивал, наследие поляков

Нет, такое не видел, увы

external ip вроде назначается cloud controller manager'ом. Но обычно это делается один раз, и потом это поле не удалить. Только если переинициализировать ноду. Покажи манифест полный такой ноды

‘’’apiVersion: v1 kind: Node metadata: annotations: csi.volume.kubernetes.io/nodeid: '{"csi.vsphere.vmware.com":"k8s-compute-2.test"}' kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: "0" projectcalico.org/IPv4Address: 10.191.120.86/27 projectcalico.org/IPv4IPIPTunnelAddr: 172.18.9.0 volumes.kubernetes.io/controller-managed-attach-detach: "true" creationTimestamp: "2020-06-01T11:57:57Z" labels: beta.kubernetes.io/arch: amd64 beta.kubernetes.io/instance-type: vsphere-vm.cpu-6.mem-40gb.os-rhel7 beta.kubernetes.io/os: linux kubernetes.io/arch: amd64 kubernetes.io/hostname: k8s-compute-2.test kubernetes.io/os: linux node-purpose: compute node-role.kubernetes.io/compute: "" name: k8s-compute-2.test resourceVersion: "202198648" selfLink: /api/v1/nodes/k8s-compute-2.test uid: 51a91f1b-9560-4dfc-82e7-d04586f33fa2 spec: podCIDR: 192.168.5.0/24 podCIDRs: - 192.168.5.0/24 providerID: vsphere://42298B0A-A938-AD52-6530-5211C0F8F104 status: addresses: - address: k8s-compute-2.test type: Hostname allocatable: cpu: 5700m ephemeral-storage: "12061472335" hugepages-1Gi: "0" hugepages-2Mi: "0" memory: 40691680Ki pods: "110" capacity: cpu: "6" ephemeral-storage: 15417704Ki hugepages-1Gi: "0" hugepages-2Mi: "0" memory: 41203680Ki pods: "110" conditions: - lastHeartbeatTime: "2021-08-27T16:03:12Z" lastTransitionTime: "2021-08-27T16:03:12Z" message: Calico is running on this node reason: CalicoIsUp status: "False" type: NetworkUnavailable - lastHeartbeatTime: "2021-08-28T07:18:01Z" lastTransitionTime: "2021-08-27T16:02:28Z" message: kubelet has sufficient memory available reason: KubeletHasSufficientMemory status: "False" type: MemoryPressure - lastHeartbeatTime: "2021-08-28T07:18:01Z" lastTransitionTime: "2021-08-27T16:02:28Z" message: kubelet has no disk pressure reason: KubeletHasNoDiskPressure status: "False" type: DiskPressure - lastHeartbeatTime: "2021-08-28T07:18:01Z" lastTransitionTime: "2021-08-27T16:02:28Z" message: kubelet has sufficient PID available reason: KubeletHasSufficientPID status: "False" type: PIDPressure - lastHeartbeatTime: "2021-08-28T07:18:01Z" lastTransitionTime: "2021-08-27T16:11:07Z" message: kubelet is posting ready status reason: KubeletReady status: "True" type: Ready daemonEndpoints: kubeletEndpoint: Port: 10250 images: - names: - docker-registry.test-starboost/payments-lcy/ml-payment-data-parser@sha256:a4a2d6d916fc14c1f3caa2ca29f460d54f3c66ac02cdc6d4ed7b50735d26948b - docker-registry.test-starboost/payments-lcy/ml-payment-data-parser:0.1.0-f45c7d0 sizeBytes: 2518440821 - names: - docker-registry.test-starboost/payments-lcy/ml-payment-data-parser@sha256:d3e083679fd2b42ce267faa63f3fa2c2158840a6de4de35c008102aaf8c395d2 - docker-registry.test-starboost/payments-lcy/ml-payment-data-parser:0.1.0-c30e94f sizeBytes: 2518440803 - names: - docker-registry.test-starboost/payments-lcy/ml-payment-data-parser@sha256:13c50009199a592112b63c4b9214ae5dafc0a94f0201369fd838157bab0c8490 - docker-registry.test-starboost/payments-lcy/ml-payment-data-parser:0.1.0-2c2d638 sizeBytes: 2516972056 - names: - docker-registry.test-starboost/clients-document/yes-nuxeo@sha256:7d617aba89fcef0e180f877c13354fe273d8b49d12a87f5682687fe1439489ec - docker-registry.test-starboost/clients-document/yes-nuxeo:0.1.0-a5c96d9 sizeBytes: 1991123989 - names: - docker-registry.test-starboost/payments/payment-orchestration@sha256:8ead30b72b6419ae663902b94d78017f261cb9f7479bcfeaceea0f7bb82ebde2 - docker-registry.test-starboost/payments/payment-orchestration:0.1.0-4068ed7 sizeBytes: 835494227 - names: - docker-registry.test-starboost/payments/payment-orchestration@sha256:d93a0434138e7cc4a8013f17ecc05f80bf07e787cf57835bdef4296943f12b7e - docker-registry.

test-starboost/payments/payment-orchestration:0.1.0-a03a077 sizeBytes: 835488303 - names: - docker-registry.test-starboost/payments/payment-orchestration@sha256:ddf5046c98198c10bf1e2356b6e39116085a8623446de3498d5c4778278775c5 - docker-registry.test-starboost/payments/payment-orchestration:0.1.0-dc64d43 sizeBytes: 835488255 - names: - docker-registry.test-starboost/payments/payment-orchestration@sha256:7cb8c3d6c9ce747bbb90067d94d4ffb8e199e2b403f5314594249a82cab4f763 - docker-registry.test-starboost/payments/payment-orchestration:0.1.0-46bf747 sizeBytes: 835354155 - names: - docker-registry.test-starboost/payments/payment-orchestration@sha256:12a60366e89a162f972179df99885121f19800994d5128dc9c4b159a7284c618 - docker-registry.test-starboost/payments/payment-orchestration:0.1.0-ab36bc6 sizeBytes: 835350267 - names: - docker-registry.test-starboost/payments/payment-orchestration@sha256:fd1bcd9d354fd83c989eb499e9204f39c428eb7f5c16f9f8a9618ef0aa7e630c - docker-registry.test-starboost/payments/payment-orchestration:0.1.0-fe65172 sizeBytes: 834749883 - names: - docker-registry.test-starboost/operations/network-utilities@sha256:99cf45ba0e1e57b4dec7f986c0c2f0696b77fcbef2bc2374b88df7c329d9dfa0 - docker-registry.test-starboost/operations/network-utilities:0.1.0-d28e757 sizeBytes: 803223146 - names: - docker-registry.test-starboost/accounting/securegate@sha256:e1a8c6807fcfaa88c25bffa6875f7eaf9463d43db9a594d24f95d4f0889e1f81 - docker-registry.test-starboost/accounting/securegate:0.1.0-0a129f3 sizeBytes: 731687513 - names: - docker-registry.test-starboost/cards/card-services-system-tests@sha256:37e28f79b9ec6e601fe4cc97c3bc5a7df5331f7c0650dc5f5a69ecafe503355f sizeBytes: 680705812 - names: - docker-registry.test-starboost/cards/card-services-system-tests@sha256:314c735778386101aa22c919d7bfe231b96405f53216d13fcdfa1812d17aecfb - docker-registry.test-starboost/cards/card-services-system-tests:latest sizeBytes: 680621192 - names: - docker-registry.test-starboost/core-banking/dls-processing@sha256:cd2a21b83bb73e18f9c21a0b1614da25ddafb49bf91c9f7526f523115b592dc1 - docker-registry.test-starboost/core-banking/dls-processing:0.1.0-1d77614 sizeBytes: 664003740 - names: - docker-registry.test-starboost/core-banking/bmrs-payment@sha256:8d07ab26aef06458ccba7091146421b866e3ab3833e69cc38321884a9b918382 - docker-registry.test-starboost/core-banking/bmrs-payment:0.1.0-426c0c2 sizeBytes: 661151119 - names: - docker-registry.test-starboost/core-banking/bmrs-loan@sha256:5ee896985ee1402b24b7d3fb822f49a0f3c2215ed6ec8a6e3ee2cbd8071252b9 - docker-registry.test-starboost/core-banking/bmrs-loan:0.1.0-aaf2a8e sizeBytes: 661011539 - names: - docker-registry.test-starboost/core-banking/loan579-processing@sha256:74107ec7c67e5fdf0ecd0b86a5e19b898009afcb99b61909f182aaf4327b7ab4 - docker-registry.test-starboost/core-banking/loan579-processing:0.1.0-6ff26a0 sizeBytes: 660239220 - names: - docker-registry.test-starboost/core-banking/bmrs-rate@sha256:9cce48a3a40494c2e426f067c8ecc7f44b2ac6e31a9c7708d47f6909f4db4f8e - docker-registry.test-starboost/core-banking/bmrs-rate:0.1.0-d8c9e24 sizeBytes: 659300830 - names: - docker-registry.test-starboost/channels/permission@sha256:86f7e0136bdc0f8d3f95913dfc01701d94927c27067e7273d981b7b6b18a1528 - docker-registry.test-starboost/channels/permission:0.1.0-0a62037 sizeBytes: 637414886 - names: - docker-registry.test-starboost/client-products/product-catalog@sha256:8fbca71cf735a53ebd81bb15ff3232e9567b6bc1e30d72999c90c080301634f8 - docker-registry.test-starboost/client-products/product-catalog:0.1.0-d54b636 sizeBytes: 629137020 - names: - docker-registry.test-starboost/channels/gateway@sha256:609a5fceee3ae3db33d7cdc29d66d9aa6dbbecbbda9397ebc9da84aa042bb73b - docker-registry.test-starboost/channels/gateway:0.1.0-8f021a7 sizeBytes: 589292808 - names: - docker-registry.

test-starboost/channels/rgo-account@sha256:1ae281a5d7561a9c671c6eda4e76d62c43675fd028a7695fca59a7a00748c1ee - docker-registry.test-starboost/channels/rgo-account:0.1.0-b93e5e5 sizeBytes: 588577460 - names: - docker-registry.test-starboost/cards/card-services-system-tests@sha256:5c68c443aa3a1e986b9b0cea6f8d1acc3bd4a54d72111881fc61c28083544b91 - docker-registry.test-starboost/cards/card-services-system-tests:KALINA-2469 sizeBytes: 569665253 - names: - docker-registry.test-starboost/cards/card-services-system-tests@sha256:cdc7013c0cedc020bdd1675705bca1d8f7ed9867d799be8de88088a15017e0a7 sizeBytes: 569662021 - names: - docker-registry.test-starboost/cards/card-services-system-tests@sha256:8e2dc7f1cc303a880a6063e605432bce3d5a115c5cd34c5078e5cb5bc9affd61 sizeBytes: 569615591 - names: - docker-registry.test-starboost/channels/bff-product-details@sha256:f7e03567269d6780a34acc59ec28aa85a41db961a858b45a71e0b6cc8c8856c6 - docker-registry.test-starboost/channels/bff-product-details:0.1.0-3eb7c1b sizeBytes: 568692066 - names: - docker-registry.test-starboost/cards/card-services-system-tests@sha256:dd870183dc4a1dabc6c83fa75c37b0583c2df299057d40d2e24136446c75d8e0 sizeBytes: 566467904 - names: - docker-registry.test-starboost/channels/channels-3ds-connect@sha256:0d6483576be0c65a8fa6065c6c5ac6440aa689949e8e1043a4f39be298dfa519 - docker-registry.test-starboost/channels/channels-3ds-connect:0.1.0-4a880f2 sizeBytes: 566297480 - names: - docker-registry.test-starboost/payments-salary/salary-api-gateway@sha256:040675dad339c8a1be83796daf160d286bbff467360b2866e707f40d477b1e52 - docker-registry.test-starboost/payments-salary/salary-api-gateway:0.1.0-76bf2f3 sizeBytes: 556977647 - names: - docker-registry.test-starboost/payments-salary/salary-api-gateway@sha256:c2bc61bf09ec038b28137eece0da4f17503077fbad5f0f66fd3af3a0722dd1bc - docker-registry.test-starboost/payments-salary/salary-api-gateway:0.1.0-c17a7cb sizeBytes: 556977396 - names: - docker-registry.test-starboost/clients/contract@sha256:3863b45a578f95486b9d33cb72b1de89f612b68c5c9630662f80f21f25820edb - docker-registry.test-starboost/clients/contract:0.1.0-6c3685e sizeBytes: 556418588 - names: - docker-registry.test-starboost/payments-fxss/fcy-settings@sha256:813db309fbcc63ae15e36b391b341c10b61e0d322d1b36c469c4256db82a7973 - docker-registry.test-starboost/payments-fxss/fcy-settings:0.1.0-286b819 sizeBytes: 552291536 - names: - docker-registry.test-starboost/payments-fxss/fcy-settings@sha256:485acb924beff6422e97df54053f0fca2d852af0fb31c28bb76f5dc06ed140e9 - docker-registry.test-starboost/payments-fxss/fcy-settings:0.1.0-792a039 sizeBytes: 552291498 - names: - docker-registry.test-starboost/payments-fxss/admin@sha256:3e9ae35bb4c52437e057a97a7ee141181f360288ecbb605cf6309978ef589e84 - docker-registry.test-starboost/payments-fxss/admin:0.1.0-40432ce sizeBytes: 552246366 - names: - docker-registry.test-starboost/accounting/unistatement@sha256:f1d1271ff4b57f6248141742499605f6277a0d15959fc72fcf99f003c8c1170a - docker-registry.test-starboost/accounting/unistatement:0.1.0-9fdc3a0 sizeBytes: 551626250 - names: - docker-registry.test-starboost/accounting/unistatement@sha256:8cbfb02b0ecc3acdb8ddbc8e734c9675115558514c8a992fdfc82201fcaea264 - docker-registry.test-starboost/accounting/unistatement:0.1.0-32aac83 sizeBytes: 551626060 - names: - docker-registry.test-starboost/payments-fxss/credential-auth@sha256:97a9c5ed5d5c68d17be3e98a2f19efbcd09ef469d41e6a48ba5cbf8df372dc06 - docker-registry.test-starboost/payments-fxss/credential-auth:0.1.0-23cc8e1 sizeBytes: 545346844 - names: - docker-registry.test-starboost/channels/cmd@sha256:7b4e33bca83122dcc7bcb06d55079bba9268c61aed037590b7cdc9db645b93ec - docker-registry.test-starboost/channels/cmd:0.1.0-eed5829 sizeBytes: 536519348 - names: - docker-registry.

test-starboost/client-products/product-catalog@sha256:95d0def02f333a5406176c74fa5bbe95ca89bfe93befa48c663c90fff713d866 - docker-registry.test-starboost/client-products/product-catalog:0.1.0-0c9a2c2 sizeBytes: 532465148 - names: - docker-registry.test-starboost/payments-fxss/gateway-fxss@sha256:d2e1ecc679ecaab78bb8e6efbd34e1b58ccd304005cb28e819b9234a49ba168e - docker-registry.test-starboost/payments-fxss/gateway-fxss:0.1.0-c25d0e3 sizeBytes: 531921786 - names: - docker-registry.test-starboost/cards/is-card@sha256:ca58d334ff25b71327faff4e39b69967ec411b8aa6337d42f5bbaaf86ad9e8f9 - docker-registry.test-starboost/cards/is-card:0.1.0-8936310 sizeBytes: 529227784 - names: - docker-registry.test-starboost/cards/is-card@sha256:93d17f7a4b687924f36fd52048cfd2e47de1f167bca8d4fad2bfbdc861c3825d - docker-registry.test-starboost/cards/is-card:0.1.0-6445d3a sizeBytes: 529227784 - names: - docker-registry.test-starboost/cards/is-card@sha256:d71d4a4e8b3f4a525989c81a07098d1ecb674faf4773b43427e4809e9a32477f - docker-registry.test-starboost/cards/is-card:0.1.0-93e6c6e sizeBytes: 529224508 - names: - docker-registry.test-starboost/channels/audit-events@sha256:2a3e5f0eb460ed024643795ce8c2f310c55e98acf545bbc463b3fc3a6a24b9b5 - docker-registry.test-starboost/channels/audit-events:0.1.0-a07fbb7 sizeBytes: 529131338 - names: - docker-registry.test-starboost/clients/client@sha256:3bbd6716a40549d3d0dbc67202099059e7a9e527255ac48267f5ef5d23db78a6 - docker-registry.test-starboost/clients/client:0.1.0-05a95ef sizeBytes: 527096711 - names: - docker-registry.test-starboost/channels/annotation@sha256:b8d6c097b6ed40a46287e1cad112808ca5963595b2160f82cc6c381ec77dcab3 - docker-registry.test-starboost/channels/annotation:0.1.0-ee2de08 sizeBytes: 524975866 - names: - docker-registry.test-starboost/payments-erp/payment-factory-tests-service@sha256:5c9e773c2be12cf3b1505d9a1a7c5d77fe3fcf4699bd71a29bb8c2b22331babe - docker-registry.test-starboost/payments-erp/payment-factory-tests-service:dev sizeBytes: 523588271 - names: - docker-registry.test-starboost/clients/onboarding@sha256:cbb7e468a3c065ea9be6b4b8566378fcb57ac375caa9b6105e99aefeffc48173 - docker-registry.test-starboost/clients/onboarding:0.1.0-5e4fc84 sizeBytes: 521765067 - names: - docker-registry.test-starboost/clients/onboarding@sha256:bb1d11eba9bf071456ca710c7a77f8f7b241f72cb143bc7f264b41f13b04b066 - docker-registry.test-starboost/clients/onboarding:0.1.0-658a42c sizeBytes: 521763161 nodeInfo: architecture: amd64 bootID: fb02c91b-cf6e-43dc-8d59-0909f131d9d1 containerRuntimeVersion: docker://19.3.4 kernelVersion: 3.10.0-1127.el7.x86_64 kubeProxyVersion: v1.18.3 kubeletVersion: v1.18.3 machineID: a76e0a52cfc445fc854e15bb616f748e operatingSystem: linux osImage: RHEL systemUUID: 42298B0A-A938-AD52-6530-5211C0F8F104 volumesAttached: - devicePath: "" name: kubernetes.io/csi/csi.vsphere.vmware.com^2e3761c4-c665-4e78-a439-729bdc36ed11 - devicePath: "" name: kubernetes.io/csi/csi.vsphere.vmware.com^ed3f1109-c3ea-4a5c-9abb-4cf020449147 - devicePath: "" name: kubernetes.io/csi/csi.vsphere.vmware.com^efe92d37-381e-4db9-b8d6-100294fb5b2b volumesInUse: - kubernetes.io/csi/csi.vsphere.vmware.com^2e3761c4-c665-4e78-a439-729bdc36ed11 - kubernetes.io/csi/csi.vsphere.vmware.com^ed3f1109-c3ea-4a5c-9abb-4cf020449147 - kubernetes.io/csi/csi.vsphere.vmware.com^efe92d37-381e-4db9-b8d6-100294fb5b2b’’’

Убейте его кто-нибудь

Гражданин, подотрите за собой

https://pastebin.com/xQuENXXx

cloud controller manager точно есть: providerID: vsphere://42298B0A-A938-AD52-6530-5211C0F8F104 проверь правильный ли это id, и можно ли вытащить из api vsphere адреса для ноды с этим ID Попробуй переинициализировать таким образом kubectl patch node k8s-compute-2.test --type='json' -p='[{"op":"add","path":"/spec/taints","value": [{"effect":"NoSchedule","key":"node.cloudprovider.kubernetes.io/uninitialized"}]}]' Посмотри что в логах пишет cloud controller manager при инициализации ноды Механизм такой, он должен увидеть эти тейнты, проинициализировать ноду (добавить лейблы, добавить адреса) а затем удалить тейнты Сам cloud controller manager может в кластере работать (проверь это). И смотри его логи при инициализации

если он работает в кластере, то у него есть секрет от API vshepre, а значит у тебя есть доступ к API vsphere

Ах вот оно чо, а я думал какого хера он лейбл не ставит

У кубелета флаг external, попробую, спасибо за наводку

Экстернал это норм, значит используется не ин-три, а отдельный клауд контроллер

да, в таком случае он сам тейнты пропишет. Но я думал он у тебя и так с этим флагом был запущен, потому что эта нода уже вроде была проинициализирована когда-то (поскольку там есть providerID)