This was originally reported in https://gitlab.com/gitlab-org/gitlab/-/issues/407161. GitLab Agent for Kubernetes, apart from other things, is a Kubernetes API reverse proxy. Users recently started getting errors when using it with kubectl v1.27.x.
Users' CI jobs get a generated kubectl-compatible config file, where server URL has a path component. For GitLab.com the address is https://kas.gitlab.com/k8s-proxy/. Looks like kubectl v1.27.x ignores the path component for some OpenAPI requests - here is an output for a kubectl apply run:
I0419 06:40:40.335640 21 loader.go:373] Config loaded from file: /builds/jeff127/master-site.tmp/KUBECONFIG
I0419 06:40:40.337257 21 round_trippers.go:463] GET https://kas.gitlab.com/k8s-proxy/openapi/v2?timeout=32s
I0419 06:40:40.337609 21 round_trippers.go:469] Request Headers:
I0419 06:40:40.337808 21 round_trippers.go:473] User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/4c94112
I0419 06:40:40.337996 21 round_trippers.go:473] Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
I0419 06:40:40.338175 21 round_trippers.go:473] Authorization: Bearer <masked>
I0419 06:40:40.705094 21 round_trippers.go:574] Response Status: 200 OK in 366 milliseconds
I0419 06:40:40.705471 21 round_trippers.go:577] Response Headers:
I0419 06:40:40.705674 21 round_trippers.go:580] Cache-Control: no-cache, private
I0419 06:40:40.705825 21 round_trippers.go:580] Audit-Id: 48e3c227-8e96-4688-8e44-c39c89f13588
I0419 06:40:40.706023 21 round_trippers.go:580] X-Varied-Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
I0419 06:40:40.706200 21 round_trippers.go:580] Date: Wed, 19 Apr 2023 06:40:40 GMT
I0419 06:40:40.706367 21 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: d8cba8a0-47c4-4945-ac5f-79f4e5e47b25
I0419 06:40:40.706555 21 round_trippers.go:580] Accept-Ranges: bytes
I0419 06:40:40.706732 21 round_trippers.go:580] Content-Type: application/octet-stream
I0419 06:40:40.706874 21 round_trippers.go:580] Etag: "90C689D93C2CBA4FB3BB7599D386DB48F5A580DFCE512B5E5594DBE2ACC8A95791BD41AE58413A6F57288874F25A4F0A7123AF5F28101087E840B3D413D8CBA4"
I0419 06:40:40.707064 21 round_trippers.go:580] Via: 2.0 gitlab-agent/v15.11.0/996c5f37
I0419 06:40:40.707231 21 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v15.11.0/v15.11.0
I0419 06:40:40.707382 21 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: 80394f72-88ac-405d-b429-678a7820d227
I0419 06:40:40.707672 21 round_trippers.go:580] Last-Modified: Sat, 15 Apr 2023 06:42:23 GMT
I0419 06:40:40.707844 21 round_trippers.go:580] Vary: Accept-Encoding
I0419 06:40:40.707980 21 round_trippers.go:580] Vary: Accept
I0419 06:40:40.708169 21 round_trippers.go:580] Vary: Accept-Encoding
I0419 06:40:40.708315 21 round_trippers.go:580] Vary: Accept
I0419 06:40:41.204663 21 request.go:1186] Response Body:
00000000 0a 03 32 2e 30 12 16 0a 0a 4b 75 62 65 72 6e 65 |..2.0....Kuberne|
00000010 74 65 73 12 08 76 31 2e 32 34 2e 31 30 42 80 a6 |tes..v1.24.10B..|
00000020 a1 01 12 8c 02 0a 22 2f 2e 77 65 6c 6c 2d 6b 6e |......"/.well-kn|
00000030 6f 77 6e 2f 6f 70 65 6e 69 64 2d 63 6f 6e 66 69 |own/openid-confi|
00000040 67 75 72 61 74 69 6f 6e 2f 12 e5 01 12 e2 01 0a |guration/.......|
00000050 09 57 65 6c 6c 4b 6e 6f 77 6e 1a 57 67 65 74 20 |.WellKnown.Wget |
00000060 73 65 72 76 69 63 65 20 61 63 63 6f 75 6e 74 20 |service account |
00000070 69 73 73 75 65 72 20 4f 70 65 6e 49 44 20 63 6f |issuer OpenID co|
00000080 6e 66 69 67 75 72 61 74 69 6f 6e 2c 20 61 6c 73 |nfiguration, als|
00000090 6f 20 6b 6e 6f 77 6e 20 61 73 20 74 68 65 20 27 |o known as the '|
000000a0 4f 49 44 43 20 64 69 73 63 6f 76 65 72 79 20 64 |OIDC discovery d|
000000b0 6f 63 27 2a 2a 67 65 74 53 65 72 76 69 63 65 41 |oc'**getServiceA|
000000c0 63 63 6f 75 6e 74 49 73 73 75 65 72 4f 70 65 6e |ccountIssuerOpe [truncated 17871695 chars]
I0419 06:40:41.278595 21 round_trippers.go:463] GET https://kas.gitlab.com/k8s-proxy/openapi/v3?timeout=32s
I0419 06:40:41.278962 21 round_trippers.go:469] Request Headers:
I0419 06:40:41.279159 21 round_trippers.go:473] Accept: application/json, */*
I0419 06:40:41.279317 21 round_trippers.go:473] User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/4c94112
I0419 06:40:41.279531 21 round_trippers.go:473] Authorization: Bearer <masked>
I0419 06:40:41.567776 21 round_trippers.go:574] Response Status: 200 OK in 288 milliseconds
I0419 06:40:41.568253 21 round_trippers.go:577] Response Headers:
I0419 06:40:41.568463 21 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: d8cba8a0-47c4-4945-ac5f-79f4e5e47b25
I0419 06:40:41.568742 21 round_trippers.go:580] Accept-Ranges: bytes
I0419 06:40:41.568917 21 round_trippers.go:580] Audit-Id: dc49e1ee-77b5-4dea-a139-9ee2156a637e
I0419 06:40:41.569081 21 round_trippers.go:580] Content-Type: text/plain; charset=utf-8
I0419 06:40:41.569249 21 round_trippers.go:580] Date: Wed, 19 Apr 2023 06:40:41 GMT
I0419 06:40:41.569373 21 round_trippers.go:580] Last-Modified: Wed, 19 Apr 2023 06:40:41 GMT
I0419 06:40:41.569587 21 round_trippers.go:580] Via: 2.0 gitlab-agent/v15.11.0/996c5f37
I0419 06:40:41.569751 21 round_trippers.go:580] Via: gRPC/1.0 gitlab-kas/v15.11.0/v15.11.0
I0419 06:40:41.569913 21 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: 80394f72-88ac-405d-b429-678a7820d227
I0419 06:40:41.570078 21 round_trippers.go:580] Cache-Control: no-cache, private
I0419 06:40:41.570252 21 round_trippers.go:580] Content-Length: 14539
I0419 06:40:41.571553 21 request.go:1188] Response Body: {"paths":{".well-known/openid-configuration":{"serverRelativeURL":"/openapi/v3/.well-known/openid-configuration?hash=BF1C8CB94CEB72F9853EDF554520390667FFB0AA92BDAD649C8B70C28C0F7594D8E877E9A0682773E8301D0289D288DF118C90A0155A647A7A27AF26C02CBEC2"},"api":{"serverRelativeURL":"/openapi/v3/api?hash=2DCF3BF7BF6FAAB0A535BC865BE6C3613FAB37EB2A5650DAD1FA4DCCF5658086E5D15896598112DAB5591BF089A327C60E7A98F88D37AEB5F99DAB7E1B2704EC"},"api/v1":{"serverRelativeURL":"/openapi/v3/api/v1?hash=17C7EEA4DFF6156193C13E441C8A8BA5D552BCA97017070ADC5688AF4B3FFCA305B4708D83CF883B522E33958A3812631C33E90C5556DDF482DB7CC4D5286C6F"},"apis":{"serverRelativeURL":"/openapi/v3/apis?hash=3D55D9A351386B8FD26B41F214DD2D0C34328BA69DB2E4E0FC54EE6FC5693DF7B5094439ABE2A3ADA31CCCBB22A9FC9372769CB2ACCD3A165CC30A6D281251D6"},"apis/admissionregistration.k8s.io":{"serverRelativeURL":"/openapi/v3/apis/admissionregistration.k8s.io?hash=D3B2D7397ED883146045CE57DD02E66B442A7ED8FF25480FAFB1238044B0F1241140493C55AADB6C87AC8BB317DB28B15CBA569930FB85747AC2EF6 [truncated 13515 chars]
I0419 06:40:41.572270 21 round_trippers.go:463] GET https://kas.gitlab.com/openapi/v3/apis/apps/v1?hash=BF53D008DE76FB14B787193AFA503792A5ABCF349C100FCAC5E22D8179272576DB7C657B60A4B6D932F1E09A3B6CFE13A74E95FF2FB91F92B1D34E6621262606&timeout=32s
I0419 06:40:41.572507 21 round_trippers.go:469] Request Headers:
I0419 06:40:41.572754 21 round_trippers.go:473] Accept: application/json
I0419 06:40:41.572926 21 round_trippers.go:473] User-Agent: kubectl/v1.27.1 (linux/amd64) kubernetes/4c94112
I0419 06:40:41.573138 21 round_trippers.go:473] Authorization: Bearer <masked>
I0419 06:40:41.596856 21 round_trippers.go:574] Response Status: 426 Upgrade Required in 23 milliseconds
I0419 06:40:41.597220 21 round_trippers.go:577] Response Headers:
I0419 06:40:41.597377 21 round_trippers.go:580] Content-Type: text/plain; charset=utf-8
I0419 06:40:41.597557 21 round_trippers.go:580] Server: gitlab-kas/v15.11.0/v15.11.0
I0419 06:40:41.597716 21 round_trippers.go:580] X-Content-Type-Options: nosniff
I0419 06:40:41.597900 21 round_trippers.go:580] Date: Wed, 19 Apr 2023 06:40:41 GMT
I0419 06:40:41.598104 21 round_trippers.go:580] Content-Length: 76
I0419 06:40:41.598559 21 request.go:1188] Response Body: WebSocket protocol violation: Connection header "" does not contain Upgrade
error: error validating "kubernetes/staging/app.yml": error validating data: the server responded with the status code 426 but did not return more information; if you choose to ignore these errors, turn validation off with --validate=false
Important bits - request is made to https://kas.gitlab.com/openapi/v3/apis/apps/v1 but should be https://kas.gitlab.com/k8s-proxy/openapi/v3/apis/apps/v1:
I0419 06:40:41.571553 21 request.go:1188] Response Body: {"paths":{".well-known/openid-configuration":{"serverRelativeURL":"/openapi/v3/.well-known/openid-configuration?hash=BF1C8CB94CEB72F9853EDF554520390667FFB0AA92BDAD649C8B70C28C0F7594D8E877E9A0682773E8301D0289D288DF118C90A0155A647A7A27AF26C02CBEC2"},"api":{"serverRelativeURL":"/openapi/v3/api?hash=2DCF3BF7BF6FAAB0A535BC865BE6C3613FAB37EB2A5650DAD1FA4DCCF5658086E5D15896598112DAB5591BF089A327C60E7A98F88D37AEB5F99DAB7E1B2704EC"},"api/v1":{"serverRelativeURL":"/openapi/v3/api/v1?hash=17C7EEA4DFF6156193C13E441C8A8BA5D552BCA97017070ADC5688AF4B3FFCA305B4708D83CF883B522E33958A3812631C33E90C5556DDF482DB7CC4D5286C6F"},"apis":{"serverRelativeURL":"/openapi/v3/apis?hash=3D55D9A351386B8FD26B41F214DD2D0C34328BA69DB2E4E0FC54EE6FC5693DF7B5094439ABE2A3ADA31CCCBB22A9FC9372769CB2ACCD3A165CC30A6D281251D6"},"apis/admissionregistration.k8s.io":{"serverRelativeURL":"/openapi/v3/apis/admissionregistration.k8s.io?hash=D3B2D7397ED883146045CE57DD02E66B442A7ED8FF25480FAFB1238044B0F1241140493C55AADB6C87AC8BB317DB28B15CBA569930FB85747AC2EF6 [truncated 13515 chars]
I0419 06:40:41.572270 21 round_trippers.go:463] GET https://kas.gitlab.com/openapi/v3/apis/apps/v1?hash=BF53D008DE76FB14B787193AFA503792A5ABCF349C100FCAC5E22D8179272576DB7C657B60A4B6D932F1E09A3B6CFE13A74E95FF2FB91F92B1D34E6621262606&timeout=32s
The error that is reported is a red herring. The problem is that the call is made not to the proxy URL but talks to something that expects a WebSocket request.
What did you expect to happen?Command should work normally.
How can we reproduce it (as minimally and precisely as possible)?Maybe write a unit test that starts apiserver with a URL path and test how discovery works?
Anything else we need to know?This is probably related to kubernetes/enhancements#3352, which was released as beta in v1.27.
Kubernetes version v1.27.x Cloud provider N/A OS version N/A Install tools Container runtime (CRI) and version (if applicable) Related plugins (CNI, CSI, ...) and versions (if applicable)RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4