Deployment with external gitlab

Renku Admin
1

Dear Renku team,
Thank you for your work. I am sorry to bother you but I am blocked.

I managed to deploy renku on a bare metal kubernetes cluster of mine. The deployment seems alright, all pods running and jobs passed. I made an installation with an external gitlab (lsms-gitlab.epfl.ch). Ingress works, keycloak was configured with the proper realms.

However the UI (enacvm0061.xaas.epfl.ch/) stays with the message: “Some of the resources on RenkuLab are temporarily unavailable.”

Also when I hit the page: enacvm0061.xaas.epfl.ch/gitlab I see a weird message with a clearly malformed uri:

{“message”:“remote https://lsms-gitlab.epfl.ch(https://lsms-gitlab.epfl.ch) unreachable, could not forward: dial tcp 10.95.33.25:443: connect: connection refused”}

Can someone give me a hand ?

kubectl get pods:

renku-amalthea-7bc7c6888f-fcmfk 1/1 Running 0 33m
renku-commit-event-service-7657ddd464-vwd9t 1/1 Running 0 33m
renku-core-v10-dfcd5df88-dd4jd 4/4 Running 3 (33m ago) 33m
renku-core-v10-dfcd5df88-zmxdg 4/4 Running 3 (33m ago) 33m
renku-core-v9-85ff4f664b-tc4nr 4/4 Running 1 (33m ago) 33m
renku-core-v9-85ff4f664b-vhbs6 4/4 Running 2 (33m ago) 33m
renku-event-log-5f489796c9-ms8bd 1/1 Running 0 33m
renku-gateway-auth-f554dfd69-nrwkn 1/1 Running 0 33m
renku-gateway-revproxy-6b4f44d888-hzwn4 1/1 Running 2 (33m ago) 33m
renku-gateway-revproxy-6b4f44d888-w4b9z 1/1 Running 0 33m
renku-jena-master-0 1/1 Running 0 33m
renku-keycloakx-0 1/1 Running 0 33m
renku-knowledge-graph-84cdd49876-8sgz2 1/1 Running 0 33m
renku-notebooks-0 1/1 Running 0 33m
renku-notebooks-1 1/1 Running 0 33m
renku-notebooks-k8s-watcher-7bcbf4fb9d-dpgb7 1/1 Running 0 33m
renku-post-install-keycloak-zsbl6 0/1 Completed 0 33m
renku-post-install-postgres-whnh8 0/1 Completed 0 33m
renku-postgresql-0 1/1 Running 0 33m
renku-redis-node-0 3/3 Running 0 33m
renku-redis-node-1 3/3 Running 0 33m
renku-redis-node-2 3/3 Running 0 33m
renku-swagger-675697d74f-qxhtm 1/1 Running 0 33m
renku-token-repository-7b5d7d79dc-5wr7h 1/1 Running 0 33m
renku-triples-generator-67766bfb94-cr8tb 1/1 Running 0 33m
renku-ui-6445f69656-89qtp 1/1 Running 0 33m
renku-uiserver-d59f58c8f-4vq5k 1/1 Running 1 (32m ago) 33m
renku-webhook-service-69848b588f-blsm5 1/1 Running 0 33m

2

Hello Guillaume,
Could you please share your values file, removing all the secrets?

3

Yes of course. I used the script generate-values.sh while following the instruction on the online documentation.


# This is an automatically generated values file to deploy Renku.
# Please scrutinize it carefully before trying to deploy.
global:
  gateway:
    cliClientSecret: XXX
    clientSecret: XXX
    gitlabClientSecret: XXX
    gitlabClientId: XXX
  gitlab:
    urlPrefix: /
    clientSecret: XXX
  renku:
    domain: enacvm0061.xaas.epfl.ch
  uiserver:
    clientSecret: XXX
  useHTTPS: true
gateway:
  gitlabUrl: https://lsms-gitlab.epfl.ch
  secretKey: XXX
gitlab:
  enabled: false
graph:
  gitlab:
    url: https://lsms-gitlab.epfl.ch
  jena:
    users:
      admin:
        password: XXX
      renku:
        password: XXX
  tokenRepository:
    tokenEncryption:
      secret: XXX
  webhookService:
    hookToken:
      secret: XXX
ingress:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
  enabled: true
  hosts:
  - enacvm0061.xaas.epfl.ch
  tls:
  - hosts:
    - enacvm0061.xaas.epfl.ch
    secretName: renku-tls
notebooks:
  amalthea:
    deployCrd: true
  gitlab:
    registry:
      host: registry.enacvm0061.xaas.epfl.ch
    url: https://lsms-gitlab.epfl.ch
  oidc:
    allowUnverifiedEmail: true
    authUrl: https://enacvm0061.xaas.epfl.ch/auth/realms/Renku/protocol/openid-connect/auth
    clientSecret: XXX
    tokenUrl: https://enacvm0061.xaas.epfl.ch/auth/realms/Renku/protocol/openid-connect/token
  sessionIngress:
    host: enacvm0061.xaas.epfl.ch
    tlsSecret: renku-tls
  userSessionPersistentVolumes:
    enabled: false
postgresql:
  persistence:
    enabled: true
  postgresqlPassword: XXX
redis:
  password: XXX
ui:
  client:
    welcomePage:
      text: |
        ## Welcome to Renku!
        Renku is software for collaborative data science.
        With Renku you can share code and data, discuss problems and solutions, and coordinate data-science projects.
        ## Template
        I am templateable, so deployment specific information can be put here by
        changing ui.client.welcomePage.text in the values file.
4

Thanks for the values.

I did some digging with a colleague and we think that the internal Renku reverse-proxy is unable to reach Gitlab.
Could you please check a few things?

  1. The Kubernetes manifest of the reverse proxy pod (should be called something like renku-gateway-revproxy-xxx), does the env variable called REVPROXY_EXTERNAL_GITLAB_URL have the correct URL to your external Gitlab?

  2. Can you check the logs of the Renku reverse-proxy (same pod as above) and see if there is something there?

  3. Can you reach your Gitlab instance from the Renku reverse-proxy pod (e.g. executing into the pod and then wget https://lsms-gitlab.epfl.ch? If you can reach it from the pod, can you reach it from the machine?

Let me know!

5

What an impressive intuition of yours…

REVPROXY_EXTERNAL_GITLAB_URL have the correct URL to your external Gitlab?

yes that is correct

Can you check the logs of the Renku reverse-proxy (same pod as above) and see if there is something there?

well that is my stupidity: I have tons of access errors.

Can you reach your Gitlab instance from the Renku reverse-proxy pod (e.g. executing into the pod and then wget https://lsms-gitlab.epfl.ch? If you can reach it from the pod, can you reach it from the machine?

Well I realize that these (virtual) hosts live in a different segmented network space… making them firewalled from each other…

This means I have to change my configuration and probably deploy a gitlab on the same k8s.

Thanks a lot for your help!

6

Glad to hear that!
Also credit were credit is due: the debugging insight comes from Tasko Olevski, so we both thank him :slight_smile:

Looking forward to the continuation of your deployment!
Best,
Alessandro.