API calls fail with timeout errors

You're viewing Apigee and Apigee hybrid documentation.
View Apigee Edge documentation.

Symptom

You may observe one of the following symptoms:

  1. The client applications get timeout errors as a response for API calls on Apigee hybrid.
  2. You observe errors such as Error from server (invalid) or The Job "apigee-resources-install" is invalid while applying configuration (overrides.yaml) to cluster during Apigee hybrid installation.

Error messages

You may observe one of the following errors:

Error response to API calls

The API requests on Apigee hybrid may fail with the following error message:

* Connection failed
* connect to 34.84.67.39 port 443 failed: Operation timed out
* Failed to connect to example.apis.com port 443: Operation timed out
* Closing connection 0
curl: (7) Failed to connect to example.apis.com port 443: Operation timed out

Errors observed while applying configuration (overrides.yaml) to clusters

You may observe one of the following errors while applying configuration (overrides.yaml file) to clusters during the installation:

Error #1

apigeectl init -f overrides/overrides.yaml

...
...
Error from server (Invalid): error when applying patch:
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "istio-init-crd-10-1.4.6", Namespace: "istio-system"
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "istio-init-crd-11-1.4.6", Namespace: "istio-system"
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "istio-init-crd-14-1.4.6", Namespace: "istio-system"

Error #2

apigeectl init -f overrides/overrides.yaml

...
...
The Job "apigee-resources-install" is invalid: spec.template: Invalid value:
core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"apigee-resources-install",
GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"",
Generation:0,

Possible causes

These errors can happen if the istio-ingressgateway service is in a pending state and unable to bind to an external IP address as shown below:

kubectl get services -n istio-system
NAME                      TYPE         CLUSTER-IP   EXTERNAL-IP  PORT(S)             AGE
istio-ingressgateway      LoadBalancer 10.198.5.104 <pending>    15020:31927/TCP,    12h
                                                                 80:31381/TCP,
                                                                 443:31391/TCP,
                                                                 31400:31401/TCP,
                                                                 15443:32623/TCP

The possible causes for the istio-ingressgateway service to be in a pending state are as follows:

Cause Description
Jobs in erroneous/pending state in istio-system namespace The incomplete/erroneous jobs in the istio-system namespace could cause the istio-ingressgateway service to be in pending state forever and unable to bind to an external IP address.
apigee-resources-install job in erroneous/pending state in apigee-system namespace The incomplete jobs in the apigee-system namespace could cause the istio- ingressgateway service to be in pending state forever and unable to bind to an external IP address.
Incorrect IP address range assigned to external load balancer An incorrect IP address range may be configured in the istio-operator.yaml file causing the istio-ingressgateway service to get into pending state forever and unable to bind to an external IP address during the installation.

Cause: Jobs in istio-system namespace in erroneous/pending state

Diagnosis

  1. Check the status of the jobs in the istio-system namespace using the following command:
    kubectl get jobs -n istio-system
    
  2. The status of the jobs must be complete. If the status of the jobs is in an erroneous/pending state, then that's the cause of this problem.

Resolution

  1. If any of the jobs are in the pending or erroneous state, delete them using the following command:
    kubectl -n istio-system delete job JOB_NAME_FROM_STEP_1
    
  2. Re-run the installation by applying the overrides.yaml file:
    apigeectl apply -f overrides.yaml
    

Cause: apigee-resources-install job in the apigee-system namespace may be in erroneous state

Diagnosis

  1. Check the status of the jobs in the apigee-system namespace using the following command:
    kubectl get jobs -n apigee-system
    
  2. The status of the jobs must be complete. If the status of the jobs is in an erroneous/pending state, then that's the cause of this problem. The following sample output shows that the job apigee-resources-install is successfully completed.
    kubectl get jobs  -n apigee-system
    NAME                       COMPLETIONS   DURATION   AGE
    apigee-resources-install   1/1           23s        16d
    

Resolution

  1. If the jobs are in the pending or erroneous state, delete them using the following command:
    kubectl -n apigee-system delete job JOB_NAME_FROM_STEP_1
    
  2. Re-run the installation by applying the overrides.yaml file:
    apigeectl apply -f overrides.yaml
    

Cause: Incorrect IP address range assigned to external load balancer

Diagnosis

  1. Check the IP address configured for the load balancer in the istio- operator.yaml file. For example, the following snippet shows the location in the istio-operator.yaml file where the IP address is configured:
    -name: istio-ingressgateway
      enabled: true
      k8s:
        service:
          type: LoadBalancer
         loadBalancerIP: 10.195.24.23
  2. The istio-ingressgateway service is configured as a load balancer (indicated by type) in the istio-operator.yaml file. During the ASM installation, a load balancer is created with the configured IP address and wired to communicate with the istio- ingressgateway service. Therefore, the IP address configured should be correct and reserved for the load balancer.
  3. Engage your network team and verify that the IP address configured for loadBalancerIP is correct. If it is incorrect, then the load balancer service will not be able to bind to the IP address. This causes the istio-ingressgateway service to be in the pending state forever.

Resolution

  1. Work with your network team and configure the correct IP address in the istio- operator.yaml file.
  2. Re-run ASM installation and apply the overrides.yaml file:
    apigeectl apply -f overrides.yaml
    

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:

  1. The Google Cloud Project ID
  2. The name of the Apigee hybrid organization
  3. Kubernetes Cluster name
  4. Google Cloud project name if kubernetes cluster resides in different Google Cloud project
  5. The overrides.yaml file
  6. The Istio-operator .yaml file used during the ASM installation.
  7. Collect the logs from each istio-ingressgateway pod in the istio-system namespace:
    kubectl logs NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system > /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.log
    
  8. Collect the description of the each pod in the istio-system namespace:
    kubectl describe pod NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system > /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.yaml
    
  9. Collect the list of services in the istio-system namespace:
    kubectl get svc -n istio-system