tjtjtjのメモ

自分のためのメモです

envoy learn/on-your-laptop

このあたり

https://www.envoyproxy.io/learn/on-your-laptop
https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/front_proxy.html

Running Envoy

https://github.com/envoyproxy/envoy の example/examples/front-proxy に今回の学習環境がある。docker-compose up で3個のコンテナが起動する。

$ git clone https://github.com/envoyproxy/envoy
$ cd envoy/examples/front-proxy
$ docker-compose up --build -d
:
$ docker-compose ps
          Name                         Command               State                            Ports
----------------------------------------------------------------------------------------------------------------------------
front-proxy_front-envoy_1   /docker-entrypoint.sh /bin ...   Up      10000/tcp, 0.0.0.0:8000->80/tcp, 0.0.0.0:8001->8001/tcp
front-proxy_service1_1      /bin/sh -c /usr/local/bin/ ...   Up      10000/tcp, 80/tcp
front-proxy_service2_1      /bin/sh -c /usr/local/bin/ ...   Up      10000/tcp, 80/tcp

この図のようになる
https://www.envoyproxy.io/docs/envoy/latest/_images/docker_compose_v0.1.svg

tcp:8080 -> [front_envoy_1(envoy)] -> [service1_1(envoy + service.py)]
                                   -> [service2_1(envoy + service.py)]

service.py

curl の前に service.py を確認しレスポンスを予想する。 service と trace がある。service_number パラメータを受け取ってはいるものの利用していないようだ?

https://github.com/envoyproxy/envoy/blob/master/examples/front-proxy/service.py

@app.route('/service/<service_number>')
def hello(service_number):
  return ('Hello from behind Envoy (service {})! hostname: {} resolved'
          'hostname: {}\n'.format(os.environ['SERVICE_NAME'], socket.gethostname(),
                                  socket.gethostbyname(socket.gethostname())))


@app.route('/trace/<service_number>')
def trace(service_number):
  headers = {}
  # call service 2 from service 1
  if int(os.environ['SERVICE_NAME']) == 1:
    for header in TRACE_HEADERS_TO_PROPAGATE:
      if header in request.headers:
        headers[header] = request.headers[header]
    ret = requests.get("http://localhost:9000/trace/2", headers=headers)
  return ('Hello from behind Envoy (service {})! hostname: {} resolved'
          'hostname: {}\n'.format(os.environ['SERVICE_NAME'], socket.gethostname(),
                                  socket.gethostbyname(socket.gethostname())))

Sending Traffic

service に curl。パラメータで得られるレスポンスが決まっていてラウンドロビンしていない。3 は 404 になった。

$ curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 0cb6d4920789 resolvedhostname: 172.18.0.2

$ curl localhost:8000/service/2
Hello from behind Envoy (service 2)! hostname: 2ccf243aa95e resolvedhostname: 172.18.0.4

$ curl localhost:8000/service/3
[root@env1 front-proxy]# curl -v localhost:8000/service/3
* About to connect() to localhost port 8000 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 8000 (#0)
> GET /service/3 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8000
> Accept: */*
>
< HTTP/1.1 404 Not Found
< date: Thu, 30 May 2019 10:30:57 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host localhost left intact

docker ps でコンテナのhostname を確認。

# docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                                     NAMES
2ccf243aa95e        front-proxy_service2      "/bin/sh -c /usr/loc…"   47 seconds ago      Up 44 seconds       80/tcp, 10000/tcp                                         front-proxy_service2_1
7d80c76c6736        front-proxy_front-envoy   "/docker-entrypoint.…"   47 seconds ago      Up 44 seconds       0.0.0.0:8001->8001/tcp, 10000/tcp, 0.0.0.0:8000->80/tcp   front-proxy_front-envoy_1
0cb6d4920789        front-proxy_service1      "/bin/sh -c /usr/loc…"   47 seconds ago      Up 45 seconds       80/tcp, 10000/tcp                                         front-proxy_service1_1

service1 の /etc/hosts

# docker exec -it 0c sh
/ # cat /etc/hosts
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.18.0.2      0cb6d4920789

service2 の /etc/hosts

# docker exec -it 2c sh
/ # cat /etc/hosts
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.18.0.4      2ccf243aa95e

Configuring Envoy

docker-compose.yaml はまあいいとして、front-envoy.yaml を確認。

https://github.com/envoyproxy/envoy/blob/master/examples/front-proxy/docker-compose.yaml

https://github.com/envoyproxy/envoy/blob/master/examples/front-proxy/front-envoy.yaml

トップレベルには adminstatic_resources がある。admin は管理側として、問題は static_resources 。前回のも static_resources だった。

このstatic_resourcesブロックには、動的に管理されていないクラスターとリスナーの定義が含まれています。クラスタはホスト/ポートの名前付きグループで、Envoyはこれを介してトラフィックの負荷を分散します。リスナーはクライアントが接続できるネットワークの場所という名前です。このadminブロックは管理サーバーを設定します。

私たちのフロントプロキシは、HTTPトラフィックを管理するためにEnvoyを設定するフィルタチェーンを持つ、ポート80でリスンするように設定された単一のリスナを持っています。

テキストとソースが若干異なるので、ソースのほうを載せる。

  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 80
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route

HTTP接続マネージャフィルタの設定には、すべてのドメイントラフィックを受け付けるように設定された単一の仮想ホストの定義があります。

/service/1 はクラスタ:service1に向いていることがわかる。

            virtual_hosts:
            - name: backend
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/service/1"
                route:
                  cluster: service1
              - match:
                  prefix: "/service/2"
                route:
                  cluster: service2

ここでルートが設定され、適切なクラスタへのトラフィック/service/1と/service/2適切なクラスタへのトラフィックマッピングされます。

次に静的なクラスタ定義があります。

  clusters:
  - name: service1
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    http2_protocol_options: {}
    load_assignment:
      cluster_name: service1
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: service1
                port_value: 80
  - name: service2
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    http2_protocol_options: {}
    load_assignment:
      cluster_name: service2
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: service2
                port_value: 80

Step 4: Test Envoy’s load balancing capabilities

こちらを試す https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/front_proxy.html

クラスタ:service1 をスケールさせる。

# docker-compose scale service1=3
WARNING: The scale command is deprecated. Use the up command with the --scale flag instead.
Starting front-proxy_service1_1 ... done
Creating front-proxy_service1_2 ... done
Creating front-proxy_service1_3 ... done

docker-compose ps と docker ps

# docker-compose ps
          Name                         Command               State                            Ports
----------------------------------------------------------------------------------------------------------------------------
front-proxy_front-envoy_1   /docker-entrypoint.sh /bin ...   Up      10000/tcp, 0.0.0.0:8000->80/tcp, 0.0.0.0:8001->8001/tcp
front-proxy_service1_1      /bin/sh -c /usr/local/bin/ ...   Up      10000/tcp, 80/tcp
front-proxy_service1_2      /bin/sh -c /usr/local/bin/ ...   Up      10000/tcp, 80/tcp
front-proxy_service1_3      /bin/sh -c /usr/local/bin/ ...   Up      10000/tcp, 80/tcp
front-proxy_service2_1      /bin/sh -c /usr/local/bin/ ...   Up      10000/tcp, 80/tcp

# docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                                     NAMES
280801f8ad72        front-proxy_service1      "/bin/sh -c /usr/loc…"   3 minutes ago       Up 3 minutes        80/tcp, 10000/tcp                                         front-proxy_service1_3
0d1c580854e8        front-proxy_service1      "/bin/sh -c /usr/loc…"   3 minutes ago       Up 3 minutes        80/tcp, 10000/tcp                                         front-proxy_service1_2
2ccf243aa95e        front-proxy_service2      "/bin/sh -c /usr/loc…"   About an hour ago   Up About an hour    80/tcp, 10000/tcp                                         front-proxy_service2_1
7d80c76c6736        front-proxy_front-envoy   "/docker-entrypoint.…"   About an hour ago   Up About an hour    0.0.0.0:8001->8001/tcp, 10000/tcp, 0.0.0.0:8000->80/tcp   front-proxy_front-envoy_1
0cb6d4920789        front-proxy_service1      "/bin/sh -c /usr/loc…"   About an hour ago   Up About an hour    80/tcp, 10000/tcp                                         front-proxy_service1_1

curlクラスタ:service1 はスケールアップしている

# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 280801f8ad72 resolvedhostname: 172.18.0.6

# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 0cb6d4920789 resolvedhostname: 172.18.0.2

# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 0d1c580854e8 resolvedhostname: 172.18.0.5

クラスタ:service2 は変わらない

# curl localhost:8000/service/2
Hello from behind Envoy (service 2)! hostname: 2ccf243aa95e resolvedhostname: 172.18.0.4

Step 5: enter containers and curl services

ここまでは、クラスタ外からフロントを通してのリクエストだったが、クラスタ内からリクエストする。

フロントコンテナからlocalhostcurl

# docker exec -it front-proxy_front-envoy_1 sh

# curl localhost:80/service/1
Hello from behind Envoy (service 1)! hostname: 0cb6d4920789 resolvedhostname: 172.18.0.2
# curl localhost:80/service/1
Hello from behind Envoy (service 1)! hostname: 0d1c580854e8 resolvedhostname: 172.18.0.5
# curl localhost:80/service/1
Hello from behind Envoy (service 1)! hostname: 280801f8ad72 resolvedhostname: 172.18.0.6
# curl localhost:80/service/2
Hello from behind Envoy (service 2)! hostname: 2ccf243aa95e resolvedhostname: 172.18.0.4

フロントコンテナからクラスタ名へcurl。このとき service_number パラメータでルーティングされていないことがわかる。

# curl service1/service/2
Hello from behind Envoy (service 1)! hostname: 280801f8ad72 resolvedhostname: 172.18.0.6
# curl service1/service/1
Hello from behind Envoy (service 1)! hostname: 0d1c580854e8 resolvedhostname: 172.18.0.5
# curl service1/service/1
Hello from behind Envoy (service 1)! hostname: 0cb6d4920789 resolvedhostname: 172.18.0.2
# curl service2/service/1
Hello from behind Envoy (service 2)! hostname: 2ccf243aa95e resolvedhostname: 172.18.0.4

service1クラスタのコンテナからlocalhostcurl。何度やってもかわらない。そうだろうね。プロンプトに/ が表示されてるがわざとかな?

# docker exec -it front-proxy_service1_1 sh
/ # curl localhost:80/service/1
Hello from behind Envoy (service 1)! hostname: 0cb6d4920789 resolvedhostname: 172.18.0.2
/ # curl localhost:80/service/1
Hello from behind Envoy (service 1)! hostname: 0cb6d4920789 resolvedhostname: 172.18.0.2

service1クラスタのコンテナからクラスタ名へcurl

/ # curl service1/service/2
Hello from behind Envoy (service 1)! hostname: 280801f8ad72 resolvedhostname: 172.18.0.6
/ # curl service1/service/1
Hello from behind Envoy (service 1)! hostname: 280801f8ad72 resolvedhostname: 172.18.0.6
/ # curl service1/service/1
Hello from behind Envoy (service 1)! hostname: 0cb6d4920789 resolvedhostname: 172.18.0.2
/ # curl service1/service/1
Hello from behind Envoy (service 1)! hostname: 0d1c580854e8 resolvedhostname: 172.18.0.5

/ # curl service2/service/2
Hello from behind Envoy (service 2)! hostname: 2ccf243aa95e resolvedhostname: 172.18.0.4

Step 6: enter containers and curl admin

フロントから admin にcurl。server_info

# docker exec -it front-proxy_front-envoy_1 sh

# curl localhost:8001/server_info
{
 "version": "4dafba65baaf9769723f895761268eed31af629b/1.11.0-dev/Clean/RELEASE/BoringSSL",
 "state": "LIVE",
 "command_line_options": {
  "base_id": "0",
  "concurrency": 2,
  "config_path": "/etc/front-envoy.yaml",
  "config_yaml": "",
  "allow_unknown_fields": false,
  "admin_address_path": "",
  "local_address_ip_version": "v4",
  "log_level": "info",
  "component_log_level": "",
  "log_format": "[%Y-%m-%d %T.%e][%t][%l][%n] %v",
  "log_path": "",
  "hot_restart_version": false,
  "service_cluster": "front-proxy",
  "service_node": "",
  "service_zone": "",
  "mode": "Serve",
  "max_stats": "0",
  "max_obj_name_len": "0",
  "disable_hot_restart": false,
  "enable_mutex_tracing": false,
  "restart_epoch": 0,
  "cpuset_threads": false,
  "file_flush_interval": "10s",
  "drain_time": "600s",
  "parent_shutdown_time": "900s"
 },
 "uptime_current_epoch": "5116s",
 "uptime_all_epochs": "5116s"
}

stats

# curl localhost:8001/stats
access_log_file.flushed_by_timer: 269
access_log_file.reopen_failed: 0
access_log_file.write_buffered: 3
access_log_file.write_completed: 3
access_log_file.write_total_buffered: 0
cluster.service1.assignment_stale: 0
cluster.service1.assignment_timeout_received: 0
cluster.service1.bind_errors: 0
cluster.service1.circuit_breakers.default.cx_open: 0
cluster.service1.circuit_breakers.default.cx_pool_open: 0
:

コンテナ止めたらどうなる?

コンテナをstopしたらどうなるか試した。1回失敗したw。その後は問題なかった。

# docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                                     NAMES
280801f8ad72        front-proxy_service1      "/bin/sh -c /usr/loc…"   30 minutes ago      Up 30 minutes       80/tcp, 10000/tcp                                         front-proxy_service1_3
0d1c580854e8        front-proxy_service1      "/bin/sh -c /usr/loc…"   30 minutes ago      Up 30 minutes       80/tcp, 10000/tcp                                         front-proxy_service1_2
2ccf243aa95e        front-proxy_service2      "/bin/sh -c /usr/loc…"   2 hours ago         Up 2 hours          80/tcp, 10000/tcp                                         front-proxy_service2_1
7d80c76c6736        front-proxy_front-envoy   "/docker-entrypoint.…"   2 hours ago         Up 2 hours          0.0.0.0:8001->8001/tcp, 10000/tcp, 0.0.0.0:8000->80/tcp   front-proxy_front-envoy_1
0cb6d4920789        front-proxy_service1      "/bin/sh -c /usr/loc…"   2 hours ago         Up 2 hours          80/tcp, 10000/tcp                                         front-proxy_service1_1
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 0d1c580854e8 resolvedhostname: 172.18.0.5
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 0cb6d4920789 resolvedhostname: 172.18.0.2
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 0d1c580854e8 resolvedhostname: 172.18.0.5
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 280801f8ad72 resolvedhostname: 172.18.0.6
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 0cb6d4920789 resolvedhostname: 172.18.0.2
# docker stop 0c
0c
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 0d1c580854e8 resolvedhostname: 172.18.0.5
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 280801f8ad72 resolvedhostname: 172.18.0.6
# curl localhost:8000/service/1
upstream connect error or disconnect/reset before headers. reset reason: connection failure
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 280801f8ad72 resolvedhostname: 172.18.0.6
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 0d1c580854e8 resolvedhostname: 172.18.0.5
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 280801f8ad72 resolvedhostname: 172.18.0.6
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 0d1c580854e8 resolvedhostname: 172.18.0.5
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 0d1c580854e8 resolvedhostname: 172.18.0.5
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 280801f8ad72 resolvedhostname: 172.18.0.6
# curl localhost:8000/service/1
Hello from behind Envoy (service 1)! hostname: 280801f8ad72 resolvedhostname: 172.18.0.6