consul_kv
Summary#
For users who used nginx-upsync-module and consul key value for service discovery way, as we Weibo Mobile Team, maybe need it.
Thanks to @fatman-x guy, who developed this module, called consul_kv, and its worker process data flow is below:
Configuration for discovery client#
Configuration for Consul KV#
Add following configuration in conf/config.yaml :
discovery: consul_kv: servers: - "http://127.0.0.1:8500" - "http://127.0.0.1:8600" prefix: "upstreams" skip_keys: # if you need to skip special keys - "upstreams/unused_api/" timeout: connect: 1000 # default 2000 ms read: 1000 # default 2000 ms wait: 60 # default 60 sec weight: 1 # default 1 fetch_interval: 5 # default 3 sec, only take effect for keepalive: false way keepalive: true # default true, use the long pull way to query consul servers default_server: # you can define default server when missing hit host: "127.0.0.1" port: 20999 metadata: fail_timeout: 1 # default 1 ms weight: 1 # default 1 max_fails: 1 # default 1 dump: # if you need, when registered nodes updated can dump into file path: "logs/consul_kv.dump" expire: 2592000 # unit sec, here is 30 dayAnd you can config it in short by default value:
discovery: consul_kv: servers: - "http://127.0.0.1:8500"The keepalive has two optional values:
true, default and recommend value, use the long pull way to query consul serversfalse, not recommend, it would use the short pull way to query consul servers, then you can set thefetch_intervalfor fetch interval
Dump Data#
When we need reload apisix online, as the consul_kv module maybe loads data from CONSUL slower than load routes from ETCD, and would get the log at the moment before load successfully from consul:
http_access_phase(): failed to set upstream: no valid upstream nodeSo, we import the dump function for consul_kv module. When reload, would load the dump file before from consul; when the registered nodes in consul been updated, would dump the upstream nodes into file automatically.
The dump has three optional values now:
path, the dump file save path- support relative path, eg:
logs/consul_kv.dump - support absolute path, eg:
/tmp/consul_kv.bin - make sure the dump file's parent path exist
- make sure the
apisixhas the dump file's read-write access permission,eg:chown www:root conf/upstream.d/
- support relative path, eg:
load_on_init, default value istrue- if
true, just try to load the data from the dump file before loading data from consul when starting, does not care the dump file exists or not - if
false, ignore loading data from the dump file - Whether
trueorfalse, we don't need to prepare a dump file for apisix at anytime
- if
expire, unit sec, avoiding load expired dump data when load- default
0, it is unexpired forever - recommend 2592000, which is 30 days(equals 3600 * 24 * 30)
- default
Register Http API Services#
Service register Key&Value template:
Key: {Prefix}/{Service_Name}/{IP}:{Port}Value: {"weight": <Num>, "max_fails": <Num>, "fail_timeout": <Num>}The register consul key use upstreams as prefix by default. The http api service name called webpages for example, and you can also use webpages/oneteam/hello as service name. The api instance of node's ip and port make up new key: <IP>:<Port>.
Now, register nodes into consul:
curl \ -X PUT \ -d ' {"weight": 1, "max_fails": 2, "fail_timeout": 1}' \ http://127.0.0.1:8500/v1/kv/upstreams/webpages/172.19.5.12:8000
curl \ -X PUT \ -d ' {"weight": 1, "max_fails": 2, "fail_timeout": 1}' \ http://127.0.0.1:8500/v1/kv/upstreams/webpages/172.19.5.13:8000In some case, same keys exist in different consul servers. To avoid confusion, use the full consul key url path as service name in practice.
Upstream setting#
Here is an example of routing a request with a URL of "/*" to a service which named "http://127.0.0.1:8500/v1/kv/upstreams/webpages/" and use consul_kv discovery client in the registry :
$ curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d '{ "uri": "/*", "upstream": { "service_name": "http://127.0.0.1:8500/v1/kv/upstreams/webpages/", "type": "roundrobin", "discovery_type": "consul_kv" }}'The format response as below:
{ "node": { "value": { "priority": 0, "update_time": 1612755230, "upstream": { "discovery_type": "consul_kv", "service_name": "http://127.0.0.1:8500/v1/kv/upstreams/webpages/", "hash_on": "vars", "type": "roundrobin", "pass_host": "pass" }, "id": "1", "uri": "/*", "create_time": 1612755230, "status": 1 }, "key": "/apisix/routes/1" }, "action": "set"}You could find more usage in the apisix/t/discovery/consul_kv.t file.
Debugging API#
It also offers control api for debugging.
Memory Dump API#
GET /v1/discovery/consul_kv/dumpFor example:
# curl http://127.0.0.1:9090/v1/discovery/consul_kv/dump | jq{ "config": { "fetch_interval": 3, "timeout": { "wait": 60, "connect": 6000, "read": 6000 }, "prefix": "upstreams", "weight": 1, "servers": [ "http://172.19.5.30:8500", "http://172.19.5.31:8500" ], "keepalive": true, "default_service": { "host": "172.19.5.11", "port": 8899, "metadata": { "fail_timeout": 1, "weight": 1, "max_fails": 1 } }, "skip_keys": [ "upstreams/myapi/gateway/apisix/" ] }, "services": { "http://172.19.5.31:8500/v1/kv/upstreams/webpages/": [ { "host": "127.0.0.1", "port": 30513, "weight": 1 }, { "host": "127.0.0.1", "port": 30514, "weight": 1 } ], "http://172.19.5.30:8500/v1/kv/upstreams/1614480/grpc/": [ { "host": "172.19.5.51", "port": 50051, "weight": 1 } ], "http://172.19.5.30:8500/v1/kv/upstreams/webpages/": [ { "host": "127.0.0.1", "port": 30511, "weight": 1 }, { "host": "127.0.0.1", "port": 30512, "weight": 1 } ] }}Show Dump File API#
It offers another control api for dump file view now. Maybe would add more api for debugging in future.
GET /v1/discovery/consul_kv/show_dump_fileFor example:
curl http://127.0.0.1:9090/v1/discovery/consul_kv/show_dump_file | jq{ "services": { "http://172.19.5.31:8500/v1/kv/upstreams/1614480/webpages/": [ { "host": "172.19.5.12", "port": 8000, "weight": 120 }, { "host": "172.19.5.13", "port": 8000, "weight": 120 } ] }, "expire": 0, "last_update": 1615877468}