exportETCDCTL_API=3ENDPOINTS=localhost:2379
etcdctl --endpoints=${ENDPOINTS} role add root
etcdctl --endpoints=${ENDPOINTS} role get root
etcdctl --endpoints=${ENDPOINTS} user add root
etcdctl --endpoints=${ENDPOINTS} user grant-role root root
etcdctl --endpoints=${ENDPOINTS} user get root
etcdctl --endpoints=${ENDPOINTS} role add role0
etcdctl --endpoints=${ENDPOINTS} role grant-permission role0 readwrite foo
etcdctl --endpoints=${ENDPOINTS} user add user0
etcdctl --endpoints=${ENDPOINTS} user grant-role user0 role0
etcdctl --endpoints=${ENDPOINTS} auth enable# now all client requests go through authetcdctl --endpoints=${ENDPOINTS} --user=user0:123 put foo bar
etcdctl --endpoints=${ENDPOINTS} get foo
# permission denied, user name is empty because the request does not issue an authentication requestetcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo
# user0 can read the key fooetcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo1
etcdは、--user username:による空のパスワードでの認証に対応していません。たとえば、etcdctl user add anonymous:''のように空のパスワードで作成したユーザーは、ユーザー名とパスワードを使うリクエストでは認証できません。etcdctl --user anonymous: get fooなどのリクエストは、user name is emptyで失敗します。
次のコマンドで、ユーザーにロールを付与したり、取り消したりできます。
$ etcdctl user grant-role myusername foo
$ etcdctl user revoke-role myusername bar
# Give read access to a key /foo
$ etcdctl role grant-permission myrolename read /foo
# Give read access to keys with a prefix /foo/. The prefix is equal to the range [/foo/, /foo0)
$ etcdctl role grant-permission myrolename --prefix=true read /foo/
# Give write-only access to the key at /foo/bar
$ etcdctl role grant-permission myrolename write /foo/bar
# Give full access to keys in a range of [key1, key5)
$ etcdctl role grant-permission myrolename readwrite key1 key5
# Give full access to keys with a prefix /pub/
$ etcdctl role grant-permission myrolename --prefix=true readwrite /pub/
付与されている権限は、いつでもロールを参照して確認できます。
$ etcdctl role get myrolename
権限の取り消しも同様に行います。
$ etcdctl role revoke-permission myrolename /foo/bar
--name 'default'
Human-readable name for this member.
--data-dir '${name}.etcd'
Path to the data directory.
--wal-dir ''
Path to the dedicated wal directory.
--snapshot-count '10000'
Number of committed transactions to trigger a snapshot to disk.
--heartbeat-interval '100'
Time (in milliseconds) of a heartbeat interval.
--election-timeout '1000'
Time (in milliseconds) for an election to timeout. See tuning documentation for details.
--initial-election-tick-advance 'true'
Whether to fast-forward initial election ticks on boot for faster election.
--listen-peer-urls 'http://localhost:2380'
List of URLs to listen on for peer traffic.
--listen-client-urls 'http://localhost:2379'
List of URLs to listen on for client grpc traffic and http as long as --listen-client-http-urls is not specified.
--listen-client-http-urls ''
List of URLs to listen on for http only client traffic. Enabling this flag removes http services from --listen-client-urls.
--max-snapshots '5'
Maximum number of snapshot files to retain (0 is unlimited).
--max-wals '5'
Maximum number of wal files to retain (0 is unlimited).
--memory-mlock
Enable to enforce etcd pages (in particular bbolt) to stay in RAM.
--quota-backend-bytes '0'
Raise alarms when backend size exceeds the given quota (0 defaults to low space quota).
--backend-bbolt-freelist-type 'map'
BackendFreelistType specifies the type of freelist that boltdb backend uses(array and map are supported types).
--backend-batch-interval ''
BackendBatchInterval is the maximum time before commit the backend transaction.
--backend-batch-limit '0'
BackendBatchLimit is the maximum operations before commit the backend transaction.
--max-txn-ops '128'
Maximum number of operations permitted in a transaction.
--max-request-bytes '1572864'
Maximum client request size in bytes the server will accept.
--grpc-keepalive-min-time '5s'
Minimum duration interval that a client should wait before pinging server.
--grpc-keepalive-interval '2h'
Frequency duration of server-to-client ping to check if a connection is alive (0 to disable).
--grpc-keepalive-timeout '20s'
Additional duration of wait before closing a non-responsive connection (0 to disable).
--socket-reuse-port 'false'
Enable to set socket option SO_REUSEPORT on listeners allowing rebinding of a port already in use.
--socket-reuse-address 'false'
Enable to set socket option SO_REUSEADDR on listeners allowing binding to an address in TIME_WAIT state.
クラスタリング
--initial-advertise-peer-urls 'http://localhost:2380'
List of this member's peer URLs to advertise to the rest of the cluster.
--initial-cluster 'default=http://localhost:2380'
Initial cluster configuration for bootstrapping.
--initial-cluster-state 'new'
Initial cluster state ('new' or 'existing').
--initial-cluster-token 'etcd-cluster'
Initial cluster token for the etcd cluster during bootstrap.
Specifying this can protect you from unintended cross-cluster interaction when running multiple clusters.
--advertise-client-urls 'http://localhost:2379'
List of this member's client URLs to advertise to the public.
The client URLs advertised should be accessible to machines that talk to etcd cluster. etcd client libraries parse these URLs to connect to the cluster.
--discovery ''
Discovery URL used to bootstrap the cluster.
--discovery-fallback 'proxy'
Expected behavior ('exit' or 'proxy') when discovery services fails.
"proxy" supports v2 API only.
--discovery-proxy ''
HTTP proxy to use for traffic to discovery service.
--discovery-srv ''
DNS srv domain used to bootstrap the cluster.
--discovery-srv-name ''
Suffix to the dns srv name queried when bootstrapping.
--strict-reconfig-check 'true'
Reject reconfiguration requests that would cause quorum loss.
--pre-vote 'true'
Enable the raft Pre-Vote algorithm to prevent disruption when a node that has been partitioned away rejoins the cluster.
--auto-compaction-retention '0'
Auto compaction retention length. 0 means disable auto compaction.
--auto-compaction-mode 'periodic'
Interpret 'auto-compaction-retention' one of: periodic|revision. 'periodic' for duration based retention, defaulting to hours if no time unit is provided (e.g. '5m'). 'revision' for revision number based retention.
--enable-v2 'false'
Accept etcd V2 client requests. Deprecated and to be decommissioned in v3.6.
--v2-deprecation 'not-yet'
Phase of v2store deprecation. Allows to opt-in for higher compatibility mode.
Supported values:
'not-yet' // Issues a warning if v2store have meaningful content (default in v3.5)
'write-only' // Custom v2 state is not allowed (default in v3.6 and v3.7)
'write-only-skip-check' // Custom v2 state is not supported and, if present, will be ignored (available in v3.5.32+, v3.6.13+, and v3.7.0+). Use this option at your own risk.
'write-only-drop-data' // Custom v2 state will get DELETED ! (planned default in v3.8)
'gone' // v2store is not maintained any longer.
セキュリティー
--cert-file ''
Path to the client server TLS cert file.
--key-file ''
Path to the client server TLS key file.
--client-cert-auth 'false'
Enable client cert authentication.
It's recommended to enable client cert authentication to prevent attacks from unauthenticated clients (e.g. CVE-2023-44487), especially when running etcd as a public service.
--client-crl-file ''
Path to the client certificate revocation list file.
--client-cert-allowed-hostname ''
Comma-separated list of SAN hostnames for client cert authentication.
--trusted-ca-file ''
Path to the client server TLS trusted CA cert file.
Note setting this parameter will also automatically enable client cert authentication no matter what value is set for `--client-cert-auth`.
--auto-tls 'false'
Client TLS using generated certificates.
--peer-cert-file ''
Path to the peer server TLS cert file.
--peer-key-file ''
Path to the peer server TLS key file.
--peer-client-cert-auth 'false'
Enable peer client cert authentication.
It's recommended to enable peer client cert authentication to prevent attacks from unauthenticated forged peers (e.g. CVE-2023-44487).
--peer-trusted-ca-file ''
Path to the peer server TLS trusted CA file.
--peer-cert-allowed-cn ''
Comma-separated list of allowed CNs for inter-peer TLS authentication.
--peer-cert-allowed-hostname ''
Comma-separated list of allowed SAN hostnames for inter-peer TLS authentication.
--peer-auto-tls 'false'
Peer TLS using self-generated certificates if --peer-key-file and --peer-cert-file are not provided.
--self-signed-cert-validity '1'
The validity period of the client and peer certificates that are automatically generated by etcd when you specify ClientAutoTLS and PeerAutoTLS, the unit is year, and the default is 1.
--peer-crl-file ''
Path to the peer certificate revocation list file.
--cipher-suites ''
Comma-separated list of supported TLS cipher suites between client/server and peers (empty will be auto-populated by Go).
--cors '*'
Comma-separated whitelist of origins for CORS, or cross-origin resource sharing, (empty or * means allow all).
--host-whitelist '*'
Acceptable hostnames from HTTP client requests, if server is not secure (empty or * means allow all).
--tls-min-version 'TLS1.2'
Minimum TLS version supported by etcd.
--tls-max-version ''
Maximum TLS version supported by etcd (empty will be auto-populated by Go).
認証
--auth-token 'simple'
Specify a v3 authentication token type and its options ('simple' or 'jwt').
--bcrypt-cost 10
Specify the cost / strength of the bcrypt algorithm for hashing auth passwords. Valid values are between 4 and 31.
--auth-token-ttl 300
Time (in seconds) of the auth-token-ttl.
プロファイリングと監視
--enable-pprof 'false'
Enable runtime profiling data via HTTP server. Address is at client URL + "/debug/pprof/"
--metrics 'basic'
Set level of detail for exported metrics, specify 'extensive' to include server side grpc histogram metrics.
--listen-metrics-urls ''
List of URLs to listen on for the metrics and health endpoints.
ログ
--logger 'zap'
Currently only supports 'zap' for structured logging.
--log-outputs 'default'
Specify 'stdout' or 'stderr' to skip journald logging even when running under systemd, or list of comma separated output targets.
--log-level 'info'
Configures log level. Only supports debug, info, warn, error, panic, or fatal.
--log-format 'json'
Configures log format. Only supports json, console.
--enable-log-rotation 'false'
Enable log rotation of a single log-outputs file target.
--log-rotation-config-json '{"maxsize": 100, "maxage": 0, "maxbackups": 0, "localtime": false, "compress": false}'
Configures log rotation if enabled with a JSON logger config. MaxSize(MB), MaxAge(days,0=no limit), MaxBackups(0=no limit), LocalTime(use computers local time), Compress(gzip)".
--warning-unary-request-duration '300ms'
Set time duration after which a warning is logged if a unary request takes more than this duration.
--enable-distributed-tracing 'false'
Enable distributed tracing.
--distributed-tracing-address 'localhost:4317'
Distributed tracing collector address.
--distributed-tracing-service-name 'etcd'
Distributed tracing service name, must be the same across all etcd instances.
--distributed-tracing-instance-id ''
Distributed tracing instance ID, must be unique for each etcd instance.
--distributed-tracing-sampling-rate '0'
Number of samples to collect per million spans for distributed tracing.
--proxy 'off'
Proxy mode setting ('off', 'readonly' or 'on').
--proxy-failure-wait 5000
Time (in milliseconds) an endpoint will be held in a failed state.
--proxy-refresh-interval 30000
Time (in milliseconds) of the endpoints refresh interval.
--proxy-dial-timeout 1000
Time (in milliseconds) for a dial to timeout.
--proxy-write-timeout 5000
Time (in milliseconds) for a write to timeout.
--proxy-read-timeout 0
Time (in milliseconds) for a read to timeout.
機能
--corrupt-check-time '0s'
Duration of time between cluster corruption check passes.
--compact-hash-check-time '1m'
Duration of time between leader checks followers compaction hashes.
--compaction-batch-limit 1000
CompactionBatchLimit sets the maximum revisions deleted in each compaction batch.
--peer-skip-client-san-verification 'false'
Skip verification of SAN field in client certificate for peer connections.
--watch-progress-notify-interval '10m'
Duration of periodical watch progress notification.
--warning-apply-duration '100ms'
Warning is generated if requests take more than this duration.
--bootstrap-defrag-threshold-megabytes
Enable the defrag during etcd server bootstrap on condition that it will free at least the provided threshold of disk space. Needs to be set to non-zero value to take effect.
--max-learners '1'
Set the max number of learner members allowed in the cluster membership.
--compaction-sleep-interval
Sets the sleep interval between each compaction batch.
--downgrade-check-time
Duration of time between two downgrade status checks.
--snapshot-catchup-entries
Number of entries for a slow follower to catch up after compacting the raft storage entries.
フィーチャーゲート
--feature-gates=AllAlpha=true|false
Enables or disables all alpha features. Default is false.
--feature-gates=AllBeta=true|false
Enables or disables all beta features. Default is false.
--feature-gates=CompactHashCheck=true
Enables leader to periodically check follower compaction hashes.
Replaces: --experimental-compact-hash-check-enabled
--feature-gates=InitialCorruptCheck=true
Enables corruption check before serving client/peer traffic.
Replaces: --experimental-initial-corrupt-check
--feature-gates=LeaseCheckpoint=true
ExperimentalEnableLeaseCheckpoint enables primary lessor to persist lease remainingTTL to prevent indefinite auto-renewal of long lived leases.
Replaces: --experimental-enable-lease-checkpoint
--feature-gates=LeaseCheckpointPersist=true
Enable persisting remainingTTL to prevent indefinite auto-renewal of long lived leases. Always enabled in v3.6. Should be used to ensure smooth upgrade from v3.5 clusters with this feature enabled.
Replaces: --experimental-enable-lease-checkpoint-persist
--feature-gates=SetMemberLocalAddr=true
Allows setting a member’s local address.
--feature-gates=StopGRPCServiceOnDefrag=true
Enable etcd gRPC service to stop serving client requests on defragmentation.
Replaces: --experimental-stop-grpc-service-on-defrag
--feature-gates=TxnModeWriteWithSharedBuffer=true
Enable the write transaction to use a shared buffer in its readonly check operations.
Replaces: --experimental-txn-mode-write-with-shared-buffer
# Correct: 10 minutes in nanosecondswatch-progress-notify-interval:600000000000# Incorrect: will produce an unmarshal errorwatch-progress-notify-interval:'10m'
v3.1.0以降（v3.2.9を除く）、SRVディスカバリーによるブートストラップは、--discovery-srvフラグのルートドメイン名でServerNameを認証します。証明書のSubject Alternative Name（SAN）フィールドに一致するルートドメイン名を要求することで、証明書を使う中間者攻撃を防ぎます。たとえば、etcd --discovery-srv=etcd.localでは、提供された証明書のSubject Alternative Name（SAN）フィールドにルートドメインetcd.localが含まれる場合にのみ、ピアやクライアントを認証します。
v3.2.0
以降、サーバーはIPのSANが正しくないピア証明書を拒否します
。たとえば、ピア証明書のSubject Alternative Name（SAN）フィールドにIPアドレスがある場合、リモートIPアドレスがそのいずれかと一致するときだけ認証します。これは、許可されていないエンドポイントがクラスターに参加することを防ぐためです。たとえば、ピアBのCSR（cfsslを使用）が次のとおりだとします。
一方、ピアBの実際のIPアドレスは10.138.0.27ではなく10.138.0.2です。Bがクラスターに参加しようとすると、ピアAはx509: certificate is valid for 10.138.0.27, not 10.138.0.2エラーで拒否します。BのリモートIPアドレスがSubject Alternative Name（SAN）フィールドのアドレスと一致しないためです。
v3.2.0
以降、サーバーはSANの検証時にTLSのDNSNamesを名前解決します
。たとえば、ピア証明書のSubject Alternative Name（SAN）フィールドにDNS名だけがあり、IPアドレスがない場合、そのDNS名を正引き（dig b.com）したIPがリモートIPアドレスに一致するときだけ、サーバーはピアを認証します。たとえば、ピアBのCSR（cfsslを使用）が次のとおりだとします。
{"CN":"etcd peer","hosts":["b.com"],
ピアBのリモートIPアドレスは10.138.0.2です。Bがクラスターに参加しようとすると、ピアAは接続してきたホストb.comを名前解決し、IPアドレスの一覧を取得します（dig b.comなど）。一覧にIPアドレス10.138.0.2がなければ、tls: 10.138.0.2 does not match any of DNSNames ["b.com"]エラーでBを拒否します。
v3.2.2
以降、IPが一致すれば、サーバーはDNSエントリーを確認せずに接続を受け入れます
。たとえば、ピア証明書のSubject Alternative Name（SAN）フィールドにIPアドレスとDNS名があり、リモートIPアドレスがそのいずれかに一致する場合、サーバーはDNS名を追加で確認せずに接続を受け入れます。たとえば、ピアBのCSR（cfsslを使用）が次のとおりだとします。
ピアBのリモートIPアドレスは10.138.0.2で、invalid.domainは無効なホストです。Bがクラスターに参加しようとすると、Subject Alternative Name（SAN）フィールドに一致する有効なIPアドレスがあるため、ピアAはBの認証に成功します。詳しくは、issue#8206
を参照してください。
v3.2.5
以降、サーバーはワイルドカードDNSのSANの逆引きに対応します
。たとえば、ピア証明書のSubject Alternative Name（SAN）フィールドにDNS名だけがあり、IPアドレスがない場合、サーバーはまずリモートIPアドレスを逆引きし、そのアドレスに対応する名前の一覧を取得します（nslookup IPADDRなど）。その名前のいずれかがピア証明書のDNS名と完全一致またはワイルドカード一致すれば、接続を受け入れます。一致しなければ、ピア証明書の各DNSエントリーを正引きし（エントリーが*.example.default.svcならexample.default.svcを検索）、名前解決したアドレスがピアのリモートIPアドレスと一致するときだけ受け入れます。たとえば、ピアBのCSR（cfsslを使用）が次のとおりだとします。
ピアBのリモートIPアドレスは10.138.0.2です。Bがクラスターに参加しようとすると、ピアAはIPアドレス10.138.0.2を逆引きしてホスト名の一覧を取得し、Bの証明書のSubject Alternative Name（SAN）フィールドのDNS名と、完全一致またはワイルドカード一致するかを確認します。逆引きでも正引きでも一致しなければ、"tls: "10.138.0.2" does not match any of DNSNames ["*.example.default.svc","*.example.default.svc.cluster.local"]というエラーを返します。詳しくは、issue#8268
を参照してください。
$ etcd --peer-cert-allowed-cn m1.etcd.local
I | embed: rejected connection from "127.0.0.1:48044"(error "CommonName authentication failed", ServerName "m1.etcd.local")I | embed: rejected connection from "127.0.0.1:55702"(error "remote error: tls: bad certificate", ServerName "m3.etcd.local")
各プロセスを次のように起動します。
etcd --peer-cert-allowed-cn etcd.local
I | pkg/netutil: resolving m3.etcd.local:32380 to 127.0.0.1:32380
I | pkg/netutil: resolving m2.etcd.local:22380 to 127.0.0.1:22380
I | pkg/netutil: resolving m1.etcd.local:2380 to 127.0.0.1:2380
I | etcdserver: published {Name:m3 ClientURLs:[https://m3.etcd.local:32379]} to cluster 9db03f09b20de32b
I | embed: ready to serve client requests
I | etcdserver: published {Name:m1 ClientURLs:[https://m1.etcd.local:2379]} to cluster 9db03f09b20de32b
I | embed: ready to serve client requests
I | etcdserver: published {Name:m2 ClientURLs:[https://m2.etcd.local:22379]} to cluster 9db03f09b20de32b
I | embed: ready to serve client requests
I | embed: serving client requests on 127.0.0.1:32379
I | embed: serving client requests on 127.0.0.1:22379
I | embed: serving client requests on 127.0.0.1:2379
$ dig +noall +answer SRV _etcd-server._tcp.example.com
_etcd-server._tcp.example.com. 300 IN SRV 0 0 2380 infra0.example.com.
_etcd-server._tcp.example.com. 300 IN SRV 0 0 2380 infra1.example.com.
_etcd-server._tcp.example.com. 300 IN SRV 0 0 2380 infra2.example.com.
$ dig +noall +answer SRV _etcd-client._tcp.example.com
_etcd-client._tcp.example.com. 300 IN SRV 0 0 2379 infra0.example.com.
_etcd-client._tcp.example.com. 300 IN SRV 0 0 2379 infra1.example.com.
_etcd-client._tcp.example.com. 300 IN SRV 0 0 2379 infra2.example.com.
$ dig +noall +answer infra0.example.com infra1.example.com infra2.example.com
infra0.example.com. 300 IN A 10.0.1.10
infra1.example.com. 300 IN A 10.0.1.11
infra2.example.com. 300 IN A 10.0.1.12
# file: etcd.yaml---apiVersion:v1kind:Servicemetadata:name:etcdnamespace:defaultspec:type:ClusterIPclusterIP:Noneselector:app:etcd#### Ideally we would use SRV records to do peer discovery for initialization.## Unfortunately discovery will not work without logic to wait for these to## populate in the container. This problem is relatively easy to overcome by## making changes to prevent the etcd process from starting until the records## have populated. The documentation on statefulsets briefly talk about it.## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-idpublishNotReadyAddresses:true#### The naming scheme of the client and server ports match the scheme that etcd## uses when doing discovery with SRV records.ports:- name:etcd-clientport:2379- name:etcd-serverport:2380- name:etcd-metricsport:8080---apiVersion:apps/v1kind:StatefulSetmetadata:namespace:defaultname:etcdspec:#### The service name is being set to leverage the service headlessly.## https://kubernetes.io/docs/concepts/services-networking/service/#headless-servicesserviceName:etcd#### If you are increasing the replica count of an existing cluster, you should## also update the --initial-cluster-state flag as noted further down in the## container configuration.replicas:3#### For initialization, the etcd pods must be available to eachother before## they are "ready" for traffic. The "Parallel" policy makes this possible.podManagementPolicy:Parallel#### To ensure availability of the etcd cluster, the rolling update strategy## is used. For availability, there must be at least 51% of the etcd nodes## online at any given time.updateStrategy:type:RollingUpdate#### This is label query over pods that should match the replica count.## It must match the pod template's labels. For more information, see the## following documentation:## https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectorsselector:matchLabels:app:etcd#### Pod configuration template.template:metadata:#### The labeling here is tied to the "matchLabels" of this StatefulSet and## "affinity" configuration of the pod that will be created.#### This example's labeling scheme is fine for one etcd cluster per## namespace, but should you desire multiple clusters per namespace, you## will need to update the labeling schema to be unique per etcd cluster.labels:app:etcdannotations:#### This gets referenced in the etcd container's configuration as part of## the DNS name. It must match the service name created for the etcd## cluster. The choice to place it in an annotation instead of the env## settings is because there should only be 1 service per etcd cluster.serviceName:etcdspec:#### Configuring the node affinity is necessary to prevent etcd servers from## ending up on the same hardware together.#### See the scheduling documentation for more information about this:## https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinityaffinity:## The podAntiAffinity is a set of rules for scheduling that describe## when NOT to place a pod from this StatefulSet on a node.podAntiAffinity:#### When preparing to place the pod on a node, the scheduler will check## for other pods matching the rules described by the labelSelector## separated by the chosen topology key.requiredDuringSchedulingIgnoredDuringExecution:## This label selector is looking for app=etcd- labelSelector:matchExpressions:- key:appoperator:Invalues:- etcd## This topology key denotes a common label used on nodes in the## cluster. The podAntiAffinity configuration essentially states## that if another pod has a label of app=etcd on the node, the## scheduler should not place another pod on the node.## https://kubernetes.io/docs/reference/labels-annotations-taints/#kubernetesiohostnametopologyKey:"kubernetes.io/hostname"#### Containers in the podcontainers:## This example only has this etcd container.- name:etcdimage:quay.io/coreos/etcd:v3.7.2imagePullPolicy:IfNotPresentports:- name:etcd-clientcontainerPort:2379- name:etcd-servercontainerPort:2380- name:etcd-metricscontainerPort:8080#### These probes will fail over TLS for self-signed certificates, so etcd## is configured to deliver metrics over port 8080 further down.#### As mentioned in the "Monitoring etcd" page, /readyz and /livez were## added in v3.5.12. Prior to this, monitoring required extra tooling## inside the container to make these probes work.#### The values in this readiness probe should be further validated, it## is only an example configuration.readinessProbe:httpGet:path:/readyzport:8080initialDelaySeconds:10periodSeconds:5timeoutSeconds:5successThreshold:1failureThreshold:30## The values in this liveness probe should be further validated, it## is only an example configuration.livenessProbe:httpGet:path:/livezport:8080initialDelaySeconds:15periodSeconds:10timeoutSeconds:5failureThreshold:3env:#### Environment variables defined here can be used by other parts of the## container configuration. They are interpreted by Kubernetes, instead## of in the container environment.#### These env vars pass along information about the pod.- name:K8S_NAMESPACEvalueFrom:fieldRef:fieldPath:metadata.namespace- name:HOSTNAMEvalueFrom:fieldRef:fieldPath:metadata.name- name:SERVICE_NAMEvalueFrom:fieldRef:fieldPath:metadata.annotations['serviceName']#### Configuring etcdctl inside the container to connect to the etcd node## in the container reduces confusion when debugging.- name:ETCDCTL_ENDPOINTSvalue:$(HOSTNAME).$(SERVICE_NAME):2379#### TLS client configuration for etcdctl in the container.## These files paths are part of the "etcd-client-certs" volume mount.# - name: ETCDCTL_KEY# value: /etc/etcd/certs/client/tls.key# - name: ETCDCTL_CERT# value: /etc/etcd/certs/client/tls.crt# - name: ETCDCTL_CACERT# value: /etc/etcd/certs/client/ca.crt#### Use this URI_SCHEME value for non-TLS clusters.- name:URI_SCHEMEvalue:"http"## TLS: Use this URI_SCHEME for TLS clusters.# - name: URI_SCHEME# value: "https"#### If you're using a different container, the executable may be in a## different location. This example uses the full path to help remove## ambiguity to you, the reader.## Often you can just use "etcd" instead of "/usr/local/bin/etcd" and it## will work because the $PATH includes a directory containing "etcd".command:- /usr/local/bin/etcd#### Arguments used with the etcd command inside the container.args:#### Configure the name of the etcd server.- --name=$(HOSTNAME)#### Configure etcd to use the persistent storage configured below.- --data-dir=/data#### In this example we're consolidating the WAL into sharing space with## the data directory. This is not ideal in production environments and## should be placed in it's own volume.- --wal-dir=/data/wal#### URL configurations are parameterized here and you shouldn't need to## do anything with these.- --listen-peer-urls=$(URI_SCHEME)://0.0.0.0:2380- --listen-client-urls=$(URI_SCHEME)://0.0.0.0:2379- --advertise-client-urls=$(URI_SCHEME)://$(HOSTNAME).$(SERVICE_NAME):2379#### This must be set to "new" for initial cluster bootstrapping. To scale## the cluster up, this should be changed to "existing" when the replica## count is increased. If set incorrectly, etcd makes an attempt to## start but fail safely.- --initial-cluster-state=new#### Token used for cluster initialization. The recommendation for this is## to use a unique token for every cluster. This example parameterized## to be unique to the namespace, but if you are deploying multiple etcd## clusters in the same namespace, you should do something extra to## ensure uniqueness amongst clusters.- --initial-cluster-token=etcd-$(K8S_NAMESPACE)#### The initial cluster flag needs to be updated to match the number of## replicas configured. When combined, these are a little hard to read.## Here is what a single parameterized peer looks like:## etcd-0=$(URI_SCHEME)://etcd-0.$(SERVICE_NAME):2380- --initial-cluster=etcd-0=$(URI_SCHEME)://etcd-0.$(SERVICE_NAME):2380,etcd-1=$(URI_SCHEME)://etcd-1.$(SERVICE_NAME):2380,etcd-2=$(URI_SCHEME)://etcd-2.$(SERVICE_NAME):2380#### The peer urls flag should be fine as-is.- --initial-advertise-peer-urls=$(URI_SCHEME)://$(HOSTNAME).$(SERVICE_NAME):2380#### This avoids probe failure if you opt to configure TLS.- --listen-metrics-urls=http://0.0.0.0:8080#### These are some configurations you may want to consider enabling, but## should look into further to identify what settings are best for you.# - --auto-compaction-mode=periodic# - --auto-compaction-retention=10m#### TLS client configuration for etcd, reusing the etcdctl env vars.# - --client-cert-auth# - --trusted-ca-file=$(ETCDCTL_CACERT)# - --cert-file=$(ETCDCTL_CERT)# - --key-file=$(ETCDCTL_KEY)#### TLS server configuration for etcdctl in the container.## These files paths are part of the "etcd-server-certs" volume mount.# - --peer-client-cert-auth# - --peer-trusted-ca-file=/etc/etcd/certs/server/ca.crt# - --peer-cert-file=/etc/etcd/certs/server/tls.crt# - --peer-key-file=/etc/etcd/certs/server/tls.key#### This is the mount configuration.volumeMounts:- name:etcd-datamountPath:/data#### TLS client configuration for etcdctl# - name: etcd-client-tls# mountPath: "/etc/etcd/certs/client"# readOnly: true#### TLS server configuration# - name: etcd-server-tls# mountPath: "/etc/etcd/certs/server"# readOnly: truevolumes:#### TLS client configuration# - name: etcd-client-tls# secret:# secretName: etcd-client-tls# optional: false#### TLS server configuration# - name: etcd-server-tls# secret:# secretName: etcd-server-tls# optional: false#### This StatefulSet will uses the volumeClaimTemplate field to create a PVC in## the cluster for each replica. These PVCs can not be easily resized later.volumeClaimTemplates:- metadata:name:etcd-dataspec:accessModes:["ReadWriteOnce"]#### In some clusters, it is necessary to explicitly set the storage class.## This example will end up using the default storage class.# storageClassName: ""resources:requests:storage:1Gi
$ etcd gateway start --endpoints=infra0.example.com:2379,infra1.example.com:2379,infra2.example.com:2379
2016-08-16 11:21:18.867350 I | tcpproxy: ready to proxy client requests to [...]
DNSでサービスディスカバリーを行う場合は、次のDNS SRVエントリーを考えます。
$ dig +noall +answer SRV _etcd-client._tcp.example.com
_etcd-client._tcp.example.com. 300 IN SRV 002379 infra0.example.com.
_etcd-client._tcp.example.com. 300 IN SRV 002379 infra1.example.com.
_etcd-client._tcp.example.com. 300 IN SRV 002379 infra2.example.com.
$ dig +noall +answer infra0.example.com infra1.example.com infra2.example.com
infra0.example.com. 300 IN A 10.0.1.10
infra1.example.com. 300 IN A 10.0.1.11
infra2.example.com. 300 IN A 10.0.1.12
ETCDCTL_API=3 etcdctl --endpoints=http://localhost:23790 member list --write-out table
+----+---------+--------------------------------+------------+-----------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |+----+---------+--------------------------------+------------+-----------------+
|0| started | Gyu-Hos-MBP.sfo.coreos.systems || 127.0.0.1:23791 ||0| started | Gyu-Hos-MBP.sfo.coreos.systems || 127.0.0.1:23790 |+----+---------+--------------------------------+------------+-----------------+
ETCDCTL_API=3 etcdctl --endpoints=http://localhost:23792 member list --write-out table
+----+---------+--------------------------------+------------+-----------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |+----+---------+--------------------------------+------------+-----------------+
|0| started | Gyu-Hos-MBP.sfo.coreos.systems || 127.0.0.1:23792 |+----+---------+--------------------------------+------------+-----------------+
$ curl -L http://localhost:2379/metrics | grep -v debugging # ignore unstable debugging metrics# HELP etcd_disk_backend_commit_duration_seconds The latency distributions of commit called by backend.# TYPE etcd_disk_backend_commit_duration_seconds histogrametcd_disk_backend_commit_duration_seconds_bucket{le="0.002"}72756etcd_disk_backend_commit_duration_seconds_bucket{le="0.004"}401587etcd_disk_backend_commit_duration_seconds_bucket{le="0.008"}405979etcd_disk_backend_commit_duration_seconds_bucket{le="0.016"}406464...
$ etcdctl member add infra3 --peer-urls=http://10.0.1.13:2380
added member 9bf1b35fc7761a23 to cluster
ETCD_NAME="infra3"ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380,infra3=http://10.0.1.13:2380"ETCD_INITIAL_CLUSTER_STATE=existing
複数のメンバーを追加する場合は、1つずつ構成し、正しく起動することを確認してから次を追加するのがベストプラクティスです。1ノードのクラスターに新しいメンバーを追加すると、コンセンサスに必要な過半数が2メンバーになるため、新しいメンバーが起動するまでクラスターは処理を進められません。この動作は、etcdctl member addが新しいメンバーをクラスターに通知してから、新しいメンバーが既存メンバーへの接続を確立するまでの間だけ発生します。
etcdctl member addに--learnerフラグを指定し、新しいメンバーを学習者メンバーとしてクラスターに追加します。
$ etcdctl member add infra3 --peer-urls=http://10.0.1.13:2380 --learner
Member 9bf1b35fc7761a23 added to cluster a7ef944b95711739
ETCD_NAME="infra3"ETCD_INITIAL_CLUSTER="infra0=http://10.0.1.10:2380,infra1=http://10.0.1.11:2380,infra2=http://10.0.1.12:2380,infra3=http://10.0.1.13:2380"ETCD_INITIAL_CLUSTER_STATE=existing
新しく追加した学習者メンバーのetcdプロセスを起動した後、etcdctl member promoteで投票メンバーに昇格させます。
$ etcdctl member promote 9bf1b35fc7761a23
Member 9e29bbaa45d74461 promoted in cluster a7ef944b95711739
