Created
October 27, 2018 22:51
-
-
Save anubsinha/bb5266360b4dd7f406251d2a71b5cf09 to your computer and use it in GitHub Desktop.
RMQ DNS issue
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2018-10-27 22:04:19.363 [info] <0.33.0> Application lager started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:19.791 [info] <0.33.0> Application crypto started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:19.792 [info] <0.33.0> Application xmerl started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:19.943 [info] <0.33.0> Application mnesia started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:19.943 [info] <0.33.0> Application cowlib started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:19.944 [info] <0.33.0> Application amqp10_common started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:19.944 [info] <0.33.0> Application recon started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.027 [info] <0.33.0> Application inets started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.032 [info] <0.33.0> Application os_mon started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.033 [info] <0.33.0> Application jsx started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.033 [info] <0.33.0> Application asn1 started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.033 [info] <0.33.0> Application public_key started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.118 [info] <0.33.0> Application ssl started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.130 [info] <0.33.0> Application ranch started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.132 [info] <0.33.0> Application cowboy started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.132 [info] <0.33.0> Application ranch_proxy_protocol started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.135 [info] <0.33.0> Application amqp10_client started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.135 [info] <0.33.0> Application rabbit_common started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.144 [info] <0.203.0> | |
| Starting RabbitMQ 3.7.8 on Erlang 20.3.4 | |
| Copyright (C) 2007-2018 Pivotal Software, Inc. | |
| Licensed under the MPL. See http://www.rabbitmq.com/ | |
| ## ## | |
| ## ## RabbitMQ 3.7.8. Copyright (C) 2007-2018 Pivotal Software, Inc. | |
| ########## Licensed under the MPL. See http://www.rabbitmq.com/ | |
| ###### ## | |
| ########## Logs: <stdout> | |
| Starting broker... | |
| 2018-10-27 22:04:20.170 [info] <0.203.0> | |
| node : rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local | |
| home dir : /var/lib/rabbitmq | |
| config file(s) : /etc/rabbitmq/rabbitmq.conf | |
| cookie hash : hPCw+wbpEtazXkCDbGUgLg== | |
| log(s) : <stdout> | |
| database dir : /var/lib/rabbitmq/mnesia/rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local | |
| 2018-10-27 22:04:22.919 [info] <0.211.0> Memory high watermark set to 244 MiB (256000000 bytes) of 985 MiB (1033609216 bytes) total | |
| 2018-10-27 22:04:22.922 [info] <0.213.0> Enabling free disk space monitoring | |
| 2018-10-27 22:04:22.922 [info] <0.213.0> Disk free limit set to 50MB | |
| 2018-10-27 22:04:22.925 [info] <0.216.0> Limiting to approx 65436 file handles (58890 sockets) | |
| 2018-10-27 22:04:22.925 [info] <0.217.0> FHC read buffering: OFF | |
| 2018-10-27 22:04:22.925 [info] <0.217.0> FHC write buffering: ON | |
| 2018-10-27 22:04:22.926 [info] <0.203.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local is empty. Assuming we need to join an existing cluster or initialise from scratch... | |
| 2018-10-27 22:04:22.926 [info] <0.203.0> Configured peer discovery backend: rabbit_peer_discovery_k8s | |
| 2018-10-27 22:04:22.926 [info] <0.203.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s | |
| 2018-10-27 22:04:22.926 [info] <0.203.0> Peer discovery backend does not support locking, falling back to randomized delay | |
| 2018-10-27 22:04:22.927 [info] <0.203.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping randomized startup delay. | |
| 2018-10-27 22:04:22.930 [info] <0.203.0> Failed to get nodes from k8s - {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}}, | |
| {inet,[inet],nxdomain}]} | |
| 2018-10-27 22:04:22.930 [error] <0.202.0> CRASH REPORT Process <0.202.0> with 0 neighbours exited with reason: no case clause matching {error,"{failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\n {inet,[inet],nxdomain}]}"} in rabbit_mnesia:init_from_config/0 line 164 in application_master:init/4 line 134 | |
| 2018-10-27 22:04:22.931 [info] <0.33.0> Application rabbit exited with reason: no case clause matching {error,"{failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\n {inet,[inet],nxdomain}]}"} in rabbit_mnesia:init_from_config/0 line 164 | |
| Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,"{failed_connect,[{to_address,{\"kubernetes.defau | |
| {"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,\"{failed_connect,[{to_address,{\\"kubernetes.default.svc.cluster.local\\",443}},\n {inet,[inet],nxdomain}]}\"}},[{rabbit_mnesia,init_from_config,0,[{file,\"src/rabbit_mnesia.erl\"},{line,164}]},{rabbit_mnesia,init_with_lock,3,[{file,\"src/rabbit_mnesia.erl\"},{line,144}]},{rabbit_mnesia,init,0,[{file,\"src/rabbit_mnesia.erl\"},{line,111}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,run_step,2,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit_boot_steps,run_boot_steps,1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit,start,2,[{file,\"src/rabbit.erl\"},{line,816}]}]}}}}}"} | |
| Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done | |
| [root@ip-10-200-1-107 ~]# docker logs 2c7e5a8bbb89 | |
| 2018-10-27 22:04:19.363 [info] <0.33.0> Application lager started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:19.791 [info] <0.33.0> Application crypto started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:19.792 [info] <0.33.0> Application xmerl started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:19.943 [info] <0.33.0> Application mnesia started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:19.943 [info] <0.33.0> Application cowlib started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:19.944 [info] <0.33.0> Application amqp10_common started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:19.944 [info] <0.33.0> Application recon started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.027 [info] <0.33.0> Application inets started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.032 [info] <0.33.0> Application os_mon started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.033 [info] <0.33.0> Application jsx started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.033 [info] <0.33.0> Application asn1 started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.033 [info] <0.33.0> Application public_key started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.118 [info] <0.33.0> Application ssl started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.130 [info] <0.33.0> Application ranch started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.132 [info] <0.33.0> Application cowboy started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.132 [info] <0.33.0> Application ranch_proxy_protocol started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.135 [info] <0.33.0> Application amqp10_client started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.135 [info] <0.33.0> Application rabbit_common started on node 'rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local' | |
| 2018-10-27 22:04:20.144 [info] <0.203.0> | |
| Starting RabbitMQ 3.7.8 on Erlang 20.3.4 | |
| Copyright (C) 2007-2018 Pivotal Software, Inc. | |
| Licensed under the MPL. See http://www.rabbitmq.com/ | |
| ## ## | |
| ## ## RabbitMQ 3.7.8. Copyright (C) 2007-2018 Pivotal Software, Inc. | |
| ########## Licensed under the MPL. See http://www.rabbitmq.com/ | |
| ###### ## | |
| ########## Logs: <stdout> | |
| Starting broker... | |
| 2018-10-27 22:04:20.170 [info] <0.203.0> | |
| node : rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local | |
| home dir : /var/lib/rabbitmq | |
| config file(s) : /etc/rabbitmq/rabbitmq.conf | |
| cookie hash : hPCw+wbpEtazXkCDbGUgLg== | |
| log(s) : <stdout> | |
| database dir : /var/lib/rabbitmq/mnesia/rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local | |
| 2018-10-27 22:04:22.919 [info] <0.211.0> Memory high watermark set to 244 MiB (256000000 bytes) of 985 MiB (1033609216 bytes) total | |
| 2018-10-27 22:04:22.922 [info] <0.213.0> Enabling free disk space monitoring | |
| 2018-10-27 22:04:22.922 [info] <0.213.0> Disk free limit set to 50MB | |
| 2018-10-27 22:04:22.925 [info] <0.216.0> Limiting to approx 65436 file handles (58890 sockets) | |
| 2018-10-27 22:04:22.925 [info] <0.217.0> FHC read buffering: OFF | |
| 2018-10-27 22:04:22.925 [info] <0.217.0> FHC write buffering: ON | |
| 2018-10-27 22:04:22.926 [info] <0.203.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@rabbit-rabbitmq-ha-0.rabbit-rabbitmq-ha-discovery.default.svc.cluster.local is empty. Assuming we need to join an existing cluster or initialise from scratch... | |
| 2018-10-27 22:04:22.926 [info] <0.203.0> Configured peer discovery backend: rabbit_peer_discovery_k8s | |
| 2018-10-27 22:04:22.926 [info] <0.203.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s | |
| 2018-10-27 22:04:22.926 [info] <0.203.0> Peer discovery backend does not support locking, falling back to randomized delay | |
| 2018-10-27 22:04:22.927 [info] <0.203.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping randomized startup delay. | |
| 2018-10-27 22:04:22.930 [info] <0.203.0> Failed to get nodes from k8s - {failed_connect,[{to_address,{"kubernetes.default.svc.cluster.local",443}}, | |
| {inet,[inet],nxdomain}]} | |
| 2018-10-27 22:04:22.930 [error] <0.202.0> CRASH REPORT Process <0.202.0> with 0 neighbours exited with reason: no case clause matching {error,"{failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\n {inet,[inet],nxdomain}]}"} in rabbit_mnesia:init_from_config/0 line 164 in application_master:init/4 line 134 | |
| 2018-10-27 22:04:22.931 [info] <0.33.0> Application rabbit exited with reason: no case clause matching {error,"{failed_connect,[{to_address,{\"kubernetes.default.svc.cluster.local\",443}},\n {inet,[inet],nxdomain}]}"} in rabbit_mnesia:init_from_config/0 line 164 | |
| Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,"{failed_connect,[{to_address,{\"kubernetes.defau | |
| {"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,\"{failed_connect,[{to_address,{\\"kubernetes.default.svc.cluster.local\\",443}},\n {inet,[inet],nxdomain}]}\"}},[{rabbit_mnesia,init_from_config,0,[{file,\"src/rabbit_mnesia.erl\"},{line,164}]},{rabbit_mnesia,init_with_lock,3,[{file,\"src/rabbit_mnesia.erl\"},{line,144}]},{rabbit_mnesia,init,0,[{file,\"src/rabbit_mnesia.erl\"},{line,111}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,run_step,2,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit_boot_steps,run_boot_steps,1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit,start,2,[{file,\"src/rabbit.erl\"},{line,816}]}]}}}}}"} | |
| Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment