linux – Rabbitmq instances are crashing

I am using docker-compose.yml file to spin 3 instances of RabbitMQ in a single host. Running docker on mac, Installed haproxy on mac. When I run docker-compose up, I see erlang cookies are matching for the instances in the cluster but not sure why instances are crashing and not able to join in cluster. Let me know if you need any other information.

version: '3'

services:

  rabbitmq1:
    image: rabbitmq:3.7.28-management
    hostname: rabbitmq1
    environment:
      - RABBITMQ_ERLANG_COOKIE=12345
      - RABBITMQ_DEFAULT_USER=guest
      - RABBITMQ_DEFAULT_PASS=guest
      - RABBITMQ_DEFAULT_VHOST=/

  rabbitmq2:
    image: rabbitmq:3.7.28-management
    hostname: rabbitmq2
    depends_on:
      - rabbitmq1
    environment:
      - RABBITMQ_ERLANG_COOKIE=12345
      - RABBITMQ_DEFAULT_USER=guest
      - RABBITMQ_DEFAULT_PASS=guest
      - RABBITMQ_DEFAULT_VHOST=/
    volumes:
      - ./cluster-entrypoint.sh:/usr/local/bin/cluster-entrypoint.sh
    entrypoint: /usr/local/bin/cluster-entrypoint.sh

  rabbitmq3:
    image: rabbitmq:3.7.28-management
    hostname: rabbitmq3
    depends_on:
      - rabbitmq1
    environment:
      - RABBITMQ_ERLANG_COOKIE=12345
      - RABBITMQ_DEFAULT_USER=guest
      - RABBITMQ_DEFAULT_PASS=guest
      - RABBITMQ_DEFAULT_VHOST=/
    volumes:
      - ./cluster-entrypoint.sh:/usr/local/bin/cluster-entrypoint.sh
    entrypoint: /usr/local/bin/cluster-entrypoint.sh
    
  haproxy:
    image: haproxy:1.7
    volumes:
      - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
    depends_on:
      - rabbitmq1
      - rabbitmq2
      - rabbitmq3
    ports:
      - 15672:15672
      - 5672:5672

Below is my cluster-entrypoint.sh file

#!/bin/bash

set -e

# Start RMQ from entry point.
# This will ensure that environment variables passed
# will be honored
/usr/local/bin/docker-entrypoint.sh rabbitmq-server -detached

# Do the cluster dance
rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@rabbitmq1

# Stop the entire RMQ server. This is done so that we
# can attach to it again, but without the -detached flag
# making it run in the forground
rabbitmqctl stop

# Wait a while for the app to really stop
sleep 2s

# Start it
rabbitmq-server

I am seeing below error:

haproxy_1    | <7>haproxy-systemd-wrapper: executing /usr/local/sbin/haproxy -p /run/haproxy.pid -db -f /usr/local/etc/haproxy/haproxy.cfg -Ds 
rabbitmq1_1  | 
rabbitmq1_1  | warning: /var/lib/rabbitmq/.erlang.cookie contents do not match RABBITMQ_ERLANG_COOKIE
rabbitmq1_1  | 
rabbitmq3_1  | 
rabbitmq3_1  | warning: /var/lib/rabbitmq/.erlang.cookie contents do not match RABBITMQ_ERLANG_COOKIE
rabbitmq3_1  | 
rabbitmq2_1  | 
rabbitmq2_1  | warning: /var/lib/rabbitmq/.erlang.cookie contents do not match RABBITMQ_ERLANG_COOKIE
rabbitmq2_1  | 
rabbitmq3_1  | Warning: PID file not written; -detached was passed.
rabbitmq2_1  | Warning: PID file not written; -detached was passed.
rabbitmq3_1  | Stopping rabbit application on node rabbit@rabbitmq3 ...
rabbitmq3_1  | Error: unable to perform an operation on node 'rabbit@rabbitmq3'. Please see diagnostics information and suggestions below.
rabbitmq3_1  | 
rabbitmq3_1  | Most common reasons for this are:
rabbitmq3_1  | 
rabbitmq3_1  |  * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
rabbitmq3_1  |  * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
rabbitmq3_1  |  * Target node is not running
rabbitmq3_1  | 
rabbitmq3_1  | In addition to the diagnostics info below:
rabbitmq3_1  | 
rabbitmq3_1  |  * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
rabbitmq3_1  |  * Consult server logs on node rabbit@rabbitmq3
rabbitmq3_1  |  * If target node is configured to use long node names, don't forget to use --longnames with CLI tools
rabbitmq3_1  | 
rabbitmq3_1  | DIAGNOSTICS
rabbitmq3_1  | ===========
rabbitmq3_1  | 
rabbitmq3_1  | attempted to contact: [rabbit@rabbitmq3]
rabbitmq3_1  | 
rabbitmq3_1  | rabbit@rabbitmq3:
rabbitmq3_1  |   * connected to epmd (port 4369) on rabbitmq3
rabbitmq3_1  |   * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic 
rabbitmq3_1  |   * TCP connection succeeded but Erlang distribution failed 
rabbitmq3_1  |   * suggestion: check if the Erlang cookie identical for all server nodes and CLI tools
rabbitmq3_1  |   * suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other
rabbitmq3_1  |   * suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
rabbitmq3_1  |    * suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
rabbitmq3_1  | 
rabbitmq3_1  | 
rabbitmq3_1  | Current node details:
rabbitmq3_1  |  * node name: 'rabbitmqcli-409-rabbit@rabbitmq3'
rabbitmq3_1  |  * effective user's home directory: /var/lib/rabbitmq
rabbitmq3_1  |  * Erlang cookie hash: gnzLDuqKcGxMNKFokfhOew==
rabbitmq3_1  | 
docker-rabbitmq-cluster_rabbitmq3_1 exited with code 69
rabbitmq2_1  | Stopping rabbit application on node rabbit@rabbitmq2 ...
rabbitmq2_1  | Error: unable to perform an operation on node 'rabbit@rabbitmq2'. Please see diagnostics information and suggestions below.
rabbitmq2_1  | 
rabbitmq2_1  | Most common reasons for this are:
rabbitmq2_1  | 
rabbitmq2_1  |  * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
rabbitmq2_1  |  * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
rabbitmq2_1  |  * Target node is not running
rabbitmq2_1  | 
rabbitmq2_1  | In addition to the diagnostics info below:
rabbitmq2_1  | 
rabbitmq2_1  |  * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
rabbitmq2_1  |  * Consult server logs on node rabbit@rabbitmq2
rabbitmq2_1  |  * If target node is configured to use long node names, don't forget to use --longnames with CLI tools
rabbitmq2_1  | 
rabbitmq2_1  | DIAGNOSTICS
rabbitmq2_1  | ===========
rabbitmq2_1  | 
rabbitmq2_1  | attempted to contact: [rabbit@rabbitmq2]
rabbitmq2_1  | 
rabbitmq2_1  | rabbit@rabbitmq2:
rabbitmq2_1  |   * connected to epmd (port 4369) on rabbitmq2
rabbitmq2_1  |   * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic 
rabbitmq2_1  |   * TCP connection succeeded but Erlang distribution failed 
rabbitmq2_1  |   * suggestion: check if the Erlang cookie identical for all server nodes and CLI tools
rabbitmq2_1  |   * suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other
rabbitmq2_1  |   * suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
rabbitmq2_1  |    * suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
rabbitmq2_1  | 
rabbitmq2_1  | 
rabbitmq2_1  | Current node details:
rabbitmq2_1  |  * node name: 'rabbitmqcli-409-rabbit@rabbitmq2'
rabbitmq2_1  |  * effective user's home directory: /var/lib/rabbitmq
rabbitmq2_1  |  * Erlang cookie hash: gnzLDuqKcGxMNKFokfhOew==
rabbitmq2_1  | 
docker-rabbitmq-cluster_rabbitmq2_1 exited with code 69
rabbitmq1_1  | 2022-07-05 02:18:35.113 [info] <0.9.0> Feature flags: list of feature flags found:
rabbitmq1_1  | 2022-07-05 02:18:35.113 [info] <0.9.0> Feature flags: feature flag states written to disk: yes
rabbitmq1_1  | 2022-07-05 02:18:35.162 [info] <0.267.0> 
rabbitmq1_1  |  Starting RabbitMQ 3.7.28 on Erlang 22.3.4.7
rabbitmq1_1  |  Copyright (c) 2007-2020 Pivotal Software, Inc.
rabbitmq1_1  |  Licensed under the MPL.  See https://www.rabbitmq.com/
rabbitmq1_1  | 
rabbitmq1_1  |   ##  ##
rabbitmq1_1  |   ##  ##      RabbitMQ 3.7.28. Copyright (c) 2007-2020 Pivotal Software, Inc.
rabbitmq1_1  |   ##########  Licensed under the MPL.  See https://www.rabbitmq.com/
rabbitmq1_1  |   ######  ##
rabbitmq1_1  |   ##########  Logs: <stdout>
rabbitmq1_1  | 
rabbitmq1_1  |               Starting broker...
rabbitmq1_1  | 2022-07-05 02:18:35.163 [info] <0.267.0> 
rabbitmq1_1  |  node           : rabbit@rabbitmq1
rabbitmq1_1  |  home dir       : /var/lib/rabbitmq
rabbitmq1_1  |  config file(s) : /etc/rabbitmq/rabbitmq.conf
rabbitmq1_1  |  cookie hash    : VlfoFK5J8f9Ln3G9sXDoPQ==
rabbitmq1_1  |  log(s)         : <stdout>
rabbitmq1_1  |  database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq1
rabbitmq1_1  | 2022-07-05 02:18:35.170 [info] <0.267.0> Running boot step pre_boot defined by app rabbit
rabbitmq1_1  | 2022-07-05 02:18:35.170 [info] <0.267.0> Running boot step rabbit_core_metrics defined by app rabbit
rabbitmq1_1  | 2022-07-05 02:18:35.170 [info] <0.267.0> Running boot step rabbit_alarm defined by app rabbit
rabbitmq1_1  | 2022-07-05 02:18:35.173 [info] <0.298.0> Memory high watermark set to 3140 MiB (3293098803 bytes) of 7851 MiB (8232747008 bytes) total
rabbitmq1_1  | 2022-07-05 02:18:35.176 [info] <0.315.0> Enabling free disk space monitoring
rabbitmq1_1  | 2022-07-05 02:18:35.176 [info] <0.315.0> Disk free limit set to 50MB
rabbitmq1_1  | 2022-07-05 02:18:35.179 [info] <0.267.0> Running boot step code_server_cache defined by app rabbit
rabbitmq1_1  | 2022-07-05 02:18:35.179 [info] <0.267.0> Running boot step file_handle_cache defined by app rabbit
rabbitmq1_1  | 2022-07-05 02:18:35.180 [info] <0.319.0> FHC read buffering:  OFF
rabbitmq1_1  | 2022-07-05 02:18:35.180 [info] <0.319.0> FHC write buffering: ON
rabbitmq1_1  | 2022-07-05 02:18:35.180 [info] <0.318.0> Limiting to approx 1048476 file handles (943626 sockets)
rabbitmq1_1  | 2022-07-05 02:18:35.181 [info] <0.267.0> Running boot step worker_pool defined by app rabbit
rabbitmq1_1  | 2022-07-05 02:18:35.181 [info] <0.276.0> Will use 5 processes for default worker pool
rabbitmq1_1  | 2022-07-05 02:18:35.181 [info] <0.276.0> Starting worker pool 'worker_pool' with 5 processes in it
rabbitmq1_1  | 2022-07-05 02:18:35.181 [info] <0.267.0> Running boot step database defined by app rabbit
rabbitmq1_1  | 2022-07-05 02:18:35.183 [warning] <0.267.0> Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `/var/lib/rabbitmq/mnesia/rabbit@rabbitmq1-feature_flags`:
rabbitmq1_1  | 2022-07-05 02:18:35.183 [warning] <0.267.0> Feature flags:   - list of previously disabled feature flags now marked as such: [drop_unroutable_metric,empty_basic_get_metric,implicit_default_bindings,maintenance_mode_status,quorum_queue,stream_queue,user_limits,virtual_host_metadata]
rabbitmq1_1  | 2022-07-05 02:18:35.192 [error] <0.266.0> CRASH REPORT Process <0.266.0> with 0 neighbours exited with reason: {{schema_integrity_check_failed,[{table_attributes_mismatch,rabbit_user,[username,password_hash,tags,hashing_algorithm],[username,password_hash,tags,hashing_algorithm,limits]},{table_attributes_mismatch,rabbit_vhost,[virtual_host,limits],[virtual_host,limits,metadata]},{table_attributes_mismatch,rabbit_durable_queue,[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,operator_policy,gm_pids,decorators,state,policy_version,slave_pids_pending_shutdown,...],...},...]},...} in application_master:init/4 line 138
rabbitmq1_1  | 2022-07-05 02:18:35.192 [info] <0.44.0> Application rabbit exited with reason: {{schema_integrity_check_failed,[{table_attributes_mismatch,rabbit_user,[username,password_hash,tags,hashing_algorithm],[username,password_hash,tags,hashing_algorithm,limits]},{table_attributes_mismatch,rabbit_vhost,[virtual_host,limits],[virtual_host,limits,metadata]},{table_attributes_mismatch,rabbit_durable_queue,[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,operator_policy,gm_pids,decorators,state,policy_version,slave_pids_pending_shutdown,...],...},...]},...}
haproxy_1    | [ALERT] 185/021830 (8) : parsing [/usr/local/etc/haproxy/haproxy.cfg:32] : 'server rabbitmq2' : could not resolve address 'rabbitmq2'.
haproxy_1    | [ALERT] 185/021830 (8) : parsing [/usr/local/etc/haproxy/haproxy.cfg:33] : 'server rabbitmq3' : could not resolve address 'rabbitmq3'.
haproxy_1    | [ALERT] 185/021830 (8) : parsing [/usr/local/etc/haproxy/haproxy.cfg:43] : 'server rabbitmq2' : could not resolve address 'rabbitmq2'.
haproxy_1    | [ALERT] 185/021830 (8) : parsing [/usr/local/etc/haproxy/haproxy.cfg:44] : 'server rabbitmq3' : could not resolve address 'rabbitmq3'.
haproxy_1    | [ALERT] 185/021830 (8) : Failed to initialize server(s) addr.
haproxy_1    | <5>haproxy-systemd-wrapper: exit, haproxy RC=1
docker-rabbitmq-cluster_haproxy_1 exited with code 1
rabbitmq1_1  | {"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{{schema_integrity_check_failed,[{table_attributes_mismatch,rabbit_user,[username,password_hash,tags,hashing_algorithm],[username,password_hash,tags,hashing_algorithm,limits]},{table_attributes_mismatch,rabbit_vhost,[virtual_host,limits],[virtual_host,limits,metadata]},{table_attributes_mismatch,rabbit_durable_queue,[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,operator_policy,gm_pids,decorators,state,policy_version,slave_pids_pending_shutdown,vhost,options],[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,operator_policy,gm_pids,decorators,state,policy_version,slave_pids_pending_shutdown,vhost,options,type,type_state]},{table_attributes_mismatch,rabbit_queue,[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,operator_policy,gm_pids,decorators,state,policy_version,slave_pids_pending_shutdown,vhost,options],[name,durable,auto_delete,exclusive_owner,arguments,pid,slave_pids,sync_slave_pids,recoverable_slaves,policy,operator_policy,gm_pids,decorators,state,policy_version,slave_pids_pending_shutdown,vhost,options,type,type_state]}]},{rabbit,start,[normal,[]]}}}"}
rabbitmq1_1  | Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{schema_integrity_check_failed,[{table_attributes_mismatch,rabbit_user,[username,password_hash,tags,hashing_algorithm
rabbitmq1_1  | 
rabbitmq1_1  | Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done
docker-rabbitmq-cluster_rabbitmq1_1 exited with code 0

erlang cookie generated is common but not sure why rabbitmq instances are crashing.

Leave a Comment