is starting with status: started, takes too much time to boot

When I want to restart the server, it takes too much time to do so.
I can immediately stop it. But when I start it, I obtain absolutely nothing in the logs and I get this:

/sbin/ejabberdctl status

The node ejabberd@localhost is starting with status: started
ejabberd is not running in that node
Check for error messages: //var/log/ejabberd/ejabberd.log
or other files in that directory.

I also noticed my log files are empty although ejabberd seems to know where to find them (the given path).
This is a 2.1.5, installed by building myself.

Here is the config file:

%%%
%%%     Debian ejabberd configuration file
%%%     This config must be in UTF-8 encoding
%%%
%%% The parameters used in this configuration file are explained in more detail
%%% in the ejabberd Installation and Operation Guide.
%%% Please consult the Guide in case of doubts, it is available at
%%% /usr/share/doc/ejabberd/guide.html

%%%   ===================================
%%%   OVERRIDE OPTIONS STORED IN DATABASE

%%
%% Override global options (shared by all ejabberd nodes in a cluster).
%%
override_global.

%%
%% Override local options (specific for this particular ejabberd node).
%%
override_local.

%%
%% Remove the Access Control Lists before new ones are added.
%%
override_acls.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Options which are set by Debconf and managed by ucf

%% Admin user
{acl, admin, {user, "jabberangel", "mydomain.com"}}.

%% Hostname
%{hosts, ["lazytune.com"]}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%% This configuration file contains Erlang terms.
%%% In case you want to understand the syntax, here are the concepts:
%%%
%%%  - The character to comment a line is %
%%%
%%%  - Each term ends in a dot, for example:
%%%      override_global.
%%%
%%%  - A tuple has a fixed definition, its elements are
%%%    enclosed in {}, and separated with commas:
%%%      {loglevel, 4}.
%%%
%%%  - A list can have as many elements as you want,
%%%    and is enclosed in [], for example:
%%%      [http_poll, web_admin, tls]
%%%
%%%  - A keyword of ejabberd is a word in lowercase.
%%%    The strings are enclosed in "" and can have spaces, dots...
%%%      {language, "en"}.
%%%      {ldap_rootdn, "dc=example,dc=com"}.
%%%
%%%  - This term includes a tuple, a keyword, a list and two strings:
%%%      {hosts, ["jabber.example.net", "im.example.com"]}.
%%%

%%%   =========
%%%   DEBUGGING

%%
%% loglevel: Verbosity of log files generated by ejabberd.
%% 0: No ejabberd log at all (not recommended)
%% 1: Critical
%% 2: Error
%% 3: Warning
%% 4: Info
%% 5: Debug
%%
{loglevel, 4}.

%%
%% watchdog_admins: If an ejabberd process consumes too much memory,
%% send live notifications to those Jabber accounts.
%%
%%{watchdog_admins, ["bob@example.com"]}.

%%%   ================
%%%   SERVED HOSTNAMES

%%
%% hosts: Domains served by ejabberd.
%% You can define one or several, for example:
%% {hosts, ["example.net", "example.com", "example.org"]}.
%%
%% (This option is defined by debconf earlier)
{hosts, ["mydomain.com", "guests.mydomain.com"]}.

%%
%% route_subdomains: Delegate subdomains to other Jabber server.
%% For example, if this ejabberd serves example.org and you want
%% to allow communication with a Jabber server called im.example.org.
%%
%%{route_subdomains, s2s}.

%%%   ===============
%%%   LISTENING PORTS

%%
%% listen: Which ports will ejabberd listen, which service handles it
%% and what options to start it with.
%%
{listen,
[
  {5222, ejabberd_c2s, [
{access, c2s},
{shaper, c2s_shaper},
{max_stanza_size, 65536},
starttls, {certfile, "/etc/ejabberd/ejabberd.pem"}
       ]},
  {4560, ejabberd_xmlrpc, []}

  %%
  %% To enable the old SSL connection method (deprecated) in port 5223:
  %%
  %%{5223, ejabberd_c2s, [
  %% {access, c2s},
  %% {shaper, c2s_shaper},
  %% {max_stanza_size, 65536},
  %% tls, {certfile, "/etc/ejabberd/ejabberd.pem"}
  %%        ]},

%  {5269, ejabberd_s2s_in, [
%    {shaper, s2s_shaper},
%    {max_stanza_size, 131072}
%   ]},

  %% External MUC jabber-muc (but internal mod_muc is better :))
  %%{5554, ejabberd_service, [
  %%     {ip, {127, 0, 0, 1}},
  %%     {access, all},
  %%     {shaper_rule, fast},
  %%     {host, "muc.localhost", [{password, "secret"}]}
  %%     ]},

  %% Jabber ICQ Transport
  %%{5555, ejabberd_service, [
  %%     {ip, {127, 0, 0, 1}},
  %%     {access, all},
  %%     {shaper_rule, fast},
  %%     {hosts, ["icq.localhost", "sms.localhost"],
  %%        [{password, "secret"}]}
  %%     ]},

  %% AIM Transport
  %%{5556, ejabberd_service, [
  %%     {ip, {127, 0, 0, 1}},
  %%     {access, all},
  %%     {shaper_rule, fast},
  %%     {host, "aim.localhost", [{password, "secret"}]}
  %%     ]},

  %% MSN Transport
  %%{5557, ejabberd_service, [
  %%     {ip, {127, 0, 0, 1}},
  %%     {access, all},
  %%     {shaper_rule, fast},
  %%     {host, "msn.localhost", [{password, "secret"}]}
  %%     ]},

  %% Yahoo! Transport
  %%{5558, ejabberd_service, [
  %%     {ip, {127, 0, 0, 1}},
  %%     {access, all},
  %%     {shaper_rule, fast},
  %%     {host, "yahoo.localhost", [{password, "secret"}]}
  %%     ]},

  %% External JUD (internal is more powerful,
  %% but doesn't allow to register users from other servers)
  %%{5559, ejabberd_service, [
  %%     {ip, {127, 0, 0, 1}},
  %%     {access, all},
  %%     {shaper_rule, fast},
  %%     {host, "jud.localhost", [{password, "secret"}]}
  %%     ]},

%  {5280, ejabberd_http, [
% http_poll,
% web_admin
% ]}

]}.

%%
%% s2s_use_starttls: Enable STARTTLS + Dialback for S2S connections.
%% Allowed values are: true or false.
%% You must specify a certificate file.
%%
{s2s_use_starttls, true}.

%%
%% s2s_certfile: Specify a certificate file.
%%
{s2s_certfile, "/etc/ejabberd/ejabberd.pem"}.

%%
%% domain_certfile: Specify a different certificate for each served hostname.
%%
%%{domain_certfile, "example.org", "/path/to/example_org.pem"}.
%%{domain_certfile, "example.com", "/path/to/example_com.pem"}.

%%
%% S2S whitelist or blacklist
%%
%% Default s2s policy for undefined hosts.
%%
%%{s2s_default_policy, allow}.
{s2s_default_policy, deny}.

%%
%% Allow or deny communication with specific servers.
%%
%%{{s2s_host, "goodhost.org"}, allow}.
%%{{s2s_host, "badhost.org"}, deny}.

%%%   ==============
%%%   AUTHENTICATION

%%
%% auth_method: Method used to authenticate the users.
%% The default method is the internal.
%% If you want to use a different method,
%% comment this line and enable the correct ones.
%%
{auth_method, internal}.

%{auth_method, external}.
%{extauth_program, "/home/flo/mydo/engine/utils/ejabberd.py"}.

%%
%% Authentication using external script
%% Make sure the script is executable by ejabberd.
%%
%%{auth_method, external}.
%%{extauth_program, "/path/to/authentication/script"}.

%%
%% Authentication using ODBC
%% Remember to setup a database in the next section.
%%
%%{auth_method, odbc}.

%%
%% Authentication using PAM
%%
%%{auth_method, pam}.
%%{pam_service, "pamservicename"}.

%%
%% Authentication using LDAP
%%
%%{auth_method, ldap}.
%%
%% List of LDAP servers:
%%{ldap_servers, ["localhost"]}.
%%
%% Encryption of connection to LDAP servers (LDAPS):
%%{ldap_encrypt, tls}.
%%
%% Port connect to LDAP server:
%%{ldap_port, 636}.
%%
%% LDAP manager:
%%{ldap_rootdn, "dc=example,dc=com"}.
%%
%% Password to LDAP manager:
%%{ldap_password, "******"}.
%%
%% Search base of LDAP directory:
%%{ldap_base, "dc=example,dc=com"}.
%%
%% LDAP attribute that holds user ID:
%%{ldap_uids, [{"mail", "%u@mail.example.org"}]}.
%%
%% LDAP filter:
%%{ldap_filter, "(objectClass=shadowAccount)"}.

%%
%% Anonymous login support:
%%   auth_method: anonymous
%%   anonymous_protocol: sasl_anon | login_anon | both
%%   allow_multiple_connections: true | false
%%
%%{host_config, "public.example.org", [{auth_method, anonymous},
%%                                     {allow_multiple_connections, false},
%%                                     {anonymous_protocol, sasl_anon}]}.
%%
%% To use both anonymous and internal authentication:
%%
%%{host_config, "public.example.org", [{auth_method, [internal, anonymous]}]}.

{host_config, "guests.mydomain.com", [{auth_method, [anonymous]},
                                  {allow_multiple_connections, true},
                                  {anonymous_protocol, login_anon}]}.

%%%   ==============
%%%   DATABASE SETUP

%% ejabberd uses by default the internal Mnesia database,
%% so you can avoid this section.
%% This section provides configuration examples in case
%% you want to use other database backends.
%% Please consult the ejabberd Guide for details about database creation.

%%
%% MySQL server:
%%
%%{odbc_server, {mysql, "server", "database", "username", "password"}}.
%%
%% If you want to specify the port:
%%{odbc_server, {mysql, "server", 1234, "database", "username", "password"}}.

%%
%% PostgreSQL server:
%%
%%{odbc_server, {pgsql, "server", "database", "username", "password"}}.
%%
%% If you want to specify the port:
%%{odbc_server, {pgsql, "server", 1234, "database", "username", "password"}}.
%%
%% If you use PostgreSQL, have a large database, and need a
%% faster but inexact replacement for "select count(*) from users"
%%
%%{pgsql_users_number_estimate, true}.

%%
%% ODBC compatible or MSSQL server:
%%
%%{odbc_server, "DSN=ejabberd;UID=ejabberd;PWD=ejabberd"}.

%%
%% Number of connections to open to the database for each virtual host
%%
%%{odbc_pool_size, 10}.

%%
%% Interval to make a dummy SQL request to keep alive the connections
%% to the database. Specify in seconds: for example 28800 means 8 hours
%%
%%{odbc_keepalive_interval, undefined}.

%%%   ===============
%%%   TRAFFIC SHAPERS

%%
%% The "normal" shaper limits traffic speed to 1.000 B/s
%%
{shaper, normal, {maxrate, 1000}}.

%%
%% The "fast" shaper limits traffic speed to 50.000 B/s
%%
{shaper, fast, {maxrate, 50000}}.

%%%   ====================
%%%   ACCESS CONTROL LISTS

%%
%% The 'admin' ACL grants administrative privileges to Jabber accounts.
%% You can put as many accounts as you want.
%%
%%{acl, admin, {user, "aleksey", "localhost"}}.
%%{acl, admin, {user, "ermine", "example.org"}}.

%%
%% Blocked users
%%
%%{acl, blocked, {user, "baduser", "example.org"}}.
%{acl, blocked, {user, "test"}}.
{acl, blocked, {user, "admin"}}.
{acl, blocked, {user, "root"}}.

%%
%% Local users: don't modify this line.
%%
{acl, local, {user_regexp, ""}}.

%%
%% More examples of ACLs
%%
%%{acl, jabberorg, {server, "jabber.org"}}.
%%{acl, aleksey, {user, "aleksey", "jabber.ru"}}.
%%{acl, test, {user_regexp, "^test"}}.
%%{acl, test, {user_glob, "test*"}}.

{acl, muc_creators, {user, "mydoangel", "mydomain.com"}}.
{acl, muc_creators, {user, "mydoangel1", "mydomain.com"}}.
{acl, muc_creators, {user, "mydoangel2", "mydomain.com"}}.

%%%   ============
%%%   ACCESS RULES

%% Define the maximum number of time a single user is allowed to connect:
{access, max_user_sessions, [{3, all}]}.

%% This rule allows access only for local users:
{access, local, [{allow, local}]}.

%% Only non-blocked users can use c2s connections:
{access, c2s, [{deny, blocked},
       {allow, all}]}.

%% For all users except admins used "normal" shaper
{access, c2s_shaper, [{none, admin},
      {normal, all}]}.

%% For all S2S connections used "fast" shaper
%{access, s2s_shaper, [{fast, all}]}.

%% Only admins can send announcement messages:
{access, announce, [{allow, admin}]}.

%% Only admins can use configuration interface:
{access, configure, [{allow, admin}]}.

%% Admins of this server are also admins of MUC service:
{access, muc_admin, [{allow, admin}]}.

%% All users are allowed to use MUC service:
{access, muc, [{allow, all}]}.
{access, muc_create, [{allow, muc_creators}]}.

%% No username can be registered via in-band registration:
%% To enable in-band registration, replace 'deny' with 'allow'
% (note that if you remove mod_register from modules list then users will not
% be able to change their password as well as register).
% This setting is default because it's more safe.
{access, register, [{deny, all}]}.

%% Everybody can create pubsub nodes
%{access, pubsub_createnode, [{allow, all}]}.
{access, pubsub_createnode, [{deny, all}]}.

%%%   ================
%%%   DEFAULT LANGUAGE

%%
%% language: Default language used for server messages.
%%
{language, "en"}.

%%%   =======
%%%   MODULES

%%
%% Modules enabled in all ejabberd virtual hosts.
%%
{modules,
[
  {mod_adhoc,    []},
  {mod_admin_extra, []},
  {mod_announce, [{access, announce}]}, % requires mod_adhoc
%  {mod_caps,     []},
  {mod_configure,[]}, % requires mod_adhoc
  {mod_ctlextra, []},
%  {mod_disco,    []},
  %%{mod_echo,   [{host, "echo.localhost"}]},
%  {mod_irc,      []},
  {mod_last,     []},
  {mod_muc,      [
  {host, "conference.mydomain.com"},
  {access, muc},
  {access_create, muc_create},
  {access_persistent, muc_create},
  {access_admin, muc_admin},
  {history_size, 0},
  {max_users, 500}
]},
  %%{mod_muc_log,[]},
  {mod_offline,  []},
  {mod_privacy,  []},
  {mod_private,  []},
%  {mod_proxy65,  [
%   {access, local},
%   {shaper, c2s_shaper}
% ]},
%  {mod_pubsub,   [ % requires mod_caps
%   {access_createnode, pubsub_createnode},
%   {plugins, ["default", "pep"]}
% ]},
  {mod_register, [
  %%
  %% After successful registration, the user receives
  %% a message with this subject and body.
  %%
  {welcome_message, none},
  %% Replace it with 'none' if you don't want to send such message:
  %%{welcome_message, none},

  %%
  %% When a user registers, send a notification to
  %% these Jabber accounts.
  %%
  %%{registration_watchers, ["admin1@example.org"]},

  {access, register}
]},
  {mod_roster,   []},
  %%{mod_service_log,[]},
  %%{mod_shared_roster,[]},
  {mod_stats,    []},
  {mod_time,     []},
  {mod_vcard,    []},
  {mod_xmlrpc,   [{port, 4560}, {ip, {127, 0, 0, 1}}, {maxsessions, 50}]}
%  {mod_version,  []}
]}.

%%% $Id: ejabberd.cfg.example 1178 2008-02-08 18:28:36Z badlop $

%%% Local Variables:
%%% mode: erlang
%%% End:
%%% vim: set filetype=erlang tabstop=8:

Do you have a Mnesia

Do you have a Mnesia (internal) database with many registered users?

Try this:
1. Stop ejabberd, kill beam, beam.smp, epmd
2. Delete log files (if any)
3. Move the Mnesia database spool files away, so ejabberd will recreate the database empty
4. Start ejabberd
5. Does it start correctly, and fast? Or it still takes long to start (or never starts)?

It runs very fast

Yes it worked nice, it started very fast.
What can I do to make it work with a lot of registered users? My old mnesia contained around 30000 users.

(I also noticed some important and non-deterministic lags in MUC rooms, could it be the same reason?)

Check tables sizes.

NecmiKara wrote:

Yes it worked nice, it started very fast.
What can I do to make it work with a lot of registered users? My old mnesia contained around 30000 users.

With 9000 registered accounts, 5000 vcards, 130000 roster items and 6000 offline messages, my server takes around one minute to load Mnesia.

Check (using Webadmin, or looking at mnesia spool files) if you have some Mnesia table with an abnormal size.

To reduce the database size, you can delete accounts unused for more than 365 days (for example) using mod_admin_extra .

NecmiKara wrote:

(I also noticed some important and non-deterministic lags in MUC rooms, could it be the same reason?)

Once the server is fully started, with minimal load, MUC rooms shouldn't lag. Unless you have millions of rooms, or hundreds of occupants in the room.

Abnormal

As file sizes, I have:

drwxr-xr-x  2 root     root        4096 2011-01-19 07:29 .
drwx--x--x 33 flo      flo         4096 2011-01-19 08:22 ..
-rw-r--r--  1 root     root         556 2011-01-09 11:44 acl.DCD
-rw-r--r--  1 root     root        1154 2011-01-09 12:06 acl.DCL
-rwxr-x---  1 ejabberd ejabberd       8 2010-10-15 01:20 caps_features.DCD
-rw-r--r--  1 root     root        1183 2011-01-09 11:44 config.DCD
-rw-r--r--  1 root     root        1925 2011-01-09 12:06 config.DCL
-rw-r--r--  1 root     root         160 2011-01-19 07:24 DECISION_TAB.LOG
-rwxr-x---  1 ejabberd ejabberd       8 2010-10-15 01:20 irc_custom.DCD
-rw-r--r--  1 root     root      253488 2011-01-18 15:42 last_activity.DCD
-rw-r--r--  1 root     root       57004 2011-01-19 07:24 last_activity.DCL
-rw-r--r--  1 root     root         668 2011-01-19 07:26 LATEST.LOG
-rw-r--r--  1 root     root        2038 2011-01-09 11:44 local_config.DCD
-rw-r--r--  1 root     root       27439 2011-01-09 12:06 local_config.DCL
-rwxr-x---  1 ejabberd ejabberd       8 2010-10-15 01:20 motd.DCD
-rwxr-x---  1 ejabberd ejabberd       8 2010-10-15 01:20 motd_users.DCD
-rwxr-x---  1 ejabberd ejabberd       8 2010-10-15 01:20 muc_registered.DCD
-rwxr-x---  1 ejabberd ejabberd       8 2010-10-15 01:20 muc_room.DCD
-rw-r--r--  1 root     root        5432 2011-01-19 06:55 offline_msg.DAT
-rwxr-x---  1 ejabberd ejabberd 2650283 2010-12-05 01:19 passwd.DCD
-rw-r--r--  1 root     root      139914 2011-01-19 07:00 passwd.DCL
-rwxr-x---  1 ejabberd ejabberd      94 2010-12-05 01:19 privacy.DCD
-rwxr-x---  1 ejabberd ejabberd    5752 2010-10-15 01:22 private_storage.DAT
-rwxr-x---  1 ejabberd ejabberd     130 2010-10-16 07:01 pubsub_index.DCD
-rwxr-x---  1 ejabberd ejabberd    5752 2010-10-15 01:22 pubsub_item.DAT
-rwxr-x---  1 ejabberd ejabberd    1239 2010-10-16 07:01 pubsub_node.DCD
-rwxr-x---  1 ejabberd ejabberd     220 2010-10-16 07:01 pubsub_state.DCD
-rwxr-x---  1 ejabberd ejabberd       8 2010-10-15 01:20 pubsub_subscription.DCD
-rwxr-x---  1 ejabberd ejabberd   88236 2010-10-19 19:31 roster.DCD
-rwxr-x---  1 ejabberd ejabberd    6968 2011-01-17 19:00 roster.DCL
-rwxr-x---  1 ejabberd ejabberd       8 2010-10-15 01:20 roster_version.DCD
-rwxr-x---  1 ejabberd ejabberd   27084 2010-10-16 07:08 schema.DAT
-rwxr-x---  1 ejabberd ejabberd       8 2010-10-15 01:20 sr_group.DCD
-rwxr-x---  1 ejabberd ejabberd       8 2010-10-15 01:20 sr_user.DCD
-rwxr-x---  1 ejabberd ejabberd 7743400 2011-01-19 07:06 vcard.DAT
-rwxr-x---  1 ejabberd ejabberd 3901699 2010-12-05 01:19 vcard_search.DCD
-rw-r--r--  1 root     root      194262 2011-01-19 07:00 vcard_search.DCL

Some of them seem too big indeed. BTW, how can I avoid these root-owned files? (I guess they appear because I execute ejabberdctl as root)

Quote:

my server takes around one minute to load Mnesia.

I am a bit disappointed by erlang then.

Quote:

To reduce the database size, you can delete accounts unused for more than 365 days (for example) using mod_admin_extra

I can't do this, I synchronize ejabberd accounts with my website's user accounts (by xmlrpc) . Should I consider external auth or clustering?

And what about the log files? Now I get them but I didn't get them at all when the server took long time to start even once it was started.

Quote:

Once the server is fully started, with minimal load, MUC rooms shouldn't lag. Unless you have millions of rooms, or hundreds of occupants in the room.

Indeed, I found the bug. The lag was not due to a ejabberd behaviour and is not related to this issue.

If problem still happens, some ideas to try

NecmiKara wrote:

As file sizes, I have:

-rw-r--r--  1 root     root      253488 2011-01-18 15:42 last_activity.DCD
-rwxr-x---  1 ejabberd ejabberd 2650283 2010-12-05 01:19 passwd.DCD
-rwxr-x---  1 ejabberd ejabberd 7743400 2011-01-19 07:06 vcard.DAT
-rwxr-x---  1 ejabberd ejabberd 3901699 2010-12-05 01:19 vcard_search.DCD

Some of them seem too big indeed.

Compare to the server that I mendioned before:
-rw-r--r-- 1 ejabberd ejabberd 957130 Jan 28 10:27 last_activity.DCD
-rw-r--r-- 1 ejabberd ejabberd 520589 Dec 27 20:08 passwd.DCD
-rw-r--r-- 1 ejabberd ejabberd 58306790 Jan 28 12:25 vcard.DAT
-rw-r--r-- 1 ejabberd ejabberd 791815 Jan 27 00:42 vcard_search.DCD

Your only file that is larger is passwd, and not that much.
The vcard table is big in my server, but it is configured to be disk-only, so I imagine it doesn't produce delay when starting Mnesia.

NecmiKara wrote:

BTW, how can I avoid these root-owned files? (I guess they appear because I execute ejabberdctl as root)

I configure ejabberd with:

./configure --enable-user=somesystemuser
make
sudo make install
sudo ejabberdctl start

This way, the actual beam program is run by somesystemuser, not by root.

NecmiKara wrote:
Quote:

my server takes around one minute to load Mnesia.

I am a bit disappointed by erlang then.

What do you prefer for a program that you start only once every few months and keeps running for months?
A) To start fast the day you start it.
B) To run fast during all the months it's running

Remember: you can reconfigure most of ejabberd usign WebAdmin, add new modules, and even reload patched modules (as long as the changes are not substantial). No need to wait 5 minutes every morning to restart ejabberd.

NecmiKara wrote:
Quote:

To reduce the database size, you can delete accounts unused for more than 365 days (for example) using mod_admin_extra

I can't do this, I synchronize ejabberd accounts with my website's user accounts (by xmlrpc) . Should I consider external auth or clustering?

Ok, then better keep the auth as now. You have many registered accounts (see 'passwd'), but few of them really login, so your Mnesia database can be considered small.

You could try this (remember to first make a copy/backup of the Mnesia spool dir):
Go to ejabberd WebAdmin -> nodes -> Your node -> Database, and change the 'passwd' table to 'Copy in disc only', like the 'vcard' table. Then press Send. Then stop ejabberd, start it and see if the start time reduces.

NecmiKara wrote:

And what about the log files? Now I get them but I didn't get them at all when the server took long time to start even once it was started.

No idea.

Syndicate content