ejabberd burning cpu and eating memory

ejabberd-2.1.3-5.fc12.i686
erlang-R13B-04.1.fc12.i686

After running less than one day, ejabberd crashed. When I restarted, I noticed it was using 100% cpu.
I have only one domain with one user. Configuration is pretty standard.
One msn transport active, using Spectrum.
Captcha enabled but not in use.

For this simple config it was using an awful lot of memory. Trying to allocate almost a gigabyte of heap memory failed.

=erl_crash_dump:0.1
Mon Apr  5 14:18:11 2010
Slogan: eheap_alloc: Cannot allocate 912262800 bytes of memory (of type "heap").
System version: Erlang R13B04 (erts-5.7.5) [source] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:true]
Compiled: Wed Mar 17 15:28:34 2010
Taints:
Atoms: 14141
=memory
total: 1128752984
processes: 1107717980
processes_used: 1107708476
system: 21035004
atom: 635329
atom_used: 628293
binary: 22296
code: 6059853
ets: 5085196
...

Crashlog is very long.
You can find the whole file here

Just downgraded to

Just downgraded to erlang-R13B-01.3.fc12.i686.rpm and everything runs fine again.
Beam.smp now uses only 26MB and 0.0% CPU.
I think there is a problem with that particular erlang version.

Re: ejabberd burning cpu and eating memory

I don't think the problem is in Erlang version. As I see from the crash dump, the problem is in your transport. Do you have max_fsm_queue option configured? Refer to http://www.process-one.net/en/ejabberd/guide_en#listened-options for the explanation of this option.

Same problem here

... and already tried with max_fsm_queue.
I've pymsn-t and pyicq-t running for years now, yesterday I installed spectrum transports (currently for testing only), and ejabberd crashed overnight.
Right now the situation is like that: when no spectrum transport is running, ejabberd uses almost 0% cpu and about 613M Ram. After starting a spectrum transport (only one, doesn't matter which one) the cpu load of ejabberd increases to 100% after a few minutes.
I played around with the configuration options in ejabberd, tried almost everything - reads like this now:

 {5351, ejabberd_service, [{access,local},{max_fsm_queue, 5},
                           {max_stanza_size, 65536},{shaper, c2s_shaper},
                           {backlog, 2},{service_check_from, false},
                           {host, "msntesting.domain.tld",
                           [{password, "random"}]}]},

I changed my ejabberd loglevel from 2 to debug and there are very much messages like:

=INFO REPORT==== 2010-04-18 14:15:37 ===
D(<0.10861.0>:ejabberd_router:301) : route
        from {jid,[],"msntesting.server.tld",[],[],
                  "msntesting.server.tld",[]}
        to {jid,[],"msntesting.server.tld",[],[],
                "msntesting.server.tld",[]}
        packet {xmlelement,"iq",
                   [{"type","error"},
                    {"to","msntesting.server.tld"},
                    {"from","msntesting.server.tld"},
                    {"id","uid:4bcaf79d:1ded0839"}],
                   [{xmlelement,"ping",[{"xmlns","urn:xmpp:ping"}],[]},
                    {xmlelement,"error",
                        [{"code","405"},{"type","cancel"}],
                        [{xmlelement,"not-allowed",
                             [{"xmlns","urn:ietf:params:xml:ns:xmpp-stanzas"}],
                             []}]},
                    {xmlelement,"error",
                        [{"code","405"},{"type","cancel"}],
                        [{xmlelement,"not-allowed",
                             [{"xmlns","urn:ietf:params:xml:ns:xmpp-stanzas"}],

Any ideas whats going on here/how to fix this?

Edit to add some additional information: I'm using Ejabberd 2.1.3 compiled with Erlang R13B04 on Gentoo. I did set max_fsm_queue to values between 5 and 1000, always the same result. Also I tried to disable mod_ping for the domain Spectrum is connecting to.
If I should post more of the logfiles or try something else (perhaps without downgrading Erlang) just tell me, I'll do ;)

You can workaround the

You can workaround the problem by deactivating the XMPP pings in spectrum, but for this you have to compile spectrum yourself. In line 1527 in the file main.cpp (current git snapshot) you can comment out "purple_timeout_add_seconds(60, &sendPing, this);", so that it never sends XMPP pings. After compiling and installing spectrum with this workaround ejabberd should not crash again. At least I could fix my server with this change.

Commenting out didn't help much...

the cpu load was a bit delayed, I didn't turn ejabberd debug log on so can't be sure if I made a mistake.

badlop wrote:

Maybe the problem is related to this one: RFC3920/9.3.1 violation, ejabberd sends error stanza as reply to error stanza.

This one helped me - and solved my problem :-) Thanks badlop, again...

If any Gentoo user has the same errors, try the updated ebuilds from my Overlay (Spectrum with 'ping' useflag, ejabberd-2.1.3-r1 with patch from EJAB-930).

I see you added the ticket

I see you added the ticket 100% CPU Usage Problem with a Transport that does XMPP Pings

Maybe the problem is related to this one: RFC3920/9.3.1 violation, ejabberd sends error stanza as reply to error stanza.

If you agree that both problems may be related, you can try the patch from that ticket, and report if it solves your problem.

Syndicate content