Resources.DocsTsCrash History

Hide minor edits - Show changes to markup

April 24, 2013, at 06:23 PM by 213.233.101.41 -
Changed lines 1-77 from:

Resources -> Documentation -> My OpenSIPS is crashing

What is the problem?

Most likely you have stumbled upon a bug in OpenSIPS, which can be caused by a variety of of issues, like invalid memory access, memory corruption, etc.

Where to look for logs

If you have log_stderror=no in your config file (opensips.cfg), all the logs from OpenSIPS will be sent to the syslog service, so you have the check into the corresponding file, typically:

  1. /var/log/syslog
  2. /var/log/messages

You can simply check system log by:
$ tail -f /var/log/messages

Note you might need root permissions to access these files! If you do not have it, set log_stderror=yes in your config and you will get the log in the console.

If you have log_stderror=yes, you should get the log in the console where you are run OpenSIPS

Reading the error logs

First, you should check the error logs to make sure this was an actual crash, as opposed to the scenario where another entity forcefully killed your OpenSIPS process. A crash is usually detected by searching for the following string in the logs :

child process 6645 exited by a signal 11

If you do not see such an error message, but rather you see things like

terminating due to SIGTERM

or you do not see anything logs, you just notice the OpenSIPS suddenly dying, then most likely your OpenSIPS was sent a SIGTERM or a SIGKILL, and you should investigate why some other entity decided to terminate your OpenSIPS process.

If you see the 'exited by a signal 11' then your OpenSIPS has crashed and you should proceed into investigating the core file. Usually, the 'exited by a signal 11' is accompanied by a 'core was generated' message, which tells you that OpenSIPS was succesful in dumping a core file

How to make sure OpenSIPS dumps a proper core file

Typically, in a crash scenario, OpenSIPS should dump a core file which contains the full memory contents at the moment of the crash.

Several things have to be taken care of to make sure that your OpenSIPS dumps a proper core file, that can be used for investigating the crash. Failing to follow these steps could lead to the fact that either the core file is not being generated ( you will see in the logs a message like 'core was not generated' ), or you could end up with a core file that gets overwritten at OpenSIPS shutdown, which would not be useful for further debugging.

1. Pass the '-w [FOLDER]' parameter to your OpenSIPS at startup. The [FOLDER] path that you provide must be write accessible by your OpenSIPS, and will be the folder that will contain the core files in the eventuality of a crash

2. Before starting OpenSIPS, make sure to run 'ulimit -c unlimited' . This tells the system to allow OpenSIPS to dump a core file of unlimited size. This is useful, because in case of a crash, the core file will be at least the size of your shared memory + private memory. The ulimit commands usually should go in your init.d script for OpenSIPS

3. If you are running OpenSIPS with a different username and group ( -u and -g params ), some kernels might need some extra configuration to allow core dumps :

  • echo 1 > /proc/sys/fs/suid_dumpable

4. Make sure your core files do not get overwritten. There are several sysctl options that can be used for this :

  • echo 1 > /proc/sys/kernel/core_uses_pid
    • When you see the message 'child process 6645 exited by a signal 11' , you should get a core file called 'core.6645' in your -w directory
  • For more customization of the core file name, you can run setup your own core name pattern with something like : echo 'core.t.sigp' > /proc/sys/kernel/core_pattern
    • This will have the core file contain the process name ( % e ), the timestamp ( % t ), the received signal ( % s ) and the pid file ( % p )

Extracting a back trace from the core file

Browse the logs for the

child process 6645 exited by a signal 11

and the run

gdb opensips core.6645 

Once in the gdb environment, run the following command

bt full

This will dump the full stack trace that lead to the crash. Send the crash to the OpenSIPS developers ( usually on the Sourceforge Bug tracker of on the DEV mailing list )

Do not delete the core file and also make sure to keep an exact copy of your OpenSIPS binary, as the OpenSIPS developers might need to extra more information from the core file, like printing various variables, etc. The core file on it's own is useless without the exact OpenSIPS binary file that lead to the crash.

to:

(:redirect Documentation.TroubleShooting-Crash quiet=1 :)

December 07, 2012, at 12:24 PM by vlad_paiu -
Changed line 1 from:

Resources -> Documentation -> OpenSIPS is crashing

to:

Resources -> Documentation -> My OpenSIPS is crashing

December 07, 2012, at 12:19 PM by vlad_paiu -
Changed line 77 from:

Do not delete the core file and also make sure to keep an exact copy of your OpenSIPS binary, as the OpenSIPS developers might need to extra more information from the core file, like printing various variables, etc.

to:

Do not delete the core file and also make sure to keep an exact copy of your OpenSIPS binary, as the OpenSIPS developers might need to extra more information from the core file, like printing various variables, etc. The core file on it's own is useless without the exact OpenSIPS binary file that lead to the crash.

December 07, 2012, at 12:18 PM by vlad_paiu -
Changed line 52 from:
  • This will have the core file contain the process name ( t ), the received signal ( p )
to:
  • This will have the core file contain the process name ( % e ), the timestamp ( % t ), the received signal ( % s ) and the pid file ( % p )
December 07, 2012, at 12:17 PM by vlad_paiu -
December 07, 2012, at 12:16 PM by vlad_paiu -
Changed line 75 from:

This will dump the full stack trace that lead to the crash. Send the crash to the OpenSIPS developer ( usually on the Sourceforge Bug tracker of on the DEV mailing list )

to:

This will dump the full stack trace that lead to the crash. Send the crash to the OpenSIPS developers ( usually on the Sourceforge Bug tracker of on the DEV mailing list )

December 07, 2012, at 12:15 PM by vlad_paiu -
Changed lines 39-40 from:

Several things have to be taken care of to make sure that your OpenSIPS dumps a proper core file, that can be used for investigating the crash. Failing to follow these steps could lead to the fact that either the core file is not being generated ( you will see in the logs a message like 'core was not generated', or you could end up with a core file that gets overwritten at OpenSIPS shutdown, which would not be useful for further debugging.

to:

Several things have to be taken care of to make sure that your OpenSIPS dumps a proper core file, that can be used for investigating the crash. Failing to follow these steps could lead to the fact that either the core file is not being generated ( you will see in the logs a message like 'core was not generated' ), or you could end up with a core file that gets overwritten at OpenSIPS shutdown, which would not be useful for further debugging.

Added lines 57-77:

Browse the logs for the

child process 6645 exited by a signal 11

and the run

gdb opensips core.6645 

Once in the gdb environment, run the following command

bt full

This will dump the full stack trace that lead to the crash. Send the crash to the OpenSIPS developer ( usually on the Sourceforge Bug tracker of on the DEV mailing list )

Do not delete the core file and also make sure to keep an exact copy of your OpenSIPS binary, as the OpenSIPS developers might need to extra more information from the core file, like printing various variables, etc.

December 07, 2012, at 12:11 PM by vlad_paiu -
Deleted line 36:
Changed lines 52-56 from:
  • This will have the core file contain the process name ( t ), the received signal ( p )
to:
  • This will have the core file contain the process name ( t ), the received signal ( p )

Extracting a back trace from the core file

December 07, 2012, at 12:09 PM by vlad_paiu -
Added lines 37-39:

Typically, in a crash scenario, OpenSIPS should dump a core file which contains the full memory contents at the moment of the crash.

Changed lines 53-57 from:
  • This will have the core file contain the process name ( t ), the received signal ( p )

Typically, in a crash scenario, OpenSIPS should dump a core file which contains the full memory contents at the moment of the crash.

to:
  • This will have the core file contain the process name ( t ), the received signal ( p )
December 07, 2012, at 12:08 PM by vlad_paiu -
Changed lines 43-46 from:

3. Make sure your core files do not get overwritten. There are several sysctl options that can be used for this :

to:

3. If you are running OpenSIPS with a different username and group ( -u and -g params ), some kernels might need some extra configuration to allow core dumps :

  • echo 1 > /proc/sys/fs/suid_dumpable

4. Make sure your core files do not get overwritten. There are several sysctl options that can be used for this :

Changed lines 48-52 from:
  • ffd
to:
  • When you see the message 'child process 6645 exited by a signal 11' , you should get a core file called 'core.6645' in your -w directory
  • For more customization of the core file name, you can run setup your own core name pattern with something like : echo 'core.t.sigp' > /proc/sys/kernel/core_pattern
    • This will have the core file contain the process name ( t ), the received signal ( p )
December 07, 2012, at 11:59 AM by vlad_paiu -
Changed lines 44-45 from:
  • fd
to:
  • echo 1 > /proc/sys/kernel/core_uses_pid
    • ffd
December 07, 2012, at 11:58 AM by vlad_paiu -
Changed lines 43-44 from:

3. Make sure your core files do not get overwritten.

to:

3. Make sure your core files do not get overwritten. There are several sysctl options that can be used for this :

  • fd
December 07, 2012, at 11:52 AM by vlad_paiu -
Added line 40:
Added line 42:
December 07, 2012, at 11:52 AM by vlad_paiu -
Changed lines 39-41 from:

1.

to:

1. Pass the '-w [FOLDER]' parameter to your OpenSIPS at startup. The [FOLDER] path that you provide must be write accessible by your OpenSIPS, and will be the folder that will contain the core files in the eventuality of a crash 2. Before starting OpenSIPS, make sure to run 'ulimit -c unlimited' . This tells the system to allow OpenSIPS to dump a core file of unlimited size. This is useful, because in case of a crash, the core file will be at least the size of your shared memory + private memory. The ulimit commands usually should go in your init.d script for OpenSIPS 3. Make sure your core files do not get overwritten.

December 07, 2012, at 11:48 AM by vlad_paiu -
Added lines 26-39:

If you do not see such an error message, but rather you see things like

terminating due to SIGTERM

or you do not see anything logs, you just notice the OpenSIPS suddenly dying, then most likely your OpenSIPS was sent a SIGTERM or a SIGKILL, and you should investigate why some other entity decided to terminate your OpenSIPS process.

If you see the 'exited by a signal 11' then your OpenSIPS has crashed and you should proceed into investigating the core file. Usually, the 'exited by a signal 11' is accompanied by a 'core was generated' message, which tells you that OpenSIPS was succesful in dumping a core file

How to make sure OpenSIPS dumps a proper core file

Several things have to be taken care of to make sure that your OpenSIPS dumps a proper core file, that can be used for investigating the crash. Failing to follow these steps could lead to the fact that either the core file is not being generated ( you will see in the logs a message like 'core was not generated', or you could end up with a core file that gets overwritten at OpenSIPS shutdown, which would not be useful for further debugging.

1.

December 07, 2012, at 11:41 AM by vlad_paiu -
Added lines 1-27:

Resources -> Documentation -> OpenSIPS is crashing

What is the problem?

Most likely you have stumbled upon a bug in OpenSIPS, which can be caused by a variety of of issues, like invalid memory access, memory corruption, etc.

Where to look for logs

If you have log_stderror=no in your config file (opensips.cfg), all the logs from OpenSIPS will be sent to the syslog service, so you have the check into the corresponding file, typically:

  1. /var/log/syslog
  2. /var/log/messages

You can simply check system log by:
$ tail -f /var/log/messages

Note you might need root permissions to access these files! If you do not have it, set log_stderror=yes in your config and you will get the log in the console.

If you have log_stderror=yes, you should get the log in the console where you are run OpenSIPS

Reading the error logs

First, you should check the error logs to make sure this was an actual crash, as opposed to the scenario where another entity forcefully killed your OpenSIPS process. A crash is usually detected by searching for the following string in the logs :

child process 6645 exited by a signal 11

Typically, in a crash scenario, OpenSIPS should dump a core file which contains the full memory contents at the moment of the crash.


Page last modified on April 24, 2013, at 06:23 PM