Linux / apache web-server segmentation fault warnings

Lately I have been receiving a lot of segmentation fault warnings on my web-server.

The warnings look like:

[notice] child pid xxxx exit signal Segmentation fault (11)

I have consulted with the server provider (it is a dedicated redhat enterprise server) and they could not find anything.

What I have done so far:

Since the error I have already tried the following:

• I have added more ram
• I have turned off / turned on several php modules (they sent me to a web-page someone had the same problem, caused by an excessive amount of php modules)

At the moments the warnings occur, there seems to be plenty of free ram left and the number of processes is very low (the number of httpd processes is about a quarter of the maximum allowed).

What can be causing these errors?

Edit: current versions

• apache: 2.0.52
• php: 5.2.8
• RHEL 4

From serverfault jeroen

• Have you updated apache or any of its modules lately? I've seen this issue when you have a mod which was built against a different interface version (expat in my case) I think. If you can narrow it down to a few pieces of software, check all the version info for this. See http://www.dscpl.com.au/wiki/ModPython/Articles/ExpatCausingApacheCrash

  jeroen : Thanks, I have not updated anything myself, but I will look into the expat thing.
  From Dana the Sane

• Often when I've seen this it's been caused by a PHP script doing infinite recursion. You can use the timestamp on your error_log to identify the requests from the access_log which might have caused the problem. If there are any PHP script requests there, you (or your developers) should take a look at those pages and see if there's any potential problems there.

  Jim.

  jeroen : Aiaiai, I should have done that already, thanks!
  From Jim OHalloran

• What can be causing these errors?

  One thing that would be helpful in diagnosing the issue is if you try and find what pages/resources are being viewed when this error happens. If you are able check your access log to see what is happening at the same time the segfault happens.

  It is possible that some php script or something else is doing something that is causing the crash. Once you figure out what page/file is causing the crash, then you can dig into figuring out what it is doing, or if there are an bug fixes available.

  jeroen : Thanks, I am going to look into that right now.
  From Zoredache

• OKay, a segmentation fault is an indication of a program bug; it's not a matter of having too little memory. It occurs when a program tries to access memory it's not allowed to access.

  If it's happening in Apache, you need to enable core dumps and restart the httpd process. There's a guide to doing this in the Apache documentation.

  jeroen : Thanks for your response. I have once myself noticed a page not loading / disconnection while opening a page heavy with php, so I will be checking that first (also I'm a bit nervous fiddling with the apache configuration as that will affect all of my sites...).
  From Charlie Martin

• Have observed "exit signal Segmentation fault (11)" error lot of times. This happens when some scripts dont complete while apache crashes.

  If you are using PHP based website on apache this is very likely to happen.

  If at all you are using PHP Check following config in php config in conf.d/php.conf:

  SetOutputFilter PHP SetInputFilter PHP LimitRequestBody 524288

  Remove the LimitRequestBody line. and observe. The error should go away.

  Also if you are using apache with php following versions work perfectly. apache 2.0.43 and php 4.3.0_RC1

  Note: This answer is purely based on my past research on the error and issue faced by me. Let me know your current configuration and feedback.

  jeroen : This sounds promising, but I'm afraid I don´t have a php.conf file. I have noticed the segmentation fault error myself while opening a php page, so it could definitely be something. I will add some version information to the question.

  viky : Checked the versions of php. The bug was mentioned with PHP 5. thats your current version. presuming thats the cause. can you revert to php RC4 and check if the error goes( only if you can ). If you do some googling on php forums you should see this error. am sorry, I am a new user to serverfault and hence it is not letting me paste links for referrance which I used.

  viky : @jeroen. any luck? did you try downgrading the php version?
  From viky

• I recently had one of those errors the other day and it was related to MySQL not having proper access to the database for Cacti.

  There was an array in the file global_forms.php which set an array as one of the values to a key. That array had a key and the value called a function to load a configuration value from the database. I would have never expected the combination to result in a segmentation fault but after tracking it down there was no other cause.

  From kovert

• Do you have the right mod_whatever downloaded? I had a similar situation where I downloaded mod_jk for Solaris, but the Apache I was running was included from Sun, and was not compiled with gcc, but with their compiler. I had to download mod\jk source and compile with the Sun compiler.

  It's a shot in the dark, I know, but since no answer is marked as chosen, I thought I'd throw it out there.

  From caseyboardman

• Check this blog entnry for details on how to work with core dumps to find the bug in PHP code:

  http://www.omh.cc/blog/2008/mar/6/fixing-apache-segmentation-faults-caused-php/

  This is not my blog post - I just stumbled upon it and found it useful.

  Scott

  jeroen : Thanks, the problem is that it is on a production machine and the I only get the error every once in a while, opening exactly the same pages.
  From Scott Davey