[Solved]-Very large python function definition in exec() call crashes Django but does not crash directly executed Python code

by

4👍

The problem might be due to int(s), float(s) and others may cause segmentation fault

As mentioned here:

Please try setting the environmental flag PYTHONMALLOC=debug

This might allow your code to run without running into segmentation errors, if you do still get an error you should be able to catch it using.

PYTHONMALLOC=debug python3 -X tracemalloc=10

You might also want to check out: faulthandler

This module contains functions to dump Python tracebacks explicitly,
on a fault, after a timeout, or on a user signal. Call
faulthandler.enable() to install fault handlers for the SIGSEGV,
SIGFPE, SIGABRT, SIGBUS, and SIGILL signals. You can also enable them
at startup by setting the PYTHONFAULTHANDLER environment variable or
by using the -X faulthandler command line option.

Adding this for more clarity since it’s related; the following is taken from the answer provided by Darrrrrren and is a tweak to make faulthandler run on threaded django applications:

So I was able to get a stack trace by initializing Python with
faulthandler, but additionally I had to run manage.py runserver
--nothreading --noreload – for some reason if you do not disable threading with Django, even faulthandler will not print a stack trace.

👤alexisdevarennes

3👍

This sounds like a job to divide and conquer!

Split your exec block up into parts to find where it fails, attempting to catch BaseException rather than Exception and dumping progress

If you believe you’re hitting a segfault, you can handle it using signal.signal(signalnum, handler) example

As they’re guaranteed to be a contained block of logic, you could begin new blocks to execute by splitting at def and if statements. If most if statements are at the highest scope, you should be able to split directly on them, otherwise some additional scope detection will be needed.

import signal
import sys

CONTENT_AND_POS = {
    "text_lines": [],    # first attempt is exec("") without if
    "block_line_no": 1,  # first block should be at line 1+
}

def report(text_lines, line_no, msg=""):
    """ display progress to the console """
    print("running code block at {}:{}\n{}".format(
        line_no, msg, text_lines))  # NOTE reordered from args

def signal_handler_segfault(signum, frame):
    """ try to show where the segfault occurred """
    report(
        "\n".join(CONTENT_AND_POS["text_lines"]),
        CONTENT_AND_POS["block_line_no"],
        "SIGNAL {}".format(signum)
    )
    sys.exit("caught segfault")

# initial setup
signal.signal(signal.SIGSEGV, signal_handler_segfault)
path_code_to_exec = sys.argv[1]  # consider argparse
print("reading from {}".format(path_code_to_exec))

# main entrypoint
with open(path_code_to_exec) as fh:
    for line_no, line in enumerate(fh, 1):  # files are iterable by-line
        if line.startswith(("def", "if")):  # new block to try
            text_exec_block = "\n".join(CONTENT_AND_POS["text_lines"])
            try:
                exec(text_exec_block, globals())
            except BaseException as ex:
                report(
                    text_exec_block,
                    CONTENT_AND_POS["block_line_no"],
                    str(repr(ex)))
                # catching BaseException will squash exit, ctrl+C, et al.
                sys.exit("caught BaseException")
            # reset for the next block
            CONTENT_AND_POS["block_line_no"] = line_no  # new block begins
            CONTENT_AND_POS["text_lines"].clear()
        # continue with new or existing block
        CONTENT_AND_POS["text_lines"].append(line)

    # execute the last block (which is otherwise missed)
    exec_text_lines(
        CONTENT_AND_POS["text_lines"],
        CONTENT_AND_POS["block_line_no"]
    )

print("successfully executed {} lines".format(line_no))

If this still ends silently, output the line number of each block before executing it. You may need to write to a file or sys.stdout/stderr to ensure output isn’t lost

👤ti7

2👍

If you’re using Python 2 (perhaps accidentally), you’re simply passing too much to exec

You can reproduce this as follows (also see a relevant codegolf!)

% python2
>>> exec(
... """if True:
...     pass
... """ * (200 * 1000)  # 400k lines
... )
segmentation fault python2

You should be able to fix this by breaking it up (described in my other answer), or by writing the code to a file and importing it instead (as suggested/already implemented in comments)

This limit on exec should be fixed in Python 3 (RecursionError), but may affect a few unlucky versions (see ticket).

👤ti7

1👍

So I was able to get a stack trace by initializing Python with faulthandler, but additionally I had to run manage.py runserver --nothreading --noreload – for some reason if you do not disable threading with Django, even faulthandler will not print a stack trace.

Fatal Python error: Segmentation fault

Current thread 0x00007fe61836b740 (most recent call first):
  File "/apps/AADD/projects/FAF/Web App/faf/modelling/views.py", line 42 in index
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/base.py", line 113 in _get_response
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/utils/deprecation.py", line 94 in __call__
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34 in inner
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/base.py", line 75 in get_response
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/handlers/wsgi.py", line 133 in __call__
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/contrib/staticfiles/handlers.py", line 68 in __call__
  File "/apps/AADD/envs/faf/lib/python3.6/wsgiref/handlers.py", line 137 in run
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/servers/basehttp.py", line 197 in handle_one_request
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/servers/basehttp.py", line 172 in handle
  File "/apps/AADD/envs/faf/lib/python3.6/socketserver.py", line 724 in __init__
  File "/apps/AADD/envs/faf/lib/python3.6/socketserver.py", line 364 in finish_request
  File "/apps/AADD/envs/faf/lib/python3.6/socketserver.py", line 351 in process_request
  File "/apps/AADD/envs/faf/lib/python3.6/socketserver.py", line 320 in _handle_request_noblock
  File "/apps/AADD/envs/faf/lib/python3.6/socketserver.py", line 241 in serve_forever
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/servers/basehttp.py", line 216 in run
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 139 in inner_run
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 104 in run
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 95 in handle
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/base.py", line 369 in execute
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 60 in execute
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/base.py", line 328 in run_from_argv
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/__init__.py", line 395 in execute
  File "/apps/AADD/envs/faf/lib/python3.6/site-packages/django/core/management/__init__.py", line 401 in execute_from_command_line
  File "manage.py", line 17 in main
  File "manage.py", line 21 in <module>
Segmentation fault

If I provide unlimited stack space, the exec() actually works in Django, but only with --nothreading . So I have a hunch that Django is somehow restricting stack size to spawned off threads.

👤Darrrrrren

Leave a comment