pcre2_set_compile_recursion_guard man page
PCRE2 — Perl-compatible regular expressions (revised API)
int pcre2_set_compile_recursion_guard(pcre2_compile_context *ccontext, int (*guard_function)(uint32_t, void *), void *user_data);
This function defines, within a compile context, a function that is called whenever pcre2_compile() starts to compile a parenthesized part of a pattern. The first argument to the function gives the current depth of parenthesis nesting, and the second is user data that is supplied when the function is set up. The callout function should return zero if all is well, or non-zero to force an error. This feature is provided so that applications can check the available system stack space, in order to avoid running out. The result of pcre2_set_compile_recursion_guard() is always zero.
There is a complete description of the PCRE2 native API in the pcre2api page and a description of the POSIX API in the pcre2posix page.