perl5420delta - Man Page

This document describes differences between the 5.42.0 release and the 5.40.0 release.

• Constant-folded strings are now shareable via the Copy-on-Write mechanism. [GH #22163 <https://github.com/Perl/perl5/pull/22163>]

  The following code would previously have allocated eleven string buffers, each containing one million "A"s:

      my @scalars; push @scalars, ("A" x 1_000_000) for 0..9;

  Now a single buffer is allocated and shared between a CONST OP and the ten scalar elements of @scalars.

  Note that any code using this sort of constant to simulate memory leaks (perhaps in test files) must now permute the string in order to trigger a string copy and the allocation of separate buffers. For example, ("A" x 1_000_000).time might be a suitable small change.

• tr/// now runs at the same speed regardless of the internal representation of its operand, as long as the only characters being translated are ASCII-range, for example tr/A-Z/a-z/.  Previously, if the internal encoding was UTF-8, a slower, more general implementation was used.

• Code that uses the indexed function from the builtin module to generate a list of index/value pairs out of an array or list which is then passed into a two-variable foreach list to unpack those again is now optimised to be more efficient.

      my @array = (...);
  
      foreach my ($idx, $val) (builtin::indexed @array) {
          ...
      }

      foreach my ($idx, $val) (builtin::indexed LIST...) {
          ...
      }

  In particular, a temporary list twice the size of the original is no longer generated.  Instead, the loop iterates down the original array or list in-place directly, in the same way that foreach (@array) or foreach (LIST) would do.

• The peephole optimizer recognises the following zero-offset substr patterns and swaps in a new dedicated operator (OP_SUBSTR_LEFT). [GH #22785 <https://github.com/Perl/perl5/issues/22785>]

      substr($x, 0, ...)
      substr($x, 0, ..., '')

• The stringification of integers by "print" in perlfunc and "say" in perlfunc, when coming from an SVt_IV, is now more efficient. [GH #22927 <https://github.com/Perl/perl5/issues/22927>]
• String reversal from a single argument, when the string buffer is not "swiped", is now done in a single pass and is noticeably faster. The extent of the improvement is compiler & hardware dependent. [GH #23012 <https://github.com/Perl/perl5/issues/23012>]

The following additions or changes have been made to diagnostic output, including warnings and fatal error messages. For the complete list of diagnostic messages, see perldiag.

• Fix compilation on platforms (e.g. "Gentoo Prefix") with only a C locale [GH #22569 <https://github.com/Perl/perl5/issues/22569>] Bug first reported downstream bugs.gentoo.org/939014 <https://bugs.gentoo.org/939014>

• The (mostly undocumented) configuration macro PERL_STRICT_CR has been removed. When enabled (e.g. with ./Configure -A ccflags=-DPERL_STRICT_CR), it would make the perl parser throw a fatal error when it encountered a CR (carriage return) character in source files. The default (and now only) behavior of the perl parser is to strip CRs paired with newline characters and otherwise treat them as whitespace.

  (PERL_STRICT_CR was originally introduced in perl 5.005 to optionally restore backward compatibility with perl 5.004, which had made CR in source files an error. Before that, CR was accepted, but retained literally in quoted multi-line constructs such as here-documents, even at the end of a line.)

• Similarly, the (even less documented) configuration macro PERL_CR_FILTER has been removed. When enabled, it would install a default source filter to strip carriage returns from source code before the parser proper got to see it.

Tests were added and changed to reflect the other additions and changes in this release.  Furthermore, these significant changes were made:

• A new t/run/todo.t test script was added as a place for TODO tests for known unfixed bugs. Patches are welcome to add to this file.
• Added testing of the perl headers against the C++ compiler corresponding to the C compiler perl is being built with. [GH #22232 <https://github.com/Perl/perl5/issues/22232>]

• The "sv_strftime_ints" in perlapi function is introduced.  This is an enhanced version of "my_strftime" in perlapi, which is retained for backwards compatibility.  Both are to call strftime(3) when you have the year, month, hour, etc.  The new function handles UTF8ness for you, and allows you to specify if you want the possibility of daylight savings time to be considered.  my_strftime never considers DST.
• The bytes_to_utf8, bytes_from_utf8, and bytes_from_utf8_loc functions are no longer experimental.
• Calls to call_argv() with the G_DISCARD flag set also ensure the SV parameters constructed from the argv parameter are released before call_argv() returns. Previously they were released on the next FREETMPS. [GH #22255 <https://github.com/Perl/perl5/issues/22255>]

• When built with the -DDEBUGGING compile option, perl API functions that take pointers to distinct types of SVs (AVs, HVs or CVs) will check the SvTYPE() of the passed values to ensure they are valid.  Additionally, internal code within core functions that attempts to extract AVs, HVs or CVs from reference values passed in will also perform such checks.

  While this has been entirely tested by normal Perl CI testing, there may still be some corner-cases where these constraints are violated in otherwise-valid calls.  These may require further investigation if they are found, and specific code to be adjusted to account for it.

• The op_dump() function has been expanded to include additional information about the recent OP_METHSTART and OP_INITFIELD ops, as well as for OP_ARGCHECK and OP_ARGELEM which had not been done previously.
• op_dump() now also has the facility to print extra debugging information about custom operators, if those operators register a helper function via the new xop_dump element of the XOP structure. For more information, see the relevant additions to perlguts.

• New API functions are introduced to convert strings encoded in UTF-8 to their ordinal code point equivalent.  These are safe to use by default, and generally more convenient to use than the existing ones.

  "utf8_to_uv" in perlapi and "utf8_to_uv_or_die" in perlapi replace "utf8_to_uvchr" in perlapi (which is retained for backwards compatibility), but you should convert to use the new forms, as likely you aren't using the old one safely.

  To convert in the opposite direction, you can now use "uv_to_utf8" in perlapi.  This is not a new function, but a new synonym for "uvchr_to_utf8" in perlapi.  It is added so you don't have to learn two sets of names.

  There are also two new functions, "strict_utf8_to_uv" in perlapi and "c9strict_utf8_to_uv" in perlapi which do the same thing except when the input string represents a code point that Unicode doesn't accept as legal for interchange, using either the strict original definition (strict_utf8_to_uv), or the looser one given by Unicode Corrigendum #9 <https://www.unicode.org/versions/corrigendum9.html> (c9strict_utf8_to_uv).  When the input string represents one of the restricted code points, these functions return the Unicode REPLACEMENT CHARACTER instead.

  Also "extended_utf8_to_uv" in perlapi is a synonym for utf8_to_uv, for use when you want to emphasize that the entire range of Perl extended UTF-8 is acceptable.

  There are also replacement functions for the three more specialized conversion functions that you are unlikely to need to use.  Again, the old forms are kept for backwards compatibility, but you should convert to use the new forms.

  "utf8_to_uv_flags" in perlapi replaces "utf8n_to_uvchr" in perlapi.

  "utf8_to_uv_errors" in perlapi replaces "utf8n_to_uvchr_error" in perlapi.

  "utf8_to_uv_msgs" in perlapi replaces "utf8n_to_uvchr_msgs" in perlapi.

  Also added are the inverse functions "uv_to_utf8_flags" in perlapi and "uv_to_utf8_msgs" in perlapi, which are synonyms for the existing functions, "uvchr_to_utf8_flags" in perlapi and "uvchr_to_utf8_flags_msgs" in perlapi respectively.  These are provided only so you don't have to learn two sets of names.

• Three new API functions are introduced to convert strings encoded in UTF-8 to native bytes format (if possible).  These are easier to use than the existing ones, and they avoid unnecessary memory allocations. The functions are "utf8_to_bytes_overwrite" in perlapi which is used when it is ok for the input string to be overwritten with the converted result; and "utf8_to_bytes_new_pv" in perlapi and "utf8_to_bytes_temp_pv" in perlapi when the original string must be preserved intact.  utf8_to_bytes_temp_pv returns the result in a temporary using perlapi/SAVEFREEPV that will automatically be destroyed.  With utf8_to_bytes_new_pv, you are responsible for freeing the newly allocated memory that is returned if the conversion is successful.

  The latter two functions are designed to replace "bytes_from_utf8" in perlapi which creates memory unnecessarily, or unnecessarily large.

• New API functions valid_identifier_pve(), valid_identifier_pvn() and valid_identifier_sv() have been added, which test if a string would be considered by Perl to be a valid identifier name.
• When assigning from an SVt_IV into a SVt_NV (or vice versa), providing that both are "bodyless" types, Perl_sv_setsv_flags will now just change the destination type to match the source type. Previously, an SVt_IV would have been upgraded to a SVt_PVNV to store an NV, and an SVt_NV would have been upgraded to a SVt_PVIV to store an IV. This change prevents the need to allocate - and later free - the relevant body struct.
• Two new API functions are introduced to convert strings encoded in native bytes format to UTF-8.  These return the string unchanged if its UTF-8 representation is the same as the original.  Otherwise, new memory is allocated to contain the converted string.  This is in contrast to the existing "bytes_to_utf8" in perlapi which always allocates new memory.  The new functions are "bytes_to_utf8_free_me" in perlapi and "bytes_to_utf8_temp_pv" in perlapi. "bytes_to_utf8_temp_pv" in perlapi arranges for the new memory to automatically be freed.  With bytes_to_utf8_free_me, you are responsible for freeing any newly allocated memory.

• The way that subroutine signatures are parsed by the parser grammar has been changed.

  Previously, when parsing individual signature parameters, the parser would accumulate an OP_ARGELEM optree fragment for each parameter on the parser stack, collecting them in an OP_LIST sequence, before finally building the complete argument handling optree itself, in a large action block defined directly in perly.y.

  In the new approach, all the optree generation is handled by newly-defined functions in op.c which are called by the action blocks in the parser. These do not keep state on the parser stack, but instead in a dedicated memory structure referenced by the main PL_parser structure.  This is intended to be largely opaque to other code, and accessed only via the new functions.

  This new arrangement is intended to allow more flexible code generation and additional features to be developed in the future.

• Three new API functions have been added to interact with the regexp global match position stored in an SV.  These are sv_regex_global_pos_get(), sv_regex_global_pos_set() and sv_regex_global_pos_clear().  Using these API functions avoids XS modules needing to know about or interact directly with the way this position is currently stored, which involves the PERL_MAGIC_regex_global magic type.

• New SvVSTRING API macro

  A new API macro has been added, which is used to obtain the second string buffer out of a "vstring" SV, in a manner similar to the SvPV macro which obtains the regular string buffer out of a regular SV.

      STRLEN len;
      const char *vstr_pv = SvVSTRING(sv, vstr_len);

  See "SvVSTRING" in perlapi.

• Fix null pointer dereference in S_SvREFCNT_dec [GH #16627 <https://github.com/Perl/perl5/issues/16627>].
• Fix feature 'class' Segmentation fault in DESTROY [GH #22278 <https://github.com/Perl/perl5/issues/22278>].

• chdir now returns real booleans (as its documentation describes), not integers. This means the result of a failed chdir now stringifies to '', not '0'.

  [GH #22365 <https://github.com/Perl/perl5/issues/22365>]

• Compound assignment operators return lvalues that can be further modified:

      ($x &= $y) += $z;
      # Equivalent to:
      #  $x &= $y;
      #  $x += $z;

  However, the separate numeric/string bitwise operators provided by the bitwise feature, &= ^= |= &.= ^.= |.=, did not return such lvalues:

      use feature qw(bitwise);
      ($x &= $y) += $z;
      # Used to die:
      #  Can't modify numeric bitwise and (&) in addition (+) at ...

  This has been corrected. [GH #22412 <https://github.com/Perl/perl5/issues/22412>]

• Starting in v5.39.8, "strftime" in POSIX would crash or produce odd errors (such as Out of memory in perl:util:safesysmalloc) when given a format string that wasn't actually a string, but a number, undef, or an object (even one with overloaded string conversion).

  Now strftime stringifies its first argument, as before. [GH #22498 <https://github.com/Perl/perl5/issues/22498>]

  Also, fix POSIX::strftime() [GH #22369 <https://github.com/Perl/perl5/issues/22369>].

• pack("p", ...) and pack("P", ...) now SvPV_force() the supplied SV unless it is read only.  This will remove CoW from the SV and prevents code that writes through the generated pointer from modifying the value of other SVs that happen the share the same CoWed string buffer.

  Note: this does not make pack("p",... ) safe, if the SV is magical then any writes to the buffer will likely be discarded on the next read.  [GH #22380 <https://github.com/Perl/perl5/issues/22380>]

• Enforce no feature "bareword_filehandles" for bareword file handles that have strictness removed because they are used in open() with a "dup" mode, such as in open my $fh, ">&", THISHANDLE. [GH #22568 <https://github.com/Perl/perl5/issues/22568>]
• Using goto to tail call, or using the call_sv() and related APIs to call, any of trim(), refaddr(), reftype(), ceil(), floor() or stringify() in the builtin:: package would crash or assert due to a TARG handling bug. [GH #22542 <https://github.com/Perl/perl5/issues/22542>]
• Fix sv_gets() to accept a SSize_t append offset instead of I32. This prevents integer overflows when appending to a large SV for readpipe aka qx// and readline. <https://www.perlmonks.org/?node_id=11161665>
• Fixed an issue where utf8n_to_uvchr failed to correctly identify certain invalid UTF-8 sequences as invalid. Specifically, sequences that start with continuation bytes or unassigned bytes could cause unexpected behavior or a panic. This fix ensures that such invalid sequences are now properly detected and handled. This correction also resolves related issues in modules that handle UTF-8 processing, such as Encode.pm.
• The perl parser would erroneously parse some POD directives as if they were =cut. Some other POD directives whose names start with cut, prematurely terminating an embedded POD section. The following cases were affected: cut followed by a digit (e.g. =cut2studio), cut followed by an underscore (e.g. =cut_grass), and in string eval, any identifier starting with cut (e.g. =cute). [GH #22759 <https://github.com/Perl/perl5/issues/22759>]
• Builds with -msse and quadmath on 32-bit x86 systems would crash with a misaligned access early in the build. [GH #22577 <https://github.com/Perl/perl5/issues/22577>]
• On threaded builds on POSIX-like systems, if the perl signal handler receives a signal, we now resend the signal to the main perl thread.  Previously this would crash.  [GH #22487 <https://github.com/Perl/perl5/issues/22487>]
• Declaring a lexically scoped array or hash using state within a subroutine and then immediately returning no longer triggers a "Bizarre copy of HASH/ARRAY in subroutine exit" error. [GH #18630 <https://github.com/Perl/perl5/issues/18630>]
• builtin::trim() didn't properly clear TARG which could result in out of date cached numeric versions of the value being used on a second evaluation.  Properly clear any cached values. [GH #22784 <https://github.com/Perl/perl5/issues/22784>]
• "shmread" in perlfunc and "shmwrite" in perlfunc are no longer limited to 31-bit values and can use all the available bits on a platform for their POS and SIZE arguments. [GH #22895 <https://github.com/Perl/perl5/issues/22895>]
• "shmread" in perlfunc is now better behaved if VAR is not a plain string.  If VAR is a tied variable, it calls STORE once; previously, it would also call FETCH, but without using the result.  If VAR is a reference, the referenced entity has its refcount properly decremented when VAR is turned into a string; previously, it would leak memory. [GH #22898 <https://github.com/Perl/perl5/issues/22898>]
• The $SIG{__DIE__} and $SIG{__WARN__} handlers can no longer be invoked recursively, either deliberately or by accident, as described in "%SIG" in perlvar. That is, when an exception (or warning) triggers a call to a $SIG{__DIE__} (or $SIG{__WARN__}) handler, further exceptions (or warnings) are processed directly, ignoring %SIG until the original $SIG{__DIE__} (or $SIG{__WARN__}) handler call returns. [GH #14527 <https://github.com/Perl/perl5/issues/14527>], [GH #22984 <https://github.com/Perl/perl5/issues/22984>], [GH #22987 <https://github.com/Perl/perl5/issues/22987>]
• The ObjectFIELDS() for an object and xhv_class_fields for the object's stash weren't always NULL or not-NULL, confusing sv_dump() (and hence Devel::Peek's Dump()) into crashing on an object with no defined fields in some cases.  [GH #22959 <https://github.com/Perl/perl5/issues/22959>]
• When comparing strings when using a UTF-8 locale, the behavior was previously undefined if either or both contained an above-Unicode code point, such as 0x110000.  Now all such code points will collate the same as the highest Unicode code point, U+10FFFF. [GH #22989 <https://github.com/Perl/perl5/issues/22989>]
• In regexes, the contents of \g{...} backreferences are now properly validated. Previously, \g{1 FOO} was silently parsed as \g{1}, ignoring everything after the first number. [GH #23050 <https://github.com/Perl/perl5/issues/23050>]

• A run-time pattern which contained a code block which recursed back to the same bit of code which ran that match, could cause a crash. [GH #22869 <https://github.com/Perl/perl5/issues/22869>]

  For example:

      my $r = qr/... (?{ foo() if ... }) .../;
      sub foo { $string =~ $r }
      foo()

• In some cases an eval would not add integer parts to the source lines saved by the debugger. [GH #23151 <https://github.com/Perl/perl5/issues/23151>]
• In debugging mode, perl saves source lines from all files (plus an indication of whether each line is breakable) for use by the debugger. The internal storage format has been optimized to use less memory. This should save 24 bytes per stored line for 64-bit systems, more for -Duselongdouble or -Dusequadmath builds. Discussed in [GH #23171 <https://github.com/Perl/perl5/issues/23171>].
• Ensure cloning the save stack for fork emulation doesn't duplicate freeing the RExC state. [GH #23022 <https://github.com/Perl/perl5/issues/23022>]
• Smartmatch against a code reference that uses a loop exit such as last would crash perl. [GH #16608 <https://github.com/Perl/perl5/issues/16608>]
• Class initializers and ADJUST blocks, per perlclass, that called last or other loop exits would crash perl.  Same cause as for [GH #16608 <https://github.com/Perl/perl5/issues/16608>].

• Exceptions thrown and caught entirely within a defer {} or finally {} block no longer stop the outer run-loop.

  Code such as the following would stop running the contents of the defer block once the inner exception in the inner try/catch block was caught. This has now been fixed, and runs as expected. ([GH #23064 <https://github.com/Perl/perl5/issues/23064>]).

      defer {
          try { die "It breaks\n"; }
          catch ($e) { warn $e }
  
          say "This line would never run";
      }

• "readline" in perlfunc now clears the error flag if an error occurs when reading and that error is EAGAIN or EWOULDBLOCK.  This allows code that depended on readline to clear all errors to ignore these relatively harmless errors.  [GH #22883 <https://github.com/Perl/perl5/issues/22883>]

• open automatically creates an anonymous temporary file when passed undef as a filename:

      open(my $fh, "+>", undef) or die ...

  This is supposed to work only when the undefined value is the one returned by the undef function.

  In perls before 5.41.3, this caused a problem due to the fact that the same undefined value can be generated by lookups of non-existent hash keys or array elements, which can lead to bugs in user-level code (reported as [GH #22385 <https://github.com/Perl/perl5/issues/22385>]).

  In 5.41.3, additional checks based on the syntax tree of the call site were added, which fixed this issue for some number of common cases, though not all of them, at the cost of breaking the ability of APIs that wrap open to expose its anonymous file mode. A notable example of such an API is autodie.

  This release reverts to the old problem in preference to the new one for the time being.

Perl 5.42.0 represents approximately 13 months of development since Perl 5.40.0 and contains approximately 280,000 lines of changes across 1,600 files from 65 authors.

Excluding auto-generated files, documentation and release tools, there were approximately 94,000 lines of changes to 860 .pm, .t, .c and .h files.

Perl continues to flourish into its fourth decade thanks to a vibrant community of users and developers. The following people are known to have contributed the improvements that became Perl 5.42.0:

Aaron Dill, Andrei Horodniceanu, Andrew Ruthven, Antanas Vaitkus, Aristotle Pagaltzis, Branislav Zahradník, brian d foy, Chad Granum, Chris 'BinGOs' Williams, Craig A. Berry, Dabrien 'Dabe' Murphy, Dagfinn Ilmari Mannsåker, Dan Book, Daniel Dragan, Dan Jacobson, David Cantrell, David Mitchell, E. Choroba, Ed J, Ed Sabol, Elvin Aslanov, Eric Herman, Erik Huelsmann, Gianni Ceccarelli, Graham Knop, hbmaclean, H.Merijn Brand, iabyn, James E Keenan, James Raspass, Johan Vromans, Karen Etheridge, Karl Williamson, Leon Timmermans, Lukas Mai, Marek Rouchal, Marin Tsanov, Mark Fowler, Masahiro Honma, Max Maischein, Paul Evans, Paul Johnson, Paul Marquess, Peter Eisentraut, Peter John Acklam, Philippe Bruhat (BooK), pyrrhlin, Reini Urban, Richard Leach, Robert Rothenberg, Robin Ragged, Russ Allbery, Scott Baker, Sergei Zhmylev, Sevan Janiyan, Sisyphus, Štěpán Němec, Steve Hay, TAKAI Kousuke, Thibault Duponchelle, Todd Rinaldo, Tony Cook, Unicode Consortium, Vladimír Marek, Yves Orton.

The list above is almost certainly incomplete as it is automatically generated from version control history. In particular, it does not include the names of the (very much appreciated) contributors who reported issues to the Perl bug tracker.

Many of the changes included in this version originated in the CPAN modules included in Perl's core. We're grateful to the entire CPAN community for helping Perl to flourish.

For a more complete list of all of Perl's historical contributors, please see the AUTHORS file in the Perl source distribution.

If you find what you think is a bug, you might check the perl bug database at <https://github.com/Perl/perl5/issues>. There may also be information at <https://www.perl.org/>, the Perl Home Page.

If you believe you have an unreported bug, please open an issue at <https://github.com/Perl/perl5/issues>. Be sure to trim your bug down to a tiny but sufficient test case.

If the bug you are reporting has security implications which make it inappropriate to send to a public issue tracker, then see "Security VULNERABILITY CONTACT INFORMATION" in perlsec for details of how to report the issue.

If you wish to thank the Perl 5 Porters for the work we had done in Perl 5, you can do so by running the perlthanks program:

    perlthanks

This will send an email to the Perl 5 Porters list with your show of thanks.

The Changes file for an explanation of how to view exhaustive details on what changed.

The INSTALL file for how to build Perl.

The README file for general stuff.

The Artistic and Copying files for copyright information.