Post-mortem: Security Releases of this weekend

The intention of the task was to create 3 new releases all at once,
as a "surprise" because they contained security-related fixes. Those
fixes had been distributed as patches to vendors in the two weeks
prior, so this was the determined time to make it public. As such, it
was a little difficult to work too much out in the open, as would be
the case for a normal blead-point or maint release. Normal maintenance
policy said we should keep 5.38 and 5.36 updated, but we decided since
we're doing two releases anyway, adding a third didn't make it much
worse, so I was building all of 5.34.2, 5.36.2 and 5.38.1.

Since I was building three sets of tarballs all at once, I got myself a
large-CPU machine and used three separate "git worktree" directories,
one on each of the maint-5.34, maint-5.36 and maint-5.38 branches. This
meant I could build them all concurrently, rather than constantly
having to switch between all three branches. This saved much time.


A timeline of what steps I took:

  1. Prepared tarballs by following RMG, these were tested as RC1
     from a private server URL.

  2. Some of these needed some minor fixes to documentation, two were
     then tested as RC2.

  3. Now found to be OK, I prepared the real tarballs. These were then
     uploaded to PAUSE.

  4. The PAUSE indexer failed on these because they were not marked as
     "stable" according to the META.json file inside them.

  5. I rebuilt the tarballs with updated META.json

  6. Because PAUSE doesn't let you create the same file again,
     these were uploaded with "a" suffix on the filename.

  7. The tests in these were still failing; specifically
     t/porting/regen.t

At this point I decided I wasn't going to have a third attempt, as I
was not in a good mental state to continue. I took a step back to have
another look the next day.

After some poking around the weekend we eventually came to the
conclusion to abandon those version numbers and rebuild the next
numbers - 5.34.3, 5.36.3, 5.38.2.

What went wrong:

  1. First time, I forgot to rebuild META.json - because it's a
     separate manual step in the RMG, somewhat hard to notice because
     of the wording of the RMG. Also I wasn't really thinking it would
     be a discrete step, since when using more regular tools for
     building CPAN modules ("./Build dist" or "make dist") that would
     be included there anyway.

  2. I rebuilt the META.json file by running "perl" not "./perl" so it
     wrote the wrong version number of JSON::PP into the file. This
     wouldn't normally really matter, except for the fact that
     t/porting/regen.t demands a byte-for-byte perfect copy to be
     recreated.

  3. Also Module::CoreList was wrong in several ways; having missed the
     update of itself in any of the releases, and also declaring
     information about the upcoming 5.39.6 release that didn't even
     exist at that point.


Extenuating factors:

  *  Many porting scripts differ in behaviour. Some work in
     workdirs, some do not. Some require `git clean -dxf`, some require
     the built perl

  *  Some porting scripts *require* the built perl but do not complain
     if not run using it, and therefore break in subtle ways

  *  Module::CoreList continues not to have any tooling around updating
     it, especially for nontrivial setups like this

  *  CI tests did in fact pick up on these issues, but I didn't notice,
     because CI tests *always* fail for various spurious reasons. I
     have taken to ignoring any failure notifications. 
     Example: i386 never works - 
              https://github.com/Perl/perl-security/actions/runs/7020092773/job/19099577640
              Win32 never works -
              https://github.com/Perl/perl-security/actions/runs/7020092773/job/19099579131


Suggestions to fix avoid in future

  *  Fix up many of the scripts to work in git worktrees

  *  Early bailout if run using systemperl for those scripts that must
     be run on the built perl

  *  Tooling to help automate the Module::CoreList process. At the very
     least its own module should always be updated in every block; as
     should B::Opcode and Config.

  *  Further consider if we need to keep the "next upcoming release" in
     blead's Module/CoreList.pm at all; as it complicates
     out-of-sequence maint releases in this process

  *  Streamlining of the RMG to reduce the number of steps overall,
     combine more testing phases, less build/clean/build/clean...

  *  Also update the wording / formatting of the RMG, making it much
     easier to see which steps apply to which kinds of release

  *  Fix up / skip / ignore the always-failing or flaky CI tests, so
     that failure messages become useful signals and not noise.

-- 
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk      |  https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/  |  https://www.tindie.com/stores/leonerd/

• Post-mortem: Security Releases of this weekend by Paul "LeoNerd" Evans
• Re: Post-mortem: Security Releases of this weekend by sisyphus
• Re: Post-mortem: Security Releases of this weekend by James E Keenan
• Re: Post-mortem: Security Releases of this weekend by Sam James