Moving the site to MediaWiki 1.43

[Dave Pattern]

This has been a long time coming, partly due to me having to delay the planned move from 1.35 to 1.39 for personal reasons, but we’re now well on the way to upgrading the whole site to the current LTS (Long Term Support) release of MediaWiki which is version 1.43 (released on 21 December 2024).

Normally the MediaWiki upgrades are straightforward, but the live site has a several issues:

• the underlying OS (Ubuntu 20.04) is only supported until April 2025 so needs to be replaced
• the “skin” used by the site is a custom one which is incompatible with more recent versions of MediaWiki
• PHP stuck on an old unsupported version due to the use of MediaWiki 1.35

I decided the best approach to the above was to migrate and upgrade the site onto new servers, and for that to be completed before April 2025 using a phased approach:

• phase 1
  1. set up a new development environment [✓]
  2. install latest version of MediaWiki and carry out test database migration and updates [✓]
  3. evaluate alternatives to the custom skin used on the live site [✓]
  4. migrate and improve the bespoke APIs which provide extra functionality to the site (e.g. maps, image archive, embedded books, etc) [✓]
  5. improve accessibility (aim for WCAG 2.2 AA if possible) and also usability on mobile devices [ongoing]
  6. beta test the site and resolve any “snagging” issues [ongoing]
  7. final database migration so that the development environment matches the live site [✓]
  8. point DNS at development environment so that it becomes the “live” site to allow user testing [✓]
• phase 2
  1. deploy new servers [✓]
  2. migrate development environment [✓]
  3. test the new site [ongoing]
  4. point DNS at new live environment [✓]

The move from phase 1 to 2 will involve locking the site down so that page editing by users is disabled. Editing will be re-enabled once phase 2 is completed.

[Dave Pattern]

Phase 1 is now nearly complete…

• the new environment includes:
  • ElasticSearch 7.17.26 (the most recent version currently supported by CirrusSearch)
  • PHP 8.4.x
  • https://beta.huddersfield.exposed/wiki/Special:Version
• the site’s “skin” has been migrated to Chameleon
• improvements have been made to the bespoke APIs, including improved caching to increase performance

Current known issues:

• image rotation option still to do on archive pages (example) still to do [✓]
• improvement to CSS for smaller screens and mobile devices
• site search currently only searching wiki pages — bento style with images and book content matches still to do
• possible issues with known items & relevancy ranking in ElasticSearch (most testing needed)
• newspaper OCR project
• book reader in new window

[Dave Pattern]

Some of the revised API code has been deployed to the new API server (“amy”) and the live site is now making use of the following API endpoints:

• /api/content/aflink/
• /api/content/books/
• /api/content/buddypress/
• /api/content/maplink/
• /api/content/mapzoom/
• /api/content/mediawiki/
• /api/content/newspapers/
• /api/content/omeka/
• /api/content/page/
• /api/content/rollofhonour/
• /api/locations/
• /api/content/security/
• /api/tiles/
• /api/content/wpsecurity/

…still to do (italics = tied to the new MW server):

• /api/content/featured/
• /api/content/gallery/

[Dave Pattern]

The live site (running on MW 1.35) is now set to “read only” so no further edits can be made.

The replacement servers are currently being configured and deployed. Unfortunately the MW 1.43 update script initially failed (due to issues with database schema changes that are incompatible with Galera’s strict primary key enforcement) so I had to adjust some of the SQL changes by hand. The update script now appears to have completed successfully:

Your composer.lock file is up to date with current dependencies!
Going to run database updates for XXXXXXXXX
Depending on the size of your database this may take a while!
…collations up-to-date.
…rd_title in table redirect already modified by patch patch-redirect-rd_title-varbinary.sql.
…pl_title field does not exist in pagelinks table, skipping modify field patch.
…tl_title field does not exist in templatelinks table, skipping modify field patch.
…il_to in table imagelinks already modified by patch patch-imagelinks-il_to-varbinary.sql.
…ll_title in table langlinks already modified by patch patch-langlinks-ll_title-varbinary.sql.
…iwl_title in table iwlinks already modified by patch patch-iwlinks-iwl_title-varbinary.sql.
…cat_title in table category already modified by patch patch-category-cat_title-varbinary.sql.
…qc_title in table querycache already modified by patch patch-querycache-qc_title-varbinary.sql.
…qcc_title in table querycachetwo already modified by patch patch-querycachetwo-qcc_title-varbinary.sql.
…wl_title in table watchlist already modified by patch patch-watchlist-wl_title-varbinary.sql.
…user_last_timestamp in table user_newtalk already modified by patch patch-user_newtalk-user_last_timestamp-binary.sql.
…pt_title in table protected_titles already modified by patch patch-protected_titles-pt_title-varbinary.sql.
…ir_type in table ipblocks_restrictions already modified by patch patch-ipblocks_restrictions-ir_type.sql.
…index wl_namespace_title already set on watchlist table.
…job_title in table job already modified by patch patch-job-job_title-varbinary.sql.
…job_timestamp in table job already modified by patch patch-job_job_timestamp.sql.
…job_token_timestamp in table job already modified by patch patch-job_job_token_timestamp.sql.
…wl_notificationtimestamp in table watchlist already modified by patch patch-watchlist-wl_notificationtimestamp.sql.
…role_id in table slot_roles already modified by patch patch-slot_roles-role_id.sql.
…model_id in table content_models already modified by patch patch-content_models-model_id.sql.
…cl_to in table categorylinks already modified by patch patch-categorylinks-cl_to-varbinary.sql.
…log_title in table logging already modified by patch patch-logging-log_title-varbinary.sql.
…us_timestamp in table uploadstash already modified by patch patch-uploadstash-us_timestamp.sql.
…index up_property already set on user_properties table.
…index site_global_key already set on sites table.
…index log_type_time already set on logging table.
…fa_name in table filearchive already modified by patch patch-filearchive-fa_name.sql.
…oi_name in table oldimage already modified by patch patch-oldimage-oi_name-varbinary.sql.
…exptime in table objectcache already modified by patch patch-objectcache-exptime-notnull.sql.
…index ar_name_title_timestamp already set on archive table.
…img_name in table image already modified by patch patch-image-img_name-varbinary.sql.
…img_timestamp in table image already modified by patch patch-image-img_timestamp.sql.
…index si_key already set on site_identifiers table.
…rc_title in table recentchanges already modified by patch patch-recentchanges-rc_title-varbinary.sql.
…rc_timestamp in table recentchanges already modified by patch patch-recentchanges-rc_timestamp.sql.
…rc_id in table recentchanges already modified by patch patch-recentchanges-rc_id.sql.
…index rc_new_name_timestamp already set on recentchanges table.
…ar_title in table archive already modified by patch patch-archive-ar_title-varbinary.sql.
…page_title in table page already modified by patch patch-page-page_title-varbinary.sql.
…user_name in table user already modified by patch patch-user_table-updates.sql.
…index rev_page_timestamp already set on revision table.
…have modtoken field in objectcache table.
…index oi_timestamp already set on oldimage table.
…index page_name_title already set on page table.
…index ct_rc_tag_id already set on change_tag table.
…page_restrictions table does not contain pr_user field.
…fa_id in table filearchive already modified by patch patch-filearchive-fa_id.sql.
…img_major_mime in table image already modified by patch patch-image-img_major_mime-default.sql.
…linktarget table already exists.
…rev_page_id key doesn’t exist.
…pr_page in table page_restrictions already modified by patch patch-page_restrictions-pr_page.sql.
…pp_page in table page_props already modified by patch patch-page_props-pp_page.sql.
…ir_value in table ipblocks_restrictions already modified by patch patch-ipblocks_restrictions-ir_value.sql.
…have tl_target_id field in templatelinks table.
…user_autocreate_serial table already exists.
…ir_ipb_id in table ipblocks_restrictions already modified by patch patch-ipblocks_restrictions-ir_ipb_id.sql.
…ipblocks table does not exist, skipping modify field patch.
…user_editcount in table user already modified by patch patch-user-user_editcount.sql.
Running MigrateRevisionActorTemp…
…Update ‘MigrateRevisionActorTemp’ already logged as completed. Use –force to run it again.
done.
…revision_actor_temp doesn’t exist.
Running UpdateRestrictions…
Migration is not needed.
done.
…page table does not contain page_restrictions field.
…templatelinks table has already been migrated.
…tl_namespace field does not exist in templatelinks table, skipping modify field patch.
…templatelinks table does not contain tl_title field.
…have el_to_path field in externallinks table.
…have user_is_temp field in user table.
Running MigrateRevisionCommentTemp…
…Update ‘MigrateRevisionCommentTemp’ already logged as completed. Use –force to run it again.
done.
…revision_comment_temp doesn’t exist.
Running MigrateExternallinks…
…Update ‘MigrateExternallinks’ already logged as completed. Use –force to run it again.
done.
…el_to field does not exist in externallinks table, skipping modify field patch.
…have pl_target_id field in pagelinks table.
…externallinks table does not contain el_to field.
Running FixInconsistentRedirects…
…Update ‘FixInconsistentRedirects’ already logged as completed. Use –force to run it again.
done.
…img_size in table image already modified by patch patch-image-img_size_to_bigint.sql.
…fa_size in table filearchive already modified by patch patch-filearchive-fa_size_to_bigint.sql.
…oi_size in table oldimage already modified by patch patch-oldimage-oi_size_to_bigint.sql.
…us_size in table uploadstash already modified by patch patch-uploadstash-us_size_to_bigint.sql.
…have uas_year field in user_autocreate_serial table.
…block_target table already exists.
…cl_collation_ext key doesn’t exist.
Running PopulateUserIsTemp…
…Update ‘PopulateUserIsTemp’ already logged as completed. Use –force to run it again.
done.
…site_type key doesn’t exist.
…iwl_prefix_from_title key doesn’t exist.
…pagelinks table has already been migrated.
…rev_id in table revision already modified by patch patch-revision-cleanup.sql.
…rc_id in table recentchanges already modified by patch patch-recentchanges-rc_id-bigint.sql.
…ct_rc_id in table change_tag already modified by patch patch-change_tag-ct_rc_id.sql.
Running MigrateBlocks…
…Update ‘MigrateBlocks’ already logged as completed. Use –force to run it again.
done.
…ipblocks doesn’t exist.
…pagelinks table does not contain pl_title field.
…page_links_updated in table page already modified by patch patch-page-page_links_updated-noinfinite.sql.
…searchindex table has already been migrated.
…site_stats is populated…done.
…Update ‘cleanup empty categories’ already logged as completed. Use –force to run it again.
…Update ‘MediaWiki\Maintenance\FixAutoblockLogTitles’ already logged as completed. Use –force to run it again.
Set the local repo temp zone container to be private.
Purging caches…done.

The new ElasticSearch/CirrusSearch server is up & running and is successfully providing search results.

Note to self — always check the MySQL bin logs size after running updates as one of the database servers nearly ran out of disk space!!

[Dave Pattern]

The replacement remote replication database server is now running. A change to the previous setup is that this server is used for the ElasticSearch/CirrusSearch full reindex which now happens daily at 2:30am (which means there’s no load on the live database & Redis servers).

Both of the new replacement MediaWiki servers (PHP-FPM and Apache/ModPerl API) are up & running and are currently being tested.

There’s still quite a bit of work to do in the background, so realistically I’m not expecting to do the final switchover to the new servers until later on in February. Until then, the live site remains “read only”.

[Dave Pattern]

Ofcom have started to publish Online Safety Act advice for small community sites, so I’ll digest that and decide what changes to the site are required.

[Dave Pattern]

Been a bit too busy for updates, but the site is now live on the replacement servers. Today’s work:

• ✓ migrated the Roll of Honour location code which parses the pages for individuals to extract location data, etc
• ✓ migrated the Omeka code so that multimedia updates now sync with the replacement servers

[Author]

Posts