Repository navigation
Conversation
…blished projects.
… Reimplemented the schema.org JSON-LD format and Dublin Core meta tag elements only.
…pulation. Added context attributes needed for proper ML processing of tag data.
…tandards compliant for Croissant and Google Dataset Search SEO.
… optimization. Will cause JSON-LD block to fail if no distribution is found for project data.
…RLs for landing page and identifier. Logs better errors on misconfig.
… project descriptions. Limit to searchable length for crawlers.
…oissant spec requirements. No actual hashes exist yet, TBD somewhere else in the codebase.
…ss the JSON-LD data.
… for page optimization.
…uded open graph and twitter card link metadata for sharing best practices.
…out a license on portals whose form requires one. It follows the license field's own "required" flag in the portal's project form (EDIT_PROJECT_ADDON in the deployment's settings_forms.py). DPMP's form already marks it required, so DPMP is covered with no deployment change. Portals whose form doesn't require a license publish without one, as before. What changed: - publications/utils.py: a new is_publication_license_required() reads that flag. - publications/views.py: a new _require_license() returns a 400 with "License is required", the same text as the client's message. It runs in three places, before any workspace is created, any task is queued or any DOI is minted: - submitting for review: checks the project's own license, before the author list is saved; - approving a publish: checks the review project's license, since that's the metadata that gets published; - approving a republish: the same check. - If the project record can't be found, the check steps aside and lets the view handle it as it already does. - A blank or whitespace-only license counts as missing, and each refusal is logged. Tests: - utils_unit_test.py: the flag check across 7 form setups, including no form config at all and a license required only on a different form. - views_unit_test.py, for each of the three endpoints: - a missing, empty or whitespace-only license gets a 400, and nothing further runs; - a project with a license goes through; - a portal that doesn't require one publishes without it, and on publish and republish doesn't even read the project for it; - a missing project isn't turned into "License is required".
…, so it can no longer cause the DOI publish to be queued again. Problem publish_project_callback runs once a publication's files have finished transferring. It used to submit the archive job (ZIP, checksum manifests and Ranch copy) itself, as its last step. If Tapis rejected that submission, the whole callback was retried by poll_tapis_file_transfer, re-running every earlier step: - the Publication row was switched again and search re-indexed; - publish_publication_doi was queued again, costing another read and write to DataCite; - if only queueing the checksum poll had failed, a second archive job was submitted for the same version. The repeated DOI publish was harmless, since publishing an already-findable DOI changes nothing, but every retry repeated the work above. Changes server/portal/apps/projects/workspace_operations/project_publish_operations.py - New Celery task, submit_publication_archive_job(archive_project_id, project_id=None, version=None), with max_retries=5: - It submits the archive job, then queues poll_publication_archive_job to load the checksums the job writes. - A failed submission is retried with growing waits (1, 2, 4, 8, 16 minutes), so the 5 retries span about half an hour. - When retries run out, it logs an error saying the ZIP, manifests and Ranch copy weren't made, how to resubmit from a Django shell, and the compute_publication_checksums --load-only command to run afterwards. - If the job was submitted but the checksum poll couldn't be queued, it logs the --load-only command and doesn't retry, since a retry would submit a second job. - Transfer polls queued before the checksum step existed have no project_id. They still get their archive job, without the checksum poll, as before. - publish_project_callback: queues the new task instead of submitting the job and queueing the poll itself. Now the only thing after the DOI publish is queued is putting a message on the broker. server/portal/apps/projects/workspace_operations/project_publish_operations_unit_test.py - The 7 callback tests that expected direct calls now expect the task to be queued with the right arguments, and that the archive job isn't submitted inside the callback. - 5 new tests for the task cover: - submitting the job and queueing the checksum poll; - skipping the poll without a project_id; - retrying with backoff without queueing the poll; - giving up at the retry limit and logging how to recover; - never resubmitting when the poll can't be queued. Not changed - poll_tapis_file_transfer still has no explicit max_retries, so it keeps Celery's default of 3. Errors earlier in the callback still retry the whole callback, which is safe because the DOI publish is only queued once those steps succeed. - compute_publication_checksums calls archive_publication_files directly and isn't affected. Testing - ruff check and ruff format --check pass. - pytest: 1134 passed, 34 skipped, 0 failed. - project_publish_operations.py has 100% line and branch coverage. Deployment notes - Deploy the portal and its Celery workers together, as usual. A callback queues the new task by name, so a worker running older code wouldn't recognize it.
…rver doesn't enforce the license. It now says DPMP enforces it in the wizard and on the server (publications/views.py). License labels comment (license_urls.py): "Core-Portal's own settings_forms.py" now reads "DPMP's settings_forms.py". Client license check (DataFilesProjectPublish.jsx): it now trims the value (values.license?.trim()), so a license of only spaces fails here as it does on the server. I added a whitespace case to its test. Cover image in the JSON-LD (schema_org.py): - The Dataset now includes image, the same cover-image URL used for og:image. - og:image now reads that same value, so the two can't disagree. - Two new tests cover it, and a one-off check confirmed mlcroissant still reports no errors or warnings with image present. Versions newer than the current one now return 404 (views.py): - Before, a URL like …v99 301'd to the current version. - The check runs whether or not the publication is withdrawn. - I had to let the 404 pass through the view's catch-all error handler, which would otherwise have swallowed it. - The existing warning test now uses an older version (v2), which still 301s, and a new test covers the 404. - I didn't do the optional part (a 404 for other unknown paths under /published-datasets/). Other portals' public-data browser and the DPMP client rely on the catch-all page, so that change is riskier than the rest. Long file lists on the landing page (schema_org.py): - A new constant, LANDING_PAGE_MAX_FILE_OBJECTS = 100, caps the per-file distribution entries in the landing page's JSON-LD. I picked 100 as a default; change it if your datasets usually have more files. - The whole-dataset ZIP is always listed after them, and croissant.json still lists every file. - The Croissant conformance claim is decided before the cap, so it doesn't change. Any recordSet whose file was cut is dropped, so nothing points at a missing entry. - Four new tests cover the cap, and the Croissant view's docstring now mentions it. I updated CHANGELOG.md to mention image, the 100-file cap and the version 404, and the get_cover_image_url docstring in links.py to mention the JSON-LD image.
Conflict in datacite_operations.py: kept this branch's creator handling, which already adds ORCID nameIdentifiers (normalized and checksum-validated), and dropped main's institutions.append(...), whose list this branch removed. projects/views_unit_test.py: dropped main's "orcid_id": None fixture keys, which duplicated the branch's existing orcid_id keys (ruff F601). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main's DPM ORCID integration (#1391) lets authors add an ORCID in the publish form (AddOrcidModal). This branch's DataCite code (get_datacite_json) cleans up and checks each ORCID before sending it. Before this change, no test used the exact values the modal saves. This test does, so a change on either side that breaks the hand-off shows up as a failing test. Changes - server/portal/apps/projects/workspace_operations/datacite_operations_unit_test.py: new test_get_datacite_json_orcid_from_add_orcid_modal. Each author is saved the way the form saves it after the modal: the member's username, first and last name and email, plus an ORCID matching the modal's format check (^\d{4}-\d{4}-\d{4}-\d{3}[0-9X]$, case-insensitive). ┌───────────────────────────────────────────────────────┬───────────────────────────────────────┐ │ ORCID saved by the modal │ Sent to DataCite │ ├───────────────────────────────────────────────────────┼───────────────────────────────────────┤ │ 0000-0002-1825-0097 │ https://orcid.org/0000-0002-1825-0097 │ ├───────────────────────────────────────────────────────┼───────────────────────────────────────┤ │ 0000-0002-9079-593x (lowercase x) │ https://orcid.org/0000-0002-9079-593X │ ├───────────────────────────────────────────────────────┼───────────────────────────────────────┤ │ 0000-0002-1825-0098 (right format, wrong check digit) │ not sent │ └───────────────────────────────────────────────────────┴───────────────────────────────────────┘ Known remaining issue: the modal only checks the format, not ORCID's check digit. So a mistyped ORCID is shown in the form, but the server leaves it out of DataCite and the landing page's JSON-LD without telling the user. A follow-up could check the check digit in the modal.
…e it's being entered. A mistyped iD gets an error and can't be saved, instead of being saved and then silently left out of DataCite and the landing page. What changed in AddOrcidModal.jsx: - A new hasValidOrcidChecksum function computes ORCID's check digit with the same algorithm the server uses in projects/schema_models/orcid.py. A lowercase x is accepted, as before. - A new validation rule uses it to reject an iD in the right format but with a wrong last character. The message is "This ORCID ID has a typo: its last character does not match the rest". - The rule only runs once the format check passes, so a half-typed iD shows only the existing format error, not both. - It runs while the person types, as the existing format check does, so once they've left the field the error updates live. - hasValidOrcidChecksum and validationSchema are now exported so the tests can use them. The component itself is unchanged, and it still saves exactly what was typed; the server makes it uppercase. New test file, AddOrcidModal.test.jsx (the modal had no tests before; 12 tests): - The check-digit function accepts two valid iDs, including one ending in X or x, and rejects wrong check digits and empty values. - The validation rules give the typo message for a wrong check digit, and only the format message for a malformed iD. - In the rendered modal, entering 0000-0002-1825-0098 shows the typo error after leaving the field, and Save doesn't call onSave. Entering 0000-0002-9079-593x saves it.
…PP_SHELL
server/portal/templates/base.html
- Description: {% block meta_description %} now sits inside {% if canonical_url or settings.PORTAL_NOINDEX_APP_SHELL %}, the same check as the share-card block. In every other case the page gets main's plain <meta name="description" content="{{ site.description }}">.
- Canonical link: it uses the landing page's own URL when there is one. Otherwise, if the flag is set, it uses the absolute URL without the query string. If not, it uses main's {{request.get_full_path}}.
server/portal/apps/public_data/views_unit_test.py
- test_index_view_app_shell_canonical_url_drops_query_string no longer runs with the flag both on and off; it now only checks the flag-on case.
- New: test_index_view_app_shell_keeps_default_canonical_unless_portal_opts_in checks that with the flag off, the canonical link is main's relative path, query string included.
- New: test_index_view_landing_page_keeps_description_and_canonical_without_portal_opt_in checks that with the flag off, a landing page still gets its absolute canonical URL and the dataset's own description.
taoteg
force-pushed
the
task/WC-494-DPMP-SEO-License-Creator
branch
from
October 13, 2026 23:33
46e7c35 to
7a07308
Compare
…bmission attempts and errors. The fix is in submit_publication_archive_job (project_publish_operations.py). When PORTAL_PUBLICATION_ARCHIVE_APP_ID is empty, it now returns right away and logs: `▎ Not archiving <id>: PORTAL_PUBLICATION_ARCHIVE_APP_ID isn't set, so its ZIP, checksum manifests and Ranch copy won't be made.` It doesn't submit a job, doesn't retry and doesn't queue the checksum poll. The publish callback still queues the task exactly as before, so the rest of local publishing (DOI, switching versions, marking pages indexable) is unaffected. This was done in the code and not in the local settings because the local settings point at pprd's published storage. Setting a real archive app id there would make local publishes run real archive jobs and Ranch transfers against pprd's data. Tests (project_publish_operations_unit_test.py): - New: test_submit_publication_archive_job_skips_without_archive_app checks that with no app id nothing is submitted, retried or polled, and that the log line appears. - The five existing submit tests now set an app id through a small archive_app fixture. The test settings leave it empty, so without that the new check would skip the code those tests exercise. Effect elsewhere: - pprd and prod set the app id, so nothing changes there. - Other portals never reach this code because they don't publish. - compute_publication_checksums calls the archive function directly, not through this task, so it still refuses with a clear error when the app isn't configured.i
… left out instead of showing up as null or "None". What changed: - schema_org.py: - A new get_publisher_name() returns PORTAL_PUBLICATION_PUBLISHER with surrounding spaces trimmed. It returns None when the setting is unset, empty or only spaces. - A small _named_node() helper builds the publisher and includedInDataCatalog entries. When there's no name, it drops the name key and keeps @type and url, so those entries stay valid. - views.py: the landing-page context gets its publisher value from get_publisher_name(). - index_base.html: the DC.publisher and citation_publisher tags are only written when there is a publisher. The page title already falls back to PORTAL_NAMESPACE through |default:, so no change there. - views_unit_test.py: 7 new tests: - The JSON-LD and the rendered page, each checked with the setting as None, empty, and only spaces. - A test that surrounding spaces are trimmed from the name. The APA and BibTeX citations already skipped an unset publisher.
…9 bytes renders as "1.0 MB" instead of "1000.0 KB". The fix is in _format_content_size (server/portal/apps/public_data/schema_org.py). - It now compares the rounded value against 1000 instead of the exact one, so "1000.0" of a unit can't appear. - Byte counts are still shown as whole numbers, and terabytes is still the largest unit, so 1000⁵ bytes still shows as "1000.0 TB". - I also added a short note about this to the docstring.
taoteg
marked this pull request as ready for review
October 14, 2026 17:13
…l publishing steps are completed succesfully. - Writing and deleting the file: project_publish_operations.py writes the publication's Croissant metadata to archive/<published workspace id>/croissant.json on corral, beside the ZIP. It's the same document the portal's croissant.json route serves, and its @id and url are the landing page. The file is written only when the publication is published and indexable, every file has a checksum, and the web.corral mirror is configured. - When it's written: after the archive checksums load and after the page is marked indexable. Those two can happen in either order, and the write is skipped until both have, so whichever comes second writes the file. - Republishing: switching to a new version deletes the previous version's copy, because its checksums would no longer match the live files. - Link to the web.corral copy: the landing page JSON-LD and the croissant.json route include a subjectOf entry pointing at it. The landing page <head> also gets a second <link rel="alternate">. Both appear only under the same conditions the file is written. - Request-less URLs: in links.py, get_publication_origin(None) falls back to VANITY_BASE_URL so URLs can be built outside a web request. The web.corral URL helper moved there from views.py. - update_datacite_metadata rewrites the file for each published publication, including ones without a DOI. --dry-run writes nothing. - withdraw_publication deletes the file, even if DataCite rejects the hide. --restore writes it again after marking the page indexable. Failures in either command are reported in the error message. - Changelog: CHANGELOG.md has entries for all of this. Changes to existing tests: - The existing publish-operations and withdraw tests now patch the new Celery and Tapis calls. - Celery doesn't run tasks inline in the test settings, so without the patches those tests try to reach the message broker. Note: - subjectOf is added the moment the conditions hold, but the write is queued. A landing page can briefly link to the web.corral copy before it exists, or for longer if the write is retrying. Fixing that would need a new field and migration recording that the file was written; I didn't add that yet.
…lisher names, enforce date type consistency and make published-datasets indexable by default.
Prod DOIs on pprd (datacite_operations.py, project_publish_operations.py, both management commands)
- New is_doi_under_shoulder(doi) checks whether a DOI's prefix is PORTAL_PUBLICATION_DATACITE_SHOULDER, ignoring case. If no shoulder is configured it always says yes, so nothing changes in that case.
- When a DOI is under a different prefix, the portal no longer calls DataCite for it:
- publish_project (under DEBUG) reuses the DOI without updating its draft.
- publish_publication_doi skips DataCite and marks the page indexable, as it already does under DEBUG, instead of retrying a call that can't succeed. pprd pages still carry the noindex header, so search engines won't index them.
- update_datacite_metadata and withdraw_publication report the DOI as skipped. A --restore still marks the page indexable.
Blank publisher
- get_publisher_name() moved to publications/utils.py so DataCite and the landing page share it.
- get_datacite_json now raises a clear ValueError when the publisher is unset or blank. DataCite would otherwise reject it.
- Small behaviour change: the publisher DataCite receives is now trimmed of surrounding spaces.
Coverage
- New tests cover the "gave up" path of write_publication_croissant and the retry in delete_publication_croissant. project_publish_operations.py is now at 100% coverage.
datePublished is date-only
- get_publication_date() in publications/utils.py is now used for both DataCite's Issued date and the landing page's datePublished, so the two can't disagree.
- A stored value that isn't an ISO date is still emitted as-is and logged. Dropping it would also remove the Croissant conformsTo claim.
Indexable catalog page
- On DPMP, /published-datasets/ is now index, follow when there's at least one published dataset.
- It gets its own title, description and share tags ("Published datasets | <publisher>"), and a canonical URL with no query string.
- The page body has a server-rendered list of every published, indexable dataset, newest first, linking to each landing page. The client app replaces it once JavaScript runs, the same way landing pages work.
- It reads only the title field from each publication rather than its whole metadata.
- It stays the noindex app shell in these cases:
- there are no datasets;
- any other path, including /public-data/ and entity pages;
- any portal whose landing pages are canonical under /public-data/.
Test changes worth knowing about
- DataCite test module: I added a fixture that sets a publisher for every test. Before, they all sent "publisher": null.
- Two app-shell tests in views_unit_test.py were requesting /published-datasets/ while a publication existed, which is now the catalog. They now request /published-datasets/not-a-real-project/, so they still test the app shell.
…and the DPMP Browse page. DC.date shows the date only (public_data/schema_org.py) - DC.date used to show the raw stored timestamp (e.g. 2024-05-01T12:30:00). It now uses the YYYY-MM-DD value already worked out for the JSON-LD datePublished. - It now matches datePublished, citation_publication_date and DataCite's Issued date. Citations read the publisher through get_publisher_name() (public_data/citations.py) - get_apa_citation and get_bibtex_citation read PORTAL_PUBLICATION_PUBLISHER directly. They now use the shared helper, which trims the name and returns None when it's blank, as the JSON-LD and meta tags already do. - A padded name like " DPMP " no longer leaves extra spaces in the visible citation or the citeAs BibTeX, and a blank one is left out. - The now-unused settings import is removed. DPMP Browse page hides first publishes that are still in progress (publications/views.py, PublishedDatasetsBrowse.jsx) - GET /api/publications now returns is_indexable for each publication. - PublishedDatasetsBrowse leaves out entries where is_indexable === false, matching the sitemap and the server-rendered catalog. Entries without the flag are still shown. - The flag is applied in the client, not as a server-side filter, because the same listing feeds the workbench's DataFilesPublicationsList. Authors and reviewers still see their in-progress publications there. Tests - public_data/views_unit_test.py: - The DC.date test now also covers a stored timestamp. - New test: the APA and BibTeX citations trim a padded publisher and leave out a blank one. - publications/views_unit_test.py: new test that the listing returns is_indexable for each publication. - New PublishedDatasetsBrowse.test.jsx: hides an in-progress dataset, and shows live and unflagged ones. Impact - Only DPMP renders these landing pages and the Browse page. - For other portals the only change is the extra is_indexable field in /api/publications, which nothing else reads. - No migrations or settings changes.
DPMP's workbench and other non-dataset pages now use Workbench for both the browser tab title and the link-preview title (og:title).
Before, the preview title was DRP Workbench. It now matches every other portal.
Changes:
- base.html:68: the default preview title is now Workbench instead of {{ settings.PORTAL_NAMESPACE }} Workbench. Only dataset pages and portals with PORTAL_NOINDEX_APP_SHELL output this tag, so DPMP is the only portal this changes.
- views_unit_test.py: test_index_view_names_publisher_on_landing_pages_only once again requires the page title and the preview title to be equal, now both Workbench. I updated its docstring to match.
Dataset pages and the catalog page are unchanged: they still use {title} | {publisher} and Published datasets | {publisher}. The site name in previews (og:site_name) still shows the portal name, DRP. That doesn't affect search, which never indexes these pages.
Changes in PublishedDatasetsBrowse.jsx: - Unused error: removed from the useSelector destructure. - Missing selectedSystem?.system dependency: pulled out as selectedSystemId and added to the fetch effect's dependency list. This is the one behaviour change: if the systems config loads after the page mounts, the listing is requested a second time with the correct system. The saga uses takeLatest, so the first request is cancelled rather than both running. - Missing isDebug dependency: added to the image-check effect. - Two setState calls run directly inside an effect: - The in-progress filter is now a useMemo (listedPublications). - In production the page lists listedPublications directly, with no state. - In debug (pprd and dev) the cover-image check still runs in an effect, but it only sets state when the check finishes. The result is stored with the list it was checked against, and the loading spinner is worked out from that, so results from an older list are never shown. - The effect now cleans up after itself, so a check that finishes after the list has changed is thrown away. New test: PublishedDatasetsBrowse.test.jsx gets a debug-mode test, since that path was rewritten and had no test before. It checks that publications with no cover image or the default logo are left out and the spinner goes away.
taoteg
changed the title
…n available.
spatialCoverage is one name-only Place per distinct location across the samples, e.g. [{"@type": "Place", "name": "Austin, TX"}]. For each sample I use geographical_location, and fall back to geographic_origin only when the location is blank. I didn't include both because the origin is coarser (the form describes it as e.g. the planet for extraterrestrial samples). Blank and repeated names are dropped, and I ignored the deprecated location field.
temporalCoverage comes from date_of_collection, and a value is used only if it parses as ISO 8601. Accepted forms are a year (2019), a year-month (2019-06), a date, a date-time, or a start/end range of those.
- One sample: the value is emitted as entered.
- Several samples: they're combined into one earliest/latest range. A year counts as running from Jan 1 to Dec 31, so 2019-06-01 and 2019 together give 2019.
- Rejected: text like June 2019 or 06/01/2019, impossible dates, ranges that end before they start, and dates with a space instead of T before the time. Rejected values are skipped without a log message, since legacy free text would otherwise log a warning on every page render.
- date_of_creation is not used, as we discussed.
How it works: sample values are read from the nodes in pub.tree, the same place the file lookup gets them, without checking whether a node is a sample. Portals with no such fields just lose the keys when empty fields are dropped. The helpers are in schema_org.py, above _named_node.
…ghwire Press tags. The dataset's JSON-LD now has "inLanguage": "en", and the Scholar tags now include citation_author_institution and citation_language. Nothing is committed. 769 tests pass across public_data, publications and workspace_operations, and ruff check and format are clean. - inLanguage (schema_org.py): set to "en", the same hardcoded value as DC.language, @language and DataCite's language. - citation_author_institution: each author's own institution goes in a tag directly after that author's citation_author tag, because readers pair them by position. Authors with no institution get no tag. To carry the institution, citation_authors in get_citation_context is now a list of {name, institution} instead of plain name strings; the template was its only user. - citation_language: <meta name="citation_language" content="en">, added after citation_publication_date inside the Scholar block. Like the other Scholar tags, it only appears when the page has authors and a date. - datacite_operations.py: I only updated a comment so it lists all the places that hardcode "en".
… it lists Summary To keep page renders small, the landing page's embedded JSON-LD lists at most 100 files (LANDING_PAGE_MAX_FILE_OBJECTS), plus the whole-publication ZIP when there is one. croissant.json always lists every file. The landing page still claimed Croissant conformance (conformsTo) after cutting the list, even when no ZIP was listed to cover the files it left out. A Croissant loader reading the embedded JSON-LD would then see only part of the dataset, with no way to reach the rest. That happens with no web mirror (pprd and local dev), and on prod after a first publish, before the archive job has stored the ZIP's sha256. The page now drops conformsTo when it cuts files and has no ZIP to cover them, but still links croissant.json, which is the full, conformant description. Changes (server/portal/apps/public_data/schema_org.py) - The 100-file cut moves out of get_schema_org_json and into get_citation_context, its only caller that needed it. get_schema_org_json no longer takes max_file_objects and always lists every file. - get_citation_context records whether the full document conforms before cutting the list: - If the cut removed files and no ZIP is listed, it drops conformsTo from the page's copy. - The <link rel="alternate"> to croissant.json now follows the full document's conformance, not the page copy's. Dropping the claim no longer removes the link to the document that has it. - _cap_file_objects returns whether it cut anything. Behaviour ┌─────────────────────────────────────────────────┬─────────────────────────────────┬─────────────────────┬───────────────────────┐ │ Case │ Page conformsTo │ croissant.json link │ croissant.json │ ├─────────────────────────────────────────────────┼─────────────────────────────────┼─────────────────────┼───────────────────────┤ │ ≤100 files │ unchanged │ unchanged │ unchanged │ ├─────────────────────────────────────────────────┼─────────────────────────────────┼─────────────────────┼───────────────────────┤ │ >100 files, ZIP listed (prod, steady state) │ kept: the ZIP covers every file │ yes │ unchanged │ ├─────────────────────────────────────────────────┼─────────────────────────────────┼─────────────────────┼───────────────────────┤ │ >100 files, no ZIP listed │ dropped │ still linked │ unchanged, conformant │ ├─────────────────────────────────────────────────┼─────────────────────────────────┼─────────────────────┼───────────────────────┤ │ Not conformant (e.g. a file without a checksum) │ absent │ no │ 404, as before │ └─────────────────────────────────────────────────┴─────────────────────────────────┴─────────────────────┴───────────────────────┘ - The page stays a valid schema.org Dataset in every case, so Google Dataset Search eligibility doesn't change. - PublicationCroissantView, the static croissant.json written to the web mirror, and the page's subjectOf link to that static copy are unchanged. Tests (server/portal/apps/public_data/views_unit_test.py) - Updated the four existing cap tests for the new code path: - A capped page with the ZIP still claims conformance and links croissant.json. - A capped page without the ZIP drops the claim and keeps the link (this test replaces test_get_citation_context_caps_landing_page_file_objects). - The recordSet and short-list tests call _cap_file_objects directly and check its return value. - Added: - A capped, non-conformant dataset gets neither the claim nor the link. - End to end through the views: the rendered page's JSON-LD lists the capped files with no claim and links croissant.json, which returns 200 with the claim and every file.
Summary Adds four cr: mappings to CROISSANT_1_1_CONTEXT, which is the @context used in the landing page's JSON-LD and in croissant.json: ┌──────────────┬─────────────────┐ │ Term │ Maps to │ ├──────────────┼─────────────────┤ │ path │ cr:path │ ├──────────────┼─────────────────┤ │ repeated │ cr:repeated │ ├──────────────┼─────────────────┤ │ replace │ cr:replace │ ├──────────────┼─────────────────┤ │ samplingRate │ cr:samplingRate │ └──────────────┴─────────────────┘ The Croissant 1.1 spec's @context (Appendix 1) doesn't include these four, but the reference context bundled with mlcroissant does. Without them, mlcroissant warned that the context didn't match its own when loading our croissant.json. Adding them stops the warning. We don't output any of these terms, and the existing 1.1 mappings are unchanged. The documents mean exactly what they did before. Any tool that compares against mlcroissant's context will now find a full match. Changes - server/portal/apps/public_data/schema_org.py: - Adds the four mappings. - Updates the comment above CROISSANT_1_1_CONTEXT: it now says the context is the 1.1 Appendix 1 context plus these four mlcroissant terms, instead of exactly Appendix 1. - server/portal/apps/public_data/views_unit_test.py: test_get_schema_org_json_uses_full_croissant_1_1_context now expects the four new keys and checks that each one maps to cr:<term>.
…ation-level institution instead of the author's institution. What changed in server/portal/apps/public_data/schema_org.py: - JSON-LD: each creator's affiliation is now the publication's institution. Whitespace is trimmed, and when there's no institution the field is left out, the same rule DataCite's creators uses. Any institution set on an individual author is ignored. - Scholar tags: citation_authors[].institution now uses that same value, so every citation_author tag is followed by a citation_author_institution tag. When the publication has no institution, those tags are left out. - I updated the comments to say why: it keeps the page consistent with DataCite and with what the dataset page shows. What changed in views_unit_test.py: - test_get_schema_org_json_success, test_get_citation_context and test_index_view_renders_citation_and_dc_meta_tags now expect "Test University", which is the default institution in the tests' sample publication. - Two tests are renamed and rewritten to check that an author's own institution is ignored: - ..._affiliation_comes_from_each_authors_own_institution is now test_get_schema_org_json_affiliation_is_the_publication_institution. - ..._follows_each_citation_author_with_their_own_institution is now test_index_view_follows_each_citation_author_with_the_publication_institution. - Two new tests cover a publication with no institution (missing, empty or whitespace-only): - test_get_schema_org_json_omits_affiliation_without_publication_institution checks that the JSON-LD has no affiliation. - test_index_view_omits_citation_author_institution_without_publication_institution checks that no citation_author_institution tag is emitted.i
… manifest Archive app 0.0.4 (WMA-Tapis-Templates) writes archive/<workspace id>/manifest-columns.txt next to the sha256 manifest. Each line is one CSV or TSV file: its path and its first line (the header row), both base64-encoded. Until now nothing read that file, so landing pages and croissant.json never had a recordSet. This PR stores those column names on the publication's files, and the existing _get_record_sets code builds the recordSet from them. Changes project_publish_operations.py - New _read_columns_manifest fetches the manifest from PORTAL_PROJECTS_PUBLISHED_ROOT_SYSTEM_NAME. It returns None when the file doesn't exist (archive app older than 0.0.4, or the job stopped early). - New _parse_columns_manifest turns each line into the file's path (relative to its published system) and its column names, in the FileObj.columns format: - The header is parsed as CSV, or as TSV for .tsv files. A leading byte-order mark is dropped. - Skipped: lines that don't decode, files outside the workspace, files that aren't CSV/TSV, and headers with no named columns. - Headers with more than MAX_STORED_COLUMNS (500) columns are skipped and logged. - load_publication_file_checksums now also stores columns: - It covers root-level files and files in entity nodes. - It stores columns even when there's no sha256 manifest. - It only saves the publication when the checksums, the columns or the ZIP checksum actually changed. - The log line now reports how many files' columns were stored. Backfill - compute_publication_checksums runs the same loader, so it backfills columns for existing publications without any code change. Only its docstring is updated. Docstrings and CHANGELOG - schema_org._get_record_sets and FileColumn no longer say nothing fills in columns. - New CHANGELOG entry. Behavior - No manifest: the result is exactly as before. One info log line, and no recordSet, which Croissant allows. - Manifest present: each CSV/TSV file with a usable header gets a cr:RecordSet whose fields point back to that file's distribution entry. - Other portals: no impact. The loader only runs after DPMP's archive job. Testing - 17 new tests in project_publish_operations_unit_test.py: - Parsing: quoted headers, TSV with an uppercase .TSV extension, BOM, a newline in a filename, another workspace's files, non-CSV files, empty and blank headers, invalid base64, non-UTF-8, wrong field count, and the column limit. - Fetching: the right system and path, and a missing manifest returning None. - Storing: columns stored without a sha256 manifest, including files in entity nodes; no save when columns haven't changed. - End to end: loaded columns produce a landing-page recordSet. Deployment notes - Register archive app 0.0.4 in Tapis and upload its ZIP to the …/digitalrocks-archive-publication/v4/ path on Corral. DPMP's pprd and prod settings already set _PORTAL_PUBLICATION_ARCHIVE_APP_VERSION = "0.0.4". - After deploying, run compute_publication_checksums on existing publications to backfill their columns.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
WC-494: SEO and discoverability for DPMP published datasets
Related / Required
PR for TAPIS App update: WC-494 DPMP SEO Croissant Compliance Updates WMA-Tapis-Templates#127
PR for Camino update: WC-494 DPMP SEO Deployment Updates Camino#88
PR for Core-Portal-Deployments update: https://github.com/TACC/Core-Portal-Deployments/pull/230
Core-CMS-Custom redirect snippets to the canonical landing pages:
projects-317: https://github.com/TACC/Core-CMS-Custom/blob/main/digitalrocks_assets/snippets/projects-317--canonical.html
projects-218: https://github.com/TACC/Core-CMS-Custom/blob/main/digitalrocks_assets/snippets/projects-218--canonical.html
Note: These CMS snippets could be dropped if instead the routes they redirect are deleted from the CMS Sitemap. I think the CMS routes were tests and can be safely removed. This was the bandaid to enable deployment until that was resolved.
Summary
Published datasets now have crawlable landing pages rendered on the server. Each page carries schema.org Dataset and Croissant 1.1 JSON-LD, plus Google Scholar, Dublin Core and Open Graph tags. The pages are listed in an XML sitemap at /published-datasets/sitemap.xml. Publishing sends DataCite complete metadata, and a DOI only becomes findable once its files are in place. Everything DPMP-specific is behind settings; other portals only get small base.html cleanups (see "Impact on other portals" below).
Required
Note: The robots files could be part of Camino, but I did not want to delay the deployment of this for DPMP to test against all portals. Could be a future task to make support for the
/published-datasets/route the default for portal exposed published project data via Camino's robots.What Changed
Landing pages (public_data/)
Sitemap and indexing
Publishing and DataCite
License and ORCID
New settings, models and commands
Fixes and security
Impact on other portals
Only DPMP configures publishing in Core-Portal-Deployments. For every other portal:
Known limitations and follow-ups
Suggested review focus
Deployment Notes