gsoc26: Declarative Workflow Engine (chaining download, deploy, delete)

[DhanashreePetare]

Pull Request

Description

Adds a declarative workflow engine: databusclient workflow run pipeline.yml runs a sequence of download/deploy/delete steps defined in YAML, with output chaining between steps (${steps.name.output_files} / ${steps.name.output_urls}) and per-step error handling (fail/continue/retry).

Related Issues
Issue #79

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• This change requires a documentation update
• Housekeeping

Checklist:

• My code follows the ruff code style of this project.
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (if applicable)
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
  • poetry run pytest - all tests passed
  • poetry run ruff check - no linting errors

One open design question raised with mentor separately: deploying a file that was downloaded and converted (--format/--compression) during a workflow step has no clean path yet, since the converted file only exists locally and deploy needs a fetchable URL. Deploying the original, unconverted download URL (via the new output_urls step output) works today and is tested.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a70fa93e-10ad-4c3f-a381-f89cca3bb2b5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Integer-Ctrl]

Only accepts a singular uri. If a user supplies a natural YAML list for multiple downloads, the existing download API receives a nested list and will fail when it treats each entry as a string.

[Integer-Ctrl]

Not urgent for this PR, but if this example is kept, it might be nicer to download a version from the Databus and then delete the same version. This better reflects a realistic archive-then-delete workflow

[Integer-Ctrl]

URGENT: Whoops, replace the api_key(s) with environment variable.
I suggest deleting the API key in your Databus account and regenerating a new one 😄

[Integer-Ctrl]

Have a look at this scenario.I just changed the URIs to use data from the Databus:

steps:
  - name: fetch_dataset
    command: download
    uri: https://databus.dev.dbpedia.link/fhofer/gsoc26/test-data/2.0
    localdir: ./workflow-output/full-pipeline

  - name: publish_dataset
    command: deploy
    version_id: https://databus.dev.dbpedia.link/fhofer/gsoc26/test-data/3.0
    title: "Workflow Demo - Full Pipeline"
    abstract: "throwaway, testing full download-deploy-delete pipeline"
    description: "throwaway version testing all three commands chained together"
    license: https://creativecommons.org/licenses/by-sa/3.0/
    api_key: ${DATABUS_API_KEY}
    files: ${steps.fetch_dataset.output_urls}
    on_error: fail

  - name: cleanup_previous_version
    command: delete
    uris:
      - https://databus.dev.dbpedia.link/fhofer/gsoc26/test-data/2.0
    api_key: ${DATABUS_API_KEY}
    on_error: continue

Since test-data/2.0 gets deleted, the download of test-data/3.0 also fails since it redirects to the Databus URL of test-data/2.0. In terminal this can be checked for example like this:

curl -I https://databus.dev.dbpedia.link/fhofer/gsoc26/test-data/2.0/test-data.yml
HTTP/1.1 307 Temporary Redirect
Server: nginx/1.27.3
Date: Thu, 06 Aug 2026 12:10:22 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 118
Connection: keep-alive
X-Powered-By: Express
Location: https://raw.githubusercontent.com/dbpedia/databus/master/server/app/api/swagger.yml
Vary: Accept
Set-Cookie: connect.sid=s%3Aokk6kjAXzqO0iL5mOZ_BJrM7vZrP6Plj.hktpYQBSb%2BX5badk74%2Bw3gs25TgZmRgf8D3ANBDFuoY; Path=/; HttpOnly

One solution to fix this, is to check whether the an URL in output_urls is a redirect and if yes, use the redirect URL in the deploy step. Download and delete do not need this. For reference, check for redirect: https://github.com/DhanashreePetare/databus-python-client/blob/136c49a35eaf33b755bb1133b53700c6a2d30687/databusclient/api/download.py#L416-L427