HPC Bioinformatics

The Problem #

A bioinformatics company runs protein structure prediction workloads on a Slurm-based HPC cluster on AWS Parallel Computing Service. They needed a full CI/CD pipeline for containerized workloads, operational scripts, and a post-deployment test framework — all in GitHub Actions.

What I Built #

• Composite GitHub Action — full container lifecycle: Docker build with ECR layer caching → Singularity/Apptainer conversion (required for Slurm compatibility) → checksum comparison to skip redundant builds → S3 upload with integrity validation → SSM deployment to cluster
• Second Action — Python package publishing to AWS CodeArtifact via OIDC keyless auth, no stored credentials
• AWS-specific Bash utility library covering cluster operations, image management, and test reporting
• Post-deployment e2e test framework: submits jobs to Slurm, writes results to S3

Outcome #

• Multiple bioinformatics workloads deploy through fully automated, keyless CI/CD
• Checksum-based skip logic eliminates redundant Apptainer conversions and cluster deployments