Status: draft. Verified steps and screenshots to come.
In this lesson you will make your scratch space visible to OnDemand apps and bring a dataset onto the cluster. Time: 10 minutes · Branch: Start here · Program: the OnDemand Files app and a terminal
| Before this | Data | You'll need |
|---|---|---|
| Nothing | Your own files. Step 3 downloads a public dataset as an example | Your account on the cluster |
Note
Cluster details. Highlighted values are placeholders. AMA27's queue names, module versions, paths and addresses go in when the cluster is deployed.
Why this matters
OnDemand apps open in your home directory, which is small. Work usually belongs in scratch or in project space, and an app will not show you those unless you can reach them from home. One link fixes that, and knowing which place to use keeps your data away from the automatic clean-up.
Key idea
Three places, three jobs. The cluster gives you three kinds of storage, and each one trades something away. Home is small and backed up, so it suits code and notes. Scratch is large and fast and not backed up, so it suits the data you are working on now. Project space is shared with your group. Files left in scratch are deleted after a set age, the purge. Anything you want to keep has to be copied elsewhere. → Glossary: scratch
Steps
Step 1: Know the three places
| Place | Path | Size | Backed up? | Use it for |
|---|---|---|---|---|
| Home | AMA27 home path | AMA27 home quota | Yes | Scripts, notes, small results you must not lose |
| Scratch | AMA27 scratch path | AMA27 scratch quota | No, and old files are deleted | Data you're working on right now |
| Project | AMA27 project path | Set per group | No | Data your group shares |
You should see: which place fits your data. Working data goes in scratch.
Step 2: Make scratch visible from home
Open a terminal: the shell on the dashboard's Clusters menu, or the terminal in a desktop session.
Run on: a login node, or the OnDemand desktop terminal. Put AMA27's scratch path in place of /path/to/scratch.
ln -s /path/to/scratch/$USER ~/scratch
You should see: no output. In the Files app, your home directory now has a scratch folder that leads to your scratch space. Apps that start in your home can follow it there.
Screenshot: the Files app, with the scratch folder in your home
Screenshot to come: The Files app showing your home directory, with the new scratch folder marked
Step 3: Download a dataset
Compute nodes may not be able to reach the internet, but login nodes can. So download on a login node, into scratch. The example here is ParaView's tutorial data.
Run on: a login node
cd ~/scratch
curl -LO https://www.paraview.org/files/data/ParaViewTutorialData-20220629.tar.gz
tar -xzf ParaViewTutorialData-20220629.tar.gz
ls ParaViewTutorialData-20220629
You should see: a folder of example datasets, disk_out_ref.ex2, can.ex2 and headsq.vti among them. Your own data arrives the same way, by download or by upload.
Step 4: Upload a file of your own
In the Files app, open scratch, click Upload and choose a small file from your computer.
You should see: your file listed in scratch. For anything over a few gigabytes, use Globus instead of the browser (see module B3).
Screenshot: the Files app's Upload button
Screenshot to come: The Files app open on your scratch folder, with the Upload button marked
Step 5: Check how much space you have
Run on: a login node. Put AMA27's quota command in place of quota-command.
quota-command
quota-command $USER-scratch
You should see: your usage and limit for home, and then for scratch.
Check your understanding
Your simulation writes 5 GB of results you'll visualize next week and want to keep for a paper. Where should they live?
Write them to scratch, which is fast and large, and visualize them from there. Then copy the results you need to keep to project space, or somewhere backed up, before the purge can delete them.
If something goes wrong
| What you see | Why | What to do |
|---|---|---|
ln: failed to create symbolic link … File exists |
You already made the link | Nothing to do: open ~/scratch |
curl: (6) Could not resolve host |
You're on a compute node without internet | Run the download on a login node instead |
Permission denied in a project folder |
Your group hasn't given you write access | Ask your group's owner (see module B2) |
Next
Credits
ParaView example data: Kitware, BSD-3-Clause. Written for SBSC; text CC-BY 4.0.