The File System

A Common Structure

The file system manages and organizes our files and directories using a common structure defined by:

• Parent-child relationships
  A “family tree” (more like a root system) of “parent” and “child” relationships (Figure 1a).

• Directionality
  Parent items are at the top/up; child items are at the bottom/ down (Figure 1a).

• Different ways to access
  Accessible via command-line (Figure 1b) and GUI (Figure 1c).

(a) A representative file system with parent-child relationships shown.

(b) Accessing the file system via the command-line.

(c) Accessing the file system via graphical interface.

Figure 1: A representative file system.

Tip

The top-most directory is called the root directory and is shown with the /.

---

Paths

The directories, files, and subdirectories of a file system are connected by paths. Paths also describe the locations within the file system.

Figure 2: The absolute path from / to mouse.gtf, highlighted in red.

Absolute and Relative Paths

There are two types of paths:

• Absolute path
  The path taken from the top-most directory (root, /), to the specified file or directory. The absolute path always starts with /.
• Relative path
  The path taken from the present working directory to the specified file or directory.

Example paths to a few items from Figure 1a are shown below.

Target	Absolute Path	Relative Path (from the /bin directory)
plot.R	/bin/plot.R	plot.R
conda	/bin/conda	conda

., .., and ~ aliases

The characters ., .. and ~ have special meaning in the unix shells.

• . – Current directory
• .. – Parent directory
• ~ – Users home directory

For example the following code means to do_the_thing in the current directory.

do_the_thing ./

The code below means to do_the_thing two directories above our current directory.

do_the_thing ../../

Finally, the code below means to do_the_thing in the user’s home directory.

do_the_thing ~

Q&A: If we are in the /tmp directory, what are the absolute and relative paths of the genome.fa file?

Click here for the answer

Target	Absolute Path	Relative Path
genome.fa	/data/genome.fa	../data/genome.fa

---

Navigating the File System

Let’s learn a few useful commands for moving around the file system.

pwd – print working directory

Prints out our current location, called our “working directory”.

Command	Options/Flags	Arguments
pwd

Run the pwd command in your terminal.

pwd

/Users/csifuentes/Desktop/shell-lesson-data

Q&A: Is the path returned an absolute or relative path?

Click here for the answer

Absolute, the path starts with /. Also, pwd will always return the absolute path (from the root directory).

---

ls – list

Lists the items in a directory.

Without a target, the command defaults to the current directory (./)

Command	Options/Flags	Arguments
ls	flags	path/to/directory

In terminal, type ls and press enter/return.

ls

exercise-data
north-pacific-gyre

Items in your home directory is listed, alphabetically. Flags/options can make the output more useful, a few are shown below.

Flag	Description
-l	Returns the results in a long format, which provides information about the item type (- for file, d for directory, l for link) item permissions thenumber of links or files inside that item the item owner the item group the time the item was created item size item name
-h	Returns the results with a human-readible size value
-a	Includes entries beginning with a ., which are not shown by default

Let’s use these 3 flags together. Type the ls -lha into your terminal.

ls -lha

total 16
drwxrwxr-x@  5 csifuentes  staff   160B Mar  2 11:54 .
drwx------@  6 csifuentes  staff   192B Feb 27 13:35 ..
-rw-r--r--@  1 csifuentes  staff   6.0K Feb 27 18:00 .DS_Store
drwxrwxr-x@  7 csifuentes  staff   224B Sep 16  2021 exercise-data
drwxrwxr-x@ 21 csifuentes  staff   672B Sep 16  2021 north-pacific-gyre

Tip

The . is also used to hide items.

---

cd – change directory

Changes our location in the file system.

Note: Without a target directory, cd will default to the user home directory.

Command	Options/Flags	Arguments
cd		path/to/directory

Q&A: Change your current working directory to be one directory above the current directory, then check the new working directory location and list it’s contents.

Click here for the answer

cd ..
pwd
ls -lha

/Users/csifuentes/Desktop
total 920
drwx------@  6 csifuentes  staff   192B Feb 27 13:35 .
drwxr-xr-x+ 73 csifuentes  staff   2.3K Mar  2 11:54 ..
-rw-r--r--@  1 csifuentes  staff   6.0K Feb 17 11:08 .DS_Store
-rw-r--r--   1 csifuentes  staff     0B Sep 22  2020 .localized
drwxrwxr-x@  5 csifuentes  staff   160B Mar  2 11:54 shell-lesson-data
-rw-r--r--@  1 csifuentes  staff   450K Feb 24 15:42 shell-lesson-data.zip

Now let’s move back into the shell-lesson-data directory.

cd ~/Desktop/shell-lesson-data

---

mkdir – make directory

Creates new directories.

Note: The -p flag will create the directory and any required intermediate directories.

Command	Options/Flags	Arguments
mkdir	flags	path/to/directory path/to/additional/directory

Let’s pretend we want to create a directory structure for our thesis work. We need the following:

1. A top-level directory.
2. Separate directories for each chapter (we have 5).
3. Directories for images, data, and text of each chapter.

We’ll do this using only the commands we’ve learned thus far (except for my use of tree to easily view directory structures). Later, we’ll learn quicker ways to do this.

Make a top-level directory.

# let's first change into our shell-lesson-data directory
cd ~/Desktop/shell-lesson-data

# make the directory
mkdir -p thesis

# look at the structure of thesis
tree thesis

thesis

0 directories, 0 files

Create a directory for each chapter.

# create all of the directories at one time
mkdir -p thesis/chapter_1 thesis/chapter_2 thesis/chapter_3 thesis/chapter_4 thesis/chapter_5

# look at the structure of thesis
tree thesis

thesis
├── chapter_1
├── chapter_2
├── chapter_3
├── chapter_4
└── chapter_5

5 directories, 0 files

Create a directory for each subsection of each chapter.

# create sub-directories in each chapter
mkdir -p thesis/chapter_1/images thesis/chapter_1/data thesis/chapter_1/text
mkdir -p thesis/chapter_2/images thesis/chapter_2/data thesis/chapter_2/text
mkdir -p thesis/chapter_3/images thesis/chapter_3/data thesis/chapter_3/text
mkdir -p thesis/chapter_4/images thesis/chapter_4/data thesis/chapter_4/text
mkdir -p thesis/chapter_5/images thesis/chapter_5/data thesis/chapter_5/text

# look at the structure of thesis
tree thesis

thesis
├── chapter_1
│   ├── data
│   ├── images
│   └── text
├── chapter_2
│   ├── data
│   ├── images
│   └── text
├── chapter_3
│   ├── data
│   ├── images
│   └── text
├── chapter_4
│   ├── data
│   ├── images
│   └── text
└── chapter_5
    ├── data
    ├── images
    └── text

20 directories, 0 files

Good file and directory names

Complicated names make it difficult when working on the CL

• Do not use spaces – bash reads these a separate arguments
• Do not begin with a dash, “-” – bash reads these a options
• Use alpha-numeric, ., -, and _

If you need refer to a file/directory that contains a space, put the entire thing in ” ”
"/root/subdir/file with spaces"

---

Working in the File System

Now let’s learn some useful ways to work in the file system.

Text Editors

Allow one to create and edit text files

• using plain characters only, unlike MS Word and Google Docs
• varying easy of use and capability of the text editors
• can use in-terminal (in the shell) or GUI (external)

In-Terminal Examples	GUI Examples
pico, nano	notepad, notepad++
emacs, Vim	Atom, Visual Studio Code

(a) In-terminal Editor – nano

(b) GUI Editor – Visual Studio Code

Figure 3: In-live vs GUI text editors

Continuing with our thesis work, let’s create a README.txt file to keep track of each chapter directory.

nano – in-line text editor

Opens the editor into a file (or new file if it doesn’t exist).

Note: Creates the target file if it does not already exist. Flags and arguments are optional here.

Command	Options/Flags	Arguments
nano	flags	path/to/file

Let’s create the README.txt file in our thesis directory.

nano ~/Desktop/shell-lesson-data/thesis/README.txt

A file will open in the editor. Follow the directions in Figure 4 below.

Figure 4: Edit and save the file as README.txt using the nano editor.

Looking in thesis, we see our new file.

ls ~/Desktop/shell-lesson-data/thesis

README.txt
chapter_1
chapter_2
chapter_3
chapter_4
chapter_5

---

cp – copy

Copies and pastes items with a single command.

Command	Options/Flags	Arguments
cp	flags	path/to/source path/to/destination

It might be nice to have a README in each chapter directory. Let’s use the cp command to do this.

# copy to each chapter directory
cp thesis/README.txt thesis/chapter_1/
cp thesis/README.txt thesis/chapter_2/
cp thesis/README.txt thesis/chapter_3/
cp thesis/README.txt thesis/chapter_4/
cp thesis/README.txt thesis/chapter_5/

# view the structure of thesis
tree thesis

thesis
├── README.txt
├── chapter_1
│   ├── README.txt
│   ├── data
│   ├── images
│   └── text
├── chapter_2
│   ├── README.txt
│   ├── data
│   ├── images
│   └── text
├── chapter_3
│   ├── README.txt
│   ├── data
│   ├── images
│   └── text
├── chapter_4
│   ├── README.txt
│   ├── data
│   ├── images
│   └── text
└── chapter_5
    ├── README.txt
    ├── data
    ├── images
    └── text

20 directories, 6 files

This was tedious. Don’t worry, we’ll learn more efficient ways to do this.

---

mv – move and rename

Moves and renames items, including files and directories. Note that the last argument is the destination.

Command	Options/Flags	Arguments
mv	flags	path/to/source path/to/other/source path/to/destination

Important

The mv command will overwrite a files without warning!

• use the -n flag to prevent overwriting existing files
• use the -i flag to prompt for confirmation before overwriting existing files

Let’s rename the README.txt file in the chapter 1 directory so that it contains the chapter number.

# rename the file
mv thesis/chapter_1/README.txt thesis/chapter_1/README_1.txt

# list files in chapter 1
ls thesis/chapter_1

README_1.txt
data
images
text

---

rm – remove

Deletes the specified target.

Command	Options/Flags	Arguments
rm	flags	path/to/target

Important

Unlike in the GUI, rm deletes items permanently!

• use the -r flag to remove files and directories recursively
• use the -i flag to prompt for confirmation before deleting each item

For fun, let’s remove the thesis directory.

# remove the items (files and directories) recursively
rm -r thesis

# list items in shell-lesson-data
ls

exercise-data
north-pacific-gyre

---

Introducing Wildcards

Wildcards represent 0 or more characters and are used for pattern matching.

• * – 0 or more characters
  
• ? – exactly 1 character

Let’s see some examples with of each. From our shell-lesson-data/exercise-data/proteins.

Listing all files.

# cd into the directory
cd ~/Desktop/shell-lesson-data/exercise-data/proteins

# list all files in proteins
ls 

cubane.pdb
ethane.pdb
methane.pdb
octane.pdb
pentane.pdb
propane.pdb

Listing files ending in ethane.pdb, using *. Note that we use the * at the end becuase all files have the same .pdb ending, so this is faster.

# cd into the directory
cd ~/Desktop/shell-lesson-data/exercise-data/proteins

ls *ethane.*

ethane.pdb
methane.pdb

Listing files ending in ethane.pdb with a preceeding character, using ?.

# cd into the directory
cd ~/Desktop/shell-lesson-data/exercise-data/proteins

ls ?ethane.*

methane.pdb

As shown above, wildcards can be used together and combined in different ways to form complex patterns.

For example, we can use ???ane.pdb together to indicate any 3 characters followed by ane.pdb.

# cd into the directory
cd ~/Desktop/shell-lesson-data/exercise-data/proteins

# list all files with 3 characters followed by ane.pdb
ls ???ane.pdb

cubane.pdb
ethane.pdb
octane.pdb

---

Quiz Time

Question 1:

Starting from /Users/amanda/data, which command(s) whould take Amanda to her home directory (/Users/amanda)?

a.) cd .

NO, will be in the same place

b.) cd /

NO, will be in root

c.) cd /home/amanda

NO, not where we want to be

d.) cd ../..

NO, will be in /Users

e.) cd ~

YES, ~ is an alias for the user’s home directory

f.) cd home

NO, not a thing

g.) cd

YES, without input cd will take you to the home directory

h.) cd ..

YES

Question 2:

With the file system shown, if pwd displays Users/thing, what will ls -F ../backup display?

Note: -F adds a / to the end of directories.

1. ../backup: No such file or directory
2. 2012-12-01 2013-01-08 2013-01-27
3. 2012-12-01/ 2013-01-08/ 2013-01-27/
4. original/ pnas_final/ pnas_sub/

Click here for the answer

4. ../backup/ refers to /Users/backup

Question 3:

With the file system below, if pwd displays /Users/backup and ls -r displays items in reverse order, what command(s) will result in the following output?

pnas_sub/ pnas_final/ original/

1. ls pwd
2. ls -r -F
3. ls -r -F /Users/backup

Click here for the answer

2. YES.
3. YES.

Question 4:

Chris runs the following commands and realizes that sucrose.dat and maltose.dat should be in the raw/ directory.

$ ls -F
 analyzed/ raw/
$ ls -F analyzed
fructose.dat glucose.dat maltose.dat sucrose.dat
$ cd analyzed

Complete the command below to move these files into the raw/ directory.

$ mv sucrose.dat maltose.dat _____/____

Click here for the answer

$ mv sucrose.dat maltose.dat ../raw

Question 5:

Chris gave you a file named file.txt, which contains a list of his favorite animals. You want to rename it to why_do_i_need_this.txt. Which of the following commands would do the trick?

1. cp file.txt why_do_i_need_this.txt
2. mv file.txt why_do_i_need_this.txt
3. mv file.txt .
4. cp file.txt .

Click here for the answer

1. No. This creates a new file instead of renaming the old file.
2. YES. This renames the file.
3. No. This moves the file to the current directory with no new file name – would throw an error.
4. No. This copies the file to the current directory with no new file name – would throw an error.

Question 6:

What is the output of the final ls command in the sequence shown below?

$ pwd
 /Users/jamie/data
$ ls 
 proteins.dat
$ mkdir recombined
$ mv proteins.dat recombined/
$ cp recombined/proteins.dat ../proteins-saved.dat
$ ls

1. proteins-saved.dat recombined
2. recombined
3. proteins.dat recombined
4. proteins-saved.dat

Click here for the answer

2. recombined

Question 7:

Chris accidentally removed a file named important_file.txt. How can the file be retrieved?

1. rm --undo
2. “^Z”, control+Z
3. Restore from the “Trash” bin
4. It can’t.

Click here for the answer

4. It can’t. Be very careful when removing files/directories.

Question 8:

When run the in proteins/ directory, which command(s) will produce the output below?

ethane.pdb methane.pdb

1. ls *t*ane.pdb
2. ls *t?ne.*
3. ls *t??ne.pdb
4. ls ethane.*

Click here for the answer

1. No. Would give ethane.pdb methane.pdb octane.pdb pentane.pdb
2. No. Would give octane.pdb pentane.pdb
3. YES.
4. No. Would give ethane.pdb

Question 9:

Sam has the following diretory structure.

.
├── 2015-10-23-calibration.txt
├── 2015-10-23-dataset1.txt
├── 2015-10-23-dataset2.txt
├── 2015-10-23-dataset_overview.txt
├── 2015-10-26-calibration.txt
├── 2015-10-26-dataset1.txt
├── 2015-10-26-dataset2.txt
├── 2015-10-26-dataset_overview.txt
├── 2015-11-23-calibration.txt
├── 2015-11-23-dataset1.txt
├── 2015-11-23-dataset2.txt
├── 2015-11-23-dataset_overview.txt
├── backup
│   ├── calibration
│   └── datasets
└── send_to_bob
    ├── all_datasets_created_on_a_23rd
    └── all_november_files

Sam uses the following commands to create a backup directory and another directory to send to her collaborator, Bob.

$ cp *dataset* backup/datasets
$ cp ____calibration____ backup/calibration
$ cp 2015-____-____ send_to_bob/all_november_files/
$ cp ____ send_to_bob/all_datasets_created_on_a_23rd/

Help Sam by filling in the blanks so that the resulting structure looks like this.

.
├── 2015-10-23-calibration.txt
├── 2015-10-23-dataset1.txt
├── 2015-10-23-dataset2.txt
├── 2015-10-23-dataset_overview.txt
├── 2015-10-26-calibration.txt
├── 2015-10-26-dataset1.txt
├── 2015-10-26-dataset2.txt
├── 2015-10-26-dataset_overview.txt
├── 2015-11-23-calibration.txt
├── 2015-11-23-dataset1.txt
├── 2015-11-23-dataset2.txt
├── 2015-11-23-dataset_overview.txt
├── backup
│   ├── calibration
│   │   ├── 2015-10-23-calibration.txt
│   │   ├── 2015-10-26-calibration.txt
│   │   └── 2015-11-23-calibration.txt
│   └── datasets
│       ├── 2015-10-23-dataset1.txt
│       ├── 2015-10-23-dataset2.txt
│       ├── 2015-10-23-dataset_overview.txt
│       ├── 2015-10-26-dataset1.txt
│       ├── 2015-10-26-dataset2.txt
│       ├── 2015-10-26-dataset_overview.txt
│       ├── 2015-11-23-dataset1.txt
│       ├── 2015-11-23-dataset2.txt
│       └── 2015-11-23-dataset_overview.txt
└── send_to_bob
    ├── all_datasets_created_on_a_23rd
    │   ├── 2015-10-23-dataset1.txt
    │   ├── 2015-10-23-dataset2.txt
    │   ├── 2015-10-23-dataset_overview.txt
    │   ├── 2015-11-23-dataset1.txt
    │   ├── 2015-11-23-dataset2.txt
    │   └── 2015-11-23-dataset_overview.txt
    └── all_november_files
        ├── 2015-11-23-calibration.txt
        ├── 2015-11-23-dataset1.txt
        ├── 2015-11-23-dataset2.txt
        └── 2015-11-23-dataset_overview.txt

Click here for the answer

$ cp *calibration.txt backup/calibration
$ cp 2015-11-* send_to_bob/all_november_files/
$ cp *-23-dataset* send_to_bob/all_datasets_created_on_a_23rd/