PNDA workshops¶

Session 1 - Bash scripting¶

Preliminary steps:¶

install docker on your system (MacOS, Windows, linux)
download the PNDA docker image:

run: docker pull sination/pnda:101
run the docker image on your system

(Linux and Mac) run: docker run -it -p 9999:9999 -v `pwd`/data:/data sination/pnda:101
once we all have jupyterlab running on our browsers

Open a terminal and we're good to go!

General notes:¶

tab is your friend (use it wisely!)
make use of google, nearly everything that you think of is possible. (don't forget stackoverflow)
this tutorial is available online at PNDA github page, however, here's another good resource to check.
feel free to ask questions if you ran into a trouble

First step¶

print something!

echo "Hello world!"

File management¶

First of all, what's a path?

address to a directory (folder) or file

Important things to know about paths: relative vs. absolute, special path characters /, ., .., ~, and metacharacters *, ?

where you are

pwd

change where you are

cd <path>

cd data
    pwd

what is there (nothing!)

ls [path]

create a folder (let's put something here)

mkdir [path/]<name> [...]

mkdir test

create multiple folders

mkdir test1 test2

create an empty file

touch [path/]<name>

touch test.txt

list more information

ls -l

ls -a

ls -la

write into file

echo "Hello World!" > [path/]<name>

echo "Byebye World!" >> [path/]<name>

vi [path/]<name>

nano [path/]<name>

nano test.txt

download a file

wget <URL>

wget "http://textfiles.com/humor/aussie.lng"

copy a file

cp <source file> <destination directory>

cp ./aussie.lng ./test2/

    cp ./aussie.lng ./aussie.txt

copy a directory (folder)

cp -r <source directory> <destination directory>

cp -r ./test2 ./test3

move a file

mv <file> <destination directory>

mv ./aussie.lng ./test/

rename a file / directory

mv <old name/path> <new name/path>

mv ./aussie.txt ./aussie_terms.txt

remove a file / directory

rm [-r] <name/path>

view a file

show all: cat <name/path>

show start: head [-n ...] <name/path>

show end: tail [-n ...] <name/path>

show less but all!: less <name/path>

also there's grep but hopefully we'll get back to it later

information of a file

wc

wc -l

compress/extract a directory

tar ...

to compress:

tar -czvf <archive_name>.tar.gz <directory>

to extract:

tar -xzvf <archive_name>.tar.gz

explanation:
- -c is for creating an archive
- -z is for compressing it using gzip
- -v is to do it in verbose mode (report what's happening)
- -f is to specify archives file name
- -x is for extracting an archive
  
  There is also unzip and unrar if you needed.

some other useful basic commands:

du and df to check sizes

date to check the date

sort to sort a list of names

ln to link directories (similar to windows shortcuts)
in case you wanter to find out more about a command:

man <command>

for example:

man wc

Remote access¶

You need to access one of the imaging computers that someone else is using to copy some data onto your own computer.

You can't access nimble and blinky through your own computer -- but you have remote access to them.

virtually sit behind a computer on a terminal!

ssh ...

more specifically:

ssh {username}@{serveraddress}

after this, you're sitting on a terminal on the server.

NOTE: if you want to ssh to a server from windows you migh need a terminal emulator or some tweaks on windows 10

you wanna get more fancy with ssh?

check this link

copy files and directories to/from a remote sever

scp ...

from server:

scp yourusername@yourserver:/home/yourusername/examplefile .

to server:

scp examplefile yourusername@yourserver:/home/yourusername/

here's a detailed blog post on scp and sss for future reference
copy only if needed

rsync

useful tools to use over ssh:

you want to check the load on an imaging machine:

top

htop

having terminals running regardless of session:

screen (we'll get back to this later)

Shell Scripting¶

using the shell commands that we have learnt in a well documented way

automate routines
reduce manual workload
run a list of commands sequentially

what can you do in a shell script:

all you could do in a terminal and more...

let's write this in a text file and name it script.sh

Note: Bash scripts usually have a .sh extension and are run in the terminal.

#!/bin/bash

# Author : ...
# Date : ...
# Script follows here:

echo "What is your name?"

we also might need to give it permission to execute! (chmod +x script.sh)

Concepts: Variables¶

Store values (names, numbers, lists) to use later

variables

#!/bin/bash

# Author : ...
# Date : ...
# Script follows here:

today=`date`

echo "Today is: ${today}"
echo "What is your name?"

#!/bin/bash

# Author : ...
# Date : ...
# Script follows here:

today=`date`

echo "Today is: ${today}"

echo "What is your name?"

read name

echo "hello ${name}"

We can have different variable types:

# a string:
name="workshop tutorial"

# a list of entries:
names=("name 1" "name 2" "name 3")

# now check these commands:
echo "First name: ${names[0]}"

echo "All names: ${names[*]}"

# a number:
i=12

# now you can do basic operations:
echo `expr $test + 2`

check here to find out more about operators.

Concepts: Inputs and outputs¶

Introduce flexibility by providing variables from outside your script.

#!/bin/bash

# Author : ...
# Date : ...
# Script follows here:

today=`date`

echo "Today is: ${today}"

name=$1

echo "hello ${name}"

Concepts: Logical Statements¶

if else - logic

variables

Concepts: Logical Statements¶

syntax:

if [ condition ]
then
    do something ...
fi

Concepts: Logical Statements¶

complete syntax:

if [ condition ]; then
    "do something ..."
elif [ condition ]; then
    "do something else ..."
else
    "what are you gonna do???"
fi

Concepts: Logical Statements¶

simple example:

#!/bin/bash

# Check if your input is "password"

if [ $1 == "password" ]
then
    echo "access granted!"
else
    echo "access denied!"
fi

Concepts: Loops¶

Why and when we use loops

variables

Concepts: Loops¶

For loops in bash; syntax:

for ITEM in item1 item2 ... itemN
do
    "do this for every item"
done

Concepts: Loops¶

Here's an example:

#!/bin/bash

for i in 1 2 3 4 5
do

    echo "Welcome $i times"

done

Bash scripting¶

Example:¶

You have several subject folders and files within those folders. You run an analysis and find that some of your subjects failed. You need to know which subjects failed in order to re-run them.

We need a script!

We can use a dummy dataset that I've already created on nimble.

Let's download it then, it sits at the following directory:

/data/nimble/work/sina/PNDA/test_subjects.tar.gz

Here's what we need to do:

scp <user>@maggie.nnf.unimelb.edu.au:/data/nimble/work/sina/PNDA/test_subjects.tar.gz ./

Bash scripting¶

We need to find the subjects who have missing files. what should we do?

#!/bin/bash

# Author : Sina M.L.
# Date : 08.04.2019
# Script to find subjects with missing files:

main_folder=$1

prefix=$2

echo "These subjects are missing functional:"
for subpath in `ls "./${main_folder}/"`; do

    if [[ ${subpath} == ${prefix}_* ]]; then

        if [ ! -f "./${main_folder}/${subpath}/func/rest.nii" ]; then
            echo ${subpath}
        fi

    fi

done

echo "These subjects are missing anatomical:"
for subpath in `ls "./${main_folder}/"`; do

    if [[ ${subpath} == ${prefix}_* ]]; then

        if [ ! -f "./${main_folder}/${subpath}/anat/T1w.nii" ]; then
            echo ${subpath}
        fi

    fi

done

Can we combine the two to find all subjects who are missing any file?

Bash scripting¶

Now that we know the pattern, can you write a script that does the same?

#!/bin/bash

# Author : Sina M.L.
# Date : 08.04.2019
# Script to generate sample dataset:

main_folder=$1

prefix=$2

mkdir "./${main_folder}"

# bash syntax
for id in {0..106}
do
    mkdir "./${main_folder}/${prefix}_${id}"
    mkdir "./${main_folder}/${prefix}_${id}/anat"
    mkdir "./${main_folder}/${prefix}_${id}/func"
    echo "functional MRI!" > "./${main_folder}/${prefix}_${id}/func/rest.nii"
    echo "T1w MRI!" > "./${main_folder}/${prefix}_${id}/anat/T1w.nii"
done

for id in {1..106..16}
do
    rm "./${main_folder}/${prefix}_${id}/func/rest.nii"
done

for id in {4..106..13}
do
    rm "./${main_folder}/${prefix}_${id}/anat/T1w.nii"
done

Screen:¶

multiple long living windows on a machine!¶

You've logged into maggie on your computer through ssh and want to copy a large dataset from one folder to another.

But you need to leave and close your computer. When you close terminal, the copying gets terminated.

You can run jobs in a screen and can check them later from wherever!

Screen:¶

multiple long living windows on a machine!¶

Screen:¶

Create a screen under your name:

screen -S <name>

Note: be carefull in using upper/lower case properly

Screen:¶

a screen is created! now what?

list existing screens:

screen -ls
re-Enter the existing screen:

screen -r <name>
detach an already attached screen:

screen -d <name>

Screen:¶

alright! I'm inside the screen, what can I do?

You can have multiple windows inside one screen! each window is a separate terminal session.

create a window:

^a c
list all windows:

^a "

Note: ^ is short for using the ctrl key with the next key.

Screen:¶

alright! I'm inside the screen, what can I do?

You can have multiple windows inside one screen! each window is a separate terminal session.

rename a window:

^a A
go to next window:

^a n
gp to previous window:

^a p

Note: ^ is short for using the ctrl key with the next key.

Screen:¶

Wait there's more...

detach the screen:

^a d
close the screen (for good! take care!):

^a \
just close a window (be careful with this one):

^d

Note: ^ is short for using the ctrl key with the next key.

Screen:¶

Want to get more fancy?

vertical split tab:

^a |
horizontal split tab:

^a S
go to next tab:

^a tab
get rid of the tabs (your windows are safe):

^a Q

Note: ^ is short for using the ctrl key with the next key.

Regular expressions¶

They help you match text against a pattern.

When can you use it?¶

There are many usecases, once you combine it with grep and metacharacters it is a powerful tool under your control!

Regular expressions¶

We will use it to play with some dictionary terms!

First make sure you have the dictionary file:

scp <user>@maggie.nnf.unimelb.edu.au:/data/nimble/work/sina/PNDA/words.txt ./

Here's the simplest usecase: (matches any string that has the expression)

grep -e "expression" words.txt

Regular expressions¶

"^expression" : matches any string that starts with expression

grep -e "^expression" words.txt

"expression$" : matches any string that ends with expression

grep -e "expression$" words.txt

"^expression$" : matches exactly with expression

grep -e "^expression$" words.txt

Regular expressions¶

"*" : matches zero or more repetitions

grep -e "ucce*" words.txt
grep -e "ap*le" words.txt

"+" : matches one or more repetitions (needs to be escaped)

grep -e "ap\+le" words.txt
grep -E "ap+le" words.txt  # this uses extended regular expressions

"?" : matches zero or one repetitions (needs to be escaped)

grep -e "appen\?s" words.txt
grep -E "appen?s" words.txt  # this uses extended regular expressions

"{}" : matches mentioned number of repetitions (needs to be escaped)

grep -E "c{2}e{2}" words.txt
grep -E "c{1,3}e{2}" words.txt

"()" : group expression (needs to be escaped)

grep -E "(ca){2}" words.txt

Regular expressions¶

"|" : OR operator (needs to be escaped)

grep -E "(b|d)lub" words.txt

"[]" : OR operator for anything inside

grep -E "[abd]lub" words.txt

"[^]" : negate OR operator for anything inside

grep -E "[^abcd]lub" words.txt

Regular expressions¶

Character classes:¶

"\d" : digits
"\w" : word
"\s" : whitespace

Note: capital for negated selection
"." : anything

Regular expressions¶

back referencing:¶

"()" : reference with paranthesis
"\1" : back reference with number

grep -E "a(.{2})a\1" words.txt

Other interesting subjects to talk about if we have time (also good for further reading):¶

metacharacters: * ? [ ] ' " \ $ ; & ( ) | ^ < > new-line space tab

should be escaped with \ if needed.
...

Other interesting subjects to talk about if we have time (also good for further reading):¶

quotes: the differences of ', ", and ```
...

Other interesting subjects to talk about if we have time (also good for further reading):¶

reverse search with ^r
...

Other interesting subjects to talk about if we have time (also good for further reading):¶

here document: giving input to interactive shells

command << delimeter
    inputs
    delimeter

here's an example:

#!/bin/bash

    filename=test.txt
    vi $filename <<EndOfCommands
    i
    This file was created automatically from
    a shell script
    ^[
    ZZ
    EndOfCommands

Other interesting subjects to talk about if we have time (also good for further reading):¶

Functions
...

Other interesting subjects to talk about if we have time (also good for further reading):¶

sed for stream editing
...

Thanks!¶

In [ ]: