Kodiak Beginner's Guide
Introduction
This is an introduction to using the Kodiak high performance computing cluster. It is intended for users with little or no experience with "Unix-like" environments such as Linux. Experienced users will probably want to skip this and read the Kodiak User's Guide which goes into more detail on how to compile and run programs on Kodiak. Although this document is not meant to be a comprehensive tutorial on Linux usage, hopefully it will be enough for you to get started.
Note: This document is a "work in progress". If anything is unclear, inaccurate, or even outright wrong please send email to Carl_Bell@baylor.edu.
Contents
- Getting Started
- Directories and Files
- Directories
- Navigating Directories
- Working With Files
- Wildcards
- Editing Files
- Permissions
- File Permissions
- Directory Permissions
- Nifty Stuff
- A Few More Useful Commands
- Standard Output and Input
- Stdout
- Stdin
- Pipes
- The Shell and Scripts
- Summary of Commands
- More Information
- Kodiak User's Guide
Getting Started
Getting An Account
You will need an account to use Kodiak. Kodiak accounts are available to Baylor faculty, graduate and undergraduate students. To request an account, contact Mike_Hutcheson@baylor.edu and/or Brian_Sitton@baylor.edu. Students should have their faculty sponsor request the account on the student's behalf. Accounts are also available to non-Baylor individuals who are collaborating with Baylor researchers. Once your account is created, you should receive an email with your username and your account's initial password that you will use to log in to Kodiak.
Logging In
To log in to Kodiak, you will need to use ssh.
Microsoft Windows users should use PuTTY which may be downloaded from their official site. When you run PuTTY, you should see the "PuTTY Configuration" window. Specify "kodiak.baylor.edu" as the Host Name (or IP address) and click Open to connect. Enter your username and password to continue.
Mac OS X users should connect with the ssh command in a Terminal window. The Terminal.app program can be found in your Utilities folder. (At the Finder, select the Go→Utilities menu.) Linux users should connect with the ssh command in an xterm terminal window or the console. The actual command is ssh username@kodiak.baylor.edu.
Note: In this document, we will use "bobby" as your username.
When you first log in to Kodiak, you will see several informational messages. Eventually you should end up with something like:
[bobby@login001 ~]$
What you are looking at is known as the command prompt or just prompt. When you see this, the system is waiting for you to tell it what to do. The default format of the prompt is:
[your_username@system_name current_working_directory]$
Why does the prompt say "login001" as the system name instead of "kodiak"? It just does. And what is meant by "current working directory" and why is it "~"? We'll get to that in a minute.
Note: To make things a bit easier to read, this document will usually omit the [user@host directory] bit and just use $ as the prompt.
General Info
So you are now sitting at the prompt. What next? To do something, type a command along with any any options and arguments that the command needs and press the return (or enter) key. For example, to see a list of users currently logged in, type who and press return.
$ who
betty pts/2 2013-10-02 13:48 (172.16.8.20)
bobby pts/3 2013-10-02 13:51 (172.16.8.24)
bubba pts/1 2013-10-02 08:16 (93.184.216.119)
Note: You always need to press return to run a command, so from now on, pressing return is implied.
Commands often have options associated with them. For example, to add column headings to the output of the who command, add a -H option.
$ who -H
USER LINE TIME COMMENT
betty pts/2 2013-10-02 13:48 (172.16.8.20)
bobby pts/3 2013-10-02 13:51 (172.16.8.24)
bubba pts/1 2013-10-02 08:16 (93.184.216.119)
Some commands specify options with a "-" and a single letter. Some commands specify options with "--" and a word. Some commands can do both. Some options take extra arguments. How do you know what's what? Well, you could try running who with an invalid option such as -x and the command might give you some helpful information. (It helps if you already know that -x is, in fact, an invalid option.)
$ who -x
who: invalid option -- x
Try `who --help' for more information.
Okay then, let's try who --help for more information...
$ who --help
Usage: who [OPTION]... [ FILE | ARG1 ARG2 ]
-a, --all same as -b -d --login -p -r -t -T -u
-b, --boot time of last system boot
-d, --dead print dead processes
-H, --heading print line of column headings
-l, --login print system login processes
--lookup attempt to canonicalize hostnames via DNS
-m only hostname and user associated with stdin
-p, --process print active processes spawned by init
-q, --count all login names and number of users logged on
-r, --runlevel print current runlevel
-s, --short print only name, line, and time (default)
-t, --time print last system clock change
-T, -w, --mesg add user's message status as +, - or ?
-u, --users list users logged in
--message same as -T
--writable same as -T
--help display this help and exit
--version output version information and exit
If FILE is not specified, use /var/run/utmp. /var/log/wtmp as FILE is common.
If ARG1 ARG2 given, -m presumed: `am i' or `mom likes' are usual.
Report bugs to .
Not all commands support the --help option. The best way to get detailed information about commands is to read the on-line manual with the man command. Entries in the on-line manual also known as man pages.
$ man who
WHO(1) User Commands WHO(1)
NAME
who - show who is logged on
SYNOPSIS
who [OPTION]... [ FILE | ARG1 ARG2 ]
DESCRIPTION
-a, --all
same as -b -d --login -p -r -t -T -u
-b, --boot
time of last system boot
-d, --dead
print dead processes
-H, --heading
print line of column headings
-l, --login
print system login processes
--lookup
attempt to canonicalize hostnames via DNS
-m only hostname and user associated with stdin
-p, --process
print active processes spawned by init
-q, --count
all login names and number of users logged on
-r, --runlevel
print current runlevel
-s, --short
print only name, line, and time (default)
-t, --time
print last system clock change
-T, -w, --mesg
add user's message status as +, - or ?
-u, --users
list users logged in
--message
same as -T
--writable
same as -T
--help display this help and exit
--version
output version information and exit
If FILE is not specified, use /var/run/utmp. /var/log/wtmp as FILE is
common. If ARG1 ARG2 given, -m presumed: 'am i' or 'mom likes' are
usual.
AUTHOR
Written by Joseph Arceneaux, David MacKenzie, and Michael Stone.
REPORTING BUGS
Report bugs to .
COPYRIGHT
Copyright © 2006 Free Software Foundation, Inc.
This is free software. You may redistribute copies of it under the
terms of the GNU General Public License
. There is NO WARRANTY, to the
extent permitted by law.
SEE ALSO
The full documentation for who is maintained as a Texinfo manual. If
the info and who programs are properly installed at your site, the com-
mand
info who
should give you access to the complete manual.
who 5.97 January 2009 WHO(1)
Changing Your Password
Actually, one of the things you should do when logging on to Kodiak the first time is change your password from the one randomly generated when your account was created. Let's go ahead and get that out of the way. To change your password, use the passwd command. (Yes, the command is "passwd" and not "password".)
When you type your current password, you will not see it displayed. Same thing with your new password. If all goes well, you should see the following:
$ passwd
Changing password for user bobby.
(current) UNIX password:
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
It's possible that the system won't accept the password you want. For example, if you were to try to set your password to "secret" (don't set your password to "secret"...) or if you were to try to set it to something very simple, the passwd command will complain:
BAD PASSWORD: it is based on a dictionary word
or
BAD PASSWORD: it is too simplistic/systematic
Password formats on Kodiak are somewhat strict, so keep trying. You will get 3 attempts before the passwd command gives up. Just try again.
Logging Out
When you are finished working on Kodiak and are ready to log out type logout or exit. You can also press control-D (^D) to log out.
Note: It is common to use the "^" character as an abbreviation for "control" key sequences and we will use it from now on. Just remember that if you see something like "^X" it means "control-X".
Directories and Files
It is quite likely that you are familiar with a desktop operating system such as Microsoft Windows or Mac OS X. If so, then you are probably also familiar with how those operating systems usually organize things, namely, folders and documents:
Linux organizes things the same way except it uses the terms "directory" and "file" instead of "folder" and "document". Actually, Windows and OS X also use "directory" and "file". Folders and documents are just metaphors used in their graphical user interfaces (GUI). There are also GUIs available for Linux that display directories and files as folders and documents. Because you will be using a command line interface on Kodiak, we'll stick with that.
Just about everything on Linux is a file. Even the commands you run are files. Files are located within directories. Directories can also contain other (sub-)directories. Directories and files are stored in file systems and there may be multiple file systems. (There are several file systems on Kodiak.)
You can visualize the file system as an upside-down tree, with the roots of the tree at the top, directories and subdirectories as the tree's branches, and files as the tree's leaves. Files (and directories) can be expressed as a path starting at the root, traversing the directories, and ending at the file itself. So using the example image above, the path to the "budget" file would be:
{ ROOT } → Bobby → Documents → Budget
Linux uses / as the root. It also uses / as the directory separator (or delimiter). So the path to the file above would be:
/Bobby/Documents/Budget
A path to a file or directory that begins with "/" (the root) is considered an absolute or full path and specifies a file or directory explicitly. If a path does not start with "/", it is considered a relative path and specifies a file or directory starting with the current working directory.
Something to keep in mind is that Linux file systems are case sensitive. In other words, the files "letter.txt" and "Letter.txt" are two distinct files. Also, some operating systems (e.g., VMS) support the concept of file versions. This is not the case with Kodiak.
Directories
When you first log in to Kodiak, you will be "in" your home directory. Whichever directory that you are currently "in" is considered your current working directory or working directory. To see the full path to your current working directory, use the pwd (print working directory) command.
[bobby@login001 documents]$ pwd
/home/bobby/documents
In this case, the working directory is "/home/bobby/documents". As you can see, the prompt displays the name of the current directory. But if your current directory were your home directory you would see:
[bobby@login001 ~]$ pwd
/home/bobby
So what's the deal with the "~" character in the prompt? That's just an abbreviation or shortcut for your home directory. You can often substitute "~" when referring to something within your home directory. For example, "~/pictures/family" would be equivalent to "/home/bobby/pictures/family".
There are two other common directory abbreviations:
. - current directory
.. - "parent" directory
In the example above, the current directory (i.e., ".") is "documents" (full path /home/bobby/documents) and the parent directory (i.e., "..") is "bobby" (full path /home/bobby). You can usually refer to a file or directory within the current working directory just by its name. So if your working directory is "/home/bobby/pictures", you can refer to the family subdirectory as simply "family". But you could also refer to it as "./family". As you will see in a bit, there are times that it will be necessary to refer to a file using "./". Also, assuming your current working directory is still "/home/bobby/pictures", you can refer to the letter file as "../documents/letter.txt".
Navigating Directories
Okay, you've logged in to Kodiak and are in your home directory. Now what? To change your current working directory, use the cd command. You can see that you specify a directory with either a relative or full path.
[bobby@login001 ~]$pwd/home/bobby [bobby@login001 ~]$cd pictures/family[bobby@login001 family]$pwd/home/bobby/pictures/family [bobby@login001 family]$cd ..[bobby@login001 pictures]$pwd/home/bobby/pictures [bobby@login001 family]$cd /home/bobby/documents[bobby@login001 documents]$pwd/home/bobby/documents
To change back to your home directory, you can use cd ~. Because your home directory is the default directory for the cd command and will be used if no directory is explicitly supplied, you can omit the ~ if you wish.
$pwd/home/bobby/downloads/foo/bar/baz $cd$pwd/home/bobby
Just using cd to navigate directories is all well and good if you happen to have all of the files and directories memorized. Since you probably don't, it would help to see a listing of a directory's contents. You can do this with the ls command.
$cd$pwd/home/bobby $lsdocuments downloads pictures $cd documents$lsbudget letter.txt
What would be printed out if there was nothing in the directory? Nothing, then just the prompt. When viewing the directory listing generated by ls, how can you tell if an entry is a directory or a file? One way to do this is to add the -F (or --classify) option.
$ls -Fdocuments/ downloads/ pictures/ $ls -F documentsbudget letter.txt
With the -F option, the names of directories have a "/" after them. The -F option will append other characters for other types but we won't list them here. You may have noticed that by adding the optional argument "documents" to the second ls command, you didn't have to cd to the directory.
Another way to get more detailed (or long) information from the ls command is to add the -l option.
$ls -ltotal 2 drwxr-xr-x 2 bobby users 2 Oct 7 16:01 documents drwxr-x--- 3 bobby users 1 Oct 7 15:02 downloads drwxr-x--- 4 bobby users 2 Oct 7 14:14 pictures $cd documents$ls -ltotal 3 -rw------- 1 bobby users 1169 Oct 7 15:54 budget -rw-r--r-- 1 bobby users 1324 Oct 7 15:56 letter.txt $ls -l -F ..total 2 drwxr-xr-x 2 bobby users 2 Oct 7 16:01 documents/ drwxr-x--- 3 bobby users 1 Oct 7 15:02 downloads/ drwxr-x--- 4 bobby users 2 Oct 7 14:14 pictures/
There's quite a bit of information here. Right now, we will just take a look at the line for "documents".
drwxr-xr-x 2 bobby users 2 Oct 7 16:01 documents
The first character tells what the entry actually is, in this case "d" for directory. An ordinary file would have a "-" as the first character. The next 9 gobbledygook looking letters (rwxr-xr-x) are the entry's permissions. We'll talk about permissions later. Just ignore the next number. The next two columns are the entry's owner (bobby) and group (users), and are related to the permissions. The remaining columns are the entry's size in bytes, the date it was last modified, and its name.
Notice how we got fancy with the last command above and used multiple options, -l and -F as well as the parent directory ".." as the argument. Another useful option is -a which tells ls to include entries that begin with a dot. When a file begins with a dot, it is considered hidden. This is not for security reasons (i.e., "security through obscurity") but is more of a way to help de-clutter things.
$ ls -laF
total 3
drwxr-x--x 5 bobby users 4 Oct 7 16:15 ./
drwxr-xr-x 11 bobby users 9 Oct 7 14:14 ../
-rw------- 1 bobby users 87 Jul 9 16:15 .bashrc
drwxr-xr-x 2 bobby users 2 Oct 7 16:01 documents/
drwxr-x--- 3 bobby users 1 Oct 7 15:02 downloads/
drwxr-x--- 4 bobby users 2 Oct 7 14:14 pictures/
You can see that both the current directory '.' and parent directory '..' are now included in the listing, along with some file named ".bashrc". Also, note that the options were combined, -laF instead of -l -a -F. Many commands can do this.
To create a directory, use the mkdir (make directory) command.
$pwd/home/bobby $ls -Fdocuments/ downloads/ pictures/ $mkdir backups$ls -Fbackups/ documents/ downloads/ pictures/
Working with Files
Now that you can navigate around directories and see what's actually in those directories, it's time to start working with files. If you want to see the contents of a file, use the cat command.
$lsbudget letter.txt $cat letter.txtLorem ipsum, Lorem ipsum dolor sit amet, et vel diam nominavi adversarium, utamur labores postulant eam in. Ad quo ubique delicatissimi, ei elitr eligendi has. Alia tollit definitiones qui in. Ea vis eripuit nusquam. Eos ceteros tibique adversarium ne, nisl tota convenire at pri. Nobis adolescens contentiones sed id, vis sonet neglegentur in, an tation consetetur qui. Duo cu aliquip molestiae, decore numquam oporteat in sit. Ut mea error option conclusionemque, in posse interesset ullamcorper eam, cum ex assum maluisset. Ea eruditi nominati ius, ea quo soleat bonorum salutandi, dictas corrumpit euripidis no vim. An per habemus omittam philosophia. Pro ad verterem salutandi. Et ipsum oratio consectetuer duo, mea ex vidit definitionem. Ad dolor albucius deleniti per, postea ocurreret definitionem ex mea. Ei putent voluptua intellegat vel, pri in liber quodsi lobortis, quis altera essent at his. Laboramus elaboraret eu usu, pro labitur adipisci at, qui congue appetere assueverit ea. Liber scribentur et mel. Ex pro nullam aliquam, in primis hendrerit per, ea est inani quando. Quo ei quis sint dicit, te atqui quando mel. An rebum sententiae incorrupte eum, sea timeam similique ne. Sincerely, Robert M. "Bobby" Baylor
Where did the name "cat" come from? It's an abbreviation of "catenate" or "concatenate". So what does that have to do with displaying a file? The cat command actually concatenates all of the requested files and prints them out. If you wanted to see both files, letter.txt and budget, you would have run cat letter.txt budget.
Sometimes, a file is too big to display all at once. Instead of cat, you can display a screen at a time with the more. When the output pauses, press the space bar to continue. There is also a similar but more powerful command, less, which you will probably end up using instead.
$ more letter.txt
Lorem ipsum,
Lorem ipsum dolor sit amet, et vel diam nominavi adversarium, utamur labores
postulant eam in. Ad quo ubique delicatissimi, ei elitr eligendi has. Alia
tollit definitiones qui in. Ea vis eripuit nusquam. Eos ceteros tibique
adversarium ne, nisl tota convenire at pri.
Nobis adolescens contentiones sed id, vis sonet neglegentur in, an tation
consetetur qui. Duo cu aliquip molestiae, decore numquam oporteat in sit. Ut
mea error option conclusionemque, in posse interesset ullamcorper eam, cum ex
assum maluisset. Ea eruditi nominati ius, ea quo soleat bonorum salutandi,
dictas corrumpit euripidis no vim.
An per habemus omittam philosophia. Pro ad verterem salutandi. Et ipsum oratio
consectetuer duo, mea ex vidit definitionem. Ad dolor albucius deleniti per,
postea ocurreret definitionem ex mea. Ei putent voluptua intellegat vel, pri
in liber quodsi lobortis, quis altera essent at his.
Laboramus elaboraret eu usu, pro labitur adipisci at, qui congue appetere
assueverit ea. Liber scribentur et mel. Ex pro nullam aliquam, in primis
hendrerit per, ea est inani quando. Quo ei quis sint dicit, te atqui quando
mel. An rebum sententiae incorrupte eum, sea timeam similique ne.
--More--(90%)
Sincerely,
Robert M. "Bobby" Baylor
If you don't want to view the entire file, you can use the head command. This will display just the first 10 lines. To see the last 10 lines of a file, use the tail command. A very useful option for tail is -f which tells tail to output appended data as the file grows. We'll use this when we see how to run programs on Kodiak in part 2.
To copy a file, use the cp command. You can simply supply file names and it will copy the file in the current directory. If you want to copy the file into another directory, use the path to the destination as the second argument. If the destination is a directory without a file name, then the file will be copied to that directory and keep the same name.
$pwd/home/bobby/downloads $ls -ltotal 13 -rw-r--r-- 1 bobby users 12424 Oct 8 13:17 report.pdf $cp report.pdf report.pdf.bak$ls -ltotal 25 -rw-r--r-- 1 bobby users 12424 Oct 8 13:17 report.pdf -rw-r--r-- 1 bobby users 12424 Oct 8 13:19 report.pdf.bak $cp report.pdf ../documents$ls -l ../documentstotal 16 -rw-r--r-- 1 bobby users 1169 Oct 7 15:54 budget -rw-r--r-- 1 bobby users 1324 Oct 7 15:56 letter.txt -rw-r--r-- 1 bobby users 12424 Oct 8 13:46 report.pdf
What if, instead of copying a file somewhere, you wanted to move it there. In that case, you would use the mv command.
$pwd/home/bobby/downloads $lsreport.pdf $ls ../documentsbudget letter.txt $mv report.pdf ../documents$ls$ls ../documentsbudget letter.txt report.pdf
The mv command is also used to rename a file. You can also move and rename a file at the same time by using a path to a file (as opposed to a path to a directory) as the destination. mv is also used to move or rename a directory.
Be careful. If you cp or mv a file to a file that already exists, you will overwrite the file. Both cp and mv have an option -i (--interactive) which will ask you to confirm if you want to overwrite an existing file.
$lsbudget letter.txt report.pdf $mv report.pdf annual_report.pdf$lsannual_report.pdf budget letter.txt $cd ../downloads$lsreport.pdf.bak $mv report.pdf.bak ../backups/annual_report.pdf$ls$ls ../backupsannual_report.pdf
To delete a file, use the rm (remove) command. To delete a directory, use the rmdir command. Note that rmdir will only delete a non-empty directory.
$lsfigure_1.pdf poster.pdf.gz report.pdf.bak $rm figure_1.pdf$lsposter.pdf.gz report.pdf.bak $ls -Fbackups/ documents/ downloads/ misc/ pictures/ $ls misc$rm miscrm: cannot remove `misc': Is a directory $rmdir misc$ls -Fbackups/ documents/ downloads/ pictures/
Editing Files
How does one edit a text file on Kodiak? There are several text editors installed: vi, emacs, and nano. The simplest editor is nano. To edit an existing file, run nano with the name of the file as its argument. To create and edit a new file, run nano without an argument or with the name of the file you want to create.
$ nano letter.txt
You will then see something like the following. You should be able to use the arrow keys to move around the document and type to modify it. Available commands can be found at the bottom of the screen. For example, to get help for using nano, press ^G.
GNU nano 1.3.12 File: letter.txt
Lorem ipsum,
Lorem ipsum dolor sit amet, et vel diam nominavi adversarium, utamur labores
postulant eam in. Ad quo ubique delicatissimi, ei elitr eligendi has. Alia
tollit definitiones qui in. Ea vis eripuit nusquam. Eos ceteros tibique
adversarium ne, nisl tota convenire at pri.
Nobis adolescens contentiones sed id, vis sonet neglegentur in, an tation
consetetur qui. Duo cu aliquip molestiae, decore numquam oporteat in sit. Ut
mea error option conclusionemque, in posse interesset ullamcorper eam, cum ex
assum maluisset. Ea eruditi nominati ius, ea quo soleat bonorum salutandi,
dictas corrumpit euripidis no vim.
An per habemus omittam philosophia. Pro ad verterem salutandi. Et ipsum oratio
consectetuer duo, mea ex vidit definitionem. Ad dolor albucius deleniti per,
postea ocurreret definitionem ex mea. Ei putent voluptua intellegat vel, pri
in liber quodsi lobortis, quis altera essent at his.
Laboramus elaboraret eu usu, pro labitur adipisci at, qui congue appetere
[ Read 27 lines ]
^G Get Help ^O WriteOut ^R Read File ^Y Prev Page ^K Cut Text ^C Cur Pos
^X Exit ^J Justify ^W Where Is ^V Next Page ^U UnCut Text^T To Spell
When you are finished editing your file, press ^X to quit and you should see the following. Press "Y" if you want to save any changes, or "N" to ignore any changes. To go back to editing the document, press ^C.
Save modified buffer (ANSWERING "No" WILL DESTROY CHANGES) ? Y Yes N No ^C Cancel
If this is a new document, you will then be prompted to enter the name of the document. Just type in the new name and press return to quit.
File Name to Write: new_letter.txt ^G Get Help ^T To Files M-M Mac Format M-P Prepend ^C Cancel M-D DOS Format M-A Append M-B Backup File
The vi and emacs editors are more powerful than nano but consequently are also more difficult to use. We won't describe how to use them here other than explain below how to quit them in case you do happen to run them. Because nano is fairly limited, it is recommended that you learn to use vi or emacs at some point. There also are graphical editors on Kodiak as well. These include gedit and the aforementioned emacs. To use them, you would need to be running an X11 server on your desktop system such as Cygwin/X or Xming for Microsoft Windows, and XQuartz for Mac OS X.
To quit vi, first make sure you are in command mode and not insert mode. If you see "-- INSERT --" at the bottom of the screen, you are in insert mode. Press the escape key to switch to command mode. Now press :q! to quit without saving. (You should see the ":q!" at the bottom of the screen.)
To quit emacs, press ^X then ^C. You might see C-x C-c at the bottom of the screen before it actually quits. It's possible that you will be asked if you want to save the file. To quit without saving, press "n". If you are told "Modified buffers exist; exit anyway?" answer "yes". You have to type "yes", not just "y".
Wildcards
Now, suppose that there are more than a few files in a directory...
$pwd/home/bobby/pictures/vacation $ls2011Dec22_01.jpg 2012Aug20_27.jpg 2013Jul03_06.jpg 2013Jul08_01.jpg 2011Dec22_02.jpg 2012Aug20_28.jpg 2013Jul03_07.jpg 2013Jul08_02.jpg 2011Dec22_03.jpg 2012Aug20_29.jpg 2013Jul03_08.jpg 2013Jul08_03.jpg 2011Dec22_04.jpg 2012Aug20_30.jpg 2013Jul03_09.jpg 2013Jul08_04.jpg 2011Dec22_05.jpg 2012Aug21_01.jpg 2013Jul04_01.jpg 2013Jul08_05.jpg 2011Dec22_06.jpg 2012Aug21_02.jpg 2013Jul04_02.jpg 2013Jul08_06.jpg 2011Dec22_07.jpg 2012Aug21_03.jpg 2013Jul04_03.jpg 2013Jul08_07.jpg 2011Dec22_08.jpg 2012Aug21_04.jpg 2013Jul04_04.jpg 2013Jul08_08.jpg 2011Dec22_09.jpg 2012Aug21_05.jpg 2013Jul04_05.jpg 2013Jul08_09.jpg 2011Dec22_10.jpg 2012Aug21_06.jpg 2013Jul04_06.jpg 2013Jul08_10.jpg 2011Dec22_11.jpg 2012Aug21_07.jpg 2013Jul04_07.jpg 2013Jul08_11.jpg 2011Dec22_12.jpg 2012Aug21_08.jpg 2013Jul04_08.jpg 2013Jul08_12.jpg 2011Dec22_13.jpg 2012Aug21_09.jpg 2013Jul04_09.jpg 2013Jul09_01.jpg 2011Dec22_14.jpg 2012Aug21_10.jpg 2013Jul04_10.jpg 2013Jul09_02.jpg 2011Dec22_15.jpg 2012Aug21_11.jpg 2013Jul04_11.jpg 2013Jul09_03.jpg 2011Dec22_16.jpg 2012Aug21_12.jpg 2013Jul04_12.jpg 2013Jul09_04.jpg 2011Dec22_17.jpg 2012Aug21_13.jpg 2013Jul04_13.jpg 2013Jul09_05.jpg 2011Dec22_18.jpg 2012Aug21_14.jpg 2013Jul04_14.jpg 2013Jul09_06.jpg 2011Dec22_19.jpg 2012Aug21_15.jpg 2013Jul04_15.jpg 2013Jul09_07.jpg 2011Dec22_20.jpg 2012Aug21_16.jpg 2013Jul04_16.jpg 2013Jul09_08.jpg 2011Dec22_21.jpg 2012Aug21_17.jpg 2013Jul04_17.jpg 2013Jul09_09.jpg 2011Dec22_22.jpg 2012Aug21_18.jpg 2013Jul04_18.jpg 2013Jul09_10.jpg 2011Dec22_23.jpg 2012Aug21_19.jpg 2013Jul04_19.jpg 2013Jul09_11.jpg 2011Dec22_24.jpg 2012Aug21_20.jpg 2013Jul04_20.jpg 2013Jul09_12.jpg 2011Jul04_01.jpg 2012Aug21_21.jpg 2013Jul04_21.jpg 2013Jul09_13.jpg 2011Jul04_02.jpg 2012Aug21_22.jpg 2013Jul04_22.jpg 2013Jul09_14.jpg 2011Jul04_03.jpg 2012Aug21_23.jpg 2013Jul04_23.jpg 2013Jul09_15.jpg 2011Jul04_04.jpg 2012Aug21_24.jpg 2013Jul04_24.jpg 2013Jul09_16.jpg 2011Jul04_05.jpg 2012Aug21_25.jpg 2013Jul04_25.jpg 2013Jul09_17.jpg 2011Jul04_06.jpg 2012Aug21_26.jpg 2013Jul04_26.jpg 2013Jul09_18.jpg 2011Jul04_07.jpg 2012Aug21_27.jpg 2013Jul04_27.jpg 2013May28_01.jpg 2011Jul04_08.jpg 2012Aug21_28.jpg 2013Jul04_28.jpg 2013May28_02.jpg 2011Jul04_09.jpg 2012Aug21_29.jpg 2013Jul04_29.jpg 2013May28_03.jpg 2011Jul04_10.jpg 2012Aug21_30.jpg 2013Jul04_30.jpg 2013May28_04.jpg 2011Jul04_11.jpg 2012Aug21_31.jpg 2013Jul04_31.jpg 2013May28_05.jpg 2011Jun02_01.jpg 2012Aug21_32.jpg 2013Jul04_32.jpg 2013May28_06.jpg 2011Jun02_02.jpg 2012Aug21_33.jpg 2013Jul04_33.jpg 2013May28_07.jpg 2011Jun02_03.jpg 2012Aug21_34.jpg 2013Jul04_34.jpg 2013May28_08.jpg 2011Jun02_04.jpg 2012Aug21_35.jpg 2013Jul04_35.jpg 2013May28_09.jpg 2011Jun02_05.jpg 2012Aug21_36.jpg 2013Jul04_36.jpg 2013May28_10.jpg 2011Jun02_06.jpg 2012Aug21_37.jpg 2013Jul04_37.jpg 2013May28_11.jpg 2011Jun02_07.jpg 2012Aug21_38.jpg 2013Jul04_38.jpg 2013May28_12.jpg 2011Jun02_08.jpg 2012Aug21_39.jpg 2013Jul04_39.jpg 2013May28_13.jpg 2011Jun02_09.jpg 2012Aug21_40.jpg 2013Jul04_40.jpg 2013May28_14.jpg 2011Jun02_10.jpg 2012Aug21_41.jpg 2013Jul04_41.jpg 2013May28_15.jpg 2011Jun02_11.jpg 2012Aug21_42.jpg 2013Jul04_42.jpg 2013May28_16.jpg 2011Jun02_12.jpg 2012Aug21_43.jpg 2013Jul04_43.jpg 2013May28_17.jpg 2011Jun02_13.jpg 2012Aug21_44.jpg 2013Jul04_44.jpg 2013May28_18.jpg 2011Jun02_14.jpg 2012Aug21_45.jpg 2013Jul04_45.jpg 2013May28_19.jpg 2011Jun02_15.jpg 2012Mar11_01.jpg 2013Jul05_01.jpg 2013May28_20.jpg 2011Jun02_16.jpg 2012Mar11_02.jpg 2013Jul05_02.jpg 2013May28_21.jpg 2011Jun02_17.jpg 2012Mar11_03.jpg 2013Jul05_03.jpg 2013May28_22.jpg 2011Jun02_18.jpg 2012Mar11_04.jpg 2013Jul05_04.jpg 2013May28_23.jpg 2011Jun02_19.jpg 2012Mar11_05.jpg 2013Jul05_05.jpg 2013May28_24.jpg 2011Jun02_20.jpg 2012Mar11_06.jpg 2013Jul05_06.jpg 2013May28_25.jpg 2011Jun02_21.jpg 2012Mar11_07.jpg 2013Jul05_07.jpg 2013May28_26.jpg 2011Jun02_22.jpg 2012Mar11_08.jpg 2013Jul05_08.jpg 2013May28_27.jpg 2011Jun02_23.jpg 2012Mar11_09.jpg 2013Jul05_09.jpg 2013May28_28.jpg 2011Jun02_24.jpg 2012Mar11_10.jpg 2013Jul05_10.jpg 2013May28_29.jpg 2011Jun02_25.jpg 2012Mar11_11.jpg 2013Jul05_11.jpg 2013May28_30.jpg 2011Jun02_26.jpg 2012Mar11_12.jpg 2013Jul05_12.jpg 2013May28_31.jpg 2011Jun02_27.jpg 2012Mar11_13.jpg 2013Jul05_13.jpg 2013May28_32.jpg 2011Jun02_28.jpg 2012Mar11_14.jpg 2013Jul05_14.jpg 2013May28_33.jpg 2011Jun02_29.jpg 2012Mar11_15.jpg 2013Jul05_15.jpg 2013May28_34.jpg 2011Jun03_01.jpg 2012Mar11_16.jpg 2013Jul05_16.jpg 2013May28_35.jpg 2011Jun03_02.jpg 2012Mar11_17.jpg 2013Jul05_17.jpg 2013May28_36.jpg 2011Jun03_03.jpg 2012Mar11_18.jpg 2013Jul05_18.jpg 2013May29_01.jpg 2011Jun03_04.jpg 2012Mar11_19.jpg 2013Jul05_19.jpg 2013May29_02.jpg 2011Jun04_01.jpg 2012Mar11_20.jpg 2013Jul05_20.jpg 2013May29_03.jpg 2011Jun04_02.jpg 2012Mar11_21.jpg 2013Jul07_01.jpg 2013May29_04.jpg 2011Jun04_03.jpg 2012Mar11_22.jpg 2013Jul07_02.jpg 2013May29_05.jpg 2011Jun04_04.jpg 2012Mar11_23.jpg 2013Jul07_03.jpg 2013May29_06.jpg 2011Jun04_05.jpg 2012Mar11_24.jpg 2013Jul07_04.jpg 2013May29_07.jpg 2011Jun04_06.jpg 2012Mar11_25.jpg 2013Jul07_05.jpg 2013May29_08.jpg 2011Jun04_07.jpg 2012Mar11_26.jpg 2013Jul07_06.jpg 2013May29_09.jpg 2011Jun04_08.jpg 2012Mar11_27.jpg 2013Jul07_07.jpg 2013May29_10.jpg 2011Jun04_09.jpg 2012Mar11_28.jpg 2013Jul07_08.jpg 2013May29_11.jpg 2011Jun04_10.jpg 2012Mar11_29.jpg 2013Jul07_09.jpg 2013May29_12.jpg 2011Jun04_11.jpg 2012Mar11_30.jpg 2013Jul07_10.jpg 2013May29_13.jpg 2011Jun04_12.jpg 2012Mar11_31.jpg 2013Jul07_11.jpg 2013May29_14.jpg 2011Jun04_13.jpg 2012Mar11_32.jpg 2013Jul07_12.jpg 2013May29_15.jpg 2011Jun04_14.jpg 2012Mar11_33.jpg 2013Jul07_13.jpg 2013May29_16.jpg 2011Jun04_15.jpg 2012Mar11_34.jpg 2013Jul07_14.jpg 2013May29_17.jpg 2011Jun04_16.jpg 2012Mar11_35.jpg 2013Jul07_15.jpg 2013May29_18.jpg 2011Jun04_17.jpg 2012Mar12_01.jpg 2013Jul07_16.jpg 2013May29_19.jpg 2011Jun04_18.jpg 2012Mar12_02.jpg 2013Jul07_17.jpg 2013May29_20.jpg 2011Jun04_19.jpg 2012Mar12_03.jpg 2013Jul07_18.jpg 2013May29_21.jpg 2011Jun04_20.jpg 2012Mar12_04.jpg 2013Jul07_19.jpg 2013May29_22.jpg 2011Jun04_21.jpg 2012Mar12_05.jpg 2013Jul07_20.jpg 2013May29_23.jpg 2011Jun04_22.jpg 2012Mar12_06.jpg 2013Jul07_21.jpg 2013May29_24.jpg 2012Aug20_01.jpg 2012Mar12_07.jpg 2013Jul07_22.jpg 2013May29_25.jpg 2012Aug20_02.jpg 2012Mar12_08.jpg 2013Jul07_23.jpg 2013May29_26.jpg 2012Aug20_03.jpg 2012Mar12_09.jpg 2013Jul07_24.jpg 2013May29_27.jpg 2012Aug20_04.jpg 2012Mar12_10.jpg 2013Jul07_25.jpg 2013May29_28.jpg 2012Aug20_05.jpg 2012Mar12_11.jpg 2013Jul07_26.jpg 2013May29_29.jpg 2012Aug20_06.jpg 2012Mar12_12.jpg 2013Jul07_27.jpg 2013May29_30.jpg 2012Aug20_07.jpg 2012Mar12_13.jpg 2013Jul07_28.jpg 2013May29_31.jpg 2012Aug20_08.jpg 2012Mar13_01.jpg 2013Jul07_29.jpg 2013May29_32.jpg 2012Aug20_09.jpg 2012Mar13_02.jpg 2013Jul07_30.jpg 2013May29_33.jpg 2012Aug20_10.jpg 2012Mar13_03.jpg 2013Jul07_31.jpg 2013May29_34.jpg 2012Aug20_11.jpg 2012Mar13_04.jpg 2013Jul07_32.jpg 2013May29_35.jpg 2012Aug20_12.jpg 2012Mar13_05.jpg 2013Jul07_33.jpg 2013May29_36.jpg 2012Aug20_13.jpg 2012Mar13_06.jpg 2013Jul07_34.jpg 2013May30_01.jpg 2012Aug20_14.jpg 2012Mar13_07.jpg 2013Jul07_35.jpg 2013May30_02.jpg 2012Aug20_15.jpg 2012Mar13_08.jpg 2013Jul07_36.jpg 2013May30_03.jpg 2012Aug20_16.jpg 2012Mar13_09.jpg 2013Jul07_37.jpg 2013May30_04.jpg 2012Aug20_17.jpg 2012Mar13_10.jpg 2013Jul07_38.jpg 2013May30_05.jpg 2012Aug20_18.jpg 2012Mar13_11.jpg 2013Jul07_39.jpg 2013May30_06.jpg 2012Aug20_19.jpg 2012Mar13_12.jpg 2013Jul07_40.jpg 2013May30_07.jpg 2012Aug20_20.jpg 2012Mar13_13.jpg 2013Jul07_41.jpg 2013May30_08.jpg 2012Aug20_21.jpg 2012Mar13_14.jpg 2013Jul07_42.jpg 2013May30_09.jpg 2012Aug20_22.jpg 2013Jul03_01.jpg 2013Jul07_43.jpg 2013May30_10.jpg 2012Aug20_23.jpg 2013Jul03_02.jpg 2013Jul07_44.jpg 2013May30_11.jpg 2012Aug20_24.jpg 2013Jul03_03.jpg 2013Jul07_45.jpg 2012Aug20_25.jpg 2013Jul03_04.jpg 2013Jul07_46.jpg 2012Aug20_26.jpg 2013Jul03_05.jpg 2013Jul07_47.jpg
Okay. That list of files is somewhat unwieldy. But by using wildcards, it's possible to prune the list. When specifying file names or paths, you can use the special character "*" which matches zero or more characters. For example, "a*" would match "a", "at", "abacab", etc. In the pictures directory above, the names of the image files happen to be yearmonthday_num.jpg. So to get a list of just the pictures from 2011...
$ ls 2011*
2011Dec22_01.jpg 2011Dec22_24.jpg 2011Jun02_12.jpg 2011Jun04_02.jpg
2011Dec22_02.jpg 2011Jul04_01.jpg 2011Jun02_13.jpg 2011Jun04_03.jpg
2011Dec22_03.jpg 2011Jul04_02.jpg 2011Jun02_14.jpg 2011Jun04_04.jpg
2011Dec22_04.jpg 2011Jul04_03.jpg 2011Jun02_15.jpg 2011Jun04_05.jpg
2011Dec22_05.jpg 2011Jul04_04.jpg 2011Jun02_16.jpg 2011Jun04_06.jpg
2011Dec22_06.jpg 2011Jul04_05.jpg 2011Jun02_17.jpg 2011Jun04_07.jpg
2011Dec22_07.jpg 2011Jul04_06.jpg 2011Jun02_18.jpg 2011Jun04_08.jpg
2011Dec22_08.jpg 2011Jul04_07.jpg 2011Jun02_19.jpg 2011Jun04_09.jpg
2011Dec22_09.jpg 2011Jul04_08.jpg 2011Jun02_20.jpg 2011Jun04_10.jpg
2011Dec22_10.jpg 2011Jul04_09.jpg 2011Jun02_21.jpg 2011Jun04_11.jpg
2011Dec22_11.jpg 2011Jul04_10.jpg 2011Jun02_22.jpg 2011Jun04_12.jpg
2011Dec22_12.jpg 2011Jul04_11.jpg 2011Jun02_23.jpg 2011Jun04_13.jpg
2011Dec22_13.jpg 2011Jun02_01.jpg 2011Jun02_24.jpg 2011Jun04_14.jpg
2011Dec22_14.jpg 2011Jun02_02.jpg 2011Jun02_25.jpg 2011Jun04_15.jpg
2011Dec22_15.jpg 2011Jun02_03.jpg 2011Jun02_26.jpg 2011Jun04_16.jpg
2011Dec22_16.jpg 2011Jun02_04.jpg 2011Jun02_27.jpg 2011Jun04_17.jpg
2011Dec22_17.jpg 2011Jun02_05.jpg 2011Jun02_28.jpg 2011Jun04_18.jpg
2011Dec22_18.jpg 2011Jun02_06.jpg 2011Jun02_29.jpg 2011Jun04_19.jpg
2011Dec22_19.jpg 2011Jun02_07.jpg 2011Jun03_01.jpg 2011Jun04_20.jpg
2011Dec22_20.jpg 2011Jun02_08.jpg 2011Jun03_02.jpg 2011Jun04_21.jpg
2011Dec22_21.jpg 2011Jun02_09.jpg 2011Jun03_03.jpg 2011Jun04_22.jpg
2011Dec22_22.jpg 2011Jun02_10.jpg 2011Jun03_04.jpg
2011Dec22_23.jpg 2011Jun02_11.jpg 2011Jun04_01.jpg
The "*" character doesn't have to be at the end and you can also use multiple wildcard characters. Want to get a list of pictures from the 4th of July no matter what year?
$ ls *Jul04*
2011Jul04_01.jpg 2013Jul04_04.jpg 2013Jul04_18.jpg 2013Jul04_32.jpg
2011Jul04_02.jpg 2013Jul04_05.jpg 2013Jul04_19.jpg 2013Jul04_33.jpg
2011Jul04_03.jpg 2013Jul04_06.jpg 2013Jul04_20.jpg 2013Jul04_34.jpg
2011Jul04_04.jpg 2013Jul04_07.jpg 2013Jul04_21.jpg 2013Jul04_35.jpg
2011Jul04_05.jpg 2013Jul04_08.jpg 2013Jul04_22.jpg 2013Jul04_36.jpg
2011Jul04_06.jpg 2013Jul04_09.jpg 2013Jul04_23.jpg 2013Jul04_37.jpg
2011Jul04_07.jpg 2013Jul04_10.jpg 2013Jul04_24.jpg 2013Jul04_38.jpg
2011Jul04_08.jpg 2013Jul04_11.jpg 2013Jul04_25.jpg 2013Jul04_39.jpg
2011Jul04_09.jpg 2013Jul04_12.jpg 2013Jul04_26.jpg 2013Jul04_40.jpg
2011Jul04_10.jpg 2013Jul04_13.jpg 2013Jul04_27.jpg 2013Jul04_41.jpg
2011Jul04_11.jpg 2013Jul04_14.jpg 2013Jul04_28.jpg 2013Jul04_42.jpg
2013Jul04_01.jpg 2013Jul04_15.jpg 2013Jul04_29.jpg 2013Jul04_43.jpg
2013Jul04_02.jpg 2013Jul04_16.jpg 2013Jul04_30.jpg 2013Jul04_44.jpg
2013Jul04_03.jpg 2013Jul04_17.jpg 2013Jul04_31.jpg 2013Jul04_45.jpg
What about using "*" by itself as in ls *? Because "*" matches zero or more characters that should match any and all files in the directory, right? Well, not necessarily. It turns out that there is an exception to wildcard name matching. Files that begin with "." (hidden files) are not included. Wildcards are not just used with the ls command. Most commands that take files as arguments can use wildcards.
$mkdir 4th_pics$mv *Jul04* 4th_pics/$ls *Jul04*ls: cannot access *Jul04*: No such file or directory $cd 4th_pics$ls2011Jul04_01.jpg 2013Jul04_04.jpg 2013Jul04_18.jpg 2013Jul04_32.jpg 2011Jul04_02.jpg 2013Jul04_05.jpg 2013Jul04_19.jpg 2013Jul04_33.jpg 2011Jul04_03.jpg 2013Jul04_06.jpg 2013Jul04_20.jpg 2013Jul04_34.jpg 2011Jul04_04.jpg 2013Jul04_07.jpg 2013Jul04_21.jpg 2013Jul04_35.jpg 2011Jul04_05.jpg 2013Jul04_08.jpg 2013Jul04_22.jpg 2013Jul04_36.jpg 2011Jul04_06.jpg 2013Jul04_09.jpg 2013Jul04_23.jpg 2013Jul04_37.jpg 2011Jul04_07.jpg 2013Jul04_10.jpg 2013Jul04_24.jpg 2013Jul04_38.jpg 2011Jul04_08.jpg 2013Jul04_11.jpg 2013Jul04_25.jpg 2013Jul04_39.jpg 2011Jul04_09.jpg 2013Jul04_12.jpg 2013Jul04_26.jpg 2013Jul04_40.jpg 2011Jul04_10.jpg 2013Jul04_13.jpg 2013Jul04_27.jpg 2013Jul04_41.jpg 2011Jul04_11.jpg 2013Jul04_14.jpg 2013Jul04_28.jpg 2013Jul04_42.jpg 2013Jul04_01.jpg 2013Jul04_15.jpg 2013Jul04_29.jpg 2013Jul04_43.jpg 2013Jul04_02.jpg 2013Jul04_16.jpg 2013Jul04_30.jpg 2013Jul04_44.jpg 2013Jul04_03.jpg 2013Jul04_17.jpg 2013Jul04_31.jpg 2013Jul04_45.jpg
Another wildcard character that is worth mentioning is "?" which matches exactly one character. There are also other wildcards that allow you to match ranges of characters, negate matches, etc. You will probably find yourself using wildcards quite often.
Permissions
Above we mentioned "permissions". Permissions are used to restrict (or allow) access to files and directories.
Each user on Kodiak has a username and is a member of default group. Here, the username is "bobby" and the default group is "users". Users can also be members of other groups, not just the default. You can use the id command to see your username and group membership information.
$ id
uid=520(bobby) gid=100(users) groups=100(users)
You can also use the whoami and groups commands to display your username and group membership. So what are those numbers, 520 and 100? Users and groups are actually identified by numeric values called user identifiers and group identifiers. These are often abbreviated as "uid" and "gid". The names "bobby" and "users" are really labels associated with uid 520 and gid 100 and are used to make things easier for humans to read. Yes, as far a Kodiak is concerned, you are just a number.
Each file (and directory) has a specific owner and group associated with it. That is, a file is owned by a specific user and assigned a group. Using ls -l we can see the files' owners and groups in the output. Want to see the numeric uid and gid instead of the username and group name? Use ls -n.
$pwd/home/bobby $ls -latotal 3 drwxr-x--x 6 bobby users 5 Oct 8 15:27 . drwxr-xr-x 115 root root 114 Oct 1 14:09 .. drwxr-x--- 2 bobby users 0 Oct 8 15:36 backups -rw------- 1 bobby users 87 Oct 7 16:15 .bashrc drwxr-xr-x 2 bobby users 3 Oct 8 15:23 documents drwxr-x--- 2 bobby users 1 Oct 8 15:36 downloads drwxr-x--- 4 bobby users 2 Oct 7 14:14 pictures $ls -n .bashrc-rw------- 1 520 501 87 Oct 7 16:15 .bashrc $ls -l documentstotal 16 -rw-r--r-- 1 bobby users 12424 Oct 8 13:17 annual_report.pdf -rw------- 1 bobby users 1169 Oct 7 15:54 budget -rw-r--r-- 1 bobby users 1324 Oct 7 15:56 letter.txt
As you can see, almost all of the items have as the owner "bobby" (that's you) and group "users". Note that the parent directory '..' (in this case, "/home") is owned by "root". The user "root" is the system administrator account and is sometimes called the "superuser".
Earlier, when describing the output from ls -l, we mentioned that rwxr-xr-x were the file's (or directory's) permissions. So what does that actually mean? There are three types of permissions:
| Permission | Abbr. | Needed to... (files) | Needed to... (directories) |
|---|---|---|---|
| Read | "r" | Read a file's contents | List the files in the directory |
| Write | "w" | Modify or delete the file | Modify the directory, i.e., create/delete files within it |
| Execute | "x" | Execute the file/program | Search the directory (see below) |
Permissions on a file or directory apply to three different classes: 1) the file's or directory's owner or user; 2) members of the file's or directory's assigned group; and 3) others (sometimes called "world"). So putting the permissions for the file "letter.txt" (rw-r--r--) and classes together, we get:
| User | Group | Others |
|---|---|---|
rw- |
r-- |
r-- |
This tells us that the owner of letter.txt (bobby) has read and write access but not execute. Members of the group "users" have read access but not write or execute. Likewise, everyone else also has read access but not write or execute.
To modify permissions of a file or directory, use the chmod (change mode) command. To set or add a permission, that is, to allow that permission, you would use the argument class+permission. For example, to give everyone/others read permission you would use chmod o+r. The possible classes are u (user), g (group), o (others). There is also a special class a (all) which means all three classes. "All" is not a synonym for others. To clear or remove a permission, use class-permission (e.g., chmod o-r).
The method above uses the symbolic mode for modifying permissions. There is second absolute (or numeric) mode as well. The permissions are actually bits and the three types of permissions (read, write, execute) for a class (user, group, others) are sets of 3 bits. The permissions (rwx) map to bit values (100, 010, 001) and their corresponding octal values (4, 2, 1). To set a permission, add the corresponding octal value, e.g., r-x would be 4+1 = 5. You need to do this for all three classes, e.g., rwxr-x--- = 4+2+1,4+1,0 = 750. (You are actually adding 400+200+100 + 040+010 + 000, but it's easier to calculate them as separate digits.) So to set those permissions, you would use chmod 750.
Absolute/numeric mode can be somewhat confusing at first but it's really not too hard once you get the hang of it. There may be times where you would be required to use absolute permissions.
The examples in this document will use symbolic mode.
File Permissions
So right now everyone can read the file letter.txt. Let's fix that.
$ls -l letter.txt-rw-r--r-- 1 bobby users 1324 Oct 7 15:56 letter.txt $chmod o-r letter.txt$ls -l letter.txt-rw-r----- 1 bobby users 1324 Oct 7 15:56 letter.txt
Now what happens when another user, betty, tries to read the file?
$whoamibetty $cat /home/bobby/documents/letter.txtcat: /home/bobby/documents/letter.txt: Permission denied
What about the execute permission. What does that mean? Earlier, we stated that just about everything in Linux is a file, including the commands you run. The execute permission is what permits the file to be run as a command as well as who can run it. So commands are files, too? Most of them, yes. The ls command is located in the "/bin" directory. You can use the which command to find the full path of a command.
$which ls/bin/ls $ls -l /bin/ls-rwxr-xr-x 1 root root 90168 Jan 21 2009 /bin/ls
Setting the execute permission does not make a file executable; it allows an executable file, such as a compiled program or shell script, to be run. (We'll talk about shell scripts later.) So happens when you try to run an executable program that doesn't have the execute permission set? Let's try it with the fictitious command howdy.
$which howdy/usr/local/bin/howdy $ls -l /usr/local/bin/howdy-rw-r--r-- 1 root root 27 Apr 01 12:01 /usr/local/bin/howdy $howdy-bash: /usr/local/bin/howdy: Permission denied
"Permission denied". That makes sense because the execute permission is not set. And if it were?
$ls -l /usr/local/bin/howdy-rwxr-xr-x 1 root root 27 Apr 01 12:01 /usr/local/bin/howdy $howdyHowdy, bobby!
What happens when you set the execute permission on a non-executable file and try to run it?
$ls -l letter.txt-rw-r----- 1 bobby users 1324 Oct 7 15:56 letter.txt $chmod u+x letter.txt$ls -l letter.txt-rwxr----- 1 bobby users 1324 Oct 7 15:56 letter.txt $./letter.txt./letter.txt: line 1: Lorem: command not found ./letter.txt: line 3: Lorem: command not found ./letter.txt: line 4: postulant: command not found ./letter.txt: line 5: tollit: command not found ./letter.txt: line 6: adversarium: command not found ./letter.txt: line 8: Nobis: command not found ./letter.txt: line 9: consetetur: command not found ./letter.txt: line 10: mea: command not found ./letter.txt: line 11: assum: command not found ./letter.txt: line 12: dictas: command not found ./letter.txt: line 14: An: command not found ./letter.txt: line 15: consectetuer: command not found ./letter.txt: line 16: postea: command not found ./letter.txt: line 17: syntax error near unexpected token `in' ./letter.txt: line 17: `in liber quodsi lobortis, quis altera essent at his.' $chmod -x letter.txt
Okay. Let's not do that again. By the way, did you happen to notice that when we tried to run the letter.txt file, we included a "./" in front of it? Sometimes you have to refer to a program using its path (full or relative) and not just by name. We'll explain why later.
Directory Permissions
Permissions have somewhat different meanings for directories. It might help to think of a directory as a special file that contains the list of all of the files and sub-directories within it.
We'll start with the execute permission. For directories, the execute permission is sometimes called "search" permission. It is needed in order to cd into the directory as well as access the files or sub-directories within it. It does not give access the actual list of files. To access a file within the directory, one would need to already know the name of the file as well as have read access to the file itself.
Below, user betty wants to allow bobby (you) to read the file "/home/betty/shared/misc.txt". You can see that although everyone can read misc.txt, the enclosing directory only has search permission for the owner. Bobby is unable to read the file.
[betty@login001 ~] $whoamibetty [betty@login001 ~]$ls -ltotal 16 drwx------ 2 betty users 4096 Oct 21 13:17 documents drwx------ 2 betty users 4096 Oct 21 13:17 downloads drwx------ 2 betty users 4096 Oct 21 13:17 pictures drwx------ 2 betty users 4096 Oct 22 09:28 shared [betty@login001 ~]$ls -l sharedtotal 4 -rw-r--r-- 1 betty users 268 Oct 21 13:18 misc.txt
[bobby@login001 ~]$ whoami bobby [bobby@login001 ~]$ cat /home/betty/shared/misc.txt cat: /home/betty/shared/misc.txt: Permission denied
Rather than give everyone access, betty can add group search permission. Both bobby and betty are members of the "users" group. Note that by adding the -d option to ls you can see the permissions for a directory itself rather than the directory's contents. So now that the directory "/home/betty/shared" has search permission set, bobby should be able to access the file within it. Correct?
[betty@login001 ~]$chmod g+x shared[betty@login001 ~]$ls -ld shareddrwx--x--- 2 betty users 4096 Oct 22 09:28 shared
[bobby@login001 ~]$ cat /home/betty/shared/misc.txt cat: /home/betty/shared/misc.txt: Permission denied
Not yet. In order to access the file misc.txt, not only does its directory (/home/betty/shared) need search permission, but so does its directory's parent directory (/home/betty), and its parent directory's parent (/home) all the way up to root (/). It's turtles all the way up. The /home and / directories already have have search permissions set (which is good because you wouldn't be able to chmod those - they are owned by user root). So adding group search permission to /home/betty should allow bobby to access it. Also because we are adding group permissions and not others permissions, user bubba (who is not a member of the "users" group) will not have access.
[betty@login001 ~]$ls -ld /home/bettydrwx------ 8 betty users 4096 Oct 21 13:57 /home/betty [betty@login001 ~]$chmod g+x /home/betty[betty@login001 ~]$ls -ld /home/bettydrwx--x--- 8 betty users 4096 Oct 21 13:57 /home/betty [betty@login001 ~]$ls -ld /homedrwxr-xr-x 6 root root 4096 Oct 17 14:05 /home [betty@login001 ~]$ls -ld /dr-xr-xr-x 26 root root 4096 Oct 17 14:22 /
[bobby@login001 ~]$ ls /home/betty/shared ls: cannot open directory /home/betty/shared: Permission denied [bobby@login001 ~]$ cat /home/betty/shared/misc.txt Lorem ipsum dolor sit amet, et vel diam nominavi adversarium, utamur labores postulant eam in. Ad quo ubique delicatissimi, ei elitr eligendi has. Alia tollit definitiones qui in. Ea vis eripuit nusquam. Eos ceteros tibique adversarium ne, nisl tota convenire at pri.
[bubba@login001 ~]$ whoami bubba [bubba@login001 ~]$ groups grads [bubba@130 ~]$ ls /home/betty/shared ls: cannot access /home/betty/shared: Permission denied [bubba@login001 ~]$ cat /home/betty/shared/misc.txt cat: /home/betty/shared/misc.txt: Permission denied
Read permission allows one to list the contents of the directory. While there are times where you want to allow search but disallow reading on a directory, you typically will not see the reverse. That is, if you set read permission, you will likely set search permission as well.
[betty@login001 ~]$ ls -ld shared
drwx--x--- 2 betty users 4096 Oct 22 09:28 shared
[bobby@login001 ~]$ cd /home/betty/shared [bobby@login001 shared]$ ls ls: cannot open directory .: Permission denied
[betty@login001 ~]$ chmod g+r shared [betty@login001 ~]$ ls -ld shared drwxr-x--- 2 betty users 4096 Oct 22 09:28 shared
[bobby@login001 shared]$ ls flag.gif misc.txt
Write permission allows one to create, delete, or rename files in the directory. The touch command is a simple way to create an empty file. (By the way, if you touch an existing file, it updates the file's modification timestamp.) As you can see below, when betty adds group write permission to the /home/betty/shared directory, bobby is able to create a file within it.
[betty@login001 shared]$ ls -ld /home/betty/shared
drwxr-x--- 2 betty users 4096 Oct 22 10:29 /home/betty/shared
[bobby@login001 ~]$ cd /home/betty/shared [bobby@login001 shared]$ ls flag.gif misc.txt [bobby@login001 shared]$ touch new_file touch: cannot touch `new_file': Permission denied
[betty@login001 shared]$ chmod g+w /home/betty/shared [betty@login001 shared]$ ls -ld /home/betty/shared drwxrwx--- 2 betty users 4096 Oct 22 10:29 /home/betty/shared
[bobby@login001 shared]$ ls -l total 16 -rw------- 1 betty users 10628 Oct 22 10:28 flag.gif -rw-r--r-- 1 betty users 268 Oct 21 13:18 misc.txt [bobby@login001 shared]$ touch new_file [bobby@login001 shared]$ ls -l total 16 -rw------- 1 betty users 10628 Oct 22 10:28 flag.gif -rw-r--r-- 1 betty users 268 Oct 21 13:18 misc.txt -rw-r--r-- 1 bobby users 0 Oct 24 11:23 new_file
So bobby is now able to create a file within /home/betty/shared. But because bobby does not have any type of access (read nor write) to the flag.gif file, you shouldn't be able to do anything to that file, right?
[bobby@login001 shared]$ls -ltotal 16 -rw------- 1 betty users 10628 Oct 22 10:28 flag.gif -rw-r--r-- 1 betty users 268 Oct 21 13:18 misc.txt -rw-r--r-- 1 bobby users 0 Oct 24 11:23 new_file [bobby@login001 shared]$mv flag.gif huh.gif[bobby@login001 shared]$ls -ltotal 16 -rw------- 1 betty users 10628 Oct 22 10:28 huh.gif -rw-r--r-- 1 betty users 268 Oct 21 13:18 misc.txt -rw-r--r-- 1 bobby users 0 Oct 24 11:23 new_file [bobby@login001 shared]$rm huh.gifrm: remove write-protected regular file `huh.gif'?y[bobby@login001 shared]$ls -ltotal 12 -rw-r--r-- 1 betty users 268 Oct 21 13:18 misc.txt -rw-r--r-- 1 bobby users 0 Oct 24 11:23 new_file
So not only were you able to rename the file, you were even able to delete it as well. What's going on here? Recall that earlier it was said that you can think of a directory as a special file that contains the list of all of the files and sub-directories within it. So adding write permission to the directory means we can modify that list. This includes renaming the entries in that list thus renaming the files in the directory. It also includes removing entries from that list thus deleting the files the files (although we did have to confirm that we wanted to delete a write-protected file). Without write permissions on the file, we would not have been able to edit and modify the contents of the file, but the ability to delete a file that we don't own might be a problem.
But what if we don't want someone to be able to rename or delete files that they don't own? It was mentioned above that the three types of permissions for the three classes were three sets of three bits. There is actually a fourth set of permission-related bits. These extra bits are called the setuid (set user id), setgid (set group id), and sticky bits. The sticky bit is the one we are interested in. If set, it prevents a user from deleting or renaming a file in the directory unless they own the file or the directory. To set or clear the sticky bit, use chmod +t or chmod -t.
[betty@login001 ~]$ls -ld shareddrwxr-xr-x 2 betty users 4096 Oct 24 15:17 shared [betty@login001 ~]$chmod +t shared[betty@login001 ~]$ls -ld shareddrwxr-xr-t 2 betty users 4096 Oct 24 15:17 shared
Notice that the world execute/search permission is now displayed as t. The sticky bit is set. If it were displayed as an uppercase T, that would have meant that the sticky bit is set but world execute/search permission was not. So now what happens when we try to delete or rename the file?
[bobby@login001 shared]$pwd/home/betty/shared [bobby@login001 shared]$ls -ld .drwxr-xr-t 2 betty users 4096 Oct 24 15:17 . [bobby@login001 shared]$ls -ltotal 16 -rw-r--r-- 1 betty users 10628 Oct 22 10:28 flag.gif -rw-r--r-- 1 betty users 186 Oct 24 11:45 misc.txt -rw-r--r-- 1 bobby users 0 Oct 24 11:23 new_file [bobby@login001 shared]$rm flag.gifrm: remove write-protected regular file `flag.gif'?yrm: cannot remove `flag.gif': Operation not permitted [bobby@login001 shared]$mv flag.gif other.gifmv: cannot move `flag.gif' to `other.gif': Operation not permitted [bobby@login001 shared]$rm new_file[bobby@login001 shared]$ls -ltotal 16 -rw-r--r-- 1 betty users 10628 Oct 22 10:28 flag.gif -rw-r--r-- 1 betty users 186 Oct 24 11:45 misc.txt
Now that the sticky bit for /home/betty/shared is set, we are unable to delete or rename the flag.gif file because we aren't the file's or enclosing directory's owner. Note that because we were the owner of the file "new_file", we were able to delete it. One of the directories that has its sticky bit set is "/tmp", a directory on the system that is writable by all users and is used to hold temporary files.
We won't discuss the other bits (setuid and setgid) here. So where did the name "sticky" bit come from? If commands are actually files, then each time someone wants to run the ls command, the system has to read the ls file from disk and place it into memory before it can run. This takes time. Because ls is a popular command and is run often, it would be faster for everybody if the ls file just remained in memory, that is, it were stuck in memory all the time. Programs with their sticky bit set got stuck in memory. Okay, that was an over-simplification but it's fairly accurate. These days, most systems ignore the the sticky bit for files and only use it for directories instead. But the name "sticky" bit has stuck.
Nifty Stuff
A Few More Useful Commands
Below are some more commands worth mentioning. Some of them, such as echo may seem very simplistic, but combined with other commands or included in shell scripts, they can be quite useful. For more detailed information on these commands, check the on-line manual pages with man.
echo — Output a line of text.
$ echo Howdy!
Howdy!
date — Prints the system date and time.
$ date
Fri Oct 25 13:46:28 CDT 2013
w — Show who is logged in and what they are doing. This is similar to the who command but displays more information. At the very top you can see how long it has been since the system was rebooted (also known as uptime) as well as how busy the system is (load average). For each user, you can see when they logged in (and from where), how long they've been idle, and the command they are currently running.
$ w
14:27:50 up 45 days, 44 min, 7 users, load average: 1.27, 1.18, 1.11
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root tty1 - 10Sep13 45days 0.01s 0.01s -bash
root pts/1 172.16.1.12 10:28 31:03 0.15s 0.15s -bash
betty pts/2 172.16.8.20 14:12 4:08 1.05s 0.02s less ./test.f
root :0 - 09Oct13 ?xdm? 2days 0.14s /usr/bin/gnome-
bobby pts/3 172.16.8.24 13:50 1.00s 0.05s 0.00s w
bubba pts/4 93.184.216.119 10:25 15:37 0.78s 0.04s man gcc
betty pts/5 172.16.8.20 10:31 24:20 0.07s 0.07s -bash
top — Display and update sorted information about processes. This command will continuously display the most CPU intensive programs currently running on the system. If the system seems slow, you can use top to see if something (or someone!) is hogging the CPU. When you are ready to quit top, press q.
$ top
top - 13:50:56 up 45 days, 8 min, 32 users, load average: 1.15, 1.14, 1.10
Tasks: 453 total, 1 running, 452 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.4%us, 0.9%sy, 0.0%ni, 97.6%id, 0.0%wa, 0.1%hi, 0.0%si, 0.0%st
Mem: 16435208k total, 5800124k used, 10635084k free, 33076k buffers
Swap: 16386292k total, 723220k used, 15663072k free, 206516k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20054 bobby 15 0 12996 1360 816 R 1.0 0.0 0:00.06 top
3407 betty 15 0 68292 2308 956 S 0.3 0.0 15:02.57 less
4080 bubba 15 0 56408 2592 1844 S 0.3 0.0 0:04.69 man
20064 betty 15 0 13776 2136 248 D 0.3 0.0 0:00.01 sshd
20065 betty 17 0 3812 460 368 S 0.3 0.0 0:00.01 wc
20066 betty 16 0 68808 2040 260 S 0.3 0.0 0:00.01 ./a.out
24945 bubba 15 0 92212 2264 840 S 0.3 0.0 35:50.09 sshd
1 root 15 0 10344 576 540 S 0.0 0.0 0:05.86 init
2 root RT -5 0 0 0 S 0.0 0.0 0:22.58 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:05.37 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT -5 0 0 0 S 0.0 0.0 2:39.97 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:07.07 ksoftirqd/1
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:05.05 migration/2
9 root 34 19 0 0 0 S 0.0 0.0 0:01.36 ksoftirqd/2
10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
wc — Print line, word, and character counts for a file.
$wc letter.txt27 189 1324 letter.txt $wc --lines letter.txt27 letter.txt
file — Determine file type.
$file *annual_report.pdf: PDF document, version 1.3 budget: ASCII text letter.txt: ASCII text $file /home/betty/shared/flag.gif/home/betty/shared/flag.gif: GIF image data, version 89a, 68 x 50 $file /tmp/tmp: sticky directory
gzip, gunzip — Compress or expand files. The gzip (GNU Zip) command will compress a file, add a .gz suffix to the compressed file, and then remove the old (uncompressed) file. The gunzip command does the opposite.
$lsposter.pdf report.pdf.bak $file *poster.pdf: PDF document, version 1.3 report.pdf.bak: PDF document, version 1.3 $gzip poster.pdf$ls -ltotal 28 -rw-r--r-- 1 bobby users 10956 Oct 17 14:30 poster.pdf.gz -rw-r--r-- 1 bobby users 12424 Oct 17 14:30 report.pdf.bak $file poster.pdf.gzposter.pdf.gz: gzip compressed data, was "poster.pdf", from Unix, last modified: Thu Oct 17 14:30:39 2013 $gunzip poster.pdf.gz$ls -ltotal 32 -rw-r--r-- 1 bobby users 12424 Oct 17 14:30 poster.pdf -rw-r--r-- 1 bobby users 12424 Oct 17 14:30 report.pdf.bak
hostname — Show the system's host name.
$ hostname
login001
ssh — Remote login program. You can use the ssh command to log in to the compute nodes if necessary.
[bobby@login001 ~]$hostnamelogin001 [bobby@login001 ~]$whoroot tty1 2013-09-10 13:48 betty pts/2 2013-10-02 13:48 (172.16.8.20) bobby pts/3 2013-10-02 13:51 (172.16.8.24) bubba pts/1 2013-10-02 08:16 (93.184.216.119) root :0 2013-10-09 15:39 [bobby@login001 ~]$ssh n007Last login: Thu Sep 12 12:59:49 2013 from 192.168.1.130 [bobby@n007 ~]$hostnamen007 [bobby@n007 ~]$whobobby pts/0 2013-10-25 15:10 (192.168.1.130) [bobby@n007 ~]$exitlogout Connection to n007 closed. [bobby@login001 ~]$
diff — Compare files line by line. A useful option is -y which displays the two files side by side.
grep — Search a file for a string or pattern. The grep command has an odd name but is very useful.
$cat budgetMonthly Income & Expense Estimates INCOME: Salary $3,000 Interest 50 Other Income 50 ------ Total Income $3,100 EXPENSES: Mortgage Payment $700 Auto Loan Payment 350 Student Load Payment 200 Food and Groceries 500 Utilities 250 Home insurance 120 Auto insurance 50 Life Insurance 20 Clothing 50 Retirement Fund 150 Education Fund 100 Auto Fuel and Maintenance 160 Entertainment 120 Miscellaneous 50 ------ Total Expenses $2,820 $grep Auto budgetAuto Loan Payment 350 Auto insurance 50 Auto Fuel and Maintenance 160
With the -n option, grep will display line numbers as well. To tell grep to ignore case when searching, add the -i option.
$grep -n Auto budget12: Auto Loan Payment 350 17: Auto insurance 50 22: Auto Fuel and Maintenance 160 $grep Insurance budgetLife Insurance 20 $grep -i Insurance budgetHome insurance 120 Auto insurance 50 Life Insurance 20
For some commands, it's obvious where their names came from. The who command shows who is logged in. The wc command displays a word count. So how in the world did a command that searches a file for some text get the name "grep"? Well, at first, the original programmers wanted to call the command "find" but there was already a command with that name. While they were sitting around trying to come up with a different name, through an open window they heard a bullfrog outside make a "grrrrrepppp" croaking sound and just decided to use "grep" for the name.
Okay, not really. So where did the name come from? Search patterns are sometimes called "regular expressions" (abbreviated "regex", "regexp", or often "re"). Regular expressions can get complicated so we won't go into detail about them here. But a very simple regular expression can just be the actual text you want search for. To search for the string "Auto", the regular expression would just be "Auto". Easy enough. There was (still is!) a line-oriented text editor called ed and you would type commands within ed to do various things. For example, to print lines 1 through 4, you would enter the command 1,4p. To search for the next line that contains some text and print it out, you would enter the command /text/p. To be more accurate, the text there is actually a regular expression so the general command to search for and print the next line that matches the regular expression would be /re/p. What if you wanted to search for and display all of the lines in the file that matches a regular expression? You would use the "global" version of the command, i.e., g/re/p. That's where the name "grep" comes from.
$ed letter.txt13241,4pLorem ipsum, Lorem ipsum dolor sit amet, et vel diam nominavi adversarium, utamur labores postulant eam in. Ad quo ubique delicatissimi, ei elitr eligendi has. Alia/quo/passum maluisset. Ea eruditi nominati ius, ea quo soleat bonorum salutandi,g/quo/ppostulant eam in. Ad quo ubique delicatissimi, ei elitr eligendi has. Alia assum maluisset. Ea eruditi nominati ius, ea quo soleat bonorum salutandi, in liber quodsi lobortis, quis altera essent at his.q$
By the way, although the story above about the bullfrog was made up, there is a similar story about how another program got its name. On some unix-like systems (BSD/Berkeley unix derived systems) you could run a command to enable notification when you received email. When email arrived for you, you would get a line displayed on your terminal telling you that "You have new mail from betty" or something like that. At U.C. Berkeley, where early versions of BSD unix were being developed, there was a dog named Biff who would bark at the mailman whenever the (paper) mail was delivered. The email notification command, biff, was named for that dog. Mac OS X is derived from BSD and does, in fact, have the biff command. If you have a Mac, open a Terminal.app window and run biff or man biff.
Standard Output and Standard Input
Stdout
As we have seen, commands have the useful habit of displaying stuff on the terminal window or screen. This output is called "standard output", often abbreviated as "stdout". The default destination for stdout is the terminal but you can redirect stdout to a file if necessary. You do this by adding "> filename" when running the command. For example, if you wanted to save the output from who to a file named "current_users"...
$lsannual_report.pdf budget letter.txt $who > current_users$lsannual_report.pdf budget current_users letter.txt $cat current_usersbobby pts/0 2013-10-30 13:20 (172.16.8.24) betty pts/1 2013-10-30 14:04 (172.16.8.20) bubba pts/2 2013-10-30 14:47 (93.184.216.119)
Be careful. If you redirect the output to an existing file with ">" you will overwrite the file with the new output. This is sometimes called "clobbering" and we'll show how to prevent it when we talk about scripts below.
So what if you need to append to the file instead? You would use ">> filename" (two ">"s).
$lsannual_report.pdf budget letter.txt $echo 'Users...' > current_users$cat current_usersUsers... $date >> current_users$who >> current_users$echo 'And what they are doing...' >> current_users$w >> current_users$cat current_usersUsers... Thu Oct 31 09:23:31 CDT 2013 bobby pts/0 2013-10-30 13:20 (172.16.8.24) betty pts/1 2013-10-30 15:09 (172.16.8.20) bubba pts/2 2013-10-30 14:47 (93.184.216.119) And what they are doing... 09:23:53 up 6 days, 23:01, 3 users, load average: 0.33, 0.39, 0.27 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT bobby pts/0 172.16.8.24 Wed13 0.00s 0.05s 0.00s w betty pts/1 172.16.8.20 Wed15 16:18m 0.00s 0.00s -bash bubba pts/2 93.184.216.119 Wed14 52.00s 0.14s 0.13s top
Stdin
Similar to standard output is "standard input" or "stdin". This is often you typing away at the keyboard. Recall our fictional program howdy from above. If you add a -q optin, the program will query the user for a name.
$howdyHowdy, bobby! $howdy -qYour name?Tim the EnchanterGreetings, Tim the Enchanter!
The howdy program is reading input from stdin, in this case, the user typing a name. But like stdout, you can redirect from a file with < filename. When you redirect stdin from a file, it may or may not be echoed to stdout. That just depends on the program itself.
$cat my_real_nameRobert M. Baylor $howdy -q < my_real_nameYour name? Greetings, Robert M. Baylor!
Often, programs that accept a file as a command line argument will use stdin if no file is specified. For example, we could use wc to count the number of characters typed at the keyboard. When entering text this way, you will need a way to tell the system that you are finished typing. Typically, you will press ^D to do this.
Below, we see an example of different ways of counting the number of characters in a sentence. The first method just specifies a file. The second method redirects stdin from that file. The commands look similar (the only difference is the < character) but way wc is reading the text is different. Notice that when you specify a file, wc includes the name of the file in the output. When stdin is redirected from the file, wc doesn't know from where the text originated so it can't print out the name of the file. The third method is using stdin from the keyboard.
$cat sentence.txtThe quick brown fox jumped over the lazy dogs. $wc -c sentence.txt47 sentence.txt $wc -c < sentence.txt47 $wc -cThe quick brown fox jumped over the lazy dogs.^D47
So a program that accepts a file name as an argument may also use standard input, e.g., the user typing at the keyboard, if the file is omitted. But users aren't going to type an entire document just to count the number of characters in it. So seriously... is this really all that useful?
Pipes and Piping
Why, yes. Yes it is. It is possible to chain multiple commands together so that one program's output (stdout) is another program's input (stdin). This is known as "piping" and is done using the "|" vertical bar character, aka "pipe".
The ps (process status) command will print out information about processes currently running on the system. There are many, many options to ps but we will just use -ef which means shows full info for processes for all users.
$ ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Oct24 ? 00:00:02 /sbin/init
root 2 0 0 Oct24 ? 00:00:00 [kthreadd]
root 3 2 0 Oct24 ? 00:00:00 [migration/0]
root 4 2 0 Oct24 ? 00:00:00 [ksoftirqd/0]
root 5 2 0 Oct24 ? 00:00:00 [migration/0]
root 6 2 0 Oct24 ? 00:00:00 [watchdog/0]
root 7 2 0 Oct24 ? 00:00:00 [migration/1]
root 8 2 0 Oct24 ? 00:00:00 [migration/1]
root 9 2 0 Oct24 ? 00:00:03 [ksoftirqd/1]
root 10 2 0 Oct24 ? 00:00:00 [watchdog/1]
root 11 2 0 Oct24 ? 00:00:00 [migration/2]
root 12 2 0 Oct24 ? 00:00:00 [migration/2]
root 13 2 0 Oct24 ? 00:00:00 [ksoftirqd/2]
root 14 2 0 Oct24 ? 00:00:00 [watchdog/2]
root 15 2 0 Oct24 ? 00:00:00 [migration/3]
root 16 2 0 Oct24 ? 00:00:00 [migration/3]
root 17 2 0 Oct24 ? 00:00:00 [ksoftirqd/3]
root 18 2 0 Oct24 ? 00:00:00 [watchdog/3]
root 19 2 0 Oct24 ? 00:00:12 [events/0]
root 20 2 0 Oct24 ? 00:00:10 [events/1]
root 21 2 0 Oct24 ? 00:00:12 [events/2]
root 22 2 0 Oct24 ? 00:00:13 [events/3]
root 23 2 0 Oct24 ? 00:00:00 [cgroup]
root 24 2 0 Oct24 ? 00:00:00 [khelper]
root 25 2 0 Oct24 ? 00:00:00 [netns]
root 26 2 0 Oct24 ? 00:00:00 [async/mgr]
root 27 2 0 Oct24 ? 00:00:00 [pm]
root 28 2 0 Oct24 ? 00:00:00 [sync_supers]
root 29 2 0 Oct24 ? 00:00:01 [bdi-default]
root 30 2 0 Oct24 ? 00:00:00 [kintegrityd/0]
root 31 2 0 Oct24 ? 00:00:00 [kintegrityd/1]
root 32 2 0 Oct24 ? 00:00:00 [kintegrityd/2]
root 33 2 0 Oct24 ? 00:00:00 [kintegrityd/3]
root 34 2 0 Oct24 ? 00:00:00 [kblockd/0]
root 35 2 0 Oct24 ? 00:00:00 [kblockd/1]
root 36 2 0 Oct24 ? 00:00:00 [kblockd/2]
root 37 2 0 Oct24 ? 00:00:00 [kblockd/3]
root 38 2 0 Oct24 ? 00:00:00 [kacpid]
root 39 2 0 Oct24 ? 00:00:00 [kacpi_notify]
root 40 2 0 Oct24 ? 00:00:00 [kacpi_hotplug]
root 41 2 0 Oct24 ? 00:00:00 [ata/0]
root 42 2 0 Oct24 ? 00:00:00 [ata/1]
root 43 2 0 Oct24 ? 00:00:00 [ata/2]
root 44 2 0 Oct24 ? 00:00:00 [ata/3]
root 45 2 0 Oct24 ? 00:00:00 [ata_aux]
root 46 2 0 Oct24 ? 00:00:00 [ksuspend_usbd]
root 47 2 0 Oct24 ? 00:00:00 [khubd]
root 48 2 0 Oct24 ? 00:00:00 [kseriod]
root 49 2 0 Oct24 ? 00:00:00 [md/0]
root 50 2 0 Oct24 ? 00:00:00 [md/1]
root 51 2 0 Oct24 ? 00:00:00 [md/2]
root 52 2 0 Oct24 ? 00:00:00 [md/3]
root 53 2 0 Oct24 ? 00:00:00 [md_misc/0]
root 54 2 0 Oct24 ? 00:00:00 [md_misc/1]
root 55 2 0 Oct24 ? 00:00:00 [md_misc/2]
root 56 2 0 Oct24 ? 00:00:00 [md_misc/3]
root 57 2 0 Oct24 ? 00:00:00 [khungtaskd]
root 58 2 0 Oct24 ? 00:00:00 [kswapd0]
root 59 2 0 Oct24 ? 00:00:00 [ksmd]
root 60 2 0 Oct24 ? 00:00:03 [khugepaged]
root 61 2 0 Oct24 ? 00:00:00 [aio/0]
root 62 2 0 Oct24 ? 00:00:00 [aio/1]
root 63 2 0 Oct24 ? 00:00:00 [aio/2]
root 64 2 0 Oct24 ? 00:00:00 [aio/3]
root 65 2 0 Oct24 ? 00:00:00 [crypto/0]
root 66 2 0 Oct24 ? 00:00:00 [crypto/1]
root 67 2 0 Oct24 ? 00:00:00 [crypto/2]
root 68 2 0 Oct24 ? 00:00:00 [crypto/3]
root 73 2 0 Oct24 ? 00:00:00 [kthrotld/0]
root 74 2 0 Oct24 ? 00:00:00 [kthrotld/1]
root 75 2 0 Oct24 ? 00:00:00 [kthrotld/2]
root 76 2 0 Oct24 ? 00:00:00 [kthrotld/3]
root 78 2 0 Oct24 ? 00:00:00 [kpsmoused]
root 79 2 0 Oct24 ? 00:00:00 [usbhid_resumer]
root 110 2 0 Oct24 ? 00:00:00 [kstriped]
root 154 2 0 Oct24 ? 00:00:00 [ttm_swap]
root 155 2 0 Oct24 ? 00:00:00 [kslowd000]
root 156 2 0 Oct24 ? 00:00:00 [kslowd001]
root 237 2 0 Oct24 ? 00:00:00 [scsi_eh_0]
root 238 2 0 Oct24 ? 00:00:00 [scsi_eh_1]
root 297 2 0 Oct24 ? 00:00:00 [scsi_eh_2]
root 298 2 0 Oct24 ? 00:00:00 [scsi_eh_3]
root 299 2 0 Oct24 ? 00:00:33 [scsi_eh_4]
root 300 2 0 Oct24 ? 00:00:33 [scsi_eh_5]
root 301 2 0 Oct24 ? 00:00:00 [scsi_eh_6]
root 302 2 0 Oct24 ? 00:00:00 [scsi_eh_7]
root 366 2 0 Oct24 ? 00:00:00 [kdmflush]
root 368 2 0 Oct24 ? 00:00:00 [kdmflush]
root 386 2 0 Oct24 ? 00:00:03 [jbd2/dm-0-8]
root 387 2 0 Oct24 ? 00:00:00 [ext4-dio-unwrit]
root 388 2 0 Oct24 ? 00:00:00 [ext4-dio-unwrit]
root 389 2 0 Oct24 ? 00:00:00 [ext4-dio-unwrit]
root 390 2 0 Oct24 ? 00:00:00 [ext4-dio-unwrit]
root 433 2 0 Oct24 ? 00:00:00 [kauditd]
root 483 1 0 Oct24 ? 00:00:00 /sbin/udevd -d
root 708 1925 0 11:26 ? 00:00:00 sleep 60
root 709 1781 0 11:26 ? 00:00:00 sshd: bobby [priv]
bobby 711 709 0 11:26 ? 00:00:00 sshd: bobby@pts/0
bobby 712 711 0 11:26 pts/0 00:00:00 -bash
root 735 2 0 Oct24 ? 00:00:00 [hd-audio0]
bobby 741 712 0 11:26 pts/0 00:00:00 ps -ef
root 985 483 0 Oct24 ? 00:00:00 /sbin/udevd -d
root 988 2 0 Oct24 ? 00:00:00 [kvm-irqfd-clean]
root 1030 2 0 Oct24 ? 00:00:00 [jbd2/sda1-8]
root 1031 2 0 Oct24 ? 00:00:00 [ext4-dio-unwrit]
root 1032 2 0 Oct24 ? 00:00:00 [ext4-dio-unwrit]
root 1033 2 0 Oct24 ? 00:00:00 [ext4-dio-unwrit]
root 1034 2 0 Oct24 ? 00:00:00 [ext4-dio-unwrit]
root 1140 2 0 Oct24 ? 00:00:03 [flush-253:0]
root 1415 1 0 Oct24 ? 00:00:00 /sbin/rsyslogd -i /var/run/syslo
root 1438 2 0 Oct24 ? 00:00:18 [kondemand/0]
root 1439 2 0 Oct24 ? 00:00:10 [kondemand/1]
root 1440 2 0 Oct24 ? 00:00:13 [kondemand/2]
root 1441 2 0 Oct24 ? 00:00:14 [kondemand/3]
root 1462 1 0 Oct24 ? 00:00:34 irqbalance
rpc 1481 1 0 Oct24 ? 00:00:00 rpcbind
dbus 1496 1 0 Oct24 ? 00:00:00 dbus-daemon --system
root 1506 1 0 Oct24 ? 00:00:01 NetworkManager --pid-file=/var/r
root 1513 1 0 Oct24 ? 00:00:00 /usr/sbin/modem-manager
avahi 1518 1 0 Oct24 ? 00:00:00 avahi-daemon: running [login001
avahi 1519 1518 0 Oct24 ? 00:00:00 avahi-daemon: chroot helper
rpcuser 1538 1 0 Oct24 ? 00:00:00 rpc.statd
root 1547 1506 0 Oct24 ? 00:00:00 /sbin/dhclient -d -4 -sf /usr/li
root 1553 1 0 Oct24 ? 00:00:00 /usr/sbin/wpa_supplicant -c /etc
root 1624 2 0 Oct24 ? 00:00:00 [rpciod/0]
root 1625 2 0 Oct24 ? 00:00:00 [rpciod/1]
root 1626 2 0 Oct24 ? 00:00:00 [rpciod/2]
root 1627 2 0 Oct24 ? 00:00:00 [rpciod/3]
root 1631 1 0 Oct24 ? 00:00:00 rpc.idmapd
root 1646 1 0 Oct24 ? 00:00:00 cupsd -C /etc/cups/cupsd.conf
root 1657 1 0 Oct24 ? 00:00:00 /usr/sbin/acpid
68 1666 1 0 Oct24 ? 00:00:02 hald
root 1667 1666 0 Oct24 ? 00:00:00 hald-runner
root 1705 1667 0 Oct24 ? 00:00:00 hald-addon-input: Listening on /
68 1706 1667 0 Oct24 ? 00:00:00 hald-addon-acpi: listening on ac
root 1707 1667 0 Oct24 ? 00:00:31 hald-addon-storage: polling /dev
root 1708 1667 0 Oct24 ? 00:00:33 hald-addon-storage: polling /dev
root 1732 1 0 Oct24 ? 00:00:00 pcscd
root 1748 1 0 Oct24 ? 00:00:02 automount --pid-file /var/run/au
root 1769 1 0 Oct24 ? 00:00:00 /usr/sbin/mcelog --daemon
root 1781 1 0 Oct24 ? 00:00:00 /usr/sbin/sshd
root 1789 1 0 Oct24 ? 00:00:00 xinetd -stayalive -pidfile /var/
ntp 1797 1 0 Oct24 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd
root 1879 1 0 Oct24 ? 00:00:01 /usr/libexec/postfix/master
postfix 1887 1879 0 Oct24 ? 00:00:00 qmgr -l -t fifo -u
root 1903 1 0 Oct24 ? 00:00:00 /usr/sbin/abrtd
root 1911 1 0 Oct24 ? 00:00:00 abrt-dump-oops -d /var/spool/abr
root 1925 1 0 Oct24 ? 00:00:13 /bin/bash /usr/sbin/ksmtuned
qpidd 1937 1 0 Oct24 ? 00:00:40 /usr/sbin/qpidd --data-dir /var/
root 1979 1 0 Oct24 ? 00:00:01 crond
root 1990 1 0 Oct24 ? 00:00:00 /usr/sbin/atd
root 2003 1 0 Oct24 ? 00:01:59 /usr/bin/python /usr/share/virt-
root 2018 1 0 Oct24 ? 00:01:35 libvirtd --daemon
root 2048 1 0 Oct24 ? 00:00:00 rhnsd
root 2102 483 0 Oct24 ? 00:00:00 /sbin/udevd -d
root 2107 1 0 Oct24 ? 00:00:00 /usr/bin/rhsmcertd
root 2135 1 0 Oct24 ? 00:00:00 /usr/sbin/gdm-binary -nodaemon
root 2142 1 0 Oct24 tty2 00:00:00 /sbin/mingetty /dev/tty2
root 2144 1 0 Oct24 tty3 00:00:00 /sbin/mingetty /dev/tty3
root 2148 1 0 Oct24 tty4 00:00:00 /sbin/mingetty /dev/tty4
root 2150 1 0 Oct24 tty5 00:00:00 /sbin/mingetty /dev/tty5
root 2152 1 0 Oct24 tty6 00:00:00 /sbin/mingetty /dev/tty6
nobody 2158 1 0 Oct24 ? 00:00:00 /usr/sbin/dnsmasq --strict-order
root 2183 2135 0 Oct24 ? 00:00:00 /usr/libexec/gdm-simple-slave --
root 2194 2183 0 Oct24 tty1 00:00:22 /usr/bin/Xorg :0 -nr -verbose -a
root 2242 1 0 Oct24 ? 00:00:00 /usr/sbin/console-kit-daemon --n
gdm 2312 1 0 Oct24 ? 00:00:00 /usr/bin/dbus-launch --exit-with
gdm 2313 1 0 Oct24 ? 00:00:00 /bin/dbus-daemon --fork --print-
gdm 2314 2183 0 Oct24 ? 00:00:00 /usr/bin/gnome-session --autosta
root 2317 1 0 Oct24 ? 00:00:00 /usr/libexec/devkit-power-daemon
gdm 2321 1 0 Oct24 ? 00:00:01 /usr/libexec/gconfd-2
gdm 2339 2314 0 Oct24 ? 00:00:01 /usr/libexec/at-spi-registryd
gdm 2340 1 0 Oct24 ? 00:10:16 /usr/libexec/gnome-settings-daem
gdm 2342 1 0 Oct24 ? 00:00:00 /usr/libexec/bonobo-activation-s
gdm 2349 1 0 Oct24 ? 00:00:00 /usr/libexec/gvfsd
gdm 2350 2314 0 Oct24 ? 00:00:02 metacity
gdm 2352 2314 0 Oct24 ? 00:00:00 /usr/libexec/polkit-gnome-authen
gdm 2353 2314 0 Oct24 ? 00:00:03 gnome-power-manager
root 2355 1 0 Oct24 ? 00:00:00 /usr/libexec/polkit-1/polkitd
gdm 2357 2314 0 Oct24 ? 00:00:11 /usr/libexec/gdm-simple-greeter
gdm 2379 1 0 Oct24 ? 00:00:00 /usr/bin/pulseaudio --start --lo
rtkit 2381 1 0 Oct24 ? 00:00:03 /usr/libexec/rtkit-daemon
root 2388 1 0 Oct24 ? 00:00:00 auditd
root 2404 2183 0 Oct24 ? 00:00:00 pam: gdm-password
root 22385 1781 0 Oct30 ? 00:00:00 sshd: bubba [priv]
bubba 22387 22385 0 Oct30 ? 00:00:00 sshd: bubba@pts/2
bubba 22388 22387 0 Oct30 pts/2 00:00:00 -bash
root 22629 1781 0 Oct30 ? 00:00:00 sshd: betty [priv]
betty 22631 22629 0 Oct30 ? 00:00:00 sshd: betty@pts/1
betty 22632 22631 0 Oct30 pts/1 00:00:00 -bash
bubba 31903 22388 0 09:23 pts/2 00:00:17 top
postfix 32714 1879 0 10:51 ? 00:00:00 pickup -l -t fifo -u
That's a lot of output. Let's say we want to get the processes related to user bubba. (There's actually a -u user option for ps but we're not going to use that here.) We could redirect the output from ps to a temporary file and grep that file for "bubba". We would need to be sure to remember to delete the file when we were finished with it.
$ps -ef > /tmp/current_processes$grep bubba /tmp/current_processesroot 22385 1781 0 Oct30 ? 00:00:00 sshd: bubba [priv] bubba 22387 22385 0 Oct30 ? 00:00:00 sshd: bubba@pts/2 bubba 22388 22387 0 Oct30 pts/2 00:00:00 -bash bubba 31903 22388 0 09:23 pts/2 00:00:16 top $rm /tmp/current_processes
Fortunately, grep will use stdin if we don't specify a file. So instead let's simplify the above into a single command.
$ ps -ef | grep bubba
bobby 661 21591 0 11:21 pts/0 00:00:00 grep bubba
root 22385 1781 0 Oct30 ? 00:00:00 sshd: bubba [priv]
bubba 22387 22385 0 Oct30 ? 00:00:00 sshd: bubba@pts/2
bubba 22388 22387 0 Oct30 pts/2 00:00:00 -bash
bubba 31903 22388 0 09:23 pts/2 00:00:16 top
Notice that in addition to bubba's processes, this included two others - our grep command and some command running as user root. This is because the grep command is searching for the text "bubba" within stdin, not listing bubba's processes.
It's also possible to chain multiple commands using pipes. To count the number of "bubba related processes" pipe the above output into wc -l (line count).
$ ps -ef | grep bubba | wc -l
5
Stderr
We'll mention one more issue concerning input/output redirection. What if we wanted to save the word count of all the files in a directory to a file, i.e., redirect the output of wc * to the file "word_counts"? We know how to do that...
$ls -ltotal 36 -rw-r--r-- 1 bobby users 12424 Oct 17 14:30 annual_report.pdf -rw------- 1 bobby users 1065 Oct 28 10:01 budget -rw-r--r-- 1 bobby users 594 Oct 31 09:24 current_users -rw-r----- 1 bobby users 1324 Oct 17 14:30 letter.txt -rw-r--r-- 1 bobby users 18 Oct 30 13:23 my_name -rw-r--r-- 1 bobby users 47 Oct 31 09:55 sentence.txt $wc * > word_counts$cat word_counts175 771 12424 annual_report.pdf 27 65 1065 budget 11 71 594 current_users 27 189 1324 letter.txt 1 3 18 my_name 1 9 47 sentence.txt 242 1108 15472 total
That worked as expected. Now it just so happens that the wc command does not like directories and it will let you know if an item is a directory. Let's create one and see what happens.
$mkdir misc$ls -ltotal 44 -rw-r--r-- 1 bobby users 12424 Oct 17 14:30 annual_report.pdf -rw------- 1 bobby users 1065 Oct 28 10:01 budget -rw-r--r-- 1 bobby users 594 Oct 31 09:24 current_users -rw-r----- 1 bobby users 1324 Oct 17 14:30 letter.txt drwxr-xr-x 2 bobby users 4096 Oct 31 14:08 misc -rw-r--r-- 1 bobby users 18 Oct 30 13:23 my_name -rw-r--r-- 1 bobby users 47 Oct 31 09:55 sentence.txt $wc * > word_countswc: misc: Is a directory $cat word_counts175 771 12424 annual_report.pdf 27 65 1065 budget 11 71 594 current_users 27 189 1324 letter.txt 0 0 0 misc 1 3 18 my_name 1 9 47 sentence.txt 242 1108 15472 total
The wc command displayed an error message. But we were directing output to a file so why did the error message display on the terminal and not get included in the file? Because in addition to standard output, there is a separate output stream for errors: "standard error" or "stderr". To redirect stderr to a file, you would use "2> filename". Note that although you can specify the same file for both stdout and stderr, you shouldn't because the file will likely end up corrupted.
$wc * > word_counts 2> word_counts.err$cat word_counts.errwc: misc: Is a directory
Why "2"? Where did that come from? These input and output streams are actually unix file descriptors and file descriptors 0, 1, and 2, are reserved for stdin, stdout, and sterr respectively. When you redirect from the command line, you are really redirecting the data for those file descriptors. So by adding "2> filename" we are redirecting data from file descriptor 2 (stderr) to the file "filename". We could actually redirect stdout with "1> filename" if we wanted but the 1 is implied and is omitted.
What if we know that wc doesn't like directories and so we don't really care about error messages and just want to ignore stderr completely. We could always redirect stderr to some file but we would still to deal with leftover error files. A better way is to redirect stderr to a special file on the system, "/dev/null". The /dev/null file is sometimes called the "null device" and just discards any data written to it.
$ wc * 2> /dev/null
175 771 12424 annual_report.pdf
27 65 1065 budget
11 71 594 current_users
27 189 1324 letter.txt
0 0 0 misc
1 3 18 my_name
1 9 47 sentence.txt
242 1108 15472 total
The wc command actually did output the error message, but it was redirected to /dev/null and so it just disappeared. By the way, it's also possible to redirect stdout to /dev/null. If we wanted to ignore any word counts but did want to see error messages...
$ wc * > /dev/null
wc: misc: Is a directory
The Shell and Scripts
The Shell
We have seen references above to something called a "shell". A shell is the command-line interpreter UI for unix like systems. When you are typing commands at the command prompt, you are actually running a shell program. Above, we also discussed wildcards, I/O redirection, and pipes. It turns out that those features are actually provided through the shell and not individual programs.
Traditionally, there are two major shell programs - the "Bourne shell" (sh) and the "C shell" (csh). Over time, these shells have been replaced by other shell programs such as the "Bourne-Again shell" (bash). These newer shells provide compatibility support for the older shells but have many other added features as well. You will still see references to sh and csh.
The bash shell is the default shell for Kodiak users. One of the useful enhancements that bash provides is command autocompletion. As you are typing a command at the prompt, you can press the tab key and the shell will use the current context, such as the items in the current directory, and complete the word for you.
$lsbackups documents downloads pictures shared $ls doc[tab]$ ls documents/ annual_report.pdf current_users misc sentence.txt budget letter.txt my_name
If there are multiple possibilities for autocompletion, the shell will show them if you press the tab key twice. Type a bit more to make it unambiguous and press tab again to finish. (If it doesn't display anything after you press tab multiple times, there are no matches.)
$ls do[tab][tab]documents/ downloads/ $ ls dow[tab]$ ls downloads/ poster.pdf report.pdf.bak
You can also use the arrow keys to recall previous commands and edit the command line. Pressing the up arrow ↑ will step backwards through the history of previous commands that you have entered. Press the down arrow ↓ to step forward through previous commands. You can use the left ← and right → arrows to move the cursor back and forth to edit the current line. A quick way to move the cursor the beginning of the line is press ^A. Press ^E to move it to the end of the line. There are many more keyboard shortcuts besides these. To see them, along with a lot more info, run man bash.
Scripts
The shell is not just a command-line user interface that runs commands one at a time. It is possible to place multiple commands in a text file, called a "script" or "shell script" and run them all by running that script as a command. But wait! There's more! The shell doesn't just run commands. It also provides many features in "real" programming languages, such as variables, control-flow constructs and loops, subroutines, etc. Although shell scripts are executable, they are interpreted and not compiled. When you run your program on Kodiak (discussed in Part 2) you will actually be running a shell script which then runs your program.
Not all scripts are unix shell scripts. There are other scripting languages, e.g., Perl, Python, and Tcl, as well. But we are only going to discuss shell scripts here.
So let's take a look at a simple shell script. Below is a script that outputs the current date and time (date), current users (w) and a blank line (echo), then waits for 30 seconds before continuing (sleep 30). It does this three times and quits.
$pwd/home/bobby/scripts $cat list_users.sh#!/bin/sh date w echo sleep 30 date w echo sleep 30 date w $sh list_users.shThu Nov 7 09:47:28 CST 2013 bobby pts/0 172.16.8.24 0.00s sh list_users.sh betty pts/1 172.16.8.20 4:33 -bash bubba pts/2 93.184.216.119 4:04 top Thu Nov 7 09:47:58 CST 2013 bobby pts/0 172.16.8.24 30.00s sh list_users.sh bubba pts/2 93.184.216.119 4:34 top Thu Nov 7 09:48:28 CST 2013 bobby pts/0 172.16.8.24 60.00s sh list_users.sh bubba pts/2 93.184.216.119 2.00s -bash
The first thing to point out is that the script file name's suffix is ".sh". But this is by convention and not really required. You may also see other suffixes for scripts as well, such as ".csh" and ".py". Ignore the first line of the script for now. The rest of the script are just commands that get run as if you had typed them at the prompt.
When we ran the script, we actually ran the sh command with the script file as an argument. Unsurprisingly, the command we ran (sh list_users.sh) is what is actually listed by the w command within the script. Now you may recall that earlier we said that the default shell on Kodiak was bash. So why did we use the Bourne shell sh instead if bash? Like many systems, on Kodiak the sh command is actually a symbolic link to the bash command. A symbolic link is special kind of file that points to or is linked to another file or directory, the target. When you access the symbolic link, the system automatically uses the target file or directory instead. There would have been no difference if we had used bash list_users.sh to run the script instead.
$which sh/bin/sh $ls -l /bin/shlrwxrwxrwx 1 root root 4 Aug 21 2009 /bin/sh -> bash $file /bin/sh/bin/sh: symbolic link to `bash' $ls -l /bin/bash-rwxr-xr-x 1 root root 801512 Jan 21 2009 /bin/bash
Now, let's modify the script a bit. We want to print the date and users just once, not three times. We could delete the second and third sets of commands but we might want to go back to printing three times later. So instead we'll "comment out" the remaining lines. We do this by inserting a "#" character at the beginning of a line. When the shell sees a "#" it considers everything on the line after it a comment and ignores it.
$ cat list_users.sh
#!/bin/sh
date
w
echo
# Comment out the remaining lines...
#sleep 30
#date
#w
#echo
#sleep 30
#date
#w
Now we want to run the script as if it were a regular command. Notice below that when we first try to run the script from the prompt, bash (our command line interface shell) gives us a "Permission denied" error. The first thing we need to do is make sure that the file is executable. While we're at it, we're going to rename the file and get rid of the .sh suffix to show we really don't need it. Also note that to run the list_users script, we still must do so with a path to the file by putting a "./" in front of it.
$./list_users.sh-bash: ./list_users.sh: Permission denied $ls -ltotal 4 -rw-r--r-- 1 bobby users 135 Nov 7 13:40 list_users.sh $chmod u+x list_users.sh-rwxr--r-- 1 bobby users 135 Nov 7 13:40 list_users.sh $mv list_users.sh list_users$./list_usersFri Nov 8 11:40:58 CST 2013 bobby pts/0 172.16.8.24 0.00s /bin/sh ./list_users bubba pts/1 93.184.216.119 16.00s -bash betty pts/2 172.16.8.20 4.00s man make
You can see above that although we ran ./list_users, the system is actually running /bin/sh ./list_users. So how exactly did it know we wanted to use sh (really, bash)? Because of the first line of the script, "#!/bin/sh". This is a special comment, sometimes called a "shebang", that tells the system what program to use to interpret the script. You might be able to get by without including a shebang line because your current interactive shell will then try to interpret the script. But to prevent compatibility issues in the future, it's good practice to always include the shebang line. Because this script is very simple and doesn't take advantage of any bash specific features, we use #!/bin/sh (the "lowest common denominator" shell) for portability and compatibility in the future. If the script did require the bash shell, we would use #!/bin/bash as the shebang line.
Why is that called a "shebang"? We saw earlier that the vertical bar "|" character is called "pipe". The "#" character is sometimes called "hash" (you may be familiar with the term "#hashtag"...) and the "!" character is sometimes called a "bang". So "#!" is sometimes called hash-bang but that is usually shortened to shebang. By the way, the "*" character is sometimes called a "splat".
PATH and Other Environment Variables
We've seen where to run a program we sometimes were required to specify it with its absolute or relative path. To run the list_users script (which happens to be in the directory /home/bobby/scripts), we must explicitly include the path to it (e.g., /home/bobby/scripts/list_users or ./list_users). So why do some programs work with just their names while others require a path? Because of the $PATH environment variable.
Like "regular" programming languages, the shell provides and uses variables or "environment variables". You can echo them to see their values. Remember how "~" is a shortcut for your home directory? There is an environment variable for this as well: $HOME.
$echo $HOME/home/bobby $ls $HOME/documentsannual_report.pdf current_users misc sentence.txt budget letter.txt my_name
To see a list of variables use the printenv, env, or set commands. You can create and set your own variables which is especially useful within shell scripts.
$echo $howdy$howdy=doody$echo $howdydoody
So back to the $PATH variable. A path variable is a list of directory paths, both absolute and relative, that the system will search for programs you run when you don't include the path in the command. The directories are separated by colon (:) characters. Shown below is a default $PATH. (Actually, the default $PATH on Kodiak contains many more directories...)
$ echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:
/sbin:/home/bobby/bin
So when you run the command list_users, the system searches each of the above directories, in order, for the executable file "list_users". Once it finds one, it runs it. If it doesn't find one, it will fail with a "command not found" error. Right now, the list_users script is located in /home/bobby/scripts which is not in $PATH. But we can see that the directory /home/bobby/bin is there. (Notice that there are several "bin" directories. These are standard locations for executable programs or binaries.) So if we move list_users to /home/bobby/bin, it should work.
$which list_users/usr/bin/which: no list_users in (/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin: /usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/bobby/bin) $mv /home/scripts/list_users /home/bin/$which list_users~/bin/list_users $list_usersMon Nov 11 14:26:21 CST 2013 bobby pts/0 172.16.8.24 0.00s /bin/sh /home/bobby/bin/list_users
But suppose we don't want to move list_users to our ~/bin directory, but want to keep it in the ~/scripts directory. We can tell the system to include /home/bobby/scripts when searching by appending it to the current $PATH.
$which list_users/usr/bin/which: no list_users in (/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin: /usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/bobby/bin) $echo $PATH/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin: /sbin:/home/bobby/bin $PATH=$PATH:/home/bobby/scripts$echo $PATH/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin: /sbin:/home/bobby/bin:/home/bobby/scripts $which list_users~/scripts/list_users
Because any changes made to environment variables disappear once the shell script exits or when logging out, we must remember to set them back each time. This would be a major hassle, but fortunately there is a solution: startup scripts. These are hidden files, located in your home directory, that run when you first log in or start a shell. The bash man page actually calls them "personal initialization files". We'll stick with "startup scripts".
$ ls -la $HOME
total 100
drwx------. 13 bobby users 4096 Nov 11 14:49 .
drwxr-xr-x. 6 root root 4096 Oct 17 14:05 ..
drwxr-x--- 2 bobby users 4096 Oct 25 14:09 backups
-rw-r--r--. 1 bobby users 18 Jul 9 08:24 .bash_logout
-rw-r--r--. 1 bobby users 176 Jul 9 08:24 .bash_profile
-rw-r--r-- 1 bobby users 150 Nov 11 14:49 .bashrc
drwxr-xr-x 2 bobby users 4096 Nov 11 14:36 bin
drwx--x--x. 3 bobby users 4096 Oct 31 14:47 documents
drwxr-x---. 2 bobby users 4096 Oct 25 14:10 downloads
drwxr-x---. 4 bobby users 4096 Oct 25 08:56 pictures
drwxr-xr-x 2 bobby users 4096 Nov 11 14:36 scripts
drwxr-x--x 2 bobby users 4096 Oct 21 12:40 shared
For bash the startup scripts are .bash_profile and .bashrc. Why two files? What's the difference? The .bash_profile script is executed for login shells, that is, you had to enter your username and password. The .bashrc script is executed for interactive non-login shells. Although these are mutually exclusive, as we'll soon see, the .bashrc script is run by the default .bash_profile so anything in .bashrc will also get run when you log in.
Right now, we'll look at .bash_profile, the script that runs whenever you log in.
$ cat .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
Notice that the first line is not a shebang, but just a regular comment line. This script is only run by bash so the shebang isn't needed. The second bit of code is what runs the .bashrc file (after first checking if it exists). Don't modify that part unless you know what you are doing.
The last part is the section that we care about. See the line "PATH=$PATH:$HOME/bin"? What that does is take the current, default, PATH variable (already set by system-wide startup scripts) and append ":$HOME/bin" to it. That's why we see /home/bobby/bin at the end when we echo $PATH. Recall that what we want is to be able to run scripts in the /home/bobby/scripts directory so we just need to append ":$HOME/scripts" to the end of that line. When changing a path variable like $PATH, always remember to separate directories with ':'. Also, you almost always want to modify PATH, either appending or prepending directories, so be sure and include the current value of $PATH there. If we were to leave out $PATH, (e.g., PATH=$HOME/bin:$HOME/scripts) then only those two directories would be in the current path and regular commands, such as ls, would not be found. Note that if you do this, you would still be able to run commands by using their full paths (e.g., /bin/ls). For any changes to .bash_profile to take affect, log out and log back in.
At some point, you might be tempted to add the current working directory "." to PATH. That way you wouldn't have to always use "./" in front of programs and scripts to run them. That sounds like it would be convenient, right? Don't do it. Adding your current directory to your PATH is potentially dangerous, especially if it appears before the system-wide directories. Remember that the directories in $PATH are searched in order. What if some ne'er-do-well were to write a script that deleted all of the files and directories in your account? And what if he named the script "ls" and placed it in the /tmp directory? If you were to cd /tmp and ls and if "." were at the beginning of your PATH, the system would find the malicious script first, run it instead of the real ls command, and all of your files would get deleted. It's safer to just type "./" before your scripts.
One more thing to mention before we do so. Note the last line in the script export PATH. By default, variables are only set and usable in the environment (shell or script) where they are created. To make them usable in sub- or child processes, that is, programs and scripts run by the current environment, they must be exported. If you make multiple changes to a variable, you can do that with multiple lines and export it after the final change. If you just need to set a variable or make one change, you could set and export it on the same line.
PATH=$PATH:$HOME/bin PATH=$PATH:$HOME/scripts export PATH export PRINTER="Some_Printer"
$cat .bash_profile# .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs PATH=$PATH:$HOME/bin:$HOME/scripts export PATH[Log out and log back in...]$echo $PATH/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin: /sbin:/home/bobby/bin:/home/bobby/scripts $which list_users~/scripts/list_users
The .bash_profile script is used to set default settings such as your $PATH variable whenever you log in. You can run other commands as well. For example, if you wanted to list the current users when you log in, add a who or w to the script. Or if you want to keep track of when you log in, add something like date >> ~/login_times.dat.
Remember how redirecting stdout with ">" to an existing file would overwrite or clobber the file? If you want to protect yourself from doing that accidentally, set the bash "noclobber option" by adding a "set -o noclobber" line to your .bash_profile. If you try to redirect output to an existing file, the command will fail. To ignore the noclobber option and force the shell to overwrite an existing file, use ">| filename".
$who > current_users-bash: current_users: cannot overwrite existing file $who >| current_users
So what is the .bashrc script used for? As we said above, it's for interactive non-login shells. What does that mean? There may be times where you may need to start a separate shell from the current one, for example, by explicitly running bash. Or you may be editing a file with vi and need to temporarily escape the editor (without quitting) to run a new shell to search for some other file or something. (In vi you do that by entering ":!bash".) You can run a single command on a compute node without logging in to it with ssh, e.g., ssh n001 'ps -fu bobby'. In all these cases, only .bashrc gets run, not .bash_profile.
Some of the things that are often placed in .bashrc are "aliases" and functions. We'll just describe aliases here. An alias is just a "shortcut" command that replaces a real one. For example, if you find yourself running the command ls -laF often, you could create an alias lf for this.
$alias lf="ls -laF"$lf[bobby@login001 documents]$ lf total 48 drwx--x--x. 3 bobby users 4096 Oct 31 14:47 ./ drwx------. 13 bobby users 4096 Nov 13 09:55 ../ -rw-r--r-- 1 bobby users 12424 Oct 17 14:30 annual_report.pdf -rw------- 1 bobby users 1065 Oct 28 10:01 budget -rw-r--r-- 1 bobby users 594 Oct 31 09:24 current_users drwxr-xr-x 2 bobby users 4096 Oct 31 14:08 misc/ -rw-r--r-- 1 bobby users 18 Oct 30 13:23 my_name -rw-r----- 1 bobby users 1324 Oct 17 14:30 letter.txt -rw-r--r-- 1 bobby users 47 Oct 31 09:55 sentence.txt
If you find yourself often (mis)typing "mroe" instead of "more", embrace your inner typo and add alias mroe="more". Or if you prefer to use less instead of more but keep typing more out of habit, alias more="less".
If you are unsure which startup script gets called when, you can place a echo 'This is bashrc/bash_profile' command at the beginning of them. Be sure to remove or comment out those lines when finished. Extraneous output, especially from .bashrc, can sometimes cause odd problems.
Summary of Commands
Below is list of commands mentioned in this document. For more information, use the man command.
| Command | Description |
|---|---|
who |
Show who is logged in |
man |
Display the on-line manual pages |
passwd |
Modify your password |
pwd |
Print name of current working directory |
cd |
Change current directory |
ls |
List directory contents |
mkdir |
Make directory |
cat |
Print file on standard output |
more |
Display a file, one screen at a time |
less |
Less is more (more or less) |
head |
Display the first part of a file |
tail |
Display the last part of a file |
cp |
Copy files and directories |
mv |
Move or rename files |
rm |
Remove/delete a file |
nano |
Simple text editor |
vi |
Text editor |
emacs |
Text editor |
gedit |
Graphical text editor |
id |
Print user identity |
whoami |
Print username |
groups |
Print the groups a user is in |
chmod |
Change file access permissions |
which |
Show the full path of a command |
touch |
Change file access and modification time |
echo |
Output a line of text |
date |
Prints the system date and time |
w |
Show who is logged in and what they are doing |
top |
Display and update sorted information about processes |
wc |
Print line, word, and character counts for a file |
file |
Determine file type |
gzip,gunzip |
Compress or expand files |
hostname |
Show the system's host name |
ssh |
Remote login program |
diff |
Compare files line by line |
grep |
Search a file for a string or pattern |
ed |
Line-oriented text editor |
ps |
Process status |
sleep |
Delay for a specified amount of time |
env,printenv,set |
Print environment variables |
More Information
You can get more information on the topics above at the following link...
Ready to learn how to compile and run programs on Kodiak? Check out the Kodiak User's Guide.
Link: