ls

Go back to your terminal window and cd (meaning change directory) into the lab1 directory:

cd anlp/labs/lab1

If you ever get confused or forget which directory you are in, use the pwd (print working directory) command. Try it now:

pwd

Now let's list the contents of the current directory using ls:

ls

You should now see a subdirectory called Providence, which contains the data you just downloaded. What do you see in it? What about in the further subdirectories?

ls Providence
ls Providence/Ethan

less

To get an idea of what is in the files you just downloaded, type:

cd Providence/Ethan
less eth01.cha

What information is in the metadata at the top of each file? (Hint: child language researchers use the format y;m.d to indicate a child's age in years;months.days)

(less may seem like a funny name for this command which shows you more of the file; it's because there was an earlier similar command called more, so when this newer version was developed the developers decided to give it a cute name. UNIX developers like cute names.)

When you are looking at a file using less, you can scroll up and down using the arrow keys or <PageUp>/<PageDown>. <space> also acts like <PageDown> and <Enter> acts like <down-arrow>.

What do you see in the rest of the file? (Hint: the string of numbers at the end of each line is a code that links to a time point in the audio recording of this data. The audio isn't included here but can be obtained from the CHILDES database.)

One final useful thing you can do with less is search for a particular string in the file by typing / followed by the string. Try it: type

/the

followed by ENTER. You should see all of the instances of the being highlighted.

To stop viewing the file and go back to the terminal command line, type q.

Actually, the eth01.cha file maybe is not so interesting. Look now at eth50.cha. What are some of the main differences between the data in these two files? Can you think of an explanation for those differences? (If you want to look at both files simultaneously, you could start a new terminal and open one file in each terminal.)

head, tail

Sometimes we just need a quick peek at part of a file. What does this command do?

head eth01.cha

What about this one?

tail eth01.cha

These commands can be useful, for example, if we want to look over the range of ages in the files we have. Try:

head *.cha

What does the * do?

man

We don't really need to look at the first 10 lines of each file to see the ages, we only need the first five lines. If we could print only the first five lines, we could see the information we want more compactly. Most UNIX commands, including head, have many possible options to change their behavior. To see what options are available, look at the manual for the command:

man head

You should see:

HEAD(1)                          User Commands                         HEAD(1)

NAME
       head - output the first part of files

SYNOPSIS
       head [OPTION]... [FILE]...

DESCRIPTION
       Print  the  first  10 lines of each FILE to standard output.  With more
       than one FILE, precede each with a header giving the file  name.   With
       no FILE, or when FILE is -, read standard input.

       Mandatory  arguments  to  long  options are mandatory for short options
       too.

       -c, --bytes=[-]K
              print the first K bytes of each  file;  with  the  leading  `-',
              print all but the last K bytes of each file

       -n, --lines=[-]K
              print  the first K lines instead of the first 10; with the lead-
              ing `-', print all but the last K lines of each file

       -q, --quiet, --silent
              never print headers giving file names

       -v, --verbose
              always print headers giving file names

       ...

Note: you should be able to move up and down in the man pages using the same keys you used for less [I hope; the default for the student account might not allow you to move upwards...], and you can also return to the command line using q.

The man page starts with the name of the command and a brief synopsis of how to use it. In this case, it says that the head command can be followed by zero or more options (sometimes called flags) and then zero or more files as arguments (things to act on). The square brackets indicate that these options and files need not be included at all, and the ... indicate that you can include more than one of each.

The description after the synopsis says what will happen if you include zero or multiple files. (Standard input refers to the text you input in the terminal. Try entering head with no filename to see how this works. You will need to enter some more text after that! To quit, type Ctrl-c.)

The next part of the description tells you what the possible options are and if they require arguments themselves. Many (in this case all) options have both a short and a long form, which are equivalent. For example, to print just the first 5 lines of each file, we could either use:

head -n5 eth01.cha

or:

head --lines=5 eth01.cha

(To try this yourself, you will need to either use another terminal or quit the man page first by typing q.)

The K specified in the description is a variable indicating a required argument to the -n option. Here we use the value 5 for K. Notice that this option also has a non-required argument [-] so if we wanted to print all but the last 5 lines we could type:

head -n-5 eth01.cha

As noted above, you can specify multiple options at once. Compare the output of the following commands:

head -n5 *.cha
head -n5 -q *.cha

Now that you know how to read a man page, you may want to look at the man pages for some of the other commands we've seen to see what options are available for them.

There is also a lot of help about these commands on the Internet. You can find more examples or maybe a better explanation than the man page has. If you know the name of the command you want to use, like grep, try searching for UNIX command grep. If you know what you want to do but not the name of the command, searching is trickier, but try searches like UNIX search regular expression or even UNIX find string in file. Get in the habit of looking for help--you will start to recognize some of the main help forums and which ones are most useful to you.

Tab completion

Type the following into your terminal. Instead of pressing <enter> at the end, press the <tab> key:

head e

You should find that the filename has been automatically extended to eth. If you type 01 and then <tab> again, you will have the complete filename. In this case the tabbing didn't save many keystrokes, but for long filenames it can save a lot. You can use tab completion for nearly anything in UNIX (commands, filenames, directories). The operating system will complete as much as it can but will stop where there are multiple possible extensions (like here, where all files in the current directory start with eth but we then had to type in 01 to disambiguate before tabbing again.) In DICE, tab completion is set up so that if you hit <tab> twice when there are multiple completions, the system will print out all possible completions for you to see.

History

Your terminal keeps track of all the commands you have entered. To access the previous command, just type the up-arrow key. If you hit <up> twice, you will get the command before that, and so forth. This can save a lot of typing.

wc

Try this command:

wc eth01.cha

What are the numbers that you got back? What do you think wc stands for? (Hint: look at the man page!)

Using a single command, can you figure out which of the first nine Ethan files has the most lines in it?

grep

Sometimes we want some more specific information. The number of lines in a file roughly indicates how much language data is in there, but what if we want to know how many utterances are spoken by a particular person? Try this command:

grep 'MOT' eth01.cha

What did this command do?

Using the man page for grep, figure out what option you can use with grep to compute the number of utterances spoken by the mother in this file.

In case you are wondering, grep stands for Globally search for a Regular Expression and Print. We won't discuss regular expressions (REs) in detail here, but they are covered in Section 2.1 of the Jurafsky and Martin textbook, and also in the Computer Programming for Speech and Language Processing course. (The grep man page also explains the syntax that grep uses for REs.) REs are a powerful way to match patterns in text and we hope you will learn to love them. For now, we will just show you a few simple REs to show some of what you can do.

First, look at this command but don't run it yet:

grep 'MOT.*the' eth01.cha

What do you think it will do? (Hint: . matches any character, and * means "zero or more repetitions of the previous character")

To see if you were right, run the command and look at the first few lines of output. Even better, let's only print out the first few lines of output. We can do that by combining two commands we already know, grep and head:

grep 'MOT.*the' eth01.cha | head

pipes and redirection

What just happened?

The | (vertical bar on the keyboard, called pipe in UNIX-world) tells the operating system to use the output of the first command as the input to the second command. (Another way to think of this is that it turns the output of the first command into "standard input", which as you will recall, is what is used as input to head if no filename is given.

Can you think of a way to use pipe to combine two commands we already know to count the number of lines that include the string MOT without using the -c option for grep?

Instead of sending the output of one command to the input of another, we can also just send the output of a command into a file. This can be useful if we want to save the output for later inspection, and we do it using the > (output redirect) character:

grep 'MOT.*the' eth01.cha > my_output.txt
less my_output.txt

grep again

Coming back to regular expressions, let's make sure you understand what the RE 'MOT.*the' is matching in the grep command. What is the difference between the following two commands?

grep 'MOT.*the' eth01.cha
grep 'MOT.*\bthe\b' eth01.cha

You may want to look at just the start of the output by piping it to head. Or try using the --color=auto option for grep (unfortunately the color is lost if you pipe the output).

In general, backslashes are used in regular expressions to indicate something special (like a word boundary or any whitespace character) and are also used to make characters that are usually special (like *) be interpreted literally. So, for example, if we want to match only the lines spoken by the mother, we should use '^\*MOT' rather than just 'MOT', because the first version only matches the exact string *MOT at the beginning of a line (indicated by the ^). The second version also matches lines that are not spoken by the mother but happen to contain the string MOT. (Are there any?)

As you can start to see, regular expressions can be very powerful but proper use takes time and practice, and we will not go any further here.