Tag Archives: tutorials

Running Shell Commands from Julia

Here are some examples of starting and interacting with other programs from Julia.
The official documentation is pretty good, but I want something with more (basic) examples and fewer words.
I do reccommend reading that to see some of the fancier tricks you can pull
(and for up-to-date documentation, when this blog post gets stale).

Running Other Programs

The easiest, most basic way to run a shell command is run.

julia> help(run)
Base.run(command)

   Run a command object, constructed with backticks. Throws an error
   if anything goes wrong, including the process exiting with a non-
   zero status.

You can’t communicate with the process at all,
and run will block until the command finishes running.

julia> run(`echo hello!`)
hello!

Notice that Command literals are written with backticks, not single quotes (').

You can run two commands in parallel using &:

julia> run(`echo hello` & `echo world`)
hello
world

If you try to cat a file that doesn’t exist, you’ll get an error.

julia> run(`cat test.txt`)
cat: test.txt: No such file or directory
ERROR: failed process: Process(`cat test.txt`, ProcessExited(1)) [1]
 in error at error.jl:22
 in pipeline_error at process.jl:430
 in run at process.jl:413

The first line is cat printing an error message.
The rest is Julia throwing an error.
Note that the command is containedin backticks, not normal single quotes.

Redirecting

Base.|>

   Redirect standard input or output of a process.

   Example: run(`ls` |> "out.log")
   Example: run("file.txt" |> `cat`)

Use |> to redirect the STDOUT of a command to a file
or to write a file to a command’s STDIN.

julia> run(`echo hello, world` |> "test.txt")

julia> run(`cat test.txt`)
hello, world

julia> run("test.txt" |> `cat`)
hello, world

You can also use |> to redirect a process’s output to another process.

julia> run(`echo hello` |> `cat`)
hello

julia> run(`echo $("hellonhinhello, world")` |> `cat` |> `grep -n o`)
1:hello
3:hello, world

String Interpolation

You can use $ to interpolate into Command literals, in the same way you can into string literals.

julia> filename = "testnnewline.txt"
"testnnewline.txt"

You can see what the interpolation expands to by just printing the Command literal.
For example, this doesn’t run or do anything, it just shows you what the command looks like.

julia> `cat $filename`
`cat 'test
newline.txt'`

If you run the command without having that file around, you’ll get an error because cat doesn’t exit cleanly.

julia> run(`cat $filename`)
cat: test
newline.txt: No such file or directory
ERROR: failed process: Process(`cat 'test
newline.txt'`, ProcessExited(1)) [1]
 in error at error.jl:22
 in pipeline_error at process.jl:430
 in run at process.jl:413

Creating the file resolves the error:

julia> run(`echo the name is $filename` |> filename)

julia> run(`cat $filename`)
the name is test
newline.txt

The rest of this section’s examples are mostly from the official documentation.

julia> names = ["foo","bar","baz"]
3-element ASCIIString Array:
 "foo"
 "bar"
 "baz"

Command interpolation is more specialized than just plain string interpolation.
Lists will become escaped and space separated, which is more reasonable in shell commands than the normal square-brackets-and-commas notation.

julia> `grep xylophone $names`
`grep xylophone foo bar baz`

If you interpolate a list as part of a word, the resulting space separated list will be versions of the complete word:

julia> `grep xylophone $names.txt`
`grep xylophone foo.txt bar.txt baz.txt`

julia> `grep xylophone hi$names.bye`
`grep xylophone hifoo.bye hibar.bye hibaz.bye`

julia> `grep xylophone hi$(names)bye`
`grep xylophone hifoobye hibarbye hibazbye`

You can do even cooler things if you have two arrays.
If you interpolate two arrays into the same word,
then you’ll get all combinations of elements in the two lists.

julia> exts = ["aux","log"]
2-element ASCIIString Array:
 "aux"
 "log"

julia> `rm -f $names.$exts`
`rm -f foo.aux foo.log bar.aux bar.log baz.aux baz.log`

Getting Output

The easiest way to call a command and get it’s output as a String is readall.

julia> readall(`echo hello?`)
"hello?n"

This words just as well with chains of commands and redirections
because the chain will become a single Cmd value
when the redirectin operators are done.

julia> readall("test.txt" |> `cat`)
"hello, worldn"

If you just want the contents of a file as a string, you don’t need to use cat or any other command. Just use readall on the file:

julia> readall("test.txt")
"hello, worldn"

However, sometimes you’d like to be able to read the output in over time, rather than all at once.
For that, you want readsfrom:

julia> help(readsfrom)
Base.readsfrom(command)

   Starts running a command asynchronously, and returns a tuple
   (stream,process). The first value is a stream reading from the
   process' standard output.

julia> (st,pr) = readsfrom(`cat poem`)
(Pipe(active, 0 bytes waiting),Process(`cat poem`, ProcessRunning))

This use of readsfrom isn’t that interesting, since cat exits right away.

julia> pr
Process(`cat poem`, ProcessExited(0))

You can use any reading stream functions you’d like: readall, readline, etc.

julia> readline(st)
" There is a place where the sidewalk endsn"

The pipe is closed because the process has exited.

julia> st
Pipe(closed, 718 bytes waiting)

This way, you can process the data one line (or whatever increment you like) at a time.

julia> readline(st)
"And before the street begins,n"

julia> readline(st)
"And there the grass grows soft and white,n"

julia> st
Pipe(closed, 646 bytes waiting)

When you run out of data in the pipe, readline will start returning empty strings: "".

Want to ls, but get the result as an array of file and directory names?
You want to use readdir()

julia> help(readdir)
# methods for generic function readdir
readdir(path::String) at file.jl:169
readdir() at file.jl:198

julia> readdir()
6-element String Array:
 "newnline"      
 "poem"           
 "snowman☃snowman"
 "sp ace"         
 "ttab"

Sending Input

Sometimes, you might want to communicate with another process in a more complex way than just commandline args.

Write to STDIN and then wait for it to finish

The writesto function will give you a Pipe hooked up to the STDIN of the process and a Process. Do not read from the Pipe.

julia> (si,pr) = writesto(`cat`)
(Pipe(open, 0 bytes waiting),Process(`cat`, ProcessRunning))

If you don’t give cat any filenames (or if you pass in a -),
then cat will read from STDIN until you send CTRL-D.

julia> write(si,"hello")
5

We can keep writing to cat for as long as we want.
As you can see, the process is still running:

julia> pr
Process(`cat`, ProcessRunning)

When you’re done writing to the process, close the Pipe.

julia> close(si)

This will make cat exit, as it’s done doing it’s work:

julia> pr
Process(`cat`, ProcessExited(0))

If you want to wait for the process to be down doing work,
you can call wait on the Process.
This will block until the process exits.

julia> wait(pr)
0

Read AND Write on the same process

Using the readandwrite function, you can get two Pipes and a Process.
The Pipes are the STDOUT and STDIN of the process.
The Process is a Julia value representing the asynchronous process started to run the command.

You can read from the STDOUT of the process, but don’t write to it.
You can write to the STDIN of the process, but don’t read from it.

julia> (so,si,pr) = readandwrite(`cat`)
(Pipe(active, 0 bytes waiting),Pipe(open, 0 bytes waiting),Process(`cat`, ProcessRunning))

julia> write(si,"hellongoodnbyen")
15

julia> so
Pipe(active, 15 bytes waiting)

julia> si
Pipe(open, 0 bytes waiting)

julia> close(si)

julia> so
Pipe(closed, 15 bytes waiting)

julia> pr
Process(`cat`, ProcessExited(0))

julia> readall(so)
"hellongoodnbyen"

julia> so
Pipe(closed, 0 bytes waiting)

REPL specialness

In the REPL, you can use a ; to run shell commands easily.

julia> ;ls
new?line  NGrep.jl  poem  snowman☃snowman  sp ace  t?ab  test.txt

julia> ;mkdir folder

julia> ;ls
folder  new?line  NGrep.jl  poem  snowman☃snowman  sp ace  t?ab  test.txt

julia> ;cd folder/
/home/leah/src/lambdajam/folder

The name of the directory (folder/) tab completed! 🙂

julia> ;ls

julia> ;touch my_new_file

julia> ;ls
my_new_file

API Summary

Setting up the process

julia> help(readsfrom)
Base.readsfrom(command)

   Starts running a command asynchronously, and returns a tuple
   (stream,process). The first value is a stream reading from the
   process' standard output.

julia> help(writesto)
Base.writesto(command)

   Starts running a command asynchronously, and returns a tuple
   (stream,process). The first value is a stream writing to the
   process' standard input.

julia> help(readandwrite)
Base.readandwrite(command)

   Starts running a command asynchronously, and returns a tuple
   (stdout,stdin,process) of the output stream and input stream of the
   process, and the process object itself.

Interacting with streams

julia> help(write)
Base.write(stream, x)

   Write the canonical binary representation of a value to the given
   stream.

julia> help(readall)
Base.readall(stream)

   Read the entire contents of an I/O stream as a string.

julia> help(readline)
Base.readline(stream)

   Read a single line of text, including a trailing newline character
   (if one is reached before the end of the input).

julia> help(close)
Base.close(stream)

   Close an I/O stream. Performs a "flush" first.

Interacting with processes

julia> help(wait)
Base.wait(x)

   Block the current task until some event occurs, depending on the
   type of the argument:

   * "RemoteRef": Wait for a value to become available for the
     specified remote reference.

   * "Condition": Wait for "notify" on a condition.

   * "Process": Wait for the process to exit, and get its exit code.

   * "Task": Wait for a "Task" to finish, returning its result
     value.

Writing FizzBuzz in Julia

One of the simplest first programming exercises to try in a new language
is the FizzBuzz problem.
It’s a simple toy problem that is less trivial than “Hello, World”
because it also involves using basic control-flow structures.

This blog post assumes that you’ve already installed the Julia programming language (hint: see GitHub)
and figured out how to open a REPL (hint: run julia).

Now that you’ve got a Julia REPL open, let’s go through the very basic syntax of the language:
its control flow structures.
After we have those under control, you’ll write FizzBuzz in Julia.

If Statements

Julia uses the keyword end to end blocks rather than using whitespace or curly braces ({}) as delimeters.
Julia is not whitespace sensitive.

A simple if statement:

julia> if 5 == 4
         print("hello world")
       end

== is the equality operator; it returns either true or false.
It is both an infix operator and a function.

The simplest possible if-statements are:

if false
end

and

if true end

Julia is not whitespace sensitive;
you can sometimes put things all on one line without changing the meaning.
This is the case with these very simple if-statements.

The tricky part to guess in the if-statement syntax is the keyword elseif,
since there’s so much variation in that keyword between languages.

julia> if 5 == 6 # this is a comment
         print("five and six are equal")
       elseif mod(5,6) == 4 # you will need the mod function later
         print("five mod six is four")
       else
         print("Stefan is awesome")
       end
Stefan is awesome

For Loops

The most straight-forward ways to write FizzBuzz involve for-looping over a range of numbers,
so here’s an example of that style of loop and its output:

julia> for x=1:5
         println(x)
       end
1
2
3
4
5

Note that the range 1:5 is inclusive; both 1 and 5 appear in the output.

As you would expect, Julia also has while loops:

julia> while false
         print("something is very wrong")
       end

Ranges

The 1:5 in the for loop is a range.
Ranges are of the form start:end.
As you can see, both the start and the end are inclusive. (both 1 and 5 are values in the range 1:5)
By default, the range increments by 1.

julia> for x=1:2:10
         println(x)
       end
1
3
5
7
9

In this more complex form of range, you have start:step:end.
The end value of the range does not have to be included; it is only included if incrementing by step lands on it.

Exercise: FizzBuzz in REPL

Now that you have if statements, for loops, and the println, mod, and == functions
at your disposal, you can write FizzBuzz in the REPL.
It should take the form of a for loop with an if statement inside.

For those unfamiliar with it, FizzBuzz is a common coding problem that involves printing something
for each of the first 100 positive integers.
You should have 100 lines of output: one for each integer.

What you print depends on the number, as follows:

  • If the number is divisible by 3, print ‘Fizz’.
  • If the number is divisible by 5, print ‘Buzz’.
  • If the number is divisible by both 3 and 5, print ‘FizzBuzz’.
  • If none of the above apply, just print the number.

In case the above explanation is unclear, the correct output for 1:20 is:

1
2
Fizz
4
Buzz
Fizz
7
8
Fizz
Buzz
11
Fizz
13
14
FizzBuzz
16
17
Fizz
19
Buzz

Solution

Don’t peek before trying it yourself, but if you feel the need to see a correct solution,
you can find one here on RosettaCode.

Using TCP Sockets in Julia

This post was updated on June 28, 2013 to reflect changes in the TcpServer/TcpSocket api in Julia.

Recently, I’ve been writing the
WebSockets implementation for Julia.

TcpSockets were not well documented
when we started using them,
so I figure a tutorial might be useful
for anyone else who might want to use TCP sockets in Julia.

REPL server

I’m still figuring out this “try things out in the REPL first” thing
(due to usually using compiled languages),
but here’s a first example of how you might play with TCP sockets
in the Julia REPL (on the right) and netcat in another terminal (on the left).

terminal1 terminal2
$ julia
> server = listen(8080)
TcpServer(listening)
> conn = accept(server)
$ nc localhost 8080
TcpSocket(connected,0 bytes waiting)
> line = readline(conn)
hi
“hi\n”
> write(conn, "Hello")
Hello 5
> close(conn)
$ false

The code-formated parts are what you type.
The normally-formated parts are what the computer prints.

Basically, you start julia, and open a TCP server for listening.
You can only accept connections on a TcpServer, not write or read.

After creating a TcpServer, you wait for a connection to accept.
At this point, you’ll need to switch terminals and open a connection from netcat.
(You can switch the order of accept and starting netcat.)

Notice that accept returns a TcpSocket.
We can do two things with a TcpSocket: read or write.

Using readline, we can block until we get a full line from the connection.
In order to stop blocking, you’ll need to enter a line in netcat.
You need to press enter in netcat for readline to return in Julia.

You can also write text to the TCP connection from Julia.
The write function takes a TcpSocket and a string to write.
The text you write from Julia will appear in netcat, as expected.

You can call close on a TcpSocket to close the TCP connection.
Closing the TCP connection makes the netcat session close.

Echo Server

A very simple server to run over TCP is an echo server.
Each time you send this server a line of text,
it will repeat that line back to you.

Here is the entire file:

server = listen(8080)
while true
  conn = accept(server)
  @async begin
    try
      while true
        line = readline(conn)
        write(conn,line)
      end
    catch err
      print("connection ended with error $err")
    end
  end
end

Running the example

Put this file in echoserver.jl.
Then, you can run it with julia echoserver.jl.

With the server running, open up netcat again (nc localhost 8080).
Type in some text and press enter.
You should see the same text printed again under what you typed.

The structure of the server

The core of this server is a pair of while true loops.
The outer one accepts each incoming connection.
The inner one reads lines from the client and echos them.

The try catch block catches any error thrown in the while loop.
The thrown error is bound to err.
The $err in the string literal is how you embed values in Julia strings.

Given two strings, s1 and s2, "$s1$s2" would be their concatenation.
You can also embed arbitrary Julia expressions: "$(2+2)" would be "4".

Handling Multiple Connections: @async

@async is a macro. It affects the begin to end block that directly follows it.
@async starts a new coroutine to handle the execution of the block.

Coroutines are like threads in that there are multiple threads of execution in the program at once,
but do not involve concurrent execution.
Only one coroutine is running at once, but the language runtime switches between them.

This means that the main connection-accepting routine will see the @async block return immediately.
This allows the outer while loop to continue accepting connections, while another routine handles each existing connection.

If you remove the @async part, your echo server will only deal with one client connection at a time.
You can test this using two (or more) instances of netcat, each in their own terminal window.

Inside the @async block, we’re in a new coroutine.
We can sit in our while true loop for as long as we want without affecting the server’s ability to accept connections.

Coroutines have very low overhead, so making a new one for each connection is reasonable.
Webstack.jl works on this same one-thread-per-connection principle.

Basic TcpSocket API

  • listen(8080) binds a TcpServer to a socket listening for connections on port 8080.
  • accept(server) blocks until there is a connection to accept on server, then accepts it and returns a TcpSocket.
  • readline(conn) blocks until a complete line comes in on conn, then returns it.
  • read(conn,Uint8) blocks until there is a byte to read from conn, and then returns it.
    You can use convert(Char,u) to convert a Uint8 into a Char. This will let you see the ASCII letter for the Uint8 you read.
  • read(conn,Char) blocks until there is a byte to read from conn, and then returns it.
  • write(conn,line) writes line to conn.