Between the Ctrl-C's

We’re Sequin. We stream data from services like Salesforce, Stripe, and AWS to messaging systems like Kafka and databases like Postgres. It’s the fastest way to build high-performance integrations you don’t need to worry about. We use Elixir. One of our favorite parts is the tooling, including the iex shell. We realized one day that we didn't really understand the Ctrl-C feature, so we had to research more and share!

One of Elixir's best features is its REPL, IEx (interactive Elixir.) The iex command spawns a shell that allows you to run Elixir code interactively.

To exit IEx, you normally press Ctrl-C twice. If you've done this before, you may have noticed an interesting menu pop-up between the Ctrl-Cs:

BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution

What's that about? What do all of those options do?

This menu is known as the BREAK menu and it allows you to interact with the running Elixir session in various ways.

The BREAK menu is part of the Erlang Virtual Machine (commonly known as the BEAM), so these options are also available when running Erlang applications and shells.

The BREAK menu is available when:

• Running iex
• Running an application locally (e.g. mix phx.server or iex -S mix)
• Connecting IEx to a remote application (e.g. iex --sname local --remsh app@host)

In the latter case, all of the menu options in the BREAK menu apply only to your local IEx session and not to the remote application. In some cases this is a good thing, as you probably don't want to accidentally shut down the running application.

Here's a quick summary of these commands (and some additional undocumented commands). We'll look at each of these in more detail in the following sections.

If you want to see how these are implemented, you can look at the C source code in the BEAM.

Simple Commands

A number of the BREAK menu options are simple commands for interacting with the running shell.

(a)bort

The a command does the same thing that hitting the second Ctrl-C does: it closes the current shell.

If you're running IEx normally, it will close the session and any application running in it.

If you're running your application directly (e.g. mix phx.server), it will shut down your application.

If you're in a remote IEx session connected to a running application, it only closes your IEx session, but leaves the remote application alone.

(A)bort with dump

This is the same as (a)bort, but also produces a BEAM crash dump file (erl_crash.dump).

Looking through long pages of information in an interactive shell on a remote system can often be quite difficult. In that case, producing a crash dump file that you can examine on your local workstation might make things easier.

See the Erlang documentation for more information on interpreting and working with crash dump files.

(c)ontinue

Exits the BREAK menu and returns to your application or shell session.

This is the command to remember when you hit Ctrl-C by accident or change your mind.

(q)uit

The q command is not listed in the BREAK menu, but is supported. It is exactly the same as the a command.

(v)ersion

Prints the current BEAM version.

Interactive Elixir (1.14.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
v
Erlang (BEAM) emulator version 12.3.2.6

This shows you the current Erlang Runtime System (ERTS) version and not the OTP version.

Informational Options

A number of the options in the BREAK menu will provide information about the running Erlang VM.

In general, it's easier to see this information by running either Observer or observer_cli. That's what we normally use. But if you don't have those tools available, the BREAK menu is always available.

As I mentioned above, these commands all print information about your local IEx session even when you've opened a remote shell to a running application.

If you're in a normal IEx session or are running your application locally, that's perfect.

But for a remote shell to a running application, you're probably more interested in seeing this information for that application instead. There doesn't seem to be a way to make the BREAK menu display information for the remote application, so you'll have to rely on using Observer or observer_cli instead.

Let's look at the commands that are available.

(D)b-tables

Prints a list of ETS tables along with some information about each one.

This can help you find a particular table, see how it's been configured, and see how big it is.

Interactive Elixir (1.14.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
D
=ets:<0.42.0>
Slot: 4710072768
Table: logger
Name: logger
Buckets: 256
Chain Length Avg: 0.011719
Chain Length Max: 1
Chain Length Min: 0
Chain Length Std Dev: 0.107617
Chain Length Expected Std Dev: 0.108042
Fixed: false
9: [{{'$handler_config$','Elixir.Logger'},#{config=>#{counter=>{atomics,#Ref<0.2287420997.2752643074.39926>},sasl=>false,thresholds=>{20,500},translators=>[{'Elixir.Logger.Translator',translate}],truncate=>8096,utc_log=>false},filter_default=>log,filters=>[],formatter=>{logger_formatter,#{}},id=>'Elixir.Logger',level=>all,module=>'Elixir.Logger.Handler'}}]
84: [{'$primary_config$',#{filter_default=>log,filters=>[{process_level,{fun 'Elixir.Logger.Filter':process_level/2,[]}}],handlers=>['Elixir.Logger'],level=>debug,metadata=>#{}}}]
116: [{'$proxy_config$',#{burst_limit_enable=>false,burst_limit_max_count=>500,burst_limit_window_time=>1000,drop_mode_qlen=>1000,flush_qlen=>5000,overload_kill_enable=>false,overload_kill_mem_size=>3000000,overload_kill_qlen=>20000,overload_kill_restart_after=>5000,sync_mode_qlen=>500}}]
Objects: 3
Words: 7709
Type: set
Protection: protected
Compressed: false
Write Concurrency: true
Read Concurrency: true

...

(d)istribution

Prints information about the current node and other nodes it is connected to.

This command shows you information about all of the nodes in your distributed Elixir/Erlang application, which can be helpful when trying to debug connection problems.

Interactive Elixir (1.14.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
d
=node:'nonode@nohost'
=no_distribution

(i)nfo

Prints information about memory usage and allocation.

This command produces a large volume of information about memory usage and can be helpful in figuring out where a memory leak might be helpful.

Interactive Elixir (1.14.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
i
=memory
total: 34928033
processes: 5723312
processes_used: 5723312
system: 29204721
atom: 336049
atom_used: 331018
binary: 25520
code: 7495815
ets: 537960
=hash_table:atom_tab
size: 8192
used: 6389
objs: 12508
depth: 9
=index_table:atom_tab
size: 13312
limit: 1048576
entries: 12509
=hash_table:module_code
size: 128
used: 103
objs: 174
depth: 5
=index_table:module_code
size: 1024
limit: 65536
entries: 174

...

(l)oaded

Prints a list of loaded modules and their size.

This can be handy for checking that all of the code is available that you thought should be there.

Interactive Elixir (1.14.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
l
Current code: 7737441
Old code: 0
erts_code_purger 13536
erl_init 2520
init 63188
prim_buffer 4984
prim_eval 1352
prim_inet 111589
prim_file 41674
zlib 20584
socket_registry 22408
prim_socket 51960
prim_net 10840
prim_zip 24120
erl_prim_loader 56864
erlang 97012
erts_internal 22920
erl_tracer 1680
erts_literal_area_collector 4928
erts_dirty_process_signal_handler 2352
atomics 3464
counters 4176
persistent_term 1456

...

p(o)rt info

This is not listed in the BREAK menu, but does appear in the source code. It prints a list of open ports (identical to what's included at the end of the p command's output).

According to the Erlang documentation, ports provide the basic mechanism for communication with the external world, from Erlang's point of view. An open port will represent a connection from the Erlang VM to another process running in the host operating system.

Listing open ports can allow you to see what other processes your application is talking to.

Interactive Elixir (1.14.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
o
=port:#Port<0.0>
State: CONNECTED
Slot: 0
Connected: <0.0.0>
Port controls forker process: forker
Input: 0
Output: 0
Queue: 0
=port:#Port<0.2>
State: CONNECTED|BINARY_IO
Slot: 16
Connected: <0.62.0>
Links: <0.62.0>
Port is UNIX fd not opened by emulator: 2/2
Input: 0
Output: 0
Queue: 0
=port:#Port<0.4>
State: CONNECTED|SOFT_EOF
Slot: 32
Connected: <0.64.0>
Links: <0.64.0>
Port controls linked-in driver: tty_sl -c -e
Input: 4
Output: 235
Queue: 0

(p)roc info

Prints a list of running processes along with a list of open ports.

Processes are a fundamental building blog of Elixir applications, so seeing what processes are running can give you a lot of insight into what's going on.

Most Elixir/Erlang applications run a large number of processes, so this list will be quite long.

BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
p
...

=proc:<0.1.0>
State: Waiting
Name: erts_code_purger
Spawned as: erts_code_purger:start/0
Current call: erlang:hibernate/3
Spawned by: <0.0.0>
Message queue length: 0
Number of heap fragments: 0
Heap fragment data: 0
Reductions: 9
Stack+heap: 4
OldHeap: 0
Heap unused: 4
OldHeap unused: 0
BinVHeap: 0
OldBinVHeap: 0
BinVHeap unused: 46422
OldBinVHeap unused: 46422
Memory: 800
Stack dump:
Program counter: 0x00000001059a6008 (unknown function)
arity = 3
   erts_code_purger
   wait_for_request
   []

0x0000000118970220 Return addr 0x00000001059a6088 (<terminate process normally>)
Internal State: ACT_PRIO_HIGH | USR_PRIO_HIGH | PRQ_PRIO_HIGH | OFF_HEAP_MSGQ
=proc:<0.2.0>
State: Waiting
Spawned as: erts_literal_area_collector:start/0
Current call: erlang:hibernate/3
Spawned by: <0.0.0>
Message queue length: 0
Number of heap fragments: 0
Heap fragment data: 0
Reductions: 8
Stack+heap: 4
OldHeap: 0
Heap unused: 4
OldHeap unused: 0
BinVHeap: 0
OldBinVHeap: 0
BinVHeap unused: 46422
OldBinVHeap unused: 46422
Memory: 800
Stack dump:
Program counter: 0x00000001059a6008 (unknown function)
arity = 3
   erts_literal_area_collector
   start
   []

0x0000000107c3a3b8 Return addr 0x00000001059a6088 (<terminate process normally>)
Internal State: ACT_PRIO_HIGH | USR_PRIO_HIGH | PRQ_PRIO_HIGH | OFF_HEAP_MSGQ

...

Interactive Command

One of the commands is an interactive command, entering a new mode with sub-commands.

(k)ill

Prints information about one of the running processes in the system, then prompts for what to do with that process:

• (k)ill: Kill the process, then display the next process
• (n)ext: Move to the next process without killing the current one
• (r)eturn: Exit the sub-menu and return to the BREAK menu

When using the p command to list processes (described above), the list can be quite long and hard to navigate.

This command will show you only a single process at a time, allowing you to decide what to do with it.

If the process is running under a supervisor, killing it will cause the supervisor to respond according to its restart policy, often be recreating the process.

Interactive Elixir (1.14.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
k


Process Information

--------------------------------------------------
=proc:<0.108.0>
State: Waiting
Spawned as: erlang:apply/2
Spawned by: <0.78.0>
Message queue length: 0
Number of heap fragments: 0
Heap fragment data: 0
Link list: [{to,<0.66.0>,#Ref<0.2287420997.2752512002.40054>}]
Reductions: 4190
Stack+heap: 233
OldHeap: 0
Heap unused: 207
OldHeap unused: 0
BinVHeap: 1
OldBinVHeap: 0
BinVHeap unused: 46421
OldBinVHeap unused: 46422
Memory: 2696
Stack dump:
Program counter: 0x0000000105dac3e8 (io:execute_request/3 + 376)
arity = 0
y(0)     #Ref<0.2287420997.2752512002.40054>
y(1)     error
y(2)     {false,{get_line,unicode,<<"iex(1)> ">>}}
y(3)     <0.66.0>

0x0000000107d82470 Return addr 0x0000000105e20998 ('Elixir.IEx.Server':io_get/4 + 136)
y(0)     <0.108.0>
y(1)     <0.78.0>

0x0000000107d82488 Return addr 0x00000001059a6088 (<terminate process normally>)
Internal State: ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL
(k)ill (n)ext (r)eturn:

Wrapping Up

I wasn't able to find a place where this information was documented, so I hope this gives you some insight into more of the power tools available in the BEAM.

When you're running your application in an environment that doesn't have access to more powerful tools like Observer or observer_cli, the BREAK menu can help you get to the bottom of a problem.