Weird socket behavior

0-problem.md

For some reason, I can't seem to serve more than a bit under 2^14 tcp requests without elixer hanging.

This happens consistently in multiple projects I have set up. Below is a boiled down version that exhibits the behavior.

00-test.exs

defmodule Counter do
  def start_link do
    Agent.start_link(fn -> 0 end)
  end

  def hit(counter) do
    Agent.update(counter, &(&1 + 1))
  end

  def get(counter) do
    Agent.get(counter, (&(&1)))
  end

  def watch(counter) do
    IO.puts "Requests served: #{get(counter)}"
    :timer.sleep(2000)
    watch(counter)
  end
end

defmodule TestServer do
  def start(port, counter) do
    {:ok, socket} = :gen_tcp.listen(port, [
      :binary, packet: :line, backlog: 2048,
      active: false, reuseaddr: true])

    do_accept(socket, counter)
  end

  defp do_accept(socket, counter) do
    {:ok, client} = :gen_tcp.accept(socket)

    Counter.hit(counter)

    body = "Something clever..."

    :gen_tcp.send(client, "HTTP/1.1 200 OK\r\n")
    :gen_tcp.send(client, "Server: MechWarrior/0.1\r\n")
    :gen_tcp.send(client, "Content-Length: #{String.length(body)}\r\n")
    :gen_tcp.send(client, "Connection: close\r\n")
    :gen_tcp.send(client, "\r\n")
    :gen_tcp.send(client, body)

    :gen_tcp.close(client)

    do_accept(socket, counter)
  end
end

{:ok, counter} = Counter.start_link

spawn_link Counter, :watch, [counter]

TestServer.start(4000, counter)

1-run.txt

$ ab -r -c 1 -n 18000 http://192.168.22.176:4000/as
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.22.176 (be patient)
Completed 1800 requests
Completed 3600 requests
Completed 5400 requests
Completed 7200 requests
Completed 9000 requests
Completed 10800 requests
Completed 12600 requests
Completed 14400 requests
Completed 16200 requests
Completed 18000 requests
Finished 18000 requests


Server Software:        MechWarrior/0.1
Server Hostname:        192.168.22.176
Server Port:            4000

Document Path:          /as
Document Length:        19 bytes

Concurrency Level:      1
Time taken for tests:   37.134 seconds
Complete requests:      18000
Failed requests:        3
   (Connect: 1, Receive: 1, Length: 1, Exceptions: 0)
Total transferred:      1835898 bytes
HTML transferred:       341981 bytes
Requests per second:    484.74 [#/sec] (mean)
Time per request:       2.063 [ms] (mean)
Time per request:       2.063 [ms] (mean, across all concurrent requests)
Transfer rate:          48.28 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0  53.0      0    7110
Processing:     0    2 193.5      0   25964
Waiting:        0    0   1.0      0     138
Total:          0    2 200.7      0   25964

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      0
 100%  25964 (longest request)

1-server.txt

$ elixir 00-test.exs
Requests served: 0
Requests served: 0
Requests served: 0
Requests served: 7164
Requests served: 15794
Requests served: 16348
Requests served: 16348
Requests served: 16348
Requests served: 16348
Requests served: 16348
Requests served: 16348
Requests served: 16348
Requests served: 16348
Requests served: 16348
Requests served: 16348
Requests served: 16348
Requests served: 16348
Requests served: 16348
Requests served: 16349
Requests served: 16349
Requests served: 16349
Requests served: 18000
Requests served: 18000
^C

2-run.txt

$ ab -r -c 1 -n 18000 http://192.168.22.176:4000/as
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.22.176 (be patient)
Completed 1800 requests
Completed 3600 requests
Completed 5400 requests
Completed 7200 requests
Completed 9000 requests
Completed 10800 requests
Completed 12600 requests
Completed 14400 requests
Completed 16200 requests
Completed 18000 requests
Finished 18000 requests


Server Software:        MechWarrior/0.1
Server Hostname:        192.168.22.176
Server Port:            4000

Document Path:          /as
Document Length:        19 bytes

Concurrency Level:      1
Time taken for tests:   37.371 seconds
Complete requests:      18000
Failed requests:        4
   (Connect: 1, Receive: 2, Length: 1, Exceptions: 0)
Total transferred:      1835898 bytes
HTML transferred:       341981 bytes
Requests per second:    481.65 [#/sec] (mean)
Time per request:       2.076 [ms] (mean)
Time per request:       2.076 [ms] (mean, across all concurrent requests)
Transfer rate:          47.97 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0  55.6      0    7455
Processing:     0    2 193.7      0   25986
Waiting:        0    0   0.0      0       1
Total:          0    2 201.5      0   25986

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      0
 100%  25986 (longest request)

2-server.txt

$ elixir 00-test.exs
Requests served: 0
Requests served: 0
Requests served: 2856
Requests served: 11161
Requests served: 16352
Requests served: 16352
Requests served: 16352
Requests served: 16352
Requests served: 16352
Requests served: 16352
Requests served: 16352
Requests served: 16352
Requests served: 16352
Requests served: 16352
Requests served: 16352
Requests served: 16352
Requests served: 16352
Requests served: 16354
Requests served: 16354
Requests served: 16354
Requests served: 16354
Requests served: 18000
Requests served: 18000
^C

observations.md

It always hangs extremely close to 2^14, which is the default Erlang max open ports

The longest request in each case is very similar, perhaps something timing out?