kevinschweikert.de

Composable SFTP streams in Elixir

2026-04-26T00:00:00Z

Composable SFTP streams in Elixir

Published on April 26, 2026 · f294096e

I'm still new to writing and I want to disclose that AI helped in creating this post. All words are my own but AI was used in spelling correction and structural discussions.

Recently I was tasked to add media file sync feature via SFTP. OTP has all the necessary tools for this so I started with:

{:ok, channel, connection_ref} =
  :ssh_sftp.start_channel("example.com", 22)

video_file = File.read!("my_video.mp4")
:ok = :ssh_sftp.write_file(channel, "/var/media/video.mp4", video_file)

But there's a big (literally!) problem. By using File.read!/1 the whole file gets loaded into memory. For small files this is fine but in our case we had intraframe heavy video files which can be hundreds of GB big. We don't want them to be loaded into RAM. But Elixir has a solution for this - lazy enumerables.

Elixir differentiates between two kinds of enumerables. Eager (Enum module) and lazy (Stream module). Both build on the Enumerable protocol but differ in how they are consumed. Lazy means the enumerable is only calculated when consumed.

We can enumerate the contents of a file lazily by opening it as a stream with File.stream!/2:

chunk_size = 2 ** 16
file_stream = File.stream!("my_big_video.mov", chunk_size)

Because this is a Stream the file will be read in 64KB chunks one after another when we enumerate it. Until then, nothing will be loaded into memory. The 64KB are a tradeoff between memory consumption and network overhead. This can be tuned for specific use cases.

We could end it here and use it like this:

{:ok, handle} =
  :ssh_sftp.open(channel, "/archive/my_big_video.mov", [:write, :binary])

for chunk <- file_stream do
  :ok = :ssh_sftp.write(channel, handle, chunk)
end

:ok = :ssh_sftp.close(channel, handle)
:ok = :ssh.close(connection_ref)

The for comprehension enumerates the stream, and now the chunks will be consumed

From a functional standpoint this works as desired. But from an architectural point of view it has several drawbacks:

No cleanup on failure. If :ssh_sftp.write/3 returns an error, we raise. If we remove the match, we'd have to inspect the resulting chunk results.
Leaks the Erlang API. Every place that uploads a file has to know about :ssh_sftp.open/3, :ssh_sftp.write/3 and :ssh_sftp.close/2
Not a value. We'd have to wrap everything in a function but now the caller has to actually call it. An external library would need to know, how to use it.

Let's go back to the file streaming example and extend it to a sink, because File.stream!/2 has one more trick up its sleeve! It also implements the Collectable protocol.

source_stream = File.stream!("my_big_video.mov", chunk_size)
destination_stream =
  File.stream!("/archive/my_big_video.mov", chunk_size)

Stream.into(source_stream, destination_stream)
|> Stream.run()

Stream.into/2 (or Enum.into/2 for the eager version) is the missing link here, and consumes the enumerable from the first argument and collects it into the second argument.

This is basically what we want to do but with the SFTP stream as the sink.

In Elixir a protocol can be implemented on any datatype including custom structs! File.stream!/2 returns a %File.Stream struct which implements Enumerable and Collectable

This gives us all we need. Protocols are an Elixir concept so we don't have a streaming collectable SFTP sink in OTP. But nothing stops us from implementing it on our own data structure.

Let's write a simple example to understand the structure of the Collectable protocol:

defmodule MyInspector do
  defstruct []

  defimpl Collectable do
    def into(struct) do
      collector_fun = fn
        _acc, {:cont, chunk} -> IO.inspect(chunk)
        acc, :done -> acc
        acc, :halt -> acc
      end

      {struct, collector_fun}
    end
  end
end

into/1 returns a tuple of the {initial accumulator, collector_fun}. The collector is a 2-arity function called for each step in the enumeration:

acc, {:cont, chunk}: called on a new chunk
acc, :done: called when enumerable has no more chunks
acc, :halt: called when enumeration is aborted

And put it to action:

source_stream = File.stream!("my_big_video.mov", chunk_size)

Stream.into(source_stream, %MyInspector{})
|> Stream.take(5)
|> Stream.run()

Let's combine everything we've learned into implementing the Collectable protocol for a custom SFTP stream.

defmodule SFTP.Stream do
  defstruct [:channel, :path, :handle, :status]

  def build(channel, path) when is_pid(channel) and is_binary(path),
    do: %__MODULE__{channel: channel, path: path}

  defimpl Collectable do
    def into(%{channel: channel, path: path} = stream) do
      {:ok, handle} = :ssh_sftp.open(channel, path, [:write, :binary])
      stream = %{stream | handle: handle, status: :ok}

      collector_fun = fn
        %__MODULE__{status: :ok, handle: handle} = stream , {:cont, chunk} ->
          case :ssh_sftp.write(channel, handle, chunk) do
            :ok -> stream
            {:error, err} -> %{stream | status: err}
          end
        
        stream , {:cont, _chunk} ->
          # status != :ok
          # we don't do anything anymore and wait for the :done call
          stream

        %__MODULE__{status: status, handle: handle, channel: channel}, :done -> 
          :ssh_sftp.close(channel, handle)
          status

        %__MODULE__{handle: handle, channel: channel}, :halt ->
          :ssh_sftp.close(channel, handle)
      end

      {stream, collector_fun}
    end
  end
end

Back to our example but this time with our new implementation:

file_stream = File.stream!("my_big_video.mov", chunk_size)

{:ok, channel, connection_ref} =
  :ssh_sftp.start_channel("example.com", 22)
sftp_stream = SFTP.Stream.build(channel, "my_big_video.mov")

Stream.into(file_stream, sftp_stream)
|> Stream.run()

:ok = :ssh.close(connection_ref)

Notice that :ssh_sftp.start_channel/2 lives in the caller's code and not inside the Collectable implementation. That's by design: opening an SSH connection is expensive (network, key exchange, auth) and we may want to transfer many files through one channel. We can always wrap it in a convenience function if needed.

Remember the file transfer example? This is exactly the same, we just swapped the destination file stream for an SFTP file stream. Without changing the code which loads from a file or how the chunks are consumed!

Why bother implementing protocols?

The protocol implementation seems more complicated than the for comprehension. So why should we go through the trouble of implementing it?

Because now it is flexible to work with. The Collectable protocol is defined in the Elixir standard library, so any other project can either implement it or work with it.

For example the Req library accepts a Collectable with the :into option. This allows us to stream an HTTP response into a file over SFTP:

sftp_stream = SFTP.Stream.build(channel, "my_downloaded_file")
Req.new(url: "https://example.com/download", into: sftp_stream)
|> Req.get!()

Remember the for comprehension from the beginning? It also has an :into option:

for chunk <- file_stream, into: sftp_stream, do: chunk

What's important is that the caller never needs to know any of this. From the outside, our SFTP destination looks identical to File.stream!/2. We can swap one for the other, compose them with Stream.into/2, pass the resulting stream to a Task or a worker queue — all without changing the pipeline.

Turns out, it wasn't really about SFTP. It was about using the right abstractions and creating a building block. Think about it next time when you upload files to S3 or create a big CSV.

If you want to play around with this concepts use this Livebook

Build Home Assistant Devices with Elixir and Nerves

2026-01-18T00:00:00Z

Build Home Assistant Devices with Elixir and Nerves

Published on January 18, 2026 · updated on January 26, 2026 · af13e2bb

I love Home Assistant. I love that I can throw in all these devices from different manufacturers, display their data in a unified dashboard and write automations that combine devices never intended to know of each other. Like switching on Philips Hue lights when the IKEA door sensor opens, or turning on the switch of a power outlet to activate a dehumidifier when an Aqara humidity sensor reports high values.

But wouldn't it be nice to integrate devices we built ourselves?

I was working through the Book Build a Weather Station with Elixir and Nerves where the authors show how to integrate sensors for temperature, humidity, pressure and air quality to then collect the measurements on a Nerves. In the second part of the book, we learn how to create a Phoenix API to receive the measurements and how to set up Grafana to visualize them. This is extremely cool but still does not show all my other devices or let's me control them.

So, last year I had the idea to combine the learnings from the book and my own Home Assistant journey and posted in the Elixir Forum my initial ideas on how to tackle this.

I've outlined three options on how to communicate and interact with Home Assistant.

I quickly discovered that you cannot create new entities via the REST/WebSocket API and that at least the first iteration should be all in Elixir instead of writing a whole new integration in Python. So I settled on MQTT.

Before we start, I want to clarify some wording i'll be using:

entity - one thing you can see/control (switch, sensor, camera)
device - a thing (often physical) that groups entities (the Nerves weather station)
integration - the code that let's Home Assistant interact with a device/brand (MQTT, Zigbee, Unifi, …)
platform - the entity type within an integration (switch, sensor, camera)

MQTT

[Elixir Application] <--> [MQTT Broker] <--> [Home Assistant]

To add our own entities, Home Assistant has a built-in auto discovery mechanism when using MQTT. There are different options like device or single component discovery but here's what we are doing.

Discovery

Home Assistant listens on a pre-configured MQTT topic for a JSON payload which is the discovery message. The default topic is:

/homeassistant/device/[YOUR_DEVICE_ID]/config

and it expects a payload like this:

{
  "device": {
    "name": "My Device",
    "identifiers": [
      "My Device"
    ]
  },
  "origin": {
    "name": "homex"
  },
  "components": {
    "my_switch_44962374": {
      "name": "my-switch",
      "platform": "switch",
      "unique_id": "my_switch_44962374",
      "state_topic": "homex/switch/my_switch_44962374",
      "command_topic": "homex/switch/my_switch_44962374/set"
    }
  }
}

platform: this is a common and required field for all components and lets Home Assistant know how to treat it and which configuration values to expect
unique_id: important when you want to have Home Assistant remember the component across restarts

All other fields are dependent on the kind of platform. A detailed documentation can be found on the Home Assistant website, e.g. MQTT Switch

Here, Home Assistant publishes commands to state_topic, and your app publishes updates to command_topic.

Client

There are only a few MQTT clients in Elixir and i've settled on emqtt which is an Erlang library made by the EMQX folks. EMQX is a MQTT broker and the client seems actively developed. It is also the only one which supports MQTT 5.0. The only current thing to look out for are some native/NIF dependencies for QUIC support which are hard to build on Nerves. But we can override if we want to when using the git dependency:

def deps() do
  ...
  {:emqtt, github: "emqx/emqtt", system_env: [{"BUILD_WITHOUT_QUIC", "1"}]}
end

Building it

The following examples focus on functionality and ignore error handling in the most places. If you want to jump to the finished version go to Homex

Manager

Let's create a GenServer, creatively called Homex.Manager to handle the client life-cycle and discovery mechanism. Here we send an empty components payload on startup:

defmodule Homex.Manager do
  use GenServer

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end
  
  @impl GenServer
  def init(_opts) do
    {:ok, emqtt_pid} = :emqtt.start_link(host: "localhost", user: "admin", password: "admin")
    :ok = :emqtt.connect(emqtt_pid)

    {:ok, emqtt_pid, {:continue, :discovery}}
  end

  @impl GenServer
  def handle_continue(:discovery, emqtt_pid) do

    # TODO: lookup entities to fill discovery components
  
    discovery_config = %{
      device: %{name: "Homex", identifiers: ["Homex"]},
      origin: %{name: "homex"},
      components: %{}
    }

    payload = Jason.encode!(discovery_config)
    :emqtt.publish(emqtt_pid, "/homeassistant/device/homex/config", payload)
  end
end

Now we need to know when entities are added or when they are removed. They need to somehow register themselves and let the manager know about them. This sounds like a job for the Elixir Registry module. We can start a new Registry in our application and set the Manager as a listener, so that we get notified when an Entity registers itself there:

{Registry,
  name: Homex.SubscriptionRegistry,
  keys: :duplicate,
  listeners: [Homex.Manager]
}

The keys will be the subscribe topics and we will receive the following events in Homex.Manager:

def handle_info({:register, _registry, topic, _pid, _value}, emqtt_pid) do
  :emqtt.subscribe(emqtt_pid, topic)
  {:noreply, emqtt_pid, {:continue, :discovery}}
end

def handle_info({:unregister, _registry, topic, _pid}, emqtt_pid) do
  :emqtt.unsubscribe(emqtt_pid, topic)
  {:noreply, emqtt_pid, {:continue, :discovery}}
end

Perfect! Now we get notified when a new Entity registers itself and we subscribe to the requested topics. If a new message on the subscribed topics arrives, we have to delegate that message to the registered entities. We can make use of Registry.dispatch/4 to implement our own PubSub mechanism:

def handle_info({:publish, %{topic: topic, payload: payload}}, emqtt_pid) do
  Registry.dispatch(Homex.SubscriptionRegistry, topic, fn registered ->
    for {entity_pid, _value} <- registered do
      send(entity_pid, {:message, topic, payload})
    end
  end)

  {:noreply, emqtt_pid}
end

We just need to give the future entities a way to publish new messages in return:

def publish(topic, message) do
  GenServer.call(__MODULE__, {:publish, topic, message})
end

def handle_call({:publish, topic, message}, _from, emqtt_pid) do
  payload = Jason.encode!(message)
  result = :emqtt.publish(emqtt_pid, topic, payload)
  {:reply, result, emqtt_pid}
end

Entity

Almost done! Before we implement the real entities, we plan ahead and define a common interface for all future entity implementations:

defmodule Homex.Entity do
  @callback unique_id() :: String.t()
  @callback component() :: Map.t()
end

Alright, final stretch! This is the base for our switch entity:

defmodule Homex.Entity.Switch do
  use GenServer

  @name "My Switch"
  @platform "switch"
  @unique_id "my_switch_#{:erlang.phash2([@platform, __MODULE__])}"
  @state_topic "/homex/switch/my_switch"
  @command_topic "/homex/switch/my_switch/set"

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end
  
  @impl GenServer
  def init(_opts) do
    Registry.register(Homex.SubscriptionRegistry, @state_topic, __MODULE__)
    {:ok, :off}
  end

  @impl GenServer
  def handle_info({:message, @state_topic, payload}, state) do
    case payload do
      "ON" -> {:noreply, :on}
      "OFF" -> {:noreply, :off}
    end
  end
end

We defined a stable unique ID, registered our process in the registry and will handle a message on the state topic. We don't have to handle the command topic, because we will only write to it. Let's also implement the Entity behaviour, we defined earlier:

@behaviour Homex.Entity

def unique_id(), do: @unique_id

def component() do
  %{
    name: @name,
    platform: @platform,
    unique_id: @unique_id,
    state_topic: @state_topic,
    command_topic: @command_topic
  }
end

This is enough to handle state updates from the Home Assistant side but we also want to manipulate the switch from our side. Let's define a public API and the necessary handlers:

def toggle() do
  GenServer.cast(__MODULE__, :toggle)
end

@impl GenServer
def handle_cast(:toggle, :on) do
  Homex.Manager.publish(@command_topic, "OFF")
  {:noreply, :off}
end

def handle_cast(:toggle, :off) do
  Homex.Manager.publish(@command_topic, "ON")
  {:noreply, :on}
end

We can start the Homex.Manager statically but the entities are more dynamic. So we will use a DynamicSupervisor. The supervisor can tell us which children are started. That's why we don't need a separate Registry to figure out which modules/components are started and should be included in the discovery message:

{DynamicSupervisor, name: Homex.EntitySupervisor, strategy: :one_for_one}

This starts a new DynamicSupervisor with the strategy :one_for_one which only restarts the crashed process and no other children:

def add_entity(module) do
  GenServer.call(__MODULE__, {:add_entity, module})  
end

def handle_call({:add_entity, module}, _from, emqtt_pid) do
  {:ok, _pid} = DynamicSupervisor.start_child(Homex.EntitySupervisor, module)
  {:reply, :ok, {:continue, :discovery}}
end

And last but not least we can update our handle_continue/2 function to send a full discovery message containing all supervised entities:

def handle_continue(:discovery, emqtt_pid) do
  entities =
    DynamicSupervisor.which_children(Homex.EntitySupervisor)
    |> Enum.map(fn {_id, _pid, _type, modules} -> modules end)
    |> List.flatten()

  components =
    for module <- entities, into: %{} do
      {module.unique_id(), module.component()}
    end

  discovery_config = %{
    device: %{name: "Homex", identifiers: ["Homex"]},
    origin: %{name: "homex"},
    components: components
  }

  payload = Jason.encode!(discovery_config)
  :emqtt.publish(emqtt_pid, "/homeassistant/device/homex/config", payload)
end

Let's try it out!

1> Homex.Manager.start_link([])
{:ok, _pid}
2> Homex.Manager.add_entity(Homex.Entity.Switch)
:ok
3> Homex.Entity.Switch.toggle()
:ok
4> Homex.Entity.Switch.toggle()
:ok

We should see the switch entity in Home Assistant and we can see that it gets toggled whenever we call toggle/0 on the switch module.

Homex

I am excited to tell you, that all the work above and more is already available as an Elixir library called homex.

A bridge between Elixir and Home Assistant

Homex takes care of

Autodiscovery
Supervision
Automatic Reconnection
Message deduplication
Timed measurements

Be it a temperature sensor or a light switch. This makes it especially interesting for Nerves devices to integrate with some hardware directly but you can also use it to switch feature flags in your day to day Elixir application.

What can you build with it?

Homex is just a generic framework to create and control entities in Home Assistant. Here are some ideas what you can build with it.

Weather station like in the book from the beginning
Surveillance camera with the Raspberry Pi Camera
Sensor hub for RF devices
Water management system - Measure water levels and control valves
Report application metrics

But in the end it is only limited by the supported MQTT platforms and your imagination

Basic usage

To create a new switch entity in Homex you define a new module using the Homex.Entity.Switch macro. After passing a name you can implement different callbacks as seen in the docs. In this demo we restrict ourselves to

handle_on/1
handle_off/1

These are called when you switch the light on or off from the Home Assistant side, either manually or via automations. Here we just print the received value to the console to see that it works:

defmodule MySwitch do
  use Homex.Entity.Switch, name: "my-switch"

  def handle_on(entity) do
    IO.puts("Switch turned on")
    entity
  end

  def handle_off(entity) do
    IO.puts("Switch turned off")
    entity
  end
end

To expose the new entity to Home Assistant you have to set up a MQTT broker (see Setting up a broker) and tell Homex how to connect. This could look like this:

config :homex,
  broker: [
    host: "localhost",
    port: 1883,
    username: "admin",
    password: "admin"
  ]

Finally we can start Homex and add the switch:

Homex.start_link([])
Homex.add_entity(MySwitch)

Typically you would start Homex in your supervision tree.

After that we can see the new entity in Home Assistant and some logs in our application when we toggle the switch on and off.

13:27:17.973 [info] added entity my-switch
Switch turned on
Switch turned off
Switch turned on

You can optionally specify a list of entities in the config and Homex will add them automatically on startup

More advanced use cases

To store data in the process state we can use put_private/3 and get_private/2. This can be a reference to a sensor or in this case the reader of a webcam. With this and some image magic we are able to stream pictures right into Home Assistant:

defmodule MyCamera do
  use Homex.Entity.Camera, 
    name: "my-camera", 
    image_encoding: "b64", 
    update_interval: 1_000

  def handle_init(entity) do
    r = Xav.Reader.new!("C922 Pro Stream Webcam", device?: true)
    entity |> put_private(:reader, r)
  end

  def handle_timer(entity) do
    r = entity |> get_private(:reader)
    {:ok, %Xav.Frame{} = frame} = Xav.Reader.next_frame(r)

    {:ok, img} = Vix.Vips.Image.new_from_binary(frame.data, frame.width, frame.height, 3, :VIPS_FORMAT_UCHAR)

    img = img |> Image.thumbnail!(200)
    binary = img |> Image.write!(:memory, suffix: ".jpg") |> Base.encode64()

    entity
    |> set_image(binary)
    |> set_attributes(%{width: Image.width(img), height: Image.height(img)})
  end
end

Closing thoughts

Playing around with home automation is fun. Playing around with Elixir is fun! Playing around with both of them is even more fun!

Homex is still young and evolving. It works, but expect some breaking changes in the early versions. I've already received some feedback and these are the things i want to tackle next

Make entities more flexible to define them at runtime and allow multiple instances
Allow support for multiple logical devices, so a hub can expose and delegate devices
Add APIs to query the state of the entities
Optional telemetry hooks
Expand support for more MQTT platforms
Make using/building emqtt easier

I’d also love to integrate Homex into existing projects, for example:

Nerves itself (system metrics)
Soleil
Govee lights

You even want to write your automations in Elixir? You're in luck - there's already an article on how to do exactly that Writing Home Assistant automations using Genservers in Elixir by Jonas Hietala.

For the long term, Matter could be an option to not just support the Home Assistant use-case but integrate in the home automation hub of your choice.

Do you have ideas where Homex could improve or be used? Let me know

Update 2026-01-26

I presented Homex in the Nerves Meetup EU. Here's the link to the recording: Bringing Nerves and Home Assistant together