Experimenting with Fedora Core OS

As part of BUGOUT, we want to attempt supporting Fedora CoreOS. We previously ran our stateful container deployment on CoreOS Container Linux and enjoyed the automatic OS upgrades. It used a relatively small amount of RAM. We were too lazy to move beyond using docker-compose, and that wasn't a problem: after writing a couple of tiny systemd configs, our system always started up reliably.

But CoreOS Container Linux is being put to pasture, so we've decided to try the new thing.

First Steps with FCCT

We need to create an ignition file which is intended to be written once, and valid for the life of the image.

See some docs for getting started with this configuration tool.

They recommend using podman instead of docker, but there's no snap install available that doesn't warn about potentially stomping on our localdev ❤️ Debian ❤️ system.

sudo snap install --edge podman
error: The publisher of snap "podman" has indicated that they do not consider this revision to be
       of production quality and that it is only meant for development or testing at this point. As
       a consequence this snap will not refresh automatically and may perform arbitrary system
       changes outside of the security sandbox snaps are generally confined to, which may put your
       system at risk.

       If you understand and want to proceed repeat the command including --devmode; if instead you
       want to install the snap forcing it into strict confinement repeat the command including
       --jailmode.

Use docker to produce the ignition file, instead. First we created an example YAML file that fcct could read. This was a bit unclear based on the current state of the documentation, as we used .yaml for the file extension instead of .fcc.

# local example.yaml
variant: fcos
version: 1.0.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa AAAAB3NzaC1yc...
docker pull quay.io/coreos/fcct:release
docker run  -i --rm quay.io/coreos/fcct:release --pretty --strict <  input.yaml > example.ign

As promised, this command output an ignition file:

{
  "ignition": {
    "config": {
      "replace": {
        "source": null,
        "verification": {}
      }
    },
    "security": {
      "tls": {}
    },
    "timeouts": {},
    "version": "3.0.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-rsa AAAAB3NzaC1yc..."
        ]
      }
    ]
  },
  "storage": {},
  "systemd": {}
}

Launching an Instance on AWS

We can launch an instance on AWS. See the operating system getting started page.

The docs currently ask you to use the aws command line interface. I'm too lazy to do that. I don't want to (re)learn the options for the ec2 run instances command. I just want to plug some values in via the web interface.

To do that, click through the downloads page, look for for your AWS region, click through to the launch instance page within AWS, then look for the "user data" details.

user data for your igntion config

You can enter your ignition config here.

Finally, you launch the instance and connect:

Fedora CoreOS 31.20200407.3.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/

Setting up the system

Right now we're manually exploring the system to see how things feel.

git clone https://github.com/Terkwood/BUGOUT.git
cd BUGOUT
sudo usermod -aG docker $USER  # we'll fix this in next section
sudo reboot   # reboot helped

You can use ctop:

docker run -ti -v /var/run/docker.sock:/var/run/docker.sock quay.io/vektorlab/ctop:latest

You cannot use toolbox. It isn't really supposed to work due to the permissions structure of FCOS.

You can use rpm-ostree to install htop:

sudo rpm-ostree install htop
sudo systemctl reboot  # you need to reboot to get access to it

Exploring Packer

We are clearly going to need to manage the creation of this image. Let's try using Packer.

Here is their basic guide to build an image.

We can move on to running something a bit more interesting.

You first need to write a program.

cat >hello.ts
console.log("Welocem Friend 🦕");

You need some env vars specified.

# set_deno_env.sh
export VPC_ID="vpc-deadbeef"
export SUBNET_ID="subnet-bad1dea5"

...then...

source set_deno_env.sh

Write some packer-example.json

{
    "variables": {
        "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
        "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
        "region":         "us-east-1",
        "vpc_id":         "{{env `VPC_ID`}}",
        "subnet_id":      "{{env `SUBNET_ID`}}"
    },
    "builders": [
        {
            "access_key": "{{user `aws_access_key`}}",
            "ami_name": "packer-linux-aws-demo-{{timestamp}}",
            "instance_type": "t3.micro",
            "region": "{{user `region`}}",
            "vpc_id": "{{user `vpc_id`}}",
	          "subnet_id": "{{user `subnet_id`}}",
            "secret_key": "{{user `aws_secret_key`}}",
            "source_ami_filter": {
              "filters": {
              "virtualization-type": "hvm",
              "name": "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*",
              "root-device-type": "ebs"
              },
              "owners": ["099720109477"],
              "most_recent": true
            },
            "ssh_username": "ubuntu",
            "type": "amazon-ebs"
        }
    ],
    "provisioners": [
        {
            "type": "file",
            "source": "./welcome.txt",
            "destination": "/home/ubuntu/"
        },
        {
            "type": "file",
            "source": "./hello.ts",
            "destination": "/home/ubuntu/"
        },
        {
            "type": "shell",
            "inline":[
                "ls -al /home/ubuntu",
                "cat /home/ubuntu/welcome.txt"
            ]
        },
        {
            "type": "shell",
            "inline": [
              "sudo apt install -y unzip",
              "curl -fsSL https://deno.land/x/install/install.sh | sh"
              
            ]
        },
        {
            "type": "shell",
            "inline":[
              "export DENO_INSTALL=\"/home/ubuntu/.deno\"",
              "export PATH=\"$DENO_INSTALL/bin:$PATH\"",
              "deno hello.ts"
            ]
        }
    ]
}

You'll see a bunch of glorious progress, and finally, an artifact:

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:
us-east-1: ami-eeeeeeeeeeeeeeeea

Don't forget to deregister the AMI after you're done!

Putting It All Together With FCOS

That's all well and good, and we're happy about packer.

But we need to usable FCOS image which has a few modifications that packer, rather than ignition is well-suited to handle.

Create an ignition file which you'll include in the packer config:

variant: fcos
version: 1.0.0
passwd:
  users:
    - name: core
      groups: [ docker ]
      ssh_authorized_keys:
        - ssh-rsa AAAA...
storage:
  files:
    - path: /opt/bin/docker-compose
      overwrite: true
      mode: 0755
      contents:
        source: https://github.com/docker/compose/releases/download/1.13.0/docker-compose-Linux-x86_64
        verification:
          hash: sha512-9d2c4317784999064ba1b71dbcb6830dba38174b63b1b0fa922a94a7ef3479f675f0569b49e0c3361011e222df41be4f1168590f7ea871bcb0f2109f5848b897
docker run  -i --rm quay.io/coreos/fcct:release --pretty --strict <  packed.yaml > packed.ign

Then write your packer config:

{
    "variables": {
        "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
        "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
        "region":         "us-east-1",
        "vpc_id":         "{{env `VPC_ID`}}",
        "subnet_id":      "{{env `SUBNET_ID`}}"
    },
    "builders": [
        {
            "access_key": "{{user `aws_access_key`}}",
            "ami_name": "fcos-linux-aws-demo-{{timestamp}}",
            "instance_type": "t3.medium",
            "region": "{{user `region`}}",
            "vpc_id": "{{user `vpc_id`}}",
            "subnet_id": "{{user `subnet_id`}}",
            "secret_key": "{{user `aws_secret_key`}}",
            "user_data_file": "packed.ign",
            "source_ami_filter": {
              "filters": {
                "virtualization-type": "hvm",
                "name": "fedora-coreos-31.20200407.3.0",
                "root-device-type": "ebs"
              },
              "owners": ["125523088429"],
              "most_recent": true
            },
            "ssh_username": "core",
            "type": "amazon-ebs"
        }
    ],
    "provisioners": [
        {
            "type": "shell",
            "inline": [
                "sudo rpm-ostree install htop"
            ]
        },
        {
            "type": "shell",
            "inline":[
                "git clone https://github.com/Terkwood/BUGOUT.git"
            ]
        }
    ]
}

...and we have a baked image!

We used docker-compose 1.13 instead of 1.25 because python3.7 fails to link to libcrypt.so.1 on Fedora 31. ☹️

Launch Templates Are Helpful

Creating a launch template makes the AWS CLI invocation less annoying.

You can launch the instance using minimal parameters:

aws ec2 run-instances --launch-template LaunchTemplateId=lt-0000,Version=2 --image-id ami-0000 --subnet-id subnet-deadbeef

In the example above, we pasted the ignition user data into the web form. But you can override user data at the command line, if desired.

Completing the ignition

Next up, write some systemd data and figure out how to seat some dev.env files as .env files.

Using journalctl for monitoring

You can follow logs with journalctl:

journalctl -u bugout -f

DigitalOcean has a nice tutorial.

Friendly Memory Footprint

We were pleasantly surprised by amount of memory saved by switching from Container OS to Fedora Core OS. We haven't done any extensive testing yet, but the /proc/meminfo shown below were for the same workload (kafka box) under the same conditions (1-3 users playing, system online for less than 15 minutes).

Our old t3.medium Container Linux VM:

MemTotal:        3979308 kB
MemFree:         1793524 kB
MemAvailable:    1920252 kB
Buffers:           67600 kB
Cached:           449764 kB
SwapCached:            0 kB
Active:          1760416 kB
Inactive:         280408 kB

The new t3.medium Fedora CoreOS VM:

MemTotal:        3962968 kB
MemFree:         2971716 kB
MemAvailable:    3439920 kB
Buffers:            1108 kB
Cached:           626192 kB
SwapCached:            0 kB
Active:           414104 kB
Inactive:         405596 kB