How to set up an Infrastructure as Code automation using Pulumi, Civo Object Store, and GitHub Actions

Infrastructure as Code (IaC) is a software development practice that manages infrastructure through code instead of through a manual process. By doing this, the infrastructure changes can be expressed in code, stored in a git repository, and reviewed together by a team in a pull request process.

One of the tools widely used for IaC is Pulumi, an open source software that can enable a team to write its infrastructure code using common programming languages like Python, Javascript, and Go. By default, Pulumi uses Pulumi Cloud, a paid service provided by Pulumi, as the backend to store the state of the infrastructure managed by Pulumi. Still, it’s possible to use S3-compatible storage as a replacement.

In this tutorial, we will use Civo Object Store as Pulumi backend, Python code to express the infrastructure, and GitHub Actions to facilitate IaC automation in a development process with pull requests.

Prerequisites

To complete this tutorial, you will need the following:

• Civo account
• Civo CLI installed
• Pulumi CLI installed
• Two GitHub accounts to simulate a pull request process: one account as the GitHub repo owner and one account as the collaborator who creates the PR.

Also, as part of this tutorial, you will create a Civo Object Store and a Civo Compute instance. The pricing for both of these can be checked on the Civo pricing page.

Creating a new Pulumi project

Before using Civo Object Store to store Pulumi state, we will store it in our local storage. To do this, run this command in your terminal:

pulumi login --local

After running that command, Pulumi will tell you that you are now logged in to your local computer in file://~ path.

Now in your home directory, there is a directory named .pulumi. Pulumi uses this directory to store your credentials and state files. You can check this by running the following command.

Next, you need to set the PULUMI_CONFIG_PASSPHRASE environment variable. Pulumi will treat the value of this environment variable as the password for your Pulumi state. The most important thing for now is to choose something memorable and keep it secret.

export PULUMI_CONFIG_PASSPHRASE=<your-secret-passphrase>

Now, create a new directory called my-infra. Here, you will store all your Pulumi codes to manage your infrastructure:

mkdir my-infra
cd my-infra

Next, you need to create a new Pulumi project before you start writing the code for your infrastructure:

pulumi new civo-python -y

In the above command, civo-python is the name of the template that we use to create our new Pulumi project. Pulumi provides templates for many cloud providers and programming languages. Once you run the command, Pulumi will create a new project using civo-python template with default settings. It will:

• Create a new project with the same name as your current directory name, my-infra
• Create a stack named dev
• Create a Python virtual environment inside venv directory
• Install necessary Python packages inside the Python virtual environment just created

In Pulumi, a stack is a configurable instance of a Pulumi program. It is usually used to denote different environments in your infrastructure, such as development, staging, and production. The basic idea is that we write the same code for all stacks, but each stack differs in its configuration.

The content in your current directory now will be like this ↓

Explanation for each of the items above:

Creating a new Civo Object Store

Replace the content of main.py with the code below:

import pulumi
import pulumi_civo as civo

region = "LON1"

obj_store_cred = civo.ObjectStoreCredential("my-obj-store-cred", region=region)

obj_store = civo.ObjectStore("pulumi-state",
                            access_key_id=obj_store_cred.access_key_id,
                            max_size_gb=500,
                            region=region)

pulumi.export('obj_store_cred_access_key', obj_store_cred.access_key_id)
pulumi.export('obj_store_url', obj_store.bucket_url)
pulumi.export('obj_store_name', obj_store.name)

Get your Civo API key from your dashboard and set it as the CIVO_TOKEN environment variable in the terminal. The key is in the security section of the Civo dashboard.

export CIVO_TOKEN=<your civo api key>

To get a preview of what changes will be applied by Pulumi, run pulumi preview.

From the output, you can see that once you apply the changes, Pulumi will create 3 things: a stack named dev, a Civo Object Store credential, and a Civo Object Store.

Now, apply the changes:

pulumi update -y

You can confirm the changes by opening your Civo dashboard. You will find a new object store and a new object store credential with the same names mentioned in the output from the previous command. For more information about Civo’s Object store, refer to our object store documentation.

Now that you already have an object store, you can use it to store the Pulumi state there.

Using Civo Object Store as Pulumi backend

In Pulumi, a state represents the resources managed by Pulumi along with their attributes. For example, in the previous section, you created an object store credential and an object store using Pulumi. The state of those two resources is now stored in a Pulumi state.

Because you ran pulumi login --local previously, you use your local machine as your Pulumi backend. This means that the files representing the Pulumi state are now in your local files. If you check the content of ~/.pulumi directory, there is now a stacks directory. Inside it, there is a directory named my-infra that represents your project. Inside that directory are all the files used by Pulumi to store the states of all the resources managed by Pulumi.

Before you switch to using Civo Object Store as your Pulumi backend, you need to export your stack into a JSON file.

pulumi stack export --file stack.json

This stack.json file now has all the information related to your current stack state.

The next thing to do is to set some environment variables so that you can use the Civo Object Store as your Pulumi backend. You will use the Civo CLI to get the information for those environment variables. Run the following command to get the information about your object store credential:

civo objectstore credential export \
  --access-key=$(pulumi stack output obj_store_cred_access_key) \
  --region LON1

The output will show you 4 lines of export commands that you need to run to set the required environment variables to use Civo Object Store as Pulumi backend. Copy those lines and run them.

You can now use the Civo Object Store as your Pulumi backend.

Run the following Pulumi command:

pulumi login "s3://$(pulumi stack output obj_store_name)?endpoint=$(pulumi stack output obj_store_url)&disableSSL=true&s3ForcePathStyle=true"

The output should show that you are successfully logged in to the Civo Object Store. The thing is the Pulumi backend knows nothing about your dev stack and the resources that you created previously using the local backend. To complete the backend switching process, run the following commands to create the dev stack and import the stack state using the stack.json file that you created before.

pulumi stack init dev
pulumi stack import --file stack.json

Run the following command to confirm that the state has been stored in your new backend. It will show you all the resources the current stack and backend manages:

pulumi stack

You have successfully migrated your state from the local backend to the Pulumi backend, so you don’t need the stack.json file anymore. You can now delete it.

rm stack.json

Creating a GitHub repo

To simulate the pull request process in this tutorial, you will create a new GitHub repo to store your infrastructure code. In my example I will use my personal GitHub user, rizaldim, for this.

Go to your GitHub dashboard. Make sure that you have signed in. At the right side of the search bar, you will find a button with a plus icon. Click that and then click New Repository.

On the new repository page, type in your repository name, such as my-infra. Choose public as the type of the repo. Click create repository.

You will now see the quick setup instructions. Go back to your terminal, then run the following commands to create a new git repo in your working directory:

git init
git add .
git commit -m ‘Initial commit’

Now go back to your browser. Find the section showing the instructions on how to push an existing repository to your new GitHub repo. Click the copy button ↓

Go back to your terminal, paste the commands, and press Enter to run them.

Go back to your browser and refresh the page. You will now see that your repository in GitHub has your Pulumi files.

Creating a GitHub Actions workflow for PR creation

The next step is to create the automation needed for your pull request process. For this, you will use GitHub Actions. You will create 2 workflows:

• A workflow for anytime a PR is created. When this happens, your automation will run pulumi preview and create a new comment with the preview output. By doing this, the PR creator and the reviewer can review what will change in the infrastructure with the Pulumi code changes.
• A workflow for whenever the PR has been approved. When this happens, your automation will run pulumi update and apply the changes in the infrastructure.

First, go back to your terminal and create a new directory for GitHub Actions workflows. Create 2 new files inside: pr-created.yaml and pr-approved.yaml.

mkdir -p .github/workflows
touch .github/workflows/{pr-created.yaml,pr-approved.yaml}

Open pr-created.yaml with your code editor and paste the following code into the file.

name: Pulumi preview
on:
  pull_request:
    type: [opened]
    branches: [main]

permissions:
  pull-requests: write

jobs:
  preview:
    name: Preview
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3

    - uses: pulumi/actions@v5
          with:
          command: preview
          stack-name: organization/my-infra/dev
          cloud-url: ${{ vars.PULUMI_BACKEND_URL }}
          comment-on-pr: true
          env:
          PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
          CIVO_TOKEN: ${{ secrets.CIVO_TOKEN }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_DEFAULT_REGION: ${{ vars.AWS_DEFAULT_REGION }}
          AWS_HOST: ${{ vars.AWS_HOST }}

Save the file and commit the changes:

git add .
git commit -m ‘Add workflow for when pr is opened’

Next, you need to create variables and secrets for your GitHub repository. Go back to your repository page in GitHub. Click Settings. In the left-side menu, click Secrets & Variables and then click Actions.

In the Actions secrets and variables page, click New repository secret.

First, you need to create the AWS_ACCESS_KEY_ID secret. Type in AWS_ACCESS_KEY_ID in the Name field. Paste the value of AWS_ACCESS_KEY_ID environment variable in your terminal in the Secret field. You can get the value by running echo $AWS_ACCESS_KEY_ID in your terminal. After pasting it in the new secret form, click Add secret.

Then, repeat the steps for creating 3 more secrets:

• AWS_SECRET_ACCESS_KEY
• PULUMI_CONFIG_PASSPHRASE
• CIVO_TOKEN

You can get the value for these secrets by inspecting your environment variables in the terminal. For example, you can get the value for CIVO_TOKEN by running echo $CIVO_TOKEN.

Now you should have four items under your repository secrets.

Next, you need to create some variables for your repository. On the same page, click the Variables tab and then click New repository variable.

Type in PULUMI_BACKEND_URL in the Name field. You need to type in the URL for your Pulumi backend for the value. You can get it by running:

pulumi whoami -v

The output will show you the backend URL with the prefix s3://. Copy the url and paste it into the Value field. Click Add variable.

Repeat the same steps for two more variables:

• AWS_DEFAULT_REGION, with value LON1
• AWS_HOST, with value https://objectstore.lon1.civo.com

You should have 3 items under Repository variables now.

Creating a GitHub Actions workflow for PR approval

Open pr-approved.yaml file in your code editor and paste the following code into it:

name: Pulumi up
on:
  pull_request_review:
    type: [submitted]
    branches: [main]

permissions:
  pull-requests: write

jobs:
  preview:
    if: github.event.review.state == 'APPROVED'
    name: Update
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: pulumi/actions@v5
        with:
        command: update
        stack-name: organization/my-infra/dev
        cloud-url: ${{ vars.PULUMI_BACKEND_URL }}
        comment-on-pr: true
        env:
        PULUMI_CONFIG_PASSPHRASE: ${{ secrets.PULUMI_CONFIG_PASSPHRASE }}
        CIVO_TOKEN: ${{ secrets.CIVO_TOKEN }}
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_DEFAULT_REGION: ${{ vars.AWS_DEFAULT_REGION }}
        AWS_HOST: ${{ vars.AWS_HOST }}

Go back to the terminal. Commit your changes and push it into GitHub:

git add .
git commit -m ‘Add workflow for pr approval’
git push

Testing GitHub Actions workflow for new PR

Invite a collaborator

To test the GitHub Actions workflow created previously, you need to invite another GitHub user as a collaborator in your GitHub repo. You will use this user to create a new branch and make some modifications in main.py file to create a Civo Compute instance. After you commit the changes, you will create a new pull request to start the workflow created earlier.

So first, using your main GitHub account, invite another GitHub user to be a collaborator in your repository. In my example, I will use a GitHub account with the username unyilops. Go to your repository's settings page, and on the left menu, click Collaborators.

Click the Add people button. A pop-up will show up. Type in the second GitHub account username that you will use as a collaborator. Click the username in the search result and then click the button to add the user to this repository.

Now, in your browser, sign out of the first account and log in using the second account. Go to the GitHub notifications page. You will find a notification for the invitation. Click on the notification.

Create a new branch

To create a new branch, first click the branch name and then click view all branches.

Click the New branch button on the branches page. A dialog will pop up. Type in your new branch name, for example add-new-server, and then click Create new branch. Refresh the page, and then you will see a new branch named add-new-server in the repo.

Create a pull request (PR)

Next, you will edit the __main__.py file to add a new Civo instance to your infrastructure. Click your new branch name in the branches page. Then click __main__.py

To edit the file, click the edit button at the top right of the page. Add this line at the end of the file:

web_server = civo.Instance("web-server", region="LON1", disk_image="ubuntu-jammy")

pulumi.export('server_id', web_server.id)

Click Commit changes. A dialog will pop up. Type in Add new server as the commit message and then click Commit changes to create a new commit.

Click the Code tab to go back to the repo’s homepage. You will see that now there is a small notification telling you that there is a recent push to add-new-server branch. Click Compare & create pull request.

In the Open a pull request form, click the gear icon for Reviewers section and then click the username for the repo owner. Then click the Create pull request button at the bottom of the page.

Click the Actions tab, and you will see that there is one workflow running. Click on the workflow if you want to see its progress.

When the workflow is finished, go back to your PR details page. You will see now that there is a new comment in the PR. The comment is from the GitHub Actions bot, showing the output of pulumi preview command.

Approve the new PR

The last step is to approve the PR. Log out from your collaborator account and log back in using the first account, the one you used to create the repo. Access your repo’s homepage, click on the Pull request tab, and click on the PR created by the collaborator account.

Click Add your review button. A page to review the PR will load. To approve the PR, click Review changes, select Approve, and click Submit review. You will be brought back to the PR details page, and you will now see one workflow queued. Click on details to open the workflow details. You will see the progress of your GitHub Actions workflow for PR approval.

Watch the page as the workflow is running. Once the Civo instance creation process is finished, you will see the output of the pulumi update command.

Go back to the PR details. There is now a new comment with pulumi update output as its content.

You can also check that a new Civo Compute instance has been created by accessing your Civo dashboard.

To finish the process, merge the pull request by clicking the Merge pull request button and then click Confirm merge.

Cleaning up

To clean up all the resources that you have just created, run the following commands:

pulumi stack export --file stack.json
pulumi login --local
pulumi stack import --file stack.json
pulumi destroy -y

You may notice that in the above commands, you switch back to using the local backend before running pulumi destroy command. This is because you can’t use the Civo Object Store bucket as the backend while destroying the bucket.

Summary

In this tutorial, you learned how to use Pulumi and Python to create resources in Civo. You also learned how to use a bucket in Civo Object Store as the backend for Pulumi. Lastly, by using GitHub Actions, you have learned how to create a development workflow with pull requests for managing your infrastructure code.

The pull request process that you learned in this tutorial is simple. You might need to add a few more GitHub Actions workflows in the real work environment. One that comes to mind is if the reviewer asks the code committer to edit their PR. In this case, there should be a workflow to run pulumi preview command anytime there is a code update in the PR.

You can check out the documentation for GitHub Actions to learn more about its features and how to customize your workflows to suit your development process. Also, don’t forget to check the documentation for the Pulumi package for Civo to learn how to manage, not just Civo Object Store and Civo Compute instance, but also other Civo products.