- Published on
Deploy BlueSKy locally and in the cloud
Web Deployment and Building
To carry out the implementation of the BlueSky project, it is necessary to clone both the app-social repository, which covers the user interaction interface, and atproto, which is responsible for managing the underlying logic of the application.
The application build process is not as straightforward as suggested by the project documentation. This process requires the prior installation of certain programs and tools, as well as the user having basic programming knowledge to carry out the proper configuration and deployment.
Necessary Programs:
- Git
- Nvm
- Jq
- Golang
- Docker
Dependencies in Linux
sudo apt install git jq golang docker
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash
sudo systemctl enable docker
sudo systemctl start docker
sudo usermod -aG docker $USER
reboot
Dependencies in MacOS
brew install git jq
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
To proceed with the installation of Golang, the package needs to be obtained from the official Go website and the provided instructions should be followed. Similarly, this process applies to the installation of Docker.
atproto Project
First, clone the atproto repository.
We start by downloading the pnpm and yarn packages specific to nodeJS, which will help manage the project's dependencies.
Subsequently, by using the make command, we can see which commands it allows to execute to build the application and which will help automate the build process.
Allowed Parameters:
- help: Shows information about all commands.
- nvm-setup: Use NVM to install and activate node and pnpm.
- deps: Installs dependent libraries using
pnpm install. - build: Compile all modules.
- run-dev-env: Run a development environment shell.
npm install --global pnpm
npm install --global yarn
cd atproto
make nvm-setup
make deps
make build
make run-dev-env
If everything goes well, the URL of the services is expected to be displayed without problems.
| Port | Service |
|---|---|
| localhost:2582 | DID Placeholder |
| localhost:2583 | Personal Data |
| localhost:34593 | Ozone server |
| localhost:2584 | Bsky Appview |
| localhost:35683 | Feed Generator |
| localhost:40975 | Feed Generator |
social-app Project
Start by cloning the social-app repository. Then, proceed to execute the following commands to create the bskyweb binary and launch the server.
cd social-app
yarn && yarn build-web
cd bskyweb
go mod tidy
go build -v -tags timetzdata -o bskyweb ./cmd/bskyweb
./bskyweb serve
Service Connection
To connect the frontend part of BlueSky, you must select the 'Choose Service' option to enter the address of the 'Personal Data', whose local domain is http://localhost:2583.
To verify the changes, you can use the default user data that is created. This can be found in the bluesky-social/atproto/packages/dev-env/src/mock/index.ts file.
...
const users: User[] = [
{
email: 'alice@test.com',
did: '',
handle: `alice.test`,
password: 'hunter2',
agent: clients.alice,
},
{
email: 'bob@test.com',
did: '',
handle: `bob.test`,
password: 'hunter2',
agent: clients.bob,
},
{
email: 'carla@test.com',
did: '',
handle: `carla.test`,
password: 'hunter2',
agent: clients.carla,
},
]
...
If everything goes as expected, we should be able to access the account and verify that all the configuration has been done correctly.
Additionally, it is possible to access the PostgreSQL database using the 'dbeaver' program and view both the created tables and user accounts.
Possible Errors
NVM Not Found
The issue is that NVM is not available in the context of the make command because NVM is a shell script and not a globally installed application. This means that NVM must be loaded in every new shell instance that is started.
Here is the error result when running make nvm-setup:
> make nvm-setup
nvm install 18
/bin/bash: line 1: nvm: command not found
make: *** [Makefile:52: nvm-setup] Error 127
To resolve this error, you need to edit the Makefile and modify it as shown in the following code snippet.
> git diff Makefile
diff --git a/Makefile b/Makefile
index 2ec47b289..b9715896d 100644
--- a/Makefile
+++ b/Makefile
@@ -49,6 +49,7 @@ deps: ## Installs dependent libs using 'pnpm install'
.PHONY: nvm-setup
nvm-setup: ## Use NVM to install and activate node+pnpm
- nvm install 18
- nvm use 18
+ . $$NVM_DIR/nvm.sh; \
+ nvm install 18; \
+ nvm use 18; \
npm install --global pnpm
Docker Compose Not Found
This error occurs because the project uses version 25.X or higher, causing the following error message to appear:
> pnpm run start
@atproto/dev-env@0.3.14 start
/home/noroot/app/atproto/packages/dev-env
../dev-infra/with-test-redis-and-db.sh node dist/bin.js
unknown flag: --file
See 'docker --help'.
Usage: docker [OPTIONS] COMMAND
...
ELIFECYCLE Command failed with exit code 125.
To resolve this issue, it is necessary to update to the latest version. Alternatively, you can modify the docker-compose.yaml file, replacing docker compose with docker-compose.
Additionally, you will need to manually create a database in PostgreSQL using the following command:
docker run --name bluesky -e POSTGRES_PASSWORD=password -e POSTGRES_USER=pg -e POSTGRES_DB=postgres -d -p 5433:5432 postgres
# Stop the container
docker stop bluesky
# Start the container
docker start bluesky
Building as a Mobile Application
The documentation related to building the mobile application from the GitHub repository focuses on the OTA (Over-the-Air Deployment) method. This method is commonly used to distribute application updates without the user needing to manually download the new version from an app store.
However, it is not possible to follow the mentioned procedures because only developers with access to the project can execute the established workflow. Therefore, the manual procedure will be explained below.
Initial Configuration
As a first step, create a new branch from an established tag of the project, this is done to avoid possible errors that may occur when a developer makes a commit. Also, create a file called .env.test based on the .env.example file.
> git tag
1.2
1.72.0
1.73.0
1.74.0
1.75.0
1.76.0
1.77.0
1.78.0
1.79
1.80.0
1.81.0
1.82.0
> git checkout -b 1.0.0-ota-1 1.82.0
> mv .env.example .env.test
Next, create an account on expo and create a project. The following image shows the project named 'temp' and the variables that are created and needed for the build part.
These variables should be modified in the app.config.js file located in the root of the project as shown in the following image.
Building
First, install the EAS CLI tool globally using npm install -g eas-cli. Then, log in to the EAS account using eas login.
To verify that you are logged in correctly, use the eas whoami command. Next, build the application for the Android platform using eas build --platform android.
Finally, execute eas build to start the full build process. These steps ensure that the application is properly compiled and ready for subsequent distribution.
> npm install -
g eas-cli
> eas login
> eas whoami
> eas build --platform android
> eas build
Performance Tests
Maestro executes tests and allows the creation of custom tests according to the user's preferences.
Flashlight measures application performance throughout the execution of tests, enabling the detection of any potential performance drop caused by the code being tested.
sudo apt install scrcpy
sudo apt-get install android-tools-adb
curl https://get.flashlight.dev | bash
curl -Ls "https://get.maestro.mobile.dev" | bash
For testing, it is necessary to have an Android device connected. You can verify the device connection by executing the adb devices command.
Thus, execute these commands to visualize in the browser.
scrcpy
maestro studio
flashlight measure
BlueSky deplouy AWS
Project Overview
This document details the necessary procedures to implement the BlueSky social network on AWS, utilizing IAM, EC2, and ECR services, and automating the workflow via GitHub Actions. Additionally, it presents a method to automate this deployment without the need for direct interaction with the AWS web interface.
Figure 1 illustrates the workflow for continuous deployment on AWS, where GitHub Actions is used to automate code deployment to ECR. However, IAM and EC2 must be configured manually.
Testing with GitHub Actions
There are two workflows for the testing process, as shown in Figure 2.
Golang Test
This GitHub Actions workflow, defined in the file golang-test-lint.yaml, is triggered by pull request and push events to the main branch and runs concurrently to perform build, test, and linting tasks in an Ubuntu environment. It consists of two jobs: build-and-test and lint.
In build-and-test, code verification is performed with make check, the binary is compiled with make build, and tests are executed with make test after configuring Go and simulating the presence of static files.
In the lint job, after configuring Go and simulating static files, make lint is executed to analyze the code for style and quality issues.
Lint
This GitHub Actions workflow, called lint.yml, runs on every pull request and push to the main branch. It has two jobs: linting and testing. In the linting job, the code is checked out from the repository, dependencies are installed with Yarn, and various checks are executed: linters, Prettier, internationalization compilation, and type checking. In the testing job, the code is also checked out and dependencies are installed, compiled, and then tests are run with the Node environment already configured.
AWS Configuration
Creating an IAM Account
To configure the cloud environment, having an AWS account is essential. Within AWS, the IAM (Identity and Access Management) service allows for the creation of users with the necessary permissions. It is crucial to create a specific user and assign the appropriate permissions. Additionally, an access key must be generated to authenticate the user via AWS CLI and Terraform.
As shown in Figure 3, full access to AWS services has been granted to avoid any authorization issues.
ECR Configuration
Amazon ECR (Elastic Container Registry) is an AWS service for storing, managing, and deploying Docker containers.
In this section, a private repository named bskyweb will be created. It is crucial that the repository is private and has exactly this name, as any discrepancy will cause errors in the GitHub configured workflow, as illustrated in Figure 4.
From this point, the workflow named build-and-push-bskyweb-aws will run automatically whenever a push is made to the main branch. It verifies credentials and the repository; if valid, it performs a series of steps in Ubuntu: code checkout, Docker configuration, registry login, metadata extraction, outputs configuration, and finally, Docker image build and push. If the repository is invalid, the process fails and shows an error. The workflow ends with the successful deployment of the image or a workflow failure. The description of this workflow is illustrated in Figure 5.
Amazon EC2 Configuration
Amazon EC2 is an AWS service that provides scalable computing capacity in the cloud, allowing for the creation and management of virtual servers. Two instances will be created: one for atproto and another for social-app.
In the atproto instance, the following build commands should be executed:
sudo apt install git jq golang docker
NVM=https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh
curl -o- $NVM | bash
sudo systemctl enable docker
sudo systemctl start docker
git clone https://github.com/bluesky-social/atproto
cd atproto
npm install --global pnpm
make nvm-setup
make deps
make build
make run-dev-env
In the social-app instance, the following commands should be executed:
sudo apt install -y docker-ce
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER
sudo docker pull public.ecr.aws/t8b2r8w9/social-app:latest
sudo docker run -d --name social-app -p 80:8100 public.ecr.aws/t8b2r8w9/social-app:latest /bin/sh -c "/usr/bin/bskyweb serve"
sudo iptables -A PREROUTING -t nat -i enX0 -p tcp --dport 80 -j REDIRECT --to-port 8100
Application Deployment
Figure 7 shows the architecture of the BlueSky application hosted on AWS. Users access the 'social-app' through port 80. Within AWS, 'social-app' communicates with 'at-proto' on port 2583, which in turn connects to an external service called Render on port 5432. Render hosts a PostgreSQL database. The 'at-proto' component also uses port 2581 internally. This structure allows the separation of the user interface, business logic, and data storage into different interconnected services.
Infrastructure as Code
Another automated way to perform the deployment is by applying Infrastructure as Code using Terraform.
This will allow the creation of the entire infrastructure without the need to manually create EC2 instances, security groups, or open ports.
To carry out this procedure, a new repository dedicated to Terraform has been established. This repository is structured into two separate folders: one for social-app and another for atproto.
It is essential to have Terraform installed on your local machine, adjust the AWS user variables, and run the commands detailed below:
terraform init
terraform plan
terraform apply
These commands will start the process of building all the necessary infrastructure without the need to access the AWS web interface.
References
- Bluesky Social. (n.d.). Bluesky Social App [GitHub repository]. GitHub. Retrieved June 5, 2024, from https://github.com/bluesky-social/social-app
- Docker. (n.d.). Compose CLI reference. Docker Documentation. Retrieved June 5, 2024, from https://docs.docker.com/reference/cli/docker/compose
- Bluesky. (n.d.). AT Protocol. GitHub. Retrieved June 5, 2024, from https://github.com/bluesky-social/atproto
- Expo. (n.d.). Expo EAS Documentation. Retrieved June 5, 2024, from https://docs.expo.dev/eas