CommCare Connect¶

CommCare Connect

Local setup¶

This assumes you will use the docker compose file in this repo to run services. If that is not the case you may need to edit some settings.

# install uv to manage python versions and dependencies
# https://docs.astral.sh/uv/getting-started/installation/
$ curl -LsSf https://astral.sh/uv/install.sh | sh

# create a venv and install requirements
$ uv sync

# This will also create a virtual environment in .venv if
# one does not already exist. To activate the environment:
source .venv/bin/activate

# For convenience, you can create an alias to activate
# virtual environments in ".venv" and "venv" directories.
# To do that, add the following to your .bashrc or .zshrc file (optional):
alias venv='if [[ -d .venv ]] ; then source .venv/bin/activate ; elif [[ -d venv ]] ; then source venv/bin/activate ; fi'

# Then you can activate virtual environments with
venv

# install git hooks
$ prek install
$ prek run -a

# create env file and edit the settings as needed (or export settings directly)
$ cp .env_template .env

# command for installing new python dependencies
$ uv add <pkg>

GeoDjango / PostGIS Setup¶

Connect uses GeoDjango and PostGIS. To learn more about the required dependencies, please refer to the official Django documentation: https://docs.djangoproject.com/en/6.0/ref/contrib/gis/install/geolibs/#geolibs-list

For macOS (Intel / Apple Silicon) – using Homebrew:

brew install gdal geos proj

Django may struggle to find the GDAL and GEOS libraries even after they are installed. Follow these steps to set your paths:

Get your local library paths

# Get GDAL path
ls $(brew --prefix gdal)/lib/libgdal.dylib

# Get GEOS path
ls $(brew --prefix geos)/lib/libgeos_c.dylib

Set it in your .env file

GDAL_LIBRARY_PATH=...
GEOS_LIBRARY_PATH=...

Install WeasyPrint on macOS (https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#macos)

brew install weasyprint

For Ubuntu / Debian Linux:

sudo apt-get install -y binutils libproj-dev gdal-bin

Enabling the PostGIS extension

No migration creates the postgis extension. The inv up setup works because the postgis/postgis image enables it for you when it initialises the database.

If you are using your own PostgreSQL instead, create the extension in the target database before migrating, or the first spatial migration will fail:

psql -d commcare_connect -c "CREATE EXTENSION IF NOT EXISTS postgis;"

# start docker services
$ inv up

# install JS dependencies
$ npm ci

# build JS (optionally watch files for changes and rebuild)
$ inv build-js [-w]

# run Django
$ ./manage.py migrate
$ ./manage.py runserver

Basic Commands¶

Some useful command are available via the tasks.py file:

$ inv -l

Setting up auth with CommCare HQ¶

Expose your local service to the internet

Create an account on ngrok and Install ngrok
https://dashboard.ngrok.com/get-started/setup/
Create a custom domain on ngrok using this link
Run ngrok http --url=[my-unique-subdomain].ngrok-free.app 8000
Update your .env file with the host:

DJANGO_ALLOWED_HOSTS=[my-unique-subdomain].ngrok-free.app

Create an OAuth2 application on CommCare HQ

Navigate to https://staging.commcarehq.org/oauth/applications/
Create a new application with the following settings:
Client Type: Confidential
Authorization Grant Type: Authorization Code
Redirect URIs: https://[my-unique-subdomain].ngrok-free.app/accounts/commcarehq/login/callback/
Pkce required: False
Copy the Client ID and Client Secret
Create a new SocialApp locally at http://localhost:8000/admin/socialaccount/socialapp/

Test the OAuth2 flow

Confirm COMMCARE_HQ_URL points at https://staging.commcarehq.org. That is the default, so you only need to set it in .env if you have overridden it.
Navigate to http://[my-unique-subdomain].ngrok-free.app/accounts/login/
Click the "Log in with CommCare HQ" button
You should be redirected to CommCare HQ to log in
After logging in, you should be redirected back to the app and logged in

Setting Up Your Users¶

No social app setup is needed: the "Log in with CommCare HQ" button only appears once a commcarehq social app exists (see Setting up auth with CommCare HQ).

To create a superuser, run the command below. --email is not accepted, because the custom User model sets REQUIRED_FIELDS = []; set the address afterwards in the admin if you need one.

$ ./manage.py createsuperuser
To create a normal user account, go to Sign Up and fill out the form. Local settings use ACCOUNT_EMAIL_VERIFICATION = "optional", so you are logged straight in; the confirmation email is still printed to the console if you want to verify the address.
To promote an existing user to superuser, use this command:

$ ./manage.py promote_user_to_superuser

For convenience, you can keep your normal user logged in on Chrome and your superuser logged in on Firefox (or similar), so that you can see how the site behaves for both kinds of users.

Sample Data¶

To populate a local database with organizations, opportunities, visits and payments:

# generate_sample_data <num_visits> <org_slug>
$ ./manage.py generate_sample_data 50 demo-org

The organization is created if the slug does not already exist. Each run first deletes the existing opportunities, visits, payments, programs and attachments belonging to the two organizations it is about to use, including the attachment files under MEDIA_ROOT. Optional flags:

--invited_org_slug — reuse a specific organization as the invited org. Without it, a new invited org with a random slug is created on every run, and the previous one is left behind.
--managed_opportunities — how many managed opportunities to create (default 3)
--image_percent — percentage of visits that get a photo (default 0, none)
--audio_percent — percentage of visits that get one to three audio recordings, each with a stand-in transcript and translation (default 0, none)
--media_dir — a directory to search (recursively) for image / audio files to use as the attachments for generated visits if requested. Otherwise placeholders are used.

Adding yourself to the generated organizations is a separate step, either through /admin/organization/userorganizationmembership/ or the organization's own member admin pages. Note there are two: the primary org from <org_slug> and the invited org, which owns the managed opportunities — join only the first and those stay hidden. Worker tabs list only workers whose OpportunityAccess is accepted, which the generated ones are.

Microplanning, without CommCare HQ¶

To seed an opportunity with work areas, groups, workers and visits for the Progress Map:

$ ./manage.py generate_microplanning_data --fake-hq

It prints the Progress Map URL and creates the superuser mpadmin with password mpadmin. --help lists the other options.

Assigning, unassigning, excluding and editing an assigned work area all update cases on CommCare HQ. --fake-hq sends those calls to a stand-in for HQ on localhost:8001 and gives each worker an HQ user there. Run the stand-in in its own terminal:

$ ./manage.py run_fake_hq

It answers every case update and prints the request with its body, so you can see what Connect sends. Start it with --fail to answer case updates with an error instead. To see the push notification that assignment sends to ConnectID too, start the Celery worker with CONNECTID_URL=http://localhost:8001/connectid.

Test coverage¶

To run the tests, check your test coverage, and generate an HTML coverage report:

$ coverage run -m pytest
$ coverage html
$ open htmlcov/index.html

Running tests with pytest¶

$ pytest

Live reloading and Sass CSS compilation¶

$ inv build-js -w

Celery¶

This app comes with Celery.

To run a celery worker:

celery -A config.celery_app worker -l info

Please note: For Celery's import magic to work, it is important where the celery commands are run. If you are in the same folder with manage.py, you should be right.

To run periodic tasks, you'll need to start the celery beat scheduler service. You can start it as a standalone process:

celery -A config.celery_app beat

or you can embed the beat service inside a worker with the -B option (not recommended for production use):

celery -A config.celery_app worker -B -l info

Deployment¶

The application runs as Docker containers on EC2, deployed with Kamal. Ansible provisions the instances and manages the container env files. See deploy/README.md for the full picture, including the tooling you need installed to deploy or change Django settings.

Deploying a new version of the app can be done via the "Deploy" workflow on GitHub Actions.

Container logs are shipped to CloudWatch, and can also be read directly with cd deploy && kamal app logs.

For details on how this action is configured see:

Deploying to the staging environment¶

The project has a staging environment at https://connect-staging.dimagi.com/, which is connected to the staging environment of CommCare HQ at https://staging.commcarehq.org/.

Update commcare-connect-staging.yml with the branches you need to include.
Run deploy/rebuildstaging to build the autostaging branch
After this, you can deploy to the staging environment by manually running the deploy workflow from here.

Remote Access to machines¶

Machines can be accessed via SSH using AWS SSM. In order for that to work, you will need to add the following lines to your ssh config (typically located at ~/.ssh/config)

Host i-*
     ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' --region us-east-1 --profile commcare-connect"

This remote access is required for the ansible commands in the tasks.py file.

Setting up logical replication¶

Create another postgres database (in a seperate cluster)
Add SECONDARY_DATABASE_URL to the .env file

Enable replication on the default database (requires restart)

      - "wal_level=logical"
      - "max_replication_slots=5"
      - "max_wal_senders=5"

Create table schemas in the secondary database
```
./manage.py migrate_multi
```
Initialize replication
```
./manage.py setup_logical_replication
```