Welcome to Ji Cloud!

This book is for internal documentation. Use the table of contents on the left :)

Frontend Pipeline

Themes

Configuring existing themes

Themes are configured in frontend/config/themes.json

The various images are stored on the CDN in the ui/theme directory

Adding new themes

Requires a couple extra steps:

  1. ThemeId in Rust: in the shared crate
  2. The as_str_id definition in the Rust themes.rs

Usage in code

The JSON is checked via Serde on the Rust side at crates/utils/src/themes.rs It is also loosely checked on the element side via Typescript definitions at elements/src/_themes/themes.ts

These respectively do a bit more processing too (the element sets CSS vars and Rust sets up helpers to access data)

Note that modules are free to use the above settings however they wish, e.g. to use a specific theme mapping based on runtime conditions

Elements and Stories

Elements and Stories

Intro and reasoning

We tend to say everything is a component. Whether it's a single line of text with a specific font, or a full page consisting of hundreds of tiny configurable pieces, the concept is all about composing pieces together in order to create larger structures, much like LEGO.

However, once we move from a visual brief to the engineering side, we have to consider the implementation details of exactly how these things compose together and the limitations of our target platform. How do properties change? How do events propogate? How do children get swapped out?

By way of example, consider a single button with a label. Now let's imagine what happens if we have a menu containing a dozen of these buttons. If we want to change the text or color of precisely one of the menu's sub-buttons, how do we know which one to target? Furthermore, if a button is clicked, how do we know which one was clicked?

In the old days, we would do this imperatively, by getting access to an element that we manually added to the DOM (e.g. via jQuery). This isn't necessarily a bad thing, but it becomes hard to maintain at scale and the more modern trend in web development is to favor declarative frameworks like React, Vue, etc.

This declarative approach requires that we compose our components from the top down, e.g. a page contains sections, sections contain inputs, and so on.

Web components

Historically speaking, the web did not have a mechanism for defining components, and so these frameworks introduce their own special domain-specific language (JSX, Vue properties, Svelte, etc.). In some cases, like JSX, this language is very close to HTML, and in others it's radically different - but in all cases, the language is tied to the framework. You can't take JSX and stick it on a webpage - you need React to render it.

This changed with the advance of "Web Components". The full spec and history of that is outside the scope here, but ultimately, it allows defining custom html elements that are literally html elements. You can stick them on a webpage and it just works, with no need for another framework.

Lit-Element

That said, authoring web components directly is annoying and prone to performance problems. For that reason, it makes sense to author the web components in a framework that removes boilerplate and performs well at scale. Our choice is lit-element and its sister lit-html.

"Elements" not "Components"

Since everything is a component, and it's also a term used in other contexts (like in the design brief), for now on we will refer to "web components" as "custom elements" or just "elements" for short.

This is technically accurate enough - "custom elements" are part of the "web component" spec, and we are ultimately rendering these custom elements.

Storybook

Creating the elements is one thing, but we still need to see how they behave for QA purposes, and it's much faster to iterate on that before it gets bogged down with all the other requirements of the live app. In order to test with mock data, and prototype before signing off on the elements for further application development, we use Storybook as a visual testing environment.

The purpose of storybook is only for QA/testing. Content created in Storybook alone is never bundled into the final app.

"Stories" not "Components"

Again, it's not wrong to think of every story as a "component", and it even uses React under the hood - but to disambiguate, we will stick to calling them "stories".

How to Compose Elements

The Zen of Composition

Developing an intuition for how to structure elements is part art and part science. A general tip is to think through the big picture, consider how the element is used, and what its purpose is. Careful thought and planning can prevent much refactoring later.

Slots

Slots are how we compose elements together.

Let's pretend we have two elements already defined: my-menu and my-button

What we want is to have this simple html:

<my-menu>
  <my-button>Button 1</my-button>
  <my-button>Button 2</my-button>
  <my-button>Button 3</my-button>
</my-menu>

This works perfectly, with no changes to the above, if my-menu exposes an unnamed slot. For example, my-menu might render:

<div>
    <slot></slot>
</div>

But let's say my-menu wants to place those buttons in some specific nested area. That's easy enough, use a named slot:

<div>
    <h1>Menu Start<h1>
    <slot name="items"></slot>
    <h1>Menu End<h1>
</div>

And then, our top-level html is:

<my-menu>
  <my-button slot="items">Button 1</my-button>
  <my-button slot="items">Button 2</my-button>
  <my-button slot="items">Button 3</my-button>
</my-menu>

Avoid uber-elements, embrace purposeful elements

Lit-element is provides a lot of expressive power by using properties to render dynamic content. In fact, if one were so inclined, they could build an entire website in one ridiculous giant element.

Of course, this is a bad idea. The goal should be make elements purposeful as well as to provide a simple API so that code is clean and elements can be re-used. Purpose here not necessarily mean "achieve the design brief", though it can be just that. It can mean "create a grid structure with slots", "control all the colors and fonts of children", etc.

A page element specifying nothing other than a grid serves a clear purpose. So does a multimedia element or a controlbar. An uber-page element that does all of that together might not serve a clear purpose, would have a confusing API, and would be much better separated into pieces.

So how do we know when to split elements apart?

Split at nested functionality

A good rule of thumb is to think about whether or not you need to get at the children from the outside. For example, to set properties or respond to events.

Let's consider our above example of my-menu and my-button. Why did we split it, instead of just slogging those buttons in between Menu Start and Menu End?

In a real-world example, it's likely that Button 1 will have its text determined at runtime. So the alternative would mean my-menu would need to provide a mapping of a property for each button text (button1Text, button2Text, etc.). Ewww. Awful.

Also, in a real-world example, we'd want to do something when the button is clicked. So from the outside, we need to know which button is clicked. It's much cleaner to attach a listener to each button directly than to inspect the event target or have some hodgepodge of custom events.

By splitting the button into its own element, and then slotting it in, we have a much cleaner API overall.

Otherwise, don't split

If there is no need to get at the children, then from the outside it's just one big opaque element and should be created as such. Breaking things into a million unnecessary pieces creates confusion and complexity.

Consider an element with some visual decorators like lines and boxes. It should all just be self-contained in that element - splitting it out and then requiring it to be slotted back in is utterly pointless.

In general, when there's a very clear mapping of properties to the element contents, or the element dispatches a very clear set of events, there is no need to split.

How to Write Stories

More detail and code guidelines will be broken down later, but for now let's talk about the overall concept.

We use Storybook for two purposes:

  1. To preview and test elements.
  2. As a reference for app development.

That means the content in Storybook is temporary

K.I.S.S.

Some general tips:

  • Do not have static, real data in a story. It should always be in an element.
  • Do have dynamic, mock data in a story - to simulate what will happen at runtime.
  • Do use Controls and props to pass that mock data in.
  • Do keep the directory structure of stories mostly synced with elements

Use the platform

We only support modern, evergreen browsers, and in some specific cases don't even require supporting Safari or mobile.

Since custom elements are part of the web spec, this means the full power of the web api is available to us.

Here's a non-exuastive list of some possibilities

Direct element references

Let's say you are rendering a canvas element like <canvas id="canvas"></canvas> and you need access to it in order to create a context.

You can call this.shadowRoot.querySelector('#canvas') in the firstUpdated() lifecycle method and it will return the HTMLCanvasElement.

The concept is identical to calling document.querySelector() on a webpage after the html has rendered, with the difference being that here it's scoped to the custom element where you call it.

CSS Grid

Since we are using encapsulated CSS, it means that we avoid frameworks that rely on global class definitions (bootstrap, tailwind, etc.)

In the next page we'll talk about how we can tackle reusable styles in a number of ways, but for now - note that CSS grid is likely to be ubiquitous for creating container structures.

Named template areas with media queries, along with generic top-level structures with slots can effectively replace - and supercede - the grid capabilities of popular libraries, and it gives us far more control to create unique reusable patterns that fit our needs.

This article on MDN shows some examples: https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Grid_Layout/Realizing_common_layouts_using_CSS_Grid_Layout

Mutation/Resize/Intersection Observers

These are powerful mechanisms that allow reacting to changes to all sorts of things, including DOM layout itself. They can be used to dynamically generate slot names based on child box size, set CSS vars to be passed down to children, and more.

Code Style - Elements

Consult the library guides for general rules

This document is a supplement, not a replacement, for the basic usage guides:

Don't name the classes

Rather, use a _ as a placeholder:

@customElement('circle-button')
export class _ extends LitElement { 
    //...
}

The exception to this is when defining a base class for the sake of inheritance (discussed below in Global styles)

Be explicit with variants

Example - a property "color" should not be defined as a string, rather it should be defined as an enum or string variants:

export type Color = "red" | "blue" | "green";

@property()
color: Color = "red";

No null values

In the example above, it could have instead be written as:

@property()
color: "red" | "blue" | "green" | null = null;

This is bad. There should always be a sane non-null value.

Favor a declarative code style

Some tips:

  • There is almost never a need for a var or let, everything should be const.
  • Instead of switch+return, use conditionals (a.k.a. ternaries) or a predefined lookup
  • Create pure functions instead of class methods (i.e. in render(), call renderFooter(args) instead of this.renderFooter())
  • Split things out into small functions as needed

When conditionals get long, use the following style:

cond1 ? option1
 : cond2 ? option2
 : cond3 ? option3
 : default

Use nothing

When interpolating the template literals, it's often required to render "nothing". Instead of null or "", lit-html exports a special nothing object for this use case. Use that instead, since it will also prevent other bugs.

See lit-html docs for details.

Don't hardcode data in the render function

Static data should be moved out of the render function and defined as a const with an all-caps variable name.

For displayable strings, we will eventually have a more complex localization solution, but for now - use the STR_ prefix in order to facilitate string replacement / localization later

Example:


const STR_HOWDY = "hello world";

@customElement('my-element')
export class _ extends LitElement {
  render() {
    return html`<div>${STR_HOWDY}</div>`;
  }
}

Dynamic styles

lit-element and lit-html provide some helpers like classMap and styleMap to make some of the code around dynamic styling cleaner.

At the end of the day, you're just returning html. Both classes and inline styles can be changed at runtime based on properties.

Reusable styles

There are a few approaches to reusable static styles: css vars, inheritance, interpolation, and mixins.

Reusable styles - CSS Vars

CSS Vars pierce down through the shadow dom, and can be used by any nested child anywhere.

It's a great way to define global themes, or, more generally, for a child element to declare which of its styles can be overridden by an ancestor

Reusable styles - Inheritance

This approach is handy when there's a clear hierarchy of global styles for particular kinds of elements, but not others.

Example:

_in styles/text.ts

export class BaseText extends LitElement {
  static get styles() {
    //Note that it's an array even with only one entry
    //This is required to keep the consistent ...super.styles in the subclass
    return [css`
        .bold{
            font-weight: 600;
        }
    `]
  }
}

anywhere


import {BaseText} from "@elements/_styles/text";

@customElement('my-element')
export class _ extends BaseText {

    static get styles() {
        return [...super.styles, css`
            :host {
                width: 100px;
            }
        `]
    }
    render() {
        //...
    }
}

Reusable styles - Mixins

Mixins are more flexible than inheritance, but can also be harder to reason about and decide what should/should not be included.

Example:

_in styles/colors.ts


import { css} from 'lit-element';

export const colorTheme = css`
    .red { rgba(161,168,173,255) }
    .blue { rgba(85,144,252,255) }
`;

anywhere


import {colorTheme} from "@elements/_styles/colors";

@customElement('circle-button')
export class _ extends LitElement {

  static get styles() {
    return [colorTheme, css`
      .foo {
        border-style: solid;
        border-width: 1px;
        border-color: ${colorValues.grey}; 
      }
    `]
  }

  render() { 
      //... 
  }
}

Of course, they are just JS objects, so they can be grouped into their own arrays and mixed in like:

//cssThemes is an array of css objects
static get styles() {
    return [...cssThemes, css`
      .foo {
        border-style: solid;
        border-width: 1px;
        border-color: ${colorValues.grey}; 
      }
    `]
}

Reusable styles - Interpolation

The static CSS getter cannot be interpolated at runtime with dynamic values, but it can be interpolated with static css literals

Example:

_in styles/colors.ts


import { css} from 'lit-element';

export const colorValues = {
  grey: css`rgba(161,168,173,255)`,
  blue: css`rgba(85,144,252,255)`
}

anywhere


import {colorValues} from "@elements/_styles/colors";

@customElement('circle-button')
export class _ extends LitElement {

  static get styles() {
    return [css`
      .foo {
        border-style: solid;
        border-width: 1px;
        border-color: ${colorValues.grey}; 
      }
    `]
  }

  render() { 
      //... 
  }
}

Code style - Stories

Here we refer explicitly to the components created in Storybook for UI/UX prototyping.

VSCode helpers

There are a couple snippets you can add to your VSCode config to automate the boilerplate for new stories:

VSCode snippets gist

Import the element dependencies

It's a straight import of the code, not a module import (because it's executed right away and defines the custom element for usage by name).

Good:

import "@elements/buttons/my-button";

Bad:

import {MyButton} from "@elements/buttons/my-button";

Move displayable strings out

Until we have the string library functionality, use the STR_ prefix in order to facilitate string replacement / localization later (a similar technique is used in Elements).

Example (not including props for the sake of simplicity):


const STR_HOWDY = "hello world";

export const MyStory = () => `<div>${STR_HOWDY}</div>`

Controls

Use Controls (via the args property) to simulate data that changes at runtime - but it's only needed in the first test of an element.

For example, a button story should have a Control to see how that button behaves with all sorts of text.

Once that button is used in another story, then there is no need to add a Control button in this other story too.

Provide arguments

(note that the above VSCode snippet makes all the boilerplate for this much simpler)

  1. Args should always be well-typed and optional (e.g. foo(args?:Partial<MyArgs>))
  2. A hardcoded default should be used as a fallback if no args are provided
  3. To implement the fallback, destructure in the component
  4. Assign the default to the components args property (this makes it part of Storybook's Controls)
  5. Enumerations should be expressed as actual enums or unions (not free-for-all strings/numbers) - and should similarly have a control type of radio, dropdown, etc.
  6. Use argsToAttrs() to make life easier

Note that for the sake of jargon, "args" and "props" are used interchangeably, but we tend to use "args" on the outside since that fits with Storybook's lingo, and "props" on the inside since that fits with React/Component lingo.

Example:

import "@elements/my-button";

export default {
  title: 'Buttons',
}

interface ButtonArgs {
  text: string
}

const DEFAULT_ARGS:ButtonArgs = {
  text: "click me"
}

export const Button = (props?:Partial<Args>) => {
    props = props ? {...DEFAULT_ARGS, ...props} : DEFAULT_ARGS;

    return `<my-button ${argsToAttrs(props)} />`
    //same as return `<my-button text="${props.text}" />`
}

Button.args = DEFAULT_ARGS;

Destructing into separate objects is straightforward:

interface ButtonArgs {
  text: string,
  src: string,
}

const DEFAULT_ARGS:ButtonArgs = {
  text: "click me",
  src: "example.jpg",
}

export const Button = (props?:Partial<Args>) => {
    props = props ? {...DEFAULT_ARGS, ...props} : DEFAULT_ARGS;

    const {src, ...buttonProps} = props;

    return `
      <my-button ${argsToAttrs(buttonProps)}>
        <my-image src="${src}" />
      </my-button>
      `
}

Sometimes controls are abstract

One use case for stories/components is to show elements 1:1. Another is to show a larger composition, where the props need to be mapped.

Example:

import "@elements/pages/user-page";
import "@elements/buttons/my-button";

export default {
  title: 'Pages',
}

interface PageArgs {
  scenario: "login" | "register"
}

const DEFAULT_ARGS:PageArgs = {
  scenario: "login"
}

export const UserPage = (props?: PageArgs) => {
    const {scenario} = props || DEFAULT_ARGS;

    const color = scenario == "login" ? "red" : "blue";

    return `
        <user-page>
            <my-button color="${color}" />
        </user-page>
    `
}

UserPage.args = DEFAULT_ARGS;

Define the control type

By default, Storybook will try to guess the control type, but it defaults to a string most of the time.

Set it explicitly for more control. For example, this creates a radio selection:


//Continuing the previous example
UserPage.argTypes = {
  scenario: {
    control: {
      type: 'inline-radio',
      options: ["login", "registration", "profile"]
    }
  }
}

The current list of available controls and annotations are here: https://storybook.js.org/docs/react/essentials/controls#annotation

Images

There is never a need to use <img> directly. We have precisely two storages:

  • <img-ui>: corresponds to the Dropbox folder for static ui images
  • <img-ji>: corresponds to uploaded images on our server.

Consult the elements for the full list of available properties

Directory Structure

Top-level directories (both in Storybook and Elements):

  • core: Used everywhere. Examples are "buttons" and "images"
  • entry: Corresponds to the actual app entry point, follows the division of Rust code and backend routes.
  • modules: Corresponds to the various module entry points

Additionally, there are some optional patterns which may appear at any level:

  • _common: used in multiple places from this directory's siblings and deeper (but not parent)
  • pages: full pages.
  • buttons, sections, widgets, etc.: exactly as they sound, used in this directory and deeper (but not parent). The names here can also be specific for a unique one-off component/element.

When there is only one page in an entry, it should be under pages/landing.ts

Checklist

The list here is meant to be consulted before a PR is submitted/merged to sandbox:

Elements

  • Headspace: "raw materials to be used in building the app"

  • All the static data for the element should be hardcoded - not properties

  • Compose elements within elements - try to keep things DRY

  • Element name matches file name

  • No unused properties

  • All properties have the correct attribute conversion type

  • Use enums or unions instead of primitive types where appropriate

  • Sane defaults (don't rely on the Storybook mock to set a property)

  • Displayable strings should be outside of the render function (or at least outside the return html) and prefixed with STR_

  • Use lit-html's nothing instead of null or "" when inside an html template

  • Use the suggested conditional syntax when there are multiple conditions

  • Use top-level CSS Vars for colors

  • Use inheritance where appropriate (such as re-using styles)

Tip:

  • Plan the architecture carefully and consider how the element is intended to be used, not just as a wrapper for a bunch of HTML/CSS. Container elements with nothing other than slots and styles are absolutely fine, as are static elements that have no properties. Complex large elements that abstract over a ton of functionality are fine too if that's what's needed. For example, when separating this way, you may find that "my-custom-page" gets broken into a generic re-usable container which has nothing to do with that specific page, and only needs to be slotted with the custom content.

Stories

  • Headspace: "temp prototypes for QA, app reference, or Element tests"

  • Should mostly be about configuring and composing elements for interactivity. Move static data to the appropriate element.

  • If it corresponds to an element, names and directory structure should match

  • Storybook title nesting should match directory structure

  • Args/Props should be defined properly (tip: use the snippet)

  • Set the appropriate Control type (e.g. radios for unions/enums)

  • For now - prefer using HTML directly instead of importing and re-using component functions (makes it easier to copy/paste the HTML for now, due to a bug in the "view source" functionality in Storybook)

  • Displayable strings should be outside of the render function (or at least outside the return html) and prefixed with STR_

Tip:

  • Think of Storybook as like a visual approach to TDD. It's only a way to test and see the elements, and how they compose together - not where you build the elements.

Both

  • File is in the correct directory structure

  • Cleanup temp code. It's okay to commit it, but clean it up with another commit before making a PR

  • Rebase before making a PR

Project Management

  • Design and high-level tasks are tracked in Monday
  • Engineering tasks are tracked in Github
  • Slack is used for ephemeral communication and daily standups

Directory structure

  • _secret-keys: contains secret keys for local development (.gitignored)

  • backend: servers and backend utils deployed to Cloud Run, Cloud Functions, etc.

    • _core: common library used between backend apps
    • api: the main api server (Rust/actix-web/sqlx)
    • pages: the main html server (Rust/askama)
    • fastly-purge: cloud functions for purging the CDN on file change
    • script: tools needed for backend stuff

  • build-utils: internal tooling and utils for the project as a whole (e.g. connecting to db)

  • documentation: this book

  • frontend: projects that get compiled to Single Page Applications (Rust->Wasm)

    • apps/crates: the SPA Rust/dominator apps
      • entry: each entry point
      • utils: common utils for each app
      • components: reusable components between apps
    • build-utils: internal tooling and utils for frontend
    • config: configurating files for frontend
    • elements: lit-element custom elements ("web components")
    • ts-utils: typescript utils shared between frontend typescript
    • storybook: mock components for quick layout development and showcase

  • shared: code that gets shared between frontend and backend

  • config: global configuration

    • (js/rust): static settings
    • .env: local settings and secrets (not checked into repo)

Setup

There are many moving parts that need to be configured in order to make it all work together. The required setup is spread across local environment settings, remote secrets, proprietary vendor administration, local configuration, and more.

The subchapters here should point to everything needed to get the project up and running - even in a complete fork.

When there are implementation details (such as how to apply IAM permissions to google, or the permitted values in local files, or guidelines for strong secrets), please consult the additional documentation (e.g. google's documentation, comments in the settings files, or industry-wide best-practice guidelines)

Frontend one-time setup

Requirements

Setup

  1. Setup your Git credentials (probably easiest via Github Desktop)
  2. Fork the repo
  3. Clone it into your local folder (using Github Desktop or manually)
  4. Install frontend dependencies
    • storybook and elements projects require a Font Awesome auth token to install fonts:
      • export FONTAWESOME_NPM_AUTH_TOKEN=<REPLACE_ME>
    • npm install in the following folders:
      • frontend/storybook
      • frontend/elements
      • frontend/apps
      • frontend/build-utils
  5. Install Dropbox / accept the invitation to ji-cloud-media  (we might move that over to a separate repo at some point...)

After all this is setup, you should be able to npm start from frontend/storybook and see it working, just without the images.

(the rest of the setup is merely setting the .env values)

This is totally out of date...

cargo install:

  • systemfd
  • watchexec-cli
  • cargo-make
  • cargo-watch

openssl:

  1. visit https://slproweb.com/products/Win32OpenSSL.html (yes, the site says win32 but it has win64 msi)
  2. after installing, add C:\Program Files\OpenSSL-Win64\bin to path
  3. add C:\Program Files\OpenSSL-Win64 to OPENSSL_DIR env var

postgres (for the client library):

  1. install via the regular installer
  2. make sure that both of the postgres bin and lib dirs are on the PATH

If not using postgres for the server (e.g. to use docker instead), make sure to manually disable it in startup so that there's no conflict with the port. On windows this conflict may not even show up as an error!

Google Cloud SDK (https://cloud.google.com/sdk/docs/quickstarts)

Google SQL Proxy (https://cloud.google.com/sql/docs/mysql/sql-proxy) Make sure to put it somewhere in the path and name it cloud_sql_proxy(.exe)

Refer to the .env.sample

Google Cloud Setup

Secrets

The runtime secrets are accessed via Gooogle Secret Manager. Some of these are also in local .env

  • DB_PASS: the database password
  • INTER_SERVER: a random string to authenticate inter-server communication
  • JWT_SECRET: a random string used to sign JWT tokens
  • SANITY_TEST: not really necessary, just to make it easier to test that things are working correctly
  • ALGOLIA_KEY: algolia key
  • ALGOLIA_PROJECT_ID: algolia project id
  • GOOGLE_S3_ACCESS_KEY: the access key for accessing cloud storage like s3
  • GOOGLE_S3_ACCESS_SECRET: the access secret for accessing cloud storage like s3
  • SENTRY_DSN_API: Sentry API
  • SENTRY_DSN_PAGES: Sentry Pages

Backend - Cloud Run

  1. Create a service account with a new name and:
  • the following roles (if not done here, add the account to IAM and do later)
    • Cloud SQL Client (optional)
    • Compute Admin
    • Service Account Token Creator
    • Cloud Run Admin
    • Secret Manager Secret Accessor
    • EventArc Admin
  • the CI service account as a user access (if not here, can be given Service Account User in permissions later)
  1. Create an initial cloud run service and assign its service account to this new one
    • If not part of the initial flow, edit and deploy a new revision and change in Security tab
    • at this point the deploy will fail - CI will fix it later
  2. If Cloud SQL access is needed, assign it
  3. If the Cloud Run instance needs to create files for a storage bucket - assign it as an admin for that bucket (via the bucket page)
  4. Assign the custom domain if it's exposed to the outside world

Backend - Cloud Functions

Used to purge FastlyCDN when certain buckets change

The domain names need to be set in index.js

Backend - Compute Engine (for media sync)

Similar to Cloud Run, but the only access the service account needs besides compute engine is to the target media bucket

Database

Remember to enable Cloud SQL Admin API

Regions

Try to keep things in the same region, for example europe-west1

Buckets

Every bucket has:

  • allUsers w/ Object Viewer permissions
  • Fastly CDN proxy
  • The service account assigned as Storage Admin if it needs write access
  • If not using a CDN, set the default index.html and 404.html (setting it anyway, via gsutil if not using an explicit domain as bucket name, doesn't hurt)

See CI/CD for more detail

Secret Keys

./_secret-keys is a .gitignored folder containing the following, for local development only:

  • gcp-dev-release.json: google cloud credentials for release project
  • gcp-dev-sandbox.json: google cloud credentials for sandbox project

These should match the values in config/.env

These secret keys should not be confused with the Google Secrets Manager keys, which are for production use (both release and sandbox)

Google Cloud

  1. Create a service account for ci/cd (e.g. github-actions)
  2. Via the IAM->Service Accounts page, allow this service to operate as the service account for each cloud run instance
  3. In cloud storage, give this service account cloud storage admin to buckets that are deployed via ci/cd:
    • frontend (release and sandbox)
    • storybook (sandbox)
    • docs (sandbox)
    • artifacts.project.appspot.com (release and sandbox, created by google - needed for cloud run deployments)
  4. Also give the github actions service Cloud Run Admin and Cloud Functions Admin permissions
  5. Also, give the github actions service account user for the AppEngine service account (seems to be required for cloud funcs even w/ cloud func admin)

Generally speaking, the very first deployment (see below) on a brand new project should be done manually via a local account first, before using ci/cd going forward.

In particular, the first cloud function deployment requires hitting "yes" on "Allow unauthenticated invocations"

Github Secrets

  • SLACK_BOT_TOKEN (the one that begins "xoxb-")
  • GOOGLE_CLOUD_SERVICE_ACCOUNT_JSON_KEY - json key for service account
  • GOOGLE_CLOUD_SERVICE_ACCOUNT_JSON_KEY_SANDBOX - same but for dev deployment
  • FIREBASE_TOKEN (run firebase login:ci)

The GOOGLE_CLOUD keys must be base64 encoded. Literally, take the json string and run it through a bas64 encoder.

Makefiles and Dockerfiles

Deployment is done via the top-level Makefile.toml as well as Dockerfiles as needed.

The PROJECT_ID and other variables are hardcoded directly in these files as needed (even if that's the process of setting as an env var)

If adjusting, remember to change sandbox vs. release :)

Github Actions

A new github action needs to be created for each frontend wasm project. Simply copy/paste from one of the existing actions

Database setup

There are 3 different database targets: local, release, and sandbox.

Local means totally local to the developer machine, local password, etc. Release and sandbox are on google cloud and require access through google-cloud-proxy

The connection string needs to be set in different places depending on the project scope, which is covered in the rest of this Setup chapter

Note that "local" is up to the dev... could be in Docker, or native, it doesn't matter

Cloud Sql Proxy

Although the username, password, and database name are set in config/.env files, the database instance name needs to be passed as a commandline arg to cloud-sql-proxy.

Set this in build-utils/package.json. Note that the port and instance should match SQL_PROXY_PORT and DB_INSTANCE_* in the config too.

Local Docker

Although local can be anything (including native), a docker setup is provided in build-utils/db_local. Simply cargo make db-local

Make sure to not conflict with other ports of other docker instances, of course :)

Fastly CDN setup

Used for all static assets.

Dynamic backends are typically through cloud run and so cannot be efficiently cached right now, at least until Google supports straight external IP or a load balancer.

Make sure to set interconnect location that's closest to storage (e.g. Amsterdam Fastly for Belgium Google)

Only each origin should have host request condition, in order for that origin to be used for the domain. e.g. req.http.host == "docs.jigzi.org"

A small VCL Snippet for the recv block is required to make it fetch index.html for plain directory requests:

if (req.url ~ "\/$") {
  set req.url = req.url "index.html";
}

Though we're not using that at the moment (pushing to firebase for docs and storybook).

We are using this VCL in order to cache things for 1 year, and then we purge things more aggressively as needed:

set beresp.ttl = 1y;
if (beresp.status >= 200 && beresp.status < 300) {
  set beresp.cacheable = true;
}

See Fastly documentation for more details

Purging

Some buckets are purged automatically via a google cloud function (see google cloud) on every file change

Others are not and require manual purging, either because they are never expected to be purged (i.e. uploads) or it would depend on the exact route and very rarely need to be purged (i.e. page templates, if/when that is supported for CDN cache)

For buckets that are purged automatically, the file's cache-control headers are set to not cache, as per Fastly's recommendation. This setting is done immediately before the file is purged

Note that purging only happens when files are changed or added, not when deleted (handling deleting and/or archiving would require an additional cloud function for each origin, and missing files aren't a common use case for the end user. a manual purge, of course, can always be done if there's a real need to remove its existence from the edge cache)

Media VM

Media is synced via Dropbox, in order to allow graphics, audio, and other media artists to contribute their work without going through git.

In order to have media updates automatically propogate all the way up to the CDN, a VM with a cron task needs to be setup:

  1. Locally install rclone. Not on the server, but on a dev machine
  2. Use it to create the authenticated dropbox remote
  3. Same for cloud storage (though no need for authentication - that will happen via automatic credentials)
  4. Create the bucket like all the others (and setup w/ fastly, etc.)
  5. Create a small Compute Engine instance (i.e. e2 micro) and give its service account owner access to the bucket
  6. Install rclone on that instance
  7. Copy the local config file (which will contain the config keys) to the remote VM (run rclone config file on the vm to get path)
  8. Write a small shell script and chmod u+x it with the following, replacing the remote and bucket names as needed:
#!/bin/sh
rclone sync dropbox:/ji-cloud-media media-storage:/ji-cloud-media-origin-eu-001/
  1. Setup a cron task to run rclone and sync at some interval. For example, every 2 minutes:
*/2 * * * * /home/david/sync.sh

TODO (when Google Supports it)

Create a load balancer for each google run endpoint

Once that works, it's likely that some origins can be proxied by CDN too

CORS

For backend servers, it's configured as part of actix.

For static media (including wasm), there is a script in frontend/build-utils. All that's needed is to npm run each of cors:frontend, cors:media, and cors:uploads

To configure the origins, see the respective *-cors.json file in the build-utils folder.

Since it runs gsutil, you may need to be careful to run it in a compatible shell (like cmd, not powershell, in windows)

Firebase

The firebase side should be connected to the google cloud, i.e. both release and sandbox projects

Make sure to follow the firebase guidelines - e.g. to allow the domains for oauth

Auth config

Edit frontend/apps/crates/entry/user/js/firebase.js and put in the config for both sandbox and release

These aren't secrets, it's okay to be checked into the repo and it will be publically viewable through the browser anyway

Boilerplate

After initial setup, some actions require additional boilerplate changes as new content or features are added.

Frontend Setup

Each frontend module needs to set the NAME const in rollup.release.js and rollup.sandbox.js, in order to match the directory in its github action

All about engineering

Start projects in dev mode

Inside the frontend/apps folder:

cargo make [target] [app]

Where target is one of:

  • local-main
  • local-iframe (will use a different port)
  • local-main-nomedia (will not start the local media server)
  • local-iframe-nomedia (different port and will not start the local media server)

app is the name of the SPA (user, asset/edit, module/memory/edit, etc.)

Available apps and modules

There are a few apps setup for scratch and showcase that are good for like a whiteboard on the dev side

Storybook

Frontend wasm and storybook:

npm start

or

npm run start:nomedia (will not start the local media server)

Hosting and Deployment High Level View

Notes:

  • The release/sandbox split is consistent. A sandbox frontend will hit sandbox server which hits sandbox sql.
  • There are several static hosting endpoints. It's primarily split to keep frontend weight down.

Frontend Templates (TODO - DEPRECATE)

The template system is based on a two-step process:

  1. Robust pre-processing at build time
  2. Simple key replacement at runtime

Build Time

The pre-processing step uses a jinja2 like system, powered by Tera

Most of the capability should work out of the box.

At build time, the templates are collated from 3 locations and then output to one place (.template_output/ which is gitignored). This output folder is made available to storybook via the @templates import alias. Original directory structure from each location is preserved.

The location of source templates are:

  • frontend/core/templates - common reusable templates for use in multiple projects
  • frontend/[APP]/templates - the templates for the specific app
  • frontend/[APP]/storybook/demo-templates - templates that are only used for demo purposes in storybook (these won't be available to the APP itself at build time)

Due to an issue in Tera, the guideline is that _core and demo-templates should be nested in subdirectories prefixed with an underscore (to avoid namespace conflicts with eachother and the app templates). This is probably a good idea anyway for _core to keep things organized (e.g. in subdirectories like _buttons, _input, etc.). Since the Tera issue might one day be resolved, it's recommended to create a _demo subdirectory in demo-templates, and then reference it in the storybook code via @templates/_demo/*. That way, we can simply move the files without changing any of the storybook code if Tera adds the feature request.

In terms of Context, the only key passed down is MEDIA_UI (and this is changed via the build system to target local, sandbox, or release)

One thing which does not work is mixing macro definitions in the same file as a template. Rather, macros should be in their own file - and these will be filtered out when rendering.

Runtime

At runtime, the only thing which is supported is simple key/value replacement. This follows a different pattern than the one supported by Jinja/Tera: ${pattern}

The different runtimes do the actual replacement via:

The JS function is potentially insecure, but it's only used in Storybook, (i.e. dev mode only where we have strict control over the values, so it's fine)

When using in the deployed app, the simple-html-template crate provides a html_map! and html_map_strong! macro which not only make it more convenient to create the hashmap, but also escape according to standard security rules (extra protection when setting attributes)

Combo

The above system can be combined to allow passing dynamic values down to a template macro, allowing for a combination of static and dynamic replacements.

For example:

Given the following macro in _core/templates/_buttons/button-macros.html:

{% macro orange(label, dataId="") %}
<button data-id="{{dataId}}">
    {{label}}
</button>
{% endmacro orange %}

We can do the following, to either hardcode a value via the template system or allow a dynamic value to pass through:

{% import "_buttons/button-macros.html" as ButtonMacros %}
{{ ButtonMacros::orange(label="HardCoded!", dataId="btn-1") }}
{{ ButtonMacros::orange(label="${dynamicButtonLabel}", dataId="btn-2") }}

Common Documentation Terms

Key

Authentication and Authorization

Signin and Registration

Endpoint Authorization

Single Signin

Single Signin is accomplished by:

  1. Taking advantage of the pre-existing auth flow on the main site
  2. Implementing a signin flow on the secondary site
  3. Leveraging the fact that GET / read-only requests are secure despite not having the CSRF header check.

In more detail, the flow is like this (assuming user has already signed in on the main auth site):

  1. Client makes a GET request to the main auth server's endpoint
  2. Main Auth Server validates the cookie (and only the cookie)
  3. Main Auth Server generates and responds with a new Signin JWT (without a CSRF token)
  4. Client passes this JWT along to Secondary Auth Server's login endpoint
  5. Secondary Auth Server validates the JWT (via a validation request to Main Auth Server)
  6. Secondary Auth Server follows the same signin flow as the Main Auth Server

Essentially, what's happening is that the user is really signing into the secondary site just like the main one. However, instead of actually entering their credentials, their credentials are proven, re-used, and supplied via the main auth site.

Session Expirey

The cookie is set for some amount of time (currently preconfigured at 2 weeks)

It can be extended at any point - however, real extending with completely fresh data would mean resetting the CSRF token as well.

Therefore two services should be exposed:

  1. extend-auth-soft - will re-use the existing CSRF token and simply re-set the cookie with an updated max-age

  2. extend-auth-hard - will also change the CSRF token and the client is expected to replace it in local storage

The idea is that it's simple for a website to fire extend-auth-soft as some Future/Promise on startup and not need to worry about handling the response

Frontend Developer's Pipeline (TODO - UPDATE/DEPRECATE

STEP 1: HTML/CSS

  1. Design gets build in Storybook and local static templates with HTML and CSS (using Tailwind CSS)
  2. Pushing this will bundle/minify the CSS and make it publically available
  3. Templates can be composed to illustrate how they fit together in the live app, with different state and contexts.

STEP 2: RUST

  1. Register the templates
  2. Implement the same structure, with event handling, logic, state management, etc.

Backend Developer's Pipeline (TODO)

Common Developer Pipeline

Shared types

Edit the shared Rust types as needed. Request/Response are the primary things that are shared between frontend/backend

SEARCH

Will be handled by Algolia

TODO: describe how we populate the data on our side and then it gets synced either immediately or via scheduled tasks

Internal tooling and various utilities

Uploads are gated by the backend for auth and media transcoding

All about design

Wireframes

Designer's Reference

Themes, Responsive approach, working with XD, etc.

All about localization

We'll be using Mozilla's Project Fluent for the translation side of localization

Engineering

Requires that all strings everywhere be written with a function wrapper and some id + optional context

The translation bundles will need to be managed and delivered such that only what's needed is loaded

Translation

Should take advantage of the adaptable system to provide first-class experiences in the native language.

Design

We'll need to figure out how to best communicate this. Right to left and different text sizes will change some flow decisions. Some things will need real references, others can be done by explanation