The Ampersand compiler is a Haskell program built with stack. Stack is a build tool for Haskell projects such as Ampersand. We have automated the building process (using stack) for the following purposes:

1. to prevent mistakes such as dependency conflicts inside and between Haskell packages, for an uninterrupted compilation process (robust building);

2. to generate ampersand compilers for different platforms (platform independence);

3. to provide a reproducible and reliable build process to developers with diverse development tools, operating systems, and working environments (uniform building);

4. to allow for generating images for docker containers (containerization);

5. to accellerate the build process to increase the release frequency of Ampersand.

Installation

comes as part of , so there is no need to install Haskell separately.

The are pretty clear for the various platforms. Make sure you read the part about the STACK_ROOT environment variable.

To compile Ampersand you need a file , which sits in the Ampersand repository. Fetch it and put it in you working directory. From the command-line, call command stack install and after a while (go get coffee!) your ampersand compiler exists! NB: If you want to build Rieks' preprocessor as well, the magic spell is stack install --flag ampersand:buildAll

Getting started with GitBook

If you want to add to the documentation, it is possible to update the repository with any tool you like. However, we strongly advise to use the GitBook editor. A lot of cumbersome work is automated for you. Here are the steps to get on your way:

• Create an account at Gitbook.

• Once you have an account get in touch with so he can grant you access to the book(s) you want to contribute to.

• You will automatically find the books at your , where you will find a button to start editing the book.

The GitBook Web Editor

The editor explains itself. At GitBook, they eat their own dogfood to write .

The GitBook desktop client

Recently, Gitbook also has a desktop client. It can be found When you install this desktop client, GitBook will make a local clone of the book. You will find it at ~home/Gitbook. As far as I know, this is not configurable, so you'll just have to deal with it. You can synchronize from within gitbook. However, if it gets a little bit complicated, I prefer sourcetree to give the correct git commands.

We use Gitbook for the following purposes:

1. to maintain documentation that changes continuously as a team, without having too much work on the coordination of this collaboration;

2. to get good looking documentation with little effort;

3. to maintain the documentation on the internet, so we can hyperlink to it from anywhere.

The documentation you are reading at this very moment is created using GitBook. It allows to write the documentation in , which is an easy to use document-format.

The documentation can be changed by the . For helping to document Ampersand, please request an authorization for Gitbook with .

The documentation is organized as a book. We prefer this over a traditional wiki, because a book contains a content part, which helps both author as well as the reader to think about structure, so everything can easily be found. Each book can be served as a (part of a) website, as well be downloaded in several forms, like pdf or e-book. We use a fully automated toolchain to build the book for each commit to the master branch of a book.

At AmpersandTarski, we currently have two repositories, each dedicated to the documentation for a specific audience:

1. The repository contains the contents of the book you are currently reading. You probably found it . This document is not about Ampersand, but about the way Ampersand is being developed

2. The repository is going to contain the documentation about Ampersand itself.

Agreements about the development of code

Working with feature branches

All changes that are made to the code is done in feature branches. Such a branch should have a specific goal. Normally, this is documented in a single github issue. While the work can take some time, eventually a pull request is made in order to merge it into main. There are several requirements that must be met before a pull request can be merged.

Releasenotes

Every feature must be briefly documented in the releasenotes. A github action is in place that will notify if the releasenotes are not changed.

Code formatting

All code must be formatted in a consistent way. We use , a standard code formatting tool for Haskell. We only accept Haskell files that are properly formatted. This is enforced by a new .

In order to use ormolu, you have to make sure it is available. Hopefully, some time in future, this will be installed automatically. Until then, you have to install it yourself:

Devcontainer

We provide a standard developer container to the developers of Ampersand. Documentation about this awsome VScode feature can be found .

How to get started using the .devcontainer stuff

Setting up a Haskell environment with awsome tooling has never been as easy as today.

• Make sure you have vscode installed.

• Install the `Remote - Containers` extention.

• Make sure Docker is running.

• Push the button `Reopen in Container` and watch the show.

• While you are waiting, you can watch the progress by inspecting the log. There will be a message containing a link to the log file.

• After everything is set up, open Main.hs . This will trigger the Haskell extention of vscode. Also here, the first time will take a while, because the project is being build. That includes all dependencies of Ampersand. If you want to see what is going on, go to the Output tab and open the dropdown called `tasks`. You will find the task building Ampersand:

Specific tools used in the Ampersand project

Generic tools used in the Ampersand project

Branching strategy

• Use your own feature branch to experiment. You will typically spawn it from the main branch. Commit your branch frequently, so other team members can see what you are doing and help out if needed.

Querying packages using Github's GraphQL API

• The query below returns a list of all packages and package versions pushed to github repo of Ampersand

• The packages api is still in development, therefore you must include a HTTP Accept header to indicate you want to use this feature

• Make sure the right access right are set for your personal access token. Inluding read repo + read/write packages

  POST https://api.github.com/graphql

headers: >

body (grapql query):

Deleting specific packages

NOTE! Doesn't work with public packages, like we have

POST https://api.github.com/graphql

headers: >

body (graphql query) >

Backend implementation (PHP)

Structure of project folder

If you want to deploy RAP4 to your laptop, this chapter tells you how.

This example has worked on my Macbook. If you have a different computer, you may either change the commands below to the commands of your own computer or set up a virtual Linux machine on your laptop.

Prerequisites

You need docker. Follow the instructions onhttps://docs.docker.com/.

If you want to deploy RAP4 on your own server, you might try to copy how I deployed RAP4 at the OUNL. This chapter tells you how.

In early 2016 the need arose to replace the RAP2 implementation of Ampersand by a RAP3 implementation, because RAP2 was insufficiently maintainable. This environment is used by students for completing the course Rule Based Design (OBR, code IM0103). This implementation is hosted by ICTS, the IT-department of the university.

This chapter is an account of the installation process. It serves the following purposes:

1. It is an example for others who want to deploy Ampersand. We get requests now and then by people who want to deploy Ampersand, so we figured it is nice to have a documented example for them.

2. It documents the installation we made for the Open University. We want maintenance of RAP4 to be transferable to other persons, so we need to document the choices made and the reasons for making them.

3. It contains all information needed to make a deployment script for automated deployment. We have automated the deployment with Docker, so that RAP4 will always be up to date with the most recent stable release of Ampersand.

Each step in the installation process gets a separate section in this text.

1. Setting up the virtual machine

I got a server from the Open University's IT-department.

The following settings apply:

Getting access to the server

At the OUNL, we need VPN to gain access with SSH to a server. This requires approval from the IT department. I got a raw Ubuntu machine, meaning that the port settings (specified above) and VPN have to be requested at the IT-servicedesk.

I can now access this machine through SSH (using PUTTY, which I downloaded from the Internet), but only after installing a VPN-tunnel to the server (using Pulse Secure). In the sequel, I will refer to this machine as "the server". This gave me access through a command line interface (CLI). Ubuntu gave me bash as its CLI.

Installing Docker

Since this is a fresh machine, docker has to be installed. By just typing docker, the server advised to install Docker by means of the command sudo apt install docker.io. This turned out to be a bad advice, because it resulted in a too old version of docker. Instead, I followed the instructions on https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/ for installing docker. The instructions for docker-compose are found on https://docs.docker.com/compose/install/ .

Then I checked that everything went successfully by means of the which-command:

Obtaining the files we need

We need only one file: docker-compose.yml To get it, I used the wget command, which gets stuff from the web:

Installing RAP4

To install RAP4:

To check whether this worked, I went to my browser and navigated to http://145.20.188.96/RAP4. It took a while to get started, because it was building a fresh database.

I checked whether the containers are running by means of the docker ps command.

Completion of this step allowed access to RAP4 from an arbitrary computer on the internet:

The database is accessible on port 8080:

We use Git for the following purposes:

1. to have a shared repository that contains our source code;

2. to work on different features in parallel without interference and without having to wait for each other;

These pages are meant to describe the functionality of RAP4. As RAP4 is still under development, some of this functionality is still work in progress. We will tell the current status of everything mentioned here. While doing so, this document will grow until it describes everything one can do with RAP4.

The purpose of RAP4 is to be a platform for people to learn how to work with Ampersand. We distinguage the following ROLEs:

• Student.

• Tutor. This role is to watch the progress of students, and being able to give guidance.

• AccountMgr. This role is to create accounts and to grant access rights to users.

• GradStudent. This role is for more advanced students.

All of the above users have access to the login page. Further functionality is based on specific roles.

Logging in

When you log in, a userid and password must be entered.

Logging out

After you have logged in, this screen turns into a logout-screen. The menu bar gives you the options you are entitled to by the role(s) you have. In this case you get MyScripts and My Account:

In the next pages, the functionality is described by role.

The magic

To get a docker container I took the file Dockerfile and built an image called "myampersand", using the following command:

It took 45 minutes to build, but I got an image called myampersand in the docker repository on my laptop.

To see it work, I executed the newly created image on a little file in my working directory, hello.adl from the command line with the following command:

The same command in the Windows command line is:

Do you want to reproduce this?

The Ampersand repository contains a file, , that contains a recipe for building an Ampersand compiler and put it in your Ampersand repository. You need the following ingredients to run it:

1. A machine to run docker, for building your docker image with. I ran it on my MacBook.

2. Docker, which you need .

3. Docker needs least 5G of memory to build Ampersand, which is more than the standard 2G. I used to increase Docker's memory on my Macbook.

To run it, I cloned the Ampersand repository, into ~/Ampersand/, and built the image with:

It runs on my Mac for over half an hour, so some patience is required. If you don't have that patience, consider using the image from docker hub. It was built for your convenience.

The resulting docker image sits in the docker repository on you laptop (placed there when docker was installed).

So which steps does Docker take?

If you want a slightly different image (for reasons of your own), you may want to repeat this process yourself. For that purpose, let us walk through the different steps described in Dockerfile.

Let us discuss the steps one-by-one. (Please check the , just in case it is inconsistent with this documentation.) All of these steps happen automatically, as they are in the docker file.

The first statement states that the compiler is built on a well-defined Haskell image.

This building stage is called buildstage because we want to use a to obtain a small Ampersand image without excess-software.

We decide to work in the build-container from a working directory called /Ampersand.

Normally we want to generate Ampersand from the source code on GitHub. For this purpose we clone the Ampersand-repository into the (working directory in the) build environment.

In this case I wanted to build from a specific feature, so I checked it out.

Now everything is in place to compile Ampersand. Running stack install results in a full-fledged Ampersand compiler in /root/.local/bin. Mind you, this takes a while...

If we were to stop here, we get an image larger than 4GB. We can do better than that by starting over with a clean machine. So we introduce a second FROM-line in the Dockerfile which starts with a clean slate. We use an empty ubuntu machine (form some reason yet unknown, the smaller alpine image doesn't work)

Now we must copy the ampersand executable to /bin, from where we can run it as though it were a normal ubuntu-command. It is the only software we will copy into this image. (Haskell and the intermediate files are all absent). This results in an image that is slightly over 220MB.

When compiling, we will work in a directory called scripts. When using this container, we will volume-map this directory to the actual working directory that contains the ampersand-files we want to compile.

The program to be called when running the container is of course ampersand (residing in /bin/). If called without arguments it will use --verbose.

RAP is a Repository for Ampersand Projects. This tool is being used by the Open University of the Netherlands in the course Rule-Based Design. It lets users analyse Ampersand models, generate functional specifications, and make prototypes of information systems. It is the primary tool for students in relation to Ampersand.

Deployment means to install RAP4 on a web server or a laptop of your own. The easiest way to try Ampersand, however, is by just using the OUNL-installation and avoid the installation.

Automated deployment

We use Docker for automating the deployment and making RAP portable over different platforms. The following three sections report three different installations we did:

• to ;

• to in the Azure cloud;

• to on a server in the OUNL-datacenter.

These three reports will most likely contain enough information to let you reproduce the installation on a computer or platform of your own.

The last section of this chapter discusses the , which you may need if the above does not work because the images are (for any reason whatsoever) unavailable on the internet.

Questions? Need help?

Don't hesitate to contact Han Joosten. He'll be glad to help out.

The purpose of automation.

The release of Ampersand is largely automated because we want:

1. to save ourselves work;

2. to release frequently in a predictable rythm, to bring new functionality to users quickly and predictably;

3. reliable releases, to prevent our mistakes to hit users and to avoid delays caused by the release process;

4. reproducible releases, to allow any team members to step in when the release is due.

What we want to achieve.

Version managment

First of all, we want to be in control of our software. We use Git(hub) to do version management. We use git flow strategy. We have a master branch that holds the code of the latest stable release. Then we have a development branch that holds the latest added features and bugfixes. We want this branch to be stable too. It must be buildable at all times, and no half-on-its-way functionality should be in it. For new features or other issues, we use feature branches. These branches are work in progress. They might be buildable, or they might not. Feature branches should be used for work in progress only. This keeps the amount of branches manageable. Tags can be created for all kind of other reference purposes to specific commits.

Automatic build & test

We use to build and test the code whenever a commit is done on github. Github actions is pretty well . Our specific code can be found in the repository at the designated directory: .github/workflows/ .

Introduction

In the early days of the development of Ampersand, it was a single person's project. Haskell was used as programming language, and the source code was kept on a single PC. Fortunately, things have evolved. Now we have a team, and we use quite a lot of open source tooling to help us along. The downside is that not everybody in the team knows all there is to know about the tooling. The purpose of this guide is to help out. It will also be a great place to start for new team members.

The up to date version of this book is published automatically every time a commit is done to the master branch.

Documentation on actually using Ampersand - e.g. for making and running prototypes (including the ways to install on your computer what it takes to do so) - is in another document, called

Working Principles

• The Ampersand project produces free open source software, available to everyone.

• "We" is the as specified on Github. We adhere to our own working principles to the best of our ability.

• We respect the intellectual contribution of all. We wish to keep it that way by acknowledging intellectual ownership wherever appropriate.

Contributions to the documentation

If you have anything to fix or details to add, just post a comment next to the paragraph. We really appreciate if you do so, because our readers know best what it is they are looking for in the docs.

You can do that by clicking the icon that appears when you hover above the paragraph. Don't be shy! Try it out! (you need to be logged in however, but it is totally free to create an account).

Licence

Unless otherwise specified, everything in this repository is covered by the following licence:

Ampersand Documentation by the is licensed under a .

Based on a work at

Requirement

The proper use of Docker ensures that your application will run on any location on the internet. Yet, when testing your application on your laptop, you may discover that your laptop is not configured as a domain on the internet. To run any Ampersand application locally without changes, your laptop must behave like an ordinary top-level domain: localhost. Your computer must believe that localhost is like com, or edu, or nl. All requests from your browser(s) to localhost must be routed to IP-address 127.0.0.1, which is your laptop.

The desired result

First demonstrate that an arbitrary internet domain (here: nu.nl) is visible.

> ping -c 1 nu.nl PING nu.nl (99.86.122.115): 56 data bytes 64 bytes from 99.86.122.115: icmp_seq=0 ttl=232 time=24.367 ms --- nu.nl ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 24.367/24.367/24.367/0.000 ms

Now demonstrate that localhost is routed to IP-address 127.0.0.1.

> ping -c 1 localhost PING localhost (127.0.0.1): 56 data bytes 64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.040 ms --- localhost ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.040/0.040/0.040/0.000 ms

Then demonstrate that an arbitrary subdomain of localhost is routed to IP-address 127.0.0.1.

> ping -c 1 rap.localhost PING rap.localhost (127.0.0.1): 56 data bytes 64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.029 ms --- rap.localhost ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.029/0.029/0.029/0.000 ms

Finally, show that this works recursively on an arbitrary sub-subdomain.

> ping -c 1 foo.rap.localhost PING foo.rap.localhost (127.0.0.1): 56 data bytes 64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.032 ms --- foo.rap.localhost ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.032/0.032/0.032/0.000 ms

Complications

So why should this NOT work?

1. Your machine might not be connected to the internet. And if it is connected, it needs access to a domain name server to find locations on the internet. For most laptops this is the case, so you have no problems here. Some corporate laptops are connected only to the local corporate network. This means that you cannot let your application communicate with the outside world, or you have to organize that your laptop gains access (e.g. through a proxy).

2. You cannot run your application locally if localhost is not configured. On some laptops there is a file called hosts or .hosts, which contains the redirection from localhost to IP-address

Solutions

• Turn to to install a local DNS-server on your Macbook.

• Turn to ??? to install a local DNS-server on your Windows laptop.

• Turn to for setting the hosts-file on your Windows laptop. (Don't do this if you are installing a domain name server)

Workarounds

If you cannot get your laptop configured, you might test your application on a separate test-server that runs Linux.

Do's and don'ts

Every commit in the origin/master branch is built.

We followed , to automate the workflow from committing to the to the corresponding docker image on .

Manual builds

If you must do things on your own, you might want to reproduce parts of the builds.

The Dockerfile for the Ampersand compiler resides in the root of the Ampersand repository. So from the command line, if you are at the root of the Ampersand repository, you can call

Permissions for log files

The docker configuration uses volumes to link to the file system of the host computer. As a result the status of RAP will survive containers (i.e. you can start, stop, kill, restart containers without losing data). However, you must ensure write permission for the container on runtime throughout all three subdirectories of the volumes directory. The pod in which RAP runs defines a volume called log in which the log files are written and a volume called scripts in which user scripts are stored. The pod in which MariaDB runs writes its mysql-directory to a volume called mysql. This means that the files are stored outside the container on the machine that hosts the docker platform.

Docker group

You have a host computer in which the docker platform runs. (In deploying RAP4 for the OUNL that would be the machine with domain name rap.cs.ou.nl). To avoid permission errors (and the use of sudo), add your user to the docker group. .

Development vs. Production versions

For developing RAP you use slightly different settings. For example you want to reset the database when developing, but this feature must be entirely impossible during production. For this purpose we have two source files: RAP4dev.adl and RAP4prod.adl. The only difference between the two is configuration settings. Follow the code to find out.

The purpose of maintaining software is to guarantee continuous operation for all users. In this chapter we use the RAP4 server at OUNL as an example, trusting that you will figure out what to do for servers of your own.

Tasks

Here is an overview of all tasks described on this page. Refer to the related section below for details on each specific task

1. Connect Before doing any maintenance, you need credentials to gain access to your server. Then you can connect to the server. We assume you gain access to a command-line interface (CLI) of your server. In this text that CLI is /bin/bash on a linux machine.

2. Check the configuration When you start a maintenance session, you may want to check on the system. RAP4 runs in containers on a docker-platform, so you can check whether the containers are running and you can check the configuration in which they should be running.

3. Upgrade to a new version of RAP

4. Upgrade to a new version of Ubuntu

5. Refresh the configuration Refreshing the RAP4 configuration is something you need to do only when developers tell you to do that.

Connecting to the RAP4 server

The RAP-server has been configured to communicate via ssh. The Open Universiteit allows ssh-connections only through VPN. So I made sure my VPN-connection is active.

I entered the server using command ssh rap.cs.ou.nl and the right user/password combination, upon which I gained access to the CLI.

The RAP4-instance is installed from directory ~\RAP4, which is the working directory from which maintenance is done.

This is what you can expect to see:

Checking which containers are running

The server should show at least two containers, a database container called rap4_db and a RAP-container called rap4. There may be a third container called phpmyadmin, which is there to gain access to the database (for maintainers only). To verify, give the command docker ps. This is what you may expect to see:

In this case it appears that a fourth container is running. This poses no problem, because other processes may run concurrently without interfering with RAP4.

The configuration file that specifies this configuration is the only file in the working directory:

By inspecting the contents you can see whether the configuration matches what you see in docker. This is what you can expect in docker-compose.yml:

The directory volumes contains the Ampersand data, which is kept outside the containers so the data persists if containers get killed or if you restart the software.

There are three data sets. The directory log contains logging information of the RAP4-server. The directory mysql contains the data from MariaDB. The directory scripts contains the student scripts and the files they generated when working in RAP4.

Upgrade to a new version of RAP

If you need to upgrade RAP4 to the latest release run this command:

As you can see in the configuration (docker-compose.yml)

Upgrade to a new version of Ubuntu

When you connect to the server, you get hints about the state your Ubuntu server is in. E.g.

In such cases you can update by giving two commands:

Sometimes a package is kept back because there is a problem with dependencies. You will have to upgrade such packages by hand.

Refreshing the code

In the rare event that the configuration of RAP4 has changed (to be announced by the developers), you must update the file docker-compose.ymlby hand, using the command:

What Docker needs to do...

Docker is sheer magic. You don't have to understand it to run it. However, if you insist on knowing how it is done...

Here are ten steps to install RAP4 manually, without Docker, on a freshly created server:

1. Setting up a server

You need a server that is connected to the internet, because RAP takes updates from a GitHub repository. A 2-core/8GB server is sufficient. A memory size under 3GB has shown to be insufficient. Ports for HTTP, HTTPS, SSL, and SFTP must be open.

2. Getting MariaDB (MySQL) and phpMyAdmin to work

You need a database and a web-server. This explains why a LAMP-server is an obvious choice if you use a preconfigured server.

3. Uploading and running RAP4

A quick way to install is to copy the source code of RAP4 on your server and compile it with Ampersand. That gives you the RAP4 webapplication. You will find the complete Ampersand source code of RAP4 on . The main file is ./RAP4/RAP4dev.adl for the development version and ./RAP4/RAP4prod.adl for the production version.

4. Filling the Git repository with Ampersand files and Ampersand models

If you don't have an Ampersand compiler, you can build one using the Haskell sources of that compiler. You will find the source code on the development branch of the Ampersand repository (). You can verify success by asking the Ampersand for its version:

If you don't have an Ampersand compiler, but docker runs on your machine, you can simply use the most recent ampersand compiler by

Use Git to create a local clone of these repositories. Git is preferred over copying the files, because you can repeatedly use it to ensure you get the right version.

5. Avoiding to install Haskell

To build an Ampersand compiler, you need Haskell. However, you can also use the latest executable Ampersand-compiler, which is published periodically on . This way, you don't need to build Ampersand and you can avoid installing Haskell.

6. Creating an Ampersand compiler

If you want, create your own Ampersand compiler. This way, you can pick a version by which to generate RAP4.

7. Installing LaTeX and GraphViz

LaTeX and GraphViz are called by RAP4 to generate documentation. Therefore, both must be installed on the server.

8. Generating the RAP4 application

You need to generate RAP4 to facilitate regular updates to the system.

9. Local Settings

Some local settings may apply, which are brought together in one file. Use this to administer things like database account(s) and PHP time limit, logging, etcetera.

10. Last minute changes before going to production

Things that are necessary for testing and development, such as logging, can be changed in the production version. These things are usually adapted shortly before going to production.

An Ampersand application may grant access to named users. We have done this before using the open standard OAuth (Open Authorization). This allows access to specific private data on a different website, without requiring the user to hand over their credentials (e.g. username/password).

Setting up with Github as access provider

I made a separate Github organization, RAP-OUNL, to serve as access provider for RAP4.

callback URL: http://example.com/AmpersandPrototypes/RAP4/api/v1/oauthlogin/callback/github

Experimenting from a laptop

For experimentation purposes, I added the following line to C:\Windows\System32\drivers\etc on my Windows 7 laptop.

Then I flushed the local name server in a command line window.

Then I brought up a browser window, to see that example.com is a valid URL. Now I can experiment with OUauth from the laptop, because the OAuth server can return messages to example.com.

In early 2017 the need arose for an Ampersand implementation in Ordina's cloud, to let young professionals get acquainted with Ampersand. We chose to implement RAP4 in Azure, because Ordina has an Azure subscription. RAP4 is the same environment that our students use at the Open Universiteit, so the maintainance of RAP4 can be reused for both target audiences.

This chapter is an account of the deployment of RAP4. You can use it as an example if you want to do it yourself.

Deployment has been automated using docker, because we want to make the deployment as much as possible automated. The required images are available on Docker-hub, so deployment is immediate. Automated deployment by docker means that we can quickly release updates too.

0. Deployment approach

We use Docker to simplify the deployment of RAP and make this installation portable to many different computers.

We took a simple Linux machine, installed docker, and built docker images for RAP4 and created three containers: db, rap4, and phpmyadmin. The first one, db, runs MariaDB and contains the administration of RAP4 and the Ampersand scripts of users. The rap4 container contains the RAP4 application proper. It is accessed on port 80. The container phpmyadmin is present for maintainers to inspect the database through phpMyAdmin. It is accessible through port 8080.

Each step in the installation process is described by a separate section in the sequel.

1. Setting up the virtual machine

I needed an Azure account to enter the Azure portal and install a server for Ampersand. I got my account from Ordina. Azure offers preconfigured installations to kick-start a virtual machine. In principle, any server that runs docker will work. I picked CoreOS, which is a minimal Linux platform and suitable for container computing with docker.

The following settings were made:

I verified completion of this step by checking the Azure dashboard. It shows machine Wolfram running.

I have verified the machine was live by logging in via Putty (a popular SSH-client). I accessed this machine, using ampersandadmin@137.117.190.28 and the correct password. In the sequel, I will refer to this machine as "the server".

2. Port settings

To perform this step, I didn't wait until step1 was completed. It is sufficient that the Netwerkbeveiligingsgroep Wolfram-nsg exists. In order to access RAP4 from the internet, we told the virtual machine to open port 80 for http and 443 for https. These settings were made in the Azure portal via Resourcegroepen > AmpersandRAP4 > Wolfram-nsg (Netwerkbeveiligingsgroep) > inkomende beveiligingsregels. A rules was added to make the server listen to port 80/TCP.

No rule was added for the database, so the database is only open to phpMyAdmin and RAP4, or from within its container.

3. Installing Docker

This step requires a server, so you must have finished section 1 successfully. (If docker and docker-compose are available, this step is superfluous.) To install docker I followed the following steps in the given order:

Of course, the last step (reboot) requires reconnecting to the machine.

Then I checked that docker is running by:

I checked that docker-compose is available by:

which docker-compose

To check whether ampersandadmin is member of the docker group, use command cat /etc/group.

4. Obtaining the install instruction

We need only one file: docker-compose.yml To get it, I used the wget command, which gets stuff from the web:

5. Installing RAP4

To install RAP4:

To check whether this worked, I went to my browser and navigated to http://137.117.190.28/. It took a while to get started, because it was building a fresh database.

I checked whether the containers are running by means of the docker ps command.

Completion of this step allowed access to RAP4 from an arbitrary computer on the internet:

The database is accessible on port 8080:

As we start fiddling around with getting a first version of VSCode extention, it turns out all kind of tooling needs to be in place. The vision of the extention is that it will eventually give ampersand modellers a rich integrated development environment, including syntax colouring, prettyprint, syntax supported editing, script execution etc etc.

At first, I had a look at language-haskell. They have some IDE support in place, so it seems a reasonable first start. I don't like to invent the wheel myself.

There is a lot of javascript going on, so we need a package manager for it.

Releasing of this vs-code extension

We have a release pipeline in place using travis-ci. See the travis.yml file in the root directory. Documentation of extentions can be found .

The personal access token in use has an expiry date on it. An email is sent to the owner of that token (currently Han), in advance of the expiry. It can be renewed or the expire date .

Docker images

This chapter discusses the process of baking (i.e. creating) images for docker hub. It also gives a recipe for creating docker images. If you are interested only in installing RAP4, you do not need this chapter.

Docker images are baked when the Ampersand source code for RAP4 is ready to be deployed. This results in three files, to be stored on Docker hub:

The release train

RAP4 is deployed as shown in this picture

A recipe for creating images

Knowing what needs to be done allows you to understand how we make Ampersand's docker images. If you just want to do it, follow the steps below. We assume that you are working on an Ubuntu machine with bash as its command line interface.

Check your Docker installation

First I checked that docker and docker-compose are installed on my computer:

If you need to install docker, follow the instructions on .

4. Building an Ampersand image

RAP is built on Ampersand and is compiled with Ampersand. For this reason, the RAP image builds on an Ampersand image. Run ./build.sh to build the initial ampersand container that serves as a base for the RAP4 application (or any other Ampersand application). This base images holds all required packages and the (at that moment) latest version of the ampersand compiler the workflow around this container can/should be improved since now the easiest way to rebuild is to remove the container (docker rmi ampersand:latest)

4. Making a Docker image

I cloned the docker files into a local directory by the following command:

and sat back to watch an image being created. This takes over an hour. I left the session up and running, because stopping the session means that the docker build process stops. After coming back a day later, I verified that the image is present by running the same command again. That produced the following output:

When you build an image where nothing changes (as shown above), this takes virtually no time at all. However, if you run a fresh image from scratch, it takes a while. Should you want to close your SSH-session in which docker is running, that docker process will be killed. If you want to leave docker running in the background, https://www.tecmint.com/keep-remote-ssh-sessions-running-after-disconnection/ will help. Best thing is to anticipate on it. However, if you didn't anticipate this, you can still send docker to the background by ^Z and give the disown -h command. (The latter is however the least elegant way.)

I checked whether all images are built by the docker images command:

5. Publishing to docker-hub

For this step, you need an account on docker-hub. Then Docker will simply push the generated images to docker hub by means of the push-command:

At this point the images are published and this chapter is done. However, it is good to discuss a few collateral issues: deployment, maintenance and security.

6. Deploying directly on this server

We are making three containers: one for the database, one for the RAP4 application, and one for PhpMyAdmin. Containers are built from images by the command docker-compose:

This command brings all three containers up and runs them in the background. For this to happen, the file docker-compose.yml must be in the current directory.

I checked whether the containers are running by means of the docker ps command.

Completion of this step allowed access to RAP4 from an arbitrary computer on the internet:

The database is accessible on port 8080:

7. Maintenance

The docker-compose up command aggregates the output of each container. When the command exits, all containers are stopped. Running docker-compose up -d starts the containers in the background and leaves them running.

To interfere with RAP4 as it is running, you need to get into the Rap4 container. It is not enough just being on the server, because all the data is in the container (rather than directly on the server). To go into the Rap4 container, use the command

10. Local Settings

To inspect and change the local settings, you need the file localsettings.php on directory ~/git/Ampersand-models/RAP4/include. This step requires section 5 to be finished successfully. This file contains comments that guide you to use the correct settings in a development situation and in a production situation. Read the file and follow the instructions it contains, especially when making the transition from development to production.

11. Last minute changes before going to production

1. In the source code of RAP4, in the file SIAM_importer.adl:

   1. disable "RAP4_LoginForDevelopment.ifc", to prevent users from seeing

   2. enable "RAP4_LoginForProduction.ifc"

This is decribed in the file docker-compose.yml

Purpose

Deploying your Ampersand application to production can be done fast and frequent. For this purpose we are working on deployment on the Kubernetes platform. This allows you to deploy easily on your laptop (for testing), in your own data center (The minimum you need is a virtual machiine that runs kubernetes), or in the cloud (to benefit from the reliability and security provided by hosting providers). We use Kubernetes because it is open, free, well known, widely used, and is continuously being impoved by a large community. It lets you deploy fast and lets you upgrade the application without taking it offline.

Way of working

1. A Kubernetes cluster is needed as the platform from which to launch and maintain the application. If it is there, that's fine. Otherwise you get one from a provider, from a data center, or you create one yourself (e.g. on your laptop). The cluster is built on top of virtual or physical machines and hosts and exposes the services that constitute the application.

2. On the platform I need a registry from which Kubernetes can pull the image(s).

I tried it out for myself. I stole my inspiration from . Thank you Kevin! I ran the following from my terminal (i.e. MacOS cli), just for the purpose of getting some hands-on experience...

Requirements

To make a Kubernetes cluster, I used Minikube. Minikube requires that VT-x/AMD-v virtualization is enabled in BIOS. To check that this is enabled on OSX / macOS, I ran:

If there's output, it works!

Prerequisites

• kubectl

• docker (for Mac)

• minikube

Verify

Start

This can take a while, expected output:

Great! You now have a running Kubernetes cluster locally. Minikube started a virtual machine for you, and a Kubernetes cluster is now running in that VM.

Check k8s

Should output something like:

Within minikube, a docker platform is running. However, on my Mac I have another docker daemon running. So which docker do we want to talk to? In this case that is definitely the docker daemon that is inside minikube...

Use minikube's built-in docker daemon

(You might add this line to .bash_profile or .zshrc or ... to use minikube's daemon by default. Or if you do not want to set this every time you open a new terminal).

If I want to talk to the docker daemon on my Mac, I have to revert back to it by running:

When running docker ps, it showed the following output:

The list was in fact a little bit longer: it contained 22 containers.

Build, deploy and run an image on your local k8s setup

To setup a local registry, so Kubernetes can pull the image(s) from there:

To test whether there is a local registry, ask docker:

Build

First of, store all files (Dockerfile, my-app.yml, index.html) in this gist locally in some new (empty) directory.

You can build the Dockerfile below locally if you want to follow this guide to the letter. Store the Dockerfile locally, preferably in an empty directory and run:

You should now have an image named 'my-app' locally, check by using docker images (or your own image of course). You can then publish it to your local docker registry:

Running docker images should now output the following:

Deploy and run from docker-compose.yml

I have a file called docker-compose.yml, which I have used to deploy RAP with docker-compose (rather than Kubernetes). I used kompose to transform that into the necessary Kubernetes .yaml files.

I deployed straight from the docker-compose.yml file:

However, I can also create the necessary Kubernetes files and deploy from kubectl:

This converts docker-compose.yml into the following files:

Having run kompose up to deploy the three RAP services, I now have the following situation on my minikube:

(I got even more information with the command kubectl get all --output=wide.)

So we see three pods (containers) running. Each of the pods is wrapped in a service of its own and exposed internally on the local network of this cluster. (In Kubernetes, , and all have their own IPs. In many cases, the node IPs, pod IPs, and some service IPs on a cluster will not be routable, so they will not be reachable from a machine outside the cluster, such as your desktop machine.)

To access these services from a browser, we now need to expose these services to localhost. I did this by exposing both services from the minikube platform:

Deploy and run from Kubernetes yml files

Store the file below my-app.yml on your system and run the following:

You should now see your pod and your service:

The configuration exposes my-app outside of the cluster, you can get the address to access it by running:

This should give an output like http://192.168.99.100:30304 (the port will most likely differ). Go there with your favorite browser, you should see "Hello world!". You just accessed your application from outside of your local Kubernetes cluster!

Kubernetes GUI

Delete deployment of my-app

You're now good to go and deploy other images!

Reset everything

Version

Last tested on 2017 October 20th macOS Sierra 10.12.6

As a student, your main objective with RAP4 is to write Ampersand scripts and have them compiled. RAP4 will help you by giving error messages when your script contains errors. If your script complies to the Ampersand syntax and has no type errors, it will let you generate artifacts from your script. Currently these artifacts are

1. A diagnosis document

2. A functional design document

3. An analysis site, containing an overview of concepts, relations and rules in your script.

You might find it interesting to know that RAP4 has been built with Ampersand itself!

Create an Ampersand script

A new script can be made by hitting the + sign in the menu and go to New Script. A new script is being made, but it doesn't have any content:

In the header of the script you can see Script

_

_

_1491765040

\

_

00901936, which is a

(

generated

)

identifier of the instance, with which you have nothing to do. The big white space in the middle is where you will write your script. You could als choose to upload a script from your computer. It is a good habit to save your scripts at your own computer, because currently we have no functionality to retrieve your script easily. Please take care of your own precious work yourself.

Each time you modify your script, you have to send it to Ampersand by pressing the big blue button below the white space. Only then Ampersand will be notified about your changes.

After you pressed the blue button, you can check the compile checkbox. Ampersand is launched and it will validate your script. If any error is found, it will notify you, as shown above.

If your script is OK however, it will also tell you, by letting you know that processing has come to an end:

Every time compilation is succesful, a new version of your script will be made. You will see the most recent version. Now you can ask for the artifacts, by checking an appropriate checkbox. This will take some time, and a link will appear at the right of the checkbox. If you follow that link, the result will be shown.