EMS - Self Hosted Guide

This section will provide information related to self-hosted deployment of Execution Management System API (EMS API) software.

info

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Configuration Manual

This section will provide the necessary information how to configure the CoinAPI EMS software.

Configuration keys and values

Configuration keys:

Are case-insensitive. For example, ExchangeId and exchangeid are treated as equivalent keys.
If a key and value is set in more than one configuration providers, the value from the last provider added is used.
Using array indices in configuration keys are allowed.
Hierarchical keys
- Within the Configuration API, a colon separator (:) works on all platforms.
- In environment variables, a colon separator may not work on all platforms. A double underscore, __, is supported by all platforms and is automatically converted into a colon :.

Configuration values:

Are strings.
Null values can't be stored in configuration.

Configuration providers

You can set the values of configuration parameters for the components using the following providers (in case of collision for any specific key):

Command-line arguments
Environment variables

Environment variables

To configure specific parameters using host or container environment variable, you must replace all occurrences of : with __ in the configuration parameter key, this is because the : separator doesn't work with environment variable hierarchical keys on all platforms. __, the double underscore, is:

Supported by all platforms. For example, the : separator is not supported by Bash, but __ is.
Automatically replaced by a :

For example, to set the value of the configuration parameter OEML:ExchangeId to BITSTAMP you would need to set the environment variable named OMS__ExchangeId to BITSTAMP.

Command-line arguments

You can set the configuration parameters by providing the arguments to the application executable.

The following example sets keys and values using the =: OEML:ExchangeId=BITSTAMP The following example sets keys and values using the /: /OEML:ExchangeId=BITSTAMP The following example sets keys and values using the --: --OEML:ExchangeId=BITSTAMP or --OEML:ExchangeId BITSTAMP

The key value:

Must follow =, or the key must have a prefix of -- or / when the value follows a space.
Isn't required if = is used. For example, MySetting=.

Within the same command, don't mix command-line argument key-value pairs that use = with key-value pairs that use a space.

Configuration parameters

Key	Type	Description	Is Required?	Default value
OEML:ServiceDiscoveryMode	string	Type of backend used for service discovery. Supported values: `COINAPI` or `CONSUL`	No	`COINAPI`
OEML:LocalAddress	string	Override the local IP address advertised to other components. The value of this parameter is used in the CoinAPI Service Discovery mode. The API is listening to all the IP addresses available. However, the software still needs a valid single IP address accessible to other components, we will use this address for advertising in the CoinAPI Service Discovery mode only.	No	IP of the first IPv4 network interface.
OEML:Port	int	Set the TCP Port for REST and WebSocket API.	No	Randomly selected free port.
OEML:FIX:SocketAcceptPort	int	FIX Acceptor/Server Port	No	3401
OEML:ExchangeId	string	Exchange identifier and optional tag identifying specific account configured when the software will be managing multiple accounts on the same exchange; for eg: `BITSTAMP` `BITSTAMP/7c177641-74bd-4dbe-9b01-2497c12a5f70` `BITSTAMP/2574`. Allowed separators between the exchange identifier and the tag: `~/.,:;\!@#$%^&*-_+=` .	Yes
CoinAPI:ApiKey	string	CoinAPI API Key	Yes
CoinAPI:UrlRest	string	CoinAPI REST API Endpoint	No	https://rest.coinapi.io/
CoinAPI:RatesUpdateIntervalSeconds	int	Update interval of the exchange rates in seconds from CoinAPI (using `/v1/exchangerate/USD` endpoint). `0` mean that rates will be pulled just on startup once, and `-1` mean that they will not be downloaded at all.	No	3600
OD:PublicApiKey	string	Customer Public Key for the Exchange	Conditionally
OD:PrivateApiKey	string	Customer Private Key for the Exchange	Conditionally
OD:PassPhrase	string	Customer passphrase for the Exchange	Conditionally
OD:CustomerId	string	CustomerId of customer exchange account	Conditionally
OD:Username	string	Exchange client username	Conditionally
OD:Password	string	Exchange client password	Conditionally
OD:Host	string	Exchange API endpoint host	Conditionally
OD:Port	int	Exchange API endpoint port	Conditionally
OD:RestApiUrl	string	Exchange REST API Url	Conditionally

info

"(ExchangeId)" must be replaced with the value of "OEML:ExchangeId"

Exchange conditional parameters

The table below explains the conditional parameters by the OEML:ExchangeId value. For example when OEML:ExchangeId == "BITSTAMP" then OEML:Exchanges:BITSTAMP:CustomerId is required.

OEML:ExchangeId	Required parameters	Optional parameters
BINANCE	OD:PublicApiKey OD:PrivateApiKey
BINANCEJE	OD:PublicApiKey OD:PrivateApiKey
BINANCEUS	OD:PublicApiKey OD:PrivateApiKey
BINANCEUAT	OD:PublicApiKey OD:PrivateApiKey
BINANCEFTS	OD:PublicApiKey OD:PrivateApiKey
BINANCEFTSUAT	OD:PublicApiKey OD:PrivateApiKey
BINANCEFTSC	OD:PublicApiKey OD:PrivateApiKey
BINANCEFTSCUAT	OD:PublicApiKey OD:PrivateApiKey
BINANCEOPTV	OD:PublicApiKey OD:PrivateApiKey
BINANCEOPTVUAT	OD:PublicApiKey OD:PrivateApiKey
BITFINEX	OD:PublicApiKey OD:PrivateApiKey
BITMEX	OD:PublicApiKey OD:PrivateApiKey
BITMEXUAT	OD:PublicApiKey OD:PrivateApiKey
BITSTAMP	OD:PublicApiKey OD:PrivateApiKey OD:CustomerId
BLOCKCHAINEXCHANGE	OD:PublicApiKey OD:PrivateApiKey
COINBASE	OD:PublicApiKey OD:PrivateApiKey OD:PassPhrase	OD:Host OD:RestApiUrl
GEMINI	OD:PublicApiKey OD:PrivateApiKey
HITBTC	OD:PublicApiKey OD:PrivateApiKey
KRAKENFTS	OD:PublicApiKey OD:PrivateApiKey
KRAKEN	OD:PublicApiKey OD:PrivateApiKey
POLONIEX	OD:PublicApiKey OD:PrivateApiKey
LMAXDIGITAL	OD:Username OD:Password OD:Host OD:Port
LMAXDIGITALUAT	OD:Username OD:Password OD:Host OD:Port
FTX	OD:PublicApiKey OD:PrivateApiKey	OD:Host OD:Port OD:RestApiUrl
FTXUS	OD:PublicApiKey OD:PrivateApiKey	OD:Host OD:Port OD:RestApiUrl
DERIBIT	OD:ClientId OD:ClientSecret	OD:Host OD:Port
DERIBITUAT	OD:ClientId OD:ClientSecret	OD:Host OD:Port
DYDX	OD:ApiKey OD:ApiSecret OD:ApiPassphrase OD:StarkPrivateKey	OD:NetworkType OD:LimitFee

Exchange specific recommendations

Kraken Futures `KRAKENFTS`

Whitelist your IP addresses

Whitelisting your IP addresses will give you the lowest possible latency and most stable connection to exchange servers in AWS EU-West 1. Whitelisting works by allowing you to skip CloudFlare. Here is an article on how to whitelist your IP addresses.

Consul Configuration (optional)

The end result should look like this:

{
  "datacenter": "hq",
  "data_dir": "C:\\consul\\data-hq",
  "log_level": "INFO",
  "server": false,
  "retry_join": [ "server-a", "server-b", "server-c" ],
  "ui": true,
  "http_config": {
    "response_headers": {
      "Access-Control-Allow-Origin": "*"
    }
  }
}

If your deployment including optional Consul Cluster for the Service Discovery layer then it's required to change CORS settings of the local Consul client on the hosts where the EMS WebUI will be used. This step is necessary as we need to discover the address of the EMS API component.

"http_config": { "response_headers": { "Access-Control-Allow-Origin": "*" } }

Installation Manual

This section will provide the necessary information how to install the CoinAPI EMS software.

Service discovery

The CoinAPI EMS API is capable of automaticaly detecting all your instances of CoinAPI EMS Edge service. This service discovery works by the one of the following methods:

Using the CoinAPI EMS Central Cloud, every time you start CoinAPI EMS Edge, it will register automatically in our central cloud infrastructure.
Using the HashiCorp Consul Cluster.

Using the CoinAPI EMS Central Cloud is a default, simple, and straightforward method. Only one disadvantage of using the CoinAPI EMS Central Cloud over Consul Cluster is that we do not measure the latencies if one order destination is registered more than one time in the cluster for failover scenarios. Then CoinAPI EMS Central Cloud will connect to the last started exchange; however, when using the Consul Cluster, it will connect to the order destination in the closest proximity data center to the datacenter where CoinAPI EMS API is run.

Running using an executable

To install the precompiled binary, download the latest version of the executable matching your system architecture from http://coinapi-releases.s3-website-us-east-1.amazonaws.com/

Our software is currently packaged as a zip file, and we currently do not plan to distribute it another way. Once the zip is downloaded, unzip it into any directory. The self-contained binary inside is necessary to run EMS, and any additional files are not required to run the software.

If you intend to access it from the command-line, make sure to place the binary somewhere on your PATH. You may require to add the executable to the system startup with all necessary parameters if you want the software to be started automatically after restart.

Running on Docker

#  example using only command line parameters
docker run -d --name=HITBTC --restart=always --net=host coinapi/oeml-api --OEML:ExchangeId HITBTC \
    --OD:PublicApiKey _public_api_key_ --OD:PrivateApiKey _private_api_key_ \
    --CoinAPI:ApiKey _coinapi_api_key_ 
docker run -d --restart=always --net=host coinapi/oeml-webui
docker run --net=host coinapi/oeml-cli

# example with enviroment variables file
cat > oeml.env << EOF
CoinAPI__ApiKey=_coinapi_api_key_

OD__PublicApiKey=_public_api_key_
OD__PrivateApiKey=_private_api_key_
EOF

docker run -d --name=HITBTC --restart=always --net=host coinapi/oeml-api --env-file oeml.env --OEML:ExchangeId HITBTC
docker run -d --name=BITFINEX --restart=always --net=host coinapi/oeml-api --env-file oeml.env --OEML:ExchangeId BITFINEX
docker run -d --name=OEML_EDGE --restart=always --net=host coinapi/oeml-api --env-file oeml.env

Our software is listed in public docker repository. To run the software on docker, use the following images:

coinapi/oeml-api
coinapi/oeml-cli
coinapi/oeml-webui

To pass the configuration parameters using the environment variables (more information about this feature is available in the Configuration Manual section of this documentation), the environment variables must be provided into the container and not set on the host machine.

It's a good practice to lock itself into the specific version. To do that you can supply the version after image name and : separator, e.g. coinapi/oeml-api:1.1255

Running on Kubernetes Cluster

Our software is compatible with Kubernetes, and you can quickly run it on the cluster using our (Helm Charts)[https://github.com/coinapi/helm-charts]. Follow steps below to perform quickstart deployment.

Add the Repository to Helm

$ helm repo add coinapi-charts https://coinapi.github.io/helm-charts/
$ helm repo update

Helm chart configuration

The example config below will create two api pods, a composite pod, and a webui pod. One exchange account corresponds to one entry in the values.yaml file. That means, for multiple exchange accounts, please add one config entry with the matching API keys for each account i.e. binance-01, binance-02, kraken-01, kraken-02, etc. A complete list of all supported exchanges can be found in the API documentation.

cat > values.yaml << EOF

# Using WebUI (oeml-api/values.yaml)
webUI:
  enabled: true

# Using composite (oeml-api/values.yaml)
oemlCompositeAPI:
  enabled: true

# Setting the CoinAPI key (oeml-api/values.yaml)
extraEnv:
  - name: CoinAPI__ApiKey
    value: "YOUR_COINAPI_APIKEY"

# Specify listing of the accounts that EMS should manage (oeml-api/acct-dev.yaml)
accounts:
  # Binance spot market 
  - name: "binance"
    env:
      - name: OEML__ExchangeId
        value: "BINANCE"
      - name: OD__PublicApiKey
        value: "XXX"
      - name: OD__PrivateApiKey
        value: "YYY"
        
  # Binance spot testnet
  - name: "binanceuat"
    env:
      - name: OEML__ExchangeId
        value: "BINANCEUAT"
      - name: OD__PublicApiKey
        value: "XXX"
      - name: OD__PrivateApiKey
        value: "YYY"
        
EOF

Install ems-api latest version

$ helm install oeml-api coinapi-charts/oeml-api -f values.yaml

Install ems-api custom version

Version listing available here.

$ helm install oeml-api coinapi-charts/oeml-api -f values.yaml --set oemlAPI.tag={version}

Verify installation

kubectl get pods | grep oeml

Expected output:

oeml-api-binance-7754fc95cc-x2gkk        1/1     Running
oeml-api-binance-test-757c8bbd95-2mxd5   1/1     Running
oeml-api-composite-d5688f8f6-csbwh       1/1     Running
oeml-webui-685cd94d77-hq68j              1/1     Running 

Inspect

Inspect binance api pod:

kubectl logs oeml-api-binance-7754fc95cc-x2gkk

Inspect testnet api pod:

kubectl logs oeml-api-binance-test-757c8bbd95-2mxd5

Expected output for both api pods:

[02:22:04 WRN] Overriding address(es) 'http://+:80'. Binding
to endpoints defined in UseKestrel() instead.
Hosting environment: Production
Content root path: /app
Now listening on: http://[::]:80
Application started. Press Ctrl+C to shut down.

List services

kubectl get svc | grep oeml

Expected output:

oeml-api-binance        ClusterIP   10.112.9.119    <none>        80/TCP
oeml-api-binance-test   ClusterIP   10.112.15.239   <none>        80/TCP
oeml-api-composite      NodePort    10.112.8.159    <none>        80:30578/TCP
oeml-webui              NodePort    10.112.4.195    <none>        80:30857/TCP              

Connect to EMS API

The composite pod gives unified access to all connected exchanges & accounts. Therefore, the oeml-api-composite is the only pod required to connect to the EMS system.

Connect from within the cluster

Deploy a pod with a shell

kubectl apply -f https://k8s.io/examples/application/shell-demo.yaml

Verify the shell pod is online

kubectl get pod shell-demo

Get api-composite IP address:

kubectl get svc | grep oeml-api-composite
>oeml-api-composite      NodePort    10.112.8.159

Exec into the pod

kubectl exec --stdin --tty shell-demo -- /bin/bash

CURL IP of the composite pod.

curl 10.112.8.159/v1/balances --header 'Accept: application/json'

CURL DNS of the composite pod.

DNS name may vary depending on cloud provider.

curl oeml-api-composite.default.svc.cluster.local/v1/balances --header 'Accept: application/json'

Expected output:

[
  {
    "type": "BALANCE_SNAPSHOT",
    "exchange_id": "BINANCE",
  },
  {
    "type": "BALANCE_SNAPSHOT",
    "exchange_id": "BINANCEUAT",
    "data": [
      {
        "id": "XRP",
        "asset_id_exchange": "XRP",
        "asset_id_coinapi": null,
        "balance": 50000.00000000,
        "available": 50000.00000000,
        "locked": 0.00000000,
        "traded": 0.0,
        "last_updated_by": "EXCHANGE",
        "rate_usd": null
      },
      // more entries
      ]
  }
]

Exit the shell pod

exit

Delete shell pod

kubectl delete pod shell-demo

Connect cluster to localhost

Port-forward from cluster port 80 to localhost port 8080

 kubectl port-forward service/oeml-api-composite 8080:80

Expected output:

Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

CURL account balance.

curl localhost:8080/v1/balances --header 'Accept: application/json'

Expected output:

[
  {
    "type": "BALANCE_SNAPSHOT",
    "exchange_id": "BINANCE",
  },
  {
    "type": "BALANCE_SNAPSHOT",
    "exchange_id": "BINANCEUAT",
    "data": [
      {
      // more entries
      },
      ]
  }
]

Management Guide

This section will provide the necessary information on how to manage the CoinAPI EMS software.

To run the EMS cluster, you will need to:

(optional) Deploy the HashiCorp Consul Cluster for Service Discovery, if you don't want or can't use our included free of charge discovery service.
Run the CoinAPI EMS Edge software instance per each managed exchange account.
Run single or multiple instances of the CoinAPI EMS API software.

If the lowest latency is required, start the instance per location or server where the API will is used to minimize latency between API client and the EMS API Server.

(optional) Run the CoinAPI EMS WebUI to have GUI to the cluster.
Done! All the exchange accounts can be managed using the single API.

Installation

CoinAPI EMS can be installed using several approaches:

Running using an executable
Running on Docker
Running on Kubernetes Cluster

Our software is compiled for the following architectures:

linux-x64
linux-arm64
osx-x64
osx-arm64
win-x64
win-arm64

Take a look at the Starter Guide of this documentation for more information how to run the software in the specific conditions.

Upgrading

Currently, all components are stateless. Upgrading stateless components is limited to replacing the binaries or running the new docker image.

Versioning & Compatibility

Our software is versioned using the Major.Minor(+branch) pattern. Major version number is incremented manually on our side, and its scope is for the EMS project. The Minor version number is incremented on every release automatically separately for every component.

You can assume that our software components will work together when their Major version numbers are the same.

Time synchronization

EMS highly depends on the current time to perform actions and correctly timestamp the data for real-time trading, risk management, compliance, or reporting.

For the time synchronization we strongly recommending:

chrony (Linux, FreeBSD, NetBSD, Solaris, macOS)
w32tm (Windows)

If you can't use any of the programs above for reasons like:

Software has not been ported to my operating system.
Software doesn't have a driver or it's not compatible with my hardware reference clock, and I can't use other programs like gpsd or ntp-refclock to provide reference time via the SHM or SOCK interface.

Then it's worth trying solutions below (ordered from the best to worst options):

ntp (Linux, FreeBSD, NetBSD, OpenBSD, Solaris, macOS, Windows)
openntpd (Linux, FreeBSD, NetBSD, OpenBSD, Solaris, macOS)

If need the sub-millisecond or sub-nanosecond accuracy take a look there:

Limits

This section will list all known software limitations.

Limit description	Limit value
Maximum number of managed exchange accounts in a single cluster	`Unlimited`
Maximum number of managed exchange accounts per cluster host when running an executable	`65,535`
Maximum number of managed exchange accounts per cluster node when running on Docker or Docker Swarm	`1024`
Maximum number of managed exchange accounts per cluster node when running on Kubernetes	`250`
Maximum number of managed exchange accounts per cluster node when running on Amazon Elastic Kubernetes Service (EKS)	`from 4 to 737`
Maximum number of managed exchange accounts per cluster node when running on Google Kubernetes Engine (GKE)	`100`
Maximum number of managed exchange accounts per cluster node when running on Azure Kubernetes Service (AKS)	`250`

tip

There is no limit of exchange accounts that can simultaneously be managed using our EMS API.

Integrations & Plugins

This section is reserved for Integrations, plugins or other external tools or modules that make it easier to work with EMS API.

Prometheus metrics

In this section, we will describe the metrics exposed by the software on the /metrics endpoint.

If you are using our Helm chart then ServiceMonitor objects required by Prometheus to pull data are automatically installed by default version of the values file (helm-charts/charts/oeml-api/values.yaml).

Published metrics:

Metric name	Parameters	Description
coinapi_oeml_total_open_order	ExchangeId	Number of all orders placed since the start of oeml
coinapi_oeml_open_order	ExchangeId	The current number of open orders
coinapi_oeml_open_order_by_status	ExchangeId, Status	Current number of order statuses
coinapi_oeml_total_order_per_symbol	ExchangeId, Symbol	Number of orders per symbol since start oeml
coinapi_oeml_open_position	ExchangeId	The current number of open posiotions
coinapi_oeml_balance_total	ExchangeId, Symbol	Current balance - total per symbol
coinapi_oeml_balance_free	ExchangeId, Symbol	Current balance - free per symbol
coinapi_oeml_balance_locked	ExchangeId, Symbol	Current balance - locked per symbol
coinapi_oeml_total_error	ExchangeId, ErrorText	Total error by type
coinapi_oeml_actual_balance_total_usd_by_symbol	ExchangeId, Symbol	Actual total balance usd by symbol
coinapi_oeml_total_balance_value_in_usd	ExchangeId	Total balance value in dollars
coinapi_oeml_orders_change_status_time	ExchangeId, Status1, Status2	Orders change status time (time in miliseconds)

Deployment

In this section we are listing tools that could be used to deploy or manage our software.

Was this section helpful?

Configuration Manual​

Configuration keys and values​

Configuration providers​

Environment variables​

Command-line arguments​

Configuration parameters​

Exchange conditional parameters​

Exchange specific recommendations​

Kraken Futures KRAKENFTS​

Whitelist your IP addresses​

Consul Configuration (optional)​

Installation Manual​

Service discovery​

Running using an executable​

Running on Docker​

Running on Kubernetes Cluster​

Add the Repository to Helm​

Helm chart configuration​

Install ems-api latest version​

Install ems-api custom version​

Verify installation​

Inspect​

List services​

Connect to EMS API​

Connect from within the cluster​

Connect cluster to localhost​

Management Guide​

Installation​

Upgrading​

Versioning & Compatibility​

Time synchronization​

Limits​

Integrations & Plugins​

Prometheus metrics​

Deployment​

Configuration Manual

Configuration keys and values

Configuration providers

Environment variables

Command-line arguments

Configuration parameters

Exchange conditional parameters

Exchange specific recommendations

Kraken Futures `KRAKENFTS`

Whitelist your IP addresses

Consul Configuration (optional)

Installation Manual

Service discovery

Running using an executable

Running on Docker

Running on Kubernetes Cluster

Add the Repository to Helm

Helm chart configuration

Install ems-api latest version

Install ems-api custom version

Verify installation

Inspect

List services

Connect to EMS API

Connect from within the cluster

Connect cluster to localhost

Management Guide

Installation

Upgrading

Versioning & Compatibility

Time synchronization

Limits

Integrations & Plugins

Prometheus metrics

Deployment