S3 API

Flat Files S3 API - Documentation

General

This section will provide necessary information about the Flat Files S3 API software and enumerate a number of features that it provides.

What is Flat Files S3 API?

Flat Files S3 API is a RESTful API that provides access to data stored in flat files. The API was designed to be compliant with AMAZON S3 giving the flexibiity to use existing infrastrucutre dedicate to use with AMAZON S3. Flat Files S3 API does not support all of the AMAZON S3 features since it was meant to be use only for listing and downloading files.

What is the flat file?

quotes

Column name	Description
id_site_coinapi	UUID of the site which received the quote. This useful information can be used to replay markets from the perspective of the algorithmic trading strategy.
time_exchange	UTC timestamp of the trade provided by the exchange or estimation of it using the delay of the exchange API (if the time_exchange is equal to the time_coinapi, then it mean that the time_exchange is not available)
time_coinapi	UTC timestamp of the trade when we first received it from the exchange in the specific site identified by the id_site_coinapi column. This useful information can be used to replay markets from the perspective of the algorithmic trading strategy. (Our clock is synchronized across data types and exchanges)
ask_px	Best ask price
ask_sx	Total amount of volume resting on best ask level in the base asset of the symbol
bid_px	Best bid price
bid_sx	Total amount of volume resting on best bid level in the base asset of the symbol

trades

Column name	Description
time_exchange	UTC timestamp of the trade provided by the exchange or estimation of it using the delay of the exchange API (if the time_exchange is equal to the time_coinapi, then it mean that the time_exchange is not available)
time_coinapi	UTC timestamp of the trade when we first received it from the exchange. This useful information can be used to replay markets from the perspective of the algorithmic trading strategy. (Our clock is synchronized across data types and exchanges)
guid	Unique identifier of the trade provided by the CoinAPI
price	Price of the transaction
base_amount	Amount of base asset traded in the transaction.
taker_side	Aggressor side of the transaction. Possible values are: BUY - Exchange has reported that transaction aggressor was buying SELL - Exchange has reported that transaction aggressor was selling BUY_ESTIMATED - Exchange has not reported transaction aggressor, we estimated that more likely it was buying SELL_ESTIMATED - Exchange has not reported transaction aggressor, we estimated that more likely it was selling UNKNOWN - Exchange has not reported transaction aggressor and we have not estimated who it was

limitbook_snapshot_X

Column name	Description
time_exchange	UTC timestamp of the book provided by the exchange or estimation of it using the delay of the exchange API (if the time_exchange is equal to the time_coinapi, then it mean that the time_exchange is not available)
time_coinapi	UTC timestamp of the book when we first received it from the exchange. This useful information can be used to replay markets from the perspective of the algorithmic trading strategy. (Our clock is synchronized across data types and exchanges)
asks[0-X].price	Asks prices (incrementing from the spread)
asks[0-X].size	Total amount of volume resting on ask levels in the base asset of the symbol
bids[0-X].price	Bid prices (decrementing from the spread)
bids[0-X].size	Total amount of volume resting on bid levels in the base asset of the symbol

limitbook_full

Column name	Description
time_exchange	UTC timestamp of the update provided by the exchange or estimation of it using the delay of the exchange API (if the time_exchange is equal to the time_coinapi, then it mean that the time_exchange is not available)
time_coinapi	UTC timestamp of the update when we first received it from the exchange. This useful information can be used to replay markets from the perspective of the algorithmic trading strategy. (Our clock is synchronized across data types and exchanges)
is_buy	Is the update related to the bid side of the book? Possible values are 0 (it's ask - sell) or 1 (it's bid - buy).
update_type	Type of the update. Possible values are: ADD - Add the order to the book with key (entry_px, order_id). SUB - Subtract the resting entry_sx volume from the book order identified by the key (entry_px, order_id). MATCH - Process the same as SUB, used to identify that subtraction was caused by the execution and not modification or self trade prevention algorithms. SET - Set the new entry_sx volume on the book order identified by the key (entry_px, order_id). DELETE - Delete the order identified by the key (entry_px, order_id) from the book. SNAPSHOT - Process the same as ADD, but before processing the first SNAPSHOT line when the last line seen was not a SNAPSHOT, then the whole book must be discarded/cleared.
entry_px	Price identifying the book level.
entry_sx	Volume associated with the specific update item.
order_id	ID of the order if the format is Level 3 (order-by-order), empty if the format is Level 2

ohlcv_active_consolidated

Column name	Description
symbol_id	Symbol identifier of requested time series. More information about the symbol are provided by the CoinAPI REST API at the https://docs.coinapi.io/#list-all-symbols
time_period_start	Period starting UTC time (range left inclusive)
time_period_end	Period ending UTC time (range right exclusive)
time_open	UTC Time of first trade inside a period range
time_close	UTC Time of last trade inside a period range
px_open	First trade price inside a period range
px_high	Highest traded price inside a period range
px_low	Lowest traded price inside a period range
px_close	Last trade price inside a period range
sx_sum	Cumulative base amount traded inside a period range
sx_cnt	Amount of trades executed inside a period range

Flat Files - REST API

RESTful endpoint provides the widest range of data, based on HTTP protocol which works in Request-Reply scheme.

Implemented Standards:

• HTTP1.0
• HTTP1.1
• HTTP2.0

Endpoints

Enviroment	Encryption	Value
Production	Yes	https://flatfiles.coinapi.io/
Production	No	http://flatfiles.coinapi.io/

HTTP Requests

Each HTTP request must contain the header Accept: application/xml as all responses are in XML format.

Authentication

If you want to learn how to authenticate to this API, you can find detailed instructions and guidance in authentication section of this documentation.

Supported operation

This section describes available Flat Files S3 API operations.

List of Amazon S3 REST API operation that are supported by Flat Files S3 API.

List objects GET

Get a list of objects in the bucket. This operations corresponds to ListObjectsV2 operation.

info

This operation does not require all of the parameters presented in ListObjectsV2 documentation. Specyfing them in request should not affect request result, nevertheless we advise to stick to request syntax presentd in HTTP Request section whenever possible.

HTTP Request

1. GET /bucket/?
2. GET /bucket/?prefix={prefix}

URL Parameters

Parameter	Type	Description
prefix	string	path to desired file

shell

curl -X GET -H "Accept: application/xml" -H "Authorization: 73034021-THIS-IS-SAMPLE-KEY" 'http://flatfiles.coinapi.io/bucket/?prefix=trades/20230530/

csharp

var client = new RestClient("https://flatfiles.coinapi.io/bucket/?prefix=trades/20230530/");
var request = new RestRequest(Method.GET);
request.AddHeader("Authorization", "73034021-THIS-IS-SAMPLE-KEY");
IRestResponse response = client.Execute(request);

php

<?php
$request = new HttpRequest();
$request->setUrl('https://flatfiles.coinapi.io/bucket/?prefix=trades/20230530/');
$request->setMethod(HTTP_METH_GET);
$request->setHeaders(array(
  'Authorization' => '73034021-THIS-IS-SAMPLE-KEY'
));

try {
  $response = $request->send();
  echo $response->getBody();
} catch (HttpException $ex) {
  echo $ex;
}
?>

python

import requests
url = 'https://flatfiles.coinapi.io/bucket/?prefix=trades/20230530/'
headers = {'Authorization' : '73034021-THIS-IS-SAMPLE-KEY'}
response = requests.get(url, headers=headers)

js

const https = require('https');

var options = {
  "method": "GET",
  "hostname": "flatfiles.coinapi.io",
  "path": "/bucket/?prefix=trades/20230530/",
  "headers": {'Authorization': '73034021-THIS-IS-SAMPLE-KEY'}
};

var request = https.request(options, function (response) {
  var chunks = [];

  response.on("data", function (chunk) {
    chunks.push(chunk);
  });

});

request.end();

go

import (
  "gopkg.in/resty.v0"
)

func main() 
{
    resp, err := resty.R().
      SetHeader("Authorization", "73034021-THIS-IS-SAMPLE-KEY").
      Get("https://flatfiles.coinapi.io/bucket/?prefix=trades/20230530/")
}

ruby

require 'uri'
require 'net/http'

url = URI("https://flatfiles.coinapi.io/bucket/?prefix=trades/20230530/")
http = Net::HTTP.new(url.host, url.port)
request = Net::HTTP::Get.new(url)
request["Authorization"] = '73034021-THIS-IS-SAMPLE-KEY'
response = http.request(request)
puts response.read_body

java

OkHttpClient client = new OkHttpClient();

Request request = new Request.Builder()
  .url("https://flatfiles.coinapi.io/bucket/?prefix=trades/20230530/")
  .post(body)
  .addHeader("Authorization", "73034021-THIS-IS-SAMPLE-KEY")
  .build();

Response response = client.newCall(request).execute();

The above command returns file content.

<ListBucketResult xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <ContentLength>0</ContentLength>
    <HttpStatusCode>OK</HttpStatusCode>
    <IsTruncated>false</IsTruncated>
    <Contents>
        <Key>trades/20230530/5677816-BINANCEFTS_PERP_BTC_USDT.csv.gz</Key>
        <LastModified>2023-05-31T00:25:20.22186Z</LastModified>
        <Size>36135124</Size>
    </Contents>
    <Contents>
        <Key>trades/20230530/5753223-BINANCEFTS_PERP_ETH_USDT.csv.gz</Key>
        <LastModified>2023-05-31T00:25:05.6151012Z</LastModified>
        <Size>24401397</Size>
    </Contents>
    <Contents>
        <Key>trades/20230530/5753224-BINANCEFTS_PERP_BCH_USDT.csv.gz</Key>
        <LastModified>2023-05-31T00:24:41.567665Z</LastModified>
        <Size>1785210</Size>
    </Contents>
    <Contents>
        <Key>trades/20230530/5753225-BINANCEFTS_PERP_XRP_USDT.csv.gz</Key>
        <LastModified>2023-05-31T00:25:06.6029577Z</LastModified>
        <Size>14217669</Size>
    </Contents>
    <Contents>
        <Key>trades/20230530/5753226-BINANCEFTS_PERP_EOS_USDT.csv.gz</Key>
        <LastModified>2023-05-31T00:25:01.1120369Z</LastModified>
        <Size>1793453</Size>
    </Contents>
    <MaxKeys>0</MaxKeys>
    <KeyCount>0</KeyCount>
</ListBucketResult>

Download object GET

Retrives desired object from given path. This operations corresponds to GetObject operation.

caution

This operation does not require all of the parameters presented in GetObject documentation. Specyfing them in request should not affect request result, nevertheless we advise to stick to request syntax presentd in HTTP Request section whenever possible.

caution

Download object does not support Multi-part download operaion.

HTTP Request

1. GET /bucket/{Key}

URL Parameters

Parameter	Type	Description
Key	string	Key of the object to get.

HTTP Errors

Error message is returned in XML structured like this:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>NoSuchKey</Code>
  <Message>The resource you requested does not exist</Message>
  <Resource>/mybucket/myfoto.jpg</Resource> 
  <RequestId>4442587FB7D0A2F9</RequestId>
</Error>

This model is compliant with Amazon S3 REST Error Responses.

All HTTP requests with response status codes different to 200 must be considered as failed and you should expect additional XML inside the body of the response with the error message encapsulated inside it as shown in the example. We use the following error codes:

Error Code	Meaning
400	Bad Request -- There is something wrong with your request
401	Unauthorized -- Your API key is wrong
403	Forbidden -- Your API key doesnt't have enough privileges to access this resource
550	No data -- You requested specific single item that we don't have at this moment.

info

Good practice is to store all error messages somewhere along with request data for further manual review.

Compatible software

The list of software that can be used to retrive data from Flat Files API.

AWS SDK

Installation

python

pip install awscli

linux

sudo apt-get install awscli

macos

brew install awscli

windows

Download the AWS CLI installer for Windows from the official AWS Command Line Interface website (https://aws.amazon.com/cli/).
Follow the installation instructions provided on the website.

Configuration

1. After installing the AWS CLI, open a terminal or command prompt.
2. Run the following command to configure the AWS CLI:

aws configure

3. You will be prompted to enter your AWS access key ID, secret access key, default region, and default output format. Provide the requested information accordingly.

AWS Access Key ID [****************Key]]: 73034021-THIS-IS-SAMPLE-KEY
AWS Secret Access Key [****************Key]]: coinapi
Default region name [None]: us-east-1
Default output format [None]:

Usage

Once the AWS CLI is installed and configured, you can use it to interact with the S3 API Coinapi using various commands. Here are some examples:

1. List S3 Buckets:

aws --endpoint-url http://flatfiles.coinapi.io s3 ls s3://coinapi
    PRE limitbook_full/
    PRE metrics/
    PRE quotes/
    PRE trades/
    PRE ohlcv_active_consolidated/

2. Download a File from S3:

aws --endpoint-url http://flatfiles.coinapi.io s3 cp s3://coinapi/trades/20200301/98819-YOBIT_SPOT_XEM_BTC.csv.gz /path/to/local/destination

S3 Browser

The most important parameters:

Field name	Value
Access Key Id	73034021-THIS-IS-SAMPLE-KEY
Secret Key	coinapi

1. Download and Install S3 Browser

• Go to the official S3 Browser website (https://s3browser.com)
• Download the latest version of S3 Browser and follow the installation instructions to install it on your computer.

2. Launch S3 Browser

• Open the S3 Browser application that you installed.

3. Create a New S3 Connection

• Click on the "File" menu at the top-left corner of the S3 Browser window.
• Select "Add New Account" from the drop-down menu.

4. Enter Connection Details

• Provide a name for your connection in the "Account Name" field (e.g., Coinapi).
• Enter your S3 API Coinapi access key and secret key credentials in the "Access Key" and "Secret Key" fields, respectively.

5. Configure S3 Endpoint

• Enter the endpoint URL for the S3 API Coinapi in the "Endpoint URL" field.
• Refer to the Coinapi documentation or contact their support for the specific URL to use.

6. Choose Connection Type

• Select either "HTTP" or "HTTPS" from the "Connection Type" drop-down menu based on Coinapi's security requirements.

7. Test Connection

• Click the "Test" button to verify the connection.
• S3 Browser will attempt to connect to the specified endpoint using the provided credentials.

8. Save Connection

• If the connection test is successful, click "OK" to save the connection details.

9. Browse S3 Buckets

• In the left-hand sidebar of the S3 Browser window, find the newly created account.
• Expand the account to view the associated S3 buckets.

Was this section helpful?

• 1
• 2
• 3
• 4
• 5