Flat Files S3 API - Documentation
General
This section will provide necessary information about the Flat Files S3 API
software and enumerate a number of features that it provides.
What is Flat Files S3 API?
Flat Files S3 API is a RESTful API that provides access to data stored in flat files. The API was designed to be compliant with AMAZON S3 giving the flexibiity to use existing infrastrucutre dedicate to use with AMAZON S3. Flat Files S3 API does not support all of the AMAZON S3 features since it was meant to be use only for listing and downloading files.
What is the flat file?
quotes
Column name | Description |
---|---|
id_site_coinapi | UUID of the site which received the quote. This useful information can be used to replay markets from the perspective of the algorithmic trading strategy. |
time_exchange | UTC timestamp of the trade provided by the exchange or estimation of it using the delay of the exchange API (if the time_exchange is equal to the time_coinapi, then it mean that the time_exchange is not available) |
time_coinapi | UTC timestamp of the trade when we first received it from the exchange in the specific site identified by the id_site_coinapi column. This useful information can be used to replay markets from the perspective of the algorithmic trading strategy. (Our clock is synchronized across data types and exchanges) |
ask_px | Best ask price |
ask_sx | Total amount of volume resting on best ask level in the base asset of the symbol |
bid_px | Best bid price |
bid_sx | Total amount of volume resting on best bid level in the base asset of the symbol |
trades
Column name | Description |
---|---|
time_exchange | UTC timestamp of the trade provided by the exchange or estimation of it using the delay of the exchange API (if the time_exchange is equal to the time_coinapi, then it mean that the time_exchange is not available) |
time_coinapi | UTC timestamp of the trade when we first received it from the exchange. This useful information can be used to replay markets from the perspective of the algorithmic trading strategy. (Our clock is synchronized across data types and exchanges) |
guid | Unique identifier of the trade provided by the CoinAPI |
price | Price of the transaction |
base_amount | Amount of base asset traded in the transaction. |
taker_side | Aggressor side of the transaction. Possible values are: BUY - Exchange has reported that transaction aggressor was buying SELL - Exchange has reported that transaction aggressor was selling BUY_ESTIMATED - Exchange has not reported transaction aggressor, we estimated that more likely it was buying SELL_ESTIMATED - Exchange has not reported transaction aggressor, we estimated that more likely it was selling UNKNOWN - Exchange has not reported transaction aggressor and we have not estimated who it was |
limitbook_snapshot_X
Column name | Description |
---|---|
time_exchange | UTC timestamp of the book provided by the exchange or estimation of it using the delay of the exchange API (if the time_exchange is equal to the time_coinapi, then it mean that the time_exchange is not available) |
time_coinapi | UTC timestamp of the book when we first received it from the exchange. This useful information can be used to replay markets from the perspective of the algorithmic trading strategy. (Our clock is synchronized across data types and exchanges) |
asks[0-X].price | Asks prices (incrementing from the spread) |
asks[0-X].size | Total amount of volume resting on ask levels in the base asset of the symbol |
bids[0-X].price | Bid prices (decrementing from the spread) |
bids[0-X].size | Total amount of volume resting on bid levels in the base asset of the symbol |
limitbook_full
Column name | Description |
---|---|
time_exchange | UTC timestamp of the update provided by the exchange or estimation of it using the delay of the exchange API (if the time_exchange is equal to the time_coinapi, then it mean that the time_exchange is not available) |
time_coinapi | UTC timestamp of the update when we first received it from the exchange. This useful information can be used to replay markets from the perspective of the algorithmic trading strategy. (Our clock is synchronized across data types and exchanges) |
is_buy | Is the update related to the bid side of the book? Possible values are 0 (it's ask - sell) or 1 (it's bid - buy). |
update_type | Type of the update. Possible values are: ADD - Add the order to the book with key (entry_px, order_id). SUB - Subtract the resting entry_sx volume from the book order identified by the key (entry_px, order_id). MATCH - Process the same as SUB, used to identify that subtraction was caused by the execution and not modification or self trade prevention algorithms. SET - Set the new entry_sx volume on the book order identified by the key (entry_px, order_id). DELETE - Delete the order identified by the key (entry_px, order_id) from the book. SNAPSHOT - Process the same as ADD, but before processing the first SNAPSHOT line when the last line seen was not a SNAPSHOT, then the whole book must be discarded/cleared. |
entry_px | Price identifying the book level. |
entry_sx | Volume associated with the specific update item. |
order_id | ID of the order if the format is Level 3 (order-by-order), empty if the format is Level 2 |
ohlcv_active_consolidated
Column name | Description |
---|---|
symbol_id | Symbol identifier of requested time series. More information about the symbol are provided by the CoinAPI REST API at the https://docs.coinapi.io/#list-all-symbols (opens in a new tab) |
time_period_start | Period starting UTC time (range left inclusive) |
time_period_end | Period ending UTC time (range right exclusive) |
time_open | UTC Time of first trade inside a period range |
time_close | UTC Time of last trade inside a period range |
px_open | First trade price inside a period range |
px_high | Highest traded price inside a period range |
px_low | Lowest traded price inside a period range |
px_close | Last trade price inside a period range |
sx_sum | Cumulative base amount traded inside a period range |
sx_cnt | Amount of trades executed inside a period range |
Flat Files - REST API
RESTful endpoint provides the widest range of data, based on HTTP protocol which works in Request-Reply scheme.
Implemented Standards:
Endpoints
Enviroment | Encryption | Value |
---|---|---|
Production | Yes | https://flatfiles.coinapi.io/ |
Production | No | http://flatfiles.coinapi.io/ |
HTTP Requests
Each HTTP request must contain the header Accept: application/xml
as all responses are in XML format.
Authentication
If you want to learn how to authenticate to this API, you can find detailed instructions and guidance in authentication section of this documentation.
Supported operation
This section describes available Flat Files S3 API operations.
List of Amazon S3 REST API operation that are supported by Flat Files S3 API.
List objects GET
Get a list of objects in the bucket. This operations corresponds to ListObjectsV2 (opens in a new tab) operation.
HTTP Request
GET /bucket/?
GET /bucket/?prefix={prefix}
URL Parameters
Parameter | Type | Description |
---|---|---|
prefix | string | path to desired file |
curl -X GET -H "Accept: application/xml" -H "Authorization: 73034021-THIS-IS-SAMPLE-KEY" 'http://flatfiles.coinapi.io/bucket/?prefix=trades/20230530/
The above command returns file content.
<ListBucketResult xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<ContentLength>0</ContentLength>
<HttpStatusCode>OK</HttpStatusCode>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>trades/20230530/5677816-BINANCEFTS_PERP_BTC_USDT.csv.gz</Key>
<LastModified>2023-05-31T00:25:20.22186Z</LastModified>
<Size>36135124</Size>
</Contents>
<Contents>
<Key>trades/20230530/5753223-BINANCEFTS_PERP_ETH_USDT.csv.gz</Key>
<LastModified>2023-05-31T00:25:05.6151012Z</LastModified>
<Size>24401397</Size>
</Contents>
<Contents>
<Key>trades/20230530/5753224-BINANCEFTS_PERP_BCH_USDT.csv.gz</Key>
<LastModified>2023-05-31T00:24:41.567665Z</LastModified>
<Size>1785210</Size>
</Contents>
<Contents>
<Key>trades/20230530/5753225-BINANCEFTS_PERP_XRP_USDT.csv.gz</Key>
<LastModified>2023-05-31T00:25:06.6029577Z</LastModified>
<Size>14217669</Size>
</Contents>
<Contents>
<Key>trades/20230530/5753226-BINANCEFTS_PERP_EOS_USDT.csv.gz</Key>
<LastModified>2023-05-31T00:25:01.1120369Z</LastModified>
<Size>1793453</Size>
</Contents>
<MaxKeys>0</MaxKeys>
<KeyCount>0</KeyCount>
</ListBucketResult>
Download object GET
Retrives desired object from given path. This operations corresponds to GetObject (opens in a new tab) operation.
HTTP Request
GET /bucket/{Key}
URL Parameters
Parameter | Type | Description |
---|---|---|
Key | string | Key of the object to get. |
HTTP Errors
Error message is returned in XML structured like this:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>NoSuchKey</Code>
<Message>The resource you requested does not exist</Message>
<Resource>/mybucket/myfoto.jpg</Resource>
<RequestId>4442587FB7D0A2F9</RequestId>
</Error>
This model is compliant with Amazon S3 REST Error Responses.
All HTTP requests with response status codes different to 200
must be considered as failed
and you should expect additional XML inside the body of the response with the error message encapsulated inside it as shown in the example.
We use the following error codes:
Error Code | Meaning |
---|---|
400 | Bad Request -- There is something wrong with your request |
401 | Unauthorized -- Your API key is wrong |
403 | Forbidden -- Your API key doesnt't have enough privileges to access this resource |
550 | No data -- You requested specific single item that we don't have at this moment. |
Compatible software
The list of software that can be used to retrive data from Flat Files API.
AWS SDK
Installation
pip install awscli
Configuration
- After installing the AWS CLI, open a terminal or command prompt.
- Run the following command to configure the AWS CLI:
aws configure
- You will be prompted to enter your AWS access key ID, secret access key, default region, and default output format. Provide the requested information accordingly.
AWS Access Key ID [****************Key]]: 73034021-THIS-IS-SAMPLE-KEY
AWS Secret Access Key [****************Key]]: coinapi
Default region name [None]: us-east-1
Default output format [None]:
Usage
Once the AWS CLI is installed and configured, you can use it to interact with the S3 API Coinapi using various commands. Here are some examples:
- List S3 Buckets:
aws --endpoint-url http://flatfiles.coinapi.io s3 ls s3://coinapi
PRE limitbook_full/
PRE metrics/
PRE quotes/
PRE trades/
PRE ohlcv_active_consolidated/
- Download a File from S3:
aws --endpoint-url http://flatfiles.coinapi.io s3 cp s3://coinapi/trades/20200301/98819-YOBIT_SPOT_XEM_BTC.csv.gz /path/to/local/destination
S3 Browser
The most important parameters:
Field name | Value |
---|---|
Access Key Id | 73034021-THIS-IS-SAMPLE-KEY |
Secret Key | coinapi |
- Download and Install S3 Browser
- Go to the official S3 Browser website (https://s3browser.com (opens in a new tab))
- Download the latest version of S3 Browser and follow the installation instructions to install it on your computer.
- Launch S3 Browser
- Open the S3 Browser application that you installed.
- Create a New S3 Connection
- Click on the "File" menu at the top-left corner of the S3 Browser window.
- Select "Add New Account" from the drop-down menu.
- Enter Connection Details
- Provide a name for your connection in the "Account Name" field (e.g., Coinapi).
- Enter your S3 API Coinapi access key and secret key credentials in the "Access Key" and "Secret Key" fields, respectively.
- Configure S3 Endpoint
- Enter the endpoint URL for the S3 API Coinapi in the "Endpoint URL" field.
- Refer to the Coinapi documentation or contact their support for the specific URL to use.
- Choose Connection Type
- Select either "HTTP" or "HTTPS" from the "Connection Type" drop-down menu based on Coinapi's security requirements.
- Test Connection
- Click the "Test" button to verify the connection.
- S3 Browser will attempt to connect to the specified endpoint using the provided credentials.
- Save Connection
- If the connection test is successful, click "OK" to save the connection details.
- Browse S3 Buckets
- In the left-hand sidebar of the S3 Browser window, find the newly created account.
- Expand the account to view the associated S3 buckets.