S3 API

Flat Files S3 API - Comprehensive Documentation

Introduction

The Flat Files S3 API is a powerful and flexible solution for accessing historical cryptocurrency market data. This document provides in-depth information about the API's features, usage, and integration capabilities.

General Overview

What is Flat Files S3 API?

Flat Files S3 API is a RESTful API designed to provide efficient access to cryptocurrency market data stored in flat files. The API is built to be compatible with Amazon S3, allowing users to leverage existing S3-compatible tools and infrastructure. While it doesn't support all Amazon S3 features, it focuses on core functionality for listing and downloading files, making it ideal for retrieving historical market data.

Key Features

1. S3 Compatibility: Seamless integration with existing S3 tools and libraries.
2. Efficient Data Retrieval: Optimized for listing and downloading large datasets.
3. Flexible Authentication: Supports multiple authentication methods for secure access.
4. Comprehensive Market Data: Access to various data types including trades, quotes, order books, and more.

API Specifications

Implemented Standards

The Flat Files S3 API adheres to the following HTTP standards:

• HTTP/1.0
• HTTP/1.1
• HTTP/2.0

This ensures broad compatibility and reliable performance across various client implementations.

Endpoints

The API is accessible through the following endpoints:

Environment	Encryption	Endpoint URL
Production	Yes	https://s3.flatfiles.coinapi.io/
Production	No	http://s3.flatfiles.coinapi.io/

We strongly recommend using the HTTPS endpoint for production environments to ensure data security.

HTTP Requests

For all requests to the Flat Files S3 API, you must include the following header:

Accept: application/xml

This header indicates that the client expects XML-formatted responses, which is the standard output format for this API.

Authentication

Secure access to the Flat Files S3 API is crucial. For detailed instructions on authentication methods, please refer to the authentication section of our documentation. This section covers various authentication techniques compatible with S3 clients, including:

• API Key usage
• AWS Signature Version 2 and 4
• Configuration examples for popular S3 clients and SDKs

Supported Operations

The Flat Files S3 API supports a subset of Amazon S3 REST API operations, tailored for efficient data retrieval. Below are the key operations available:

List Objects GET

This operation allows you to retrieve a list of objects (files) in the bucket. It corresponds to the ListObjectsV2 operation in Amazon S3.

Usage Notes

• This operation is optimized for listing files within the Flat Files S3 API context.
• While it doesn't require all parameters from the Amazon S3 ListObjectsV2 operation, including additional parameters should not affect the result.
• For best practices, stick to the request syntax presented in the HTTP Request section.

HTTP Request

1. To list all objects:

   GET /bucket/?

2. To list objects with a specific prefix:

   GET /bucket/?prefix={prefix}

URL Parameters

Parameter	Type	Description
prefix	string	Path prefix to filter the desired files

Example Request

shell

curl -X GET -H "Accept: application/xml" -H "Authorization: 73034021-THIS-IS-SAMPLE-KEY" 'http://s3.flatfiles.coinapi.io/bucket/?prefix=trades/20230530/

csharp

var client = new RestClient("https://s3.flatfiles.coinapi.io/bucket/?prefix=trades/20230530/");
var request = new RestRequest(Method.GET);
request.AddHeader("Authorization", "73034021-THIS-IS-SAMPLE-KEY");
IRestResponse response = client.Execute(request);

php

<?php
$request = new HttpRequest();
$request->setUrl('https://s3.flatfiles.coinapi.io/bucket/?prefix=trades/20230530/');
$request->setMethod(HTTP_METH_GET);
$request->setHeaders(array(
  'Authorization' => '73034021-THIS-IS-SAMPLE-KEY'
));
try {
  $response = $request->send();
  echo $response->getBody();
} catch (HttpException $ex) {
  echo $ex;
}
?>

python

import requests
url = 'https://s3.flatfiles.coinapi.io/bucket/?prefix=trades/20230530/'
headers = {'Authorization' : '73034021-THIS-IS-SAMPLE-KEY'}
response = requests.get(url, headers=headers)

js

const https = require('https');
var options = {
  "method": "GET",
  "hostname": "flatfiles.coinapi.io",
  "path": "/bucket/?prefix=trades/20230530/",
  "headers": {'Authorization': '73034021-THIS-IS-SAMPLE-KEY'}
};
var request = https.request(options, function (response) {
  var chunks = [];
  response.on("data", function (chunk) {
    chunks.push(chunk);
  });
});
request.end();

go

import (
  "gopkg.in/resty.v0"
)
func main() 
{
    resp, err := resty.R().
      SetHeader("Authorization", "73034021-THIS-IS-SAMPLE-KEY").
      Get("https://s3.flatfiles.coinapi.io/bucket/?prefix=trades/20230530/")
}

ruby

require 'uri'
require 'net/http'
url = URI("https://s3.flatfiles.coinapi.io/bucket/?prefix=trades/20230530/")
http = Net::HTTP.new(url.host, url.port)
request = Net::HTTP::Get.new(url)
request["Authorization"] = '73034021-THIS-IS-SAMPLE-KEY'
response = http.request(request)
puts response.read_body

java

OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
  .url("https://s3.flatfiles.coinapi.io/bucket/?prefix=trades/20230530/")
  .post(body)
  .addHeader("Authorization", "73034021-THIS-IS-SAMPLE-KEY")
  .build();
Response response = client.newCall(request).execute();

Example Response

<ListBucketResult xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <ContentLength>0</ContentLength>
    <HttpStatusCode>OK</HttpStatusCode>
    <IsTruncated>false</IsTruncated>
    <Contents>
        <Key>trades/20230530/5677816-BINANCEFTS_PERP_BTC_USDT.csv.gz</Key>
        <LastModified>2023-05-31T00:25:20.22186Z</LastModified>
        <Size>36135124</Size>
    </Contents>
    <!-- Additional <Contents> elements -->
    <MaxKeys>0</MaxKeys>
    <KeyCount>0</KeyCount>
</ListBucketResult>

Download Object GET

This operation allows you to retrieve a specific object (file) from the bucket. It corresponds to the GetObject operation in Amazon S3.

Usage Notes

• This operation is streamlined for the Flat Files S3 API and may not support all parameters from the Amazon S3 GetObject operation.
• Multi-part download operations are not supported.
• For optimal performance, adhere to the request syntax provided in the HTTP Request section.

HTTP Request

GET /bucket/{Key}

URL Parameters

Parameter	Type	Description
Key	string	The key (path) of the object to retrieve

Error Handling

The Flat Files S3 API uses standard HTTP status codes and provides detailed error messages in XML format, consistent with Amazon S3 REST Error Responses.

Error Response Format

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>ErrorCode</Code>
  <Message>Error message description</Message>
  <Resource>/bucket/object-key</Resource> 
  <RequestId>UniqueRequestIdentifier</RequestId>
</Error>

Common Error Codes

Error Code	HTTP Status	Meaning
400	Bad Request	The request was invalid or cannot be served
401	Unauthorized	Authentication failed or user doesn't have permissions for the requested operation
403	Forbidden	Access denied due to insufficient privileges
550	No Data	The requested item is not available

info

Best Practice: Implement robust error handling in your application. Store all error messages along with the corresponding request data for troubleshooting and analysis.

Compatible Software and Tools

The Flat Files S3 API is designed to work with a wide range of S3-compatible software and tools. Here are some popular options:

AWS CLI

The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services, including S3-compatible storage like our Flat Files S3 API.

Installation

python

pip install awscli

linux

sudo apt-get install awscli

macos

brew install awscli

windows

Download and install the AWS CLI from https://aws.amazon.com/cli/

Configuration

1. Open a terminal or command prompt.
2. Run:

   aws configure
3. Enter the following information when prompted:

   AWS Access Key ID: 73034021-THIS-IS-SAMPLE-KEY
   AWS Secret Access Key: coinapi
   Default region name: us-east-1
   Default output format: [Leave blank]

Usage Examples

1. List buckets:

   aws --endpoint-url http://s3.flatfiles.coinapi.io s3 ls s3://coinapi

2. Download a file:

   aws --endpoint-url http://s3.flatfiles.coinapi.io s3 cp s3://coinapi/trades/20200301/98819-YOBIT_SPOT_XEM_BTC.csv.gz /local/path/

S3 Browser

S3 Browser is a popular Windows client for Amazon S3 and compatible services.

Configuration Steps

1. Download and install S3 Browser from https://s3browser.com
2. Launch S3 Browser
3. Add a new account:
   • Click "File" > "Add New Account"
   • Enter account details:
     • Account Name: CoinAPI Flat Files
     • Access Key ID: 73034021-THIS-IS-SAMPLE-KEY
     • Secret Access Key: coinapi
     • REST Endpoint: http://s3.flatfiles.coinapi.io
   • Choose HTTP or HTTPS for the connection type
   • Test the connection
   • Save the account settings
4. Browse the S3 buckets in the left sidebar

Best Practices and Optimization

To make the most of the Flat Files S3 API, consider the following best practices:

1. Use efficient querying: Utilize the prefix parameter when listing objects to narrow down results.
2. Implement robust error handling: Always check for and appropriately handle error responses.
3. Use HTTPS for production: Ensure data security by using the HTTPS endpoint in production environments.
4. Optimize data retrieval: Download only the data you need, using date ranges and specific symbol filters when available.
5. Cache frequently accessed data: Implement local caching to reduce API calls for repetitive queries.
6. Monitor your usage: Keep track of your API usage to stay within rate limits and optimize costs.

Conclusion

The Flat Files S3 API provides a powerful and flexible way to access historical cryptocurrency market data. By leveraging S3 compatibility, it allows for easy integration with existing tools and workflows while offering optimized performance for large-scale data retrieval.

For further assistance, feature requests, or to report issues, please contact our support team.

Was this section helpful?

• 1
• 2
• 3
• 4
• 5