Skip to main content

URL Table Engine

Queries data to/from a remote HTTP/HTTPS server. This engine is similar to the File engine.

Syntax: URL(URL [,Format] [,CompressionMethod])

  • The URL parameter must conform to the structure of a Uniform Resource Locator. The specified URL must point to a server that uses HTTP or HTTPS. This does not require any additional headers for getting a response from the server.

  • The Format must be one that ClickHouse can use in SELECT queries and, if necessary, in INSERTs. For the full list of supported formats, see Formats.

  • CompressionMethod indicates that whether the HTTP body should be compressed. If the compression is enabled, the HTTP packets sent by the URL engine contain 'Content-Encoding' header to indicate which compression method is used.

To enable compression, please first make sure the remote HTTP endpoint indicated by the URL parameter supports corresponding compression algorithm.

The supported CompressionMethod should be one of following:

  • gzip or gz
  • deflate
  • brotli or br
  • lzma or xz
  • zstd or zst
  • lz4
  • bz2
  • snappy
  • none

Usage​

INSERT and SELECT queries are transformed to POST and GET requests, respectively. For processing POST requests, the remote server must support Chunked transfer encoding.

You can limit the maximum number of HTTP GET redirect hops using the max_http_get_redirects setting.

Example​

1. Create a url_engine_table table on the server :

CREATE TABLE url_engine_table (word String, value UInt64)
ENGINE=URL('http://127.0.0.1:12345/', CSV)

2. Create a basic HTTP server using the standard Python 3 tools and start it:

from http.server import BaseHTTPRequestHandler, HTTPServer

class CSVHTTPServer(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/csv')
        self.end_headers()

        self.wfile.write(bytes('Hello,1\nWorld,2\n', "utf-8"))

if __name__ == "__main__":
    server_address = ('127.0.0.1', 12345)
    HTTPServer(server_address, CSVHTTPServer).serve_forever()
$ python3 server.py

3. Request data:

SELECT * FROM url_engine_table
┌─word──┬─value─┐
│ Hello │     1 │
│ World │     2 │
└───────┴───────┘

Details of Implementation​

  • Reads and writes can be parallel
  • Not supported:
    • ALTER and SELECT...SAMPLE operations.
    • Indexes.
    • Replication.