Running SQL Concurrently Across Three Remote DuckDB Servers with Quack

, the good folks at DuckDB released a database communication protocol called Quack. Its main aim was to allow DuckDB databases held on different servers to communicate with each other over HTTP in a client/server arrangement and allow them to read and write each others data. In other words, using the Quack protocol, DuckDB sitting on server A could now query or write to a DuckDB database on remote server B. This might sound a bit like distributed data processing, but it’s not and the DuckDB team was at pains to emphasize that Quack does not facilitate distributed query processing. Even so, I was intrigued and could see many uses for Quack. In particular, I was interested to find out if it was possible to fire off parallel SQL statements on each server and have the outputs of each query gathered and made available for use or display on a coordinating server. Note that this is a different proposition from simply joining tables across different servers in a single SQL statement. DuckDB can do that by attaching a database from server B to server A, for example, then running SQL on server A that can read data on server B as if it were local. So, to investigate the concurrent reads and writes that Quack should enable, I created a GitHub repo called cluster-duck, and no, it’s not a distributed DuckDB cluster. It’s just a comic take on the common, rude expression you probably already know. But it will let you do concurrent reads and writes on remote DuckDB databases. Note: Before proceeding I’d like to state that I have no affiliation or commercial asscociation with any of the products, systems or their creators that are mentioned in this article. The set up To test Quack out, I set up 3 AWS EC2 servers via a CloudFormation stack. Each server holds one DuckDB database with one of the servers also acting as a coordinator node. You can find the CF stack on my GitHub repo. From AWS CloudShell (or locally if you have the AWS CLI installed), with the repository checked out, deploy the stack with: aws cloudformation deploy \ --region us-east-2 \ --stack-name cluster-duck-test-v2 \ --template-file python-reference/infra/aws/cluster-duck-3-node.yaml \ --capabilities CAPABILITY_NAMED_IAM For all three EC2 servers, the following was installed, Amazon Linux 2023 ARM64. A 512 MB swap file. Python 3.12 and pip. A Python virtual environment at: /opt/cluster-duck-venv duckdb==1.5.5 boto3 DuckDB 1.5.5 ARM64 CLI at: /usr/local/bin/duckdb The official DuckDB Quack extension, loaded by the worker process. The Quack server program at: /opt/cluster-duck/quack_server.py The worker database at: /var/lib/cluster-duck/worker.duckdb A systemd service named: cluster-duck-quack.service An automatic shutdown timer, four hours by default. Quack listens on port 9494 by default. Its authentication token is retrieved from an encrypted SSM Parameter Store parameter. The Python source is copied onto all three servers because they share the same CloudFormation launch template. However, the coordinator is only made executable as a command on one server – usually worker 1. These files are installed on every server: /opt/cluster-duck/quack_server.py /opt/cluster-duck/seed_related_data.py /opt/cluster-duck/related_cluster_sql.py /opt/cluster-duck/cluster_duck/sql_api.py The coordination server (Worker 1) additionally gets these command launchers: /usr/local/bin/cluster-duck /usr/local/bin/cluster-duck-sql When you run cluster-duck-sql on the coordinator, this eventually starts: /opt/cluster-duck-venv/bin/python /opt/cluster-duck/related_cluster_sql.py The Python source is compressed and embedded directly inside the CloudFormation template as a Base64-encoded archive. During EC2 bootstrap, the user-data script: Decodes the embedded archive. Creates /opt/cluster-duck. Extracts the Python files into that directory. Creates the command launchers on worker 1. Starts the Quack service on every worker. Everything needed is contained in the CloudFormation file. How the coordination works Quack carries each SQL statement to the chosen DuckDB server and returns its result. The part that coordinates the three calls is the Python code running on Worker 1. First, every — query or — query-file argument is validated as a single SQL statement and turned into a QueryFragment. The fragments are labelled in the order in which they were supplied: fragments.append( QueryFragment(worker_id, f"query-{index}", sql) ) The coordinator then creates one thread per fragment and a barrier with the same number of participants: barrier = threading.Barrier(len(fragments)) epoch = time.perf_counter() def run_fragment(fragment): barrier.wait() started = time.perf_counter() raw_result = self.executor(fragment.worker_id, fragment.sql) finished = time.perf_counter() return { "start_offset_ms": (started - epoch) * 1000, "duration_ms": (finished - started) * 1000, "result": raw_result, } with ThreadPoolExecutor(max_workers=len(fragments)) as pool: futures = { pool.submit(run_fragment, fragment): fragment for fragment in fragments } The barrier holds the threads until every fragment is ready, then releases them together. They will not start on precisely the same CPU cycle because normal operating-system scheduling still applies, which is why the output includes a measured start spread. Each fragment gets its own local DuckDB client connection on the coordinator. That connection loads Quack, attaches one remote worker and sends the statement through remote.query(): with duckdb.connect() as connection: connection.execute("INSTALL quack") connection.execute("LOAD quack") connection.execute( f"ATTACH {endpoint} AS remote " f"(TYPE quack, TOKEN {token}, DISABLE_SSL true)" ) cursor = connection.execute( f"SELECT * FROM remote.query({sql_string(sql)})" ) columns = tuple(description[0] for description in cursor.description) return FragmentResult(columns, cursor.fetchall()) The call to fetchall() materialises each result on Worker 1. The coordinator waits for all the futures, records when each one started and how long it took, and then presents the separate results in one output. Quack is doing the remote execution and transport; the fragment construction, simultaneous release, timing and result collection are all being done by Python. Creating our test data Each of the three servers has a different DuckDB database as follows. Worker Database file Generated table --------------------------------------------------------------- Worker 1 /var/lib/cluster-duck/worker.duckdb sales Worker 2 /var/lib/cluster-duck/worker.duckdb customers Worker 3 /var/lib/cluster-duck/worker.duckdb products Each generated table contains an appropriate synthetic data set of ten million records. Here are the first 5 records of each to give you a better idea what’s in them. query-1 (worker-1) - select * from sales limit 5 sale_id customer_id product_id quantity sales_channel payment_method sale_status catalogue_price sold_unit_price discount_pct sale_date ------- ----------- ---------- -------- ------------- -------------- ----------- --------------- --------------- ------------ ---------- 1 7,920 104,730 2 store bank_transfer shipped 1,052.3 999.69 0.05 2024-01-02 2 15,839 209,459 3 marketplace wallet processing 99.59 89.63 0.1 2024-01-03 3 23,758 314,188 4 telephone invoice returned 1,146.88 974.85 0.15 2024-01-04 4 31,677 418,917 5 online card cancelled 194.17 194.17 0 2024-01-05 5 39,596 523,646 1 store bank_transfer completed 1,241.46 1,179.39 0.05 2024-01-06 query-2 (worker-2) - select * from customers limit 5 customer_id customer_code country segment membership_tier is_active credit_limit joined_date last_seen_at ----------- --------------- ------- -------------- --------------- --------- ------------ ----------- ------------------- 1 CUST-0000000001 US small_business silver True 250.10 2015-01-02 2025-01-01 00:00:01 2 CUST-0000000002 DE enterprise gold True 250.20 2015-01-03 2025-01-01 00:00:02 3 CUST-0000000003 FR public_sector standard True 250.30 2015-01-04 2025-01-01 00:00:03 4 CUST-0000000004 CA consumer silver True 250.40 2015-01-05 2025-01-01 00:00:04 5 CUST-0000000005 AU small_business gold True 250.50 2015-01-06 2025-01-01 00:00:05 query-3 (worker-3) - select * from products limit 5 product_id sku category brand supplier_region catalogue_price stock_quantity discontinued introduced_date ---------- -------------- -------- ------- --------------- --------------- -------------- ------------ --------------- 1 SKU-0000000001 home Bramble EU 5.01 13 False 2020-01-02 2 SKU-0000000002 garden Cobalt US 5.02 26 False 2020-01-03 3 SKU-0000000003 sports Dove APAC 5.03 39 False 2020-01-04 4 SKU-0000000004 clothing Elm UK 5.04 52 False 2020-01-05 5 SKU-0000000005 food Aster EU 5.05 65 False 2020-01-06 The data is generated directly inside each DuckDB database during the first EC2 bootstrap. It isn’t uploaded from your computer or copied between servers. Each server follows this sequence. 1. Determine which worker it is CloudFormation gives every EC2 instance a WorkerIndex tag: Worker 1 → WorkerIndex=1 Worker 2 → WorkerIndex=2 Worker 3 → WorkerIndex=3 The bootstrap script reads that tag through the EC2 Instance Metadata Service: WORKER_INDEX=$(curl -fsS \ -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \ http://169.254.169.254/latest/meta-data/tags/instance/WorkerIndex) 2. Run the data-generation program CloudFormation installs this program on every server: /opt/cluster-duck/seed_related_data.py It then runs: /opt/cluster-duck-venv/bin/python /opt/cluster-duck/seed_related_data.py \ --worker "$WORKER_INDEX" \ --rows 10000000 The row count comes from the CloudFormation RowCount parameter, which defaults to 10 million. 3. Open the worker’s DuckDB file The program opens: /var/lib/cluster-duck/worker.duckdb which contains this code. with duckdb.connect(str(args.database)) as connection: connection.execute( CREATE_SQL[args.worker], {"row_count": args.rows}, ) Every server uses the same database filename, but it’s a different file on a different EC2 instance. Accessing your coordinator terminal window We want to run some demos, and for that you need to be able to access the CLI terminal of your coordinator EC2 server. To do that open the AWS console and go to the EC2 console. You’ll see a screen like this. Click the Instance ID that corresponds to your coordinator instance. On the next screen there will be a Connect button towards the top right corner. Click that. You’ll see this screen Make sure you’ve selected the SSM Session Manager radio button, then click the Connect button at the bottom right of the screen. That should give you access to the CLI terminal window like this, Examples In the following examples, to make things as clear as possible I use the raw SQL text in the code snippets, however it’s also possible to store the SQL in separate files and use those files as input. For example, [root@ip-10-42-0-10 ~]# cluster-duck-sql \ --query-file "worker-1=/root/cluster-duck-sql/sales.sql" \ --query-file "worker-2=/root/cluster-duck-sql/customers.sql" \ --query-file "worker-3=/root/cluster-duck-sql/products.sql" 1. Running some simple SQL statements In the terminal CLI, type in the following code. sh-5.2$ sudo -i [root@ip-10-42-0-10 ~]# cluster-duck-sql \ --query "worker-1=SELECT sale_status, COUNT(*) FROM sales GROUP BY sale_status ORDER BY sale_status" \ --query "worker-2=SELECT country, COUNT(*) FROM customers GROUP BY country ORDER BY country" \ --query "worker-3=SELECT category, COUNT(*) FROM products GROUP BY category ORDER BY category" # output # Concurrent remote queries worker table start_offset_ms duration_seconds -------- ------- --------------- ---------------- worker-1 query-1 0.996 0.572 worker-2 query-2 5.514 0.572 worker-3 query-3 0.738 0.54 Start spread: 4.776 ms query-1 (worker-1) sale_status count_star() ----------- ------------ cancelled 2,000,000 completed 2,000,000 processing 2,000,000 returned 2,000,000 shipped 2,000,000 query-2 (worker-2) country count_star() ------- ------------ AU 1,666,666 CA 1,666,667 DE 1,666,667 FR 1,666,667 UK 1,666,666 US 1,666,667 query-3 (worker-3) category count_star() ----------- ------------ clothing 1,666,667 electronics 1,666,666 food 1,666,666 garden 1,666,667 home 1,666,667 sports 1,666,667 2. Some complex SQL (I’ve cut out some of the output to save space) [root@ip-10-42-0-10 ~]# time cluster-duck-sql \ --query "worker-1=WITH daily AS ( SELECT sale_date, sales_channel, payment_method, sale_status, COUNT(*) AS transaction_count, SUM(quantity) AS units, SUM(quantity * sold_unit_price) AS revenue, AVG(discount_pct) AS average_discount, QUANTILE_CONT(sold_unit_price, 0.50) AS median_price, QUANTILE_CONT(sold_unit_price, 0.95) AS p95_price FROM sales GROUP BY ALL ), analysed AS ( SELECT *, SUM(revenue) OVER ( PARTITION BY sales_channel ORDER BY sale_date ROWS BETWEEN 29 PRECEDING AND CURRENT ROW ) AS rolling_30_row_revenue, RANK() OVER ( PARTITION BY sale_date ORDER BY revenue DESC ) AS daily_revenue_rank FROM daily ) SELECT * FROM analysed WHERE daily_revenue_rank <= 3 ORDER BY sale_date DESC, daily_revenue_rank LIMIT 100" \ --query "worker-2=WITH customer_groups AS ( SELECT country, segment, membership_tier, is_active, YEAR(joined_date) AS joined_year, CASE WHEN credit_limit < 2500 THEN 'under_2500' WHEN credit_limit < 5000 THEN '2500_to_4999' WHEN credit_limit < 7500 THEN '5000_to_7499' ELSE '7500_plus' END AS credit_band, COUNT(*) AS customer_count, AVG(credit_limit) AS average_credit_limit, STDDEV_POP(credit_limit) AS credit_limit_stddev, QUANTILE_CONT(credit_limit, 0.50) AS median_credit_limit, QUANTILE_CONT(credit_limit, 0.95) AS p95_credit_limit, MIN(joined_date) AS first_joined, MAX(last_seen_at) AS most_recent_activity FROM customers GROUP BY ALL ), ranked AS ( SELECT *, SUM(customer_count) OVER ( PARTITION BY country ) AS country_total, RANK() OVER ( PARTITION BY country ORDER BY customer_count DESC ) AS group_rank FROM customer_groups ) SELECT *, ROUND(100.0 * customer_count / country_total, 2) AS percentage_of_country FROM ranked WHERE group_rank <= 10 ORDER BY country, group_rank LIMIT 100" \ --query "worker-3=WITH inventory_groups AS ( SELECT category, brand, supplier_region, discontinued, YEAR(introduced_date) AS introduced_year, COUNT(*) AS product_count, SUM(stock_quantity) AS stock_units, SUM(stock_quantity * catalogue_price) AS inventory_value, AVG(catalogue_price) AS average_price, STDDEV_POP(catalogue_price) AS price_stddev, QUANTILE_CONT(catalogue_price, 0.50) AS median_price, QUANTILE_CONT(catalogue_price, 0.95) AS p95_price FROM products GROUP BY ALL ), ranked AS ( SELECT *, SUM(inventory_value) OVER ( PARTITION BY category ) AS category_inventory_value, RANK() OVER ( PARTITION BY category ORDER BY inventory_value DESC ) AS inventory_rank FROM inventory_groups ) SELECT *, ROUND( 100.0 * inventory_value / category_inventory_value, 2 ) AS percentage_of_category_value FROM ranked WHERE inventory_rank <= 10 ORDER BY category, inventory_rank LIMIT 100" # # Output Concurrent remote queries worker table start_offset_ms duration_seconds -------- ------- --------------- ---------------- worker-1 query-1 1.673 15.491 worker-2 query-2 1.84 4.045 worker-3 query-3 1.42 4.045 Start spread: 0.420 ms query-1 (worker-1) sale_date sales_channel payment_method sale_status transaction_count units revenue average_discount median_price p95_price rolling_30_row_revenue daily_revenue_rank ---------- ------------- -------------- ----------- ----------------- ------ ------------- ---------------- ------------ --------- ---------------------- ------------------ 2025-12-30 store bank_transfer cancelled 6,849 34,245 32,722,337.65 0.05 955.72 1,810.568 588,590,148.25 1 ... ... ... 2025-11-11 marketplace wallet completed 6,849 6,849 6,199,619.04 0.1 905.5 1,714.096 557,458,798.88 2 query-2 (worker-2) country segment membership_tier is_active joined_year credit_band customer_count average_credit_limit credit_limit_stddev median_credit_limit p95_credit_limit first_joined most_recent_activity country_total group_rank percentage_of_country ------- -------------- --------------- --------- ----------- ----------- -------------- -------------------- ------------------- ------------------- ---------------- ------------ -------------------- ------------- ---------- --------------------- AU small_business gold True 2,017 7500_plus 21,659 8,874.317 794.81 8878.10 10115.70 2017-01-01 2025-04-26 17:20:41 1,666,666 1 1.3 AU public_sector gold True 2,017 7500_plus 21,649 8,873.263 794.868 8876.70 10114.30 2017-01-01 2025-04-26 17:20:35 1,666,666 2 1.3 ... ... ... US small_business silver True 2,022 7500_plus 21,605 8,876.671 792.939 8873.30 10110.10 2022-01-01 2025-04-26 17:46:37 1,666,667 10 1.3 query-3 (worker-3) category brand supplier_region discontinued introduced_year product_count stock_units inventory_value average_price price_stddev median_price p95_price category_inventory_value inventory_rank percentage_of_category_value ----------- ------- --------------- ------------ --------------- ------------- ----------- --------------- ------------- ------------ ------------ --------- ------------------------ -------------- ---------------------------- clothing Aster US False 2,020 33,457 83,743,010 84191813103.00 1,004.833 577.373 1004.90 1904.90 4188462626332.28 1 2.01 clothing Aster UK False 2,020 33,458 83,228,180 83715767600.00 1,005.04 577.369 1005.20 1905.42 4188462626332.28 2 2 ... ... ... home Dove APAC False 2,024 32,999 82,797,401 83251410292.23 1,004.931 577.217 1005.23 1904.83 4190160306717.15 8 1.99 home Dove APAC False 2,020 33,008 82,760,072 83227553226.96 1,005.078 577.503 1005.33 1905.43 4190160306717.15 9 1.99 ... ... sports Elm EU False 2,020 33,006 82,731,982 83205651717.58 1,005.109 577.373 1005.19 1905.44 4190156973777.15 9 1.99 sports Bramble EU False 2,020 33,005 82,700,285 83167881906.05 1,004.964 577.415 1005.21 1905.36 4190156973777.15 10 1.98 real 0m17.664s user 0m1.643s sys 0m0.389s [root@ip-10-42-0-10 ~]# 3. Concurrent writes/reads To show this we’ll concurrently write 20 new records into our sales table and fire off three reads. Each read sees a consistent snapshot containing whichever independent insert transactions had committed before that read began. It may therefore see none, some or all of the new rows, but it will not see half of an individual insert or a result that changes while the SELECT is being scanned. Every insert in this example is a separate autocommit transaction. If 19 inserts succeed and one fails, the 19 successful writes remain committed. There is no distributed transaction and Cluster-Duck does not roll successful statements back. To make the demonstration repeatable, first remove any rows left by any earlier runs: [root@ip-10-42-0-10 ~]# cluster-duck-sql \ --allow-write \ --query "worker-1=DELETE FROM sales WHERE sale_id BETWEEN 30000001 AND 30000020" Note also that in order to make changes to data in a database we should supply the — allow-write argument. Now run the 20 inserts and three reads together. To see what SQL is run for each of the query labels in the output we can use the — show-sql argument. [root@ip-10-42-0-10 ~]# cluster-duck-sql \ --show-sql \ --allow-write \ --query "worker-1=INSERT INTO sales VALUES (30000001,1,1,1,'online','card','completed',100.00,100.00,0.0,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000002,2,2,2,'store','bank_transfer','processing',110.00,104.50,0.05,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000003,3,3,3,'marketplace','wallet','shipped',120.00,108.00,0.10,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000004,4,4,4,'telephone','invoice','completed',130.00,110.50,0.15,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000005,5,5,5,'online','card','processing',140.00,112.00,0.20,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000006,6,6,1,'store','bank_transfer','shipped',150.00,150.00,0.0,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000007,7,7,2,'marketplace','wallet','completed',160.00,152.00,0.05,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000008,8,8,3,'telephone','invoice','processing',170.00,153.00,0.10,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000009,9,9,4,'online','card','shipped',180.00,153.00,0.15,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000010,10,10,5,'store','bank_transfer','completed',190.00,152.00,0.20,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000011,11,11,1,'marketplace','wallet','processing',200.00,200.00,0.0,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000012,12,12,2,'telephone','invoice','shipped',210.00,199.50,0.05,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000013,13,13,3,'online','card','completed',220.00,198.00,0.10,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000014,14,14,4,'store','bank_transfer','processing',230.00,195.50,0.15,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000015,15,15,5,'marketplace','wallet','shipped',240.00,192.00,0.20,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000016,16,16,1,'telephone','invoice','completed',250.00,250.00,0.0,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000017,17,17,2,'online','card','processing',260.00,247.00,0.05,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000018,18,18,3,'store','bank_transfer','shipped',270.00,243.00,0.10,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000019,19,19,4,'marketplace','wallet','completed',280.00,238.00,0.15,DATE '2026-08-09')" \ --query "worker-1=INSERT INTO sales VALUES (30000020,20,20,5,'telephone','invoice','processing',290.00,232.00,0.20,DATE '2026-08-09')" \ --query "worker-1=SELECT COUNT(*) AS visible_rows FROM sales WHERE sale_id BETWEEN 30000001 AND 30000020" \ --query "worker-1=SELECT COUNT(*) AS visible_rows FROM sales WHERE sale_id BETWEEN 30000001 AND 30000020" \ --query "worker-1=SELECT COUNT(*) AS visible_rows FROM sales WHERE sale_id BETWEEN 30000001 AND 30000020" # # Output Concurrent remote queries worker table start_offset_ms duration_seconds -------- -------- --------------- ---------------- worker-1 query-19 77.923 3.648 worker-1 query-11 26.242 3.705 worker-1 query-23 4.64 3.727 worker-1 query-5 8.747 3.691 worker-1 query-1 4.832 3.744 worker-1 query-2 5.243 3.903 worker-1 query-15 113.315 3.823 worker-1 query-6 15.126 3.923 worker-1 query-22 84.907 3.853 worker-1 query-14 35.127 3.907 worker-1 query-18 71.325 3.871 worker-1 query-21 56.68 3.888 worker-1 query-12 28.05 3.936 worker-1 query-7 15.349 3.949 worker-1 query-20 89.082 3.875 worker-1 query-10 36.901 3.93 worker-1 query-16 77.02 3.895 worker-1 query-13 19.964 3.954 worker-1 query-3 5.632 3.968 worker-1 query-8 58.938 3.916 worker-1 query-9 15.774 3.959 worker-1 query-4 50.36 3.954 worker-1 query-17 84.575 3.936 Start spread: 108.675 ms query-19 (worker-1) SQL: INSERT INTO sales VALUES (30000019,19,19,4,'marketplace','wallet','completed',280.00,238.00,0.15,DATE '2026-08-09') Result: Count ----- 1 query-11 (worker-1) SQL: INSERT INTO sales VALUES (30000011,11,11,1,'marketplace','wallet','processing',200.00,200.00,0.0,DATE '2026-08-09') Result: Count ----- 1 query-23 (worker-1) SQL: SELECT COUNT(*) AS visible_rows FROM sales WHERE sale_id BETWEEN 30000001 AND 30000020 Result: visible_rows ------------ 4 ... ... query-22 (worker-1) SQL: SELECT COUNT(*) AS visible_rows FROM sales WHERE sale_id BETWEEN 30000001 AND 30000020 Result: visible_rows ------------ 16 ... ... query-18 (worker-1) SQL: INSERT INTO sales VALUES (30000018,18,18,3,'store','bank_transfer','shipped',270.00,243.00,0.10,DATE '2026-08-09') Result: Count ----- 1 query-21 (worker-1) SQL: SELECT COUNT(*) AS visible_rows FROM sales WHERE sale_id BETWEEN 30000001 AND 30000020 Result: visible_rows ------------ 13 ... ... query-17 (worker-1) SQL: INSERT INTO sales VALUES (30000017,17,17,2,'online','card','processing',260.00,247.00,0.05,DATE '2026-08-09') Result: Count ----- 1 [root@ip-10-42-0-10 ~]# The output shows that by the time query-23 ran, 4 records had been inserted. By the time query-22 ran, 16 records had been inserted and for query-21, 13 new records had been written. This makes sense as we can see from the start_offset_ms timings that the order of running for the queries was query-23, then query21, and finally query-22. 5. You can run DDL too Create 3 new tables, one in each database , then query them. [root@ip-10-42-0-10 ~]# cluster-duck-sql \ --allow-write \ --query "worker-1=CREATE OR REPLACE TABLE sales_agg AS SELECT sale_status, COUNT(*) AS sale_count FROM sales WHERE sale_status = 'cancelled' GROUP BY sale_status" \ --query "worker-2=CREATE OR REPLACE TABLE customers_agg AS SELECT country, COUNT(*) AS customer_count FROM customers WHERE country = 'UK' GROUP BY country" \ --query "worker-3=CREATE OR REPLACE TABLE products_agg AS SELECT category, COUNT(*) AS product_count FROM products WHERE category = 'sports' GROUP BY category" cluster-duck-sql \ --query "worker-1=SELECT * FROM sales_agg ORDER BY sale_count DESC" \ --query "worker-2=SELECT * FROM customers_agg ORDER BY customer_count DESC" \ --query "worker-3=SELECT * FROM products_agg ORDER BY product_count DESC" # # Output Concurrent remote queries worker table start_offset_ms duration_seconds -------- ------- --------------- ---------------- worker-1 query-1 1.141 0.599 worker-2 query-2 1.033 0.603 worker-3 query-3 0.755 0.622 Start spread: 0.386 ms query-1 (worker-1) Count ----- 1 query-2 (worker-2) Count ----- 1 query-3 (worker-3) Count ----- 1 Concurrent remote queries worker table start_offset_ms duration_seconds -------- ------- --------------- ---------------- worker-1 query-1 0.925 0.464 worker-2 query-2 1.178 0.448 worker-3 query-3 0.675 0.467 Start spread: 0.503 ms query-1 (worker-1) sale_status sale_count ----------- ---------- cancelled 2,000,000 query-2 (worker-2) country customer_count ------- -------------- UK 1,666,666 query-3 (worker-3) category product_count -------- ------------- sports 1,666,667 The cost of all this. The cost of this set-up shouldn’t be a concern. For a start, DuckDB and Quack are free to download and use. The three EC2 servers that I’m standing up are tiny t4g.nano instances. As well as that, we have three 8 GB gp3 volumes, public IPv4 addresses, Systems Manager, Parameter Store and a Lambda custom resource. The Lambda invocation is short-lived, but remains deployed until the stack is deleted. This Lambda is called TokenManagerFunction in the CloudFormation template. Its only job is to manage the three Quack authentication tokens. It works like this, CloudFormation stack creation ↓ Invoke TokenManager Lambda ↓ Generate three random 64-character tokens ↓ Store them as SecureString parameters in SSM ↓ Return success and stop Here is an estimated cost if we ran this whole set-up for 4 hours. Component Approximate cost Four hours of EBS $0.011 Four hours of EC2 $0.050 Four hours of public IPv4 $0.060 Total $0.121 The $0.121 figure is an estimate for us-east-2, before any credits or free-tier allowances, tax and data-transfer charges. Qualifying AWS Free Tier users may receive some public IPv4 hours at no charge. AWS bills gp3 storage in per-second increments, with a 60-second minimum. For peace of mind, though, I would always advise tearing down any AWS infrastructure created after you’re done with it. This is easily done if you use CloudFormation by running the following command with the AWS CLI. aws cloudformation delete-stack \ --region us-east-2 \ --stack-name cluster-duck-test-v2 Summary I created the “cluster-duck” repo to test DuckDB’s new Quack communications protocol. Quack allows DuckDB databases on different servers to “talk” to each other over HTTP, and DuckDB is positioning it as an enabler of client/server communications between DuckDB databases. This development is potentially very useful and I was particularly interested to see how well Quack handled concurrent reads and writes to and from a remote database. In my tests, and in the example I demonstrated, the answer seems to be it handles it pretty well. The DuckDB team have stated that Quack is an experimental feature and very much a work-in-progress. To that point, you can expect potential changes to the protocol, function names, settings and defaults, so definitely do not use Quack for any production systems. Hopefully the concepts I’ve outlined in this article will be useful if you have a need to run parallel queries or other SQL statements moere generally against DuckDB databases running on different servers. I can’t help but wonder what future plans DuckDB have for Quack. If it becomes a fully supported part of the DuckDB eco-system I could definitely see Quack supplying the transport and session handling to a future distributed DuckDB engine, but that’s probably a long way off. But, even it just does what its capable of now, Quack will be useful in its own right. That’s all from me for now. You can access all the code, CloudFormation template etc… in my GitHub repo at: https://github.com/taupirho/cluster-duck More information on DuckDB and Quack can be found in the DuckDB online documentation at this link, https://duckdb.org/docs/current PS I’m on the market for contract work just now. If you or someone you know is looking for an experienced data engineer, either remote or based in Edinburgh, UK, with skills in AWS, AI, Python, SQL, PySpark, DuckDB, etc., let me know via LinkedIn