1
0
Fork 0

Change CSV export to buffer instead of streaming

The idea behind a streaming CSV export was to reduce the amount of
memory used, by avoiding building the entire CSV file in memory before
sending it to the client. However, it didn't work out this way in
practice: the query objects that were created to represent each line
caused Postgres to generate a very large (~200MB on bookwyrm.social)
temp file, not to mention the memory being used by the Query object
likely being similar to, if not larger than that used by the finalized
CSV row.

While we should in the long term run our CSV exports as a Celery task,
this change should allow CSV exports to work on large servers without
causing disk-space problems.

Fixes: #2157
This commit is contained in:
Wesley Aptekar-Cassels 2023-03-07 17:30:19 -05:00
parent 00666c4f52
commit 50a81bdfdd
2 changed files with 51 additions and 64 deletions

View file

@ -1,7 +1,7 @@
""" test for app action functionality """
from unittest.mock import patch
from django.http import StreamingHttpResponse
from django.http import HttpResponse
from django.test import TestCase
from django.test.client import RequestFactory
@ -57,13 +57,12 @@ class ExportViews(TestCase):
request = self.factory.post("")
request.user = self.local_user
export = views.Export.as_view()(request)
self.assertIsInstance(export, StreamingHttpResponse)
self.assertIsInstance(export, HttpResponse)
self.assertEqual(export.status_code, 200)
result = list(export.streaming_content)
# pylint: disable=line-too-long
self.assertEqual(
result[0],
b"title,author_text,remote_id,openlibrary_key,inventaire_id,librarything_key,goodreads_key,bnf_id,viaf,wikidata,asin,aasin,isfdb,isbn_10,isbn_13,oclc_number,rating,review_name,review_cw,review_content\r\n",
export.content,
b"title,author_text,remote_id,openlibrary_key,inventaire_id,librarything_key,goodreads_key,bnf_id,viaf,wikidata,asin,aasin,isfdb,isbn_10,isbn_13,oclc_number,rating,review_name,review_cw,review_content\r\nTest Book,,"
+ self.book.remote_id.encode("utf-8")
+ b",,,,,beep,,,,,,123456789X,9781234567890,,,,,\r\n",
)
expected = f"Test Book,,{self.book.remote_id},,,,,beep,,,,,,123456789X,9781234567890,,,,,\r\n"
self.assertEqual(result[1].decode("utf-8"), expected)