Rewrite File Upload and Download using RandomAccessFile #135

Open
opened 2 months ago by PapaTutuWawa · 5 comments
Owner

This allows us to only load something like 8KiB into memory for the upload and save on memory.

This allows us to only load something like 8KiB into memory for the upload and save on memory.
PapaTutuWawa added the
good first issue
backend/service
Improvement
labels 2 months ago
PapaTutuWawa added this to the Alpha Release 0.4.0 project 2 months ago

Is it the idea to switch send and load in chunks?

Btw, I like that you tag "good first issues".

Is it the idea to switch send and load in chunks? Btw, I like that you tag "good first issues".
Poster
Owner

My idea is to use RandomAccessFile instead of the "regular" Dart File because it allows us to read only a part of the file into memory for upload. By doing this, we can keep the system from killing the app because we use too much memory (Think of someone sending a 1 GB file to their friend).

Essentially, I was thinking of something like:

  • Get file size
  • Using 8KiB as a block size: n = ceil(size / 8KiB) is the amount of blocks we give Dio
  • When Dio has uploaded or downloaded that block, load in or write out the next 8KiB block until we're done.

In my head, this seems fairly easy. Just a bit of digging into Dio might be required.

My idea is to use RandomAccessFile instead of the "regular" Dart File because it allows us to read only a part of the file into memory for upload. By doing this, we can keep the system from killing the app because we use too much memory (Think of someone sending a 1 GB file to their friend). Essentially, I was thinking of something like: - Get file size - Using 8KiB as a block size: `n = ceil(size / 8KiB)` is the amount of blocks we give Dio - When Dio has uploaded or downloaded that block, load in or write out the next 8KiB block until we're done. In my head, this seems fairly easy. Just a bit of digging into Dio might be required.

If I understand you correctly, this would mean that we do chunked uploads, but this has to be supported by the server? I couldn't find anything in XEP-0363, are there business standards? Like you send a "chunkCount" field in the multipart and then the server knows to expect a chunked upload?

If I understand you correctly, this would mean that we do chunked uploads, but this has to be supported by the server? I couldn't find anything in XEP-0363, are there business standards? Like you send a "chunkCount" field in the multipart and then the server knows to expect a chunked upload?
Poster
Owner

I think I did not express myself clearly. Sorry for that...

What I mean is:

  • Read a chunk (something like 8 or 16KiB) from the file into memory
  • Send it to the server
  • Read another chunk into memory
  • ...

But we do this as a "continous stream" instead of actual chunked uploads. I just want to avoid having to load the entire file into memory.

With "finding out how many chunks we have" I was thinking ahead as to how to trick Dio into not requiring the entire file in memory.

I think I did not express myself clearly. Sorry for that... What I mean is: - Read a chunk (something like 8 or 16KiB) from the file into memory - Send it to the server - Read another chunk into memory - ... But we do this as a "continous stream" instead of actual chunked uploads. I just want to avoid having to load the entire file into memory. With "finding out how many chunks we have" I was thinking ahead as to how to trick Dio into not requiring the entire file in memory.

Nah, that was my bad. I was thinking too much in web stuff 😅. I wrapped my head around this and came up with sth, but I'm still testing.

Nah, that was my bad. I was thinking too much in web stuff 😅. I wrapped my head around this and came up with sth, but I'm still testing.
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

No dependencies set.

Reference: moxxy/moxxyv2#135
Loading…
There is no content yet.