Forum OpenACS Improvement Proposals (TIPs): Tip #59 (Implemented) Change default behavior to store files in file system

1: Tip #59 (Implemented) Change default behavior to store files in file system

Posted by Tracy Adams on 04/25/04 12:53 PM

I had this conversation with a couple of people, most recently with Cato from Bergen. They saw their database growing due to the files from file storage. At first they were confused and then they had to do a lot of effort to move to store the files in the file system. We at Sloan also have the problem that our database is getting very big and have to take care of switching.

I'm wondering, should we change the default to store the files in the file system instead of the database.

Thanks,
Tracy

2: Re: Tip #59: Change default behavior to store files in file system (response to 1)

Posted by Don Baccus on 04/25/04 01:40 PM

Just to refresh memories on the pros and cons of storing file storage items in the file system ...

Pro

Considerably improved performance, especially with PostgreSQL, with large files, and with AOLserver 4.

Con

You need to back up both filesystem and database, and there's a risk of inconsistency between the two.

In the past, the fact that backups might be inconsistent have led us to store files in the database by default. However, those with large installations and large numbers of users seem to universally find the performance issue to be of more importance.

3: Re: Tip #59: Change default behavior to store files in file system (response to 2)

Posted by Malte Sussdorff on 04/26/04 09:15 AM

How does the risk of inconsitency really affect the site.

Option 1): There is a file backed up whose revision_id in the database has not been recorded. You loose the file, but have it on disk lingering around and it might be restored.

Option 2): There is a file recorded in the database, but it is no longer there. In this case you can ask the users to upload their files again (wording: "If you uploaded the file within the last @backup_interval@, please revisit our site as it might have been lost").

4: Re: Tip #59: Change default behavior to store files in file system (response to 1)

Posted by Malte Sussdorff on 04/26/04 09:16 AM

Though agreeing with Don's caution, I don't think it matters so gravely to have our users go through the experience of storing all the files in the database.

Approve.

5: Re: Tip #59: Change default behavior to store files in file system (response to 1)

Posted by Caroline Meeks on 04/26/04 09:19 AM

Approve

6: Re: Tip #59: Change default behavior to store files in file system (response to 5)

Posted by Jeff Davis on 04/26/04 09:31 AM

approve

7: Re: Tip #59: Change default behavior to store files in file system (response to 6)

Posted by Jeff Davis on 04/26/04 09:33 AM

Also, if someone has written code to move a file-storage
instance to storing in the filesystem it would be useful
to pull that in to the admin functionality for file-storage.

8: Approve: Tip #59: Change default behavior to store files in file system (response to 1)

Posted by Don Baccus on 04/26/04 10:10 AM

I wasn't stating any personal caution, just pointing out the pro/con argument. I personally approve of the change, but wanted to state the pro/con sides objectively without giving my opinion...

9: Re: Approve: Tip #59: Change default behavior to store files in file system (response to 8)

Posted by Lars Pind on 04/26/04 01:41 PM

Approved

10: Re: Approve: Tip #59: Change default behavior to store files in file system (response to 9)

Posted by Jeff Davis on 05/05/04 11:35 AM

I made this change on both oacs-5-1 and HEAD.