Is there anyway to upload a file more than 2 GB, using simple html form upload? Previously I have been uploading large files through silverlight using chunking (dividing a large file into segments and then uploading segments one by one &amp; then reassemble segments at server).

Now, we have a requirement that we just have to use simple html (though GWT) form uploads. Please guide me if there is any way to achieve large file upload this way.

If it is impossible to do it using simple html, can anyone guide me about how I can divide &amp; upload a file in segments using flex?


Large file upload though html form (more than 2 GB)

Situation
---------
I have two servers, Production and Development. On Production server, there are two applications and multiple (6) databases (MySQL) which I need to distribute to developers for testing. All source codes are stored in [GitLab](http://gitlab.org) on Development server and developers are working only with this server and don&#39;t have access to production server. When we release an application, master logs into production and pulls new version from Git. The databases are large (over 500M each and counting) and I need to distribute them as easy as possible to developers for testing.

Possible solutions
------------------

 *  &lt;strike&gt;After a backup script which dumps databases, each to a single file, execute a script which pushes each database to its own branch. A developer pulls one of these branches if he wants to update his local copy.&lt;/strike&gt;

    This one was found non working.

 *  Cron on production server saves binary logs every day and pushes them into the branch of that database. So, in the branch, there are files with daily changes and developer pulls the files he doesn&#39;t have. The current SQL dump will be sent to the developer another way. And when the size of the repository becomes too large, we will send full dump to the developers and flush all data in the repository and start from the beginning.

Questions
---------

 - Is the solution possible?
 - If git is pushing/pulling to/from repository, does it upload/download whole files, or just changes in them (i.e. adds new lines or edits the current ones)?
 - &lt;strike&gt;Can Git manage so large files?&lt;/strike&gt; No.
 - &lt;strike&gt;How to set how many revisions are preserved in a repository?&lt;/strike&gt; Doesn&#39;t matter with the new solution.
 - Is there any better solution? I don&#39;t want to force the developers to download such large files over FTP or anything similar.

Git with large files

I mean 100+ MB big; such text files can push the envelope of editors.

I need to look through a large XML file, but cannot if the editor is buggy.

Any suggestions?

Text editor to open big (giant, huge, large) text files

I use huge data files, sometimes I only need to know the number of lines in these files, usually I open them up and read them line by line until I reach the end of the file

I was wondering if there is a smarter way to do that

Number of lines in a file in Java

I have to sync large files across some machines. The files can be up to 6GB in size. The sync will be done manually every few weeks. I cant take the filename into consideration because they can change anytime. 

My plan is to create checksums on the destination PC and on the source PC and then copy all files with a checksum, which are not already in the destination, to the destination.
My first attempt was something like this:

    using System.IO;
    using System.Security.Cryptography;
    
    private static string GetChecksum(string file)
    {
        using (FileStream stream = File.OpenRead(file))
        {
            SHA256Managed sha = new SHA256Managed();
            byte[] checksum = sha.ComputeHash(stream);
            return BitConverter.ToString(checksum).Replace(&quot;-&quot;, String.Empty);
        }
    }

The Problem was the runtime:&lt;br&gt;
- with SHA256 with a 1,6 GB File -&gt; 20 minutes&lt;br&gt;
- with MD5 with a 1,6 GB File -&gt; 6.15 minutes

Is there a better - faster - way to get the checksum (maybe with a better hash function)?


What is the fastest way to create a checksum for large files in C#

I tried opening a huge (~2GB) file in VIM but it choked. I don&#39;t actually need to edit the file, just jump around efficiently.

How can I go about working with very large files in VIM?

Working with huge files in VIM

We&#39;re just starting a UNIX class and are learning a variety of Bash commands. Our assignment involves performing various commands on a directory that has a number of folders under it as well. 

I know how to list and count all the regular files from the root folder using: 

    find . -type l | wc -l

But I&#39;d like to know where to go from there in order to find the largest file in the whole directory. I&#39;ve seen somethings regarding a `du` command, but we haven&#39;t learned that, so in the repertoire of things we&#39;ve learned I assume we need to somehow connect it to the `ls -t` command.


And pardon me if my &#39;lingo&#39; isn&#39;t correct, I&#39;m still getting used to it!

How to find the largest file in a directory and its subdirectories?

I am looking for opinions of how to handle large binary files on which my source code (web application) is dependent. We are currently discussing several alternatives:

1. Copy the binary files by hand.
 - Pro: Not sure.
 - Contra: I am strongly against this, as it increases the likelihood of errors when setting up a new site/migrating the old one. Builds up another hurdle to take.
2. Manage them all with _Git_.
 - Pro: Removes the possibility to &#39;forget&#39; to copy a important file
 - Contra: Bloats the repository and decreases flexibility to manage the code-base and checkouts, clones, etc. will take quite a while.
3. Separate repositories.
 - Pro: Checking out/cloning the source code is fast as ever, and the images are properly archived in their own repository.
 - Contra: Removes the simpleness of having _the one and only_ Git repository on the project. It surely introduces some other things I haven&#39;t thought about.

What are your experiences/thoughts regarding this?

Also: Does anybody have experience with multiple Git repositories and managing them in one project?

The files are images for a program which generates PDFs with those files in it. The files will not change very often (as in years), but they are very relevant to a program. The program will not work without the files.


Managing large binary files with Git

I&#39;ve got the lovely task of working out how to handle large files being loaded into our application&#39;s script editor (it&#39;s like [VBA][1] for our internal product for quick macros). Most files are about 300-400&amp;nbsp;KB which is fine loading. But when they go beyond 100&amp;nbsp;MB the process has a hard time (as you&#39;d expect).

What happens is that the file is read and shoved into a RichTextBox which is then navigated - don&#39;t worry too much about this part.

The developer who wrote the initial code is simply using a StreamReader and doing

    [Reader].ReadToEnd()

which could take quite a while to complete.

My task is to break this bit of code up, read it in chunks into a buffer and show a progressbar with an option to cancel it.

Some assumptions:

 - Most files will be 30-40&amp;nbsp;MB
 - The contents of the file is text (not binary), some are Unix format, some are DOS.
 - Once the contents is retrieved we work out what terminator is used.
 - No-one&#39;s concerned once it&#39;s loaded the time it takes to render in the richtextbox. It&#39;s just the initial load of the text.

Now for the questions:

  - Can I simply use StreamReader, then check the Length property (so ProgressMax) and issue a Read for a set buffer size and iterate through in a while loop *WHILST* inside a background worker, so it doesn&#39;t block the main UI thread? Then return the stringbuilder to the main thread once it&#39;s completed.
  - The contents will be going to a StringBuilder. can I initialise the StringBuilder with the size of the stream if the length is available?

Are these (in your professional opinions) good ideas? I&#39;ve had a few issues in the past with reading content from Streams, because it will always miss the last few bytes or something, but I&#39;ll ask another question if this is the case.

  [1]: http://en.wikipedia.org/wiki/Visual_Basic_for_Applications


Reading large text files with streams in C#

I&#39;m trying to setup a very basic html5 page that loads a .mp4 video that is 20MB.  It appears that the browser needs to download the entire thing rather than just playing the first part of the video and streaming in the rest.

[This post][1] is the closest thing I&#39;ve found while searching... I tried both Hand Brake and Data Go Round by neither appeared to make a difference: 

Any ideas on how to do this or if it&#39;s possible?

Here is the code I&#39;m using:

&lt;!-- language: lang-html --&gt;

    &lt;video controls=&quot;controls&quot;&gt;
        &lt;source src=&quot;/video.mp4&quot; type=&quot;video/mp4&quot; /&gt;
        Your browser does not support the video tag.
    &lt;/video&gt;


  [1]: https://stackoverflow.com/questions/6683418/inline-video-large-file-with-buffer

HTML5 - How to stream large .mp4 files?

I have some json files with 500MB. 
If I use the &quot;trivial&quot; `json.load()` to load its content all at once, it will consume a lot of memory. 

Is there a way to read partially the file? If it was a text, line delimited file, I would be able to iterate over the lines. I am looking for analogy to it.


Is there a memory efficient and fast way to load big JSON files?

Is it easy to read a line from a gz-compressed text file using python without extracting the file completely? I have a text.gz file which is aroud 200mb. When I extract it, it becomes 7.4gb. And this is not the only file I have to read. For the total process, I have to read 10 files. Although this will be a sequential job, I think it will a smart thing to do it without extarcting the whole information. I do not even know that it is possible. How can it be done using python? I need to read a text file line-by-line.

python: read lines from compressed text files

How can I import a large (14 GB) MySQL dump file into a new MySQL database?

I am looking for a text editor that will be able to load a 4+ Gigabyte file into it.  Textpad doesn&#39;t work.  I own a copy of it and have been to its support site, it just doesn&#39;t do it.  Maybe I need new hardware, but that&#39;s a different question.   The editor needs to be free OR, if its going to cost me, then no more than $30.  For Windows.

Best Free Text Editor Supporting *More Than* 4GB Files?

I have a large server log file (~750 MB) which I can&#39;t open with either Notepad or Notepad++ (they both say the file is too large).

Can anyone suggest a program (for Windows) that will only read a small part of the file into memory at a time?

Or do I need to write my own app to parse this file?

How to read large text file on windows?

I&#39;m trying to implement Git to manage creative assets (Photoshop, Illustrator, Maya, etc.), and I&#39;d like to exclude files from Git based on file size rather than extension, location, etc.

For example, I don&#39;t want to exclude all .avi files, but there are a handful of massive +1GB avi files in random directories that I don&#39;t want to commit. 

Any suggestions?

gitignore by file size?

What is the most efficient way to display the last 10 lines of a very large text file (this particular file is over 10GB). I was thinking of just writing a simple C# app but I&#39;m not sure how to do this effectively.



Get last 10 lines of very large text file &gt; 10GB

I have some csv files that are larger than github&#39;s file size limit of 100.00 MB. I have been trying to use the Git Large File Storage extension.

https://git-lfs.github.com/

From LFS - `&quot;Large file versioning- Version large files—even those as large as a couple GB in size—with Git.&quot;`

I have applied the following on the folders of concern:

    git lfs track &quot;*.csv&quot;
  
However, when I push:

    remote: error: File Time-Delay-ftn/Raw-count-data-minor-roads1.csv is 445.93 MB; this exceeds GitHub&#39;s file size limit of 100.00 MB
    remote: error: File Time-Delay-ftn/Raw-count-data-major-roads.csv is 295.42 MB; this exceeds GitHub&#39;s file size limit of 100.00 MB

When I look at the folder in question:

    -rw-r-----   1 user  staff    42B 23 Oct 12:34 .gitattributes
    -rw-r--r--   1 user  staff   1.3K 19 Oct 14:32 DfT_raw_major_manipulation.py
    -rw-r--r--   1 user  staff   1.2K 16 Oct 15:08 DfT_raw_minor_manipulation.py
    drwxr-xr-x  21 user  staff   714B 22 Oct 11:35 Driving/
    -rwxr-xr-x@  1 user  staff   295M 19 Oct 14:47 Raw-count-data-major-roads1.csv*
    -rwxr-xr-x@  1 user  staff   446M 16 Oct 14:52 Raw-count-data-minor-roads1.csv*

when I vim the .gitattributes file you can see the lfs setup:

    *.csv filter=lfs diff=lfs merge=lfs -text

What am I doing wrong?

UPDATE

When I query 

    git lfs ls-files

I get nothing returned. This indicates that despite the .csv filter being successfully applied to the .gitattributes file the csv files are not being picked up by lfs

Git lfs - &quot;this exceeds GitHub&#39;s file size limit of 100.00 MB&quot;

I&#39;m trying to parse the DMOZ content/structures XML files into MySQL, but all existing scripts to do this are very old and don&#39;t work well. How can I go about opening a large (+1GB) XML file in PHP for parsing?

Parsing Huge XML Files in PHP

I have spent the best part of two days &quot;faffing&quot; about with code samples and etc., trying to read a very large JSON file into an array in c# so I can later split it up into a 2d array for processing.

The problem I was having was I could not find any examples of people doing what I was trying to do. This meant I was just editing code a little an hoping for the best.

I have managed to get something working that will:

 - Read the file Miss out headers and only read values into array.
 - Place a certain amount of values on each line of an array. (So I
    could later split it an put into 2d array)

This was done with the code below but it crashes the program after entering a few lines into the array. This might have to do with the file size.

    // If the file extension was a jave file the following 
    // load method will be use else it will move on to the 
    // next else if statement
    if (fileExtension == &quot;.json&quot;) 
    {
        int count = 0;
        int count2 = 0;
        int inOrOut = 0;
        int nRecords=1; 
        JsonTextReader reader = new JsonTextReader(new StreamReader(txtLoaction.Text));
        string[] rawData = new string[5];
        while (reader.Read())
        {
            if (reader.Value != null)
                if (inOrOut == 1)
                {
                    if (count == 6)
                    {
                        nRecords++;
                        Array.Resize(ref rawData, nRecords);
                        //textBox1.Text += &quot;\r\n&quot;;
                        count = 0;
                    }
                    rawData[count2] += reader.Value + &quot;,&quot;; //+&quot;\r\n&quot;
                    inOrOut = 0;
                    count++;
                    if (count2 == 500)
                    {
                        MessageBox.Show(rawData[499]);
                    }
                }
                else
                {
                    inOrOut = 1;
                }
        } 
    }

A snippet of the JSON I am working with is:

    [ 
        { &quot;millis&quot;: &quot;1000&quot;, 
          &quot;stamp&quot;: &quot;1273010254&quot;, 
          &quot;datetime&quot;: &quot;2010/5/4 21:57:34&quot;, 
          &quot;light&quot;: &quot;333&quot;, 
          &quot;temp&quot;: &quot;78.32&quot;, 
          &quot;vcc&quot;: &quot;3.54&quot; }, 
    ] 
I need the values out of this JSON.  For example, I need &quot;3.54&quot;, but I would not want it to print the &quot;vcc&quot;.

I am hoping someone can show me how to read a JSON file in and only extract the data that I need and put it into an array or something that I can use to later put into an array.

 


All Large Files Solutions on Gang of Coders

Total of 20 Large Files Solutions