Git lfs - "this exceeds GitHub's file size limit of 100.00 MB"
GitGithubLarge FilesGit LfsGit Problem Overview
I have some csv files that are larger than github's file size limit of 100.00 MB. I have been trying to use the Git Large File Storage extension.
From LFS - "Large file versioning- Version large files—even those as large as a couple GB in size—with Git."
I have applied the following on the folders of concern:
git lfs track "*.csv"
However, when I push:
remote: error: File Time-Delay-ftn/Raw-count-data-minor-roads1.csv is 445.93 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File Time-Delay-ftn/Raw-count-data-major-roads.csv is 295.42 MB; this exceeds GitHub's file size limit of 100.00 MB
When I look at the folder in question:
-rw-r----- 1 user staff 42B 23 Oct 12:34 .gitattributes
-rw-r--r-- 1 user staff 1.3K 19 Oct 14:32 DfT_raw_major_manipulation.py
-rw-r--r-- 1 user staff 1.2K 16 Oct 15:08 DfT_raw_minor_manipulation.py
drwxr-xr-x 21 user staff 714B 22 Oct 11:35 Driving/
-rwxr-xr-x@ 1 user staff 295M 19 Oct 14:47 Raw-count-data-major-roads1.csv*
-rwxr-xr-x@ 1 user staff 446M 16 Oct 14:52 Raw-count-data-minor-roads1.csv*
when I vim the .gitattributes file you can see the lfs setup:
*.csv filter=lfs diff=lfs merge=lfs -text
What am I doing wrong?
UPDATE
When I query
git lfs ls-files
I get nothing returned. This indicates that despite the .csv filter being successfully applied to the .gitattributes file the csv files are not being picked up by lfs
Git Solutions
Solution 1 - Git
Since Git LFS 2.2.0 you can use git lfs migrate command. In your case it would look this way:
git lfs migrate import --include="*.csv"
After this, you should be able to push changes into origin.
More information about git lfs migrate is available on Git LFS 2.2.0 release note site and git-lfs-migrate command documentation.
Solution 2 - Git
Simply adding git-lfs configuration to an existing repository will not retroactively convert your large files to LFS support. Those large files will remain in your history and GitHub will refuse your pushes.
You need to rewrite your history to introduce git-lfs to your existing commits. I recommend the BFG repo cleaner tool, which added LFS support recently.
You should be able to convert historical usage of your CSV files by:
$ java -jar ~/bfg-1.12.5.jar --convert-to-git-lfs '*.csv' --no-blob-protection
Solution 3 - Git
i had same issue, but it's resolved using filter-branch
git filter-branch --tree-filter 'rm -rf path/to/your/file' HEAD
it'll take some time if you have big project, then push it
git push
Solution 4 - Git
I had this error:
> remote: error: File client/static/static-version/20171221_221446.psd is 223.61 MB; this exceeds GitHub's file size limit of 100.00 MB
And because I already removed this file from this folder, created .gitignore file and tried to commit couple times, I didn't know that it was cached, I could not push to github. In my case helped:
> git filter-branch --index-filter 'git rm --cached --ignore-unmatch client/static/static-version/20171221_221446.psd'
Where I placed full file path(from error above) to remove it from cache. After that, push was made successfully
Solution 5 - Git
I hit the same problem yesterday and cracked it. I was unable to push, and it appeared that none of my big files were in lfs.
There is probably a better way, but this worked for me. I have a large repo with 2.5 gigs of data.
I setup a new repo then setup lfs in it.
git lfs init
I then configured my various file types
git lfs track "*.pdb"
git lfs track "*.dll"
I then commmited my changes and pushed.
I then added my big files. I used sourcetree, and in the output notes it would state for the big files matching my wildcards that it was committing tiny txt file instead. (sorry, I didn't record these, but it should be obvious).
Then I pushed, and I saw 'skipping files', and the push succeeded quickly.
so the problem is probably trying to add files to lfs that are already in your history. You can only add new files. You can probably clean your repo of these files.
Note: I did find that quite a few files that matched my wildcards were not picked up by lfs. Similar files in different folders were picked up, but not all. I tried explicitly adding these files using the full path.
git lfs track "Windows/bin/myBigFile.dll"
but that didn't help either. In the end I gave up due to time constraints.
You should also check your storage limit with gitHub. I purchased the extra 50gig to cover my requirements.
Cloning the repo now downloads the files separately and everything is finally working well.
Solution 6 - Git
I have faced the same problem while some of my csv file was merged in previous commit and github rejected my push. Later I found this command which worked for me.
git filter-branch -f --index-filter 'git rm --cached --ignore-unmatch fixtures/11_user_answer.json'
# for my case
git filter-branch -f --index-filter 'git rm --cached --ignore-unmatch tensorflow.csv'
Solution 7 - Git
Try the following command in your terminal. It will resolve the lfs issue.
git lfs migrate import --include="*.csv"
Solution 8 - Git
If you know which commit introduced the large commit, you could also try squashing that commit with the subsequent commits that introduced Git LFS.
For example, if the large commit was three commits ago (as revealed by git status), you could do the following:
git rebase -i HEAD~3
Then, replace all "pick" usages after the first one with "squash" in the interactive dialog.
Then,
git push origin --force
Solution 9 - Git
This solution worked for me on Ubuntu 20.04
Install the Git Large File Storage (git-lfs)
sudo apt-get install git-lfs
git-lfs install
To convert any pre-existing files to Git LFS, such as files on other branches or in your prior commit history us the git lfs migrate command
git lfs migrate import --include="*.ipynb"
Select the file types to be tracked
git lfs track "*.ipynb"
update the gitattributes
git add .gitattributes
Now push to the git git commit -m "commit message" git push
And for the storage and bandwidth usage please refer
Solution 10 - Git
This may be help you
Only install lfs to a exist repo may be not enough. Your may also change the commit history. Hope this is work for you .