I&#39;m currently doing this, but it&#39;s VERY slow since I have several terabytes of data in the bucket:

    gsutil du -sh gs://my-bucket-1/

And the same for a sub-folder:

    gsutil du -sh gs://my-bucket-1/folder

Is it possible to somehow obtain the total size of a complete bucket (or a sub-folder) elsewhere or in some other fashion which is much faster?

Fastest way to get Google Storage bucket size?

Some functions in the Google Developers Console, like the Analytics API, are free until you reach a quota. Other functions, like Google Cloud Storage, create costs from the first click.

When I upload a file under `https://console.developers.google.com/ &gt; Storage &gt; Cloud Storage &gt; Storage Browser` and I make this file publicly available, I pay about $0.12 per GB traffic.

But theoretically the traffic to this link could explode, e.g. because of sudden popularity. Therefore I would like to set something like a daily or monthly cost limit. 

Q: How do I protect myself from overly high costs in the Google Developers Console?

How do I set a cost limit in Google Developers Console

Is there an option to count the number of files in bucket-folders?  

Like: 

    gsutil ls -count -recursive gs://bucket/folder

    Result:   666 files

I just want an total number of files to compare the
amount to the sync-folder on my server.

I don&#39;t get it in the manual.

How to count number of file in a bucket-folder with gsutil

Is there any way to serve a static website (SPA actually) located on Google Cloud Storage via SSL, for that nice SSL address and icon for users to see?

Amazon allows this via CloudFront SNI.

Setting up SSL for Google Cloud Storage static website?

I have a Google App Engine app, which connects to Google Cloud Storage. 

I noticed that the amount of data stored was unreasonably high (4.01 GB, when it should be 100MB or so). 

So, I looked at how much each bucket was storing, and I found that there was an automatically created bucket called `us.artificats.` that was taking up most of the space.

I looked inside, and all it has is one folder: `containers/images/`. 

From what I&#39;ve Googled, it seems like these images come from Google Cloud Build.

My question is, can I delete them without compromising my entire application?

Can I delete container images from Google Cloud Storage artifacts bucket?

We have our application hosted on Google app engine for Java and we have a requirement where we want to store the blood donor appreciation certificates (`html` files) somewhere.

So, we can use either Google Blobstore or Google Cloud Storage. I know both of these solutions are feasible to do using gae for Java

However, the problem for us is to decide which one to use. What are the pros and cons of these two approaches?

We are a non profit and cannot pay a lot. 

Google Blobstore versus Google Cloud Storage

TL;DR. I&#39;m lost as to how to access the data after deleting a PVC, as well as why PV wouldn&#39;t go away after deleting a PVC.

Steps I&#39;m taking:

1. created a disk in GCE manually:

        gcloud compute disks create --size 5Gi disk-for-rabbitmq --zone europe-west1-b

2. ran:
  
        kubectl apply -f /tmp/pv-and-pvc.yaml

    with the following config:

        # /tmp/pv-and-pvc.yaml
        apiVersion: v1
        kind: PersistentVolume
        metadata:
          name: pv-for-rabbitmq
        spec:
          accessModes:
          - ReadWriteOnce
          capacity:
            storage: 5Gi
          gcePersistentDisk:
            fsType: ext4
            pdName: disk-for-rabbitmq
          persistentVolumeReclaimPolicy: Delete
          storageClassName: standard
        ---
        apiVersion: v1
        kind: PersistentVolumeClaim
        metadata:
          name: pvc-for-rabbitmq
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 5Gi
          storageClassName: standard
          volumeName: pv-for-rabbitmq

3. deleted a PVC manually (on a high level: I&#39;m simulating a disastrous scenario here, like accidental deletion or misconfiguration of a `helm` release):

        kubectl delete pvc pvc-for-rabbitmq

At this point I see the following:

    $ kubectl get pv
    NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                      STORAGECLASS   REASON   AGE
    pv-for-rabbitmq   5Gi        RWO            Delete           Released   staging/pvc-for-rabbitmq   standard                8m
    $

&gt; A side question, just improve my understanding: **why PV is still there, even though it has a reclaim policy set to `Delete`?** Isn&#39;t this what the [docs](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#delete) say for the `Delete` reclaim policy?

Now if I try to re-create the PVC to regain access to the data in PV:

    $ kubectl apply -f /tmp/pv-and-pvc.yaml
    persistentvolume &quot;pv-for-rabbitmq&quot; configured
    persistentvolumeclaim &quot;pvc-for-rabbitmq&quot; created
    $

I still get this for `pv`s, e.g. a PV is stuck in `Released` state:

    $
    kubectl get pv
    NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                             STORAGECLASS   REASON    AGE
    pv-for-rabbitmq                            5Gi        RWO            Delete           Released   staging/pvc-for-rabbitmq          standard                 15m
    $

...and I get this for `pvc`s:

    $
    kubectl get pvc
    NAME               STATUS    VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    pvc-for-rabbitmq   Pending   pv-for-rabbitmq   0                         standard       1m
    $

Looks like my PV is stuck in `Released` status, and PVC cannot access the PV which is not in `Available` status.

So, why the same PV and PVC cannot be friends again? **How do I make a PVC to regain access to data in the existing PV?**

What to do with Released persistent volume?

Currently, I am using the [@google-cloud/storage](https://www.npmjs.com/package/@google-cloud/storage) NPM package to upload a file directly to a Google Cloud Storage bucket. This requires some trickery as I only have the image&#39;s base64 encoded string. I have to:

- Decode the string
- Save it as a file
- Send the file path to the below script to upload to Google Cloud Storage
- Delete the local file

I&#39;d like to avoid storing the file in the filesystem altogether since I am using Google App Engine and I don&#39;t want to overload the filesystem / leave junk files there if the delete operation doesn&#39;t work for whatever reason. This is what my upload script looks like right now:

    // Convert the base64 string back to an image to upload into the Google Cloud Storage bucket
    var base64Img = require(&#39;base64-img&#39;);
    var filePath = base64Img.imgSync(req.body.base64Image, &#39;user-uploads&#39;, &#39;image-name&#39;);

    // Instantiate the GCP Storage instance
    var gcs = require(&#39;@google-cloud/storage&#39;)(),
        bucket = gcs.bucket(&#39;google-cloud-storage-bucket-name&#39;);

    // Upload the image to the bucket
    bucket.upload(__dirname.slice(0, -15) + filePath, {
        destination: &#39;profile-images/576dba00c1346abe12fb502a-original.jpg&#39;,
        public: true,
        validation: &#39;md5&#39;
    }, function(error, file) {

        if (error) {
            sails.log.error(error);
        }

        return res.ok(&#39;Image uploaded&#39;);
    });

Is there anyway to directly upload the base64 encoded string of the image instead of having to convert it to a file and then upload using the path?

How do I upload a base64 encoded image (string) directly to a Google Cloud Storage bucket using Node.js?

I am on a mac and am trying to install the Google Cloud SDK (including the gcloud command line utility) using this command in terminal

    curl https://sdk.cloud.google.com | bash
as seen at https://cloud.google.com/sdk/

It got all the way to the end and finished but even after I restarted my shell, the `gcloud` command still says it&#39;s not found.

Why isn&#39;t this installation working?


gcloud command not found - while installing Google Cloud SDK

I&#39;m trying to download files from Firebase Storage through a XMLHttpRequest, but Access-Control-Allow-Origin is not set on the resource, so it&#39;s not possible. Is there any way to set this header on the storage server?

      (let [xhr (js/XMLHttpRequest.)]
        (.open xhr &quot;GET&quot; url)
        (aset xhr &quot;responseType&quot; &quot;arraybuffer&quot;)
        (aset xhr &quot;onload&quot; #(js/console.log &quot;bin&quot; (.-response xhr)))
        (.send xhr)))

Chrome error message:

&gt; XMLHttpRequest cannot load
&gt; https://firebasestorage.googleapis.com/[EDITED]
&gt; No &#39;Access-Control-Allow-Origin&#39; header is present on the requested
&gt; resource. Origin &#39;http://localhost:3449&#39; is therefore not allowed
&gt; access.

Firebase Storage and Access-Control-Allow-Origin

I am logged in to a GCE instance via SSH. From there I would like to access the Storage with the help of a Service Account:

    GCE&gt; gcloud auth list
    Credentialed accounts:
     - 1234567890-compute@developer.gserviceaccount.com (active)

I first made sure that this Service account is flagged &quot;Can edit&quot; in the permissions of the project I am working in. I also made sure to give him the Write ACL on the bucket I would like him to copy a file:

    local&gt; gsutil acl ch -u 1234567890-compute@developer.gserviceaccount.com:W gs://mybucket

But then the following command fails:

    GCE&gt; gsutil cp test.txt gs://mybucket/logs

(I also made sure that &quot;logs&quot; is created under &quot;mybucket&quot;).

The error message I get is:

    Copying file://test.txt [Content-Type=text/plain]...
    AccessDeniedException: 403 Insufficient Permission               0 B  

What am I missing?


gsutil copy returning &quot;AccessDeniedException: 403 Insufficient Permission&quot; from GCE

I have 1000 files in Google Cloud Storage to make public, or in general a directory hierarchy.

In the Web UI, I can only seem to make one at a time public.

Is there some way to do this.

How do you make many files public in Google Cloud Storage?

I&#39;m working on the [Google Cloud Storage .NET client
library][1]. There are three features (between .NET, my client
library, and the Storage service) that are combining in an
unpleasant way:

- When downloading files (objects in Google Cloud Storage
terminology), the server includes a hash of the stored data. My
client code then validates that hash against the data it&#39;s
downloaded.

- A separate feature of Google Cloud Storage is that the user can
set the Content-Encoding of the object, and that&#39;s included as a
header when downloading, when the request contains a matching
Accept-Encoding. (For the moment, let&#39;s ignore the behavior when the
request doesn&#39;t include that...)

- `HttpClientHandler` can decompress gzip (or deflate) content
automatically and transparently.

When all three of these are combined, we get into trouble. Here&#39;s a
short but complete program demonstrating that, but without using my
client library (and hitting a publicly accessible file):

    using System;
    using System.Linq;
    using System.Net;
    using System.Net.Http;
    using System.Security.Cryptography;
    using System.Text;
    using System.Threading.Tasks;
    
    class Program
    {
        static async Task Main()
        {
            string url = &quot;https://www.googleapis.com/download/storage/v1/b/&quot;
                + &quot;storage-library-test-bucket/o/gzipped-text.txt?alt=media&quot;;
            var handler = new HttpClientHandler
            {
                AutomaticDecompression = DecompressionMethods.GZip
            };
            var client = new HttpClient(handler);
            
            var response = await client.GetAsync(url);
            byte[] content = await response.Content.ReadAsByteArrayAsync();
            string text = Encoding.UTF8.GetString(content);
            Console.WriteLine($&quot;Content: {text}&quot;);
            
            var hashHeader = response.Headers.GetValues(&quot;X-Goog-Hash&quot;).FirstOrDefault();
            Console.WriteLine($&quot;Hash header: {hashHeader}&quot;);
            
            using (var md5 = MD5.Create())
            {
                var md5Hash = md5.ComputeHash(content);
                var md5HashBase64 = Convert.ToBase64String(md5Hash);
                Console.WriteLine($&quot;MD5 of content: {md5HashBase64}&quot;);
            }
        }
    }

.NET Core project file:

    &lt;Project Sdk=&quot;Microsoft.NET.Sdk&quot;&gt;
      &lt;PropertyGroup&gt;
        &lt;OutputType&gt;Exe&lt;/OutputType&gt;
        &lt;TargetFramework&gt;netcoreapp2.0&lt;/TargetFramework&gt;
        &lt;LangVersion&gt;7.1&lt;/LangVersion&gt;
      &lt;/PropertyGroup&gt;
    &lt;/Project&gt;

Output:

    Content: hello world
    Hash header: crc32c=T1s5RQ==,md5=xhF4M6pNFRDQnvaRRNVnkA==
    MD5 of content: XrY7u+Ae7tCTyyK7j1rNww==

As you can see, the MD5 of the content isn&#39;t the same as the MD5
part of the `X-Goog-Hash` header. (In my client library I&#39;m using the crc32c
hash, but that shows the same behavior.)

This isn&#39;t a bug in `HttpClientHandler` - it&#39;s expected, but a pain
when I want to validate the hash. Basically, I need to at the
content before *and* after decompression. And I can&#39;t find any way
of doing that.

To clarify my requirements somewhat, I know how to prevent the decompression in `HttpClient` and instead decompress afterwards when reading from the stream - but I need to be able to do this without changing any the code that uses the resulting `HttpResponseMessage` from the `HttpClient`. (There&#39;s a lot of code that deals with responses, and I want to only make the change in one central place.)

I have a plan, which I&#39;ve prototyped and which works as far as I&#39;ve
found so far, but is a bit ugly. It involves creating a three-layer
handler:

- `HttpClientHandler` with automatic decompression disabled.
- A new handler which replaces the content stream with a new `Stream` subclass 
  which delegates to the original content stream, but hashes the data as it&#39;s read.
- A decompression-only handler, based on the Microsoft [`DecompressionHandler`][2] code.
  
While this works, it has disadvantages of:

- Open source licensing: checking exactly what I need to do in order
  to create a new file in my repo based on the MIT-licensed
  Microsoft code
- Effectively forking the MS code, which means I should probably
  make a regular check to see if any bugs have been found in it
- The Microsoft code uses internal members of the assembly, so it
  doesn&#39;t port as cleanly as it might.
  
If Microsoft made `DecompressionHandler` public, that would help a
lot - but that&#39;s likely to be in a longer timeframe than I need.

What I&#39;m looking for is an alternative approach if possible -
something I&#39;ve missed that lets me get at the content before
decompression. I don&#39;t want to reinvent `HttpClient` - the response
is often chunked for example, and I don&#39;t want to have to get into
that side of things. It&#39;s a pretty specific interception point that
I&#39;m looking for.

  [1]: https://googlecloudplatform.github.io/google-cloud-dotnet/docs/Google.Cloud.Storage.V1/index.html
  [2]: https://github.com/dotnet/corefx/blob/master/src/System.Net.Http/src/System/Net/Http/Managed/DecompressionHandler.cs


Is it possible to access the compressed data before decompression in HttpClient?

I am trying to create an internal app to upload files to google cloud. I don&#39;t want each individual user or this app to log in so I&#39;m using a service account. I login into the service account and everything is ok, but when I try to upload it gives me this error:
ServiceException: 401 Anonymous caller does not have storage.objects.list access to bucket

[![This is the error that I get][1]][1]


  [1]: https://i.stack.imgur.com/ELTTA.png



As you can see I am logged in with a service account and my account and(neither service or personal) works

gsutil ServiceException: 401 Anonymous caller does not have storage.objects.list access to bucket even though I&#39;m loggedin in gcloud

I am trying to read a csv file present on the Google Cloud Storage bucket onto a panda dataframe. 

    import pandas as pd
    import matplotlib.pyplot as plt
    import seaborn as sns
    %matplotlib inline
    from io import BytesIO

    from google.cloud import storage

    storage_client = storage.Client()
    bucket = storage_client.get_bucket(&#39;createbucket123&#39;)
    blob = bucket.blob(&#39;my.csv&#39;)
    path = &quot;gs://createbucket123/my.csv&quot;
    df = pd.read_csv(path)

It shows this error message:

    FileNotFoundError: File b&#39;gs://createbucket123/my.csv&#39; does not exist

What am I doing wrong, I am not able to find any solution which does not involve google datalab?

Read csv from Google Cloud storage to pandas dataframe

How can I upload a file to [Google Cloud Storage](https://cloud.google.com/storage/) from Python 3? Eventually Python 2, if it&#39;s infeasible from Python 3.

I&#39;ve looked and looked, but haven&#39;t found a solution that actually works. I tried [boto](https://github.com/boto/boto), but when I try to generate the necessary .boto file through `gsutil config -e`, it keeps saying that I need to configure authentication through `gcloud auth login`. However, I have done the latter a number of times, without it helping.


How to upload a file to Google Cloud Storage on Python 3?

When I attempt load data into BigQuery from Google Cloud Storage it asks for the Google Cloud Storage URI (gs://).  I have reviewed all of your online support as well as stackoverflow  and cannot find a way to identify the URL for my uploaded data via the browser based Google Developers Console.  The only way I see to find the URL is via gsutil and I have not been able to get gsutil to work on my machine.

Is there a way to determine the URL via the browser based Google Developers Console?


How do I identify the Google Cloud Storage URI from my Google Developers Console?

How can I move data directly from one Google Cloud Storage project to another?
I understand how to upload and how to download, but I want to transfer directly between projects.

All Google Cloud-Storage Solutions on Gang of Coders

Total of 18 Google Cloud-Storage Solutions