I am reading some values in data attribute fields. I have seen two easy ways to read the data as shown below:

    var webappData = document.getElementById(&#39;web-app-data&#39;),
        rating = webappData.dataset.rating;

OR

    var effectData = $(&#39;.effects-list li&#39;),
        creative = effectData.filter(&#39;[data-creative]&#39;).data(&quot;creative&quot;);

My question is which of these has better performance or do they really differ?

I have a page with many data attributes that I am accessing and I would like to use the method that has the best performance. 

Any guidance on understanding the difference between the two would be appreciated. While I am looking at performance specifically if there are other reasons to use one over the other I would like to know this as well. 

dataset vs .data - Difference?

Is it possible to use any datasets available via the `kaggle` API in Google Colab? I see the Kaggle API is used in [this Colab notebook][1], but it&#39;s a bit unclear to me what datasets it provides access to. 


  [1]: https://colab.research.google.com/drive/1eufc8aNCdjHbrBhuy7M7X6BGyzAyRbrF
 

Using Kaggle Datasets in Google Colab

What is the most direct route to get a DataSet if I have a sql command?

    string sqlCommand = &quot;SELECT * FROM TABLE&quot;;
    string connectionString = &quot;blahblah&quot;;
    
    DataSet = GetDataSet(sqlCommand,connectionString);
    
    GetDataSet()
    {
       //...?
    }

I started with `SqlConnection` and `SqlCommand`, but the closest thing I see in the API is `SqlCommand.ExecuteReader()`.  With this method, I&#39;ll need to get a `SqlDataReader` and then convert this to a `DataSet` manually.  I figure there is a more direct route to accomplish the task.

If easier, a `DataTable` will also fit my goal.

Direct method from SQL command text to DataSet

I&#39;m modifying someone else&#39;s code where a query is performed using the following:

    DataSet ds = new DataSet();
    SqlDataAdapter da = new SqlDataAdapter(sqlString, sqlConn);
    da.Fill(ds);

How can I tell if the DataSet is empty (i.e. no results were returned)?

How to test if a DataSet is empty?

I have an Excel worksheet I want to read into a datatable - all is well except for one particular column in my Excel sheet. The column, &#39;ProductID&#39;, is a mix of values like `##########` and `n#########`.

I tried to [let OleDB handle everything by itself automatically][1] by reading it into a dataset/datatable, but any values in &#39;ProductID&#39; like `n######` are missing, ignored, and left blank.  I tried manually creating my DataTable by looping through each row with a datareader, but with the exact same results.

Here&#39;s the code :

&lt;!-- language: lang-cs --&gt;

    // add the column names manually to the datatable as column_1, column_2, ...
    for (colnum = 0; colnum &lt; num_columns; colnum ++){
      ds.Tables[&quot;products&quot;].Columns.Add(&quot;column_&quot; +colnum , System.Type.GetType(&quot;System.String&quot;)); 
    }
    while(myDataReader.Read()){
      // loop through each excel row adding a new respective datarow to my datatable 
      DataRow a_row = ds.Tables[&quot;products&quot;].NewRow();
      for (col = 0; col &lt; num_columns; col ++){
        try {  a_row[col] = rdr.GetString(col);  }
        catch {  a_row[col] = rdr.GetValue(col).ToString(); }
      }
      ds.Tables[&quot;products&quot;].Rows.Add(a_row);
    }

I don&#39;t understand why it won&#39;t let me read in values like `n######`. How can I do this? 

  [1]: http://support.microsoft.com/kb/306572

OleDB &amp; mixed Excel datatypes : missing data

Can someone please help how to get the list of built-in data sets and their dependency packages?

How do I get a list of built-in data sets in R?

DataSet and DataTable both implement IDisposable, so, by conventional best practices, I should call their Dispose() methods.

However, from what I&#39;ve read so far, DataSet and DataTable don&#39;t actually have any unmanaged resources, so Dispose() doesn&#39;t actually do much.

Plus, I can&#39;t just use `using(DataSet myDataSet...)` because DataSet has a collection of DataTables.

So, to be safe, I&#39;d need to iterate through myDataSet.Tables, dispose of each of the DataTables, then dispose of the DataSet.

So, is it worth the hassle to call Dispose() on all of my DataSets and DataTables?

**Addendum:**

For those of you who think that DataSet should be disposed:
In general, the pattern for disposing is to use `using` or `try..finally`, because you want to guarantee that Dispose() will be called.

However, this gets ugly real fast for a collection.  For example, what do you do if one of the calls to Dispose() thrown an exception?  Do you swallow it (which is &quot;bad&quot;) so that you can continue on to dispose the next element?

Or, do you suggest that I just call myDataSet.Dispose(), and forget about disposing the DataTables in myDataSet.Tables? 

Should I Dispose() DataSet and DataTable?

How do I convert data from a Scikit-learn Bunch object to a Pandas DataFrame?

    from sklearn.datasets import load_iris
    import pandas as pd
    data = load_iris()
    print(type(data))
    data1 = pd. # Is there a Pandas method to accomplish this?


How to convert a Scikit-learn dataset to a Pandas dataset

I currently use a DataTable to get results from a database which I can use in my code.

However, many example on the web show using a DataSet instead and accessing the table(s) through the collections method.

Is there any advantage, performance wise or otherwise, of using DataSets or DataTables as a storage method for SQL results?

Datatable vs Dataset

I am finding it hard to understand the process of Naive Bayes, and I was wondering if someone could explain it with a simple step by step process in English. I understand it takes comparisons by times occurred as a probability, but I have no idea how the training data is related to the actual dataset.

Please give me an explanation of what role the training set plays. I am giving a very simple example for fruits here, like banana for example

    training set---
    round-red
    round-orange
    oblong-yellow
    round-red
    
    dataset----
    round-red
    round-orange
    round-red
    round-orange
    oblong-yellow
    round-red
    round-orange
    oblong-yellow
    oblong-yellow
    round-red

A simple explanation of Naive Bayes Classification

When using R it&#39;s handy to load &quot;practice&quot; datasets using 

    data(iris)
or

    data(mtcars)

Is there something similar for Pandas? I know I can load using any other method, just curious if there&#39;s anything builtin.

Sample datasets in Pandas

This is possibly a simple question, but I do not know how to order columns alphabetically.

    test = data.frame(C = c(0, 2, 4, 7, 8), A = c(4, 2, 4, 7, 8), B = c(1, 3, 8, 3, 2))
    
    #   C A B
    # 1 0 4 1
    # 2 2 2 3
    # 3 4 4 8
    # 4 7 7 3
    # 5 8 8 2

I like to order the columns by column names alphabetically, to achieve 
    
    #   A B C
    # 1 4 1 0
    # 2 2 3 2
    # 3 4 8 4
    # 4 7 3 7
    # 5 8 2 8

For others I want my own defined order:

    #   B A C
    # 1 4 1 0
    # 2 2 3 2
    # 3 4 8 4
    # 4 7 3 7
    # 5 8 2 8

Please note that my datasets are huge, with 10000 variables. So the process needs to be more automated.

Sort columns of a dataframe by column name

I&#39;ve been working for quite a while now with LINQ. However, it remains a bit of a mystery what the real differences are between the mentioned flavours of LINQ. 

**The successful answer will contain a short differentiation between them. What is the main goal of each flavor, what is the benefit, and is there a performance impact...**

P.S.
I know that there are a lot of information sources out there, but I&#39;m looking for a kind of a &quot;cheat sheet&quot; which instructs a newbie where to head for a specific goal.

What is the difference between &quot;LINQ to Entities&quot;, &quot;LINQ to SQL&quot; and &quot;LINQ to Dataset&quot;

I have a dataset with 11 columns with over a 1000 rows each. The columns were labeled V1, V2, V11, etc..
I replaced the names with something more useful to me using the &quot;c&quot; command.
I didn&#39;t realize that row 1 also contained labels for each column and my actual data starts on row 2.

Is there a way to delete row 1 and decrement?

How to delete the first row of a dataframe in R?

I&#39;m just getting started using ADO.NET and DataSets and DataTables. One problem I&#39;m having is it seems pretty hard to tell what values are in the data table when trying to debug.

What are some of the easiest ways of quickly seeing what values have been saved in a DataTable? Is there someway to see the contents in Visual Studio while debugging or is the only option to write the data out to a file?

I&#39;ve created a little utility function that will write a DataTable out to a CSV file. Yet the the resulting CSV file created was cut off. About 3 lines from what should have been the last line in the middle of writing out a System.Guid the file just stops. I can&#39;t tell if this is an issue with my CSV conversion method, or the original population of the DataTable.

**Update**

Forget the last part I just forgot to flush my stream writer.



How to view a DataTable while debugging

I use a DataTable with Information about Users and I want search a user or a list of users in this DataTable. I try it butit don&#39;t work :( 

Here is my c# code:

     public DataTable GetEntriesBySearch(string username,string location,DataTable table)
            {
                list = null;
                list = table;
    
                string expression;
                string sortOrder;
    
                expression = &quot;Nachname = &#39;test&#39;&quot;;
                sortOrder = &quot;nachname DESC&quot;;
    
                DataRow[] rows =  list.Select(expression, sortOrder);
    
                list = null; // for testing
                list = new DataTable(); // for testing
     
                foreach (DataRow row in rows)
                {
                    list.ImportRow(row);
                }
    
                return list; 
            }

How I can filter a Datatable?

Say I have large datasets in R and I just want to know whether two of them they are the same. I use this often when I&#39;m experimenting different algorithms to achieve the same result. For example, say we have the following datasets:

    df1 &lt;- data.frame(num = 1:5, let = letters[1:5])
    df2 &lt;- df1
    df3 &lt;- data.frame(num = c(1:5, NA), let = letters[1:6])
    df4 &lt;- df3

So this is what I do to compare them:

    table(x == y, useNA = &#39;ifany&#39;)

Which works great when the datasets have no NAs:

    &gt; table(df1 == df2, useNA = &#39;ifany&#39;)
    TRUE 
      10 

But not so much when they have NAs:

    &gt; table(df3 == df4, useNA = &#39;ifany&#39;)
    TRUE &lt;NA&gt; 
      11    1 

In the example, it&#39;s easy to dismiss the `NA` as not a problem since we know that both dataframes *are* equal. The problem is that `NA == &lt;anything&gt;` yields `NA`, so whenever one of the datasets has an `NA`, it doesn&#39;t matter what the other one has on that same position, the result is always going to be `NA`.

So using `table()` to compare datasets doesn&#39;t seem ideal to me. **How can I better check if two data frames are identical?**

P.S.: Note this is not a duplicate of https://stackoverflow.com/questions/12567524/r-comparing-several-datasets, https://stackoverflow.com/questions/7459138/comparing-2-datasets-in-r or https://stackoverflow.com/questions/8872333/compare-datasets-in-r

How to check if two data frames are equal

At the moment, when I iterate over the *DataRow* instances, I do this.

    foreach(DataRow row in table)
      return yield new Thingy { Name = row[&quot;hazaa&quot;] };

Sooner of later (i.e. sooner), I&#39;ll get the *table* to be missing the column *donkey* and the poo will hit the fan. After some extensive googling (about 30 seconds) I discovered the following protection syntax.

    foreach(DataRow row in table)
      if(row.Table.Columns.Contains(&quot;donkey&quot;))
        return yield new Thingy { Name = row[&quot;hazaa&quot;] };
      else
        return null;

Now - is this the simplest syntax?! Really? I was expecting a method that gets me the field if it exists or *null* otherwise. Or at least a *Contains* method directly on the *row*.

Am I missing something? I&#39;ll be mapping in many fields that way so the code will look dreadfully unreadable...



Best practice to check if DataRow contains a certain column

I&#39;m currently learning TensorFlow but I came across a confusion in the below code snippet:
   
    dataset = dataset.shuffle(buffer_size = 10 * batch_size) 
    dataset = dataset.repeat(num_epochs).batch(batch_size)
    return dataset.make_one_shot_iterator().get_next()

I know that first the dataset will hold all the data but what `shuffle()`,`repeat()`, and `batch()` do to the dataset? 
Please help me with an example and explanation.

What does batch, repeat, and shuffle do with TensorFlow Dataset?

I am a little bit confused about the data augmentation performed in PyTorch. Now, as far as I know, when we are performing data augmentation, we are KEEPING our original dataset, and then adding other versions of it (Flipping, Cropping...etc). But that doesn&#39;t seem like happening in PyTorch. As far as I understood from the references, when we use `data.transforms` in PyTorch, then it applies them one by one. So for example:

&lt;!-- language: lang-python --&gt;

    data_transforms = {
        &#39;train&#39;: transforms.Compose([
            transforms.RandomResizedCrop(224),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]),
        &#39;val&#39;: transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]),
    }

Here , for the training, we are first randomly cropping the image and resizing it to shape `(224,224)`. Then we are taking these `(224,224)` images and horizontally flipping them. Therefore, our dataset is now containing ONLY the horizontally flipped images, so our original images are lost in this case. 

Am I right? Is this understanding correct? If not, then where do we tell PyTorch in this code above (taken from Official Documentation) to keep the original images and resize them to the expected shape `(224,224)`?

Thanks 
 

Data Augmentation in PyTorch

I am looking for some large public datasets, in particular:

1. Large sample web server logs that have been anonymized.

2. Datasets used for database performance benchmarking.

Any other links to large public datasets would be appreciated.  I already know about Amazon&#39;s public datasets at:  http://aws.amazon.com/publicdatasets/



Large public datasets?

I have an asp.net application, and now I am using datasets for data manipulation. I recently started to convert this dataset to a List collection. But, in some places it doesn&#39;t work. One is that in my old version I am using `datarow[] drow =  dataset.datatable.select(searchcriteria)`. But in the List collection there is no method available for finding particular values. Is there any way for me to select some values according with my search criteria? I want to know if this is possible. Please help me.


Select method in List&lt;t&gt; Collection

Here is my c# code

    Employee objEmp = new Employee();
    List&lt;Employee&gt; empList = new List&lt;Employee&gt;();
    foreach (DataRow dr in ds.Tables[0].Rows)
    {
        empList.Add(new Employee { Name = Convert.ToString(dr[&quot;Name&quot;]), Age = Convert.ToInt32(dr[&quot;Age&quot;]) });
    }

It uses a loop to create a List from a dataset.Is there any direct method or shorter method or one line code to convert dataset to list

Convert DataSet to List

Just having some problems running a simulation on some weather data in Python. The data was supplied in a .tif format, so I used the following code to try to open the image to extract the data into a numpy array.

    from PIL import Image
    
    im = Image.open(&#39;jan.tif&#39;)

But when I run this code I get the following error:

    PIL.Image.DecompressionBombError: Image size (933120000 pixels) exceeds limit of 178956970 pixels, could be decompression bomb DOS attack.

It looks like this is just some kind of protection against this type of attack, but I actually need the data and it is from a reputable source. Is there any way to get around this or do I have to look for another way to do this?


Pillow in Python won&#39;t let me open image (&quot;exceeds limit&quot;)

I was trying to generate a Report using Export to Excell, PDF, TextFile. Well I am doing this in MVC. I have a class which I named SPBatch (which is the exact name of my Stored Procedure in my SQL) and it contains the following:

    public string BatchNo { get; set; }
    public string ProviderName { get; set; }
    public Nullable&lt;System.Int32&gt; NoOfClaims { get; set; }
    public Nullable&lt;System.Int32&gt; TotalNoOfClaims { get; set; }
    public Nullable&lt;System.Decimal&gt; TotalBilled { get; set; }
    public Nullable&lt;System.Decimal&gt; TotalInputtedBill { get; set; }
    public Nullable&lt;System.DateTime&gt; DateCreated { get; set; }
    public Nullable&lt;System.DateTime&gt; DateSubmitted { get; set; }
    public Nullable&lt;System.DateTime&gt; DueDate { get; set; }
    public string Status { get; set; }
    public string RefNo { get; set; }
    public string BatchStatus { get; set; }
    public string ClaimType { get; set; }

as you can see some of my Columns are declared as Nullable. It went smoothly from searching and displaying the results in a table. I have several buttons below which are image buttons for export and every time I try to export in Excel, I always get the problem &quot;**DataSet does not support System.Nullable&lt;&gt;**&quot; in this part of my code:

    foreach (MemberInfo mi in miArray)
    {
        if (mi.MemberType == MemberTypes.Property)
        {
            PropertyInfo pi = mi as PropertyInfo;
            dt.Columns.Add(pi.Name, pi.PropertyType); //where the error pop&#39;s up.
    
        }
        else if (mi.MemberType == MemberTypes.Field)
        {
            FieldInfo fi = mi as FieldInfo;
            dt.Columns.Add(fi.Name, fi.FieldType);
        }
    }

the error shows up on the one with a comment. Can you help me what to do? I tried adding  DBNull in my code but still I get the same error. I tried removing Nullable in my SPBatch but I get an error that some tables are need to be declared as Nullable. 

What should I do?


DataSet does not support System.Nullable&lt;&gt; in Export

I am reading an XML file into a DataSet and need to get the data out of the DataSet. Since it is a user-editable config file the fields may or may not be there. To handle missing fields well I&#39;d like to make sure each column in the DataRow exists and is not DBNull. 

I already check for DBNull but I don&#39;t know how to make sure the column exists without having it throw an exception or using a function that loops over all the column names. What is the best method to do this?

How do I find out if a column exists in a VB.Net DataRow

I&#39;ve been working with data imported from a CSV. Pandas changed some columns to float, so now the numbers in these columns get displayed as floating points! However, I need them to be displayed as integers or without comma. Is there a way to convert them to integers or not display the comma?



All Dataset Solutions on Gang of Coders

Total of 27 Dataset Solutions