What version of pandas are you using? For me your code works fine (i`m on git master).

Another approach could be:

    In [117]: import pandas

    In [118]: import random

    In [119]: df = pandas.DataFrame(np.random.randn(100, 4), columns=list(&#39;ABCD&#39;))

    In [120]: rows = random.sample(df.index, 10)

    In [121]: df_10 = df.ix[rows]

    In [122]: df_90 = df.drop(rows)

Newer version (from 0.16.1 on) supports this directly: 
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sample.html

I have found that `np.random.choice()` new in NumPy 1.7.0 works quite well for this.

For example you can pass the index values from a DataFrame and and the integer 10 to select 10 random uniformly sampled rows.

    rows = np.random.choice(df.index.values, 10)
    sampled_df = df.ix[rows]



New in version 0.16.1:

    sample_dataframe = your_dataframe.sample(n=how_many_rows_you_want)

doc here: http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.DataFrame.sample.html

Pandas 0.16.1 have a [sample][1] method for that.


  [1]: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sample.html

If you&#39;re using pandas.read_csv you can directly sample when loading the data, by using the skiprows parameter. Here is a short article I&#39;ve written on this - https://nikolaygrozev.wordpress.com/2015/06/16/fast-and-simple-sampling-in-pandas-when-loading-data-from-files/

I have a django application which has two models like this:

    class MyModel(models.Model):
        name = models.CharField()
        country = models.ForeignKey(&#39;Country&#39;)

    class Country(models.Model):
        code2 = models.CharField(max_length=2, primary_key=True)
        name = models.CharField()


The admin class for `MyModel` looks like this:

    class MyModelAdmin(admin.ModelAdmin):
        list_display = (&#39;name&#39;, &#39;country&#39;,)
        list_filter = (&#39;country&#39;,)
    admin.site.register(models.MyModel, MyModelAdmin)

The `Country` table contains ~250 countries. Only a handful of countries are actually referenced by some `MyModel` instance.

The problem is that the list filter in the django admin **lists ALL countries** in the filter panel. Listing all countries (and not just those that are referenced by an instance) pretty much defeats the purpose of having the list filter in this case.

Is there some to only display the countries referenced by `MyModel` as choices in the list filter? (I use Django 1.3.)

Can I make list_filter in django admin to only show referenced ForeignKeys?

I am trying to validate the password using regular expression. The password is getting updated if we have all the characters as alphabets. Where am i going wrong ? is the regular expression right ?

   

    function validatePassword() {
        var newPassword = document.getElementById(&#39;changePasswordForm&#39;).newPassword.value;
        var minNumberofChars = 6;
        var maxNumberofChars = 16;
        var regularExpression  = /^[a-zA-Z0-9!@#$%^&amp;*]{6,16}$/;
        alert(newPassword); 
        if(newPassword.length &lt; minNumberofChars || newPassword.length &gt; maxNumberofChars){
            return false;
        }
        if(!regularExpression.test(newPassword)) {
            alert(&quot;password should contain atleast one number and one special character&quot;);
            return false;
        }
    }

Javascript regular expression password validation having special characters

I&#39;m trying to read a fairly large CSV file with Pandas and split it up into two random chunks, one of which being 10% of the data and the other being 90%.

Here&#39;s my current attempt:

    rows = data.index
    row_count = len(rows)
    random.shuffle(list(rows))
    
    data.reindex(rows)
    
    training_data = data[row_count // 10:]
    testing_data = data[:row_count // 10]

For some reason, `sklearn` throws this error when I try to use one of these resulting DataFrame objects inside of a SVM classifier:

    IndexError: each subindex must be either a slice, an integer, Ellipsis, or newaxis

I think I&#39;m doing it wrong. Is there a better way to do this?

Pandas: Sampling a DataFrame

I'm trying to read a fairly large CSV file with Pandas and split it up into two random chunks, one of which being 10% of the data and the other being 90%.
Here's my current attempt:
<pre><code class="hljs language-ini">rows = data.index
row_count = len(rows)
random.shuffle(list(rows))

data.reindex(rows)

training_data = data[row_count // 10:]
testing_data = data[:row_count // 10]
</code></pre>
For some reason, <code>sklearn</code> throws this error when I try to use one of these resulting DataFrame objects inside of a SVM classifier:
<pre><code class="hljs language-vbnet">IndexError: each subindex must be either a slice, an integer, Ellipsis, or newaxis
</code></pre>
I think I'm doing it wrong. Is there a better way to do this?

Is it possible to do this on one line in Python?

    if &lt;condition&gt;:
        myList.append(&#39;myString&#39;)

I have tried the ternary operator:

    myList.append(&#39;myString&#39; if &lt;condition&gt;)

but my IDE ([MyEclipse][1]) didn&#39;t like it, without an `else`.


  [1]: https://www.genuitec.com/products/myeclipse/

Python Ternary Operator Without else

[numpy.amax()](http://docs.scipy.org/doc/numpy/reference/generated/numpy.amax.html#numpy.amax) will find the max value in an array, and [numpy.amin()](http://docs.scipy.org/doc/numpy/reference/generated/numpy.amin.html) does the same for the min value.  If I want to find both max and min, I have to call both functions, which requires passing over the (very big) array twice, which seems slow.

Is there a function in the numpy API that finds both max and min with only a single pass through the data?


NumPy: function for simultaneous max() and min()

I have the following data frame in IPython, where each row is a single stock:

    In [261]: bdata
    Out[261]:
    &lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;
    Int64Index: 21210 entries, 0 to 21209
    Data columns:
    BloombergTicker      21206  non-null values
    Company              21210  non-null values
    Country              21210  non-null values
    MarketCap            21210  non-null values
    PriceReturn          21210  non-null values
    SEDOL                21210  non-null values
    yearmonth            21210  non-null values
    dtypes: float64(2), int64(1), object(4)


I want to apply a groupby operation that computes cap-weighted average return across everything, per each date in the &quot;yearmonth&quot; column.

This works as expected:

    In [262]: bdata.groupby(&quot;yearmonth&quot;).apply(lambda x: (x[&quot;PriceReturn&quot;]*x[&quot;MarketCap&quot;]/x[&quot;MarketCap&quot;].sum()).sum())
    Out[262]:
    yearmonth
    201204      -0.109444
    201205      -0.290546

But then I want to sort of &quot;broadcast&quot; these values back to the indices in the original data frame, and save them as constant columns where the dates match.

    In [263]: dateGrps = bdata.groupby(&quot;yearmonth&quot;)
    
    In [264]: dateGrps[&quot;MarketReturn&quot;] = dateGrps.apply(lambda x: (x[&quot;PriceReturn&quot;]*x[&quot;MarketCap&quot;]/x[&quot;MarketCap&quot;].sum()).sum())
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    /mnt/bos-devrnd04/usr6/home/espears/ws/Research/Projects/python-util/src/util/&lt;ipython-input-264-4a68c8782426&gt; in &lt;module&gt;()
    ----&gt; 1 dateGrps[&quot;MarketReturn&quot;] = dateGrps.apply(lambda x: (x[&quot;PriceReturn&quot;]*x[&quot;MarketCap&quot;]/x[&quot;MarketCap&quot;].sum()).sum())
    
    TypeError: &#39;DataFrameGroupBy&#39; object does not support item assignment

I realize this naive assignment should not work. But what is the &quot;right&quot; Pandas idiom for assigning the result of a groupby operation into a new column on the parent dataframe?

In the end, I want a column called &quot;MarketReturn&quot; than will be a repeated constant value for all indices that have matching date with the output of the groupby operation.


One hack to achieve this would be the following:

    marketRetsByDate  = dateGrps.apply(lambda x: (x[&quot;PriceReturn&quot;]*x[&quot;MarketCap&quot;]/x[&quot;MarketCap&quot;].sum()).sum())
    
    bdata[&quot;MarketReturn&quot;] = np.repeat(np.NaN, len(bdata))
    
    for elem in marketRetsByDate.index.values:
        bdata[&quot;MarketReturn&quot;][bdata[&quot;yearmonth&quot;]==elem] = marketRetsByDate.ix[elem]

But this is slow, bad, and unPythonic.

Python Pandas How to assign groupby operation results back to columns in parent dataframe?

I&#39;m trying to use `matplotlib` to read in an RGB image and convert it to grayscale.

In matlab I use this:

    img = rgb2gray(imread(&#39;image.png&#39;));

In the [matplotlib tutorial][1] they don&#39;t cover it. They just read in the image

    import matplotlib.image as mpimg
    img = mpimg.imread(&#39;image.png&#39;)

and then they slice the array, but that&#39;s not the same thing as converting RGB to grayscale from what I understand.

    lum_img = img[:,:,0]

I find it hard to believe that numpy or matplotlib doesn&#39;t have a built-in function to convert from rgb to gray. Isn&#39;t this a common operation in image processing?

I wrote a very simple function that works with the image imported using `imread` in 5 minutes. It&#39;s horribly inefficient, but that&#39;s why I was hoping for a professional implementation built-in. 

Sebastian has improved my function, but I&#39;m still hoping to find the built-in one.

matlab&#39;s (NTSC/PAL) implementation:

    import numpy as np
            
    def rgb2gray(rgb):
    
        r, g, b = rgb[:,:,0], rgb[:,:,1], rgb[:,:,2]
        gray = 0.2989 * r + 0.5870 * g + 0.1140 * b
    
        return gray




  [1]: http://matplotlib.sourceforge.net/users/image_tutorial.html

How can I convert an RGB image into grayscale in Python?

For some reason my code is having trouble opening a simple file:

This is the code:

    file1 = open(&#39;recentlyUpdated.yaml&#39;)

And the error is: 

```none
IOError: [Errno 2] No such file or directory: &#39;recentlyUpdated.yaml&#39;
```

-  Naturally I checked that this is the correct name of the file.
-  I have tried moving around the file, giving `open()` the full path to the file and none of it seems to work.


open() gives FileNotFoundError/IOError: Errno 2 No such file or directory

In Kafka, I would like to use only a single broker, single topic and a single partition having one producer and multiple consumers (each consumer getting its own copy of data from the broker). Given this, I do not want the overhead of using Zookeeper; Can I not just use the broker only? Why is a Zookeeper must?

Is Zookeeper a must for Kafka?

I&#39;ve started using Spark SQL and DataFrames in Spark 1.4.0.  I&#39;m wanting to define a custom partitioner on DataFrames, in Scala, but not seeing how to do this.

One of the data tables I&#39;m working with contains a list of transactions, by account, silimar to the following example.

    Account   Date       Type       Amount
    1001    2014-04-01  Purchase    100.00
    1001    2014-04-01  Purchase     50.00
    1001    2014-04-05  Purchase     70.00
    1001    2014-04-01  Payment    -150.00
    1002    2014-04-01  Purchase     80.00
    1002    2014-04-02  Purchase     22.00
    1002    2014-04-04  Payment    -120.00
    1002    2014-04-04  Purchase     60.00
    1003    2014-04-02  Purchase    210.00
    1003    2014-04-03  Purchase     15.00

At least initially, most of the calculations will occur between the transactions within an account.  So I would want to have the data partitioned so that all of the transactions for an account are in the same Spark partition.

But I&#39;m not seeing a way to define this.  The DataFrame class has a method called &#39;repartition(Int)&#39;, where you can specify the number of partitions to create.  But I&#39;m not seeing any method available to define a custom partitioner for a DataFrame, such as can be specified for an RDD.

The source data is stored in Parquet.  I did see that when writing a DataFrame to Parquet, you can specify a column to partition by, so presumably I could tell Parquet to  partition it&#39;s data by the &#39;Account&#39; column.  But there could be millions of accounts, and if I&#39;m understanding Parquet correctly, it would create a distinct directory for each Account, so that didn&#39;t sound like a reasonable solution.

Is there a way to get Spark to partition this DataFrame so that all data for an Account is in the same partition?



How to define partitioning of DataFrame?

I read up on the documentation of [`HashPartitioner`][1]. Unfortunately nothing much was explained except for the API calls. I am under the assumption that `HashPartitioner` partitions the distributed set based on the hash of the keys. For example if my data is like 

    (1,1), (1,2), (1,3), (2,1), (2,2), (2,3)

So partitioner would put this into different partitions with same keys falling in the same partition. However I do not understand the significance of the constructor argument 

    new HashPartitoner(numPartitions) //What does numPartitions do?

For the above dataset how would the results differ if I did 

    new HashPartitoner(1)
    new HashPartitoner(2)
    new HashPartitoner(10)

So how does `HashPartitioner` work actually?

  [1]: http://spark.apache.org/docs/1.3.1/api/java/index.html?org/apache/spark/HashPartitioner.html

How does HashPartitioner work?

Sorry for the long post!

I have a database containing ~30 tables (InnoDB engine). Only two of these tables, namely, &quot;transaction&quot; and &quot;shift&quot; are quite large (the first one have 1.5 million rows and shift has 23k rows). Now everything works fine and I don&#39;t have problem with the current database size. 

However, we will have a similar database (same datatypes, design ,..) but much larger, e.g., the &quot;transaction&quot; table will have about **1 billion records** (about 2,3 million transaction per day) and we are thinking about how we should deal with such volume of data in MySQL? (it is both read and write intensive). I read a lot of related posts to see if Mysql (and more specifically InnoDB engine) can perform well with billions of records, but still I have some questions. Some of those related posts that I&#39;ve read are in the following:

 - [Can MySQL reasonably perform queries on billions of rows?][1]
 - [Is InnoDB (MySQL 5.5.8) the right choice for multi-billion rows?][2]
 - [Best data store for billions of rows][3]
 - [How big can a MySQL database get before performance starts to degrade][4]
 - [Why MySQL could be slow with large tables?][5]
 - [Can Mysql handle tables which will hold about 300 million records?][6]

What I&#39;ve understood so far to improve the performance for very large tables:

 1. (for innoDB tables which is my case) increasing the `innodb_buffer_pool_size` (e.g., up to 80% of RAM).
    Also, I found some other MySQL performance tunning settings [here in
    percona blog][7]
 2. having proper indexes on the table (using EXPLAN on queries)
 3. partitioning the table
 4. MySQL Sharding or clustering

Here are my questions/confusions:

 - About partitioning, I have some doubts whether we should use it or not. On one hand many people suggested it to improve performance when table is very large. On the other hand, I&#39;ve read many posts saying it does not improve query performance and it does not make queries run faster (e.g., [here][8] and [here][9]). Also, I read in [MySQL Reference Manual][10] that **InnoDB foreign keys and MySQL partitioning are not compatible** (we have foreign keys). 

 - Regarding indexes, right now they perform well, but as far as I understood, for very large tables indexing is more restrictive (as Kevin Bedell mentioned in his answer [here][1]). Also, indexes speed up reads while slow down write (insert/update). So, for the new similar project that we will have this large DB, should we first insert/load all the data and then create indexes? (to speed up the insert) 

 - If we cannot use partitioning for our big table (&quot;transaction&quot; table), what is an alternative option to improve the performance? (except MySQl variable settings such as `innodb_buffer_pool_size`). Should we use Mysql clusters? (we have also lots of joins)


&lt;h1&gt;EDIT&lt;/h1&gt;


This is the `show create table` statement for our largest table named &quot;transaction&quot;:  

      CREATE TABLE `transaction` (
     `id` int(11) NOT NULL AUTO_INCREMENT,
     `terminal_transaction_id` int(11) NOT NULL,
     `fuel_terminal_id` int(11) NOT NULL,
     `fuel_terminal_serial` int(11) NOT NULL,
     `xboard_id` int(11) NOT NULL,
     `gas_station_id` int(11) NOT NULL,
     `operator_id` text NOT NULL,
     `shift_id` int(11) NOT NULL,
     `xboard_total_counter` int(11) NOT NULL,
     `fuel_type` int(11) NOT NULL,
     `start_fuel_time` int(11) NOT NULL,
     `end_fuel_time` int(11) DEFAULT NULL,
     `preset_amount` int(11) NOT NULL,
     `actual_amount` int(11) DEFAULT NULL,
     `fuel_cost` int(11) DEFAULT NULL,
     `payment_cost` int(11) DEFAULT NULL,
     `purchase_type` int(11) NOT NULL,
     `payment_ref_id` text,
     `unit_fuel_price` int(11) NOT NULL,
     `fuel_status_id` int(11) DEFAULT NULL,
     `fuel_mode_id` int(11) NOT NULL,
     `payment_result` int(11) NOT NULL,
     `card_pan` text,
     `state` int(11) DEFAULT NULL,
     `totalizer` int(11) NOT NULL DEFAULT &#39;0&#39;,
     `shift_start_time` int(11) DEFAULT NULL,
     PRIMARY KEY (`id`),
     UNIQUE KEY `terminal_transaction_id` (`terminal_transaction_id`,`fuel_terminal_id`,`start_fuel_time`) USING BTREE,
     KEY `start_fuel_time_idx` (`start_fuel_time`),
     KEY `fuel_terminal_idx` (`fuel_terminal_id`),
     KEY `xboard_idx` (`xboard_id`),
     KEY `gas_station_id` (`gas_station_id`) USING BTREE,
     KEY `purchase_type` (`purchase_type`) USING BTREE,
     KEY `shift_start_time` (`shift_start_time`) USING BTREE,
     KEY `fuel_type` (`fuel_type`) USING BTREE
    ) ENGINE=InnoDB AUTO_INCREMENT=1665335 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT

Thanks for your time,
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        


  [1]: https://dba.stackexchange.com/questions/20335/can-mysql-reasonably-perform-queries-on-billions-of-rows
  [2]: https://stackoverflow.com/questions/6121905/is-innodb-mysql-5-5-8-the-right-choice-for-multi-billion-rows
  [3]: https://stackoverflow.com/questions/2794736/best-data-store-for-billions-of-rows?noredirect=1&amp;lq=1
  [4]: https://stackoverflow.com/questions/1276/how-big-can-a-mysql-database-get-before-performance-starts-to-degrade
  [5]: https://www.percona.com/blog/2006/06/09/why-mysql-could-be-slow-with-large-tables/
  [6]: https://stackoverflow.com/questions/14733462/can-mysql-handle-tables-which-will-hold-about-300-million-records?noredirect=1&amp;lq=1
  [7]: https://www.percona.com/blog/2014/01/28/10-mysql-performance-tuning-settings-after-installation/
  [8]: https://stackoverflow.com/questions/4249073/partitioning-for-query-performance-in-sql-server-2008/4249447#4249447
  [9]: https://dba.stackexchange.com/questions/141175/partitioning-a-large-table-has-not-improved-performance-why
  [10]: https://dev.mysql.com/doc/refman/5.6/en/partitioning-limitations-storage-engines.html

Handling very large data with mysql

One last newbie pandas question for the day:  How do I generate a table for a single Series?

For example:

    my_series = pandas.Series([1,2,2,3,3,3])
    pandas.magical_frequency_function( my_series )

    &gt;&gt; {
         1 : 1,
         2 : 2, 
         3 : 3
       }

Lots of googling has led me to Series.describe() and pandas.crosstabs, but neither of these does quite what I need: one variable, counts by categories.  Oh, and it&#39;d be nice if it worked for different data types: strings, ints, etc.



Frequency table for a single variable

I&#39;m currently using pandas to read an Excel file and present its sheet names to the user, so he can select which sheet he would like to use. The problem is that the files are really big (70 columns x 65k rows), taking up to 14s to load on a notebook (the same data in a CSV file is taking 3s).

My code in panda goes like this:

    xls = pandas.ExcelFile(path)
    sheets = xls.sheet_names

I tried xlrd before, but obtained similar results. This was my code with xlrd:

    xls = xlrd.open_workbook(path)
    sheets = xls.sheet_names

So, can anybody suggest a faster way to retrieve the sheet names from an Excel file than reading the whole file?

How to obtain sheet names from XLS files without loading the whole file?

I have a dataframe generated from Python&#39;s Pandas package. How can I generate  heatmap using DataFrame from pandas package. 


    import numpy as np 
    from pandas import *

    Index= [&#39;aaa&#39;,&#39;bbb&#39;,&#39;ccc&#39;,&#39;ddd&#39;,&#39;eee&#39;]
    Cols = [&#39;A&#39;, &#39;B&#39;, &#39;C&#39;,&#39;D&#39;]
    df = DataFrame(abs(np.random.randn(5, 4)), index= Index, columns=Cols)

    &gt;&gt;&gt; df
              A         B         C         D
    aaa  2.431645  1.248688  0.267648  0.613826
    bbb  0.809296  1.671020  1.564420  0.347662
    ccc  1.501939  1.126518  0.702019  1.596048
    ddd  0.137160  0.147368  1.504663  0.202822
    eee  0.134540  3.708104  0.309097  1.641090
    &gt;&gt;&gt; 






Making heatmap from pandas DataFrame

Assume I have a pandas DataFrame with two columns, A and B. I&#39;d like to modify this DataFrame (or create a copy) so that B is always NaN whenever A is 0. How would I achieve that?

I tried the following

    df[&#39;A&#39;==0][&#39;B&#39;] = np.nan

and

    df[&#39;A&#39;==0][&#39;B&#39;].values.fill(np.nan)

without success.


Content Type	Original Author	Original Content on Stackoverflow
Question	Blender	View Question on Stackoverflow
Solution 1 - Python	Wouter Overmeire	View Answer on Stackoverflow
Solution 2 - Python	dragoljub	View Answer on Stackoverflow
Solution 3 - Python	dval	View Answer on Stackoverflow
Solution 4 - Python	hurrial	View Answer on Stackoverflow
Solution 5 - Python	Nikolay	View Answer on Stackoverflow

Pandas: Sampling a DataFrame

Python Problem Overview

Python Solutions

Solution 1 - Python

Solution 2 - Python

Solution 3 - Python

Solution 4 - Python

Solution 5 - Python

Javascript regular expression password validation having special characters

Can I make list_filter in django admin to only show referenced ForeignKeys?

Attributions