Use the [`Series.quantile()`][1] method:

    In [48]: cols = list(&#39;abc&#39;)
    
    In [49]: df = DataFrame(randn(10, len(cols)), columns=cols)

    In [50]: df.a.quantile(0.95)
    Out[50]: 1.5776961953820687
    
To filter out rows of `df` where `df.a` is greater than or equal to the 95th percentile do:

    In [72]: df[df.a &lt; df.a.quantile(.95)]
    Out[72]:
           a      b      c
    0 -1.044 -0.247 -1.149
    2  0.395  0.591  0.764
    3 -0.564 -2.059  0.232
    4 -0.707 -0.736 -1.345
    5  0.978 -0.099  0.521
    6 -0.974  0.272 -0.649
    7  1.228  0.619 -0.849
    8 -0.170  0.458 -0.515
    9  1.465  1.019  0.966


  [1]: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.quantile.html

numpy is much faster than Pandas for this kind of things :

    numpy.percentile(df.a,95) # attention : the percentile is given in percent (5 = 5%)
is equivalent but 3 times faster than :

    df.a.quantile(.95)  # as you already noticed here it is &quot;.95&quot; not &quot;95&quot;


so for your code, it gives :

    df[df.a &lt; np.percentile(df.a,95)]



You can use [query][1] for a more concise option:

    df.query(&#39;ms &lt; ms.quantile(.95)&#39;)


  [1]: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html

Is there a POSIX Compliant way to limit the scope of a variable to the function it is declared in? i.e.:

    Testing()
    {
        TEST=&quot;testing&quot;
    }

    Testing
    echo &quot;Test is: $TEST&quot;

should print &quot;Test is:&quot;. I&#39;ve read about the declare, local, and typeset keywords, but it doesn&#39;t look like they are required POSIX built-ins.

POSIX-Compliant Way to Scope Variables to a Function in a Shell Script

I just read [RabbitMQ&#39;s Java API docs](http://www.rabbitmq.com/api-guide.html), and found it very informative and straight-forward. The example for how to set up a simple `Channel` for publishing/consuming is very easy to follow and understand. But it&#39;s a very simple/basic example, and it left me with an important question: **How can I set up 1+ `Channels` to publish/consume to and from multiple queues?**

Let&#39;s say I have a RabbitMQ server with 3 queues on it: `logging`, `security_events` and `customer_orders`. So we&#39;d either need a single `Channel` to have the ability to publish/consume to all 3 queues, or more likely, have 3 separate `Channels`, each dedicated to a single queue.

On top of this, RabbitMQ&#39;s best practices dictate that we set up 1 `Channel` per consumer thread. For this example, let&#39;s say `security_events` is fine with only 1 consumer thread, but `logging` and `customer_order` both need 5 threads to handle the volume. So, if I understand correctly, does that mean we need:

* 1 `Channel` and 1 consumer thread for publishing/consuming to and from `security_events`; and
* 5 `Channels` and 5 consumer threads for publishing/consuming to and from `logging`; and
* 5 `Channels` and 5 consumer threads for publishing/consuming to and from `customer_orders`?

If my understanding is misguided here, please begin by correcting me. Either way, could some battle-weary RabbitMQ veteran **help me &quot;connect the dots&quot; with a decent code example for setting up publishers/consumers that meet my requirements here?** Thanks in advance!

RabbitMQ by Example: Multiple Threads, Channels and Queues

I have a pandas `DataFrame` called `data` with a column called `ms`.  I want to eliminate all the rows where `data.ms` is above the 95% percentile.  For now, I&#39;m doing this:

    limit = data.ms.describe(90)[&#39;95%&#39;]
    valid_data = data[data[&#39;ms&#39;] &lt; limit]

which works, but I want to generalize that to any percentile.  What&#39;s the best way to do that?


Eliminating all data over a given percentile

I have a pandas <code>DataFrame</code> called <code>data</code> with a column called <code>ms</code>. I want to eliminate all the rows where <code>data.ms</code> is above the 95% percentile. For now, I'm doing this:
<pre><code class="hljs language-ini">limit = data.ms.describe(90)['95%']
valid_data = data[data['ms'] &#x3C; limit]
</code></pre>
which works, but I want to generalize that to any percentile. What's the best way to do that?

I&#39;d like to pass a math operator, along with the numeric values to compare, to a function. Here is my broken code:

    def get_truth(inp,relate,cut):    
        if inp print(relate) cut:
            return True
        else:
            return False

and call it with

    get_truth(1.0,&#39;&gt;&#39;,0.0)

which should return True.

How to pass an operator to a python function?

What is the best way of establishing the sum of all counts in a `collections.Counter` object?

I&#39;ve tried:

`sum(Counter([1,2,3,4,5,1,2,1,6]))` 

but this gives **21** instead of **9**?







Sum of all counts in a collections.Counter

I have the following code:

    # initialize
    a = []
    
    # create the table (name, age, job)
    a.append([&quot;Nick&quot;, 30, &quot;Doctor&quot;])
    a.append([&quot;John&quot;,  8, &quot;Student&quot;])
    a.append([&quot;Paul&quot;, 22, &quot;Car Dealer&quot;])
    a.append([&quot;Mark&quot;, 66, &quot;Retired&quot;])    
    
    # sort the table by age
    import operator
    a.sort(key=operator.itemgetter(1))    
    
    # print the table
    print(a)

It creates a 4x3 table and then it sorts it by age. My question is, what exactly `key=operator.itemgetter(1)` does? Does the `operator.itemgetter` function return the item&#39;s value? Why can&#39;t I just type something like `key=a[x][1]` there? Or can I? How could with operator print a certain value of the form like `3x2` which is `22`?

1. How does exactly Python sort the table? Can I reverse-sort it?

2. How can I sort it based on two columns like first age, and then if age is the same b name?

3. How could I do it without `operator`?


How do operator.itemgetter() and sort() work?

I have a need to watch a log file for changes.  After looking through stackoverflow questions, I see people recommending `watchdog`.  So I&#39;m trying to test, and am not sure where to add the code for when files change:
```python
import time
from watchdog.observers import Observer
from watchdog.events import LoggingEventHandler

if __name__ == &quot;__main__&quot;:
    event_handler = LoggingEventHandler()
    observer = Observer()
    observer.schedule(event_handler, path=&#39;.&#39;, recursive=False)
    observer.start()
    try:
        while True:
            time.sleep(1)
        else:
            print &quot;got it&quot;
    except KeyboardInterrupt:
        observer.stop()
    observer.join()
```

Where do I add the &quot;got it&quot; — in the `while` loop if the files have been added/changed?


watchdog monitoring file for changes

I regularly perform pandas operations on data frames in excess of 15 million or so rows and I&#39;d love to have access to a progress indicator for particular operations.

Does a text based progress indicator for pandas split-apply-combine operations exist?

For example, in something like:

    df_users.groupby([&#39;userID&#39;, &#39;requestDate&#39;]).apply(feature_rollup)

where `feature_rollup` is a somewhat involved function that take many DF columns and creates new user columns through various methods.  These operations can take a while for large data frames so I&#39;d like to know if it is possible to have text based output in an iPython notebook that updates me on the progress.

So far, I&#39;ve tried canonical loop progress indicators for Python but they don&#39;t interact with pandas in any meaningful way.

I&#39;m hoping there&#39;s something I&#39;ve overlooked in the pandas library/documentation that allows one to know the progress of a split-apply-combine.  A simple implementation would maybe look at the total number of data frame subsets upon which the `apply` function is working and report progress as the completed fraction of those subsets.

Is this perhaps something that needs to be added to the library?

Progress indicator during pandas operations

I have a Pandas data frame object of shape (X,Y) that looks like this:

    [[1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]]

and a numpy sparse matrix (CSC) of shape (X,Z) that looks something like this

    [[0, 1, 0],
    [0, 0, 1],
    [1, 0, 0]]

How can I add the content from the matrix to the data frame in a new named column such that the data frame will end up like this:

    [[1, 2, 3, [0, 1, 0]],
    [4, 5, 6, [0, 0, 1]],
    [7, 8, 9, [1, 0, 0]]]

Notice the data frame now has shape (X, Y+1) and rows from the matrix are elements in the data frame.

Add numpy array as column to Pandas data frame

Can I insert a column at a specific column index in pandas? 

    import pandas as pd
    df = pd.DataFrame({&#39;l&#39;:[&#39;a&#39;,&#39;b&#39;,&#39;c&#39;,&#39;d&#39;], &#39;v&#39;:[1,2,1,2]})
    df[&#39;n&#39;] = 0

This will put column `n` as the last column of `df`, but isn&#39;t there a way to tell `df` to put `n` at the beginning?

how do I insert a column at a specific column index in pandas?

My numpy arrays use `np.nan` to designate missing values. As I iterate over the data set, I need to detect such missing values and handle them in special ways.

Naively I used `numpy.isnan(val)`, which works well unless `val` isn&#39;t among the subset of types supported by `numpy.isnan()`. For example, missing data can occur in string fields, in which case I get:

    &gt;&gt;&gt; np.isnan(&#39;some_string&#39;)
    Traceback (most recent call last):
      File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
    TypeError: Not implemented for this type

Other than writing an expensive wrapper that catches the exception and returns `False`, is there a way to handle this elegantly and efficiently?

Efficiently checking if arbitrary object is NaN in Python / numpy / pandas?

I&#39;ve got a pandas DataFrame filled mostly with real numbers, but there is a few `nan` values in it as well.

How can I replace the `nan`s with averages of columns where they are?

This question is very similar to this one: https://stackoverflow.com/questions/18689235/numpy-array-replace-nan-values-with-average-of-columns  but, unfortunately, the solution given there doesn&#39;t work for a pandas DataFrame.


pandas DataFrame: replace nan values with average of columns

I&#39;m trying to create a sub-table from another table of all the last name fields sorted A-Z which have a phone number field that isn&#39;t null. I could do this pretty easy with SQL, but I have no clue how to go about running a SQL query within Excel. I&#39;m tempted to import the data into postgresql and just query it there, but that seems a little excessive. 

For what I&#39;m trying to do, the SQL query `SELECT lastname, firstname, phonenumber WHERE phonenumber IS NOT NULL ORDER BY lastname` would do the trick. It seems too simple for it to be something that Excel can&#39;t do natively. How can I run a SQL query like this from within Excel?

How to run a SQL query on an Excel table?

I&#39;m working with a Boolean index in Pandas.

The question is why the statement:

    a[(a[&#39;some_column&#39;]==some_number) &amp; (a[&#39;some_other_column&#39;]==some_other_number)]

works fine whereas

    a[(a[&#39;some_column&#39;]==some_number) and (a[&#39;some_other_column&#39;]==some_other_number)]

exits with error?

Example:

    a = pd.DataFrame({&#39;x&#39;:[1,1],&#39;y&#39;:[10,20]})

    In: a[(a[&#39;x&#39;]==1)&amp;(a[&#39;y&#39;]==10)]
    Out:    x   y
         0  1  10

    In: a[(a[&#39;x&#39;]==1) and (a[&#39;y&#39;]==10)]
    Out: ValueError: The truth value of an array with more than one element is ambiguous.     Use a.any() or a.all()




Logical operators for Boolean indexing in Pandas

I have a Pandas DataFrame with a &#39;date&#39; column. Now I need to filter out all rows in the DataFrame that have dates outside of the next two months. Essentially, I only need to retain the rows that are within the next two months. 

What is the best way to achieve this?

Filtering Pandas DataFrames on dates

I have a pandas data frame with few columns.

Now I know that certain rows are outliers based on a certain column value.

For instance

&gt; column &#39;Vol&#39; has all values around `12xx` and one value is `4000` (outlier).

Now I would like to exclude those rows that have `Vol` column like this.

So, essentially I need to put a filter on the data frame such that we select all rows where the values of a certain column are within, say, 3 standard deviations from mean.

What is an elegant way to achieve this?

Detect and exclude outliers in a pandas DataFrame

I&#39;m a C coder developing something in python. I know how to do the following in C (and hence in C-like logic applied to python), but I&#39;m wondering what the &#39;Python&#39; way of doing it is.

I have a dictionary d, and I&#39;d like to operate on a subset of the items, only those who&#39;s key (string) contains a specific substring.

i.e. the C logic would be:

    for key in d:
        if filter_string in key:
            # do something
        else
            # do nothing, continue

I&#39;m imagining the python version would be something like

    filtered_dict = crazy_python_syntax(d, substring)
    for key,value in filtered_dict.iteritems():
        # do something


I&#39;ve found a lot of posts on here regarding filtering dictionaries, but couldn&#39;t find one which involved exactly this.

My dictionary is not nested and i&#39;m using python 2.7



Content Type	Original Author	Original Content on Stackoverflow
Question	Roy Smith	View Question on Stackoverflow
Solution 1 - Python	Phillip Cloud	View Answer on Stackoverflow
Solution 2 - Python	2diabolos.com	View Answer on Stackoverflow
Solution 3 - Python	hellpanderr	View Answer on Stackoverflow

Eliminating all data over a given percentile

Python Problem Overview

Python Solutions

Solution 1 - Python

Solution 2 - Python

Solution 3 - Python

RabbitMQ by Example: Multiple Threads, Channels and Queues

POSIX-Compliant Way to Scope Variables to a Function in a Shell Script

Attributions