imploding a list for use in a python MySQLDB IN clause

PythonMysql

Python Problem Overview


I know how to map a list to a string:

foostring = ",".join( map(str, list_of_ids) )

And I know that I can use the following to get that string into an IN clause:

cursor.execute("DELETE FROM foo.bar WHERE baz IN ('%s')" % (foostring))

What I need is to accomplish the same thing SAFELY (avoiding SQL injection) using MySQLDB. In the above example because foostring is not passed as an argument to execute, it is vulnerable. I also have to quote and escape outside of the mysql library.

(There is a related SO question, but the answers listed there either do not work for MySQLDB or are vulnerable to SQL injection.)

Python Solutions


Solution 1 - Python

Use the list_of_ids directly:

format_strings = ','.join(['%s'] * len(list_of_ids))
cursor.execute("DELETE FROM foo.bar WHERE baz IN (%s)" % format_strings,
                tuple(list_of_ids))

That way you avoid having to quote yourself, and avoid all kinds of sql injection.

Note that the data (list_of_ids) is going directly to mysql's driver, as a parameter (not in the query text) so there is no injection. You can leave any chars you want in the string, no need to remove or quote chars.

Solution 2 - Python

Though this question is quite old, thought it would be better to leave a response in case someone else was looking for what I wanted

Accepted answer gets messy when we have a lot of the params or if we want to use named parameters

After some trials

ids = [5, 3, ...]  # list of ids
cursor.execute('''
SELECT 
...
WHERE
  id IN %(ids)s
  AND created_at > %(start_dt)s
''', {
  'ids': tuple(ids), 'start_dt': '2019-10-31 00:00:00'
})

Tested with python2.7, pymysql==0.7.11

Solution 3 - Python

This appears to still be a problem with Python3 in 2021, as pointed out in the comment by Rubms to the answer by markk.

Adding about 9 lines of code to the method "_process_params_dict" in "cursor.py" in the mysql connector package to handle tuples solved the problem for me:

def _process_params_dict(self, params):
    """Process query parameters given as dictionary"""
    try:
        to_mysql = self._connection.converter.to_mysql
        escape = self._connection.converter.escape
        quote = self._connection.converter.quote
        res = {}
        for key, value in list(params.items()):
            if type(value) is tuple: ### BEGIN MY ADDITIONS
                res[key.encode()] = b''
                for subvalue in value:
                    conv = subvalue
                    conv = to_mysql(conv)
                    conv = escape(conv)
                    conv = quote(conv)
                    res[key.encode()] = res[key.encode()] + b',' + conv if len(res[key.encode()]) else conv
            else: ### END MY ADDITIONS
                conv = value
                conv = to_mysql(conv)
                conv = escape(conv)
                conv = quote(conv)
                res[key.encode()] = conv
    except Exception as err:
        raise errors.ProgrammingError(
            "Failed processing pyformat-parameters; %s" % err)
    else:
        return res

Solution 4 - Python

If you use Django 2.0 or 2.1 and Python 3.6, this is the right way:

from django.db import connection
RESULT_COLS = ['col1', 'col2', 'col3']
RESULT_COLS_STR = ', '.join(['a.'+'`'+i+'`' for i in RESULT_COLS])
QUERY_INDEX = RESULT_COLS[0]

TABLE_NAME = 'test'
search_value = ['ab', 'cd', 'ef']  # <-- a list
query = (
    f'SELECT DISTINCT {RESULT_COLS_STR} FROM {TABLE_NAME} a '
    f'WHERE a.`{RESULT_COLS[0]}` IN %s '
    f'ORDER BY a.`{RESULT_COLS[0]}`;'
)  # <- 'SELECT DISTINCT a.`col1`, a.`col2`, a.`col3` FROM test a WHERE a.`col1` IN %s ORDER BY a.`col1`;'
with connection.cursor() as cursor:
    cursor.execute(query, params=[search_value])  # params is a list with a list as its element

ref: https://stackoverflow.com/a/23891759/2803344 https://docs.djangoproject.com/en/2.1/topics/db/sql/#passing-parameters-into-raw

Solution 5 - Python

list_of_ids = [ 1, 2, 3]
query = "select * from table where x in %s" % str(tuple(list_of_ids))
print query

This could work for some use-cases if you don't wish to be concerned with the method in which you have to pass arguments to complete the query string and would like to invoke just cursror.execute(query).

Another way could be:

"select * from table where x in (%s)" % ', '.join(str(id) for id in list_of_ids)

Solution 6 - Python

Another simple solution using list comprehension:

# creating a new list of strings and convert to tuple
sql_list = tuple([ key.encode("UTF-8") for key in list_of_ids ])

# replace "{}" with "('id1','id2',...'idlast')"
cursor.execute("DELETE FROM foo.bar WHERE baz IN {}".format(sql_list))

Solution 7 - Python

Though this question is quite old. I am sharing my solution if it can help someone.

list_to_check = ['A', 'B'] cursor.execute("DELETE FROM foo.bar WHERE baz IN ({})".format(str(list_to_check)[1:-1])

Tested with Python=3.6

Solution 8 - Python

Very simple: Just use the below formation

rules_id = ["9","10"]

sql1 = "SELECT * FROM attendance_rules_staff WHERE id in("+", ".join(map(str, rules_id))+")"

", ".join(map(str, rules_id))

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionmluebkeView Question on Stackoverflow
Solution 1 - PythonnoskloView Answer on Stackoverflow
Solution 2 - PythonmarkkView Answer on Stackoverflow
Solution 3 - PythonPhilipView Answer on Stackoverflow
Solution 4 - PythonBelterView Answer on Stackoverflow
Solution 5 - PythonAnurag NileshView Answer on Stackoverflow
Solution 6 - PythonchenchukView Answer on Stackoverflow
Solution 7 - PythonAditya SahuView Answer on Stackoverflow
Solution 8 - PythonMizanur RahmanView Answer on Stackoverflow