python-pandas和类似mysql的数据库

问题:python-pandas和类似mysql的数据库

Pandas的文档提供了许多使用各种格式存储的最佳实践示例。

但是,我找不到用于处理像MySQL这样的数据库的任何好的示例。

谁能指向我链接或提供一些代码片段,说明如何使用mysql-python将查询结果有效地转换为Pandas中的数据帧?

The documentation for Pandas has numerous examples of best practices for working with data stored in various formats.

However, I am unable to find any good examples for working with databases like MySQL for example.

Can anyone point me to links or give some code snippets of how to convert query results using mysql-python to data frames in Pandas efficiently ?


回答 0

正如Wes所说,一旦使用DBI兼容库建立了数据库连接,io / sql的read_sql就可以完成。我们可以看两个使用MySQLdbcx_Oracle库连接到Oracle和MySQL并查询其数据字典的简短示例。这是以下示例cx_Oracle

import pandas as pd
import cx_Oracle

ora_conn = cx_Oracle.connect('your_connection_string')
df_ora = pd.read_sql('select * from user_objects', con=ora_conn)    
print 'loaded dataframe from Oracle. # Records: ', len(df_ora)
ora_conn.close()

这是等效的示例MySQLdb

import MySQLdb
mysql_cn= MySQLdb.connect(host='myhost', 
                port=3306,user='myusername', passwd='mypassword', 
                db='information_schema')
df_mysql = pd.read_sql('select * from VIEWS;', con=mysql_cn)    
print 'loaded dataframe from MySQL. records:', len(df_mysql)
mysql_cn.close()

As Wes says, io/sql’s read_sql will do it, once you’ve gotten a database connection using a DBI compatible library. We can look at two short examples using the MySQLdb and cx_Oracle libraries to connect to Oracle and MySQL and query their data dictionaries. Here is the example for cx_Oracle:

import pandas as pd
import cx_Oracle

ora_conn = cx_Oracle.connect('your_connection_string')
df_ora = pd.read_sql('select * from user_objects', con=ora_conn)    
print 'loaded dataframe from Oracle. # Records: ', len(df_ora)
ora_conn.close()

And here is the equivalent example for MySQLdb:

import MySQLdb
mysql_cn= MySQLdb.connect(host='myhost', 
                port=3306,user='myusername', passwd='mypassword', 
                db='information_schema')
df_mysql = pd.read_sql('select * from VIEWS;', con=mysql_cn)    
print 'loaded dataframe from MySQL. records:', len(df_mysql)
mysql_cn.close()

回答 1

对于这个问题的新读者:熊猫在其版本14.0文档中具有以下警告:

警告:一些现有功能或功能别名已被弃用,并将在以后的版本中删除。这包括:tquery,uquery,read_frame,frame_query,write_frame。

和:

警告:不建议使用DBAPI连接对象时支持’mysql’风格。SQLAlchemy引擎(GH6900)将进一步支持MySQL。

这使得这里的许多答案已经过时了。您应该使用sqlalchemy

from sqlalchemy import create_engine
import pandas as pd
engine = create_engine('dialect://user:pass@host:port/schema', echo=False)
f = pd.read_sql_query('SELECT * FROM mytable', engine, index_col = 'ID')

For recent readers of this question: pandas have the following warning in their docs for version 14.0:

Warning: Some of the existing functions or function aliases have been deprecated and will be removed in future versions. This includes: tquery, uquery, read_frame, frame_query, write_frame.

And:

Warning: The support for the ‘mysql’ flavor when using DBAPI connection objects has been deprecated. MySQL will be further supported with SQLAlchemy engines (GH6900).

This makes many of the answers here outdated. You should use sqlalchemy:

from sqlalchemy import create_engine
import pandas as pd
engine = create_engine('dialect://user:pass@host:port/schema', echo=False)
f = pd.read_sql_query('SELECT * FROM mytable', engine, index_col = 'ID')

回答 2

作为记录,这是一个使用sqlite数据库的示例:

import pandas as pd
import sqlite3

with sqlite3.connect("whatever.sqlite") as con:
    sql = "SELECT * FROM table_name"
    df = pd.read_sql_query(sql, con)
    print df.shape

For the record, here is an example using a sqlite database:

import pandas as pd
import sqlite3

with sqlite3.connect("whatever.sqlite") as con:
    sql = "SELECT * FROM table_name"
    df = pd.read_sql_query(sql, con)
    print df.shape

回答 3

我更喜欢使用SQLAlchemy创建查询,然后从中创建一个DataFrame。如果您打算一遍又一遍地混合和匹配内容,SQLAlchemy可以更轻松地通过Python 组合SQL条件。

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Table
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from pandas import DataFrame
import datetime

# We are connecting to an existing service
engine = create_engine('dialect://user:pwd@host:port/db', echo=False)
Session = sessionmaker(bind=engine)
session = Session()
Base = declarative_base()

# And we want to query an existing table
tablename = Table('tablename', 
    Base.metadata, 
    autoload=True, 
    autoload_with=engine, 
    schema='ownername')

# These are the "Where" parameters, but I could as easily 
# create joins and limit results
us = tablename.c.country_code.in_(['US','MX'])
dc = tablename.c.locn_name.like('%DC%')
dt = tablename.c.arr_date >= datetime.date.today() # Give me convenience or...

q = session.query(tablename).\
            filter(us & dc & dt) # That's where the magic happens!!!

def querydb(query):
    """
    Function to execute query and return DataFrame.
    """
    df = DataFrame(query.all());
    df.columns = [x['name'] for x in query.column_descriptions]
    return df

querydb(q)

I prefer to create queries with SQLAlchemy, and then make a DataFrame from it. SQLAlchemy makes it easier to combine SQL conditions Pythonically if you intend to mix and match things over and over.

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Table
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from pandas import DataFrame
import datetime

# We are connecting to an existing service
engine = create_engine('dialect://user:pwd@host:port/db', echo=False)
Session = sessionmaker(bind=engine)
session = Session()
Base = declarative_base()

# And we want to query an existing table
tablename = Table('tablename', 
    Base.metadata, 
    autoload=True, 
    autoload_with=engine, 
    schema='ownername')

# These are the "Where" parameters, but I could as easily 
# create joins and limit results
us = tablename.c.country_code.in_(['US','MX'])
dc = tablename.c.locn_name.like('%DC%')
dt = tablename.c.arr_date >= datetime.date.today() # Give me convenience or...

q = session.query(tablename).\
            filter(us & dc & dt) # That's where the magic happens!!!

def querydb(query):
    """
    Function to execute query and return DataFrame.
    """
    df = DataFrame(query.all());
    df.columns = [x['name'] for x in query.column_descriptions]
    return df

querydb(q)

回答 4

MySQL示例:

import MySQLdb as db
from pandas import DataFrame
from pandas.io.sql import frame_query

database = db.connect('localhost','username','password','database')
data     = frame_query("SELECT * FROM data", database)

MySQL example:

import MySQLdb as db
from pandas import DataFrame
from pandas.io.sql import frame_query

database = db.connect('localhost','username','password','database')
data     = frame_query("SELECT * FROM data", database)

回答 5

相同的语法也适用于使用podbc的Ms SQL服务器。

import pyodbc
import pandas.io.sql as psql

cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=servername;DATABASE=mydb;UID=username;PWD=password') 
cursor = cnxn.cursor()
sql = ("""select * from mytable""")

df = psql.frame_query(sql, cnxn)
cnxn.close()

The same syntax works for Ms SQL server using podbc also.

import pyodbc
import pandas.io.sql as psql

cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=servername;DATABASE=mydb;UID=username;PWD=password') 
cursor = cnxn.cursor()
sql = ("""select * from mytable""")

df = psql.frame_query(sql, cnxn)
cnxn.close()

回答 6

这就是您使用psycopg2驱动程序连接到PostgreSQL的方式(如果您在Debian Linux派生OS上,则使用“ apt-get install python-psycopg2”安装)。

import pandas.io.sql as psql
import psycopg2

conn = psycopg2.connect("dbname='datawarehouse' user='user1' host='localhost' password='uberdba'")

q = """select month_idx, sum(payment) from bi_some_table"""

df3 = psql.frame_query(q, conn)

And this is how you connect to PostgreSQL using psycopg2 driver (install with “apt-get install python-psycopg2” if you’re on Debian Linux derivative OS).

import pandas.io.sql as psql
import psycopg2

conn = psycopg2.connect("dbname='datawarehouse' user='user1' host='localhost' password='uberdba'")

q = """select month_idx, sum(payment) from bi_some_table"""

df3 = psql.frame_query(q, conn)

回答 7

对于Sybase,以下工作原理(使用http://python-sybase.sourceforge.net

import pandas.io.sql as psql
import Sybase

df = psql.frame_query("<Query>", con=Sybase.connect("<dsn>", "<user>", "<pwd>"))

For Sybase the following works (with http://python-sybase.sourceforge.net)

import pandas.io.sql as psql
import Sybase

df = psql.frame_query("<Query>", con=Sybase.connect("<dsn>", "<user>", "<pwd>"))

回答 8

pandas.io.sql.frame_query不推荐使用。使用pandas.read_sql代替。

pandas.io.sql.frame_query is deprecated. Use pandas.read_sql instead.


回答 9

导入模块

import pandas as pd
import oursql

连接

conn=oursql.connect(host="localhost",user="me",passwd="mypassword",db="classicmodels")
sql="Select customerName, city,country from customers order by customerName,country,city"
df_mysql = pd.read_sql(sql,conn)
print df_mysql

可以正常工作,并使用pandas.io.sql frame_works(带有弃用警告)。使用的数据库是mysql教程中的示例数据库。

import the module

import pandas as pd
import oursql

connect

conn=oursql.connect(host="localhost",user="me",passwd="mypassword",db="classicmodels")
sql="Select customerName, city,country from customers order by customerName,country,city"
df_mysql = pd.read_sql(sql,conn)
print df_mysql

That works just fine and using pandas.io.sql frame_works (with the deprecation warning). Database used is the sample database from mysql tutorial.


回答 10

这应该很好。

import MySQLdb as mdb
import pandas as pd
con = mdb.connect(‘127.0.0.1’, root’, password’, database_name’);
with con:
 cur = con.cursor()
 cur.execute(“select random_number_one, random_number_two, random_number_three from randomness.a_random_table”)
 rows = cur.fetchall()
 df = pd.DataFrame( [[ij for ij in i] for i in rows] )
 df.rename(columns={0: Random Number One’, 1: Random Number Two’, 2: Random Number Three’}, inplace=True);
 print(df.head(20))

This should work just fine.

import MySQLdb as mdb
import pandas as pd
con = mdb.connect(‘127.0.0.1’, ‘root’, ‘password’, ‘database_name’);
with con:
 cur = con.cursor()
 cur.execute(“select random_number_one, random_number_two, random_number_three from randomness.a_random_table”)
 rows = cur.fetchall()
 df = pd.DataFrame( [[ij for ij in i] for i in rows] )
 df.rename(columns={0: ‘Random Number One’, 1: ‘Random Number Two’, 2: ‘Random Number Three’}, inplace=True);
 print(df.head(20))

回答 11

这对我帮助从基于python 3.xlambda函数连接到AWS MYSQL(RDS)并加载到pandas DataFrame中

import json
import boto3
import pymysql
import pandas as pd
user = 'username'
password = 'XXXXXXX'
client = boto3.client('rds')
def lambda_handler(event, context):
    conn = pymysql.connect(host='xxx.xxxxus-west-2.rds.amazonaws.com', port=3306, user=user, passwd=password, db='database name', connect_timeout=5)
    df= pd.read_sql('select * from TableName limit 10',con=conn)
    print(df)
    # TODO implement
    #return {
    #    'statusCode': 200,
    #    'df': df
    #}

This helped for me for connecting to AWS MYSQL(RDS) from python 3.x based lambda function and loading into a pandas DataFrame

import json
import boto3
import pymysql
import pandas as pd
user = 'username'
password = 'XXXXXXX'
client = boto3.client('rds')
def lambda_handler(event, context):
    conn = pymysql.connect(host='xxx.xxxxus-west-2.rds.amazonaws.com', port=3306, user=user, passwd=password, db='database name', connect_timeout=5)
    df= pd.read_sql('select * from TableName limit 10',con=conn)
    print(df)
    # TODO implement
    #return {
    #    'statusCode': 200,
    #    'df': df
    #}

回答 12

对于Postgres用户

import psycopg2
import pandas as pd

conn = psycopg2.connect("database='datawarehouse' user='user1' host='localhost' password='uberdba'")

customers = 'select * from customers'

customers_df = pd.read_sql(customers,conn)

customers_df

For Postgres users

import psycopg2
import pandas as pd

conn = psycopg2.connect("database='datawarehouse' user='user1' host='localhost' password='uberdba'")

customers = 'select * from customers'

customers_df = pd.read_sql(customers,conn)

customers_df